Compare commits
5
Commits
ec4679cd17
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
06be075b70 | ||
|
|
8406c77949 | ||
|
|
ded39aa158 | ||
|
|
0cdda43e6c | ||
|
|
f10440b7cc |
@@ -1346,11 +1346,11 @@ webhooker solves this by acting as a durable intermediary:
|
||||
**That guarantee is at-least-once, not exactly-once.** When a send reaches
|
||||
its target but the write recording that outcome fails, the delivery is
|
||||
deliberately left in a recoverable state rather than marked done — losing a
|
||||
delivery is the worse failure — so the pending sweep picks it up about
|
||||
fifteen minutes later, or the next restart does, and the target receives a
|
||||
payload it already got. A send still in flight when the service stops is
|
||||
left the same way: its attempt is not recorded, and the next start sends it
|
||||
again (see Shutdown). webhooker adds no delivery identifier of its own to an
|
||||
delivery is the worse failure — so the sweep picks it up about fifteen
|
||||
minutes later, or the next restart does, and the target receives a payload
|
||||
it already got. A send still in flight when the service stops is left the
|
||||
same way: its attempt is not recorded, and the next start sends it again
|
||||
(see Shutdown). webhooker adds no delivery identifier of its own to an
|
||||
outbound request, so **make your receiver idempotent** against whatever the
|
||||
payload itself carries.
|
||||
|
||||
@@ -1579,15 +1579,20 @@ did not arrive on the URL and counts in neither.
|
||||
A delivery destination for events. Each target defines where and how events
|
||||
should be forwarded.
|
||||
|
||||
| Field | Type | Description |
|
||||
| -------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `id` | UUID | Primary key |
|
||||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||||
| `name` | string | Human-readable name |
|
||||
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
|
||||
| `active` | boolean | Whether deliveries are enabled (default: true) |
|
||||
| `config` | JSON text | Type-specific configuration |
|
||||
| `max_attempts` | integer | How many times an `http` or `slack` target attempts each delivery in all: at least 1, and 1 by default. Ignored by `database` and `log` targets. The web UI labels it Delivery attempts |
|
||||
| Field | Type | Description |
|
||||
| ------------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `id` | UUID | Primary key |
|
||||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||||
| `name` | string | Human-readable name |
|
||||
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
|
||||
| `active` | boolean | Whether deliveries are enabled (default: true) |
|
||||
| `url` | string | Where an `http` or `slack` target sends each event. It is a credential: the target list shows only its scheme and host, and only the target edit form shows it whole |
|
||||
| `headers` | JSON text | An `http` target's extra request headers, a JSON object of name to value |
|
||||
| `timeout_seconds` | integer | An `http` target's per-request timeout; 0 uses the shared 30-second client timeout |
|
||||
| `forward_query` | boolean | Whether an `http` target is sent the query string each event arrived with |
|
||||
| `archive_expiry` | integer | How long a `database` target keeps each archived event, as a Go `time.Duration` (nanoseconds); 0 keeps them forever |
|
||||
| `archive_rotation` | string | How often a `database` target starts a new archive file: `none`, `monthly`, `daily` or `hourly` |
|
||||
| `max_attempts` | integer | How many times an `http` or `slack` target attempts each delivery in all: at least 1, and 1 by default. Ignored by `database` and `log` targets. The web UI labels it Delivery attempts |
|
||||
|
||||
**Relations:** Belongs to Webhook. Has many Deliveries.
|
||||
|
||||
@@ -1598,10 +1603,10 @@ should be forwarded.
|
||||
failed delivery with exponential backoff, up to that many attempts in all,
|
||||
behind a per-target circuit breaker.
|
||||
- **`slack`** — Post the event as a formatted message to a Slack-compatible
|
||||
incoming webhook URL (`webhookUrl` in `config`). It is built on the same HTTP
|
||||
core as `http` and honours `max_attempts` identically, circuit breaker
|
||||
included. See the Slack target section under "Per-Webhook Event Databases" for
|
||||
the message format.
|
||||
incoming webhook URL, its `url`. It is built on the same HTTP core as `http`
|
||||
and honours `max_attempts` identically, circuit breaker included. See the
|
||||
Slack target section under "Per-Webhook Event Databases" for the message
|
||||
format.
|
||||
- **`database`** — Archive the full event as a row into the target's own archive
|
||||
database (`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for
|
||||
long-term retention, with an optional creation-validated expiry (default: keep
|
||||
@@ -1617,29 +1622,28 @@ should be forwarded.
|
||||
- **`log`** — Write the event to the application log (stdout). Useful for
|
||||
debugging.
|
||||
|
||||
The `config` field stores type-specific configuration as JSON (e.g., destination
|
||||
URL, custom headers, timeout settings).
|
||||
Each type uses only its own settings, and the ones it does not use stay empty:
|
||||
`http` uses `url`, `headers`, `timeout_seconds` and `forward_query`; `slack`
|
||||
uses `url`; `database` uses `archive_expiry` and `archive_rotation`; `log` uses
|
||||
none. The add target form, the new webhook form and the target edit form check
|
||||
every setting before it is saved, and each setting is a column of its own, so
|
||||
nothing that reads a target parses its settings or has to handle one it cannot
|
||||
read.
|
||||
|
||||
**`http` target configuration:**
|
||||
An `http` target POSTs the event to its `url`. Its `headers` are applied last,
|
||||
so they win over the event's own forwarded headers.
|
||||
|
||||
| Key | Type | Description |
|
||||
| -------------- | ------------- | ----------------------------------------------------------------------------------------- |
|
||||
| `url` | string | Destination the event is POSTed to |
|
||||
| `headers` | object | Extra request headers, applied last so they win over the event's own forwarded headers |
|
||||
| `timeout` | integer (sec) | Per-target request timeout; unset (or 0) uses the shared 30-second client timeout |
|
||||
| `forwardQuery` | boolean | Pass the query string each event arrived with on to the target; unset (or false) does not |
|
||||
|
||||
`forwardQuery` is off by default, and the target URL is then sent exactly as
|
||||
`forward_query` is off by default, and the target URL is then sent exactly as
|
||||
configured. On, each delivery appends the event's query string to the target
|
||||
URL, joined with `&` when the URL already has a query string of its own; a
|
||||
replayed delivery and a resubmitted event's deliveries do the same. Both target
|
||||
forms offer it as "Pass the query string on to this target", and the target list
|
||||
shows it when it is on.
|
||||
|
||||
`timeout` is capped at **300 seconds**, and the form rejects anything above it
|
||||
rather than substituting the cap. A delivery attempt holds one of the bounded
|
||||
pool's workers for its whole duration, so an unbounded timeout would let a
|
||||
single unresponsive destination stall the queue.
|
||||
`timeout_seconds` is capped at **300 seconds**, and the form rejects anything
|
||||
above it rather than substituting the cap. A delivery attempt holds one of the
|
||||
bounded pool's workers for its whole duration, so an unbounded timeout would let
|
||||
a single unresponsive destination stall the queue.
|
||||
|
||||
`headers` rejects the names the delivery path or `net/http` writes regardless of
|
||||
what is configured: `Host`, `Content-Length`, `Transfer-Encoding`, `Connection`,
|
||||
@@ -1713,7 +1717,7 @@ The event log and the event's own page show the query string with the rest of
|
||||
the request. The event log leaves out one larger than 32 KiB, as it does request
|
||||
headers, and links to the event's page, which shows it whole. The `database` and
|
||||
`log` targets carry it with the rest of the event. An `http` target receives it
|
||||
only when its `forwardQuery` setting is on.
|
||||
only when its `forward_query` setting is on.
|
||||
|
||||
#### Delivery
|
||||
|
||||
@@ -1865,15 +1869,15 @@ These indexes on the per-webhook event databases are created by
|
||||
`internal/db/migrations/events/001_schema.sql`, and declared in the model tags
|
||||
to match:
|
||||
|
||||
| Table | Columns | Serves |
|
||||
| ------------------ | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, the webhook page's statistics and target list and the webhook list, which count each target's deliveries by status and when they finished, and the event log, which lists and counts the events with a failed delivery or one pending or retrying |
|
||||
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
|
||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
||||
| `events` | `entrypoint_id`, `deleted_at`, `resubmitted_from_id`, `created_at` | The webhook page's entrypoint list, which counts the events that arrived on each entrypoint's URL within the retention period |
|
||||
| `events` | `created_at` | Retention, which selects expired events by age |
|
||||
| Table | Columns | Serves |
|
||||
| ------------------ | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Delivery recovery at start and every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, the webhook page's statistics and target list and the webhook list, which count each target's deliveries by status and when they finished, and the event log, which lists and counts the events with a failed delivery or one pending or retrying |
|
||||
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
|
||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
||||
| `events` | `entrypoint_id`, `deleted_at`, `resubmitted_from_id`, `created_at` | The webhook page's entrypoint list, which counts the events that arrived on each entrypoint's URL within the retention period |
|
||||
| `events` | `created_at` | Retention, which selects expired events by age |
|
||||
|
||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention leaves
|
||||
it out. SQLite keeps no statistics on these tables, and without them it rates
|
||||
@@ -1992,11 +1996,10 @@ target UUID keeps the file name unique. A webhook named `Orders (EU)` with a
|
||||
target named `Long-term archive` archives into
|
||||
`archive-orders-eu-long-term-archive-{target_uuid}.db`.
|
||||
|
||||
An optional `rotation` in the target's config JSON (e.g. `{"rotation":"daily"}`)
|
||||
is `none`, the default, which keeps the one file, or `monthly`, `daily` or
|
||||
`hourly`. A target that rotates writes each event to a file named for the period
|
||||
of the event's receive time, in UTC, put before the `.db`:
|
||||
`archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
|
||||
A target's `archive_rotation` is `none`, the default, which keeps the one file,
|
||||
or `monthly`, `daily` or `hourly`. A target that rotates writes each event to a
|
||||
file named for the period of the event's receive time, in UTC, put before the
|
||||
`.db`: `archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
|
||||
`…-2026-10-01.db` daily and `…-2026-10-01-19.db` hourly. Each file holds exactly
|
||||
its period's events, and the first event of a new period starts the next file,
|
||||
so a finished period's file can be moved away like any archive. A changed
|
||||
@@ -2031,12 +2034,11 @@ archiving without stopping the service; a moved or removed archive file is
|
||||
recreated automatically on the next write, from the SQL files in
|
||||
`internal/db/migrations/archive/`. A zero-length archive file is written to as a
|
||||
new archive: SQLite opens it as an empty database, so it holds nothing to lose.
|
||||
An optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is
|
||||
validated when the target is created — the default (unset or the literal
|
||||
`never`) keeps rows forever — and rows older than the expiry are pruned each
|
||||
time the archive is (re)opened. An archive write failure is never silent
|
||||
success: the delivery records a failed attempt with the error and is marked
|
||||
failed.
|
||||
A target's `archive_expiry` is checked by the form that saves it — the default,
|
||||
`never` on the forms and 0 in the column, keeps rows forever — and rows older
|
||||
than the expiry are pruned each time the archive is (re)opened. An archive write
|
||||
failure is never silent success: the delivery records a failed attempt with the
|
||||
error and is marked failed.
|
||||
|
||||
Because reopens only happen on writes, an archive whose target has stopped
|
||||
receiving events would never be pruned. A background **archive sweeper** closes
|
||||
@@ -2047,8 +2049,7 @@ traffic. The sweep never creates an archive — a target whose archive file does
|
||||
not yet exist is skipped, not initialised — it takes the same per-target lock
|
||||
the write path uses, so it can never interleave with a write, and it leaves the
|
||||
archive closed afterwards so the move-the-file-away workflow keeps working.
|
||||
Archives with no expiry, or the expiry `never`, are not touched by the sweep at
|
||||
all.
|
||||
Archives with the expiry `never` are not touched by the sweep at all.
|
||||
|
||||
For a target with several files, the write path prunes only the file it writes
|
||||
to, and the sweep prunes every one of them, taking the target's lock for one
|
||||
@@ -2100,15 +2101,15 @@ writes go on meanwhile, since under WAL a reader never blocks a writer. Until
|
||||
the open file is closed its `-wal` cannot be checkpointed past what the download
|
||||
reads, so a long download lets that `-wal` grow.
|
||||
|
||||
Deleting a webhook releases its archives: the delivery engine's cached archive
|
||||
writers are dropped and their file handles closed, so nothing lingers after the
|
||||
webhook is gone. The archive **files themselves are deliberately left on disk**.
|
||||
Unlike the event database — per-webhook working storage that is hard-deleted
|
||||
with the webhook — an archive is long-term storage an operator may still want to
|
||||
keep or move away for offline retention, and destroying it as a side effect of
|
||||
deleting a webhook would be unrecoverable. Removing an `archive-….db` is the
|
||||
operator's call. Deleting a `database` target releases its writer the same way,
|
||||
and for the same reason leaves its file alone.
|
||||
Deleting a webhook releases its archives: its cached archive writers are dropped
|
||||
and their file handles closed, so nothing lingers after the webhook is gone. The
|
||||
archive **files themselves are deliberately left on disk**. Unlike the event
|
||||
database — per-webhook working storage that is hard-deleted with the webhook —
|
||||
an archive is long-term storage an operator may still want to keep or move away
|
||||
for offline retention, and destroying it as a side effect of deleting a webhook
|
||||
would be unrecoverable. Removing an `archive-….db` is the operator's call.
|
||||
Deleting a `database` target releases its writer the same way, and for the same
|
||||
reason leaves its file alone.
|
||||
|
||||
The **Slack target type** sends webhook events as formatted messages to any
|
||||
Slack-compatible incoming webhook URL (works with Slack, Mattermost, and other
|
||||
@@ -2116,9 +2117,8 @@ compatible services). Each message includes event metadata (HTTP method, content
|
||||
type, timestamp, body size) and the payload pretty-printed in a code block. JSON
|
||||
payloads are automatically formatted with indentation for readability; non-JSON
|
||||
payloads are shown as raw text. Large payloads are truncated to keep messages
|
||||
reasonable. Config stores `webhookUrl` — the Slack/Mattermost incoming webhook
|
||||
endpoint. That is the JSON key; the error text for a missing one reads
|
||||
`webhook_url is required`, which is the message, not the key.
|
||||
reasonable. The target's `url` is the Slack/Mattermost incoming webhook
|
||||
endpoint.
|
||||
|
||||
The database uses the
|
||||
[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) driver.
|
||||
@@ -2153,12 +2153,8 @@ External Service
|
||||
1. Look up Entrypoint by UUID — 404 if unknown,
|
||||
410 if inactive
|
||||
2. Read the body under the 1 MB cap
|
||||
3. Capture full request as Event
|
||||
4. Create Delivery records for each active Target
|
||||
5. Build self-contained delivery.Task structs
|
||||
(target config + event data inline for
|
||||
bodies < 16 KiB)
|
||||
6. Notify Engine via channel (no DB read needed)
|
||||
3. Hand the full request, as an Event, to
|
||||
Engine.StoreEvent
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
@@ -2167,6 +2163,13 @@ External Service
|
||||
│ (worker │
|
||||
│ pool) │
|
||||
└──────┬───────┘
|
||||
4. Store the Event, a pending Delivery for each
|
||||
active Target, and the totals, in one
|
||||
transaction
|
||||
5. Build a delivery.Task for each Delivery
|
||||
(target settings + event data, the body
|
||||
inline when it is under 16 KiB)
|
||||
6. Queue the tasks on the delivery channel
|
||||
│
|
||||
┌── bounded worker pool (N workers) ──┐
|
||||
▼ ▼ ▼
|
||||
@@ -2180,6 +2183,12 @@ External Service
|
||||
└────────────┘
|
||||
```
|
||||
|
||||
A **Resubmit** stores its copy of an event through the same `Engine.StoreEvent`,
|
||||
and a **Replay** adds one delivery of a stored event through
|
||||
`Engine.CreateReplay`, which writes the delivery and its target's totals in one
|
||||
transaction and queues it. Those two methods are the only code that creates a
|
||||
delivery; the handlers read and check the request, then make the one call.
|
||||
|
||||
### Bounded Worker Pool
|
||||
|
||||
The delivery engine uses a **fixed-size worker pool** (default: 10 workers) to
|
||||
@@ -2189,8 +2198,9 @@ preventing goroutine explosions regardless of queue depth.
|
||||
**Architecture:**
|
||||
|
||||
- **Channels as queues:** Two buffered channels serve as bounded queues: a
|
||||
delivery channel (new tasks from the webhook handler) and a retry channel
|
||||
(tasks from backoff timers). Both are buffered to 10,000.
|
||||
delivery channel (the tasks of newly stored deliveries and of the `pending`
|
||||
deliveries the recovery routine hands back) and a retry channel (tasks from
|
||||
backoff timers). Both are buffered to 10,000.
|
||||
- **Fan-out via channel, not goroutines:** When an event arrives with multiple
|
||||
targets, each `delivery.Task` is sent to the delivery channel. Workers pick
|
||||
them up and process them — no goroutine-per-target.
|
||||
@@ -2199,7 +2209,7 @@ preventing goroutine explosions regardless of queue depth.
|
||||
Workers are the ONLY goroutines doing actual HTTP delivery.
|
||||
- **Retry backpressure with DB fallback:** When a retry timer fires and the
|
||||
retry channel is full, the timer is dropped — the delivery stays in `retrying`
|
||||
status in the database. A periodic sweep (every 60s) scans for these
|
||||
status in the database. The sweep (every 60s, see Recovery below) finds these
|
||||
"orphaned" retries and re-queues them. No blocked goroutines, no unbounded
|
||||
timer chains.
|
||||
- **Bounded concurrency:** At most N deliveries (N = number of workers) are
|
||||
@@ -2226,24 +2236,36 @@ This means:
|
||||
that did not finish are still running — an unclean shutdown is reported rather
|
||||
than hidden.
|
||||
|
||||
**Recovery paths:**
|
||||
**Recovery:** one routine finds the deliveries the engine is not working on and
|
||||
hands them back. It runs once when the engine starts, and then every 60 seconds;
|
||||
that periodic run is the sweep. For every webhook it reads the `pending` and
|
||||
`retrying` deliveries. One that already has a successful attempt is marked
|
||||
`delivered` and not sent again. A `retrying` delivery gets a backoff timer for
|
||||
what is left of its backoff, and is queued at once if nothing is left. A
|
||||
`pending` delivery is queued on the delivery channel. A delivery the engine has
|
||||
queued, is attempting, or holds a backoff timer for is left alone, at start as
|
||||
on every sweep. This is what catches "orphaned" retries, whose in-memory timer
|
||||
was dropped because the retry channel was full, and deliveries a stop or a crash
|
||||
cut off: the database is the durable fallback that ensures no delivery is
|
||||
permanently lost, even under extreme backpressure.
|
||||
|
||||
1. **Startup recovery:** When the engine starts, it scans all per-webhook
|
||||
databases for `pending` and `retrying` deliveries. Pending deliveries are
|
||||
sent to the delivery channel; retrying deliveries get backoff timers
|
||||
scheduled.
|
||||
2. **Periodic retry sweep (DB-mediated fallback):** Every 60 seconds the engine
|
||||
scans for `retrying` deliveries whose backoff period has elapsed. This
|
||||
catches "orphaned" retries — ones whose in-memory timer was dropped because
|
||||
the retry channel was full. The database is the durable fallback that ensures
|
||||
no retry is permanently lost, even under extreme backpressure.
|
||||
The run at start and the sweep differ in two ways:
|
||||
|
||||
- At start every `pending` delivery is taken, however recent, so one a stop cut
|
||||
off moments ago is sent again at once (see Shutdown). The sweep takes only
|
||||
`pending` deliveries untouched for 15 minutes, at most 500 per webhook each
|
||||
time, so a delivery whose outcome the database will not record is sent again
|
||||
every 15 minutes rather than every minute.
|
||||
- At start every webhook's database is opened, so that a missing one is reported
|
||||
then (see Per-Webhook Event Databases). The sweep skips a webhook whose
|
||||
database file is missing rather than create it.
|
||||
|
||||
**Changing a target's type does not migrate in-flight deliveries.** Only `http`
|
||||
and `slack` targets own durable retries; `database` and `log` targets are
|
||||
fire-and-forget and never produce a `retrying` delivery. If a target's `type` is
|
||||
edited from a retrying type to a non-retrying (or unknown) one while one of its
|
||||
deliveries is still `retrying`, both recovery paths above terminally mark that
|
||||
delivery `failed` and record a `DeliveryResult` naming the current target type
|
||||
deliveries is still `retrying`, the recovery routine above terminally marks that
|
||||
delivery `failed` and records a `DeliveryResult` naming the current target type
|
||||
as the reason, logging it at warn level. The delivery is not re-dispatched under
|
||||
the new type — the operator never asked for that delivery — and the event itself
|
||||
remains stored in the per-webhook event database. Nothing is lost: the delivery
|
||||
@@ -2256,8 +2278,8 @@ re-inject it to every currently active target.
|
||||
`http` and `slack` targets with `max_attempts` > 1 are protected by a
|
||||
**per-target circuit breaker** that prevents hammering a down target with
|
||||
repeated failed delivery attempts. The circuit breaker is in-memory only and
|
||||
resets on restart (which is fine — startup recovery rescans the database
|
||||
anyway).
|
||||
resets on restart (which is fine — the recovery routine rescans the database at
|
||||
start anyway).
|
||||
|
||||
**States:**
|
||||
|
||||
@@ -2997,6 +3019,12 @@ webhooker/
|
||||
├── cmd/webhooker/
|
||||
│ └── main.go # Entry point: subcommand dispatch; no args locks DATA_DIR and wires fx
|
||||
├── internal/
|
||||
│ ├── archive/
|
||||
│ │ ├── archive.go # The archive writer of each database target, eviction, rename, and the rename lock
|
||||
│ │ ├── writer.go # Archive file lifecycle and pruning
|
||||
│ │ ├── rotation.go # Archive rotation and file names
|
||||
│ │ ├── download.go # Archive download as gzipped JSON
|
||||
│ │ └── sweeper.go # Periodic pruning of idle archives
|
||||
│ ├── banner/
|
||||
│ │ └── banner.go # Ruled block for the one credential shown in the clear
|
||||
│ ├── resetpw/
|
||||
@@ -3036,24 +3064,21 @@ webhooker/
|
||||
│ │ └── logfield.go # Encoded-byte budget for client-supplied log values
|
||||
│ ├── delivery/
|
||||
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
|
||||
│ │ ├── store.go # Storing an event with its deliveries, and a replay, then queueing them
|
||||
│ │ ├── circuit_breaker.go # Per-target circuit breaker for http/slack targets with retries
|
||||
│ │ ├── target.go # Target interface, Task, Scheduler
|
||||
│ │ ├── target_http.go # HTTP target (retries, circuit breaker)
|
||||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||||
│ │ ├── target_database.go # Database archive target
|
||||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||||
│ │ ├── target_database_rotation.go # Archive rotation and file names
|
||||
│ │ ├── target_database_export.go # Archive download as gzipped JSON
|
||||
│ │ ├── target_database.go # Database archive target, writing through internal/archive
|
||||
│ │ ├── target_log.go # Log target (stdout)
|
||||
│ │ ├── target_config_view.go # Masked target config for templates
|
||||
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
||||
│ │ ├── target_config_view.go # Masked target settings for templates
|
||||
│ │ ├── queue_depth.go # Periodic sampler behind the queue-depth gauges
|
||||
│ │ ├── url_mask.go # Strips credentials from *url.Error
|
||||
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
|
||||
│ ├── handlers/
|
||||
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
|
||||
│ │ ├── auth.go # Login, logout handlers
|
||||
│ │ ├── delivery_replay.go # Per-delivery replay: new delivery, current target config
|
||||
│ │ ├── delivery_replay.go # Per-delivery replay: new delivery, current target settings
|
||||
│ │ ├── event_resubmit.go # Event resubmit: new event, all currently active targets
|
||||
│ │ ├── entrypoint_view.go # Masked entrypoint view for templates
|
||||
│ │ ├── event_log_view.go # Event log projection, byte-capped in SQL
|
||||
@@ -3068,7 +3093,7 @@ webhooker/
|
||||
│ │ ├── webhook_delete.go # Webhook delete, event database and archive writer removal
|
||||
│ │ ├── event_log.go # Event log page: loaders, filters, delivery views
|
||||
│ │ ├── entrypoint.go # Entrypoint create, edit, delete and toggle
|
||||
│ │ ├── target_create.go # Target create, per-type config builders
|
||||
│ │ ├── target_create.go # Target create, each type's settings from the form
|
||||
│ │ ├── target_delete.go # Target delete
|
||||
│ │ ├── target_toggle.go # Target toggle
|
||||
│ │ ├── shared.go # Helpers shared by several handlers
|
||||
@@ -3136,21 +3161,26 @@ Components are wired via Uber fx in this order:
|
||||
10. `metrics.NewRegistry` — The registry `/metrics` serves
|
||||
11. `metrics.New` — The delivery collectors, registered on that registry
|
||||
12. `middleware.New` — HTTP middleware
|
||||
13. `delivery.New` — Event-driven delivery engine
|
||||
14. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||||
15. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||||
16. `delivery.Engine` → `delivery.Archives` — interface bridge so deleting or
|
||||
13. `delivery.New` — Event-driven delivery engine, which also stores each event
|
||||
with its deliveries and creates replay deliveries
|
||||
14. `archive.NewWriters` — The database targets' archive writers, and the lock
|
||||
every rename of their files runs under
|
||||
15. `archive.NewSweeper` — Periodic pruning of idle archives
|
||||
16. `archive.Writers` → `archive.Archives` — interface bridge so deleting or
|
||||
renaming a webhook or target reaches its archive files
|
||||
17. `server.New` — HTTP server and router
|
||||
17. `delivery.Engine` → `delivery.CircuitBreakers` — interface bridge so the
|
||||
pages can show a target whose deliveries are paused
|
||||
18. `server.New` — HTTP server and router
|
||||
|
||||
The server starts via
|
||||
`fx.Invoke(func(*server.Server, *delivery.Engine, *database.RetentionReaper, *delivery.ArchiveSweeper) {})`,
|
||||
which triggers the fx lifecycle hooks in dependency order. The
|
||||
`delivery.Notifier` interface allows the webhook handler to send self-contained
|
||||
`delivery.Task` slices to the engine without a direct package dependency. Each
|
||||
task carries all target config and event data inline (for bodies under 16 KiB,
|
||||
`delivery.MaxInlineBodySize`), so the engine can deliver without reading from
|
||||
any database — it only writes to record results.
|
||||
`fx.Invoke(func(*server.Server, *delivery.Engine, *database.RetentionReaper, *archive.Sweeper) {})`,
|
||||
which triggers the fx lifecycle hooks in dependency order. The handlers take
|
||||
`*delivery.Engine` itself: the receiver and resubmit handlers store an event
|
||||
through `StoreEvent`, and the replay handler adds a delivery through
|
||||
`CreateReplay`. Each task the engine queues carries the target's settings and
|
||||
the event's data, and the body too when it is under 16 KiB
|
||||
(`delivery.MaxInlineBodySize`); a larger body is read from the event row when
|
||||
the attempt runs.
|
||||
|
||||
### Middleware Stack
|
||||
|
||||
@@ -3313,7 +3343,7 @@ order under a **5 second budget** (`fx.StopTimeout` in `cmd/webhooker/main.go`).
|
||||
That budget covers the whole sequence, not each hook. The order, read off the fx
|
||||
stop-hook log:
|
||||
|
||||
1. `ArchiveSweeper`
|
||||
1. `archive.Sweeper`
|
||||
2. `RetentionReaper`
|
||||
3. `server` — the HTTP drain, bounded by `server.ShutdownTimeout` (**3
|
||||
seconds**) and by what the hooks before it left, then a Sentry flush if
|
||||
@@ -3332,9 +3362,9 @@ normal operation.
|
||||
|
||||
The delivery engine cancels a delivery request in flight. That attempt is not
|
||||
recorded or counted in the metrics, and the delivery keeps its `pending` or
|
||||
`retrying` status, so startup recovery sends it again on the next start (see
|
||||
Recovery paths under Bounded Worker Pool). If the receiver already had it, it
|
||||
receives it twice: the at-least-once duplicate described under Rationale.
|
||||
`retrying` status, so recovery at the next start sends it again (see Recovery
|
||||
under Bounded Worker Pool). If the receiver already had it, it receives it
|
||||
twice: the at-least-once duplicate described under Rationale.
|
||||
|
||||
The HTTP drain budget is deliberately **shorter** than the sequence budget. Were
|
||||
the two equal, a drain that used its whole budget would exhaust the sequence
|
||||
@@ -3356,7 +3386,7 @@ skipped when that leaves too little to be worth attempting — so a full-length
|
||||
drain means Sentry events are dropped rather than the database close being
|
||||
skipped.
|
||||
|
||||
This does not make the database close unconditional. A slow `ArchiveSweeper` or
|
||||
This does not make the database close unconditional. A slow `archive.Sweeper` or
|
||||
`RetentionReaper` is enough to cut the shutdown short, not only one that
|
||||
consumes the whole budget: what they spend comes out of the drain first, so
|
||||
after 2 seconds of theirs a request still in flight gets 1 second to finish, and
|
||||
|
||||
+9
-10
@@ -10,6 +10,7 @@ import (
|
||||
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxevent"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
@@ -49,7 +50,7 @@ import (
|
||||
// before the server may already have spent.
|
||||
//
|
||||
// This does not make the database close unconditional: the
|
||||
// ArchiveSweeper and RetentionReaper hooks run before the server.
|
||||
// archive.Sweeper and RetentionReaper hooks run before the server.
|
||||
// What they spend comes out of the drain first, but past 3s it comes
|
||||
// out of the reserve, and they can consume the whole budget.
|
||||
const stopTimeout = 5 * time.Second
|
||||
@@ -188,8 +189,8 @@ func newApp() *fx.App {
|
||||
fx.Provide(
|
||||
globals.New,
|
||||
logger.New,
|
||||
// The plain logger the session, the middleware and the
|
||||
// webhook database manager take.
|
||||
// The plain logger the session, the middleware, the
|
||||
// webhook database manager and the archive writers take.
|
||||
func(l *logger.Logger) *slog.Logger { return l.Get() },
|
||||
config.New,
|
||||
database.New,
|
||||
@@ -208,14 +209,12 @@ func newApp() *fx.App {
|
||||
// disagree about a destination.
|
||||
delivery.NewGuard,
|
||||
delivery.New,
|
||||
delivery.NewArchiveSweeper,
|
||||
// Wire *delivery.Engine as delivery.Notifier so the
|
||||
// webhook handler can notify the engine of new deliveries.
|
||||
func(e *delivery.Engine) delivery.Notifier { return e },
|
||||
// Wire *delivery.Engine as delivery.Archives so deleting
|
||||
archive.NewWriters,
|
||||
archive.NewSweeper,
|
||||
// Wire *archive.Writers as archive.Archives so deleting
|
||||
// or renaming a webhook or target reaches its archive
|
||||
// files.
|
||||
func(e *delivery.Engine) delivery.Archives { return e },
|
||||
func(w *archive.Writers) archive.Archives { return w },
|
||||
// Wire *delivery.Engine as delivery.CircuitBreakers so
|
||||
// the pages can show a target whose deliveries are
|
||||
// paused.
|
||||
@@ -227,7 +226,7 @@ func newApp() *fx.App {
|
||||
*server.Server,
|
||||
*delivery.Engine,
|
||||
*database.RetentionReaper,
|
||||
*delivery.ArchiveSweeper,
|
||||
*archive.Sweeper,
|
||||
) {
|
||||
},
|
||||
),
|
||||
|
||||
@@ -252,7 +252,7 @@ const tailHeadroom = 2 * time.Second
|
||||
// can produce, since a shorter drain leaves the flush more room and
|
||||
// the worst case is not necessarily at either extreme.
|
||||
//
|
||||
// Nor does the hook start on a full budget: the ArchiveSweeper and
|
||||
// Nor does the hook start on a full budget: the archive.Sweeper and
|
||||
// RetentionReaper hooks run before it, and whatever they spent is
|
||||
// gone. The outer sweep walks every amount they can spend. Once they
|
||||
// have eaten into the headroom themselves, the hook must spend
|
||||
|
||||
@@ -0,0 +1,446 @@
|
||||
// Package archive manages the SQLite archive files the database target
|
||||
// writes: how they are named and rotated, the writers that open, write,
|
||||
// close and reopen them, pruning the rows past a target's expiry,
|
||||
// renaming a target's files and moving them back when that fails, the
|
||||
// gzipped JSON export, the idle sweep, and the locks that keep a rename
|
||||
// apart from archive writes, from other edits and from exports.
|
||||
package archive
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// archiveNameMaxLen is how many characters of a webhook or target
|
||||
// name an archive file name keeps.
|
||||
const archiveNameMaxLen = 40
|
||||
|
||||
// Archives is how the handlers keep the database targets' archive
|
||||
// files in step with the configuration. Deleting a webhook or a
|
||||
// target releases the cached archive writers, whose open file
|
||||
// handles would otherwise outlive them; renaming one renames the
|
||||
// archive files, which are named for the webhook and the target
|
||||
// (see FileName). RenameLock returns the lock every rename runs under
|
||||
// (see Writers.RenameLock).
|
||||
//
|
||||
// It is an interface so that the handlers' tests can stand in for
|
||||
// Writers.
|
||||
//
|
||||
// Neither eviction deletes an archive file. Both are idempotent
|
||||
// and are no-ops for a webhook or target with no writer.
|
||||
type Archives interface {
|
||||
RenameLock() *sync.Mutex
|
||||
EvictWebhook(webhookID string)
|
||||
EvictTarget(targetID string)
|
||||
Rename(targetID, webhookName, targetName string) error
|
||||
}
|
||||
|
||||
// WritersParams holds the fx dependencies for Writers.
|
||||
type WritersParams struct {
|
||||
fx.In
|
||||
|
||||
Database *database.Database
|
||||
DBManager *database.WebhookDBManager
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// Writers holds the archive writer of each database target, and the
|
||||
// lock every rename of archive files runs under. The database target
|
||||
// writes an event through it, the handlers rename and evict through
|
||||
// it, the archive sweeper prunes through it, and the delivery engine
|
||||
// evicts every writer when it stops. See archiveWriter for the
|
||||
// close/reopen, auto-recreate, and expiry semantics.
|
||||
type Writers struct {
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
log *slog.Logger
|
||||
|
||||
// writers holds one archive writer per database target, keyed
|
||||
// by target ID.
|
||||
mu sync.Mutex
|
||||
writers map[string]*archiveWriter
|
||||
|
||||
// renameMu makes the webhook edit, the target edit and target
|
||||
// creation run one at a time, each held from loading the stored
|
||||
// names through the archive rename, the save and any move back.
|
||||
// Interleaved, one could rename an archive between another's
|
||||
// rename and save, leaving the file named for one edit and the
|
||||
// stored names from the other. An archive download holds it while
|
||||
// it reads the stored names and lists the files they give, and
|
||||
// again for each file while it finds the file under the names
|
||||
// stored then and opens it. RenameLock returns it.
|
||||
renameMu sync.Mutex
|
||||
}
|
||||
|
||||
// NewWriters builds the archive writers. It opens no archive file:
|
||||
// each target's writer is made on its first use.
|
||||
func NewWriters(params WritersParams) *Writers {
|
||||
return &Writers{
|
||||
db: params.Database,
|
||||
dbMgr: params.DBManager,
|
||||
log: params.Logger,
|
||||
}
|
||||
}
|
||||
|
||||
// RenameLock implements Archives. It returns the lock every rename of
|
||||
// archive files runs under: see renameMu for who holds it and for how
|
||||
// long. Archive writes and Rename itself do not take it.
|
||||
func (ws *Writers) RenameLock() *sync.Mutex {
|
||||
return &ws.renameMu
|
||||
}
|
||||
|
||||
// FileName returns the file name of a database target's
|
||||
// archive: archive-WEBHOOKNAME-TARGETNAME-TARGETID.db, with both
|
||||
// names passed through archiveNamePart. The target ID keeps the
|
||||
// name unique when two targets' names come out the same.
|
||||
func FileName(webhookName, targetName, targetID string) string {
|
||||
return "archive-" + archiveNamePart(webhookName) + "-" +
|
||||
archiveNamePart(targetName) + "-" + targetID + ".db"
|
||||
}
|
||||
|
||||
// archiveNamePart makes a webhook or target name safe to put in a
|
||||
// file name. It is lowercased; ASCII letters and digits are kept,
|
||||
// every other run of characters becomes a single "-", and no "-" is
|
||||
// left at either end. It is cut to archiveNameMaxLen characters, and
|
||||
// a name with nothing left is "unnamed".
|
||||
func archiveNamePart(name string) string {
|
||||
var b strings.Builder
|
||||
|
||||
dash := false
|
||||
|
||||
for _, r := range strings.ToLower(name) {
|
||||
if (r < 'a' || r > 'z') && (r < '0' || r > '9') {
|
||||
dash = b.Len() > 0
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if dash {
|
||||
b.WriteByte('-')
|
||||
|
||||
dash = false
|
||||
}
|
||||
|
||||
b.WriteRune(r)
|
||||
}
|
||||
|
||||
part := b.String()
|
||||
if len(part) > archiveNameMaxLen {
|
||||
part = strings.TrimRight(part[:archiveNameMaxLen], "-")
|
||||
}
|
||||
|
||||
if part == "" {
|
||||
return "unnamed"
|
||||
}
|
||||
|
||||
return part
|
||||
}
|
||||
|
||||
// Write writes the full event of a delivery to a database target as
|
||||
// a row into the target's archive database, honouring the target's
|
||||
// ArchiveExpiry and ArchiveRotation. With rotation, the event goes to
|
||||
// the file for the period of its receive time.
|
||||
func (ws *Writers) Write(d *database.Delivery) error {
|
||||
webhookID := d.Event.WebhookID
|
||||
if webhookID == "" {
|
||||
return errArchiveMissingWebhookID
|
||||
}
|
||||
|
||||
// An event whose stored row was gone before its delivery ran has
|
||||
// no receive time (see delivery's Engine.hydrateEvent), and goes
|
||||
// to the file for now.
|
||||
receivedAt := d.Event.CreatedAt
|
||||
if receivedAt.IsZero() {
|
||||
receivedAt = time.Now()
|
||||
}
|
||||
|
||||
w, err := ws.writerFor(d.TargetID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
row := archivedEvent{
|
||||
EventID: d.Event.ID,
|
||||
WebhookID: webhookID,
|
||||
EntrypointID: d.Event.EntrypointID,
|
||||
Method: d.Event.Method,
|
||||
RawQuery: d.Event.RawQuery,
|
||||
Headers: d.Event.Headers,
|
||||
Body: d.Event.Body,
|
||||
ContentType: d.Event.ContentType,
|
||||
}
|
||||
|
||||
return w.write(
|
||||
row, d.Target.ArchiveExpiry,
|
||||
archivePeriod(d.Target.ArchiveRotation, receivedAt),
|
||||
)
|
||||
}
|
||||
|
||||
// Rename implements Archives. It renames every one of a database
|
||||
// target's archive files to FileName(webhookName, targetName,
|
||||
// targetID), each keeping the period in its name. It goes through the
|
||||
// target's writer, so the move holds the lock that the target's
|
||||
// archive writes and the idle sweep take, and later writes use the
|
||||
// new name. It never replaces a file: if one already has a new name,
|
||||
// the error is ErrNameTaken.
|
||||
//
|
||||
// The writer is created if there is none, and it stays cached. The
|
||||
// handlers rename before they save the new name, so until the save
|
||||
// the main database still has the old one; a delivery in that window
|
||||
// must find this writer rather than build one from the old name.
|
||||
func (ws *Writers) Rename(
|
||||
targetID, webhookName, targetName string,
|
||||
) error {
|
||||
w, err := ws.writerFor(targetID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return w.rename(FileName(webhookName, targetName, targetID))
|
||||
}
|
||||
|
||||
// EvictTarget implements Archives. It drops a database target's
|
||||
// archive writer from the registry and closes its handle, so a
|
||||
// deleted target does not leave a writer (and an open archive handle
|
||||
// within its debounce window) alive for the process lifetime.
|
||||
//
|
||||
// The map entry is removed under the registry lock, which is
|
||||
// then released before the handle is closed under the writer's
|
||||
// own lock: that ordering keeps the registry available to other
|
||||
// targets while an in-flight write on this one drains, and
|
||||
// closing under the writer's lock means eviction can never race
|
||||
// a write.
|
||||
//
|
||||
// Eviction is idempotent and silent for a target with no writer,
|
||||
// which is the common case: only a database target that has
|
||||
// received an event or been renamed has one. It never deletes the
|
||||
// archive file.
|
||||
func (ws *Writers) EvictTarget(targetID string) {
|
||||
ws.mu.Lock()
|
||||
|
||||
w, ok := ws.writers[targetID]
|
||||
if ok {
|
||||
delete(ws.writers, targetID)
|
||||
}
|
||||
|
||||
ws.mu.Unlock()
|
||||
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
w.evict()
|
||||
|
||||
ws.log.Info(
|
||||
"evicted archive writer",
|
||||
"target_id", targetID,
|
||||
"path", w.path,
|
||||
)
|
||||
}
|
||||
|
||||
// EvictWebhook implements Archives. It evicts, exactly as EvictTarget
|
||||
// does, the writer of every database target of a webhook. The archive
|
||||
// files themselves are left on disk — they are long-term storage the
|
||||
// operator owns.
|
||||
func (ws *Writers) EvictWebhook(webhookID string) {
|
||||
ws.mu.Lock()
|
||||
|
||||
var gone []*archiveWriter
|
||||
|
||||
for targetID, w := range ws.writers {
|
||||
if w.webhookID == webhookID {
|
||||
delete(ws.writers, targetID)
|
||||
|
||||
gone = append(gone, w)
|
||||
}
|
||||
}
|
||||
|
||||
ws.mu.Unlock()
|
||||
|
||||
for _, w := range gone {
|
||||
w.evict()
|
||||
|
||||
ws.log.Info(
|
||||
"evicted archive writer",
|
||||
"webhook_id", webhookID,
|
||||
"path", w.path,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// EvictAll evicts every cached archive writer, exactly as EvictTarget
|
||||
// does for one target. The delivery engine calls it at shutdown, once
|
||||
// its workers have returned. Closing the last handle on an archive
|
||||
// moves the contents of its -wal into the .db and removes the
|
||||
// -wal, so a clean stop leaves each archive as a single file.
|
||||
func (ws *Writers) EvictAll() {
|
||||
ws.mu.Lock()
|
||||
|
||||
writers := ws.writers
|
||||
ws.writers = nil
|
||||
|
||||
ws.mu.Unlock()
|
||||
|
||||
for _, w := range writers {
|
||||
w.evict()
|
||||
}
|
||||
}
|
||||
|
||||
// writerFor returns the archive writer for a database target,
|
||||
// creating and caching it on first use. Each target has one writer
|
||||
// so its close/reopen debounce state is shared across concurrent
|
||||
// deliveries, and so a rename and the idle sweep take the same lock
|
||||
// as its writes.
|
||||
func (ws *Writers) writerFor(
|
||||
targetID string,
|
||||
) (*archiveWriter, error) {
|
||||
ws.mu.Lock()
|
||||
defer ws.mu.Unlock()
|
||||
|
||||
w, ok := ws.writers[targetID]
|
||||
if !ok {
|
||||
var err error
|
||||
|
||||
w, err = ws.newWriter(targetID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if ws.writers == nil {
|
||||
ws.writers = make(map[string]*archiveWriter)
|
||||
}
|
||||
|
||||
ws.writers[targetID] = w
|
||||
}
|
||||
|
||||
// A delivery claims the entry: even if the idle sweep created
|
||||
// it moments ago, it now belongs to the registry proper and
|
||||
// the sweep must leave it in place when it finishes.
|
||||
w.sweepOwned = false
|
||||
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// sweepWriterFor returns the archive writer the idle sweep should
|
||||
// prune a target's archive through, together with whether the sweep
|
||||
// itself created the registry entry.
|
||||
//
|
||||
// The sweep must route its prune through the registered writer so
|
||||
// the writer's mutex orders it against concurrent writes, but it
|
||||
// must never leave a registry entry behind: a sweep that ran
|
||||
// concurrently with the target's deletion would otherwise
|
||||
// re-create an entry that nothing will ever evict again, which is
|
||||
// exactly the leak eviction exists to prevent. An entry the sweep
|
||||
// creates is therefore marked sweep-owned and handed back to
|
||||
// releaseSweepWriter when the sweep is done.
|
||||
func (ws *Writers) sweepWriterFor(
|
||||
targetID string,
|
||||
) (*archiveWriter, bool, error) {
|
||||
ws.mu.Lock()
|
||||
defer ws.mu.Unlock()
|
||||
|
||||
w, ok := ws.writers[targetID]
|
||||
if ok {
|
||||
return w, false, nil
|
||||
}
|
||||
|
||||
w, err := ws.newWriter(targetID)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
|
||||
if ws.writers == nil {
|
||||
ws.writers = make(map[string]*archiveWriter)
|
||||
}
|
||||
|
||||
w.sweepOwned = true
|
||||
ws.writers[targetID] = w
|
||||
|
||||
return w, true, nil
|
||||
}
|
||||
|
||||
// releaseSweepWriter drops a registry entry that the idle sweep
|
||||
// created, so a sweep leaves the registry exactly as it found it.
|
||||
//
|
||||
// The entry is removed only if it is still the very writer the
|
||||
// sweep installed and no delivery has claimed it in the meantime
|
||||
// (writerFor clears sweepOwned when it hands a writer to the
|
||||
// write path). Both conditions are evaluated under the registry
|
||||
// lock, so an eviction that raced the sweep — which removes the
|
||||
// entry outright — simply finds nothing left to do here, and a
|
||||
// delivery that adopted the writer keeps a registered, evictable
|
||||
// one.
|
||||
func (ws *Writers) releaseSweepWriter(
|
||||
targetID string, w *archiveWriter,
|
||||
) {
|
||||
ws.mu.Lock()
|
||||
defer ws.mu.Unlock()
|
||||
|
||||
cur, ok := ws.writers[targetID]
|
||||
if !ok || cur != w || !cur.sweepOwned {
|
||||
return
|
||||
}
|
||||
|
||||
delete(ws.writers, targetID)
|
||||
}
|
||||
|
||||
// newWriter builds the writer for a database target's archive. Its
|
||||
// path is the one Path gives for the webhook and the target as
|
||||
// the main database names them now; from then on only rename changes
|
||||
// the name the writer uses. It does not touch the archive files.
|
||||
func (ws *Writers) newWriter(
|
||||
targetID string,
|
||||
) (*archiveWriter, error) {
|
||||
if ws.dbMgr == nil {
|
||||
return nil, errArchiveNoDataDir
|
||||
}
|
||||
|
||||
var target database.Target
|
||||
|
||||
err := ws.db.DB().
|
||||
Preload("Webhook").
|
||||
First(&target, "id = ?", targetID).Error
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"loading database target %s: %w", targetID, err,
|
||||
)
|
||||
}
|
||||
|
||||
w := newArchiveWriter(
|
||||
Path(ws.dbMgr, &target.Webhook, &target),
|
||||
ws.log,
|
||||
)
|
||||
w.webhookID = target.WebhookID
|
||||
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// sweepArchive prunes one database target's archive of rows older
|
||||
// than expiry, without requiring a write. A missing archive file is
|
||||
// left missing (see sweepExpired), so a sweep never creates an
|
||||
// archive for a target that has never received an event.
|
||||
//
|
||||
// It also never leaves a registry entry behind: an entry it had
|
||||
// to create to reach the writer's mutex is released again once
|
||||
// the prune is done, so a sweep racing a target deletion cannot
|
||||
// resurrect the writer the eviction just dropped.
|
||||
func (ws *Writers) sweepArchive(
|
||||
targetID string, expiry time.Duration,
|
||||
) error {
|
||||
w, created, err := ws.sweepWriterFor(targetID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if created {
|
||||
defer ws.releaseSweepWriter(targetID, w)
|
||||
}
|
||||
|
||||
return w.sweepExpired(expiry)
|
||||
}
|
||||
@@ -0,0 +1,757 @@
|
||||
package archive_test
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
const (
|
||||
// testContentType is the event content type used in tests.
|
||||
testContentType = "application/json"
|
||||
|
||||
// eventQuery is the query string an archived event carries.
|
||||
eventQuery = "a=1&b=2"
|
||||
)
|
||||
|
||||
func archiveTestLogger() *slog.Logger {
|
||||
return slog.New(slog.NewTextHandler(
|
||||
os.Stderr,
|
||||
&slog.HandlerOptions{Level: slog.LevelDebug},
|
||||
))
|
||||
}
|
||||
|
||||
// openArchiveDBForRead opens an archive file read-only so a
|
||||
// test can inspect the rows the writer persisted.
|
||||
func openArchiveDBForRead(
|
||||
t *testing.T, path string,
|
||||
) *gorm.DB {
|
||||
t.Helper()
|
||||
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite",
|
||||
fmt.Sprintf("file:%s?mode=ro", path),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
return gdb
|
||||
}
|
||||
|
||||
// archiveFileSuffixes returns the archive file itself and the
|
||||
// SQLite sidecars that accompany an open database. A test that
|
||||
// asserts no archive was created has to check all of them.
|
||||
func archiveFileSuffixes() []string {
|
||||
return []string{"", "-wal", "-shm"}
|
||||
}
|
||||
|
||||
// removeArchiveFiles simulates an operator moving the archive
|
||||
// away by deleting the SQLite file and its sidecar files.
|
||||
func removeArchiveFiles(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
|
||||
for _, suffix := range []string{
|
||||
"", "-wal", "-shm", "-journal",
|
||||
} {
|
||||
err := os.Remove(path + suffix)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
t.Fatalf("removing %s%s: %v", path, suffix, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// assertStoredTime asserts that text, a timestamp as SQLite holds it,
|
||||
// is in the form GORM stores created_at in: the layout of
|
||||
// time.Time.String, with no " m=+…" monotonic clock reading after it.
|
||||
func assertStoredTime(t *testing.T, text string) {
|
||||
t.Helper()
|
||||
|
||||
_, err := time.Parse(
|
||||
"2006-01-02 15:04:05.999999999 -0700 MST", text,
|
||||
)
|
||||
assert.NoError(t, err, "stored as %q", text)
|
||||
}
|
||||
|
||||
// newEvent returns an event received now with the given body, as the
|
||||
// webhook handler stores it.
|
||||
func newEvent(body string) database.Event {
|
||||
return database.Event{
|
||||
BaseModel: database.BaseModel{
|
||||
ID: uuid.New().String(),
|
||||
CreatedAt: time.Now(),
|
||||
},
|
||||
WebhookID: uuid.New().String(),
|
||||
EntrypointID: uuid.New().String(),
|
||||
Method: http.MethodPost,
|
||||
Headers: `{"Content-Type":["application/json"]}`,
|
||||
Body: body,
|
||||
ContentType: testContentType,
|
||||
}
|
||||
}
|
||||
|
||||
// databaseTargetDelivery returns a delivery of event to a database
|
||||
// target, as the database target hands it to Writers.Write.
|
||||
func databaseTargetDelivery(
|
||||
event database.Event, tgt *database.Target,
|
||||
) *database.Delivery {
|
||||
return &database.Delivery{
|
||||
EventID: event.ID,
|
||||
TargetID: tgt.ID,
|
||||
Event: event,
|
||||
Target: database.Target{
|
||||
Name: tgt.Name,
|
||||
Type: database.TargetTypeDatabase,
|
||||
ArchiveExpiry: tgt.ArchiveExpiry,
|
||||
ArchiveRotation: tgt.ArchiveRotation,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveWriter_WritesRow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
row := archive.ExportArchivedEvent{
|
||||
EventID: "ev-1",
|
||||
WebhookID: "wh-1",
|
||||
EntrypointID: "ep-1",
|
||||
Method: "POST",
|
||||
Headers: `{"X":"Y"}`,
|
||||
Body: `{"hello":"world"}`,
|
||||
ContentType: "application/json",
|
||||
}
|
||||
|
||||
require.NoError(t, w.Write(row, 0))
|
||||
assert.FileExists(t, path)
|
||||
|
||||
var got []archive.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, w.DB().Find(&got).Error)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "ev-1", got[0].EventID)
|
||||
assert.Equal(t, "wh-1", got[0].WebhookID)
|
||||
assert.Equal(t, "ep-1", got[0].EntrypointID)
|
||||
assert.Equal(t, row.Method, got[0].Method)
|
||||
assert.Equal(t, row.ContentType, got[0].ContentType)
|
||||
assert.JSONEq(t, `{"hello":"world"}`, got[0].Body)
|
||||
assert.False(t, got[0].ArchivedAt.IsZero())
|
||||
|
||||
var archivedAt string
|
||||
|
||||
require.NoError(t, w.DB().Raw(
|
||||
"SELECT CAST(archived_at AS TEXT) FROM archived_events",
|
||||
).Row().Scan(&archivedAt))
|
||||
assertStoredTime(t, archivedAt)
|
||||
}
|
||||
|
||||
func TestArchiveWriter_RecreatesAfterRemoval(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Write(
|
||||
archive.ExportArchivedEvent{EventID: "a"}, 0,
|
||||
))
|
||||
assert.FileExists(t, path)
|
||||
|
||||
// The operator moves the archive away while the handle is
|
||||
// still open.
|
||||
removeArchiveFiles(t, path)
|
||||
require.NoFileExists(t, path)
|
||||
|
||||
// The next write recreates the file with a fresh schema and
|
||||
// only the new row.
|
||||
require.NoError(t, w.Write(
|
||||
archive.ExportArchivedEvent{EventID: "b"}, 0,
|
||||
))
|
||||
assert.FileExists(t, path)
|
||||
|
||||
var got []archive.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, w.DB().Find(&got).Error)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "b", got[0].EventID)
|
||||
}
|
||||
|
||||
// TestArchiveHandleIsWAL closes the last gap in the durability
|
||||
// evidence: the main and per-webhook tiers each assert their journal
|
||||
// mode on a live handle, and the archive tier gets its settings from
|
||||
// the same code path but nothing checked the running file.
|
||||
func TestArchiveHandleIsWAL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w := archive.NewExportArchiveWriter(
|
||||
filepath.Join(t.TempDir(), "archive-wal.db"),
|
||||
archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Open(0))
|
||||
|
||||
var mode string
|
||||
|
||||
row := w.DB().Raw("pragma journal_mode").Row()
|
||||
require.NoError(t, row.Scan(&mode))
|
||||
assert.Equal(t, "wal", strings.ToLower(mode))
|
||||
|
||||
var busy string
|
||||
|
||||
row = w.DB().Raw("pragma busy_timeout").Row()
|
||||
require.NoError(t, row.Scan(&busy))
|
||||
assert.Equal(t, "10000", busy)
|
||||
}
|
||||
|
||||
// TestStatArchive proves Stat finds no file before the first
|
||||
// write; after a write still held in the -wal, counts the -wal in the
|
||||
// size and takes its later time as the last write; and finds no file
|
||||
// again once the file has been moved away.
|
||||
func TestStatArchive(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
|
||||
_, err := archive.Stat(path)
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
|
||||
// With the clock stopped, the reopen debounce never passes, so
|
||||
// the handle stays open after the write.
|
||||
stopped := time.Now()
|
||||
w := archive.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w.SetNow(func() time.Time { return stopped })
|
||||
|
||||
require.NoError(t, w.Write(archive.ExportArchivedEvent{EventID: "a"}, 0))
|
||||
|
||||
written := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
earlier := written.Add(-time.Hour)
|
||||
require.NoError(t, os.Chtimes(path, earlier, earlier))
|
||||
require.NoError(t, os.Chtimes(path+"-wal", written, written))
|
||||
|
||||
file, err := os.Stat(path)
|
||||
require.NoError(t, err)
|
||||
|
||||
wal, err := os.Stat(path + "-wal")
|
||||
require.NoError(t, err)
|
||||
require.Positive(t, wal.Size())
|
||||
|
||||
got, err := archive.Stat(path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, got.Files)
|
||||
assert.Equal(t, file.Size()+wal.Size(), got.Size)
|
||||
assert.True(t, written.Equal(got.Written), got.Written)
|
||||
|
||||
removeArchiveFiles(t, path)
|
||||
|
||||
_, err = archive.Stat(path)
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
}
|
||||
|
||||
func TestArchiveWriter_ReopenDebounce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const debounce = 2 * time.Second
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), debounce,
|
||||
)
|
||||
|
||||
// The writer measures its reopen debounce on this clock, which
|
||||
// only the test moves, so how long the host takes between
|
||||
// writes cannot change the result.
|
||||
now := time.Now()
|
||||
|
||||
w.SetNow(func() time.Time { return now })
|
||||
|
||||
require.NoError(t, w.Write(
|
||||
archive.ExportArchivedEvent{EventID: "a"}, 0,
|
||||
))
|
||||
require.NoError(t, w.Write(
|
||||
archive.ExportArchivedEvent{EventID: "b"}, 0,
|
||||
))
|
||||
|
||||
// Two writes inside the debounce window trigger only the
|
||||
// initial open — no extra close/reopen.
|
||||
assert.Equal(t, 1, w.Reopens())
|
||||
|
||||
now = now.Add(debounce)
|
||||
|
||||
require.NoError(t, w.Write(
|
||||
archive.ExportArchivedEvent{EventID: "c"}, 0,
|
||||
))
|
||||
|
||||
// A write after the window elapses closes and reopens once.
|
||||
assert.Equal(t, 2, w.Reopens())
|
||||
}
|
||||
|
||||
func TestArchiveWriter_ExpiryPrune(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Open(0))
|
||||
|
||||
old := archive.ExportArchivedEvent{
|
||||
EventID: "old",
|
||||
ArchivedAt: time.Now().Add(-2 * time.Hour),
|
||||
}
|
||||
fresh := archive.ExportArchivedEvent{
|
||||
EventID: "fresh",
|
||||
ArchivedAt: time.Now(),
|
||||
}
|
||||
|
||||
require.NoError(t, w.DB().Create(&old).Error)
|
||||
require.NoError(t, w.DB().Create(&fresh).Error)
|
||||
|
||||
// Reopening with a one-hour expiry prunes the old row.
|
||||
require.NoError(t, w.Reopen(time.Hour))
|
||||
|
||||
var got []archive.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, w.DB().Find(&got).Error)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "fresh", got[0].EventID)
|
||||
}
|
||||
|
||||
func TestParseArchiveExpiry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
in string
|
||||
want time.Duration
|
||||
wantErr bool
|
||||
}{
|
||||
{"empty", "", 0, false},
|
||||
{"explicit never", "never", 0, false},
|
||||
{"hour", "1h", time.Hour, false},
|
||||
{"thirty days", "720h", 720 * time.Hour, false},
|
||||
{"minutes", "30m", 30 * time.Minute, false},
|
||||
{"unparseable", "nonsense", 0, true},
|
||||
{"days are not a Go unit", "7d", 0, true},
|
||||
{"zero duration", "0s", 0, true},
|
||||
{"bare zero", "0", 0, true},
|
||||
{"negative duration", "-5h", 0, true},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := archive.ParseExpiry(tc.in)
|
||||
if tc.wantErr {
|
||||
require.Error(t, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tc.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestArchiveFileName pins the archive file name and the rules
|
||||
// that make a webhook or target name safe to put in it.
|
||||
func TestArchiveFileName(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const id = "3f2a1c9e-8d4b-4c1a-9e2f-0a1b2c3d4e5f"
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
webhook string
|
||||
target string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"plain names", "orders", "archive",
|
||||
"archive-orders-archive-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"lowercased", "Orders", "Main Archive",
|
||||
"archive-orders-main-archive-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"a run of other characters is one dash",
|
||||
`a /\..b`, "c__--d",
|
||||
"archive-a-b-c-d-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"no dash at either end", " --orders!! ", "(archive)",
|
||||
"archive-orders-archive-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"path separators", "../../etc/passwd", "a/b",
|
||||
"archive-etc-passwd-a-b-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"letters outside ASCII are dropped",
|
||||
"Bestellungen Größe", "café",
|
||||
"archive-bestellungen-gr-e-caf-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"nothing left is unnamed", "", "!!!",
|
||||
"archive-unnamed-unnamed-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"cut to 40 characters", strings.Repeat("a", 50), "x",
|
||||
"archive-" + strings.Repeat("a", 40) + "-x-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"no dash left by the cut",
|
||||
strings.Repeat("a", 39) + " b", "x",
|
||||
"archive-" + strings.Repeat("a", 39) + "-x-" + id + ".db",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assert.Equal(
|
||||
t, tc.want,
|
||||
archive.FileName(tc.webhook, tc.target, id),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestWrite_EachTargetHasItsOwnArchive proves two
|
||||
// database targets of one webhook archive into separate files.
|
||||
func TestWrite_EachTargetHasItsOwnArchive(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
first := env.seedDatabaseTarget(t, 0, "")
|
||||
second := env.addDatabaseTarget(t, first.WebhookID, 0, "")
|
||||
|
||||
event := newEvent(`{"n":1}`)
|
||||
|
||||
for _, tgt := range []*database.Target{first, second} {
|
||||
require.NoError(t, env.writers.Write(
|
||||
databaseTargetDelivery(event, tgt),
|
||||
))
|
||||
}
|
||||
|
||||
require.NotEqual(
|
||||
t, env.archivePath(first), env.archivePath(second),
|
||||
)
|
||||
assert.Equal(
|
||||
t, []string{event.ID},
|
||||
archivedEventIDs(t, env.archivePath(first)),
|
||||
)
|
||||
assert.Equal(
|
||||
t, []string{event.ID},
|
||||
archivedEventIDs(t, env.archivePath(second)),
|
||||
)
|
||||
}
|
||||
|
||||
// TestRename_MovesTheFile proves a rename moves the archive, rows
|
||||
// and all, and that later writes go to the new name.
|
||||
func TestRename_MovesTheFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
oldPath := env.archivePath(tgt)
|
||||
|
||||
first := newEvent(`{"n":1}`)
|
||||
require.NoError(t, env.writers.Write(databaseTargetDelivery(first, tgt)))
|
||||
require.FileExists(t, oldPath)
|
||||
|
||||
require.NoError(
|
||||
t, env.writers.Rename(tgt.ID, "Orders", "Long Term"),
|
||||
)
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
)
|
||||
|
||||
assert.NoFileExists(t, oldPath)
|
||||
assert.Equal(t, []string{first.ID}, archivedEventIDs(t, newPath))
|
||||
|
||||
second := newEvent(`{"n":2}`)
|
||||
require.NoError(t, env.writers.Write(databaseTargetDelivery(second, tgt)))
|
||||
|
||||
assert.ElementsMatch(
|
||||
t, []string{first.ID, second.ID},
|
||||
archivedEventIDs(t, newPath),
|
||||
)
|
||||
assert.NoFileExists(
|
||||
t, oldPath, "a write after the rename must use the new name",
|
||||
)
|
||||
}
|
||||
|
||||
// TestRename_NeverReplacesAFile plants a file at the new name, once
|
||||
// the .db alone, once a lone -wal and once a lone -shm, and proves
|
||||
// each time that the rename is refused, the planted file survives,
|
||||
// and the archive keeps its name and its rows.
|
||||
func TestRename_NeverReplacesAFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
t.Run("planted .db"+suffix, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
oldPath := env.archivePath(tgt)
|
||||
|
||||
first := newEvent(`{"n":1}`)
|
||||
require.NoError(
|
||||
t, env.writers.Write(databaseTargetDelivery(first, tgt)),
|
||||
)
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
)
|
||||
plantedPath := newPath + suffix
|
||||
require.NoError(
|
||||
t, os.WriteFile(plantedPath, []byte("planted"), 0o600),
|
||||
)
|
||||
|
||||
require.ErrorIs(
|
||||
t, env.writers.Rename(tgt.ID, "Orders", "Long Term"),
|
||||
archive.ErrNameTaken,
|
||||
)
|
||||
|
||||
//nolint:gosec // reads the file the test planted under t.TempDir()
|
||||
planted, err := os.ReadFile(plantedPath)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "planted", string(planted))
|
||||
|
||||
second := newEvent(`{"n":2}`)
|
||||
require.NoError(
|
||||
t, env.writers.Write(databaseTargetDelivery(second, tgt)),
|
||||
)
|
||||
|
||||
assert.ElementsMatch(
|
||||
t, []string{first.ID, second.ID},
|
||||
archivedEventIDs(t, oldPath),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRename_BeforeTheNameIsSaved covers the order the handlers
|
||||
// use: they rename before they save the new name, so a delivery in
|
||||
// between must write under the new name although the main database
|
||||
// still has the old one. It also shows that renaming an archive that
|
||||
// does not exist yet is not an error.
|
||||
func TestRename_BeforeTheNameIsSaved(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
|
||||
require.NoError(
|
||||
t, env.writers.Rename(tgt.ID, "Orders", "Archive"),
|
||||
)
|
||||
|
||||
event := newEvent(`{"n":1}`)
|
||||
require.NoError(t, env.writers.Write(databaseTargetDelivery(event, tgt)))
|
||||
|
||||
assert.FileExists(
|
||||
t,
|
||||
filepath.Join(
|
||||
env.dataDir, "archive-orders-archive-"+tgt.ID+".db",
|
||||
),
|
||||
)
|
||||
assert.NoFileExists(t, env.archivePath(tgt))
|
||||
}
|
||||
|
||||
// TestArchiveWriter_RenameMovesSidecars proves a rename carries
|
||||
// the -wal and -shm a crash can leave beside an archive no handle
|
||||
// has opened since. SQLite finds them by name, so a -wal left
|
||||
// behind would lose the transactions it holds.
|
||||
func TestArchiveWriter_RenameMovesSidecars(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
oldPath := filepath.Join(dir, "archive-old.db")
|
||||
newPath := filepath.Join(dir, "archive-new.db")
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
require.NoError(
|
||||
t, os.WriteFile(oldPath+suffix, []byte(suffix), 0o600),
|
||||
)
|
||||
}
|
||||
|
||||
w := archive.NewExportArchiveWriter(
|
||||
oldPath, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Rename("archive-new.db"))
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
assert.NoFileExists(t, oldPath+suffix)
|
||||
assert.FileExists(t, newPath+suffix)
|
||||
}
|
||||
|
||||
assert.Equal(t, newPath, w.Path())
|
||||
}
|
||||
|
||||
// TestArchiveWriter_RenameMovesBackOnFailure makes the -wal fail to
|
||||
// move after the .db has moved, and proves the .db is moved back, so
|
||||
// the archive is never split across two names. The new name is 255
|
||||
// bytes, the longest a file name may be, so the .db can take it but
|
||||
// the -wal, four bytes longer, cannot.
|
||||
func TestArchiveWriter_RenameMovesBackOnFailure(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
oldPath := filepath.Join(dir, "archive-old.db")
|
||||
newName := strings.Repeat("a", 252) + ".db"
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
require.NoError(
|
||||
t, os.WriteFile(oldPath+suffix, []byte(suffix), 0o600),
|
||||
)
|
||||
}
|
||||
|
||||
w := archive.NewExportArchiveWriter(
|
||||
oldPath, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.Error(t, w.Rename(newName))
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
assert.FileExists(t, oldPath+suffix)
|
||||
}
|
||||
|
||||
assert.NoFileExists(t, filepath.Join(dir, newName))
|
||||
assert.Equal(t, oldPath, w.Path())
|
||||
}
|
||||
|
||||
// TestArchiveWriter_RenameMovesBackEveryFile renames a target with a
|
||||
// file without a period and a file for a month, and proves that when
|
||||
// the month's file fails to move, the file already moved is moved back:
|
||||
// both files are under the old name with their rows, and nothing is
|
||||
// under the new name. The new name is 251 bytes, so the file without a
|
||||
// period, with its -wal and -shm, can take it, but the month's file,
|
||||
// eight bytes longer, cannot.
|
||||
func TestArchiveWriter_RenameMovesBackEveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
oldPath := filepath.Join(dir, "archive-old.db")
|
||||
monthPath := filepath.Join(dir, "archive-old-2026-03.db")
|
||||
w := archive.NewExportArchiveWriter(oldPath, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.WritePeriod(
|
||||
archive.ExportArchivedEvent{EventID: "in-none"}, 0, "",
|
||||
))
|
||||
require.NoError(t, w.WritePeriod(
|
||||
archive.ExportArchivedEvent{EventID: "in-month"}, 0, "2026-03",
|
||||
))
|
||||
|
||||
require.Error(t, w.Rename(strings.Repeat("a", 248)+".db"))
|
||||
|
||||
assert.Equal(t, []string{"in-none"}, archivedEventIDs(t, oldPath))
|
||||
assert.Equal(t, []string{"in-month"}, archivedEventIDs(t, monthPath))
|
||||
|
||||
entries, err := os.ReadDir(dir)
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, entry := range entries {
|
||||
assert.True(t, strings.HasPrefix(entry.Name(), "archive-old"),
|
||||
"%s is not under the old name", entry.Name())
|
||||
}
|
||||
|
||||
assert.Equal(t, oldPath, w.Path())
|
||||
}
|
||||
|
||||
// TestRenameLock_HeldByOneEditAtATime holds the rename lock, as an edit
|
||||
// does from loading the stored names until it has saved them. While it
|
||||
// is held, the edit's rename and an archive write still finish, since
|
||||
// neither takes it, and the event ends up under the new name whichever
|
||||
// ran first. A second edit waits for the lock until the first lets go.
|
||||
func TestRenameLock_HeldByOneEditAtATime(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
event := newEvent(`{"n":1}`)
|
||||
|
||||
env.writers.RenameLock().Lock()
|
||||
|
||||
finished := make(chan error, 2)
|
||||
|
||||
go func() {
|
||||
finished <- env.writers.Rename(tgt.ID, "Orders", "Long Term")
|
||||
}()
|
||||
go func() {
|
||||
finished <- env.writers.Write(databaseTargetDelivery(event, tgt))
|
||||
}()
|
||||
|
||||
for range 2 {
|
||||
select {
|
||||
case err := <-finished:
|
||||
require.NoError(t, err)
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("a rename or a write waited for the rename lock")
|
||||
}
|
||||
}
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
)
|
||||
assert.Equal(t, []string{event.ID}, archivedEventIDs(t, newPath))
|
||||
|
||||
secondEdit := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
env.writers.RenameLock().Lock()
|
||||
close(secondEdit)
|
||||
env.writers.RenameLock().Unlock()
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-secondEdit:
|
||||
t.Fatal("a second edit took the rename lock while it was held")
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
|
||||
env.writers.RenameLock().Unlock()
|
||||
|
||||
select {
|
||||
case <-secondEdit:
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("a second edit did not get the rename lock once released")
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package delivery
|
||||
package archive
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
@@ -28,25 +28,25 @@ import (
|
||||
const archiveTableQuery = "SELECT count(*) FROM sqlite_master " +
|
||||
"WHERE type = 'table' AND name = 'archived_events'"
|
||||
|
||||
// ArchivePath returns where a database target's archive file is: in
|
||||
// Path returns where a database target's archive file is: in
|
||||
// the data directory, beside the webhook's event database, under the
|
||||
// name ArchiveFileName gives it.
|
||||
func ArchivePath(
|
||||
// name FileName gives it.
|
||||
func Path(
|
||||
dbMgr *database.WebhookDBManager,
|
||||
webhook *database.Webhook,
|
||||
target *database.Target,
|
||||
) string {
|
||||
return filepath.Join(
|
||||
filepath.Dir(dbMgr.DBPath(webhook.ID)),
|
||||
ArchiveFileName(webhook.Name, target.Name, target.ID),
|
||||
FileName(webhook.Name, target.Name, target.ID),
|
||||
)
|
||||
}
|
||||
|
||||
// ArchiveExportFileName returns the name a database target's archive
|
||||
// ExportFileName returns the name a database target's archive
|
||||
// downloads under:
|
||||
// archive-WEBHOOKNAME-TARGETNAME-YYYYMMDDTHHMMSSZ.json.gz, the names
|
||||
// made safe as in ArchiveFileName and the time in UTC.
|
||||
func ArchiveExportFileName(
|
||||
// made safe as in FileName and the time in UTC.
|
||||
func ExportFileName(
|
||||
webhookName, targetName string, at time.Time,
|
||||
) string {
|
||||
return "archive-" + archiveNamePart(webhookName) + "-" +
|
||||
@@ -54,7 +54,7 @@ func ArchiveExportFileName(
|
||||
at.UTC().Format("20060102T150405Z") + ".json.gz"
|
||||
}
|
||||
|
||||
// ArchiveExport is a database target's archive listed for download. It
|
||||
// Export is a database target's archive listed for download. It
|
||||
// opens one of the target's files at a time, only when its rows are
|
||||
// about to be written out, and closes it before it opens the next, so
|
||||
// an export holds at most one file open however many the target has.
|
||||
@@ -66,7 +66,7 @@ func ArchiveExportFileName(
|
||||
// is open, and the export does not see them. SQLite cannot checkpoint
|
||||
// a -wal past an open snapshot, so the open file's -wal grows until the
|
||||
// export has written that file out.
|
||||
type ArchiveExport struct {
|
||||
type Export struct {
|
||||
// periods are the periods of the target's files when the export
|
||||
// was listed, "" for the file without one, in the order
|
||||
// archiveFiles lists them.
|
||||
@@ -77,7 +77,7 @@ type ArchiveExport struct {
|
||||
// between.
|
||||
lock sync.Locker
|
||||
|
||||
// currentPath returns the path ArchivePath gives the target under
|
||||
// currentPath returns the path Path gives the target under
|
||||
// the names stored for it now, which a rename may have changed since
|
||||
// the export was listed.
|
||||
currentPath func() (string, error)
|
||||
@@ -103,11 +103,12 @@ type exportedName struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// NewArchiveExport lists a database target's archive files for export,
|
||||
// given the path ArchivePath gives it (see archiveFiles). It opens none
|
||||
// NewExport lists a database target's archive files for export,
|
||||
// given the path Path gives it (see archiveFiles). It opens none
|
||||
// of them. Its caller holds lock, which every rename of the target's
|
||||
// files runs under, from reading the names path is made of until it
|
||||
// returns, so the files it lists are the ones those names give.
|
||||
// files runs under (Writers.RenameLock returns it), from reading the names
|
||||
// path is made of until it returns, so the files it lists are the ones
|
||||
// those names give.
|
||||
//
|
||||
// WriteGzipJSON, called without lock held, finds each file again by its
|
||||
// period under the path currentPath gives, holding lock while it does
|
||||
@@ -115,18 +116,18 @@ type exportedName struct {
|
||||
// file. A file that is gone by then, emptied by the sweep or moved
|
||||
// away, is skipped. The export never creates a file: with no files, it
|
||||
// has no rows.
|
||||
func NewArchiveExport(
|
||||
func NewExport(
|
||||
path string,
|
||||
lock sync.Locker,
|
||||
currentPath func() (string, error),
|
||||
log *slog.Logger,
|
||||
) (*ArchiveExport, error) {
|
||||
) (*Export, error) {
|
||||
files, err := archiveFiles(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
x := &ArchiveExport{lock: lock, currentPath: currentPath, log: log}
|
||||
x := &Export{lock: lock, currentPath: currentPath, log: log}
|
||||
|
||||
for _, file := range files {
|
||||
x.periods = append(x.periods, file.period)
|
||||
@@ -199,7 +200,7 @@ func openExportFile(
|
||||
// closed once its rows are written, before the next is opened. When it
|
||||
// returns, no file is open. After an error the gzip stream is left
|
||||
// unfinished, so what was written does not decompress as a whole file.
|
||||
func (x *ArchiveExport) WriteGzipJSON(
|
||||
func (x *Export) WriteGzipJSON(
|
||||
ctx context.Context,
|
||||
w io.Writer,
|
||||
webhook *database.Webhook,
|
||||
@@ -228,7 +229,7 @@ func (x *ArchiveExport) WriteGzipJSON(
|
||||
// openFile finds the target's archive file for period under the path
|
||||
// currentPath gives now, and opens it for the export, holding x.lock
|
||||
// for both. For a file that is gone, the error wraps fs.ErrNotExist.
|
||||
func (x *ArchiveExport) openFile(
|
||||
func (x *Export) openFile(
|
||||
ctx context.Context, period string,
|
||||
) (*exportFile, error) {
|
||||
x.lock.Lock()
|
||||
@@ -258,7 +259,7 @@ func (f *exportFile) close() error {
|
||||
|
||||
// writeJSON writes head with archived_events added as its last key,
|
||||
// the rows going into it one at a time.
|
||||
func (x *ArchiveExport) writeJSON(
|
||||
func (x *Export) writeJSON(
|
||||
ctx context.Context, w io.Writer, head []byte,
|
||||
) error {
|
||||
// head goes out without its closing brace, so that
|
||||
@@ -286,7 +287,7 @@ func (x *ArchiveExport) writeJSON(
|
||||
// writeRows writes the archived rows of each file to w, one per line,
|
||||
// separated by commas, opening each file in turn and closing it once
|
||||
// its rows are written.
|
||||
func (x *ArchiveExport) writeRows(ctx context.Context, w io.Writer) error {
|
||||
func (x *Export) writeRows(ctx context.Context, w io.Writer) error {
|
||||
sep := "\n"
|
||||
|
||||
for _, period := range x.periods {
|
||||
+27
-27
@@ -1,4 +1,4 @@
|
||||
package delivery_test
|
||||
package archive_test
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
@@ -19,8 +19,8 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// The webhook and the target the export tests' archives belong to.
|
||||
@@ -37,7 +37,7 @@ const binaryBody = "\xff\xfe\x00\x01binary\x80"
|
||||
// writeExportTo writes export to w as the archive of the export tests'
|
||||
// webhook and target, exported at 2026-10-02T12:03:04Z.
|
||||
func writeExportTo(
|
||||
t *testing.T, export *delivery.ArchiveExport, w io.Writer,
|
||||
t *testing.T, export *archive.Export, w io.Writer,
|
||||
) error {
|
||||
t.Helper()
|
||||
|
||||
@@ -57,7 +57,7 @@ func writeExportTo(
|
||||
|
||||
// listExport lists the archive at path for export, as the archive of a
|
||||
// target whose names do not change.
|
||||
func listExport(t *testing.T, path string) *delivery.ArchiveExport {
|
||||
func listExport(t *testing.T, path string) *archive.Export {
|
||||
t.Helper()
|
||||
|
||||
return newExport(t, path, &sync.Mutex{}, func() (string, error) {
|
||||
@@ -73,10 +73,10 @@ func newExport(
|
||||
path string,
|
||||
lock sync.Locker,
|
||||
currentPath func() (string, error),
|
||||
) *delivery.ArchiveExport {
|
||||
) *archive.Export {
|
||||
t.Helper()
|
||||
|
||||
export, err := delivery.NewArchiveExport(
|
||||
export, err := archive.NewExport(
|
||||
path, lock, currentPath, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
@@ -96,7 +96,7 @@ func exportArchive(t *testing.T, path string) map[string]any {
|
||||
// decompressed and parsed. Reading to the end makes the gzip reader
|
||||
// check that the stream was finished.
|
||||
func writeExport(
|
||||
t *testing.T, export *delivery.ArchiveExport,
|
||||
t *testing.T, export *archive.Export,
|
||||
) map[string]any {
|
||||
t.Helper()
|
||||
|
||||
@@ -157,11 +157,11 @@ func TestArchiveExport_MatchesStoredRows(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w := archive.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
bodies := []string{`{"order":1}`, "plain text", "", binaryBody}
|
||||
|
||||
for i, body := range bodies {
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{
|
||||
require.NoError(t, w.Write(archive.ExportArchivedEvent{
|
||||
EventID: fmt.Sprintf("ev-%d", i),
|
||||
WebhookID: exportWebhookID,
|
||||
EntrypointID: "ep-1",
|
||||
@@ -173,7 +173,7 @@ func TestArchiveExport_MatchesStoredRows(t *testing.T) {
|
||||
}, 0))
|
||||
}
|
||||
|
||||
var stored []delivery.ExportArchivedEvent
|
||||
var stored []archive.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, openArchiveDBForRead(t, path).
|
||||
Order("id").Find(&stored).Error)
|
||||
@@ -201,7 +201,7 @@ func TestArchiveExport_MatchesStoredRows(t *testing.T) {
|
||||
// assertExportedRow checks that ev, from an export, holds every column
|
||||
// of the stored row.
|
||||
func assertExportedRow(
|
||||
t *testing.T, row delivery.ExportArchivedEvent, ev map[string]any,
|
||||
t *testing.T, row archive.ExportArchivedEvent, ev map[string]any,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
@@ -248,7 +248,7 @@ func TestArchiveExport_Empty(t *testing.T) {
|
||||
|
||||
require.NoError(t, os.WriteFile(noTable, nil, 0o600))
|
||||
require.NoError(t,
|
||||
delivery.NewExportArchiveWriter(noRows, archiveTestLogger(), 0).
|
||||
archive.NewExportArchiveWriter(noRows, archiveTestLogger(), 0).
|
||||
Open(0),
|
||||
)
|
||||
|
||||
@@ -284,18 +284,18 @@ func TestArchiveExport_ReadsOneSnapshot(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w := archive.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "listed"}, 0))
|
||||
require.NoError(t, w.Write(archive.ExportArchivedEvent{EventID: "listed"}, 0))
|
||||
|
||||
opened := &unlockHook{fn: func() {
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "during"}, 0))
|
||||
require.NoError(t, w.Write(archive.ExportArchivedEvent{EventID: "during"}, 0))
|
||||
}}
|
||||
export := newExport(t, path, opened, func() (string, error) {
|
||||
return path, nil
|
||||
})
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "before-open"}, 0))
|
||||
require.NoError(t, w.Write(archive.ExportArchivedEvent{EventID: "before-open"}, 0))
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{"listed", "before-open"},
|
||||
@@ -305,7 +305,7 @@ func TestArchiveExport_ReadsOneSnapshot(t *testing.T) {
|
||||
var stored int64
|
||||
|
||||
require.NoError(t, openArchiveDBForRead(t, path).
|
||||
Model(&delivery.ExportArchivedEvent{}).Count(&stored).Error)
|
||||
Model(&archive.ExportArchivedEvent{}).Count(&stored).Error)
|
||||
assert.Equal(t, int64(3), stored)
|
||||
}
|
||||
|
||||
@@ -318,11 +318,11 @@ func TestArchiveExport_FindsFilesAfterRename(t *testing.T) {
|
||||
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "archive-old.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w := archive.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
for _, period := range []string{"", dayPeriod, hourPeriod} {
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-" + period}, 0, period,
|
||||
archive.ExportArchivedEvent{EventID: "in-" + period}, 0, period,
|
||||
))
|
||||
}
|
||||
|
||||
@@ -353,19 +353,19 @@ func TestArchiveExport_EveryFileOldestFirst(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w := archive.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
// Written in an order that is not the export's.
|
||||
for _, period := range []string{nextDayPeriod, "", hourPeriod, "2026-03"} {
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-" + period}, 0, period,
|
||||
archive.ExportArchivedEvent{EventID: "in-" + period}, 0, period,
|
||||
))
|
||||
}
|
||||
|
||||
export := listExport(t, path)
|
||||
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "later"}, 0, "2026-03-06",
|
||||
archive.ExportArchivedEvent{EventID: "later"}, 0, "2026-03-06",
|
||||
))
|
||||
|
||||
events := exportedEvents(t, writeExport(t, export))
|
||||
@@ -434,13 +434,13 @@ func TestArchiveExport_OneFileOpenAtATime(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
path := filepath.Join(dir, "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w := archive.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
random := make([]byte, 36<<10)
|
||||
|
||||
for hour := range 24 {
|
||||
_, _ = rand.Read(random)
|
||||
|
||||
require.NoError(t, w.WritePeriod(delivery.ExportArchivedEvent{
|
||||
require.NoError(t, w.WritePeriod(archive.ExportArchivedEvent{
|
||||
Body: base64.StdEncoding.EncodeToString(random),
|
||||
}, 0, fmt.Sprintf("2026-10-01-%02d", hour)))
|
||||
}
|
||||
@@ -489,7 +489,7 @@ func exportHeapGrowth(t *testing.T, rows, bodySize int) uint64 {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w := archive.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
// Base64 makes four characters of every three bytes.
|
||||
random := make([]byte, bodySize/4*3)
|
||||
@@ -497,7 +497,7 @@ func exportHeapGrowth(t *testing.T, rows, bodySize int) uint64 {
|
||||
for range rows {
|
||||
_, _ = rand.Read(random)
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{
|
||||
require.NoError(t, w.Write(archive.ExportArchivedEvent{
|
||||
Body: base64.StdEncoding.EncodeToString(random),
|
||||
}, 0))
|
||||
}
|
||||
@@ -565,7 +565,7 @@ func TestArchiveExportFileName(t *testing.T) {
|
||||
|
||||
assert.Equal(t,
|
||||
"archive-orders-eu-long-term-archive-20261002T120304Z.json.gz",
|
||||
delivery.ArchiveExportFileName(
|
||||
archive.ExportFileName(
|
||||
exportWebhookName, exportTargetName,
|
||||
time.Date(2026, 10, 2, 14, 3, 4, 0, cest),
|
||||
),
|
||||
@@ -1,7 +1,6 @@
|
||||
package delivery_test
|
||||
package archive_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
@@ -13,8 +12,8 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// deliverTo archives one event to a database target, leaving the
|
||||
@@ -24,12 +23,9 @@ func deliverTo(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"archived":true}`)
|
||||
event := newEvent(`{"archived":true}`)
|
||||
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB, seedDatabaseTargetDelivery(t, webhookDB, event, tgt),
|
||||
)
|
||||
require.NoError(t, env.writers.Write(databaseTargetDelivery(event, tgt)))
|
||||
}
|
||||
|
||||
// TestEvictWebhook_ClosesAndRemovesWriter proves that evicting
|
||||
@@ -41,28 +37,28 @@ func TestEvictWebhook_ClosesAndRemovesWriter(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
first := env.seedDatabaseTarget(t, "")
|
||||
second := env.addDatabaseTarget(t, first.WebhookID, "")
|
||||
other := env.seedDatabaseTarget(t, "")
|
||||
first := env.seedDatabaseTarget(t, 0, "")
|
||||
second := env.addDatabaseTarget(t, first.WebhookID, 0, "")
|
||||
other := env.seedDatabaseTarget(t, 0, "")
|
||||
|
||||
for _, tgt := range []*database.Target{first, second, other} {
|
||||
deliverTo(t, env, tgt)
|
||||
|
||||
require.True(
|
||||
t, env.eng.ExportArchiveHandleOpen(tgt.ID),
|
||||
t, env.writers.ExportArchiveHandleOpen(tgt.ID),
|
||||
"the writer should hold an open handle after a write",
|
||||
)
|
||||
}
|
||||
|
||||
env.eng.EvictWebhook(first.WebhookID)
|
||||
env.writers.EvictWebhook(first.WebhookID)
|
||||
|
||||
for _, tgt := range []*database.Target{first, second} {
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"eviction should remove the registry entry",
|
||||
)
|
||||
assert.False(
|
||||
t, env.eng.ExportArchiveHandleOpen(tgt.ID),
|
||||
t, env.writers.ExportArchiveHandleOpen(tgt.ID),
|
||||
"eviction should close the archive handle",
|
||||
)
|
||||
assert.FileExists(
|
||||
@@ -72,7 +68,7 @@ func TestEvictWebhook_ClosesAndRemovesWriter(t *testing.T) {
|
||||
}
|
||||
|
||||
assert.True(
|
||||
t, env.eng.ExportArchiveHandleOpen(other.ID),
|
||||
t, env.writers.ExportArchiveHandleOpen(other.ID),
|
||||
"another webhook's writer must be left alone",
|
||||
)
|
||||
}
|
||||
@@ -84,21 +80,21 @@ func TestEvictTarget_LeavesOtherTargets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
doomed := env.seedDatabaseTarget(t, "")
|
||||
kept := env.addDatabaseTarget(t, doomed.WebhookID, "")
|
||||
doomed := env.seedDatabaseTarget(t, 0, "")
|
||||
kept := env.addDatabaseTarget(t, doomed.WebhookID, 0, "")
|
||||
|
||||
deliverTo(t, env, doomed)
|
||||
deliverTo(t, env, kept)
|
||||
|
||||
env.eng.EvictTarget(doomed.ID)
|
||||
env.writers.EvictTarget(doomed.ID)
|
||||
|
||||
assert.False(t, env.eng.ExportHasArchiveWriter(doomed.ID))
|
||||
assert.False(t, env.writers.ExportHasArchiveWriter(doomed.ID))
|
||||
assert.FileExists(
|
||||
t, env.archivePath(doomed),
|
||||
"eviction must not delete the archive file",
|
||||
)
|
||||
assert.True(
|
||||
t, env.eng.ExportArchiveHandleOpen(kept.ID),
|
||||
t, env.writers.ExportArchiveHandleOpen(kept.ID),
|
||||
"the other target's writer must be left alone",
|
||||
)
|
||||
}
|
||||
@@ -112,21 +108,21 @@ func TestEvictWebhook_UnknownWebhookIsNoOp(t *testing.T) {
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
assert.NotPanics(t, func() {
|
||||
env.eng.EvictWebhook("no-such-webhook")
|
||||
env.eng.EvictWebhook("no-such-webhook")
|
||||
env.eng.EvictTarget("no-such-target")
|
||||
env.eng.EvictTarget("no-such-target")
|
||||
env.writers.EvictWebhook("no-such-webhook")
|
||||
env.writers.EvictWebhook("no-such-webhook")
|
||||
env.writers.EvictTarget("no-such-target")
|
||||
env.writers.EvictTarget("no-such-target")
|
||||
})
|
||||
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter("no-such-target"),
|
||||
t, env.writers.ExportHasArchiveWriter("no-such-target"),
|
||||
"eviction must not create a writer",
|
||||
)
|
||||
}
|
||||
|
||||
// evictTestRow builds an archive row for the eviction tests.
|
||||
func evictTestRow(eventID string) delivery.ExportArchivedEvent {
|
||||
return delivery.ExportArchivedEvent{
|
||||
func evictTestRow(eventID string) archive.ExportArchivedEvent {
|
||||
return archive.ExportArchivedEvent{
|
||||
EventID: eventID,
|
||||
WebhookID: "wh-evict",
|
||||
Method: http.MethodPost,
|
||||
@@ -147,7 +143,7 @@ func TestEvictedWriter_WriteDoesNotReopenFile(t *testing.T) {
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-evicted.db")
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
@@ -164,7 +160,7 @@ func TestEvictedWriter_WriteDoesNotReopenFile(t *testing.T) {
|
||||
err := w.Write(evictTestRow("ev-2"), 0)
|
||||
|
||||
require.ErrorIs(
|
||||
t, err, delivery.ErrExportArchiveWriterEvicted,
|
||||
t, err, archive.ErrExportArchiveWriterEvicted,
|
||||
"an evicted writer must refuse writes",
|
||||
)
|
||||
assert.NoFileExists(
|
||||
@@ -187,7 +183,7 @@ func TestEvictedWriter_SweepDoesNotReopenFile(t *testing.T) {
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-evicted.db")
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
@@ -199,7 +195,7 @@ func TestEvictedWriter_SweepDoesNotReopenFile(t *testing.T) {
|
||||
err := w.SweepExpired(time.Hour)
|
||||
|
||||
require.ErrorIs(
|
||||
t, err, delivery.ErrExportArchiveWriterEvicted,
|
||||
t, err, archive.ErrExportArchiveWriterEvicted,
|
||||
"an evicted writer must refuse an idle sweep",
|
||||
)
|
||||
assert.False(
|
||||
@@ -230,7 +226,7 @@ const racingWriteGoroutines = 4
|
||||
// against a hang, not a timing assumption: the first write after
|
||||
// the eviction is refused.
|
||||
func startRacingWrites(
|
||||
w *delivery.ExportArchiveWriter,
|
||||
w *archive.ExportArchiveWriter,
|
||||
) *racingWrites {
|
||||
r := &racingWrites{
|
||||
started: make(chan struct{}, racingWriteGoroutines),
|
||||
@@ -276,7 +272,7 @@ func (r *racingWrites) record(err error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
if errors.Is(err, delivery.ErrExportArchiveWriterEvicted) {
|
||||
if errors.Is(err, archive.ErrExportArchiveWriterEvicted) {
|
||||
r.sawEvicted = true
|
||||
|
||||
return
|
||||
@@ -314,13 +310,13 @@ func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
|
||||
// Prime the registry so the test can hold the very writer the
|
||||
// eviction is about to detach.
|
||||
deliverTo(t, env, tgt)
|
||||
|
||||
w := env.eng.ExportArchiveWriterFor(tgt.ID)
|
||||
w := env.writers.ExportArchiveWriterFor(tgt.ID)
|
||||
require.NotNil(t, w)
|
||||
require.True(t, w.HandleOpen())
|
||||
|
||||
@@ -330,7 +326,7 @@ func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
|
||||
// eviction has to contend for the writer's mutex.
|
||||
race.awaitFirstWrite()
|
||||
|
||||
env.eng.EvictWebhook(tgt.WebhookID)
|
||||
env.writers.EvictWebhook(tgt.WebhookID)
|
||||
|
||||
sawEvicted, otherErr := race.wait()
|
||||
|
||||
@@ -345,7 +341,7 @@ func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
|
||||
"been evicted",
|
||||
)
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"the registry entry must stay gone",
|
||||
)
|
||||
}
|
||||
@@ -359,45 +355,45 @@ func TestEvictWebhook_LaterDeliveryRecreatesWriter(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
|
||||
deliverTo(t, env, tgt)
|
||||
require.True(t, env.eng.ExportHasArchiveWriter(tgt.ID))
|
||||
require.True(t, env.writers.ExportHasArchiveWriter(tgt.ID))
|
||||
|
||||
env.eng.EvictWebhook(tgt.WebhookID)
|
||||
env.writers.EvictWebhook(tgt.WebhookID)
|
||||
|
||||
// A fresh delivery for the same target gets a brand new
|
||||
// writer from the registry, so archiving keeps working.
|
||||
deliverTo(t, env, tgt)
|
||||
|
||||
assert.True(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"a later delivery should recreate the writer",
|
||||
)
|
||||
}
|
||||
|
||||
// TestEngineStop_WriteAfterStopIsRefused proves the engine's stop
|
||||
// closes each archive writer the way deleting its webhook does: a
|
||||
// write that reaches a writer after the stop is refused, reopens
|
||||
// nothing and adds no row.
|
||||
func TestEngineStop_WriteAfterStopIsRefused(t *testing.T) {
|
||||
// TestEvictAll_WriteAfterIsRefused proves that EvictAll, which the
|
||||
// delivery engine calls when it stops, closes each archive writer the
|
||||
// way deleting its webhook does: a write that reaches a writer after
|
||||
// it is refused, reopens nothing and adds no row.
|
||||
func TestEvictAll_WriteAfterIsRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
|
||||
deliverTo(t, env, tgt)
|
||||
|
||||
w := env.eng.ExportArchiveWriterFor(tgt.ID)
|
||||
w := env.writers.ExportArchiveWriterFor(tgt.ID)
|
||||
require.NotNil(t, w)
|
||||
require.True(t, w.HandleOpen())
|
||||
|
||||
require.NoError(t, env.eng.ExportStop(context.Background()))
|
||||
env.writers.EvictAll()
|
||||
|
||||
err := w.Write(evictTestRow("ev-after-stop"), 0)
|
||||
|
||||
require.ErrorIs(
|
||||
t, err, delivery.ErrExportArchiveWriterEvicted,
|
||||
t, err, archive.ErrExportArchiveWriterEvicted,
|
||||
"a write after the stop must be refused",
|
||||
)
|
||||
assert.False(
|
||||
@@ -405,7 +401,7 @@ func TestEngineStop_WriteAfterStopIsRefused(t *testing.T) {
|
||||
"a refused write must not reopen the archive",
|
||||
)
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"the stop should empty the registry",
|
||||
)
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
package archive
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// ErrExportArchiveWriterEvicted exposes the sentinel returned by
|
||||
// an evicted archive writer. It carries the Err prefix rather
|
||||
// than this file's usual Export one because it is a sentinel
|
||||
// error.
|
||||
var ErrExportArchiveWriterEvicted = errArchiveWriterEvicted
|
||||
|
||||
// ExportArchivedEvent aliases the archive row type so black-box
|
||||
// tests can construct and read archive rows.
|
||||
type ExportArchivedEvent = archivedEvent
|
||||
|
||||
// ExportArchiveWriter wraps an archiveWriter so black-box tests
|
||||
// can exercise the archive file mechanics.
|
||||
type ExportArchiveWriter struct {
|
||||
w *archiveWriter
|
||||
}
|
||||
|
||||
// NewExportArchiveWriter builds an archive writer for tests,
|
||||
// optionally overriding the reopen debounce (a non-positive
|
||||
// debounce keeps the production default).
|
||||
func NewExportArchiveWriter(
|
||||
path string, log *slog.Logger, debounce time.Duration,
|
||||
) *ExportArchiveWriter {
|
||||
w := newArchiveWriter(path, log)
|
||||
if debounce > 0 {
|
||||
w.debounce = debounce
|
||||
}
|
||||
|
||||
return &ExportArchiveWriter{w: w}
|
||||
}
|
||||
|
||||
// Write archives a row through the writer, into the file named
|
||||
// without a period.
|
||||
func (e *ExportArchiveWriter) Write(
|
||||
row ExportArchivedEvent, expiry time.Duration,
|
||||
) error {
|
||||
return e.w.write(row, expiry, "")
|
||||
}
|
||||
|
||||
// WritePeriod archives a row through the writer, into the file for
|
||||
// period.
|
||||
func (e *ExportArchiveWriter) WritePeriod(
|
||||
row ExportArchivedEvent, expiry time.Duration, period string,
|
||||
) error {
|
||||
return e.w.write(row, expiry, period)
|
||||
}
|
||||
|
||||
// Open opens the archive file, pruning when expiry is positive.
|
||||
func (e *ExportArchiveWriter) Open(expiry time.Duration) error {
|
||||
return e.w.open(e.w.path, expiry)
|
||||
}
|
||||
|
||||
// Reopen closes and reopens the archive file.
|
||||
func (e *ExportArchiveWriter) Reopen(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.reopen(e.w.path, expiry)
|
||||
}
|
||||
|
||||
// SetNow replaces the clock the writer measures its reopen
|
||||
// debounce on.
|
||||
func (e *ExportArchiveWriter) SetNow(now func() time.Time) {
|
||||
e.w.now = now
|
||||
}
|
||||
|
||||
// Reopens reports how many times the file has been opened.
|
||||
func (e *ExportArchiveWriter) Reopens() int {
|
||||
return e.w.reopens
|
||||
}
|
||||
|
||||
// DB returns the writer's current open handle for row
|
||||
// inspection in tests.
|
||||
func (e *ExportArchiveWriter) DB() *gorm.DB {
|
||||
return e.w.db
|
||||
}
|
||||
|
||||
// Path returns the archive file the writer owns.
|
||||
func (e *ExportArchiveWriter) Path() string {
|
||||
return e.w.path
|
||||
}
|
||||
|
||||
// OpenExisting opens the archive without permitting creation,
|
||||
// the way the idle sweep does.
|
||||
func (e *ExportArchiveWriter) OpenExisting(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.openMode(e.w.path, archiveModeExisting, expiry)
|
||||
}
|
||||
|
||||
// SweepExpired runs an idle sweep of the archive.
|
||||
func (e *ExportArchiveWriter) SweepExpired(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.sweepExpired(expiry)
|
||||
}
|
||||
|
||||
// Evict marks the writer evicted and closes its handle, exactly
|
||||
// as leaving the registry does.
|
||||
func (e *ExportArchiveWriter) Evict() {
|
||||
e.w.evict()
|
||||
}
|
||||
|
||||
// Rename gives the archive file a new name in the same directory,
|
||||
// as a rename of the webhook or target does.
|
||||
func (e *ExportArchiveWriter) Rename(name string) error {
|
||||
return e.w.rename(name)
|
||||
}
|
||||
|
||||
// HandleOpen reports whether the writer currently holds an open
|
||||
// archive handle.
|
||||
func (e *ExportArchiveWriter) HandleOpen() bool {
|
||||
e.w.mu.Lock()
|
||||
defer e.w.mu.Unlock()
|
||||
|
||||
return e.w.db != nil
|
||||
}
|
||||
|
||||
// Same reports whether both wrappers refer to the very same
|
||||
// underlying archive writer, so a test can prove a registry entry
|
||||
// is the writer it was handed rather than a replacement.
|
||||
func (e *ExportArchiveWriter) Same(
|
||||
other *ExportArchiveWriter,
|
||||
) bool {
|
||||
return other != nil && e.w == other.w
|
||||
}
|
||||
|
||||
// ExportArchiveWriterFor returns the archive writer the registry
|
||||
// currently caches for a database target, or nil when none is
|
||||
// cached. It never creates one, so a test can hold a reference to
|
||||
// the very writer an eviction is about to detach.
|
||||
func (ws *Writers) ExportArchiveWriterFor(
|
||||
targetID string,
|
||||
) *ExportArchiveWriter {
|
||||
ws.mu.Lock()
|
||||
defer ws.mu.Unlock()
|
||||
|
||||
w, ok := ws.writers[targetID]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &ExportArchiveWriter{w: w}
|
||||
}
|
||||
|
||||
// ExportHasArchiveWriter reports whether the registry currently
|
||||
// caches an archive writer for a target.
|
||||
func (ws *Writers) ExportHasArchiveWriter(
|
||||
targetID string,
|
||||
) bool {
|
||||
ws.mu.Lock()
|
||||
defer ws.mu.Unlock()
|
||||
|
||||
_, ok := ws.writers[targetID]
|
||||
|
||||
return ok
|
||||
}
|
||||
|
||||
// ExportArchiveHandleOpen reports whether the cached archive
|
||||
// writer for a target holds an open database handle. It
|
||||
// returns false when no writer is cached.
|
||||
func (ws *Writers) ExportArchiveHandleOpen(
|
||||
targetID string,
|
||||
) bool {
|
||||
ws.mu.Lock()
|
||||
w, ok := ws.writers[targetID]
|
||||
ws.mu.Unlock()
|
||||
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
return w.db != nil
|
||||
}
|
||||
|
||||
// ExportEnsureArchiveWriter creates (if needed) and returns the
|
||||
// archive file path of the cached writer for a target, so a
|
||||
// test can prime the registry the way a delivery would.
|
||||
func (ws *Writers) ExportEnsureArchiveWriter(
|
||||
targetID string,
|
||||
) (string, error) {
|
||||
w, err := ws.writerFor(targetID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return w.path, nil
|
||||
}
|
||||
|
||||
// ExportSweepWriterFor takes a target's registry writer exactly
|
||||
// as the idle sweep does, reporting whether the sweep had to
|
||||
// create the entry. It lets a test drive the registry through the
|
||||
// sweep's own entry point instead of choreographing goroutines.
|
||||
func (ws *Writers) ExportSweepWriterFor(
|
||||
targetID string,
|
||||
) (*ExportArchiveWriter, bool, error) {
|
||||
w, created, err := ws.sweepWriterFor(targetID)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
|
||||
return &ExportArchiveWriter{w: w}, created, nil
|
||||
}
|
||||
|
||||
// ExportReleaseSweepWriter releases a sweep-created registry entry
|
||||
// exactly as a finished sweep does.
|
||||
func (ws *Writers) ExportReleaseSweepWriter(
|
||||
targetID string, w *ExportArchiveWriter,
|
||||
) {
|
||||
ws.releaseSweepWriter(targetID, w.w)
|
||||
}
|
||||
|
||||
// NewTestSweeper builds a Sweeper backed by the given main
|
||||
// database and archive writers, without the fx lifecycle.
|
||||
// Intended for tests.
|
||||
func NewTestSweeper(
|
||||
db *database.Database,
|
||||
writers *Writers,
|
||||
log *slog.Logger,
|
||||
) *Sweeper {
|
||||
return &Sweeper{
|
||||
db: db,
|
||||
writers: writers,
|
||||
log: log,
|
||||
interval: time.Hour,
|
||||
}
|
||||
}
|
||||
|
||||
// ExportSweep runs a single archive sweep synchronously for
|
||||
// tests.
|
||||
func (s *Sweeper) ExportSweep(ctx context.Context) {
|
||||
s.sweep(ctx)
|
||||
}
|
||||
|
||||
// ExportStart starts the sweeper's background loop for tests.
|
||||
func (s *Sweeper) ExportStart() {
|
||||
s.start()
|
||||
}
|
||||
|
||||
// ExportRegisterHooks registers the sweeper's real fx lifecycle
|
||||
// hooks on a lifecycle supplied by a test, so a test can drive
|
||||
// the exact OnStart/OnStop functions the application runs and
|
||||
// hand OnStart the kind of context fx actually supplies.
|
||||
func (s *Sweeper) ExportRegisterHooks(lc fx.Lifecycle) {
|
||||
s.registerHooks(lc)
|
||||
}
|
||||
|
||||
// ExportStop stops the sweeper's background loop for tests.
|
||||
func (s *Sweeper) ExportStop(ctx context.Context) error {
|
||||
return s.stop(ctx)
|
||||
}
|
||||
|
||||
// ExportWedgeLoop adds a goroutine to the sweeper's WaitGroup
|
||||
// that never observes cancellation and returns only when release
|
||||
// is closed. It stands in for a prune stuck on a locked archive.
|
||||
func (s *Sweeper) ExportWedgeLoop(
|
||||
release <-chan struct{},
|
||||
) {
|
||||
s.wg.Go(func() {
|
||||
<-release
|
||||
})
|
||||
}
|
||||
|
||||
// ExportSetInterval overrides the sweep interval for tests.
|
||||
func (s *Sweeper) ExportSetInterval(d time.Duration) {
|
||||
s.interval = d
|
||||
}
|
||||
@@ -1,7 +1,6 @@
|
||||
package delivery
|
||||
package archive
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -41,10 +40,10 @@ type archiveFile struct {
|
||||
period string
|
||||
}
|
||||
|
||||
// ValidateArchiveRotation checks a user-supplied archive rotation for
|
||||
// ValidateRotation checks a user-supplied archive rotation for
|
||||
// a database target: empty or none (both meaning one file), monthly,
|
||||
// daily or hourly.
|
||||
func ValidateArchiveRotation(rotation string) error {
|
||||
func ValidateRotation(rotation string) error {
|
||||
switch rotation {
|
||||
case "", archiveRotationNone, archiveRotationMonthly,
|
||||
archiveRotationDaily, archiveRotationHourly:
|
||||
@@ -54,35 +53,10 @@ func ValidateArchiveRotation(rotation string) error {
|
||||
}
|
||||
}
|
||||
|
||||
// parseArchiveRotation reads the rotation from a database target's
|
||||
// config JSON. An empty config or an empty rotation is none.
|
||||
func parseArchiveRotation(configJSON string) (string, error) {
|
||||
if configJSON == "" {
|
||||
return archiveRotationNone, nil
|
||||
}
|
||||
|
||||
var cfg databaseTargetConfig
|
||||
|
||||
err := json.Unmarshal([]byte(configJSON), &cfg)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("parsing database target config: %w", err)
|
||||
}
|
||||
|
||||
err = ValidateArchiveRotation(cfg.Rotation)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if cfg.Rotation == "" {
|
||||
return archiveRotationNone, nil
|
||||
}
|
||||
|
||||
return cfg.Rotation, nil
|
||||
}
|
||||
|
||||
// archivePeriod returns the period, in UTC, that a rotation puts an
|
||||
// event received at receivedAt in: "2026-10" for monthly,
|
||||
// "2026-10-01" for daily, "2026-10-01-19" for hourly, and "" for none.
|
||||
// "2026-10-01" for daily, "2026-10-01-19" for hourly, and "" for none
|
||||
// or an empty rotation.
|
||||
func archivePeriod(rotation string, receivedAt time.Time) string {
|
||||
switch rotation {
|
||||
case archiveRotationMonthly:
|
||||
@@ -97,7 +71,7 @@ func archivePeriod(rotation string, receivedAt time.Time) string {
|
||||
}
|
||||
|
||||
// archivePeriodPath returns the path of a database target's archive
|
||||
// file for a period: path, as ArchivePath gives it, with "-" and the
|
||||
// file for a period: path, as Path gives it, with "-" and the
|
||||
// period put before its ".db". The period "" gives path itself.
|
||||
func archivePeriodPath(path, period string) string {
|
||||
if period == "" {
|
||||
@@ -107,28 +81,23 @@ func archivePeriodPath(path, period string) string {
|
||||
return strings.TrimSuffix(path, ".db") + "-" + period + ".db"
|
||||
}
|
||||
|
||||
// ArchivePathAt returns the archive file a database target writes an
|
||||
// event received at receivedAt to: ArchivePath's file, with the
|
||||
// PathAt returns the archive file a database target writes an
|
||||
// event received at receivedAt to: Path's file, with the
|
||||
// period in its name when the target rotates.
|
||||
func ArchivePathAt(
|
||||
func PathAt(
|
||||
dbMgr *database.WebhookDBManager,
|
||||
webhook *database.Webhook,
|
||||
target *database.Target,
|
||||
receivedAt time.Time,
|
||||
) (string, error) {
|
||||
rotation, err := parseArchiveRotation(target.Config)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
) string {
|
||||
return archivePeriodPath(
|
||||
ArchivePath(dbMgr, webhook, target),
|
||||
archivePeriod(rotation, receivedAt),
|
||||
), nil
|
||||
Path(dbMgr, webhook, target),
|
||||
archivePeriod(target.ArchiveRotation, receivedAt),
|
||||
)
|
||||
}
|
||||
|
||||
// archiveFiles lists the database target's archive files that exist,
|
||||
// given the path ArchivePath gives it: the file at path, then each
|
||||
// given the path Path gives it: the file at path, then each
|
||||
// file archivePeriodPath names for path and a period, oldest period
|
||||
// first. Which rotation wrote a file does not matter, so the files of
|
||||
// an earlier rotation setting are listed too.
|
||||
+42
-57
@@ -1,4 +1,4 @@
|
||||
package delivery_test
|
||||
package archive_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
@@ -9,21 +9,17 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// The archive rotations, and the configs of a daily target and of one
|
||||
// whose rotation is not one of the four.
|
||||
// The archive rotations.
|
||||
const (
|
||||
rotationNone = "none"
|
||||
rotationMonthly = "monthly"
|
||||
rotationDaily = "daily"
|
||||
rotationHourly = "hourly"
|
||||
|
||||
dailyConfig = `{"rotation":"daily"}`
|
||||
weeklyConfig = `{"rotation":"weekly"}`
|
||||
)
|
||||
|
||||
// The periods the tests archive into most: two days, and an hour of
|
||||
@@ -40,32 +36,30 @@ func periodPath(path, period string) string {
|
||||
return strings.TrimSuffix(path, ".db") + "-" + period + ".db"
|
||||
}
|
||||
|
||||
// deliverReceivedAt delivers to a database target an event whose
|
||||
// receive time is receivedAt, and returns the event's id. The receive
|
||||
// time is what decides a rotated archive's file, so setting it is how
|
||||
// these tests move the clock across a period boundary.
|
||||
// deliverReceivedAt archives, as a delivery to a database target
|
||||
// does, an event whose receive time is receivedAt, and returns the
|
||||
// event's id. The receive time is what decides a rotated archive's
|
||||
// file, so setting it is how these tests move the clock across a
|
||||
// period boundary.
|
||||
func (env *archiveEnv) deliverReceivedAt(
|
||||
t *testing.T, tgt *database.Target, receivedAt time.Time,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
event := newEvent(`{"n":1}`)
|
||||
event.CreatedAt = receivedAt
|
||||
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB, seedDatabaseTargetDelivery(t, webhookDB, event, tgt),
|
||||
)
|
||||
require.NoError(t, env.writers.Write(databaseTargetDelivery(event, tgt)))
|
||||
|
||||
return event.ID
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_RotatesAtEachPeriodBoundary delivers, for each
|
||||
// TestWrite_RotatesAtEachPeriodBoundary writes, for each
|
||||
// rotation, an event received in the last second of a period and one
|
||||
// received in the first second of the next, and checks each lands in
|
||||
// the file named for its own period, in UTC. Rotation none keeps both
|
||||
// in the one file.
|
||||
func TestDeliverDatabase_RotatesAtEachPeriodBoundary(t *testing.T) {
|
||||
func TestWrite_RotatesAtEachPeriodBoundary(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
berlin := time.FixedZone("CEST", 2*60*60)
|
||||
@@ -110,7 +104,7 @@ func TestDeliverDatabase_RotatesAtEachPeriodBoundary(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+tc.rotation+`"}`)
|
||||
tgt := env.seedDatabaseTarget(t, 0, tc.rotation)
|
||||
path := env.archivePath(tgt)
|
||||
|
||||
first := env.deliverReceivedAt(t, tgt, tc.before)
|
||||
@@ -129,7 +123,7 @@ func TestDeliverDatabase_RotatesAtEachPeriodBoundary(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+rotationNone+`"}`)
|
||||
tgt := env.seedDatabaseTarget(t, 0, rotationNone)
|
||||
|
||||
first := env.deliverReceivedAt(t, tgt, cases[0].before)
|
||||
second := env.deliverReceivedAt(t, tgt, cases[0].after)
|
||||
@@ -139,14 +133,14 @@ func TestDeliverDatabase_RotatesAtEachPeriodBoundary(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_EventWithoutReceiveTime proves an event whose
|
||||
// TestWrite_EventWithoutReceiveTime proves an event whose
|
||||
// receive time is not known goes to the file for the time it is
|
||||
// archived, rather than to one for the year 1.
|
||||
func TestDeliverDatabase_EventWithoutReceiveTime(t *testing.T) {
|
||||
func TestWrite_EventWithoutReceiveTime(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+rotationMonthly+`"}`)
|
||||
tgt := env.seedDatabaseTarget(t, 0, rotationMonthly)
|
||||
path := env.archivePath(tgt)
|
||||
|
||||
before := time.Now().UTC().Format("2006-01")
|
||||
@@ -164,20 +158,20 @@ func TestDeliverDatabase_EventWithoutReceiveTime(t *testing.T) {
|
||||
assert.NoFileExists(t, periodPath(path, "0001-01"))
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_RotationChangeKeepsOldFiles changes a target's
|
||||
// TestWrite_RotationChangeKeepsOldFiles changes a target's
|
||||
// rotation from none to daily between two events, and checks the
|
||||
// second goes to the daily file while the first stays where it was.
|
||||
func TestDeliverDatabase_RotationChangeKeepsOldFiles(t *testing.T) {
|
||||
func TestWrite_RotationChangeKeepsOldFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
tgt := env.seedDatabaseTarget(t, 0, rotationNone)
|
||||
path := env.archivePath(tgt)
|
||||
at := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
first := env.deliverReceivedAt(t, tgt, at)
|
||||
|
||||
tgt.Config = dailyConfig
|
||||
tgt.ArchiveRotation = rotationDaily
|
||||
second := env.deliverReceivedAt(t, tgt, at)
|
||||
|
||||
assert.Equal(t, []string{first}, archivedEventIDs(t, path))
|
||||
@@ -195,9 +189,7 @@ func TestArchiveSweep_PrunesEveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(
|
||||
t, `{"expiry":"1h","rotation":"`+rotationDaily+`"}`,
|
||||
)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, rotationDaily)
|
||||
path := env.archivePath(tgt)
|
||||
emptied := periodPath(path, dayPeriod)
|
||||
kept := periodPath(path, nextDayPeriod)
|
||||
@@ -227,17 +219,17 @@ func TestRename_MovesEveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
tgt := env.seedDatabaseTarget(t, 0, rotationNone)
|
||||
oldPath := env.archivePath(tgt)
|
||||
day := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
unrotated := env.deliverReceivedAt(t, tgt, day)
|
||||
|
||||
tgt.Config = dailyConfig
|
||||
tgt.ArchiveRotation = rotationDaily
|
||||
first := env.deliverReceivedAt(t, tgt, day)
|
||||
second := env.deliverReceivedAt(t, tgt, day.Add(24*time.Hour))
|
||||
|
||||
require.NoError(t, env.eng.Rename(tgt.ID, "Orders", "Long Term"))
|
||||
require.NoError(t, env.writers.Rename(tgt.ID, "Orders", "Long Term"))
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
@@ -269,7 +261,7 @@ func TestRename_NeverReplacesARotatedFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, dailyConfig)
|
||||
tgt := env.seedDatabaseTarget(t, 0, rotationDaily)
|
||||
oldPath := env.archivePath(tgt)
|
||||
day := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
@@ -283,8 +275,8 @@ func TestRename_NeverReplacesARotatedFile(t *testing.T) {
|
||||
require.NoError(t, os.WriteFile(planted, []byte("planted"), 0o600))
|
||||
|
||||
require.ErrorIs(
|
||||
t, env.eng.Rename(tgt.ID, "Orders", "Long Term"),
|
||||
delivery.ErrArchiveNameTaken,
|
||||
t, env.writers.Rename(tgt.ID, "Orders", "Long Term"),
|
||||
archive.ErrNameTaken,
|
||||
)
|
||||
|
||||
assert.FileExists(t, periodPath(oldPath, dayPeriod))
|
||||
@@ -292,7 +284,7 @@ func TestRename_NeverReplacesARotatedFile(t *testing.T) {
|
||||
assert.NoFileExists(t, periodPath(newPath, dayPeriod))
|
||||
}
|
||||
|
||||
// TestStatArchive_EveryFile proves StatArchive counts and adds up every
|
||||
// TestStatArchive_EveryFile proves Stat counts and adds up every
|
||||
// one of a target's files, takes the latest write of any of them, and
|
||||
// leaves out files whose names only look like the target's.
|
||||
func TestStatArchive_EveryFile(t *testing.T) {
|
||||
@@ -324,7 +316,7 @@ func TestStatArchive_EveryFile(t *testing.T) {
|
||||
os.WriteFile(filepath.Join(dir, other), []byte("x"), 0o600))
|
||||
}
|
||||
|
||||
got, err := delivery.StatArchive(path)
|
||||
got, err := archive.Stat(path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, len(files), got.Files)
|
||||
assert.Equal(t, size, got.Size)
|
||||
@@ -343,34 +335,27 @@ func TestArchivePathAt(t *testing.T) {
|
||||
at := time.Date(2026, 10, 1, 19, 30, 0, 0, time.UTC)
|
||||
|
||||
cases := map[string]string{
|
||||
"": "",
|
||||
`{"rotation":"` + rotationNone + `"}`: "",
|
||||
`{"rotation":"` + rotationMonthly + `"}`: "-2026-10",
|
||||
dailyConfig: "-2026-10-01",
|
||||
`{"rotation":"` + rotationHourly + `"}`: "-2026-10-01-19",
|
||||
"": "",
|
||||
rotationNone: "",
|
||||
rotationMonthly: "-2026-10",
|
||||
rotationDaily: "-2026-10-01",
|
||||
rotationHourly: "-2026-10-01-19",
|
||||
}
|
||||
|
||||
for config, period := range cases {
|
||||
for rotation, period := range cases {
|
||||
target := &database.Target{
|
||||
BaseModel: database.BaseModel{ID: "tgt-id"},
|
||||
Name: "Archive",
|
||||
Config: config,
|
||||
BaseModel: database.BaseModel{ID: "tgt-id"},
|
||||
Name: "Archive",
|
||||
ArchiveRotation: rotation,
|
||||
}
|
||||
|
||||
got, err := delivery.ArchivePathAt(dbMgr, webhook, target, at)
|
||||
require.NoError(t, err, config)
|
||||
assert.Equal(t,
|
||||
filepath.Join(
|
||||
dataDir, "archive-orders-archive-tgt-id"+period+".db",
|
||||
),
|
||||
got, config,
|
||||
archive.PathAt(dbMgr, webhook, target, at), rotation,
|
||||
)
|
||||
}
|
||||
|
||||
_, err := delivery.ArchivePathAt(dbMgr, webhook, &database.Target{
|
||||
Config: weeklyConfig,
|
||||
}, at)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
// TestValidateArchiveRotation accepts the four rotations, and empty,
|
||||
@@ -381,10 +366,10 @@ func TestValidateArchiveRotation(t *testing.T) {
|
||||
for _, ok := range []string{
|
||||
"", rotationNone, rotationMonthly, rotationDaily, rotationHourly,
|
||||
} {
|
||||
require.NoError(t, delivery.ValidateArchiveRotation(ok), ok)
|
||||
require.NoError(t, archive.ValidateRotation(ok), ok)
|
||||
}
|
||||
|
||||
for _, bad := range []string{"weekly", "Daily", "hourly "} {
|
||||
require.Error(t, delivery.ValidateArchiveRotation(bad), bad)
|
||||
require.Error(t, archive.ValidateRotation(bad), bad)
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package delivery
|
||||
package archive
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -15,18 +15,18 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
)
|
||||
|
||||
// ArchiveSweeperParams holds the fx dependencies for the
|
||||
// ArchiveSweeper.
|
||||
type ArchiveSweeperParams struct {
|
||||
// SweeperParams holds the fx dependencies for the
|
||||
// Sweeper.
|
||||
type SweeperParams struct {
|
||||
fx.In
|
||||
|
||||
Config *config.Config
|
||||
Database *database.Database
|
||||
Engine *Engine
|
||||
Writers *Writers
|
||||
Logger *logger.Logger
|
||||
}
|
||||
|
||||
// ArchiveSweeper periodically prunes expired rows from the
|
||||
// Sweeper periodically prunes expired rows from the
|
||||
// archive databases of database targets that carry a positive
|
||||
// expiry.
|
||||
//
|
||||
@@ -40,9 +40,9 @@ type ArchiveSweeperParams struct {
|
||||
// It reuses Config.RetentionSweepInterval rather than
|
||||
// introducing a second interval: this is a retention sweep with
|
||||
// the same semantics as the event retention reaper.
|
||||
type ArchiveSweeper struct {
|
||||
type Sweeper struct {
|
||||
db *database.Database
|
||||
eng *Engine
|
||||
writers *Writers
|
||||
log *slog.Logger
|
||||
interval time.Duration
|
||||
|
||||
@@ -54,16 +54,16 @@ type ArchiveSweeper struct {
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
// NewArchiveSweeper creates the archive sweeper and registers
|
||||
// NewSweeper creates the archive sweeper and registers
|
||||
// its fx lifecycle hooks. The background sweep loop starts on
|
||||
// OnStart and stops cleanly on OnStop via context cancellation.
|
||||
func NewArchiveSweeper(
|
||||
func NewSweeper(
|
||||
lc fx.Lifecycle,
|
||||
params ArchiveSweeperParams,
|
||||
) *ArchiveSweeper {
|
||||
s := &ArchiveSweeper{
|
||||
params SweeperParams,
|
||||
) *Sweeper {
|
||||
s := &Sweeper{
|
||||
db: params.Database,
|
||||
eng: params.Engine,
|
||||
writers: params.Writers,
|
||||
log: params.Logger.Get(),
|
||||
interval: params.Config.RetentionSweepInterval,
|
||||
}
|
||||
@@ -77,7 +77,7 @@ func NewArchiveSweeper(
|
||||
// lifecycle. The start hook's context is deliberately ignored
|
||||
// (see start for why the background loop must not inherit it);
|
||||
// the stop hook's context is honoured (see stop).
|
||||
func (s *ArchiveSweeper) registerHooks(lc fx.Lifecycle) {
|
||||
func (s *Sweeper) registerHooks(lc fx.Lifecycle) {
|
||||
lc.Append(fx.Hook{
|
||||
//nolint:contextcheck // Not passing the hook context is
|
||||
// the point: see start.
|
||||
@@ -103,7 +103,7 @@ func (s *ArchiveSweeper) registerHooks(lc fx.Lifecycle) {
|
||||
// goroutine must outlive the startup phase, so its lifetime is
|
||||
// bounded by OnStop instead: stop cancels this context and waits
|
||||
// on the WaitGroup.
|
||||
func (s *ArchiveSweeper) start() {
|
||||
func (s *Sweeper) start() {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s.cancel = cancel
|
||||
|
||||
@@ -121,7 +121,7 @@ func (s *ArchiveSweeper) start() {
|
||||
// exit, bounded by the stop hook's context: a prune wedged on a
|
||||
// locked archive must not hang the process past fx's stop
|
||||
// timeout.
|
||||
func (s *ArchiveSweeper) stop(ctx context.Context) error {
|
||||
func (s *Sweeper) stop(ctx context.Context) error {
|
||||
s.log.Info("archive sweeper stopping")
|
||||
|
||||
if s.cancel != nil {
|
||||
@@ -140,7 +140,7 @@ func (s *ArchiveSweeper) stop(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *ArchiveSweeper) run(ctx context.Context) {
|
||||
func (s *Sweeper) run(ctx context.Context) {
|
||||
defer s.wg.Done()
|
||||
|
||||
ticker := time.NewTicker(s.interval)
|
||||
@@ -164,7 +164,7 @@ func (s *ArchiveSweeper) run(ctx context.Context) {
|
||||
// A failure for one target is logged and the sweep continues,
|
||||
// matching how the write path already treats a prune error as
|
||||
// non-fatal.
|
||||
func (s *ArchiveSweeper) sweep(ctx context.Context) {
|
||||
func (s *Sweeper) sweep(ctx context.Context) {
|
||||
var targets []database.Target
|
||||
|
||||
err := s.db.DB().
|
||||
@@ -199,31 +199,20 @@ func (s *ArchiveSweeper) sweep(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// sweepTarget prunes the archive of a single database target.
|
||||
// A missing, empty, or "never" expiry parses as a zero duration
|
||||
// and is skipped entirely, so those archives are never pruned.
|
||||
func (s *ArchiveSweeper) sweepTarget(target *database.Target) {
|
||||
expiry, err := parseArchiveExpiry(target.Config)
|
||||
if err != nil {
|
||||
s.log.Error(
|
||||
"archive sweep: invalid database target config",
|
||||
"webhook_id", target.WebhookID,
|
||||
"target_id", target.ID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// sweepTarget prunes the archive of a single database target. A
|
||||
// target with an ArchiveExpiry of 0 keeps its archive forever and is
|
||||
// skipped entirely, so that archive is never pruned.
|
||||
func (s *Sweeper) sweepTarget(target *database.Target) {
|
||||
expiry := target.ArchiveExpiry
|
||||
if expiry <= 0 {
|
||||
return
|
||||
}
|
||||
|
||||
if s.eng == nil || s.eng.dbTarget == nil {
|
||||
if s.writers == nil {
|
||||
return
|
||||
}
|
||||
|
||||
err = s.eng.dbTarget.sweepArchive(target.ID, expiry)
|
||||
err := s.writers.sweepArchive(target.ID, expiry)
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package delivery_test
|
||||
package archive_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -16,14 +16,15 @@ import (
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.uber.org/fx"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
@@ -37,6 +38,18 @@ const (
|
||||
// sweepConcurrentWrites is how many deliveries the
|
||||
// concurrent write-plus-sweep test races against the sweep.
|
||||
sweepConcurrentWrites = 20
|
||||
|
||||
// hookStopTimeout bounds how long a lifecycle test waits for
|
||||
// the sweeper's OnStop hook to return before declaring the
|
||||
// shutdown hung.
|
||||
hookStopTimeout = 10 * time.Second
|
||||
|
||||
// wedgeStopTimeout is the stop timeout a wedged-shutdown test
|
||||
// hands OnStop, standing in for fx's StopTimeout. The test
|
||||
// asserts only that the hook returns at all, and allows it
|
||||
// hookStopTimeout — forty times this budget — to do so, so no
|
||||
// assertion here races the wall clock.
|
||||
wedgeStopTimeout = 250 * time.Millisecond
|
||||
)
|
||||
|
||||
// archiveTestWebhookName is the name of every webhook
|
||||
@@ -45,12 +58,12 @@ const (
|
||||
const archiveTestWebhookName = "Sweep Test!"
|
||||
|
||||
// archiveEnv bundles the pieces an archive test drives: a main
|
||||
// configuration database holding webhooks and targets, a delivery
|
||||
// engine owning the archive writer registry, the archive sweeper,
|
||||
// and the data directory the archive files live in.
|
||||
// configuration database holding webhooks and targets, the archive
|
||||
// writers, the archive sweeper, and the data directory the archive
|
||||
// files live in.
|
||||
type archiveEnv struct {
|
||||
sweeper *delivery.ArchiveSweeper
|
||||
eng *delivery.Engine
|
||||
sweeper *archive.Sweeper
|
||||
writers *archive.Writers
|
||||
mainDB *database.Database
|
||||
dataDir string
|
||||
}
|
||||
@@ -66,39 +79,37 @@ func setupArchiveTest(t *testing.T) *archiveEnv {
|
||||
|
||||
t.Cleanup(func() { _ = mainDB.Close() })
|
||||
|
||||
eng := delivery.NewTestEngineWithDB(
|
||||
mainDB,
|
||||
databasetest.NewWebhookDBManager(t, dataDir),
|
||||
log,
|
||||
&http.Client{Timeout: 5 * time.Second},
|
||||
1,
|
||||
)
|
||||
writers := archive.NewWriters(archive.WritersParams{
|
||||
Database: mainDB,
|
||||
DBManager: databasetest.NewWebhookDBManager(t, dataDir),
|
||||
Logger: log,
|
||||
})
|
||||
|
||||
return &archiveEnv{
|
||||
sweeper: delivery.NewTestArchiveSweeper(
|
||||
mainDB, eng, log,
|
||||
sweeper: archive.NewTestSweeper(
|
||||
mainDB, writers, log,
|
||||
),
|
||||
eng: eng,
|
||||
writers: writers,
|
||||
mainDB: mainDB,
|
||||
dataDir: dataDir,
|
||||
}
|
||||
}
|
||||
|
||||
// archivePath returns where the engine keeps a database target's
|
||||
// archivePath returns where the writers keep a database target's
|
||||
// archive file, for the names seedDatabaseTarget gave it.
|
||||
func (env *archiveEnv) archivePath(tgt *database.Target) string {
|
||||
return filepath.Join(
|
||||
env.dataDir,
|
||||
delivery.ArchiveFileName(
|
||||
archive.FileName(
|
||||
archiveTestWebhookName, tgt.Name, tgt.ID,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
// seedDatabaseTarget creates a webhook with one database target
|
||||
// carrying the given target config JSON, and returns the target.
|
||||
// seedDatabaseTarget creates a webhook with one database target with
|
||||
// the given archive expiry and rotation, and returns the target.
|
||||
func (env *archiveEnv) seedDatabaseTarget(
|
||||
t *testing.T, configJSON string,
|
||||
t *testing.T, expiry time.Duration, rotation string,
|
||||
) *database.Target {
|
||||
t.Helper()
|
||||
|
||||
@@ -113,23 +124,27 @@ func (env *archiveEnv) seedDatabaseTarget(
|
||||
Create(wh).Error,
|
||||
)
|
||||
|
||||
return env.addDatabaseTarget(t, wh.ID, configJSON)
|
||||
return env.addDatabaseTarget(t, wh.ID, expiry, rotation)
|
||||
}
|
||||
|
||||
// addDatabaseTarget creates one more database target on an
|
||||
// existing webhook and returns it.
|
||||
func (env *archiveEnv) addDatabaseTarget(
|
||||
t *testing.T, webhookID, configJSON string,
|
||||
t *testing.T,
|
||||
webhookID string,
|
||||
expiry time.Duration,
|
||||
rotation string,
|
||||
) *database.Target {
|
||||
t.Helper()
|
||||
|
||||
tgt := &database.Target{
|
||||
WebhookID: webhookID,
|
||||
Name: "Archive",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Active: true,
|
||||
Config: configJSON,
|
||||
MaxAttempts: 1,
|
||||
WebhookID: webhookID,
|
||||
Name: "Archive",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Active: true,
|
||||
ArchiveExpiry: expiry,
|
||||
ArchiveRotation: rotation,
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
require.NoError(
|
||||
t,
|
||||
@@ -177,7 +192,7 @@ func seedArchiveFile(
|
||||
require.NoError(t, migrations.Apply(gdb, migrations.Archive))
|
||||
|
||||
for i, at := range archivedAt {
|
||||
row := delivery.ExportArchivedEvent{
|
||||
row := archive.ExportArchivedEvent{
|
||||
EventID: fmt.Sprintf("ev-%d", i),
|
||||
WebhookID: webhookID,
|
||||
Method: http.MethodPost,
|
||||
@@ -197,7 +212,7 @@ func archivedEventIDs(
|
||||
) []string {
|
||||
t.Helper()
|
||||
|
||||
var rows []delivery.ExportArchivedEvent
|
||||
var rows []archive.ExportArchivedEvent
|
||||
|
||||
rdb := openArchiveDBForRead(t, path)
|
||||
require.NoError(t, rdb.Order("event_id").Find(&rows).Error)
|
||||
@@ -234,7 +249,7 @@ func countArchivedRows(path string) (int64, error) {
|
||||
|
||||
var count int64
|
||||
|
||||
err = gdb.Model(&delivery.ExportArchivedEvent{}).
|
||||
err = gdb.Model(&archive.ExportArchivedEvent{}).
|
||||
Count(&count).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
@@ -243,6 +258,55 @@ func countArchivedRows(path string) (int64, error) {
|
||||
return count, nil
|
||||
}
|
||||
|
||||
// recordingLifecycle is a minimal fx.Lifecycle that records the
|
||||
// hooks a component registers, so a test can invoke the real
|
||||
// OnStart/OnStop functions with a context of its choosing.
|
||||
type recordingLifecycle struct {
|
||||
hooks []fx.Hook
|
||||
}
|
||||
|
||||
func (l *recordingLifecycle) Append(h fx.Hook) {
|
||||
l.hooks = append(l.hooks, h)
|
||||
}
|
||||
|
||||
// requireStopHookExpires drives hook.OnStop with a stop context
|
||||
// that expires while a wedged goroutine is still running, and
|
||||
// requires the hook to return the deadline error naming
|
||||
// component instead of blocking on the WaitGroup forever.
|
||||
func requireStopHookExpires(
|
||||
t *testing.T, hook fx.Hook, component string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
stopCtx, cancel := context.WithTimeout(
|
||||
context.Background(), wedgeStopTimeout,
|
||||
)
|
||||
defer cancel()
|
||||
|
||||
var stopErr error
|
||||
|
||||
stopped := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
defer close(stopped)
|
||||
|
||||
stopErr = hook.OnStop(stopCtx)
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-stopped:
|
||||
case <-time.After(hookStopTimeout):
|
||||
t.Fatal(
|
||||
"OnStop did not return: it discarded the stop " +
|
||||
"context and is waiting on a wedged goroutine " +
|
||||
"that will never observe cancellation",
|
||||
)
|
||||
}
|
||||
|
||||
require.ErrorIs(t, stopErr, context.DeadlineExceeded)
|
||||
require.ErrorContains(t, stopErr, component)
|
||||
}
|
||||
|
||||
// TestArchiveSweeper_LoopOutlivesStartHookContext is the
|
||||
// regression test for a sweeper that never swept. fx calls
|
||||
// OnStart with a context carrying the application's start
|
||||
@@ -263,7 +327,7 @@ func TestArchiveSweeper_LoopOutlivesStartHookContext(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
|
||||
now := time.Now()
|
||||
path := env.seedArchiveRows(
|
||||
@@ -317,7 +381,7 @@ func TestArchiveSweep_DoesNotResurrectEvictedWriter(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
env.seedArchiveRows(
|
||||
t, tgt, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
@@ -326,16 +390,16 @@ func TestArchiveSweep_DoesNotResurrectEvictedWriter(
|
||||
// the deletion path does. The target row is deliberately left
|
||||
// in place: this is the tick that listed the target before
|
||||
// the deletion committed.
|
||||
_, err := env.eng.ExportEnsureArchiveWriter(tgt.ID)
|
||||
_, err := env.writers.ExportEnsureArchiveWriter(tgt.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
env.eng.EvictTarget(tgt.ID)
|
||||
require.False(t, env.eng.ExportHasArchiveWriter(tgt.ID))
|
||||
env.writers.EvictTarget(tgt.ID)
|
||||
require.False(t, env.writers.ExportHasArchiveWriter(tgt.ID))
|
||||
|
||||
env.sweeper.ExportSweep(context.Background())
|
||||
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"a sweep must never re-register a writer for a target "+
|
||||
"whose registry entry has already been released",
|
||||
)
|
||||
@@ -351,14 +415,14 @@ func TestArchiveSweep_LeavesNoRegistryEntry(t *testing.T) {
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
path := env.seedArchiveRows(
|
||||
t, tgt,
|
||||
time.Now().Add(-48*time.Hour),
|
||||
time.Now().Add(-time.Minute),
|
||||
)
|
||||
|
||||
require.False(t, env.eng.ExportHasArchiveWriter(tgt.ID))
|
||||
require.False(t, env.writers.ExportHasArchiveWriter(tgt.ID))
|
||||
|
||||
env.sweeper.ExportSweep(context.Background())
|
||||
|
||||
@@ -367,7 +431,7 @@ func TestArchiveSweep_LeavesNoRegistryEntry(t *testing.T) {
|
||||
"the sweep must still prune an idle archive",
|
||||
)
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"the sweep must release the registry entry it created",
|
||||
)
|
||||
}
|
||||
@@ -384,29 +448,27 @@ func TestArchiveSweep_KeepsWriterAdoptedByDelivery(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
env.seedArchiveRows(
|
||||
t, tgt, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
d := seedDatabaseTargetDelivery(t, webhookDB, event, tgt)
|
||||
d := databaseTargetDelivery(newEvent(`{"n":1}`), tgt)
|
||||
|
||||
env.sweeper.ExportSweep(context.Background())
|
||||
require.False(t, env.eng.ExportHasArchiveWriter(tgt.ID))
|
||||
require.False(t, env.writers.ExportHasArchiveWriter(tgt.ID))
|
||||
|
||||
env.eng.ExportDeliverDatabase(webhookDB, d)
|
||||
require.NoError(t, env.writers.Write(d))
|
||||
|
||||
assert.True(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"a delivery's writer must stay registered",
|
||||
)
|
||||
|
||||
env.sweeper.ExportSweep(context.Background())
|
||||
|
||||
assert.True(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"a sweep must not drop a writer a delivery owns",
|
||||
)
|
||||
}
|
||||
@@ -440,12 +502,12 @@ func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
env.seedArchiveRows(
|
||||
t, tgt, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
|
||||
sweepWriter, created, err := env.eng.ExportSweepWriterFor(
|
||||
sweepWriter, created, err := env.writers.ExportSweepWriterFor(
|
||||
tgt.ID,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
@@ -455,36 +517,34 @@ func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
|
||||
)
|
||||
|
||||
// The delivery lands mid-sweep and adopts the entry.
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
d := seedDatabaseTargetDelivery(t, webhookDB, event, tgt)
|
||||
d := databaseTargetDelivery(newEvent(`{"n":1}`), tgt)
|
||||
|
||||
env.eng.ExportDeliverDatabase(webhookDB, d)
|
||||
require.NoError(t, env.writers.Write(d))
|
||||
|
||||
adopted := env.eng.ExportArchiveWriterFor(tgt.ID)
|
||||
adopted := env.writers.ExportArchiveWriterFor(tgt.ID)
|
||||
require.NotNil(t, adopted)
|
||||
require.True(
|
||||
t, sweepWriter.Same(adopted),
|
||||
"the delivery must have adopted the sweep's writer",
|
||||
)
|
||||
require.True(
|
||||
t, env.eng.ExportArchiveHandleOpen(tgt.ID),
|
||||
t, env.writers.ExportArchiveHandleOpen(tgt.ID),
|
||||
"the delivery leaves the archive handle open",
|
||||
)
|
||||
|
||||
// The sweep finishes.
|
||||
env.eng.ExportReleaseSweepWriter(tgt.ID, sweepWriter)
|
||||
env.writers.ExportReleaseSweepWriter(tgt.ID, sweepWriter)
|
||||
|
||||
require.True(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"a writer adopted by a delivery during a sweep must "+
|
||||
"stay registered, or its open handle is unreachable",
|
||||
)
|
||||
|
||||
env.eng.EvictTarget(tgt.ID)
|
||||
env.writers.EvictTarget(tgt.ID)
|
||||
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"the adopted writer must still be evictable",
|
||||
)
|
||||
assert.False(
|
||||
@@ -495,8 +555,8 @@ func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
|
||||
|
||||
// TestArchiveSweep_ContinuesAfterPerTargetFailure proves a
|
||||
// failure for one target does not abort the sweep for the
|
||||
// others: an unparseable expiry and an unreadable archive both
|
||||
// have to be logged and stepped over.
|
||||
// others: an unreadable archive has to be logged and stepped
|
||||
// over.
|
||||
func TestArchiveSweep_ContinuesAfterPerTargetFailure(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -504,21 +564,16 @@ func TestArchiveSweep_ContinuesAfterPerTargetFailure(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
// Seeded first so the sweep reaches them before the healthy
|
||||
// Seeded first so the sweep reaches it before the healthy
|
||||
// target: targets come back in insertion order.
|
||||
badConfig := env.seedDatabaseTarget(t, `{"expiry":"!!!"}`)
|
||||
env.seedArchiveRows(
|
||||
t, badConfig, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
|
||||
corrupt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
corrupt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
require.NoError(t, os.WriteFile(
|
||||
env.archivePath(corrupt),
|
||||
[]byte("this is not a sqlite database"),
|
||||
0o600,
|
||||
))
|
||||
|
||||
healthy := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
healthy := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
healthyPath := env.seedArchiveRows(
|
||||
t, healthy,
|
||||
time.Now().Add(-48*time.Hour),
|
||||
@@ -548,7 +603,7 @@ func TestArchiveSweep_OpenExistingDoesNotCreateFile(
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "archive-absent.db")
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
@@ -575,7 +630,7 @@ func TestArchiveSweep_PrunesIdleArchive(t *testing.T) {
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
|
||||
now := time.Now()
|
||||
path := env.seedArchiveRows(
|
||||
@@ -614,12 +669,12 @@ func TestArchiveSweep_LeavesArchiveClosed(t *testing.T) {
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
path := env.seedArchiveRows(
|
||||
t, tgt, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
@@ -654,30 +709,28 @@ func TestArchiveSweep_ClosesHandleOfRegisteredWriter(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
env.seedArchiveRows(
|
||||
t, tgt, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
d := seedDatabaseTargetDelivery(t, webhookDB, event, tgt)
|
||||
d := databaseTargetDelivery(newEvent(`{"n":1}`), tgt)
|
||||
|
||||
env.eng.ExportDeliverDatabase(webhookDB, d)
|
||||
require.NoError(t, env.writers.Write(d))
|
||||
|
||||
require.True(
|
||||
t, env.eng.ExportArchiveHandleOpen(tgt.ID),
|
||||
t, env.writers.ExportArchiveHandleOpen(tgt.ID),
|
||||
"the delivery must leave the archive handle open",
|
||||
)
|
||||
|
||||
env.sweeper.ExportSweep(context.Background())
|
||||
|
||||
require.True(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"the delivery's registry entry must survive the sweep",
|
||||
)
|
||||
assert.False(
|
||||
t, env.eng.ExportArchiveHandleOpen(tgt.ID),
|
||||
t, env.writers.ExportArchiveHandleOpen(tgt.ID),
|
||||
"the sweep must leave the archive closed",
|
||||
)
|
||||
}
|
||||
@@ -693,7 +746,7 @@ func TestArchiveSweep_ClosesHandleBeforeReopening(t *testing.T) {
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
w := archive.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
@@ -721,8 +774,8 @@ func TestArchiveSweep_CancelledSweepLogsNoError(t *testing.T) {
|
||||
|
||||
var errorLines bytes.Buffer
|
||||
|
||||
sweeper := delivery.NewTestArchiveSweeper(
|
||||
env.mainDB, env.eng,
|
||||
sweeper := archive.NewTestSweeper(
|
||||
env.mainDB, env.writers,
|
||||
slog.New(slog.NewTextHandler(
|
||||
&errorLines,
|
||||
&slog.HandlerOptions{Level: slog.LevelError},
|
||||
@@ -741,36 +794,29 @@ func TestArchiveSweep_CancelledSweepLogsNoError(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestArchiveSweep_NeverExpiryUntouched proves the sweep is a
|
||||
// no-op for the default retention policy, so archives with no
|
||||
// expiry (or the literal "never") behave exactly as before.
|
||||
// no-op for the default retention policy, so an archive with an
|
||||
// expiry of 0, which keeps it forever, is left alone.
|
||||
func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, configJSON := range []string{
|
||||
`{"expiry":"never"}`,
|
||||
`{"expiry":""}`,
|
||||
"",
|
||||
} {
|
||||
env := setupArchiveTest(t)
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, configJSON)
|
||||
path := env.seedArchiveRows(
|
||||
t, tgt,
|
||||
time.Now().Add(-10000*time.Hour),
|
||||
)
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
path := env.seedArchiveRows(
|
||||
t, tgt,
|
||||
time.Now().Add(-10000*time.Hour),
|
||||
)
|
||||
|
||||
env.sweeper.ExportSweep(context.Background())
|
||||
env.sweeper.ExportSweep(context.Background())
|
||||
|
||||
assert.Equal(
|
||||
t, []string{sweepRowOld}, archivedEventIDs(t, path),
|
||||
"config %q must keep rows forever", configJSON,
|
||||
)
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
"config %q must leave no registry entry behind",
|
||||
configJSON,
|
||||
)
|
||||
}
|
||||
assert.Equal(
|
||||
t, []string{sweepRowOld}, archivedEventIDs(t, path),
|
||||
"an expiry of 0 must keep rows forever",
|
||||
)
|
||||
assert.False(
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"an expiry of 0 must leave no registry entry behind",
|
||||
)
|
||||
}
|
||||
|
||||
// TestArchiveSweep_NeverExpirySkipsBeforeOpening pins the
|
||||
@@ -791,7 +837,7 @@ func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"never"}`)
|
||||
tgt := env.seedDatabaseTarget(t, 0, "")
|
||||
path := env.archivePath(tgt)
|
||||
|
||||
seedUnmigratedArchive(t, path)
|
||||
@@ -831,7 +877,7 @@ func archiveTableExists(t *testing.T, path string) bool {
|
||||
|
||||
return openArchiveDBForRead(t, path).
|
||||
Migrator().
|
||||
HasTable(&delivery.ExportArchivedEvent{})
|
||||
HasTable(&archive.ExportArchivedEvent{})
|
||||
}
|
||||
|
||||
// TestArchiveSweep_DoesNotCreateArchiveFile proves the sweep
|
||||
@@ -843,7 +889,7 @@ func TestArchiveSweep_DoesNotCreateArchiveFile(t *testing.T) {
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
path := env.archivePath(tgt)
|
||||
|
||||
require.NoFileExists(t, path)
|
||||
@@ -858,7 +904,7 @@ func TestArchiveSweep_DoesNotCreateArchiveFile(t *testing.T) {
|
||||
}
|
||||
|
||||
assert.False(
|
||||
t, env.eng.ExportHasArchiveWriter(tgt.ID),
|
||||
t, env.writers.ExportHasArchiveWriter(tgt.ID),
|
||||
"the sweep must leave no registry entry behind",
|
||||
)
|
||||
}
|
||||
@@ -874,9 +920,9 @@ func TestArchiveSweep_DoesNotCreateAfterWriterExists(
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
|
||||
path, err := env.eng.ExportEnsureArchiveWriter(tgt.ID)
|
||||
path, err := env.writers.ExportEnsureArchiveWriter(tgt.ID)
|
||||
require.NoError(t, err)
|
||||
require.NoFileExists(t, path)
|
||||
|
||||
@@ -893,7 +939,7 @@ func TestArchiveSweep_SkipsDeletedWebhookTargets(t *testing.T) {
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
path := env.seedArchiveRows(
|
||||
t, tgt, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
@@ -921,23 +967,16 @@ func TestArchiveSweep_ConcurrentWrites(t *testing.T) {
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
|
||||
// The deliveries are seeded up front, on the test's own
|
||||
// goroutine: the seed helpers assert, and testify assertions
|
||||
// must not run off the test goroutine.
|
||||
deliveries := make(
|
||||
[]*database.Delivery, 0, sweepConcurrentWrites,
|
||||
)
|
||||
|
||||
for range sweepConcurrentWrites {
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
|
||||
deliveries = append(
|
||||
deliveries,
|
||||
seedDatabaseTargetDelivery(t, webhookDB, event, tgt),
|
||||
databaseTargetDelivery(newEvent(`{"n":1}`), tgt),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -949,7 +988,7 @@ func TestArchiveSweep_ConcurrentWrites(t *testing.T) {
|
||||
defer wg.Done()
|
||||
|
||||
for _, d := range deliveries {
|
||||
env.eng.ExportDeliverDatabase(webhookDB, d)
|
||||
_ = env.writers.Write(d)
|
||||
}
|
||||
}()
|
||||
|
||||
@@ -973,7 +1012,7 @@ func TestArchiveSweeper_StopsCleanly(t *testing.T) {
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
|
||||
tgt := env.seedDatabaseTarget(t, time.Hour, "")
|
||||
env.seedArchiveRows(
|
||||
t, tgt, time.Now().Add(-48*time.Hour),
|
||||
)
|
||||
@@ -1,7 +1,6 @@
|
||||
package delivery
|
||||
package archive
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
@@ -18,9 +17,10 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
// archiveExpiryNever is the expiry sentinel (and default) that
|
||||
// disables pruning so archived rows are kept forever.
|
||||
const archiveExpiryNever = "never"
|
||||
// ExpiryNever is the archive expiry, as the target forms take it,
|
||||
// that disables pruning so archived rows are kept forever. It is
|
||||
// stored as an ArchiveExpiry of 0.
|
||||
const ExpiryNever = "never"
|
||||
|
||||
// archiveReopenDebounce bounds how often an archive file is
|
||||
// closed and reopened. After each write the handle is closed
|
||||
@@ -71,27 +71,14 @@ var (
|
||||
"archive writer has been evicted",
|
||||
)
|
||||
|
||||
// ErrArchiveNameTaken is returned when an archive cannot be
|
||||
// ErrNameTaken is returned when an archive cannot be
|
||||
// renamed because a file already has the new name. That file may
|
||||
// be an archive with rows of its own, so it is never replaced.
|
||||
ErrArchiveNameTaken = errors.New(
|
||||
ErrNameTaken = errors.New(
|
||||
"a file already has the archive's new name",
|
||||
)
|
||||
)
|
||||
|
||||
// databaseTargetConfig is the optional per-target JSON config
|
||||
// for a database (archive) target.
|
||||
type databaseTargetConfig struct {
|
||||
// Expiry is a Go duration (e.g. "720h") after which
|
||||
// archived rows are pruned, or "never" (the default) to
|
||||
// keep them forever.
|
||||
Expiry string `json:"expiry"`
|
||||
|
||||
// Rotation is none (the default), monthly, daily or hourly: see
|
||||
// archivePeriod.
|
||||
Rotation string `json:"rotation"`
|
||||
}
|
||||
|
||||
// archivedEvent is one fully captured webhook event stored in a
|
||||
// database target's archive for long-term retention. It is a
|
||||
// self-contained copy — independent of the per-webhook event
|
||||
@@ -112,76 +99,33 @@ type archivedEvent struct {
|
||||
ArchivedAt time.Time `gorm:"index"`
|
||||
}
|
||||
|
||||
// parseArchiveExpiry reads the optional expiry from a database
|
||||
// target's config JSON. An empty config, an empty expiry, or
|
||||
// the literal "never" all mean keep forever, returned as a zero
|
||||
// duration. Any other value must parse as a positive Go
|
||||
// duration; a set-but-invalid value (unparseable, zero, or
|
||||
// negative) is an error rather than a silent default, matching
|
||||
// ValidateArchiveExpiry at target creation.
|
||||
func parseArchiveExpiry(
|
||||
configJSON string,
|
||||
) (time.Duration, error) {
|
||||
if configJSON == "" {
|
||||
// ParseExpiry reads a user-supplied archive expiry for a database
|
||||
// target, as the target forms submit it, into the ArchiveExpiry the
|
||||
// target stores. Empty and "never" both mean keep forever, which is
|
||||
// 0; anything else must be a positive Go duration such as "720h". Any
|
||||
// other value is an error, so a bad expiry is refused at the form
|
||||
// rather than stored.
|
||||
func ParseExpiry(expiry string) (time.Duration, error) {
|
||||
if expiry == "" || expiry == ExpiryNever {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
var cfg databaseTargetConfig
|
||||
|
||||
err := json.Unmarshal([]byte(configJSON), &cfg)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf(
|
||||
"parsing database target config: %w", err,
|
||||
)
|
||||
}
|
||||
|
||||
if cfg.Expiry == "" || cfg.Expiry == archiveExpiryNever {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
dur, err := time.ParseDuration(cfg.Expiry)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf(
|
||||
"parsing archive expiry %q: %w", cfg.Expiry, err,
|
||||
)
|
||||
}
|
||||
|
||||
if dur <= 0 {
|
||||
return 0, fmt.Errorf(
|
||||
"%w: %q", errArchiveExpiryNotPositive, cfg.Expiry,
|
||||
)
|
||||
}
|
||||
|
||||
return dur, nil
|
||||
}
|
||||
|
||||
// ValidateArchiveExpiry checks a user-supplied archive expiry
|
||||
// for a database target at configuration time. Valid values are
|
||||
// empty, "never" (both meaning keep forever), or a positive Go
|
||||
// duration such as "720h". Anything else is an error, so a bad
|
||||
// expiry is rejected when the target is created rather than
|
||||
// failing every subsequent delivery.
|
||||
func ValidateArchiveExpiry(expiry string) error {
|
||||
if expiry == "" || expiry == archiveExpiryNever {
|
||||
return nil
|
||||
}
|
||||
|
||||
dur, err := time.ParseDuration(expiry)
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
return 0, fmt.Errorf(
|
||||
"expiry must be %q or a Go duration "+
|
||||
"such as \"720h\": %w",
|
||||
archiveExpiryNever, err,
|
||||
ExpiryNever, err,
|
||||
)
|
||||
}
|
||||
|
||||
if dur <= 0 {
|
||||
return fmt.Errorf(
|
||||
return 0, fmt.Errorf(
|
||||
"%w: %q", errArchiveExpiryNotPositive, expiry,
|
||||
)
|
||||
}
|
||||
|
||||
return nil
|
||||
return dur, nil
|
||||
}
|
||||
|
||||
// archiveWriter owns one database target's archive SQLite files.
|
||||
@@ -194,7 +138,7 @@ func ValidateArchiveExpiry(expiry string) error {
|
||||
type archiveWriter struct {
|
||||
mu sync.Mutex
|
||||
|
||||
// path is the target's archive file as ArchivePath names it. A
|
||||
// path is the target's archive file as Path names it. A
|
||||
// target that rotates writes to the files archivePeriodPath names
|
||||
// for path and a period instead.
|
||||
path string
|
||||
@@ -232,7 +176,7 @@ type archiveWriter struct {
|
||||
// registry proper.
|
||||
//
|
||||
// Unlike every other field here it is guarded by
|
||||
// databaseTarget.mu, not by this writer's mu: it describes the
|
||||
// Writers.mu, not by this writer's mu: it describes the
|
||||
// registry entry rather than the file.
|
||||
sweepOwned bool
|
||||
}
|
||||
@@ -499,7 +443,7 @@ func (w *archiveWriter) sweepFile(
|
||||
// the next write creates its file under the new name.
|
||||
//
|
||||
// If a file already has one of the new names, nothing is moved and
|
||||
// the error is ErrArchiveNameTaken. If one file fails to move, those
|
||||
// the error is ErrNameTaken. If one file fails to move, those
|
||||
// already moved are moved back before the error is returned, so the
|
||||
// archive is never split across two names.
|
||||
func (w *archiveWriter) rename(name string) error {
|
||||
@@ -537,7 +481,7 @@ func (w *archiveWriter) rename(name string) error {
|
||||
for _, taken := range to {
|
||||
if fileExists(taken) {
|
||||
return fmt.Errorf(
|
||||
"%w: %s", ErrArchiveNameTaken, filepath.Base(taken),
|
||||
"%w: %s", ErrNameTaken, filepath.Base(taken),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -590,7 +534,7 @@ func (w *archiveWriter) evict() {
|
||||
// each row's archived time. It runs on every (re)open, so a
|
||||
// steadily written archive is swept by its own write traffic. An
|
||||
// archive that goes idle receives no further reopens, which is
|
||||
// why ArchiveSweeper exists to drive sweepExpired on a timer.
|
||||
// why Sweeper exists to drive sweepExpired on a timer.
|
||||
// Failures are logged, not fatal: a prune error must not stop
|
||||
// archiving.
|
||||
func (w *archiveWriter) prune(expiry time.Duration) {
|
||||
@@ -617,9 +561,9 @@ func (w *archiveWriter) prune(expiry time.Duration) {
|
||||
}
|
||||
}
|
||||
|
||||
// ArchiveFileInfo is what the metadata of a database target's archive
|
||||
// FileInfo is what the metadata of a database target's archive
|
||||
// files says about them.
|
||||
type ArchiveFileInfo struct {
|
||||
type FileInfo struct {
|
||||
// Files counts the files.
|
||||
Files int
|
||||
|
||||
@@ -631,18 +575,18 @@ type ArchiveFileInfo struct {
|
||||
Written time.Time
|
||||
}
|
||||
|
||||
// StatArchive reads the metadata of a database target's archive
|
||||
// files, given the path ArchivePath gives it (see archiveFiles), and
|
||||
// Stat reads the metadata of a database target's archive
|
||||
// files, given the path Path gives it (see archiveFiles), and
|
||||
// of their -wal, without opening them. With no files, which is so
|
||||
// before the first write and after the operator moved them away, the
|
||||
// error wraps fs.ErrNotExist.
|
||||
func StatArchive(path string) (ArchiveFileInfo, error) {
|
||||
func Stat(path string) (FileInfo, error) {
|
||||
files, err := archiveFiles(path)
|
||||
if err != nil {
|
||||
return ArchiveFileInfo{}, err
|
||||
return FileInfo{}, err
|
||||
}
|
||||
|
||||
var info ArchiveFileInfo
|
||||
var info FileInfo
|
||||
|
||||
for _, file := range files {
|
||||
db, err := os.Stat(file.path)
|
||||
@@ -651,7 +595,7 @@ func StatArchive(path string) (ArchiveFileInfo, error) {
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return ArchiveFileInfo{}, err
|
||||
return FileInfo{}, err
|
||||
}
|
||||
|
||||
info.Files++
|
||||
@@ -667,7 +611,7 @@ func StatArchive(path string) (ArchiveFileInfo, error) {
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return ArchiveFileInfo{}, err
|
||||
return FileInfo{}, err
|
||||
}
|
||||
|
||||
info.Size += wal.Size()
|
||||
@@ -678,7 +622,7 @@ func StatArchive(path string) (ArchiveFileInfo, error) {
|
||||
}
|
||||
|
||||
if info.Files == 0 {
|
||||
return ArchiveFileInfo{}, fmt.Errorf(
|
||||
return FileInfo{}, fmt.Errorf(
|
||||
"no archive file for %s: %w", path, fs.ErrNotExist,
|
||||
)
|
||||
}
|
||||
+6
-6
@@ -1,4 +1,4 @@
|
||||
package delivery_test
|
||||
package archive_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
gormlogger "gorm.io/gorm/logger"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
)
|
||||
|
||||
@@ -57,7 +57,7 @@ func (g *gormDefaultBuf) String() string {
|
||||
// without exporting production code to carry it, and a logging
|
||||
// detector is not worth a production symbol. What it detects is the
|
||||
// third gorm.Open in this service, at
|
||||
// internal/delivery/target_database_archive.go — the archive writer,
|
||||
// internal/archive/writer.go — the archive writer,
|
||||
// whose type is unexported, so nothing outside this package can drive
|
||||
// it.
|
||||
func captureArchiveGORMDefault(t *testing.T) *gormDefaultBuf {
|
||||
@@ -85,7 +85,7 @@ func captureArchiveGORMDefault(t *testing.T) *gormDefaultBuf {
|
||||
// writer's gorm.Open to the adapter.
|
||||
//
|
||||
// Restore a bare &gorm.Config{} at
|
||||
// internal/delivery/target_database_archive.go and this fails: the
|
||||
// internal/archive/writer.go and this fails: the
|
||||
// default logger prints the fully interpolated SELECT on every
|
||||
// ErrRecordNotFound, so the client-chosen event id below arrives whole
|
||||
// and unbounded on stdout, answering to no level the operator set.
|
||||
@@ -100,7 +100,7 @@ func TestArchiveWriter_NeverUsesGORMsDefaultLogger(t *testing.T) {
|
||||
|
||||
gormDefault := captureArchiveGORMDefault(t)
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
w := archive.NewExportArchiveWriter(
|
||||
filepath.Join(t.TempDir(), "archive.db"),
|
||||
slog.New(slog.NewTextHandler(
|
||||
&captured, &slog.HandlerOptions{Level: slog.LevelDebug},
|
||||
@@ -118,7 +118,7 @@ func TestArchiveWriter_NeverUsesGORMsDefaultLogger(t *testing.T) {
|
||||
value := strings.Repeat("\x01", archiveGORMFillBytes) +
|
||||
archiveGORMTailMarker
|
||||
|
||||
var row delivery.ExportArchivedEvent
|
||||
var row archive.ExportArchivedEvent
|
||||
|
||||
err := w.DB().Where("event_id = ?", value).First(&row).Error
|
||||
require.ErrorIs(t, err, gorm.ErrRecordNotFound)
|
||||
@@ -28,7 +28,7 @@ func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
||||
WebhookID: webhookID,
|
||||
Name: "leaky-target",
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"webhookUrl":"https://hooks.example/secret"}`,
|
||||
URL: "https://hooks.example/secret",
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
target.ID = uuid.New().String()
|
||||
|
||||
@@ -79,8 +79,9 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
||||
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
|
||||
byEvent := "idx_deliveries_event_id (event_id=? AND deleted_at=?)"
|
||||
|
||||
// The delivery engine: recovery and the retry sweep, the sweep for
|
||||
// stranded pending deliveries, and the queue depth count.
|
||||
// The delivery engine: the recovery routine, which reads the
|
||||
// retrying deliveries and, on a tick of the sweep, the pending ones
|
||||
// untouched for a while; and the queue depth count.
|
||||
assertPlanUses(t, db, dry.Where(
|
||||
"status = ?", database.DeliveryStatusRetrying,
|
||||
).Find(&deliveries), byStatus)
|
||||
|
||||
@@ -29,8 +29,10 @@ func marshalModel(t *testing.T, v any) string {
|
||||
// these tags the first handler that marshals a model serialises the
|
||||
// secret with it. Each field below is a live credential:
|
||||
//
|
||||
// - Target.Config holds an incoming-webhook URL whose path segments
|
||||
// are the bearer token.
|
||||
// - Target.URL is an incoming-webhook URL whose path segments are
|
||||
// the bearer token.
|
||||
// - Target.Headers holds header values, routinely authorization
|
||||
// tokens.
|
||||
// - APIKey.Key is a bearer token outright.
|
||||
// - Setting.Value holds the session encryption key.
|
||||
// - User.Password holds the Argon2 hash, and was already tagged.
|
||||
@@ -44,11 +46,20 @@ func TestModelsDoNotMarshalTheirSecrets(t *testing.T) {
|
||||
model any
|
||||
}{
|
||||
{
|
||||
name: "target config",
|
||||
name: "target url",
|
||||
model: database.Target{
|
||||
Name: keptField,
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"webhookUrl":"https://h/s/` + marker + `"}`,
|
||||
URL: "https://h/s/" + marker,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "target headers",
|
||||
model: database.Target{
|
||||
Name: keptField,
|
||||
Type: database.TargetTypeHTTP,
|
||||
Headers: map[string]string{"Authorization": marker},
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
},
|
||||
@@ -87,10 +98,10 @@ func TestModelsDoNotMarshalTheirSecrets(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestWebhookMarshalsNoTargetConfig covers the nested case: a webhook
|
||||
// TestWebhookMarshalsNoTargetURL covers the nested case: a webhook
|
||||
// marshalled with its targets preloaded must not carry the credential
|
||||
// through the association either.
|
||||
func TestWebhookMarshalsNoTargetConfig(t *testing.T) {
|
||||
func TestWebhookMarshalsNoTargetURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const marker = "QQNESTEDMARKERQQ"
|
||||
@@ -99,7 +110,7 @@ func TestWebhookMarshalsNoTargetConfig(t *testing.T) {
|
||||
Name: keptField,
|
||||
Targets: []database.Target{{
|
||||
Name: "slack",
|
||||
Config: `{"webhookUrl":"https://h/s/` + marker + `"}`,
|
||||
URL: "https://h/s/" + marker,
|
||||
MaxAttempts: 1,
|
||||
}},
|
||||
})
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
package database
|
||||
|
||||
import "time"
|
||||
|
||||
// TargetType represents the type of delivery target
|
||||
type TargetType string
|
||||
|
||||
@@ -11,7 +13,9 @@ const (
|
||||
TargetTypeSlack TargetType = "slack"
|
||||
)
|
||||
|
||||
// Target represents a delivery target for a webhook
|
||||
// Target represents a delivery target for a webhook. Its settings
|
||||
// are the fields from URL to ArchiveRotation; each type uses some of
|
||||
// them, and the ones a type does not use stay empty.
|
||||
type Target struct {
|
||||
BaseModel
|
||||
|
||||
@@ -20,14 +24,35 @@ type Target struct {
|
||||
Type TargetType `gorm:"not null" json:"type"`
|
||||
Active bool `gorm:"default:true" json:"active"`
|
||||
|
||||
// Configuration fields (JSON stored based on type).
|
||||
// URL is where an http or slack target sends each event.
|
||||
//
|
||||
// json:"-" because the blob holds the target's credential — a
|
||||
// Slack incoming-webhook URL, or an http destination whose path
|
||||
// segments are the secret. delivery.TargetView is the masking
|
||||
// barrier for the HTML path; this tag is the barrier for any
|
||||
// handler that marshals the model itself.
|
||||
Config string `gorm:"type:text" json:"-"` // JSON configuration
|
||||
// json:"-" because the URL is the target's credential: a Slack
|
||||
// incoming-webhook URL, or an http destination whose path segments
|
||||
// are the secret. delivery.TargetView is the masking barrier for
|
||||
// the HTML path; this tag is the barrier for any handler that
|
||||
// marshals the model itself.
|
||||
URL string `json:"-"`
|
||||
|
||||
// Headers is an http target's extra request headers, name to
|
||||
// value, stored as JSON text. json:"-" for the same reason as URL:
|
||||
// a header value is routinely an authorization token.
|
||||
Headers map[string]string `gorm:"serializer:json" json:"-"`
|
||||
|
||||
// TimeoutSeconds is an http target's per-request timeout; 0 uses
|
||||
// the delivery engine's shared client timeout.
|
||||
TimeoutSeconds int `json:"timeoutSeconds,omitempty"`
|
||||
|
||||
// ForwardQuery passes each event's query string on to an http
|
||||
// target, appended to URL. Off, URL is sent exactly as stored.
|
||||
ForwardQuery bool `json:"forwardQuery,omitempty"`
|
||||
|
||||
// ArchiveExpiry is how long a database target keeps each archived
|
||||
// event; 0 keeps them forever.
|
||||
ArchiveExpiry time.Duration `json:"archiveExpiry,omitempty"`
|
||||
|
||||
// ArchiveRotation is how often a database target starts a new
|
||||
// archive file: none, monthly, daily or hourly.
|
||||
ArchiveRotation string `json:"archiveRotation,omitempty"`
|
||||
|
||||
// MaxAttempts is how many times an HTTP or Slack target attempts
|
||||
// each delivery in all, at least 1. 1 is a single attempt with no
|
||||
|
||||
@@ -16,7 +16,7 @@ func newHTTPTarget(maxAttempts int) *database.Target {
|
||||
WebhookID: uuid.New().String(),
|
||||
Name: "target",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"https://example.com/"}`,
|
||||
URL: "https://example.com/",
|
||||
MaxAttempts: maxAttempts,
|
||||
}
|
||||
}
|
||||
@@ -42,3 +42,22 @@ func TestTargetCreate_AcceptsOneAttempt(t *testing.T) {
|
||||
|
||||
require.NoError(t, db.Create(newHTTPTarget(1)).Error)
|
||||
}
|
||||
|
||||
// TestTargetHeaders_StoredAsJSON reads the headers column back as
|
||||
// text, to show it holds the header map as JSON.
|
||||
func TestTargetHeaders_StoredAsJSON(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := startedTestDB(t)
|
||||
|
||||
target := newHTTPTarget(1)
|
||||
target.Headers = map[string]string{"Authorization": "Bearer x"}
|
||||
require.NoError(t, db.Create(target).Error)
|
||||
|
||||
var headers string
|
||||
|
||||
require.NoError(t, db.Raw(
|
||||
"SELECT headers FROM targets WHERE id = ?", target.ID,
|
||||
).Scan(&headers).Error)
|
||||
assert.JSONEq(t, `{"Authorization":"Bearer x"}`, headers)
|
||||
}
|
||||
|
||||
@@ -184,8 +184,9 @@ func (r *RetentionReaper) sweep(ctx context.Context) {
|
||||
|
||||
wh := webhooks[i]
|
||||
|
||||
// A missing database has nothing to reap. Restart recovery
|
||||
// reports a lost one (see WebhookDBManager.GetDB).
|
||||
// A missing database has nothing to reap. The delivery
|
||||
// engine's recovery routine reports a lost one at start (see
|
||||
// WebhookDBManager.GetDB).
|
||||
if !r.dbManager.DBExists(wh.ID) {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -87,13 +87,18 @@ func TestMainSchemaHoldsEveryModel(t *testing.T) {
|
||||
Active: true,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Target{
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
Name: "target",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Active: true,
|
||||
Config: `{"url":"https://example.com/"}`,
|
||||
MaxAttempts: 3,
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
Name: "target",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Active: true,
|
||||
URL: "https://example.com/",
|
||||
Headers: map[string]string{"X-Api-Key": "key"},
|
||||
TimeoutSeconds: 30,
|
||||
ForwardQuery: true,
|
||||
ArchiveExpiry: 720 * time.Hour,
|
||||
ArchiveRotation: "daily",
|
||||
MaxAttempts: 3,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -137,7 +137,7 @@ func TestPerWebhookEventDatabaseFilesAreOwnerOnly(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestArchiveDatabaseFilesAreOwnerOnly covers the archive-*.db tier.
|
||||
// internal/delivery builds that path and opens it through OpenSQLite,
|
||||
// internal/archive builds that path and opens it through OpenSQLite,
|
||||
// the same single open path exercised here, so the mode is settled for
|
||||
// all three tiers in one place.
|
||||
func TestArchiveDatabaseFilesAreOwnerOnly(t *testing.T) {
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
-- Migration 001: the schema of a database target's archive file. The
|
||||
-- archivedEvent model in internal/delivery must agree with it.
|
||||
-- archivedEvent model in internal/archive must agree with it.
|
||||
|
||||
CREATE TABLE archived_events (
|
||||
id integer PRIMARY KEY AUTOINCREMENT,
|
||||
|
||||
@@ -79,7 +79,12 @@ CREATE TABLE targets (
|
||||
name text NOT NULL,
|
||||
type text NOT NULL,
|
||||
active numeric DEFAULT true,
|
||||
config text,
|
||||
url text,
|
||||
headers text,
|
||||
timeout_seconds integer,
|
||||
forward_query numeric,
|
||||
archive_expiry integer,
|
||||
archive_rotation text,
|
||||
max_attempts integer NOT NULL,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_webhooks_targets
|
||||
|
||||
@@ -46,12 +46,7 @@ func TestClientForRequest_TimeoutKeepsSSRFGuard(t *testing.T) {
|
||||
t.Run(target, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := &delivery.HTTPTargetConfig{
|
||||
URL: target,
|
||||
Timeout: 5,
|
||||
}
|
||||
|
||||
client := engine.ExportClientForRequest(cfg, nil)
|
||||
client := engine.ExportClientForRequest(5, nil)
|
||||
|
||||
require.NotSame(t, engine.ExportClient(), client,
|
||||
"a per-target timeout must yield a "+
|
||||
@@ -101,11 +96,7 @@ func TestClientForRequest_NoTimeoutUnchanged(t *testing.T) {
|
||||
|
||||
engine := newSSRFTestEngine()
|
||||
|
||||
cfg := &delivery.HTTPTargetConfig{
|
||||
URL: "https://example.com/hook",
|
||||
}
|
||||
|
||||
client := engine.ExportClientForRequest(cfg, nil)
|
||||
client := engine.ExportClientForRequest(0, nil)
|
||||
|
||||
assert.Same(t, engine.ExportClient(), client,
|
||||
"without a per-target timeout the shared client "+
|
||||
@@ -123,15 +114,8 @@ func TestClientForRequest_HeadersKeepSSRFGuard(t *testing.T) {
|
||||
|
||||
engine := newSSRFTestEngine()
|
||||
|
||||
cfg := &delivery.HTTPTargetConfig{
|
||||
URL: "https://example.com/with-headers",
|
||||
Headers: map[string]string{
|
||||
"X-Api-Key": "configured",
|
||||
},
|
||||
}
|
||||
|
||||
client := engine.ExportClientForRequest(
|
||||
cfg, []string{"X-Api-Key"},
|
||||
0, []string{probeHeaderName},
|
||||
)
|
||||
|
||||
require.NotNil(t, client.CheckRedirect,
|
||||
|
||||
+263
-519
File diff suppressed because it is too large
Load Diff
@@ -3,7 +3,6 @@ package delivery_test
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
@@ -97,17 +96,6 @@ func iSeedWebhookDB(
|
||||
return db
|
||||
}
|
||||
|
||||
func iHTTPConfig(url string) string {
|
||||
cfg := delivery.HTTPTargetConfig{URL: url}
|
||||
|
||||
data, err := json.Marshal(cfg)
|
||||
if err != nil {
|
||||
panic("failed to marshal HTTPTargetConfig")
|
||||
}
|
||||
|
||||
return string(data)
|
||||
}
|
||||
|
||||
func iEngine(
|
||||
t *testing.T, workers int,
|
||||
) *delivery.Engine {
|
||||
@@ -169,24 +157,24 @@ func iSeedDelivery(
|
||||
func iTask(
|
||||
d database.Delivery,
|
||||
event database.Event,
|
||||
webhookID, targetID, name, config string,
|
||||
webhookID, targetID, name, targetURL string,
|
||||
maxAttempts, attemptNum int,
|
||||
body *string,
|
||||
) delivery.Task {
|
||||
return delivery.Task{
|
||||
DeliveryID: d.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: webhookID,
|
||||
TargetID: targetID,
|
||||
TargetName: name,
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetConfig: config,
|
||||
MaxAttempts: maxAttempts,
|
||||
Method: event.Method,
|
||||
Headers: event.Headers,
|
||||
ContentType: event.ContentType,
|
||||
Body: body,
|
||||
AttemptNum: attemptNum,
|
||||
DeliveryID: d.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: webhookID,
|
||||
TargetID: targetID,
|
||||
TargetName: name,
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetURL: targetURL,
|
||||
MaxAttempts: maxAttempts,
|
||||
Method: event.Method,
|
||||
Headers: event.Headers,
|
||||
ContentType: event.ContentType,
|
||||
Body: body,
|
||||
AttemptNum: attemptNum,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -239,10 +227,9 @@ func TestProcessNewTask_InlineBody(t *testing.T) {
|
||||
)
|
||||
|
||||
bodyStr := event.Body
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"test-target", cfg, 1, 1, &bodyStr,
|
||||
"test-target", ts.URL, 1, 1, &bodyStr,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(
|
||||
@@ -290,10 +277,9 @@ func TestProcessNewTask_LargeBody_FetchFromDB(
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"test-large", cfg, 1, 1, nil,
|
||||
"test-large", ts.URL, 1, 1, nil,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(
|
||||
@@ -313,16 +299,16 @@ func TestProcessNewTask_InvalidWebhookID(t *testing.T) {
|
||||
s := newISetup(t)
|
||||
|
||||
task := delivery.Task{
|
||||
DeliveryID: uuid.New().String(),
|
||||
EventID: uuid.New().String(),
|
||||
WebhookID: uuid.New().String(),
|
||||
TargetID: uuid.New().String(),
|
||||
TargetName: "test",
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetConfig: iHTTPConfig("http://localhost:9999"),
|
||||
MaxAttempts: 1,
|
||||
Body: nil,
|
||||
AttemptNum: 1,
|
||||
DeliveryID: uuid.New().String(),
|
||||
EventID: uuid.New().String(),
|
||||
WebhookID: uuid.New().String(),
|
||||
TargetID: uuid.New().String(),
|
||||
TargetName: "test",
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetURL: "http://localhost:9999",
|
||||
MaxAttempts: 1,
|
||||
Body: nil,
|
||||
AttemptNum: 1,
|
||||
}
|
||||
|
||||
s.Engine.ExportProcessNewTask(
|
||||
@@ -363,7 +349,6 @@ func TestProcessRetryTask_SuccessfulRetry(t *testing.T) {
|
||||
)
|
||||
|
||||
bodyStr := event.Body
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
|
||||
// The target row exists because the engine confirms a scheduled
|
||||
// retry's target has not been deleted before it runs it. A retry
|
||||
@@ -372,12 +357,12 @@ func TestProcessRetryTask_SuccessfulRetry(t *testing.T) {
|
||||
// task. See https://git.eeqj.de/sneak/webhooker/issues/107.
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "retry-target",
|
||||
database.TargetTypeHTTP, cfg, 5,
|
||||
database.TargetTypeHTTP, ts.URL, 5,
|
||||
)
|
||||
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"retry-target", cfg, 5, 2, &bodyStr,
|
||||
"retry-target", ts.URL, 5, 2, &bodyStr,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(
|
||||
@@ -410,10 +395,9 @@ func TestProcessRetryTask_SkipsNonRetryingDelivery(
|
||||
)
|
||||
|
||||
bodyStr := event.Body
|
||||
cfg := iHTTPConfig("http://localhost:1")
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"skip-target", cfg, 5, 2, &bodyStr,
|
||||
"skip-target", "http://localhost:1", 5, 2, &bodyStr,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(
|
||||
@@ -460,16 +444,14 @@ func TestProcessRetryTask_LargeBody_FetchFromDB(
|
||||
database.DeliveryStatusRetrying,
|
||||
)
|
||||
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "retry-large",
|
||||
database.TargetTypeHTTP, cfg, 5,
|
||||
database.TargetTypeHTTP, ts.URL, 5,
|
||||
)
|
||||
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"retry-large", cfg, 5, 2, nil,
|
||||
"retry-large", ts.URL, 5, 2, nil,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(
|
||||
@@ -510,7 +492,7 @@ func TestWorkerLifecycle_StartStop(t *testing.T) {
|
||||
)
|
||||
task.TargetType = database.TargetTypeLog
|
||||
|
||||
s.Engine.Notify([]delivery.Task{task})
|
||||
s.Engine.ExportNotify([]delivery.Task{task})
|
||||
|
||||
iWaitForDelivered(t, s.WebhookDB, d.ID)
|
||||
|
||||
@@ -570,16 +552,15 @@ func TestWorkerLifecycle_ProcessesRetryChannel(
|
||||
s.Engine.ExportStart()
|
||||
|
||||
bodyStr := event.Body
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "retry-chan-test",
|
||||
database.TargetTypeHTTP, cfg, 5,
|
||||
database.TargetTypeHTTP, ts.URL, 5,
|
||||
)
|
||||
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"retry-chan-test", cfg, 5, 2, &bodyStr,
|
||||
"retry-chan-test", ts.URL, 5, 2, &bodyStr,
|
||||
)
|
||||
|
||||
s.Engine.ExportRetryCh() <- task
|
||||
@@ -659,7 +640,7 @@ func TestRecoverPendingDeliveries(t *testing.T) {
|
||||
|
||||
s.Engine.ExportRecoverPendingDeliveries(
|
||||
context.Background(), s.WebhookDB,
|
||||
s.WebhookID,
|
||||
s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
for i := range 3 {
|
||||
@@ -685,7 +666,7 @@ func iCreateTarget(
|
||||
mainDB *gorm.DB,
|
||||
targetID, webhookID, name string,
|
||||
targetType database.TargetType,
|
||||
config string,
|
||||
targetURL string,
|
||||
maxAttempts int,
|
||||
) {
|
||||
t.Helper()
|
||||
@@ -695,7 +676,7 @@ func iCreateTarget(
|
||||
Name: name,
|
||||
Type: targetType,
|
||||
Active: true,
|
||||
Config: config,
|
||||
URL: targetURL,
|
||||
MaxAttempts: maxAttempts,
|
||||
}
|
||||
target.ID = targetID
|
||||
@@ -736,7 +717,7 @@ func TestRecoverWebhookDeliveries_RetryingDeliveries(
|
||||
iCreateTarget(t, s.MainDB, targetID,
|
||||
s.WebhookID, "retry-recovery",
|
||||
database.TargetTypeHTTP,
|
||||
iHTTPConfig("http://example.com/hook"), 5,
|
||||
"http://example.com/hook", 5,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
@@ -756,7 +737,7 @@ func TestRecoverWebhookDeliveries_RetryingDeliveries(
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID,
|
||||
context.Background(), s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
select {
|
||||
@@ -796,7 +777,7 @@ func iSeedRetryingWithType(
|
||||
|
||||
iCreateTarget(t, s.MainDB, targetID,
|
||||
s.WebhookID, "mutated-target", targetType,
|
||||
iHTTPConfig("http://example.com/hook"), 5,
|
||||
"http://example.com/hook", 5,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
@@ -880,7 +861,7 @@ func TestRecoverSingleRetry_TypeNoLongerRetries(
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID,
|
||||
context.Background(), s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
iAssertTerminallyFailed(
|
||||
@@ -888,7 +869,7 @@ func TestRecoverSingleRetry_TypeNoLongerRetries(
|
||||
)
|
||||
}
|
||||
|
||||
func TestSweepSingleRetry_TypeNoLongerRetries(
|
||||
func TestRecoverSingleRetry_OnTick_TypeNoLongerRetries(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
@@ -903,8 +884,8 @@ func TestSweepSingleRetry_TypeNoLongerRetries(
|
||||
t, s, database.TargetTypeDatabase,
|
||||
)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
iAssertTerminallyFailed(
|
||||
@@ -934,14 +915,14 @@ func TestFailUnretryableRetry_WritesNoTargetRow(
|
||||
|
||||
targetID := uuid.New().String()
|
||||
|
||||
// A Slack incoming-webhook URL: the target config IS the
|
||||
// A Slack incoming-webhook URL: the target's URL IS the
|
||||
// credential, which is what makes a leaked target row a
|
||||
// disclosure rather than a curiosity.
|
||||
hookURL := "https://hooks.slack.com/services/T00/B00/x"
|
||||
|
||||
iCreateTarget(t, s.MainDB, targetID,
|
||||
s.WebhookID, "credential-bearing",
|
||||
database.TargetTypeLog, iHTTPConfig(hookURL), 5,
|
||||
database.TargetTypeLog, hookURL, 5,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
@@ -955,8 +936,8 @@ func TestFailUnretryableRetry_WritesNoTargetRow(
|
||||
|
||||
iSeedFailedResult(t, s.WebhookDB, d.ID)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
iAssertStatus(t, s.WebhookDB, d.ID,
|
||||
@@ -987,13 +968,13 @@ func TestRecoverSingleRetry_UnknownTargetType(
|
||||
deliveryID := iSeedRetryingWithType(t, s, unknown)
|
||||
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID,
|
||||
context.Background(), s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
iAssertTerminallyFailed(t, s, deliveryID, unknown)
|
||||
}
|
||||
|
||||
func TestSweepSingleRetry_UnknownTargetType(
|
||||
func TestRecoverSingleRetry_OnTick_UnknownTargetType(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
@@ -1008,8 +989,8 @@ func TestSweepSingleRetry_UnknownTargetType(
|
||||
|
||||
deliveryID := iSeedRetryingWithType(t, s, unknown)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
iAssertTerminallyFailed(t, s, deliveryID, unknown)
|
||||
@@ -1051,19 +1032,19 @@ func iCreateWebhook(
|
||||
require.NoError(t, mainDB.Create(&webhook).Error)
|
||||
}
|
||||
|
||||
// --- recoverInFlight Tests ---
|
||||
// --- recoverDeliveries Tests ---
|
||||
|
||||
func TestRecoverInFlight_NoWebhooks(t *testing.T) {
|
||||
func TestRecoverDeliveries_NoWebhooks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
|
||||
s.Engine.ExportRecoverInFlight(
|
||||
context.Background(),
|
||||
s.Engine.ExportRecoverDeliveries(
|
||||
context.Background(), atStart,
|
||||
)
|
||||
}
|
||||
|
||||
func TestRecoverInFlight_WithPendingDeliveries(
|
||||
func TestRecoverDeliveries_WithPendingDeliveries(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
@@ -1090,8 +1071,8 @@ func TestRecoverInFlight_WithPendingDeliveries(
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverInFlight(
|
||||
context.Background(),
|
||||
s.Engine.ExportRecoverDeliveries(
|
||||
context.Background(), atStart,
|
||||
)
|
||||
|
||||
select {
|
||||
@@ -1102,16 +1083,16 @@ func TestRecoverInFlight_WithPendingDeliveries(
|
||||
database.TargetTypeLog, task.TargetType,
|
||||
)
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("expected task from recoverInFlight")
|
||||
t.Fatal("expected task from recovery at start")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecoverInFlight_ReportsAMissingWebhookDatabase covers a webhook
|
||||
// whose database file is gone, after a partial restore say. Restart
|
||||
// recovery opens every webhook's database, so the empty one made in its
|
||||
// TestRecoverDeliveries_ReportsAMissingWebhookDatabase covers a webhook
|
||||
// whose database file is gone, after a partial restore say. Recovery at
|
||||
// start opens every webhook's database, so the empty one made in its
|
||||
// place is reported at start, naming the file
|
||||
// (https://git.eeqj.de/sneak/webhooker/issues/290).
|
||||
func TestRecoverInFlight_ReportsAMissingWebhookDatabase(t *testing.T) {
|
||||
func TestRecoverDeliveries_ReportsAMissingWebhookDatabase(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mainDB := iMainDB(t)
|
||||
@@ -1131,7 +1112,7 @@ func TestRecoverInFlight_ReportsAMissingWebhookDatabase(t *testing.T) {
|
||||
&http.Client{Timeout: 5 * time.Second}, 1,
|
||||
)
|
||||
|
||||
engine.ExportRecoverInFlight(context.Background())
|
||||
engine.ExportRecoverDeliveries(context.Background(), atStart)
|
||||
|
||||
assert.Contains(
|
||||
t, logs.String(),
|
||||
@@ -1140,10 +1121,11 @@ func TestRecoverInFlight_ReportsAMissingWebhookDatabase(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestRecoverInFlight_SkipsAWebhookDeletedAfterTheListIsRead covers a
|
||||
// webhook deleted from the web interface while restart recovery runs.
|
||||
// Its database file is gone, and recovery must not create it again.
|
||||
func TestRecoverInFlight_SkipsAWebhookDeletedAfterTheListIsRead(
|
||||
// TestRecoverDeliveries_SkipsAWebhookDeletedAfterTheListIsRead covers a
|
||||
// webhook deleted from the web interface while the recovery routine
|
||||
// runs. Its database file is gone, and recovery must not create it
|
||||
// again.
|
||||
func TestRecoverDeliveries_SkipsAWebhookDeletedAfterTheListIsRead(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
@@ -1179,13 +1161,13 @@ func TestRecoverInFlight_SkipsAWebhookDeletedAfterTheListIsRead(
|
||||
&http.Client{Timeout: 5 * time.Second}, 1,
|
||||
)
|
||||
|
||||
engine.ExportRecoverInFlight(context.Background())
|
||||
engine.ExportRecoverDeliveries(context.Background(), atStart)
|
||||
|
||||
require.True(t, deleted)
|
||||
assert.False(t, dbMgr.DBExists(webhookID))
|
||||
}
|
||||
|
||||
// --- HTTP Config with custom headers ---
|
||||
// --- HTTP target with custom headers ---
|
||||
|
||||
func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -1207,16 +1189,6 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
|
||||
)
|
||||
defer ts.Close()
|
||||
|
||||
cfg := delivery.HTTPTargetConfig{
|
||||
URL: ts.URL,
|
||||
Headers: map[string]string{
|
||||
"Authorization": "Bearer secret-token",
|
||||
},
|
||||
}
|
||||
|
||||
cfgJSON, err := json.Marshal(cfg)
|
||||
require.NoError(t, err)
|
||||
|
||||
event := iSeedEvent(
|
||||
t, s.WebhookDB, s.WebhookID,
|
||||
`{"auth":"test"}`,
|
||||
@@ -1231,9 +1203,12 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
|
||||
bodyStr := event.Body
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"auth-target", string(cfgJSON),
|
||||
"auth-target", ts.URL,
|
||||
1, 1, &bodyStr,
|
||||
)
|
||||
task.TargetHeaders = map[string]string{
|
||||
headerAuthorization: "Bearer secret-token",
|
||||
}
|
||||
|
||||
s.Engine.ExportProcessNewTask(
|
||||
context.TODO(), &task,
|
||||
@@ -1246,11 +1221,14 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
|
||||
|
||||
// --- HTTP delivery with custom timeout ---
|
||||
|
||||
// TestDeliverHTTP_TargetTimeout proves a task's TimeoutSeconds reaches
|
||||
// the delivery through the engine. The destination answers after two
|
||||
// seconds, inside the engine's own five-second client timeout, so only
|
||||
// the task's one second can fail the delivery.
|
||||
func TestDeliverHTTP_TargetTimeout(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := testWebhookDB(t)
|
||||
e := iEngine(t, 1)
|
||||
s := newISetup(t)
|
||||
|
||||
ts := httptest.NewServer(
|
||||
http.HandlerFunc(
|
||||
@@ -1262,31 +1240,30 @@ func TestDeliverHTTP_TargetTimeout(t *testing.T) {
|
||||
)
|
||||
defer ts.Close()
|
||||
|
||||
cfg := delivery.HTTPTargetConfig{
|
||||
URL: ts.URL,
|
||||
Timeout: 1,
|
||||
}
|
||||
event := iSeedEvent(
|
||||
t, s.WebhookDB, s.WebhookID, `{"timeout":"test"}`,
|
||||
)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
cfgJSON, err := json.Marshal(cfg)
|
||||
require.NoError(t, err)
|
||||
|
||||
event, del := iSeedEventAndDelivery(
|
||||
t, db, `{"timeout":"test"}`,
|
||||
string(cfgJSON),
|
||||
d := iSeedDelivery(
|
||||
t, s.WebhookDB, event.ID, targetID,
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
task, d := iHTTPTaskAndDelivery(
|
||||
event, del, "timeout-target",
|
||||
string(cfgJSON), 1, 1,
|
||||
bodyStr := event.Body
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"timeout-target", ts.URL, 1, 1, &bodyStr,
|
||||
)
|
||||
task.TimeoutSeconds = 1
|
||||
|
||||
e.ExportDeliverHTTP(context.TODO(), db, d, task)
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
|
||||
iAssertStatus(t, db, del.ID,
|
||||
iAssertStatus(t, s.WebhookDB, d.ID,
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
iAssertResultFailed(t, db, del.ID)
|
||||
iAssertResultFailed(t, s.WebhookDB, d.ID)
|
||||
}
|
||||
|
||||
// TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog pins the size
|
||||
@@ -1324,8 +1301,6 @@ func TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog(
|
||||
))
|
||||
defer ts.Close()
|
||||
|
||||
cfgJSON := iHTTPConfig(ts.URL)
|
||||
|
||||
event := iSeedEvent(
|
||||
t, s.WebhookDB, s.WebhookID, `{"cut":"test"}`,
|
||||
)
|
||||
@@ -1339,7 +1314,7 @@ func TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog(
|
||||
bodyStr := event.Body
|
||||
task := iTask(
|
||||
del, event, s.WebhookID, targetID,
|
||||
"cut-target", cfgJSON, 1, 1, &bodyStr,
|
||||
"cut-target", ts.URL, 1, 1, &bodyStr,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
@@ -1371,7 +1346,7 @@ func TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog(
|
||||
func iSeedEventAndDelivery(
|
||||
t *testing.T,
|
||||
db *gorm.DB,
|
||||
body, _ string,
|
||||
body string,
|
||||
) (database.Event, database.Delivery) {
|
||||
t.Helper()
|
||||
|
||||
@@ -1402,19 +1377,19 @@ func iSeedEventAndDelivery(
|
||||
func iHTTPTaskAndDelivery(
|
||||
event database.Event,
|
||||
del database.Delivery,
|
||||
name, config string,
|
||||
name, targetURL string,
|
||||
maxAttempts, attemptNum int,
|
||||
) (*delivery.Task, *database.Delivery) {
|
||||
task := &delivery.Task{
|
||||
DeliveryID: del.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: event.WebhookID,
|
||||
TargetID: del.TargetID,
|
||||
TargetName: name,
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetConfig: config,
|
||||
MaxAttempts: maxAttempts,
|
||||
AttemptNum: attemptNum,
|
||||
DeliveryID: del.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: event.WebhookID,
|
||||
TargetID: del.TargetID,
|
||||
TargetName: name,
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetURL: targetURL,
|
||||
MaxAttempts: maxAttempts,
|
||||
AttemptNum: attemptNum,
|
||||
}
|
||||
|
||||
d := &database.Delivery{
|
||||
@@ -1425,7 +1400,7 @@ func iHTTPTaskAndDelivery(
|
||||
Target: database.Target{
|
||||
Name: name,
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: config,
|
||||
URL: targetURL,
|
||||
MaxAttempts: maxAttempts,
|
||||
},
|
||||
}
|
||||
@@ -1454,56 +1429,30 @@ func iAssertResultFailed(
|
||||
assert.NotEmpty(t, result.Error)
|
||||
}
|
||||
|
||||
// --- HTTP request with invalid config ---
|
||||
// --- HTTP request with no URL ---
|
||||
|
||||
func TestDeliverHTTP_InvalidConfig(t *testing.T) {
|
||||
// TestDeliverHTTP_NoURL proves an http target without a URL, which
|
||||
// the forms never store, fails its delivery and records why.
|
||||
func TestDeliverHTTP_NoURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := testWebhookDB(t)
|
||||
e := iEngine(t, 1)
|
||||
|
||||
event, del := iSeedEventAndDelivery(
|
||||
t, db, `{"config":"invalid"}`, "",
|
||||
)
|
||||
event, del := iSeedEventAndDelivery(t, db, `{"url":"none"}`)
|
||||
|
||||
task, d := iHTTPTaskAndDelivery(
|
||||
event, del, "bad-config", `not-json`, 1, 1,
|
||||
)
|
||||
task, d := iHTTPTaskAndDelivery(event, del, "no-url", "", 1, 1)
|
||||
|
||||
e.ExportDeliverHTTP(context.TODO(), db, d, task)
|
||||
|
||||
iAssertStatus(t, db, del.ID,
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
iAssertResultFailed(t, db, del.ID)
|
||||
}
|
||||
|
||||
// TestDeliverHTTP_InvalidConfigUnrecordedStaysPending: a delivery is
|
||||
// failed for an invalid config only once the reason is recorded.
|
||||
// Unrecorded, it stays pending, where the sweep finds it again.
|
||||
func TestDeliverHTTP_InvalidConfigUnrecordedStaysPending(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := testWebhookDB(t)
|
||||
e := testEngine(t, 1)
|
||||
|
||||
event, del := iSeedEventAndDelivery(
|
||||
t, db, `{"config":"invalid"}`, "",
|
||||
)
|
||||
|
||||
task, d := iHTTPTaskAndDelivery(
|
||||
event, del, "bad-config", `not-json`, 1, 1,
|
||||
)
|
||||
|
||||
require.NoError(t, db.Exec("drop table delivery_results").Error)
|
||||
|
||||
e.ExportDeliverHTTP(context.TODO(), db, d, task)
|
||||
|
||||
iAssertStatus(t, db, del.ID,
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
}
|
||||
|
||||
// --- Notify batching ---
|
||||
// --- notify batching ---
|
||||
|
||||
func TestNotify_MultipleTasks(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -1518,7 +1467,7 @@ func TestNotify_MultipleTasks(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
e.Notify(tasks)
|
||||
e.ExportNotify(tasks)
|
||||
|
||||
for i := range 5 {
|
||||
select {
|
||||
|
||||
@@ -29,7 +29,7 @@ const (
|
||||
// OnStart before the caller may enqueue work. A worker pool
|
||||
// wrongly rooted in the already-done hook context has nothing
|
||||
// but ctx.Done() ready in its select, so it is deterministically
|
||||
// gone by the end of this window. Without the wait, Notify would
|
||||
// gone by the end of this window. Without the wait, notify would
|
||||
// race the pool's very first select, in which a ready ctx.Done()
|
||||
// and a ready deliveryCh are chosen between at random and a
|
||||
// doomed pool still delivers.
|
||||
@@ -99,7 +99,7 @@ func requireStopHookExpires(
|
||||
// the caller can drive OnStop too.
|
||||
//
|
||||
// Callers must not seed pending or retrying deliveries before
|
||||
// calling this: restart recovery enqueues those during startup,
|
||||
// calling this: the recovery routine's run at start enqueues those,
|
||||
// which would put work in the queue while the pool is still
|
||||
// racing its first select.
|
||||
func startEngineViaHook(
|
||||
@@ -178,11 +178,11 @@ func TestEngine_WorkersOutliveStartHookContext(t *testing.T) {
|
||||
_ = lc.hooks[0].OnStop(context.Background())
|
||||
})
|
||||
|
||||
// Seeded only after the pool has settled, so restart recovery
|
||||
// cannot enqueue it during startup.
|
||||
// Seeded only after the pool has settled, so the recovery
|
||||
// routine's run at start cannot enqueue it.
|
||||
deliveryID, task := seedLogTask(t, s)
|
||||
|
||||
s.Engine.Notify([]delivery.Task{task})
|
||||
s.Engine.ExportNotify([]delivery.Task{task})
|
||||
|
||||
iWaitForDelivered(t, s.WebhookDB, deliveryID)
|
||||
}
|
||||
@@ -202,7 +202,7 @@ func TestEngine_StopHookStopsWorkers(t *testing.T) {
|
||||
// Let the pool prove it is running before stopping it, so a
|
||||
// fast OnStop cannot pass by stopping something already dead.
|
||||
firstID, firstTask := seedLogTask(t, s)
|
||||
s.Engine.Notify([]delivery.Task{firstTask})
|
||||
s.Engine.ExportNotify([]delivery.Task{firstTask})
|
||||
iWaitForDelivered(t, s.WebhookDB, firstID)
|
||||
|
||||
var stopErr error
|
||||
@@ -232,7 +232,7 @@ func TestEngine_StopHookStopsWorkers(t *testing.T) {
|
||||
// With every worker gone, a freshly notified task must sit
|
||||
// untouched in the queue rather than being delivered.
|
||||
secondID, secondTask := seedLogTask(t, s)
|
||||
s.Engine.Notify([]delivery.Task{secondTask})
|
||||
s.Engine.ExportNotify([]delivery.Task{secondTask})
|
||||
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
|
||||
@@ -309,13 +309,12 @@ func seedHeldDelivery(
|
||||
))
|
||||
t.Cleanup(ts.Close)
|
||||
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
iCreateWebhook(t, s.MainDB, s.WebhookID, "hook")
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "held",
|
||||
database.TargetTypeHTTP, cfg, maxAttempts,
|
||||
database.TargetTypeHTTP, ts.URL, maxAttempts,
|
||||
)
|
||||
|
||||
event := iSeedEvent(t, s.WebhookDB, s.WebhookID, `{"held":true}`)
|
||||
@@ -324,7 +323,7 @@ func seedHeldDelivery(
|
||||
bodyStr := event.Body
|
||||
|
||||
return iTask(
|
||||
d, event, s.WebhookID, targetID, "held", cfg,
|
||||
d, event, s.WebhookID, targetID, "held", ts.URL,
|
||||
maxAttempts, attemptNum, &bodyStr,
|
||||
), arrived
|
||||
}
|
||||
@@ -345,15 +344,15 @@ func stopDuringAttempt(
|
||||
require.NoError(t, lc.hooks[0].OnStop(context.Background()))
|
||||
}
|
||||
|
||||
// requireRequeued runs restart recovery on the stopped engine and
|
||||
// requires it to queue task's attempt again on ch.
|
||||
// requireRequeued runs the recovery routine as at start on the stopped
|
||||
// engine and requires it to queue task's attempt again on ch.
|
||||
func requireRequeued(
|
||||
t *testing.T, s iSetup, ch <-chan delivery.Task, task delivery.Task,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID,
|
||||
context.Background(), s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
select {
|
||||
@@ -361,7 +360,7 @@ func requireRequeued(
|
||||
assert.Equal(t, task.DeliveryID, got.DeliveryID)
|
||||
assert.Equal(t, task.AttemptNum, got.AttemptNum)
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("restart recovery did not queue the delivery again")
|
||||
t.Fatal("recovery at start did not queue the delivery again")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -371,7 +370,8 @@ func requireRequeued(
|
||||
// no retries that was the delivery's only attempt, recorded as failed
|
||||
// with "context canceled", and nothing ever sent it again. The
|
||||
// delivery must instead stay pending, with no result and no count in
|
||||
// the attempt metrics, for restart recovery to send.
|
||||
// the attempt metrics, for the recovery routine to send at the next
|
||||
// start.
|
||||
func TestEngine_StopLeavesFireAndForgetAttemptPending(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -383,7 +383,7 @@ func TestEngine_StopLeavesFireAndForgetAttemptPending(t *testing.T) {
|
||||
t, s, 1, database.DeliveryStatusPending, 1,
|
||||
)
|
||||
|
||||
s.Engine.Notify([]delivery.Task{task})
|
||||
s.Engine.ExportNotify([]delivery.Task{task})
|
||||
stopDuringAttempt(t, lc, arrived)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -399,8 +399,8 @@ func TestEngine_StopLeavesFireAndForgetAttemptPending(t *testing.T) {
|
||||
// TestEngine_StopLeavesLastRetryAttemptRetrying is the same regression
|
||||
// for a target with retries, on its last attempt: the cancelled attempt
|
||||
// used to exhaust the retries and end the delivery failed. It must stay
|
||||
// retrying with only its earlier attempt recorded, and restart recovery
|
||||
// must send that last attempt again.
|
||||
// retrying with only its earlier attempt recorded, and the recovery
|
||||
// routine at the next start must send that last attempt again.
|
||||
func TestEngine_StopLeavesLastRetryAttemptRetrying(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -427,9 +427,9 @@ func TestEngine_StopLeavesLastRetryAttemptRetrying(t *testing.T) {
|
||||
|
||||
// deliverToArchive gives the setup's webhook a database target,
|
||||
// runs one delivery to it through the running engine, and returns
|
||||
// the target's ID and archive file path. The archive writer holds
|
||||
// the file open afterwards.
|
||||
func deliverToArchive(t *testing.T, s iSetup) (string, string) {
|
||||
// the target's archive file path. The archive writer holds the file
|
||||
// open afterwards.
|
||||
func deliverToArchive(t *testing.T, s iSetup) string {
|
||||
t.Helper()
|
||||
|
||||
iCreateWebhook(t, s.MainDB, s.WebhookID, "hook")
|
||||
@@ -448,11 +448,11 @@ func deliverToArchive(t *testing.T, s iSetup) (string, string) {
|
||||
task.TargetID = tgt.ID
|
||||
task.TargetType = database.TargetTypeDatabase
|
||||
|
||||
s.Engine.Notify([]delivery.Task{task})
|
||||
s.Engine.ExportNotify([]delivery.Task{task})
|
||||
|
||||
iWaitForDelivered(t, s.WebhookDB, deliveryID)
|
||||
|
||||
return tgt.ID, filepath.Join(
|
||||
return filepath.Join(
|
||||
filepath.Dir(s.DBMgr.DBPath(s.WebhookID)),
|
||||
"archive-hook-archive-"+tgt.ID+".db",
|
||||
)
|
||||
@@ -471,7 +471,7 @@ func TestEngine_StopHookClosesArchives(t *testing.T) {
|
||||
|
||||
lc := startEngineViaHook(t, s.Engine)
|
||||
|
||||
_, path := deliverToArchive(t, s)
|
||||
path := deliverToArchive(t, s)
|
||||
require.FileExists(
|
||||
t, path+"-wal",
|
||||
"an open archive should have a -wal for the stop to remove",
|
||||
@@ -505,21 +505,23 @@ func TestEngine_StopHookTimeoutLeavesArchivesOpen(t *testing.T) {
|
||||
|
||||
lc := startEngineViaHook(t, s.Engine)
|
||||
|
||||
targetID, _ := deliverToArchive(t, s)
|
||||
path := deliverToArchive(t, s)
|
||||
|
||||
release := make(chan struct{})
|
||||
|
||||
t.Cleanup(func() {
|
||||
close(release)
|
||||
s.Engine.EvictWebhook(s.WebhookID)
|
||||
s.Engine.ExportArchives().EvictWebhook(s.WebhookID)
|
||||
})
|
||||
|
||||
s.Engine.ExportWedgeWorker(release)
|
||||
|
||||
requireStopHookExpires(t, lc.hooks[0], "delivery engine")
|
||||
|
||||
require.True(
|
||||
t, s.Engine.ExportArchiveHandleOpen(targetID),
|
||||
// Closing the last handle on an archive removes its -wal, so a
|
||||
// -wal still there is a writer still open.
|
||||
require.FileExists(
|
||||
t, path+"-wal",
|
||||
"a stop that timed out must not close archive writers",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -65,20 +65,6 @@ func testEngine(
|
||||
)
|
||||
}
|
||||
|
||||
func newHTTPTargetConfig(url string) string {
|
||||
cfg := delivery.HTTPTargetConfig{URL: url}
|
||||
|
||||
data, err := json.Marshal(cfg)
|
||||
if err != nil {
|
||||
panic(
|
||||
"failed to marshal HTTPTargetConfig: " +
|
||||
err.Error(),
|
||||
)
|
||||
}
|
||||
|
||||
return string(data)
|
||||
}
|
||||
|
||||
func seedEvent(
|
||||
t *testing.T, db *gorm.DB, body string,
|
||||
) database.Event {
|
||||
@@ -141,19 +127,19 @@ type httpDeliveryFixture struct {
|
||||
func buildHTTPFixture(
|
||||
dlv database.Delivery,
|
||||
event database.Event,
|
||||
targetID, name, config string,
|
||||
targetID, name, targetURL string,
|
||||
maxAttempts, attemptNum int,
|
||||
) httpDeliveryFixture {
|
||||
task := &delivery.Task{
|
||||
DeliveryID: dlv.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: event.WebhookID,
|
||||
TargetID: targetID,
|
||||
TargetName: name,
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetConfig: config,
|
||||
MaxAttempts: maxAttempts,
|
||||
AttemptNum: attemptNum,
|
||||
DeliveryID: dlv.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: event.WebhookID,
|
||||
TargetID: targetID,
|
||||
TargetName: name,
|
||||
TargetType: database.TargetTypeHTTP,
|
||||
TargetURL: targetURL,
|
||||
MaxAttempts: maxAttempts,
|
||||
AttemptNum: attemptNum,
|
||||
}
|
||||
|
||||
d := &database.Delivery{
|
||||
@@ -164,7 +150,7 @@ func buildHTTPFixture(
|
||||
Target: database.Target{
|
||||
Name: name,
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: config,
|
||||
URL: targetURL,
|
||||
MaxAttempts: maxAttempts,
|
||||
},
|
||||
}
|
||||
@@ -232,7 +218,7 @@ func TestNotify_NonBlocking(t *testing.T) {
|
||||
done := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
e.Notify([]delivery.Task{
|
||||
e.ExportNotify([]delivery.Task{
|
||||
{DeliveryID: "overflow-1"},
|
||||
{DeliveryID: "overflow-2"},
|
||||
})
|
||||
@@ -243,7 +229,7 @@ func TestNotify_NonBlocking(t *testing.T) {
|
||||
case <-done:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal(
|
||||
"Notify blocked when delivery channel was full",
|
||||
"notify blocked when delivery channel was full",
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -275,9 +261,8 @@ func TestDeliverHTTP_Success(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := newHTTPTargetConfig(ts.URL)
|
||||
fix := buildHTTPFixture(
|
||||
dlv, event, targetID, "test-http", cfg, 1, 1,
|
||||
dlv, event, targetID, "test-http", ts.URL, 1, 1,
|
||||
)
|
||||
|
||||
e.ExportDeliverHTTP(
|
||||
@@ -321,10 +306,9 @@ func TestDeliverHTTP_Failure(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := newHTTPTargetConfig(ts.URL)
|
||||
fix := buildHTTPFixture(
|
||||
dlv, event, targetID,
|
||||
"test-http-fail", cfg, 1, 1,
|
||||
"test-http-fail", ts.URL, 1, 1,
|
||||
)
|
||||
|
||||
e.ExportDeliverHTTP(
|
||||
@@ -351,7 +335,7 @@ func TestDeliverDatabase_ImmediateSuccess(
|
||||
// The database target archives for real, so the engine needs
|
||||
// the target in the main database and a data directory.
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
tgt := env.seedDatabaseTarget(t)
|
||||
|
||||
event := seedEvent(t, db, `{"db":"target"}`)
|
||||
d := seedDatabaseTargetDelivery(t, db, event, tgt)
|
||||
@@ -452,10 +436,9 @@ func TestDeliverHTTP_WithRetries_Success(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := newHTTPTargetConfig(ts.URL)
|
||||
fix := buildHTTPFixture(
|
||||
dlv, event, targetID,
|
||||
"test-http-retry", cfg, 5, 1,
|
||||
"test-http-retry", ts.URL, 5, 1,
|
||||
)
|
||||
|
||||
e.ExportDeliverHTTP(
|
||||
@@ -510,13 +493,12 @@ func TestDeliverHTTP_MaxAttemptsExhausted(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := newHTTPTargetConfig(ts.URL)
|
||||
sched := &recordingScheduler{}
|
||||
|
||||
for attempt := 1; attempt <= maxAttempts; attempt++ {
|
||||
fix := buildHTTPFixture(
|
||||
dlv, event, targetID,
|
||||
"test-http-exhaust", cfg, maxAttempts, attempt,
|
||||
"test-http-exhaust", ts.URL, maxAttempts, attempt,
|
||||
)
|
||||
|
||||
e.ExportDeliverHTTPWithScheduler(
|
||||
@@ -574,10 +556,9 @@ func TestDeliverHTTP_SingleAttemptSkipsCircuitBreaker(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := newHTTPTargetConfig(ts.URL)
|
||||
fix := buildHTTPFixture(
|
||||
dlv, event, targetID,
|
||||
"test-http-single", cfg, 1, 1,
|
||||
"test-http-single", ts.URL, 1, 1,
|
||||
)
|
||||
sched := &recordingScheduler{}
|
||||
|
||||
@@ -624,10 +605,9 @@ func TestDeliverHTTP_SchedulesRetryOnFailure(
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := newHTTPTargetConfig(ts.URL)
|
||||
fix := buildHTTPFixture(
|
||||
dlv, event, targetID,
|
||||
"test-http-schedule", cfg, 5, 1,
|
||||
"test-http-schedule", ts.URL, 5, 1,
|
||||
)
|
||||
|
||||
e.ExportDeliverHTTP(
|
||||
@@ -789,7 +769,7 @@ func seedConcurrencyTasks(
|
||||
t *testing.T,
|
||||
db *gorm.DB,
|
||||
n int,
|
||||
targetCfg string,
|
||||
targetURL string,
|
||||
) ([]database.Delivery, []delivery.Task) {
|
||||
t.Helper()
|
||||
|
||||
@@ -815,21 +795,21 @@ func seedConcurrencyTasks(
|
||||
Target: database.Target{
|
||||
Name: fmt.Sprintf("task-%d", i),
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: targetCfg,
|
||||
URL: targetURL,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
}
|
||||
tasks[i].ID = dlv.ID
|
||||
|
||||
dtasks[i] = delivery.Task{
|
||||
DeliveryID: tasks[i].ID,
|
||||
EventID: tasks[i].EventID,
|
||||
TargetID: tasks[i].TargetID,
|
||||
TargetName: tasks[i].Target.Name,
|
||||
TargetType: tasks[i].Target.Type,
|
||||
TargetConfig: tasks[i].Target.Config,
|
||||
MaxAttempts: 1,
|
||||
AttemptNum: 1,
|
||||
DeliveryID: tasks[i].ID,
|
||||
EventID: tasks[i].EventID,
|
||||
TargetID: tasks[i].TargetID,
|
||||
TargetName: tasks[i].Target.Name,
|
||||
TargetType: tasks[i].Target.Type,
|
||||
TargetURL: tasks[i].Target.URL,
|
||||
MaxAttempts: 1,
|
||||
AttemptNum: 1,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -865,9 +845,8 @@ func TestWorkerPool_BoundedConcurrency(t *testing.T) {
|
||||
defer ts.Close()
|
||||
|
||||
e := testEngine(t, numWorkers)
|
||||
cfg := newHTTPTargetConfig(ts.URL)
|
||||
tasks, dtasks := seedConcurrencyTasks(
|
||||
t, db, numTasks, cfg,
|
||||
t, db, numTasks, ts.URL,
|
||||
)
|
||||
|
||||
runConcurrencyWorkers(
|
||||
@@ -948,12 +927,9 @@ func TestDeliverHTTP_CircuitBreakerBlocks(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg := newHTTPTargetConfig(
|
||||
"http://will-not-be-called.invalid",
|
||||
)
|
||||
fix := buildHTTPFixture(
|
||||
dlv, event, targetID,
|
||||
"test-cb-block", cfg, 5, 1,
|
||||
"test-cb-block", "http://will-not-be-called.invalid", 5, 1,
|
||||
)
|
||||
|
||||
e.ExportDeliverHTTP(
|
||||
@@ -1016,9 +992,8 @@ func TestDeliverHTTP_HalfOpenBreakerDelaysQueuedTasks(t *testing.T) {
|
||||
require.True(t, cb.Allow(), "the probe delivery should go through")
|
||||
require.Equal(t, delivery.CircuitHalfOpen, cb.State())
|
||||
|
||||
cfg := newHTTPTargetConfig(
|
||||
"http://will-not-be-called.invalid",
|
||||
)
|
||||
const targetURL = "http://will-not-be-called.invalid"
|
||||
|
||||
sched := &recordingScheduler{}
|
||||
|
||||
const queued, passes = 3, 4
|
||||
@@ -1040,7 +1015,7 @@ func TestDeliverHTTP_HalfOpenBreakerDelaysQueuedTasks(t *testing.T) {
|
||||
|
||||
fix := buildHTTPFixture(
|
||||
row, event, targetID,
|
||||
"test-cb-half-open", cfg, 5, 1,
|
||||
"test-cb-half-open", targetURL, 5, 1,
|
||||
)
|
||||
|
||||
e.ExportDeliverHTTPWithScheduler(
|
||||
@@ -1155,62 +1130,6 @@ func TestStateAndCooldown_ReadsHTTPAndSlackBreakers(t *testing.T) {
|
||||
assert.Zero(t, cooldown, "half-open")
|
||||
}
|
||||
|
||||
func TestParseHTTPConfig_Valid(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
cfg, err := e.ExportParseHTTPConfig(
|
||||
`{"url":"https://example.com/hook",` +
|
||||
`"headers":{"X-Token":"secret"}}`,
|
||||
)
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "https://example.com/hook", cfg.URL)
|
||||
assert.Equal(t, "secret", cfg.Headers["X-Token"])
|
||||
}
|
||||
|
||||
func TestParseHTTPConfig_Empty(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
_, err := e.ExportParseHTTPConfig("")
|
||||
|
||||
assert.Error(t, err,
|
||||
"empty config should return error",
|
||||
)
|
||||
}
|
||||
|
||||
func TestParseHTTPConfig_MissingURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
_, err := e.ExportParseHTTPConfig(
|
||||
`{"headers":{"X-Token":"secret"}}`,
|
||||
)
|
||||
|
||||
assert.Error(t, err,
|
||||
"config without URL should return error",
|
||||
)
|
||||
}
|
||||
|
||||
func TestParseHTTPConfig_Undecodable(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
_, err := e.ExportParseHTTPConfig(
|
||||
`{"url":"https://example.com/hook","timeout":"soon"}`,
|
||||
)
|
||||
|
||||
assert.Error(t, err,
|
||||
"config that does not decode should return error, "+
|
||||
"even when the part that did names a URL",
|
||||
)
|
||||
}
|
||||
|
||||
func TestScheduleRetry_SendsToRetryChannel(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -1358,7 +1277,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
cfg := &delivery.HTTPTargetConfig{
|
||||
target := &database.Target{
|
||||
URL: ts.URL,
|
||||
Headers: map[string]string{
|
||||
"X-Target-Auth": "bearer xyz",
|
||||
@@ -1373,7 +1292,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
|
||||
}
|
||||
|
||||
statusCode, _, _, err := e.ExportDoHTTPRequest(
|
||||
context.TODO(), cfg, event,
|
||||
context.TODO(), target, event,
|
||||
)
|
||||
|
||||
require.NoError(t, err)
|
||||
@@ -1415,7 +1334,7 @@ func TestDoHTTPRequest_CutShortResponseIsAnError(t *testing.T) {
|
||||
|
||||
_, body, _, err := e.ExportDoHTTPRequest(
|
||||
context.TODO(),
|
||||
&delivery.HTTPTargetConfig{URL: ts.URL},
|
||||
&database.Target{URL: ts.URL},
|
||||
&database.Event{},
|
||||
)
|
||||
|
||||
@@ -1467,9 +1386,9 @@ func TestApplyRequestHeaders_SendsOneContentType(t *testing.T) {
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
cfg := &delivery.HTTPTargetConfig{}
|
||||
var configured map[string]string
|
||||
if tc.configured != "" {
|
||||
cfg.Headers = map[string]string{
|
||||
configured = map[string]string{
|
||||
headerContentType: tc.configured,
|
||||
}
|
||||
}
|
||||
@@ -1488,7 +1407,7 @@ func TestApplyRequestHeaders_SendsOneContentType(t *testing.T) {
|
||||
Headers: string(inbound),
|
||||
ContentType: tc.event,
|
||||
},
|
||||
cfg,
|
||||
configured,
|
||||
"webhooker/dev",
|
||||
)
|
||||
|
||||
@@ -1519,7 +1438,7 @@ func TestApplyRequestHeaders_UndecodableInboundForwardsNothing(
|
||||
&database.Event{
|
||||
Headers: `{"X-Custom":["value1"],"X-Broken":"not a list"}`,
|
||||
},
|
||||
&delivery.HTTPTargetConfig{},
|
||||
nil,
|
||||
"webhooker/dev",
|
||||
)
|
||||
|
||||
@@ -1537,7 +1456,7 @@ func TestProcessDelivery_RoutesToCorrectHandler(
|
||||
// The database target archives for real, so the engine needs
|
||||
// the target in the main database and a data directory.
|
||||
env := setupArchiveTest(t)
|
||||
archive := env.seedDatabaseTarget(t, "")
|
||||
archive := env.seedDatabaseTarget(t)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -1622,51 +1541,6 @@ func TestMaxInlineBodySize_Constant(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
func TestParseSlackConfig_Valid(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
cfg, err := e.ExportParseSlackConfig(
|
||||
`{"webhookUrl":"https://hooks.slack.com/services/T00/B00/xxx"}`,
|
||||
)
|
||||
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t,
|
||||
"https://hooks.slack.com/services/T00/B00/xxx",
|
||||
cfg.WebhookURL,
|
||||
)
|
||||
}
|
||||
|
||||
func TestParseSlackConfig_Empty(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
_, err := e.ExportParseSlackConfig("")
|
||||
|
||||
assert.Error(t, err,
|
||||
"empty config should return error",
|
||||
)
|
||||
}
|
||||
|
||||
func TestParseSlackConfig_MissingWebhookURL(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
_, err := e.ExportParseSlackConfig(
|
||||
`{"other":"field"}`,
|
||||
)
|
||||
|
||||
assert.Error(t, err,
|
||||
"config without webhook_url should return error",
|
||||
)
|
||||
}
|
||||
|
||||
func TestFormatSlackMessage_JSONBody(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1763,11 +1637,11 @@ func TestFormatSlackMessage_LargeJSONTruncated(
|
||||
}
|
||||
|
||||
// buildSlackDelivery creates a database.Delivery
|
||||
// with a Slack target config for testing.
|
||||
// to a Slack target with the given webhook URL for testing.
|
||||
func buildSlackDelivery(
|
||||
dlv database.Delivery,
|
||||
event database.Event,
|
||||
targetID, name, slackCfg string,
|
||||
targetID, name, webhookURL string,
|
||||
) *database.Delivery {
|
||||
d := &database.Delivery{
|
||||
EventID: event.ID,
|
||||
@@ -1777,7 +1651,7 @@ func buildSlackDelivery(
|
||||
Target: database.Target{
|
||||
Name: name,
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: slackCfg,
|
||||
URL: webhookURL,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
}
|
||||
@@ -1815,11 +1689,10 @@ func TestDeliverSlack_Success(t *testing.T) {
|
||||
)
|
||||
defer ts.Close()
|
||||
|
||||
e, targetID, slackCfg, event, dlv :=
|
||||
setupSlackTest(t, db, ts.URL)
|
||||
e, targetID, event, dlv := setupSlackTest(t, db)
|
||||
|
||||
d := buildSlackDelivery(
|
||||
dlv, event, targetID, "test-slack", slackCfg,
|
||||
dlv, event, targetID, "test-slack", ts.URL,
|
||||
)
|
||||
|
||||
e.ExportDeliverSlack(context.TODO(), db, d)
|
||||
@@ -1838,9 +1711,8 @@ func TestDeliverSlack_Success(t *testing.T) {
|
||||
func setupSlackTest(
|
||||
t *testing.T,
|
||||
db *gorm.DB,
|
||||
serverURL string,
|
||||
) (
|
||||
*delivery.Engine, string, string,
|
||||
*delivery.Engine, string,
|
||||
database.Event, database.Delivery,
|
||||
) {
|
||||
t.Helper()
|
||||
@@ -1848,13 +1720,6 @@ func setupSlackTest(
|
||||
e := testEngine(t, 1)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
cfgBytes, err := json.Marshal(
|
||||
delivery.SlackTargetConfig{
|
||||
WebhookURL: serverURL,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
event := seedEvent(
|
||||
t, db, `{"action":"test","data":"value"}`,
|
||||
)
|
||||
@@ -1864,7 +1729,7 @@ func setupSlackTest(
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
return e, targetID, string(cfgBytes), event, dlv
|
||||
return e, targetID, event, dlv
|
||||
}
|
||||
|
||||
func assertSlackPayload(
|
||||
@@ -1913,13 +1778,6 @@ func TestDeliverSlack_Failure(t *testing.T) {
|
||||
e := testEngine(t, 1)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
slackCfg, err := json.Marshal(
|
||||
delivery.SlackTargetConfig{
|
||||
WebhookURL: ts.URL,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
event := seedEvent(t, db, `{"test":true}`)
|
||||
|
||||
dlv := seedDelivery(
|
||||
@@ -1929,7 +1787,7 @@ func TestDeliverSlack_Failure(t *testing.T) {
|
||||
|
||||
d := buildSlackDelivery(
|
||||
dlv, event, targetID,
|
||||
"test-slack-fail", string(slackCfg),
|
||||
"test-slack-fail", ts.URL,
|
||||
)
|
||||
|
||||
e.ExportDeliverSlack(context.TODO(), db, d)
|
||||
@@ -1943,7 +1801,10 @@ func TestDeliverSlack_Failure(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
func TestDeliverSlack_InvalidConfig(t *testing.T) {
|
||||
// TestDeliverSlack_NoURL proves a Slack target without a URL, which
|
||||
// the forms never store, fails its delivery rather than reporting it
|
||||
// delivered.
|
||||
func TestDeliverSlack_NoURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := testWebhookDB(t)
|
||||
@@ -1964,7 +1825,6 @@ func TestDeliverSlack_InvalidConfig(t *testing.T) {
|
||||
Target: database.Target{
|
||||
Name: "test-slack-bad",
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"not_webhook_url":"missing"}`,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
}
|
||||
@@ -2002,13 +1862,6 @@ func TestProcessDelivery_RoutesToSlack(t *testing.T) {
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
slackCfg, err := json.Marshal(
|
||||
delivery.SlackTargetConfig{
|
||||
WebhookURL: ts.URL,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
event := seedEvent(t, db, `{"route":"slack"}`)
|
||||
|
||||
dlv := seedDelivery(
|
||||
@@ -2018,7 +1871,7 @@ func TestProcessDelivery_RoutesToSlack(t *testing.T) {
|
||||
|
||||
d := buildSlackDelivery(
|
||||
dlv, event, dlv.TargetID,
|
||||
"test-slack-route", string(slackCfg),
|
||||
"test-slack-route", ts.URL,
|
||||
)
|
||||
|
||||
task := &delivery.Task{
|
||||
@@ -2133,7 +1986,7 @@ func TestDeliverLog_LogsFullContent(t *testing.T) {
|
||||
func buildSlackRetryDelivery(
|
||||
dlv database.Delivery,
|
||||
event database.Event,
|
||||
targetID, cfg string,
|
||||
targetID, webhookURL string,
|
||||
) *database.Delivery {
|
||||
d := &database.Delivery{
|
||||
EventID: event.ID,
|
||||
@@ -2143,7 +1996,7 @@ func buildSlackRetryDelivery(
|
||||
Target: database.Target{
|
||||
Name: "test-slack-retry",
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: cfg,
|
||||
URL: webhookURL,
|
||||
MaxAttempts: 5,
|
||||
},
|
||||
}
|
||||
@@ -2162,11 +2015,6 @@ func TestDeliverSlack_WithRetries_SchedulesRetry(
|
||||
e := testEngine(t, 1)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
slackCfg, err := json.Marshal(
|
||||
delivery.SlackTargetConfig{WebhookURL: ts.URL},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
event := seedEvent(t, db, `{"slack":"retry"}`)
|
||||
|
||||
dlv := seedDelivery(
|
||||
@@ -2175,7 +2023,7 @@ func TestDeliverSlack_WithRetries_SchedulesRetry(
|
||||
)
|
||||
|
||||
d := buildSlackRetryDelivery(
|
||||
dlv, event, targetID, string(slackCfg),
|
||||
dlv, event, targetID, ts.URL,
|
||||
)
|
||||
|
||||
task := &delivery.Task{
|
||||
|
||||
@@ -69,7 +69,6 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
|
||||
))
|
||||
defer ts.Close()
|
||||
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
targetID := uuid.New().String()
|
||||
dbPath := s.DBMgr.DBPath(s.WebhookID)
|
||||
|
||||
@@ -85,7 +84,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
|
||||
)
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"leaky-target", cfg, 5, 1, &body,
|
||||
"leaky-target", ts.URL, 5, 1, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
@@ -102,7 +101,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/107.
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "leaky-target",
|
||||
database.TargetTypeHTTP, cfg, 5,
|
||||
database.TargetTypeHTTP, ts.URL, 5,
|
||||
)
|
||||
|
||||
rd := iSeedDelivery(
|
||||
@@ -111,7 +110,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
|
||||
)
|
||||
rTask := iTask(
|
||||
rd, event, s.WebhookID, targetID,
|
||||
"leaky-target", cfg, 5, 2, &body,
|
||||
"leaky-target", ts.URL, 5, 2, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(context.TODO(), &rTask)
|
||||
@@ -138,7 +137,6 @@ func TestEventDBHoldsNoTargetRowsOnFailedDelivery(t *testing.T) {
|
||||
))
|
||||
defer ts.Close()
|
||||
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
event := iSeedEvent(
|
||||
@@ -152,7 +150,7 @@ func TestEventDBHoldsNoTargetRowsOnFailedDelivery(t *testing.T) {
|
||||
)
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"leaky-target", cfg, 1, 1, &body,
|
||||
"leaky-target", ts.URL, 1, 1, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
|
||||
@@ -93,17 +93,6 @@ func (s *tsSink) text(t *testing.T) string {
|
||||
}
|
||||
}
|
||||
|
||||
func tsSlackConfig(t *testing.T, url string) string {
|
||||
t.Helper()
|
||||
|
||||
data, err := json.Marshal(
|
||||
delivery.SlackTargetConfig{WebhookURL: url},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
return string(data)
|
||||
}
|
||||
|
||||
// tsSeedEvent writes an event whose CreatedAt is tsEventCreatedAt
|
||||
// rather than the write time, so an assertion on the rendered
|
||||
// timestamp cannot pass by accident against "roughly now".
|
||||
@@ -142,7 +131,7 @@ func tsSeedEvent(
|
||||
// tsSeedTarget writes the slack target row into the main database.
|
||||
// The retry path confirms the target still exists before sending.
|
||||
func tsSeedTarget(
|
||||
t *testing.T, mainDB *gorm.DB, webhookID, config string,
|
||||
t *testing.T, mainDB *gorm.DB, webhookID, webhookURL string,
|
||||
) database.Target {
|
||||
t.Helper()
|
||||
|
||||
@@ -150,7 +139,7 @@ func tsSeedTarget(
|
||||
WebhookID: webhookID,
|
||||
Name: "slack-sink",
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: config,
|
||||
URL: webhookURL,
|
||||
Active: true,
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
@@ -176,7 +165,7 @@ func tsTask(
|
||||
TargetID: target.ID,
|
||||
TargetName: target.Name,
|
||||
TargetType: database.TargetTypeSlack,
|
||||
TargetConfig: target.Config,
|
||||
TargetURL: target.URL,
|
||||
MaxAttempts: 1,
|
||||
Method: event.Method,
|
||||
Headers: event.Headers,
|
||||
@@ -226,8 +215,7 @@ func (c tsCase) run(t *testing.T) (iSetup, database.Delivery, string) {
|
||||
s := newISetup(t)
|
||||
sink := newTSSink(t)
|
||||
|
||||
cfg := tsSlackConfig(t, sink.URL)
|
||||
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
|
||||
target := tsSeedTarget(t, s.MainDB, s.WebhookID, sink.URL)
|
||||
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
|
||||
|
||||
d := iSeedDelivery(
|
||||
@@ -338,8 +326,7 @@ func TestFormatSlackMessageOverTaskReconstructedEvent(
|
||||
|
||||
s := newISetup(t)
|
||||
|
||||
cfg := tsSlackConfig(t, tsUndeliverableHook)
|
||||
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
|
||||
target := tsSeedTarget(t, s.MainDB, s.WebhookID, tsUndeliverableHook)
|
||||
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
|
||||
|
||||
d := iSeedDelivery(
|
||||
@@ -410,8 +397,7 @@ func TestEventReconstructionSurvivesAReapedRow(t *testing.T) {
|
||||
|
||||
s := newISetup(t)
|
||||
|
||||
cfg := tsSlackConfig(t, tsUndeliverableHook)
|
||||
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
|
||||
target := tsSeedTarget(t, s.MainDB, s.WebhookID, tsUndeliverableHook)
|
||||
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
|
||||
|
||||
d := iSeedDelivery(
|
||||
|
||||
@@ -12,16 +12,11 @@ import (
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"go.uber.org/fx"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
)
|
||||
|
||||
// ErrExportArchiveWriterEvicted exposes the sentinel returned by
|
||||
// an evicted archive writer. It carries the Err prefix rather
|
||||
// than this file's usual Export one because it is a sentinel
|
||||
// error.
|
||||
var ErrExportArchiveWriterEvicted = errArchiveWriterEvicted
|
||||
|
||||
// Exported constants for test access.
|
||||
const (
|
||||
ExportDeliveryChannelSize = deliveryChannelSize
|
||||
@@ -82,10 +77,10 @@ func ExportIsForwardableHeader(name string) bool {
|
||||
func ExportApplyRequestHeaders(
|
||||
req *http.Request,
|
||||
event *database.Event,
|
||||
cfg *HTTPTargetConfig,
|
||||
configured map[string]string,
|
||||
userAgent string,
|
||||
) []string {
|
||||
return applyRequestHeaders(req, event, cfg, userAgent)
|
||||
return applyRequestHeaders(req, event, configured, userAgent)
|
||||
}
|
||||
|
||||
// ExportTruncate exposes truncate for testing.
|
||||
@@ -163,6 +158,11 @@ func (e *Engine) ExportUpdateDeliveryStatus(
|
||||
return e.updateDeliveryStatus(webhookDB, d, "", status)
|
||||
}
|
||||
|
||||
// ExportNotify exposes notify.
|
||||
func (e *Engine) ExportNotify(tasks []Task) {
|
||||
e.notify(tasks)
|
||||
}
|
||||
|
||||
// ExportProcessNewTask exposes processNewTask.
|
||||
func (e *Engine) ExportProcessNewTask(
|
||||
ctx context.Context, task *Task,
|
||||
@@ -221,37 +221,23 @@ func (e *Engine) ExportSetSlackCircuitBreaker(
|
||||
e.slackTarget.circuitBreakers.Store(targetID, cb)
|
||||
}
|
||||
|
||||
// ExportParseHTTPConfig exposes parseHTTPConfig.
|
||||
func (e *Engine) ExportParseHTTPConfig(
|
||||
configJSON string,
|
||||
) (*HTTPTargetConfig, error) {
|
||||
return parseHTTPConfig(configJSON)
|
||||
}
|
||||
|
||||
// ExportParseSlackConfig exposes parseSlackConfig.
|
||||
func (e *Engine) ExportParseSlackConfig(
|
||||
configJSON string,
|
||||
) (*SlackTargetConfig, error) {
|
||||
return parseSlackConfig(configJSON)
|
||||
}
|
||||
|
||||
// ExportDoHTTPRequest exposes the http target's
|
||||
// doHTTPRequest.
|
||||
func (e *Engine) ExportDoHTTPRequest(
|
||||
ctx context.Context,
|
||||
cfg *HTTPTargetConfig,
|
||||
target *database.Target,
|
||||
event *database.Event,
|
||||
) (int, string, int64, error) {
|
||||
return e.httpTarget.doHTTPRequest(ctx, cfg, event)
|
||||
return e.httpTarget.doHTTPRequest(ctx, target, event)
|
||||
}
|
||||
|
||||
// ExportClientForRequest exposes the http target's
|
||||
// clientForRequest.
|
||||
func (e *Engine) ExportClientForRequest(
|
||||
cfg *HTTPTargetConfig,
|
||||
timeoutSeconds int,
|
||||
originScoped []string,
|
||||
) *http.Client {
|
||||
return e.httpTarget.clientForRequest(cfg, originScoped)
|
||||
return e.httpTarget.clientForRequest(timeoutSeconds, originScoped)
|
||||
}
|
||||
|
||||
// ErrExportTooManyRedirects exposes the sentinel the redirect
|
||||
@@ -286,32 +272,27 @@ func (e *Engine) ExportRecoverPendingDeliveries(
|
||||
ctx context.Context,
|
||||
webhookDB *gorm.DB,
|
||||
webhookID string,
|
||||
atStart bool,
|
||||
) {
|
||||
e.recoverPendingDeliveries(
|
||||
ctx, webhookDB, webhookID,
|
||||
ctx, webhookDB, webhookID, atStart,
|
||||
)
|
||||
}
|
||||
|
||||
// ExportRecoverWebhookDeliveries exposes
|
||||
// recoverWebhookDeliveries.
|
||||
func (e *Engine) ExportRecoverWebhookDeliveries(
|
||||
ctx context.Context, webhookID string,
|
||||
ctx context.Context, webhookID string, atStart bool,
|
||||
) {
|
||||
e.recoverWebhookDeliveries(ctx, webhookID)
|
||||
e.recoverWebhookDeliveries(ctx, webhookID, atStart)
|
||||
}
|
||||
|
||||
// ExportRecoverInFlight exposes recoverInFlight.
|
||||
func (e *Engine) ExportRecoverInFlight(
|
||||
ctx context.Context,
|
||||
// ExportRecoverDeliveries exposes recoverDeliveries, the recovery
|
||||
// routine.
|
||||
func (e *Engine) ExportRecoverDeliveries(
|
||||
ctx context.Context, atStart bool,
|
||||
) {
|
||||
e.recoverInFlight(ctx)
|
||||
}
|
||||
|
||||
// ExportSweepWebhookRetries exposes sweepWebhookRetries.
|
||||
func (e *Engine) ExportSweepWebhookRetries(
|
||||
ctx context.Context, webhookID string,
|
||||
) {
|
||||
e.sweepWebhookRetries(ctx, webhookID)
|
||||
e.recoverDeliveries(ctx, atStart)
|
||||
}
|
||||
|
||||
// ExportStart exposes start for testing.
|
||||
@@ -358,9 +339,9 @@ func (e *Engine) ExportRetainDelivery(deliveryID string) bool {
|
||||
|
||||
// ExportRecoverRetryingDeliveries exposes recoverRetryingDeliveries.
|
||||
func (e *Engine) ExportRecoverRetryingDeliveries(
|
||||
webhookDB *gorm.DB, webhookID string,
|
||||
ctx context.Context, webhookDB *gorm.DB, webhookID string,
|
||||
) {
|
||||
e.recoverRetryingDeliveries(webhookDB, webhookID)
|
||||
e.recoverRetryingDeliveries(ctx, webhookDB, webhookID)
|
||||
}
|
||||
|
||||
// ExportFailMissingTarget exposes failMissingTarget, so a test can hand
|
||||
@@ -399,7 +380,8 @@ func (e *Engine) ExportRetryCh() chan Task {
|
||||
}
|
||||
|
||||
// NewTestEngine creates an Engine for unit tests without
|
||||
// database dependencies.
|
||||
// database dependencies. Its archive writers have no data
|
||||
// directory, so a delivery to a database target fails.
|
||||
func NewTestEngine(
|
||||
log *slog.Logger,
|
||||
client *http.Client,
|
||||
@@ -411,6 +393,7 @@ func NewTestEngine(
|
||||
retryCh: make(chan Task, retryChannelSize),
|
||||
workers: workers,
|
||||
mtr: metrics.New(prometheus.NewRegistry()),
|
||||
archives: archive.NewWriters(archive.WritersParams{Logger: log}),
|
||||
}
|
||||
e.initTargets(client)
|
||||
|
||||
@@ -423,9 +406,10 @@ func NewTestEngineSmallRetry(
|
||||
log *slog.Logger,
|
||||
) *Engine {
|
||||
e := &Engine{
|
||||
log: log,
|
||||
retryCh: make(chan Task, 1),
|
||||
mtr: metrics.New(prometheus.NewRegistry()),
|
||||
log: log,
|
||||
retryCh: make(chan Task, 1),
|
||||
mtr: metrics.New(prometheus.NewRegistry()),
|
||||
archives: archive.NewWriters(archive.WritersParams{Logger: log}),
|
||||
}
|
||||
e.initTargets(nil)
|
||||
|
||||
@@ -433,7 +417,8 @@ func NewTestEngineSmallRetry(
|
||||
}
|
||||
|
||||
// NewTestEngineWithDB creates an Engine with a real
|
||||
// database and dbManager for integration tests.
|
||||
// database and dbManager for integration tests, and archive
|
||||
// writers of its own over the same two.
|
||||
func NewTestEngineWithDB(
|
||||
db *database.Database,
|
||||
dbMgr *database.WebhookDBManager,
|
||||
@@ -449,12 +434,22 @@ func NewTestEngineWithDB(
|
||||
retryCh: make(chan Task, retryChannelSize),
|
||||
workers: workers,
|
||||
mtr: metrics.New(prometheus.NewRegistry()),
|
||||
archives: archive.NewWriters(archive.WritersParams{
|
||||
Database: db,
|
||||
DBManager: dbMgr,
|
||||
Logger: log,
|
||||
}),
|
||||
}
|
||||
e.initTargets(client)
|
||||
|
||||
return e
|
||||
}
|
||||
|
||||
// ExportArchives returns the engine's archive writers.
|
||||
func (e *Engine) ExportArchives() *archive.Writers {
|
||||
return e.archives
|
||||
}
|
||||
|
||||
// ExportSetMetrics substitutes the engine's metric set, so a test can
|
||||
// assert on collectors registered on a registry it holds.
|
||||
func (e *Engine) ExportSetMetrics(mtr *metrics.Set) {
|
||||
@@ -477,273 +472,3 @@ func NewTestCircuitBreaker(
|
||||
cooldown: cooldown,
|
||||
}
|
||||
}
|
||||
|
||||
// ExportArchivedEvent aliases the archive row type so black-box
|
||||
// tests can construct and read archive rows.
|
||||
type ExportArchivedEvent = archivedEvent
|
||||
|
||||
// ExportArchiveWriter wraps an archiveWriter so black-box tests
|
||||
// can exercise the archive file mechanics.
|
||||
type ExportArchiveWriter struct {
|
||||
w *archiveWriter
|
||||
}
|
||||
|
||||
// NewExportArchiveWriter builds an archive writer for tests,
|
||||
// optionally overriding the reopen debounce (a non-positive
|
||||
// debounce keeps the production default).
|
||||
func NewExportArchiveWriter(
|
||||
path string, log *slog.Logger, debounce time.Duration,
|
||||
) *ExportArchiveWriter {
|
||||
w := newArchiveWriter(path, log)
|
||||
if debounce > 0 {
|
||||
w.debounce = debounce
|
||||
}
|
||||
|
||||
return &ExportArchiveWriter{w: w}
|
||||
}
|
||||
|
||||
// Write archives a row through the writer, into the file named
|
||||
// without a period.
|
||||
func (e *ExportArchiveWriter) Write(
|
||||
row ExportArchivedEvent, expiry time.Duration,
|
||||
) error {
|
||||
return e.w.write(row, expiry, "")
|
||||
}
|
||||
|
||||
// WritePeriod archives a row through the writer, into the file for
|
||||
// period.
|
||||
func (e *ExportArchiveWriter) WritePeriod(
|
||||
row ExportArchivedEvent, expiry time.Duration, period string,
|
||||
) error {
|
||||
return e.w.write(row, expiry, period)
|
||||
}
|
||||
|
||||
// Open opens the archive file, pruning when expiry is positive.
|
||||
func (e *ExportArchiveWriter) Open(expiry time.Duration) error {
|
||||
return e.w.open(e.w.path, expiry)
|
||||
}
|
||||
|
||||
// Reopen closes and reopens the archive file.
|
||||
func (e *ExportArchiveWriter) Reopen(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.reopen(e.w.path, expiry)
|
||||
}
|
||||
|
||||
// SetNow replaces the clock the writer measures its reopen
|
||||
// debounce on.
|
||||
func (e *ExportArchiveWriter) SetNow(now func() time.Time) {
|
||||
e.w.now = now
|
||||
}
|
||||
|
||||
// Reopens reports how many times the file has been opened.
|
||||
func (e *ExportArchiveWriter) Reopens() int {
|
||||
return e.w.reopens
|
||||
}
|
||||
|
||||
// DB returns the writer's current open handle for row
|
||||
// inspection in tests.
|
||||
func (e *ExportArchiveWriter) DB() *gorm.DB {
|
||||
return e.w.db
|
||||
}
|
||||
|
||||
// Path returns the archive file the writer owns.
|
||||
func (e *ExportArchiveWriter) Path() string {
|
||||
return e.w.path
|
||||
}
|
||||
|
||||
// OpenExisting opens the archive without permitting creation,
|
||||
// the way the idle sweep does.
|
||||
func (e *ExportArchiveWriter) OpenExisting(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.openMode(e.w.path, archiveModeExisting, expiry)
|
||||
}
|
||||
|
||||
// SweepExpired runs an idle sweep of the archive.
|
||||
func (e *ExportArchiveWriter) SweepExpired(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.sweepExpired(expiry)
|
||||
}
|
||||
|
||||
// Evict marks the writer evicted and closes its handle, exactly
|
||||
// as leaving the registry does.
|
||||
func (e *ExportArchiveWriter) Evict() {
|
||||
e.w.evict()
|
||||
}
|
||||
|
||||
// Rename gives the archive file a new name in the same directory,
|
||||
// as a rename of the webhook or target does.
|
||||
func (e *ExportArchiveWriter) Rename(name string) error {
|
||||
return e.w.rename(name)
|
||||
}
|
||||
|
||||
// HandleOpen reports whether the writer currently holds an open
|
||||
// archive handle.
|
||||
func (e *ExportArchiveWriter) HandleOpen() bool {
|
||||
e.w.mu.Lock()
|
||||
defer e.w.mu.Unlock()
|
||||
|
||||
return e.w.db != nil
|
||||
}
|
||||
|
||||
// Same reports whether both wrappers refer to the very same
|
||||
// underlying archive writer, so a test can prove a registry entry
|
||||
// is the writer it was handed rather than a replacement.
|
||||
func (e *ExportArchiveWriter) Same(
|
||||
other *ExportArchiveWriter,
|
||||
) bool {
|
||||
return other != nil && e.w == other.w
|
||||
}
|
||||
|
||||
// ExportArchiveWriterFor returns the archive writer the registry
|
||||
// currently caches for a database target, or nil when none is
|
||||
// cached. It never creates one, so a test can hold a reference to
|
||||
// the very writer an eviction is about to detach.
|
||||
func (e *Engine) ExportArchiveWriterFor(
|
||||
targetID string,
|
||||
) *ExportArchiveWriter {
|
||||
e.dbTarget.mu.Lock()
|
||||
defer e.dbTarget.mu.Unlock()
|
||||
|
||||
w, ok := e.dbTarget.writers[targetID]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &ExportArchiveWriter{w: w}
|
||||
}
|
||||
|
||||
// ExportHasArchiveWriter reports whether the database target
|
||||
// type currently caches an archive writer for a target.
|
||||
func (e *Engine) ExportHasArchiveWriter(
|
||||
targetID string,
|
||||
) bool {
|
||||
e.dbTarget.mu.Lock()
|
||||
defer e.dbTarget.mu.Unlock()
|
||||
|
||||
_, ok := e.dbTarget.writers[targetID]
|
||||
|
||||
return ok
|
||||
}
|
||||
|
||||
// ExportArchiveHandleOpen reports whether the cached archive
|
||||
// writer for a target holds an open database handle. It
|
||||
// returns false when no writer is cached.
|
||||
func (e *Engine) ExportArchiveHandleOpen(
|
||||
targetID string,
|
||||
) bool {
|
||||
e.dbTarget.mu.Lock()
|
||||
w, ok := e.dbTarget.writers[targetID]
|
||||
e.dbTarget.mu.Unlock()
|
||||
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
return w.db != nil
|
||||
}
|
||||
|
||||
// ExportEnsureArchiveWriter creates (if needed) and returns the
|
||||
// archive file path of the cached writer for a target, so a
|
||||
// test can prime the registry the way a delivery would.
|
||||
func (e *Engine) ExportEnsureArchiveWriter(
|
||||
targetID string,
|
||||
) (string, error) {
|
||||
w, err := e.dbTarget.writerFor(targetID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return w.path, nil
|
||||
}
|
||||
|
||||
// ExportSweepWriterFor takes a target's registry writer exactly
|
||||
// as the idle sweep does, reporting whether the sweep had to
|
||||
// create the entry. It lets a test drive the registry through the
|
||||
// sweep's own entry point instead of choreographing goroutines.
|
||||
func (e *Engine) ExportSweepWriterFor(
|
||||
targetID string,
|
||||
) (*ExportArchiveWriter, bool, error) {
|
||||
w, created, err := e.dbTarget.sweepWriterFor(targetID)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
|
||||
return &ExportArchiveWriter{w: w}, created, nil
|
||||
}
|
||||
|
||||
// ExportReleaseSweepWriter releases a sweep-created registry entry
|
||||
// exactly as a finished sweep does.
|
||||
func (e *Engine) ExportReleaseSweepWriter(
|
||||
targetID string, w *ExportArchiveWriter,
|
||||
) {
|
||||
e.dbTarget.releaseSweepWriter(targetID, w.w)
|
||||
}
|
||||
|
||||
// NewTestArchiveSweeper builds an ArchiveSweeper backed by the
|
||||
// given main database and engine, without the fx lifecycle.
|
||||
// Intended for tests.
|
||||
func NewTestArchiveSweeper(
|
||||
db *database.Database,
|
||||
eng *Engine,
|
||||
log *slog.Logger,
|
||||
) *ArchiveSweeper {
|
||||
return &ArchiveSweeper{
|
||||
db: db,
|
||||
eng: eng,
|
||||
log: log,
|
||||
interval: time.Hour,
|
||||
}
|
||||
}
|
||||
|
||||
// ExportSweep runs a single archive sweep synchronously for
|
||||
// tests.
|
||||
func (s *ArchiveSweeper) ExportSweep(ctx context.Context) {
|
||||
s.sweep(ctx)
|
||||
}
|
||||
|
||||
// ExportStart starts the sweeper's background loop for tests.
|
||||
func (s *ArchiveSweeper) ExportStart() {
|
||||
s.start()
|
||||
}
|
||||
|
||||
// ExportRegisterHooks registers the sweeper's real fx lifecycle
|
||||
// hooks on a lifecycle supplied by a test, so a test can drive
|
||||
// the exact OnStart/OnStop functions the application runs and
|
||||
// hand OnStart the kind of context fx actually supplies.
|
||||
func (s *ArchiveSweeper) ExportRegisterHooks(lc fx.Lifecycle) {
|
||||
s.registerHooks(lc)
|
||||
}
|
||||
|
||||
// ExportStop stops the sweeper's background loop for tests.
|
||||
func (s *ArchiveSweeper) ExportStop(ctx context.Context) error {
|
||||
return s.stop(ctx)
|
||||
}
|
||||
|
||||
// ExportWedgeLoop adds a goroutine to the sweeper's WaitGroup
|
||||
// that never observes cancellation and returns only when release
|
||||
// is closed. It stands in for a prune stuck on a locked archive.
|
||||
func (s *ArchiveSweeper) ExportWedgeLoop(
|
||||
release <-chan struct{},
|
||||
) {
|
||||
s.wg.Go(func() {
|
||||
<-release
|
||||
})
|
||||
}
|
||||
|
||||
// ExportSetInterval overrides the sweep interval for tests.
|
||||
func (s *ArchiveSweeper) ExportSetInterval(d time.Duration) {
|
||||
s.interval = d
|
||||
}
|
||||
|
||||
// ExportParseArchiveExpiry exposes parseArchiveExpiry.
|
||||
func ExportParseArchiveExpiry(
|
||||
configJSON string,
|
||||
) (time.Duration, error) {
|
||||
return parseArchiveExpiry(configJSON)
|
||||
}
|
||||
|
||||
@@ -6,10 +6,10 @@ import "sync"
|
||||
//
|
||||
// A delivery is owned from the moment a task for it is handed to a
|
||||
// channel or to a retry timer until the engine has no further plan for
|
||||
// it in memory. Restart recovery and both arms of the periodic sweep
|
||||
// re-dispatch only deliveries the set does not hold, which is what
|
||||
// makes them exact rather than a guess about how long a row has sat at
|
||||
// pending.
|
||||
// it in memory. The recovery routine, at start and on every tick of the
|
||||
// sweep, re-dispatches only deliveries the set does not hold, which is
|
||||
// what makes it exact rather than a guess about how long a row has sat
|
||||
// at pending.
|
||||
//
|
||||
// This replaces reasoning from timestamps. A delivery's row says
|
||||
// pending from creation until its outcome is written, which covers
|
||||
@@ -25,8 +25,8 @@ import "sync"
|
||||
// In-memory state is sufficient because a data directory admits one
|
||||
// process: internal/datadir takes an flock on it at startup and a
|
||||
// second instance refuses to run. Deliveries owned by a process that
|
||||
// died are not in any successor's set, and restart recovery is what
|
||||
// picks those up.
|
||||
// died are not in any successor's set, and the recovery routine's run
|
||||
// at start is what picks those up.
|
||||
//
|
||||
// References are counted rather than held as a plain set because
|
||||
// ownership outlives the worker that took it. A target that schedules
|
||||
@@ -61,9 +61,9 @@ func (s *inflightSet) retain(deliveryID string) {
|
||||
|
||||
// retainIdle takes the first reference on a delivery, and reports
|
||||
// whether it got it. It fails when the engine already owns the
|
||||
// delivery, which is what makes two claimants — restart recovery and
|
||||
// the sweep run concurrently, or two sweep arms — mutually exclusive
|
||||
// rather than merely atomic.
|
||||
// delivery, which is what makes two claimants — notify and the
|
||||
// recovery routine, say — mutually exclusive rather than merely
|
||||
// atomic.
|
||||
func (s *inflightSet) retainIdle(deliveryID string) bool {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
@@ -2,8 +2,6 @@ package delivery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -66,33 +64,6 @@ func fDrain(e *delivery.Engine) []delivery.Task {
|
||||
}
|
||||
}
|
||||
|
||||
// TestArchiveHandleIsWAL closes the last gap in the durability
|
||||
// evidence: the main and per-webhook tiers each assert their journal
|
||||
// mode on a live handle, and the archive tier gets its settings from
|
||||
// the same code path but nothing checked the running file.
|
||||
func TestArchiveHandleIsWAL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
filepath.Join(t.TempDir(), "archive-wal.db"),
|
||||
archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Open(0))
|
||||
|
||||
var mode string
|
||||
|
||||
row := w.DB().Raw("pragma journal_mode").Row()
|
||||
require.NoError(t, row.Scan(&mode))
|
||||
assert.Equal(t, "wal", strings.ToLower(mode))
|
||||
|
||||
var busy string
|
||||
|
||||
row = w.DB().Raw("pragma busy_timeout").Row()
|
||||
require.NoError(t, row.Scan(&busy))
|
||||
assert.Equal(t, "10000", busy)
|
||||
}
|
||||
|
||||
// TestSweepLeavesAQueuedDeliveryAlone is the case the age bound cannot
|
||||
// see. The delivery is queued and untouched, so its row is arbitrarily
|
||||
// old and still perfectly healthy; only ownership distinguishes it
|
||||
@@ -113,9 +84,9 @@ func TestSweepLeavesAQueuedDeliveryAlone(t *testing.T) {
|
||||
)
|
||||
rAgePending(t, s.WebhookDB, d.ID)
|
||||
|
||||
// Queued exactly as the receiver queues it, and never dequeued:
|
||||
// Queued exactly as StoreEvent queues it, and never dequeued:
|
||||
// no workers are running in this engine.
|
||||
s.Engine.Notify([]delivery.Task{{
|
||||
s.Engine.ExportNotify([]delivery.Task{{
|
||||
DeliveryID: d.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: s.WebhookID,
|
||||
@@ -124,8 +95,8 @@ func TestSweepLeavesAQueuedDeliveryAlone(t *testing.T) {
|
||||
|
||||
require.Equal(t, 1, s.Engine.ExportInflightHeld())
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
tasks := fDrain(s.Engine)
|
||||
@@ -136,9 +107,11 @@ func TestSweepLeavesAQueuedDeliveryAlone(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestRecoveryAndSweepDoNotDoubleDispatch drives the two entry points
|
||||
// the engine starts concurrently against one aged pending row. Before
|
||||
// ownership they both dispatched it.
|
||||
// TestRecoveryAndSweepDoNotDoubleDispatch drives the recovery routine's
|
||||
// run at start and a tick of the sweep at once against one aged pending
|
||||
// row. The engine runs them one after the other, but notify can race
|
||||
// either, and ownership is what keeps any two claimants from both
|
||||
// sending. Before ownership the two runs both dispatched the row.
|
||||
func TestRecoveryAndSweepDoNotDoubleDispatch(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -162,12 +135,12 @@ func TestRecoveryAndSweepDoNotDoubleDispatch(t *testing.T) {
|
||||
|
||||
wg.Go(func() {
|
||||
s.Engine.ExportRecoverPendingDeliveries(
|
||||
ctx, s.WebhookDB, s.WebhookID,
|
||||
ctx, s.WebhookDB, s.WebhookID, atStart,
|
||||
)
|
||||
})
|
||||
wg.Go(func() {
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
ctx, s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
ctx, s.WebhookID, onTick,
|
||||
)
|
||||
})
|
||||
wg.Wait()
|
||||
@@ -251,7 +224,7 @@ func TestOwnershipIsReleasedAfterDelivery(t *testing.T) {
|
||||
|
||||
body := `{"released":true}`
|
||||
|
||||
s.Engine.Notify([]delivery.Task{{
|
||||
s.Engine.ExportNotify([]delivery.Task{{
|
||||
DeliveryID: d.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: s.WebhookID,
|
||||
@@ -275,10 +248,11 @@ func TestOwnershipIsReleasedAfterDelivery(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestNotifyAfterRecoveryDoesNotSendAgain is the startup race of
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/299. The receiver has
|
||||
// written a delivery, restart recovery finds it pending, sends it and
|
||||
// releases it, and only then does the receiver's Notify for it arrive.
|
||||
// Nothing owns the delivery by then, so Notify takes it.
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/299. StoreEvent has
|
||||
// written a delivery, the recovery routine's run at start finds it
|
||||
// pending, sends it and releases it, and only then does StoreEvent's
|
||||
// notify for it arrive. Nothing owns the delivery by then, so notify
|
||||
// takes it.
|
||||
func TestNotifyAfterRecoveryDoesNotSendAgain(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -302,7 +276,8 @@ func TestNotifyAfterRecoveryDoesNotSendAgain(t *testing.T) {
|
||||
)
|
||||
}()
|
||||
|
||||
// Restart recovery sends the delivery and lets it go.
|
||||
// The recovery routine's run at start sends the delivery and lets
|
||||
// it go.
|
||||
iWaitForDelivered(t, s.WebhookDB, d.ID)
|
||||
require.Eventually(
|
||||
t,
|
||||
@@ -314,7 +289,7 @@ func TestNotifyAfterRecoveryDoesNotSendAgain(t *testing.T) {
|
||||
|
||||
body := event.Body
|
||||
|
||||
s.Engine.Notify([]delivery.Task{{
|
||||
s.Engine.ExportNotify([]delivery.Task{{
|
||||
DeliveryID: d.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: s.WebhookID,
|
||||
@@ -325,7 +300,7 @@ func TestNotifyAfterRecoveryDoesNotSendAgain(t *testing.T) {
|
||||
EntrypointID: event.EntrypointID,
|
||||
}})
|
||||
|
||||
// Notify took the delivery, and a worker releases it once it has
|
||||
// notify took the delivery, and a worker releases it once it has
|
||||
// run the task.
|
||||
require.Eventually(
|
||||
t,
|
||||
@@ -363,7 +338,7 @@ func TestRetryingRecoverySkipsASuccessfulResult(t *testing.T) {
|
||||
rSeedResult(t, s.WebhookDB, d.ID, 2, true)
|
||||
|
||||
s.Engine.ExportRecoverRetryingDeliveries(
|
||||
s.WebhookDB, s.WebhookID,
|
||||
context.Background(), s.WebhookDB, s.WebhookID,
|
||||
)
|
||||
|
||||
assert.Empty(
|
||||
@@ -378,8 +353,8 @@ func TestRetryingRecoverySkipsASuccessfulResult(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestRetryingSweepSkipsASuccessfulResult is the same rule on the
|
||||
// periodic sweep's retrying arm.
|
||||
// TestRetryingSweepSkipsASuccessfulResult is the same rule on a tick of
|
||||
// the sweep.
|
||||
func TestRetryingSweepSkipsASuccessfulResult(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -397,8 +372,8 @@ func TestRetryingSweepSkipsASuccessfulResult(t *testing.T) {
|
||||
rSeedResult(t, s.WebhookDB, d.ID, 1, false)
|
||||
rSeedResult(t, s.WebhookDB, d.ID, 2, true)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
assert.Empty(t, fDrain(s.Engine))
|
||||
@@ -449,8 +424,8 @@ func TestScheduledRetryIsNotSweptDuringBackoff(t *testing.T) {
|
||||
|
||||
require.Equal(t, 1, s.Engine.ExportInflightHeld())
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
assert.Empty(
|
||||
@@ -482,7 +457,7 @@ func TestRedispatchStampsTheRow(t *testing.T) {
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
|
||||
s.Engine.ExportRecoverWebhookDeliveries(ctx, s.WebhookID, onTick)
|
||||
require.Len(t, fDrain(s.Engine), 1)
|
||||
|
||||
var row database.Delivery
|
||||
|
||||
@@ -192,7 +192,7 @@ func mDeliverOK(t *testing.T, s iSetup) {
|
||||
body := event.Body
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-ok", iHTTPConfig(ts.URL), 3, 1, &body,
|
||||
"metrics-ok", ts.URL, 3, 1, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
@@ -226,18 +226,17 @@ func mExhaustRetries(t *testing.T, s iSetup) {
|
||||
)
|
||||
|
||||
body := event.Body
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
|
||||
// The retry below is only run if its target still exists; see
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/107.
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "metrics-fail",
|
||||
database.TargetTypeHTTP, cfg, 2,
|
||||
database.TargetTypeHTTP, ts.URL, 2,
|
||||
)
|
||||
|
||||
first := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-fail", cfg, 2, 1, &body,
|
||||
"metrics-fail", ts.URL, 2, 1, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &first)
|
||||
@@ -251,7 +250,7 @@ func mExhaustRetries(t *testing.T, s iSetup) {
|
||||
// deterministic and off the wall clock.
|
||||
second := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-fail", cfg, 2, 2, &body,
|
||||
"metrics-fail", ts.URL, 2, 2, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(
|
||||
@@ -289,7 +288,6 @@ func TestDeliveryMetrics_CircuitBreakerGauge(t *testing.T) {
|
||||
)
|
||||
|
||||
body := event.Body
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
|
||||
// An attempt budget above the failure threshold, so the breaker
|
||||
// rather than the budget is what stops the delivery.
|
||||
@@ -299,12 +297,12 @@ func TestDeliveryMetrics_CircuitBreakerGauge(t *testing.T) {
|
||||
// see https://git.eeqj.de/sneak/webhooker/issues/107.
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "metrics-trip",
|
||||
database.TargetTypeHTTP, cfg, maxAttempts,
|
||||
database.TargetTypeHTTP, ts.URL, maxAttempts,
|
||||
)
|
||||
|
||||
first := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-trip", cfg, maxAttempts, 1, &body,
|
||||
"metrics-trip", ts.URL, maxAttempts, 1, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &first)
|
||||
@@ -316,7 +314,7 @@ func TestDeliveryMetrics_CircuitBreakerGauge(t *testing.T) {
|
||||
ExportDefaultFailureThreshold; attempt++ {
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-trip", cfg, maxAttempts, attempt, &body,
|
||||
"metrics-trip", ts.URL, maxAttempts, attempt, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(
|
||||
@@ -363,17 +361,16 @@ func TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt(
|
||||
)
|
||||
|
||||
body := event.Body
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
maxAttempts := delivery.ExportDefaultFailureThreshold + 5
|
||||
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "metrics-blocked",
|
||||
database.TargetTypeHTTP, cfg, maxAttempts,
|
||||
database.TargetTypeHTTP, ts.URL, maxAttempts,
|
||||
)
|
||||
|
||||
first := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-blocked", cfg, maxAttempts, 1, &body,
|
||||
"metrics-blocked", ts.URL, maxAttempts, 1, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &first)
|
||||
@@ -382,7 +379,7 @@ func TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt(
|
||||
ExportDefaultFailureThreshold; attempt++ {
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-blocked", cfg, maxAttempts, attempt, &body,
|
||||
"metrics-blocked", ts.URL, maxAttempts, attempt, &body,
|
||||
)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(context.TODO(), &task)
|
||||
@@ -405,7 +402,7 @@ func TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt(
|
||||
|
||||
blocked := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"metrics-blocked", cfg, maxAttempts,
|
||||
"metrics-blocked", ts.URL, maxAttempts,
|
||||
delivery.ExportDefaultFailureThreshold+1, &body,
|
||||
)
|
||||
|
||||
@@ -444,8 +441,8 @@ func TestDeliveryMetrics_OrphanedRetryFailureLabelled(
|
||||
t, s, database.TargetTypeLog,
|
||||
)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
iAssertStatus(t, s.WebhookDB, deliveryID,
|
||||
@@ -474,7 +471,7 @@ func TestDeliveryMetrics_QueueDepthGauges(t *testing.T) {
|
||||
|
||||
iCreateTarget(t, s.MainDB, targetID, s.WebhookID,
|
||||
"queue-depth-target", database.TargetTypeHTTP,
|
||||
iHTTPConfig("https://example.com/hook"), 3,
|
||||
"https://example.com/hook", 3,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
|
||||
@@ -20,7 +20,8 @@ const queueDepthSampleInterval = 30 * time.Second
|
||||
// to be seeded correctly at startup from rows written by a previous
|
||||
// process, and would drift permanently on any transition that failed
|
||||
// to persist. Counting is the measurement that cannot go wrong, and
|
||||
// it is the same whole-database walk the retry sweep already makes.
|
||||
// it is the same whole-database walk each tick of the sweep already
|
||||
// makes.
|
||||
func (e *Engine) queueDepthSampler(ctx context.Context) {
|
||||
defer e.wg.Done()
|
||||
|
||||
|
||||
@@ -102,7 +102,7 @@ func TestQueueDepthSample_LogsNoBoundValue(t *testing.T) {
|
||||
|
||||
iCreateTarget(t, mainDB.DB(), targetID, webhookID,
|
||||
"queue-depth-gormlog-target", database.TargetTypeHTTP,
|
||||
iHTTPConfig("https://example.com/hook"), 3,
|
||||
"https://example.com/hook", 3,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
|
||||
@@ -88,7 +88,7 @@ func TestRecoverySkipsPendingWithSuccessfulResult(
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverPendingDeliveries(
|
||||
context.Background(), s.WebhookDB, s.WebhookID,
|
||||
context.Background(), s.WebhookDB, s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
select {
|
||||
@@ -145,7 +145,7 @@ func TestRecoveryContinuesTheAttemptNumbering(t *testing.T) {
|
||||
rSeedResult(t, s.WebhookDB, d.ID, 2, false)
|
||||
|
||||
s.Engine.ExportRecoverPendingDeliveries(
|
||||
context.Background(), s.WebhookDB, s.WebhookID,
|
||||
context.Background(), s.WebhookDB, s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
select {
|
||||
@@ -183,8 +183,8 @@ func TestSweepRecoversStrandedPending(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
select {
|
||||
@@ -230,7 +230,7 @@ func TestSweepClaimsAStrandedDeliveryOnlyOnce(t *testing.T) {
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
|
||||
s.Engine.ExportRecoverWebhookDeliveries(ctx, s.WebhookID, onTick)
|
||||
|
||||
select {
|
||||
case task := <-s.Engine.ExportDeliveryCh():
|
||||
@@ -246,7 +246,7 @@ func TestSweepClaimsAStrandedDeliveryOnlyOnce(t *testing.T) {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
|
||||
s.Engine.ExportRecoverWebhookDeliveries(ctx, s.WebhookID, onTick)
|
||||
|
||||
select {
|
||||
case task := <-s.Engine.ExportDeliveryCh():
|
||||
@@ -258,9 +258,9 @@ func TestSweepClaimsAStrandedDeliveryOnlyOnce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSweepSettlesStrandedPendingWithoutResending is the sweep's own
|
||||
// version of the reconcile: a stranded delivery holding a successful
|
||||
// result is settled where it stands, and the receiver hears nothing.
|
||||
// TestSweepSettlesStrandedPendingWithoutResending is the reconcile on a
|
||||
// tick of the sweep: a stranded delivery holding a successful result is
|
||||
// settled where it stands, and the receiver hears nothing.
|
||||
func TestSweepSettlesStrandedPendingWithoutResending(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -280,8 +280,8 @@ func TestSweepSettlesStrandedPendingWithoutResending(
|
||||
rSeedResult(t, s.WebhookDB, d.ID, 1, true)
|
||||
rAgePending(t, s.WebhookDB, d.ID)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
select {
|
||||
@@ -312,8 +312,9 @@ func TestSweepSettlesStrandedPendingWithoutResending(
|
||||
|
||||
// TestFailedResultWriteLeavesDeliveryRecoverable is the rule the
|
||||
// targets now follow: a bookkeeping write that fails must not advance
|
||||
// the status, because pending and retrying are the states the sweeps
|
||||
// recover and delivered is a claim the database refused to record.
|
||||
// the status, because pending and retrying are the states the recovery
|
||||
// routine recovers and delivered is a claim the database refused to
|
||||
// record.
|
||||
func TestFailedResultWriteLeavesDeliveryRecoverable(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -356,7 +357,7 @@ func TestFailedResultWriteLeavesDeliveryRecoverable(
|
||||
Target: database.Target{
|
||||
Name: "unwritable",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: iHTTPConfig(ts.URL),
|
||||
URL: ts.URL,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
}
|
||||
@@ -447,7 +448,7 @@ func TestFailedResultWriteWithRetriesLeavesDeliveryRecoverable(
|
||||
Target: database.Target{
|
||||
Name: "unwritable",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: iHTTPConfig(ts.URL),
|
||||
URL: ts.URL,
|
||||
MaxAttempts: 3,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -0,0 +1,389 @@
|
||||
package delivery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// These tests cover the recovery routine, which the engine runs once
|
||||
// when it starts and then on every tick of the sweep
|
||||
// (https://git.eeqj.de/sneak/webhooker/issues/542). Every case runs it
|
||||
// both ways, each on a webhook of its own.
|
||||
|
||||
// atStart and onTick are the two values of the recovery routine's
|
||||
// atStart parameter: its run when the engine starts, and a tick of the
|
||||
// sweep.
|
||||
const (
|
||||
atStart = true
|
||||
onTick = false
|
||||
)
|
||||
|
||||
// recoveryRun names one of the two runs of the recovery routine.
|
||||
type recoveryRun struct {
|
||||
name string
|
||||
atStart bool
|
||||
}
|
||||
|
||||
// recoveryRuns returns the routine's run at start and a tick.
|
||||
func recoveryRuns() []recoveryRun {
|
||||
return []recoveryRun{
|
||||
{"at start", atStart},
|
||||
{"on a tick", onTick},
|
||||
}
|
||||
}
|
||||
|
||||
// recoverySetup gives a fresh webhook its main-database row and an http
|
||||
// target with retries, and returns the setup and the target's ID. No
|
||||
// worker runs, so whatever the routine queues stays in the channels for
|
||||
// the test to read, and nothing is sent to the target.
|
||||
func recoverySetup(t *testing.T) (iSetup, string) {
|
||||
t.Helper()
|
||||
|
||||
s := newISetup(t)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
iCreateWebhook(t, s.MainDB, s.WebhookID, "recovery")
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "recovery",
|
||||
database.TargetTypeHTTP,
|
||||
"http://example.com/hook", 20,
|
||||
)
|
||||
|
||||
return s, targetID
|
||||
}
|
||||
|
||||
// seedRecoveryDelivery seeds a delivery at status with the given
|
||||
// recorded attempts, failed but for the last when lastSucceeded is set,
|
||||
// all recorded now.
|
||||
func seedRecoveryDelivery(
|
||||
t *testing.T,
|
||||
s iSetup,
|
||||
targetID string,
|
||||
status database.DeliveryStatus,
|
||||
attempts int,
|
||||
lastSucceeded bool,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
event := iSeedEvent(t, s.WebhookDB, s.WebhookID, `{"recovery":true}`)
|
||||
d := iSeedDelivery(t, s.WebhookDB, event.ID, targetID, status)
|
||||
|
||||
for n := 1; n <= attempts; n++ {
|
||||
rSeedResult(
|
||||
t, s.WebhookDB, d.ID, n, lastSucceeded && n == attempts,
|
||||
)
|
||||
}
|
||||
|
||||
return d.ID
|
||||
}
|
||||
|
||||
// backdateAttempts moves every recorded attempt of a delivery an hour
|
||||
// into the past, long past the backoff after its first attempt.
|
||||
func backdateAttempts(t *testing.T, s iSetup, deliveryID string) {
|
||||
t.Helper()
|
||||
|
||||
require.NoError(t, s.WebhookDB.Model(&database.DeliveryResult{}).
|
||||
Where("delivery_id = ?", deliveryID).
|
||||
UpdateColumn("created_at", s.WebhookDB.NowFunc().Add(-time.Hour)).
|
||||
Error)
|
||||
}
|
||||
|
||||
// requireQueued waits for a task on ch. A retrying delivery reaches the
|
||||
// retry channel through a timer, even one due now, so it is waited for.
|
||||
func requireQueued(
|
||||
t *testing.T, ch <-chan delivery.Task,
|
||||
) delivery.Task {
|
||||
t.Helper()
|
||||
|
||||
select {
|
||||
case task := <-ch:
|
||||
return task
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("the recovery routine queued nothing")
|
||||
|
||||
return delivery.Task{}
|
||||
}
|
||||
}
|
||||
|
||||
// assertNothingQueued asserts the engine queues nothing, giving a retry
|
||||
// timer that is due now the moment it needs to fire.
|
||||
func assertNothingQueued(t *testing.T, e *delivery.Engine) {
|
||||
t.Helper()
|
||||
|
||||
select {
|
||||
case task := <-e.ExportDeliveryCh():
|
||||
t.Errorf("queued pending delivery %s", task.DeliveryID)
|
||||
case task := <-e.ExportRetryCh():
|
||||
t.Errorf("queued retrying delivery %s", task.DeliveryID)
|
||||
case <-time.After(200 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecovery_RetryingBackoffPassed: a retrying delivery whose backoff
|
||||
// is over is queued for its next attempt at once.
|
||||
func TestRecovery_RetryingBackoffPassed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, run := range recoveryRuns() {
|
||||
t.Run(run.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, targetID := recoverySetup(t)
|
||||
id := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusRetrying, 1, false,
|
||||
)
|
||||
backdateAttempts(t, s, id)
|
||||
|
||||
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
|
||||
|
||||
task := requireQueued(t, s.Engine.ExportRetryCh())
|
||||
assert.Equal(t, id, task.DeliveryID)
|
||||
assert.Equal(t, 2, task.AttemptNum)
|
||||
iAssertStatus(t, s.WebhookDB, id, database.DeliveryStatusRetrying)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecovery_RetryingBackoffNotPassed: a retrying delivery whose last
|
||||
// attempt was just now waits out the rest of its backoff, ten attempts
|
||||
// in, eight and a half minutes. It is not sent now, and the engine holds
|
||||
// it, so a later tick leaves it to the timer.
|
||||
func TestRecovery_RetryingBackoffNotPassed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, run := range recoveryRuns() {
|
||||
t.Run(run.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, targetID := recoverySetup(t)
|
||||
id := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusRetrying, 10, false,
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
|
||||
|
||||
assertNothingQueued(t, s.Engine)
|
||||
iAssertStatus(t, s.WebhookDB, id, database.DeliveryStatusRetrying)
|
||||
assert.Equal(t, 1, s.Engine.ExportInflightHeld(),
|
||||
"no retry timer holds the delivery",
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecovery_PendingOlderThanTheAgeBound: a pending delivery untouched
|
||||
// for longer than the sweep's age bound is sent by either run.
|
||||
func TestRecovery_PendingOlderThanTheAgeBound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, run := range recoveryRuns() {
|
||||
t.Run(run.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, targetID := recoverySetup(t)
|
||||
id := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusPending, 0, false,
|
||||
)
|
||||
rAgePending(t, s.WebhookDB, id)
|
||||
|
||||
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
|
||||
|
||||
task := requireQueued(t, s.Engine.ExportDeliveryCh())
|
||||
assert.Equal(t, id, task.DeliveryID)
|
||||
assert.Equal(t, 1, task.AttemptNum)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecovery_TaskCarriesEveryTargetSetting: the task the recovery
|
||||
// routine queues carries every one of the target's settings, all six
|
||||
// set here.
|
||||
// They are copied into the task one field at a time, and a field left
|
||||
// out is sent empty without any error.
|
||||
func TestRecovery_TaskCarriesEveryTargetSetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, run := range recoveryRuns() {
|
||||
t.Run(run.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
iCreateWebhook(t, s.MainDB, s.WebhookID, "recovery")
|
||||
|
||||
target := database.Target{
|
||||
WebhookID: s.WebhookID,
|
||||
Name: "recovery",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Active: true,
|
||||
URL: "http://example.com/hook",
|
||||
Headers: map[string]string{headerAuthorization: "Bearer t"},
|
||||
TimeoutSeconds: 7,
|
||||
ForwardQuery: true,
|
||||
ArchiveExpiry: time.Hour,
|
||||
ArchiveRotation: "daily",
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
require.NoError(t, s.MainDB.Create(&target).Error)
|
||||
|
||||
id := seedRecoveryDelivery(
|
||||
t, s, target.ID, database.DeliveryStatusPending, 0, false,
|
||||
)
|
||||
rAgePending(t, s.WebhookDB, id)
|
||||
|
||||
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
|
||||
|
||||
task := requireQueued(t, s.Engine.ExportDeliveryCh())
|
||||
assert.Equal(t, target.ID, task.TargetID)
|
||||
assert.Equal(t, target.URL, task.TargetURL)
|
||||
assert.Equal(t, target.Headers, task.TargetHeaders)
|
||||
assert.Equal(t, target.TimeoutSeconds, task.TimeoutSeconds)
|
||||
assert.Equal(t, target.ForwardQuery, task.ForwardQuery)
|
||||
assert.Equal(t, target.ArchiveExpiry, task.ArchiveExpiry)
|
||||
assert.Equal(t, target.ArchiveRotation, task.ArchiveRotation)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecovery_PendingYoungerThanTheAgeBound: a pending delivery touched
|
||||
// a moment ago, as one a stop cut off is, is sent at start, which is how
|
||||
// it reaches its target on the next start
|
||||
// (https://git.eeqj.de/sneak/webhooker/issues/527). A tick leaves it.
|
||||
func TestRecovery_PendingYoungerThanTheAgeBound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, run := range recoveryRuns() {
|
||||
t.Run(run.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, targetID := recoverySetup(t)
|
||||
id := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusPending, 0, false,
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
|
||||
|
||||
if run.atStart {
|
||||
task := requireQueued(t, s.Engine.ExportDeliveryCh())
|
||||
assert.Equal(t, id, task.DeliveryID)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
assertNothingQueued(t, s.Engine)
|
||||
iAssertStatus(t, s.WebhookDB, id, database.DeliveryStatusPending)
|
||||
assert.Zero(t, s.Engine.ExportInflightHeld())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecovery_AlreadySucceeded: a pending or retrying delivery whose
|
||||
// success is recorded but whose status write was lost is marked
|
||||
// delivered where it stands, and not sent again.
|
||||
func TestRecovery_AlreadySucceeded(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, run := range recoveryRuns() {
|
||||
t.Run(run.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, targetID := recoverySetup(t)
|
||||
pending := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusPending, 1, true,
|
||||
)
|
||||
rAgePending(t, s.WebhookDB, pending)
|
||||
|
||||
retrying := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusRetrying, 2, true,
|
||||
)
|
||||
backdateAttempts(t, s, retrying)
|
||||
|
||||
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
|
||||
|
||||
assertNothingQueued(t, s.Engine)
|
||||
|
||||
for id, attempts := range map[string]int{pending: 1, retrying: 2} {
|
||||
iAssertStatus(
|
||||
t, s.WebhookDB, id, database.DeliveryStatusDelivered,
|
||||
)
|
||||
assert.Len(t, iResults(t, s.WebhookDB, id), attempts,
|
||||
"marking the delivery delivered recorded an attempt",
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecovery_MissingDatabase: a tick leaves a webhook whose database
|
||||
// file is missing alone, and the run at start opens it, which creates an
|
||||
// empty one and reports the loss (see
|
||||
// TestRecoverDeliveries_ReportsAMissingWebhookDatabase).
|
||||
func TestRecovery_MissingDatabase(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mainDB := iMainDB(t)
|
||||
webhookID := uuid.New().String()
|
||||
iCreateWebhook(t, mainDB.DB(), webhookID, "missing-database")
|
||||
|
||||
dbMgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||
t.Cleanup(func() { _ = dbMgr.CloseAll() })
|
||||
|
||||
engine := delivery.NewTestEngineWithDB(
|
||||
mainDB, dbMgr,
|
||||
slog.New(slog.DiscardHandler),
|
||||
&http.Client{Timeout: 5 * time.Second}, 1,
|
||||
)
|
||||
|
||||
engine.ExportRecoverDeliveries(context.Background(), onTick)
|
||||
assert.False(t, dbMgr.DBExists(webhookID),
|
||||
"a tick created the missing database",
|
||||
)
|
||||
|
||||
engine.ExportRecoverDeliveries(context.Background(), atStart)
|
||||
assert.True(t, dbMgr.DBExists(webhookID),
|
||||
"the run at start did not open the missing database",
|
||||
)
|
||||
}
|
||||
|
||||
// TestRecovery_LeavesADeliveryTheEngineHolds: a delivery the engine is
|
||||
// working on, has queued, or holds a retry timer for is left alone by
|
||||
// either run, pending and retrying alike, however old.
|
||||
func TestRecovery_LeavesADeliveryTheEngineHolds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, run := range recoveryRuns() {
|
||||
t.Run(run.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, targetID := recoverySetup(t)
|
||||
pending := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusPending, 0, false,
|
||||
)
|
||||
rAgePending(t, s.WebhookDB, pending)
|
||||
|
||||
retrying := seedRecoveryDelivery(
|
||||
t, s, targetID, database.DeliveryStatusRetrying, 1, false,
|
||||
)
|
||||
backdateAttempts(t, s, retrying)
|
||||
|
||||
require.True(t, s.Engine.ExportRetainDelivery(pending))
|
||||
require.True(t, s.Engine.ExportRetainDelivery(retrying))
|
||||
|
||||
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
|
||||
|
||||
assertNothingQueued(t, s.Engine)
|
||||
iAssertStatus(t, s.WebhookDB, pending, database.DeliveryStatusPending)
|
||||
iAssertStatus(t, s.WebhookDB, retrying, database.DeliveryStatusRetrying)
|
||||
assert.Equal(t, 2, s.Engine.ExportInflightHeld())
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -85,19 +85,14 @@ func deliverWithProbeHeaders(
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
cfg, err := json.Marshal(delivery.HTTPTargetConfig{
|
||||
URL: targetURL,
|
||||
Headers: map[string]string{
|
||||
probeHeaderName: probeHeaderValue,
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
body := event.Body
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, targetID,
|
||||
"redirect-target", string(cfg), 1, 1, &body,
|
||||
"redirect-target", targetURL, 1, 1, &body,
|
||||
)
|
||||
task.TargetHeaders = map[string]string{
|
||||
probeHeaderName: probeHeaderValue,
|
||||
}
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
|
||||
@@ -309,8 +304,7 @@ func TestRedirectPolicy_StopsAtHopCap(t *testing.T) {
|
||||
)
|
||||
|
||||
client := engine.ExportClientForRequest(
|
||||
&delivery.HTTPTargetConfig{URL: srv.URL},
|
||||
[]string{probeHeaderName},
|
||||
0, []string{probeHeaderName},
|
||||
)
|
||||
require.NotNil(t, client.CheckRedirect)
|
||||
|
||||
@@ -350,7 +344,7 @@ func TestApplyRequestHeaders_ReportsOriginScopedNames(t *testing.T) {
|
||||
inbound, err := json.Marshal(map[string][]string{
|
||||
inboundHeaderName: {inboundHeaderValue},
|
||||
"Content-Type": {testContentType},
|
||||
"User-Agent": {"curl/8.7.1"},
|
||||
headerUserAgent: {"curl/8.7.1"},
|
||||
"Host": {"inbound.example.com"},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
@@ -369,11 +363,9 @@ func TestApplyRequestHeaders_ReportsOriginScopedNames(t *testing.T) {
|
||||
Headers: string(inbound),
|
||||
ContentType: testContentType,
|
||||
},
|
||||
&delivery.HTTPTargetConfig{
|
||||
Headers: map[string]string{
|
||||
probeHeaderName: probeHeaderValue,
|
||||
"Content-Type": testContentType,
|
||||
},
|
||||
map[string]string{
|
||||
probeHeaderName: probeHeaderValue,
|
||||
"Content-Type": testContentType,
|
||||
},
|
||||
"webhooker/dev",
|
||||
)
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// StoreEvent stores event with one pending delivery for each of its
|
||||
// webhook's active targets, and adds them to the totals, in one
|
||||
// transaction, then queues the deliveries. It returns how many
|
||||
// deliveries it created.
|
||||
//
|
||||
// The receiver stores every event it accepts through here, and a
|
||||
// resubmit stores its copy of a stored event, so every event is
|
||||
// stored, retried, SSRF-guarded and circuit-broken the same way however
|
||||
// it entered the system. CreateReplay is the only other place a
|
||||
// delivery is created.
|
||||
//
|
||||
// The targets are the webhook's active ones as they stand now: a
|
||||
// target created after a resubmitted event first arrived is included,
|
||||
// and an inactive one is skipped. An event with no active target is
|
||||
// stored with no deliveries.
|
||||
func (e *Engine) StoreEvent(event *database.Event) (int, error) {
|
||||
var targets []database.Target
|
||||
|
||||
err := e.database.DB().Where(
|
||||
"webhook_id = ? AND active = ?", event.WebhookID, true,
|
||||
).Find(&targets).Error
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("loading active targets: %w", err)
|
||||
}
|
||||
|
||||
webhookDB, err := e.dbManager.GetDB(event.WebhookID)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("getting webhook database: %w", err)
|
||||
}
|
||||
|
||||
event.BodyBytes = int64(len(event.Body))
|
||||
tasks := make([]Task, 0, len(targets))
|
||||
|
||||
err = webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||
err := tx.Create(event).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating event: %w", err)
|
||||
}
|
||||
|
||||
for i := range targets {
|
||||
dlv, err := createDelivery(tx, event.ID, targets[i].ID, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tasks = append(tasks, buildTask(
|
||||
dlv, event.WebhookID, event, &targets[i], 1,
|
||||
))
|
||||
}
|
||||
|
||||
err = database.AddEventTotals(tx, database.EventTotals{
|
||||
Events: 1, LastEventAt: &event.CreatedAt,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// A resubmitted copy did not arrive on its entrypoint's URL, so
|
||||
// it leaves the entrypoint's last event as it is.
|
||||
if event.ResubmittedFromID != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return database.AddEntrypointTotals(tx, database.EntrypointTotals{
|
||||
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
|
||||
})
|
||||
})
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
// Counted here, after the commit: an event exists once it is
|
||||
// durably stored, which is what the delivery counters are compared
|
||||
// against on a dashboard. A resubmitted event counts too, because
|
||||
// it produces deliveries that the delivery side counts; the
|
||||
// resubmit counter is what separates the two.
|
||||
e.mtr.EventReceived()
|
||||
|
||||
e.notify(tasks)
|
||||
|
||||
return len(tasks), nil
|
||||
}
|
||||
|
||||
// CreateReplay creates one more pending delivery of the stored event
|
||||
// eventID to target, marked as a replay, adds it to the target's totals
|
||||
// in the same transaction, and queues it. It returns the new
|
||||
// delivery's id.
|
||||
//
|
||||
// The caller passes the target as it stands now and has already
|
||||
// decided the replay may go ahead. The delivery being replayed is not
|
||||
// touched.
|
||||
func (e *Engine) CreateReplay(
|
||||
webhookID, eventID string, target *database.Target,
|
||||
) (string, error) {
|
||||
webhookDB, err := e.dbManager.GetDB(webhookID)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("getting webhook database: %w", err)
|
||||
}
|
||||
|
||||
event, err := e.loadEvent(webhookDB, eventID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
var dlv *database.Delivery
|
||||
|
||||
err = webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||
var err error
|
||||
|
||||
dlv, err = createDelivery(tx, event.ID, target.ID, true)
|
||||
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
e.notify([]Task{buildTask(dlv, webhookID, &event, target, 1)})
|
||||
|
||||
return dlv.ID, nil
|
||||
}
|
||||
|
||||
// createDelivery writes a pending delivery of the event to the target
|
||||
// on the caller's transaction, and adds it to the target's totals.
|
||||
//
|
||||
// Neither Event nor Target is set on the row, and the per-webhook
|
||||
// database omits associations on every write in any case (see
|
||||
// internal/database/event_db_isolation.go), so the target's settings
|
||||
// are never copied into the event database.
|
||||
func createDelivery(
|
||||
tx *gorm.DB, eventID, targetID string, replay bool,
|
||||
) (*database.Delivery, error) {
|
||||
dlv := &database.Delivery{
|
||||
EventID: eventID,
|
||||
TargetID: targetID,
|
||||
Status: database.DeliveryStatusPending,
|
||||
Replay: replay,
|
||||
}
|
||||
|
||||
err := tx.Create(dlv).Error
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"creating delivery for target %s: %w", targetID, err,
|
||||
)
|
||||
}
|
||||
|
||||
err = database.AddTargetTotals(tx, database.TargetTotals{
|
||||
TargetID: targetID, Deliveries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return dlv, nil
|
||||
}
|
||||
@@ -0,0 +1,386 @@
|
||||
package delivery_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// storeTestTarget creates an active http target of the setup's webhook
|
||||
// with every one of a target's settings set, so a task that drops one
|
||||
// of them differs from the target.
|
||||
func storeTestTarget(
|
||||
t *testing.T, s iSetup, name string,
|
||||
) database.Target {
|
||||
t.Helper()
|
||||
|
||||
target := database.Target{
|
||||
WebhookID: s.WebhookID,
|
||||
Name: name,
|
||||
Type: database.TargetTypeHTTP,
|
||||
Active: true,
|
||||
URL: "http://93.184.216.34/" + name,
|
||||
Headers: map[string]string{"X-Api-Key": "k"},
|
||||
TimeoutSeconds: 7,
|
||||
ForwardQuery: true,
|
||||
ArchiveExpiry: time.Hour,
|
||||
ArchiveRotation: rotationDaily,
|
||||
MaxAttempts: 3,
|
||||
}
|
||||
|
||||
require.NoError(t, s.MainDB.Create(&target).Error)
|
||||
|
||||
return target
|
||||
}
|
||||
|
||||
// newStoreTestEvent returns an event of the setup's webhook as the
|
||||
// receiver builds one, not yet stored.
|
||||
func newStoreTestEvent(s iSetup, body string) *database.Event {
|
||||
return &database.Event{
|
||||
WebhookID: s.WebhookID,
|
||||
EntrypointID: uuid.New().String(),
|
||||
Method: http.MethodPost,
|
||||
RawQuery: "a=1&b=2",
|
||||
Headers: `{"X-Test":["yes"]}`,
|
||||
ContentType: testContentType,
|
||||
Body: body,
|
||||
}
|
||||
}
|
||||
|
||||
// queuedTasks takes every task the engine has queued so far. The
|
||||
// engine in these tests is never started, so nothing else takes them.
|
||||
func queuedTasks(e *delivery.Engine) []delivery.Task {
|
||||
var tasks []delivery.Task
|
||||
|
||||
for {
|
||||
select {
|
||||
case task := <-e.ExportDeliveryCh():
|
||||
tasks = append(tasks, task)
|
||||
default:
|
||||
return tasks
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// assertTaskFor proves task carries delivery deliveryID of event to
|
||||
// target: every one of the target's settings and every event field.
|
||||
// They are copied into the task one field at a time, and a field left
|
||||
// out is sent empty without any error.
|
||||
func assertTaskFor(
|
||||
t *testing.T,
|
||||
task delivery.Task,
|
||||
deliveryID string,
|
||||
event *database.Event,
|
||||
target *database.Target,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
assert.Equal(t, deliveryID, task.DeliveryID)
|
||||
assert.Equal(t, event.ID, task.EventID)
|
||||
assert.Equal(t, event.WebhookID, task.WebhookID)
|
||||
assert.Equal(t, event.EntrypointID, task.EntrypointID)
|
||||
|
||||
assert.Equal(t, target.ID, task.TargetID)
|
||||
assert.Equal(t, target.Name, task.TargetName)
|
||||
assert.Equal(t, target.Type, task.TargetType)
|
||||
assert.Equal(t, target.URL, task.TargetURL)
|
||||
assert.Equal(t, target.Headers, task.TargetHeaders)
|
||||
assert.Equal(t, target.TimeoutSeconds, task.TimeoutSeconds)
|
||||
assert.Equal(t, target.ForwardQuery, task.ForwardQuery)
|
||||
assert.Equal(t, target.ArchiveExpiry, task.ArchiveExpiry)
|
||||
assert.Equal(t, target.ArchiveRotation, task.ArchiveRotation)
|
||||
assert.Equal(t, target.MaxAttempts, task.MaxAttempts)
|
||||
|
||||
assert.Equal(t, event.Method, task.Method)
|
||||
assert.Equal(t, event.RawQuery, task.RawQuery)
|
||||
assert.Equal(t, event.Headers, task.Headers)
|
||||
assert.Equal(t, event.ContentType, task.ContentType)
|
||||
require.NotNil(t, task.Body)
|
||||
assert.Equal(t, event.Body, *task.Body)
|
||||
|
||||
assert.Equal(t, 1, task.AttemptNum)
|
||||
}
|
||||
|
||||
// storedDeliveries reads every delivery of an event.
|
||||
func storedDeliveries(
|
||||
t *testing.T, db *gorm.DB, eventID string,
|
||||
) []database.Delivery {
|
||||
t.Helper()
|
||||
|
||||
var deliveries []database.Delivery
|
||||
|
||||
require.NoError(t, db.Where("event_id = ?", eventID).
|
||||
Find(&deliveries).Error)
|
||||
|
||||
return deliveries
|
||||
}
|
||||
|
||||
// storedEventTotals reads the webhook's event totals.
|
||||
func storedEventTotals(t *testing.T, db *gorm.DB) database.EventTotals {
|
||||
t.Helper()
|
||||
|
||||
var totals database.EventTotals
|
||||
|
||||
require.NoError(t, db.First(&totals).Error)
|
||||
|
||||
return totals
|
||||
}
|
||||
|
||||
// countRows counts the rows of model in db.
|
||||
func countRows(t *testing.T, db *gorm.DB, model any) int64 {
|
||||
t.Helper()
|
||||
|
||||
var n int64
|
||||
|
||||
require.NoError(t, db.Model(model).Count(&n).Error)
|
||||
|
||||
return n
|
||||
}
|
||||
|
||||
// TestStoreEvent_StoresDeliveriesToActiveTargets proves StoreEvent
|
||||
// stores the event with one pending delivery to each active target and
|
||||
// none to an inactive one, adds them to the totals, and queues one task
|
||||
// per delivery.
|
||||
func TestStoreEvent_StoresDeliveriesToActiveTargets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
targets := []database.Target{
|
||||
storeTestTarget(t, s, "first"),
|
||||
storeTestTarget(t, s, "second"),
|
||||
}
|
||||
off := storeTestTarget(t, s, "off")
|
||||
|
||||
require.NoError(t, s.MainDB.Model(&database.Target{}).
|
||||
Where("id = ?", off.ID).Update("active", false).Error)
|
||||
|
||||
event := newStoreTestEvent(s, `{"hello":"world"}`)
|
||||
|
||||
created, err := s.Engine.StoreEvent(event)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, created)
|
||||
|
||||
var stored database.Event
|
||||
|
||||
require.NoError(t, s.WebhookDB.First(&stored, "id = ?", event.ID).Error)
|
||||
assert.Equal(t, event.Body, stored.Body)
|
||||
assert.Equal(t, int64(len(event.Body)), stored.BodyBytes)
|
||||
|
||||
rows := storedDeliveries(t, s.WebhookDB, event.ID)
|
||||
queued := queuedTasks(s.Engine)
|
||||
|
||||
require.Len(t, rows, 2)
|
||||
require.Len(t, queued, 2)
|
||||
|
||||
deliveries := map[string]database.Delivery{}
|
||||
for _, dlv := range rows {
|
||||
deliveries[dlv.TargetID] = dlv
|
||||
}
|
||||
|
||||
tasks := map[string]delivery.Task{}
|
||||
for _, task := range queued {
|
||||
tasks[task.TargetID] = task
|
||||
}
|
||||
|
||||
for i := range targets {
|
||||
dlv, ok := deliveries[targets[i].ID]
|
||||
require.True(t, ok, "no delivery to %s", targets[i].Name)
|
||||
assert.Equal(t, database.DeliveryStatusPending, dlv.Status)
|
||||
assert.False(t, dlv.Replay)
|
||||
assert.Equal(t, int64(1),
|
||||
targetTotals(t, s.WebhookDB, targets[i].ID).Deliveries)
|
||||
|
||||
assertTaskFor(t, tasks[targets[i].ID], dlv.ID, event, &targets[i])
|
||||
}
|
||||
|
||||
totals := storedEventTotals(t, s.WebhookDB)
|
||||
assert.Equal(t, int64(1), totals.Events)
|
||||
require.NotNil(t, totals.LastEventAt)
|
||||
assert.True(t, totals.LastEventAt.Equal(stored.CreatedAt))
|
||||
|
||||
var entrypoint database.EntrypointTotals
|
||||
|
||||
require.NoError(t, s.WebhookDB.First(
|
||||
&entrypoint, "entrypoint_id = ?", event.EntrypointID,
|
||||
).Error)
|
||||
assert.True(t, entrypoint.LastEventAt.Equal(stored.CreatedAt))
|
||||
}
|
||||
|
||||
// TestStoreEvent_ResubmittedCopyLeavesTheEntrypoint proves a
|
||||
// resubmitted copy is counted among the webhook's events but leaves its
|
||||
// entrypoint's last event as it is: it did not arrive on that URL.
|
||||
func TestStoreEvent_ResubmittedCopyLeavesTheEntrypoint(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
storeTestTarget(t, s, "only")
|
||||
|
||||
sourceID := uuid.New().String()
|
||||
event := newStoreTestEvent(s, `{"copy":true}`)
|
||||
event.ResubmittedFromID = &sourceID
|
||||
|
||||
created, err := s.Engine.StoreEvent(event)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, created)
|
||||
assert.Len(t, queuedTasks(s.Engine), 1)
|
||||
|
||||
assert.Equal(t, int64(1), storedEventTotals(t, s.WebhookDB).Events)
|
||||
assert.Zero(t, countRows(t, s.WebhookDB, &database.EntrypointTotals{}))
|
||||
}
|
||||
|
||||
// TestStoreEvent_NoActiveTarget proves an event of a webhook with no
|
||||
// active target is stored with no delivery and nothing queued.
|
||||
func TestStoreEvent_NoActiveTarget(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
event := newStoreTestEvent(s, `{"to":"nobody"}`)
|
||||
|
||||
created, err := s.Engine.StoreEvent(event)
|
||||
require.NoError(t, err)
|
||||
assert.Zero(t, created)
|
||||
|
||||
assert.Equal(t, int64(1), countRows(t, s.WebhookDB, &database.Event{}))
|
||||
assert.Empty(t, storedDeliveries(t, s.WebhookDB, event.ID))
|
||||
assert.Empty(t, queuedTasks(s.Engine))
|
||||
}
|
||||
|
||||
// TestStoreEvent_LargeBodyIsNotCarried proves a body at the inline
|
||||
// limit is stored whole but left off the task, for the attempt to read
|
||||
// from the event row.
|
||||
func TestStoreEvent_LargeBodyIsNotCarried(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
storeTestTarget(t, s, "only")
|
||||
|
||||
event := newStoreTestEvent(
|
||||
s, strings.Repeat("x", delivery.MaxInlineBodySize),
|
||||
)
|
||||
|
||||
_, err := s.Engine.StoreEvent(event)
|
||||
require.NoError(t, err)
|
||||
|
||||
tasks := queuedTasks(s.Engine)
|
||||
require.Len(t, tasks, 1)
|
||||
assert.Nil(t, tasks[0].Body)
|
||||
|
||||
var stored database.Event
|
||||
|
||||
require.NoError(t, s.WebhookDB.First(&stored, "id = ?", event.ID).Error)
|
||||
assert.Equal(t, event.Body, stored.Body)
|
||||
}
|
||||
|
||||
// TestStoreEvent_RollsBackWhenAWriteFails proves the event, its
|
||||
// deliveries and its totals are one transaction: when the entrypoint
|
||||
// totals write fails, the last one, after the event, its delivery, the
|
||||
// target totals and the event totals were written, none of them is left
|
||||
// and nothing is queued.
|
||||
func TestStoreEvent_RollsBackWhenAWriteFails(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
storeTestTarget(t, s, "only")
|
||||
|
||||
require.NoError(t,
|
||||
s.WebhookDB.Migrator().DropTable(&database.EntrypointTotals{}))
|
||||
|
||||
_, err := s.Engine.StoreEvent(newStoreTestEvent(s, `{"lost":true}`))
|
||||
require.Error(t, err)
|
||||
|
||||
assert.Zero(t, countRows(t, s.WebhookDB, &database.Event{}))
|
||||
assert.Zero(t, countRows(t, s.WebhookDB, &database.Delivery{}))
|
||||
assert.Zero(t, countRows(t, s.WebhookDB, &database.TargetTotals{}))
|
||||
|
||||
totals := storedEventTotals(t, s.WebhookDB)
|
||||
assert.Zero(t, totals.Events)
|
||||
assert.Nil(t, totals.LastEventAt)
|
||||
|
||||
assert.Empty(t, queuedTasks(s.Engine))
|
||||
assert.Zero(t, s.Engine.ExportInflightHeld())
|
||||
}
|
||||
|
||||
// TestCreateReplay_CreatesAndQueuesAReplay proves CreateReplay adds a
|
||||
// new pending delivery marked as a replay, counts it in the target's
|
||||
// totals, queues its task built from the target it was given and the
|
||||
// stored event, and leaves the delivery it repeats as it was. The
|
||||
// stored event has every event field set, so a task that drops one of
|
||||
// them differs from the event.
|
||||
func TestCreateReplay_CreatesAndQueuesAReplay(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
target := storeTestTarget(t, s, "replayed")
|
||||
event := newStoreTestEvent(s, `{"replay":"me"}`)
|
||||
require.NoError(t, s.WebhookDB.Create(event).Error)
|
||||
|
||||
original := iSeedDelivery(
|
||||
t, s.WebhookDB, event.ID, target.ID,
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
deliveryID, err := s.Engine.CreateReplay(s.WebhookID, event.ID, &target)
|
||||
require.NoError(t, err)
|
||||
|
||||
replay := storedDelivery(t, s.WebhookDB, deliveryID)
|
||||
assert.Equal(t, event.ID, replay.EventID)
|
||||
assert.Equal(t, target.ID, replay.TargetID)
|
||||
assert.Equal(t, database.DeliveryStatusPending, replay.Status)
|
||||
assert.True(t, replay.Replay)
|
||||
|
||||
assert.Equal(t, database.DeliveryStatusFailed,
|
||||
storedDelivery(t, s.WebhookDB, original.ID).Status)
|
||||
assert.Equal(t, int64(1),
|
||||
targetTotals(t, s.WebhookDB, target.ID).Deliveries)
|
||||
|
||||
tasks := queuedTasks(s.Engine)
|
||||
require.Len(t, tasks, 1)
|
||||
assertTaskFor(t, tasks[0], deliveryID, event, &target)
|
||||
}
|
||||
|
||||
// TestCreateReplay_RollsBackWhenAWriteFails proves the replay delivery
|
||||
// and its target totals are one transaction: when the totals write
|
||||
// fails, after the delivery was written, no new delivery is left and
|
||||
// nothing is queued.
|
||||
func TestCreateReplay_RollsBackWhenAWriteFails(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
target := storeTestTarget(t, s, "replayed")
|
||||
event := iSeedEvent(t, s.WebhookDB, s.WebhookID, `{"replay":"me"}`)
|
||||
iSeedDelivery(
|
||||
t, s.WebhookDB, event.ID, target.ID,
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
require.NoError(t,
|
||||
s.WebhookDB.Migrator().DropTable(&database.TargetTotals{}))
|
||||
|
||||
_, err := s.Engine.CreateReplay(s.WebhookID, event.ID, &target)
|
||||
require.Error(t, err)
|
||||
|
||||
assert.Len(t, storedDeliveries(t, s.WebhookDB, event.ID), 1)
|
||||
assert.Empty(t, queuedTasks(s.Engine))
|
||||
assert.Zero(t, s.Engine.ExportInflightHeld())
|
||||
}
|
||||
|
||||
// storedDelivery reads one delivery back.
|
||||
func storedDelivery(
|
||||
t *testing.T, db *gorm.DB, deliveryID string,
|
||||
) database.Delivery {
|
||||
t.Helper()
|
||||
|
||||
var dlv database.Delivery
|
||||
|
||||
require.NoError(t, db.First(&dlv, "id = ?", deliveryID).Error)
|
||||
|
||||
return dlv
|
||||
}
|
||||
+10
-22
@@ -13,8 +13,8 @@ import (
|
||||
// The engine provides one to each target so a target can own
|
||||
// its retries durably: it records the attempt, marks the
|
||||
// delivery retrying, and asks the Scheduler to deliver the
|
||||
// next attempt after delay — exactly what the engine does for
|
||||
// its own restart recovery.
|
||||
// next attempt after delay — exactly what the engine's
|
||||
// recovery routine does for a retrying delivery.
|
||||
type Scheduler interface {
|
||||
ScheduleRetry(task Task, delay time.Duration)
|
||||
}
|
||||
@@ -45,29 +45,20 @@ type Target interface {
|
||||
}
|
||||
|
||||
// rescheduler is implemented by targets that own durable
|
||||
// retries. The engine's restart recovery and periodic sweep
|
||||
// use it to let the target recompute the schedule for an
|
||||
// orphaned retrying delivery, keeping the retry schedule
|
||||
// target-owned. Fire-and-forget targets do not implement it
|
||||
// and their (never-occurring) retrying deliveries are
|
||||
// skipped.
|
||||
// retries. The engine's recovery routine uses it to let the
|
||||
// target recompute the schedule for an orphaned retrying
|
||||
// delivery, keeping the retry schedule target-owned.
|
||||
// Fire-and-forget targets do not implement it, and a retrying
|
||||
// delivery found for one is terminally failed.
|
||||
type rescheduler interface {
|
||||
// remainingBackoff returns how long to wait before the
|
||||
// next attempt of a recovered retrying delivery.
|
||||
// next attempt of a recovered retrying delivery: zero once
|
||||
// its backoff has passed.
|
||||
remainingBackoff(
|
||||
webhookDB *gorm.DB,
|
||||
deliveryID string,
|
||||
attemptNum int,
|
||||
) time.Duration
|
||||
|
||||
// backoffElapsed reports whether the backoff window for
|
||||
// the last attempt has already passed, so the periodic
|
||||
// sweep can re-enqueue the delivery now.
|
||||
backoffElapsed(
|
||||
webhookDB *gorm.DB,
|
||||
deliveryID string,
|
||||
attemptNum int,
|
||||
) bool
|
||||
}
|
||||
|
||||
// attemptResult is the outcome of a single delivery attempt,
|
||||
@@ -103,16 +94,13 @@ func (e *Engine) initTargets(client *http.Client) {
|
||||
client: client,
|
||||
}
|
||||
|
||||
dbT := &databaseTarget{eng: e}
|
||||
|
||||
e.httpTarget = httpT
|
||||
e.slackTarget = slackT
|
||||
e.dbTarget = dbT
|
||||
|
||||
e.targets = map[database.TargetType]Target{
|
||||
database.TargetTypeHTTP: httpT,
|
||||
database.TargetTypeSlack: slackT,
|
||||
database.TargetTypeDatabase: dbT,
|
||||
database.TargetTypeDatabase: &databaseTarget{eng: e},
|
||||
database.TargetTypeLog: &logTarget{eng: e},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,136 +0,0 @@
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// errUnknownTargetTypeForEdit is returned when a stored target has a
|
||||
// type the edit form has no field set for.
|
||||
var errUnknownTargetTypeForEdit = errors.New(
|
||||
"unknown target type",
|
||||
)
|
||||
|
||||
// TargetConfigForm is the UNMASKED projection of a target's stored
|
||||
// configuration, for pre-filling the target edit form.
|
||||
//
|
||||
// It is the deliberate exception to the rule TargetView enforces
|
||||
// everywhere else: TargetView exists so that no template can render
|
||||
// a target's stored blob, because a destination URL's path segments
|
||||
// and a header value are both routinely the credential. An operator
|
||||
// cannot correct a value they cannot see, so the edit form — and
|
||||
// only the edit form — is shown the full value.
|
||||
//
|
||||
// Everything that keeps that exception narrow lives at the call
|
||||
// site: the route is behind RequireAuth and the webhook's ownership
|
||||
// check, and its group sets NoCache so the rendered secret is not
|
||||
// written to a shared cache. Do not reach for this type from any
|
||||
// other page.
|
||||
type TargetConfigForm struct {
|
||||
// URL is the destination for an HTTP target and the webhook
|
||||
// URL for a Slack target.
|
||||
URL string
|
||||
// Headers is the HTTP target's configured headers in the
|
||||
// textarea representation, one "Name: value" per line.
|
||||
Headers string
|
||||
// Timeout is the HTTP target's per-request timeout in seconds,
|
||||
// empty when unset.
|
||||
Timeout string
|
||||
// ForwardQuery is the HTTP target's setting that passes each
|
||||
// event's query string on to it.
|
||||
ForwardQuery bool
|
||||
// Expiry is the database (archive) target's row expiry.
|
||||
Expiry string
|
||||
// Rotation is the database (archive) target's rotation.
|
||||
Rotation string
|
||||
}
|
||||
|
||||
// NewTargetConfigForm parses a target's stored configuration into
|
||||
// the edit form's fields.
|
||||
//
|
||||
// A configuration that does not parse is an error rather than a
|
||||
// zero-valued form that silently looks like a target with no
|
||||
// settings. The caller shows the operator that the stored value
|
||||
// could not be read, so that saving the form is understood as
|
||||
// replacing it rather than preserving it.
|
||||
func NewTargetConfigForm(
|
||||
t *database.Target,
|
||||
) (TargetConfigForm, error) {
|
||||
switch t.Type {
|
||||
case database.TargetTypeHTTP:
|
||||
cfg, err := parseHTTPConfig(t.Config)
|
||||
if err != nil {
|
||||
return TargetConfigForm{}, err
|
||||
}
|
||||
|
||||
return TargetConfigForm{
|
||||
URL: cfg.URL,
|
||||
Headers: FormatTargetHeaders(cfg.Headers),
|
||||
Timeout: FormatTargetTimeout(cfg.Timeout),
|
||||
ForwardQuery: cfg.ForwardQuery,
|
||||
}, nil
|
||||
case database.TargetTypeSlack:
|
||||
cfg, err := parseSlackConfig(t.Config)
|
||||
if err != nil {
|
||||
return TargetConfigForm{}, err
|
||||
}
|
||||
|
||||
return TargetConfigForm{URL: cfg.WebhookURL}, nil
|
||||
case database.TargetTypeDatabase:
|
||||
return databaseConfigForm(t.Config)
|
||||
case database.TargetTypeLog:
|
||||
// The log target takes no configuration.
|
||||
return TargetConfigForm{}, nil
|
||||
default:
|
||||
return TargetConfigForm{}, fmt.Errorf(
|
||||
"%w: %q", errUnknownTargetTypeForEdit, t.Type,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// databaseConfigForm parses an archive target's optional expiry and
|
||||
// rotation. An absent, empty or never expiry yields an empty expiry,
|
||||
// on which the edit form starts at never; saving it unchanged stores
|
||||
// never, which means the same as an empty expiry. An absent rotation
|
||||
// is empty too, and the form starts at none. An expiry that is set
|
||||
// but not a valid duration, or a rotation that is not one of the
|
||||
// four, is an error, not a blank field.
|
||||
func databaseConfigForm(
|
||||
configJSON string,
|
||||
) (TargetConfigForm, error) {
|
||||
if configJSON == "" {
|
||||
return TargetConfigForm{}, nil
|
||||
}
|
||||
|
||||
var cfg databaseTargetConfig
|
||||
|
||||
err := json.Unmarshal([]byte(configJSON), &cfg)
|
||||
if err != nil {
|
||||
return TargetConfigForm{}, fmt.Errorf(
|
||||
"parsing config JSON: %w", err,
|
||||
)
|
||||
}
|
||||
|
||||
err = ValidateArchiveRotation(cfg.Rotation)
|
||||
if err != nil {
|
||||
return TargetConfigForm{}, err
|
||||
}
|
||||
|
||||
form := TargetConfigForm{Rotation: cfg.Rotation}
|
||||
|
||||
if cfg.Expiry == "" || cfg.Expiry == archiveExpiryNever {
|
||||
return form, nil
|
||||
}
|
||||
|
||||
err = ValidateArchiveExpiry(cfg.Expiry)
|
||||
if err != nil {
|
||||
return TargetConfigForm{}, err
|
||||
}
|
||||
|
||||
form.Expiry = cfg.Expiry
|
||||
|
||||
return form, nil
|
||||
}
|
||||
@@ -5,19 +5,21 @@ import (
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// configUnavailable is what a target's configuration renders
|
||||
// as when it is absent, of an unknown type, or does not
|
||||
// parse. The stored blob is never shown as a fallback: it can
|
||||
// hold a credential (a Slack incoming webhook URL is a bearer
|
||||
// token) and a UI that prints it leaks that credential into
|
||||
// browser history, screenshots and screen shares.
|
||||
// configUnavailable is what a target's settings render as when
|
||||
// the target is of an unknown type, and what its URL renders as
|
||||
// when it does not parse into a scheme and host. The stored URL is
|
||||
// never shown as a fallback: it can be a credential (a Slack
|
||||
// incoming webhook URL is a bearer token) and a UI that prints it
|
||||
// leaks that credential into browser history, screenshots and
|
||||
// screen shares.
|
||||
const configUnavailable = "(unavailable)"
|
||||
|
||||
// ConfigField is one labelled, display-safe value derived
|
||||
// from a target's stored configuration.
|
||||
// from a target's settings.
|
||||
type ConfigField struct {
|
||||
Label string
|
||||
Value string
|
||||
@@ -31,9 +33,9 @@ type ConfigField struct {
|
||||
const deletedNameSuffix = " (deleted)"
|
||||
|
||||
// TargetView is the display-safe projection of a target for
|
||||
// the UI. It deliberately has no raw configuration field, so
|
||||
// no template — present or future — can render the stored
|
||||
// blob.
|
||||
// the UI. It deliberately has no URL or header field, so no
|
||||
// template — present or future — can render the stored
|
||||
// credential.
|
||||
type TargetView struct {
|
||||
ID string
|
||||
Name string
|
||||
@@ -60,8 +62,7 @@ func (v TargetView) DisplayName() string {
|
||||
}
|
||||
|
||||
// NewTargetViews projects targets for rendering, replacing
|
||||
// each stored configuration blob with named, display-safe
|
||||
// fields.
|
||||
// each target's settings with named, display-safe fields.
|
||||
//
|
||||
// A soft-deleted row projects exactly as a live one does, minus
|
||||
// the deleted marker on its name: masking is a property of the
|
||||
@@ -90,8 +91,8 @@ func NewTargetViews(
|
||||
}
|
||||
|
||||
// targetConfigFields returns the display-safe fields for a
|
||||
// target's configuration. Anything it cannot parse becomes
|
||||
// the neutral placeholder.
|
||||
// target's settings. A target of an unknown type gets the
|
||||
// neutral placeholder.
|
||||
func targetConfigFields(
|
||||
t *database.Target,
|
||||
) []ConfigField {
|
||||
@@ -101,36 +102,25 @@ func targetConfigFields(
|
||||
case database.TargetTypeHTTP:
|
||||
return httpConfigFields(t)
|
||||
case database.TargetTypeDatabase:
|
||||
return databaseConfigFields(t.Config)
|
||||
return databaseConfigFields(t)
|
||||
case database.TargetTypeLog:
|
||||
// The log target takes no configuration.
|
||||
// The log target takes no settings.
|
||||
return nil
|
||||
default:
|
||||
return unavailableConfigFields()
|
||||
return []ConfigField{{
|
||||
Label: "Configuration",
|
||||
Value: configUnavailable,
|
||||
}}
|
||||
}
|
||||
}
|
||||
|
||||
// unavailableConfigFields is the neutral placeholder shown
|
||||
// for a configuration that could not be presented.
|
||||
func unavailableConfigFields() []ConfigField {
|
||||
return []ConfigField{{
|
||||
Label: "Configuration",
|
||||
Value: configUnavailable,
|
||||
}}
|
||||
}
|
||||
|
||||
// slackConfigFields describes a Slack target: its masked
|
||||
// webhook URL and its number of delivery attempts. Only the
|
||||
// masked URL is shown; the full URL is the credential.
|
||||
func slackConfigFields(t *database.Target) []ConfigField {
|
||||
cfg, err := parseSlackConfig(t.Config)
|
||||
if err != nil {
|
||||
return unavailableConfigFields()
|
||||
}
|
||||
|
||||
return []ConfigField{{
|
||||
Label: "Webhook URL",
|
||||
Value: cfg.MaskedWebhookURL(),
|
||||
Value: MaskURL(t.URL),
|
||||
}, maxAttemptsField(t)}
|
||||
}
|
||||
|
||||
@@ -145,33 +135,28 @@ func slackConfigFields(t *database.Target) []ConfigField {
|
||||
// whose path segments are the credential, and the field takes
|
||||
// an arbitrary URL, so no segment can be assumed non-secret.
|
||||
func httpConfigFields(t *database.Target) []ConfigField {
|
||||
cfg, err := parseHTTPConfig(t.Config)
|
||||
if err != nil {
|
||||
return unavailableConfigFields()
|
||||
}
|
||||
|
||||
fields := []ConfigField{{
|
||||
Label: "Destination URL",
|
||||
Value: MaskURL(cfg.URL),
|
||||
Value: MaskURL(t.URL),
|
||||
}}
|
||||
|
||||
if cfg.Timeout > 0 {
|
||||
if t.TimeoutSeconds > 0 {
|
||||
fields = append(fields, ConfigField{
|
||||
Label: "Timeout",
|
||||
Value: strconv.Itoa(cfg.Timeout) + "s",
|
||||
Value: strconv.Itoa(t.TimeoutSeconds) + "s",
|
||||
})
|
||||
}
|
||||
|
||||
if len(cfg.Headers) > 0 {
|
||||
if len(t.Headers) > 0 {
|
||||
fields = append(fields, ConfigField{
|
||||
Label: "Headers",
|
||||
Value: fmt.Sprintf(
|
||||
"%d configured", len(cfg.Headers),
|
||||
"%d configured", len(t.Headers),
|
||||
),
|
||||
})
|
||||
}
|
||||
|
||||
if cfg.ForwardQuery {
|
||||
if t.ForwardQuery {
|
||||
fields = append(fields, ConfigField{
|
||||
Label: "Query string",
|
||||
Value: "passed on to this target",
|
||||
@@ -184,8 +169,7 @@ func httpConfigFields(t *database.Target) []ConfigField {
|
||||
}
|
||||
|
||||
// maxAttemptsField describes a target's number of delivery
|
||||
// attempts, which lives on the target row rather than in its
|
||||
// configuration blob.
|
||||
// attempts.
|
||||
func maxAttemptsField(t *database.Target) ConfigField {
|
||||
return ConfigField{
|
||||
Label: "Delivery attempts",
|
||||
@@ -195,24 +179,11 @@ func maxAttemptsField(t *database.Target) ConfigField {
|
||||
|
||||
// databaseConfigFields describes an archive target by its
|
||||
// expiry in plain units, such as "30 days", or "never" when
|
||||
// the archive is kept forever, and by its rotation. An expiry
|
||||
// that is set but not a valid duration, or a rotation that is
|
||||
// not one of the four, is reported as unavailable rather than
|
||||
// echoed back.
|
||||
func databaseConfigFields(configJSON string) []ConfigField {
|
||||
expiry, err := parseArchiveExpiry(configJSON)
|
||||
if err != nil {
|
||||
return unavailableConfigFields()
|
||||
}
|
||||
|
||||
rotation, err := parseArchiveRotation(configJSON)
|
||||
if err != nil {
|
||||
return unavailableConfigFields()
|
||||
}
|
||||
|
||||
value := archiveExpiryNever
|
||||
if expiry > 0 {
|
||||
value = plainDuration(expiry)
|
||||
// the archive is kept forever, and by its rotation.
|
||||
func databaseConfigFields(t *database.Target) []ConfigField {
|
||||
value := archive.ExpiryNever
|
||||
if t.ArchiveExpiry > 0 {
|
||||
value = plainDuration(t.ArchiveExpiry)
|
||||
}
|
||||
|
||||
return []ConfigField{{
|
||||
@@ -220,7 +191,7 @@ func databaseConfigFields(configJSON string) []ConfigField {
|
||||
Value: value,
|
||||
}, {
|
||||
Label: "Archive rotation",
|
||||
Value: rotation,
|
||||
Value: t.ArchiveRotation,
|
||||
}}
|
||||
}
|
||||
|
||||
@@ -256,14 +227,3 @@ func plainDuration(d time.Duration) string {
|
||||
|
||||
return d.String()
|
||||
}
|
||||
|
||||
// MaskedWebhookURL returns the Slack webhook URL reduced to
|
||||
// its scheme and host, with the path, query and any userinfo
|
||||
// elided. The path segments are the credential, so none of
|
||||
// them is shown: the field accepts an arbitrary URL, so no
|
||||
// segment can be assumed non-secret. A URL that does not
|
||||
// parse into a scheme and host yields the neutral
|
||||
// placeholder, never the raw string.
|
||||
func (c *SlackTargetConfig) MaskedWebhookURL() string {
|
||||
return MaskURL(c.WebhookURL)
|
||||
}
|
||||
|
||||
@@ -35,7 +35,17 @@ const (
|
||||
viewMaxAttempts = "Delivery attempts"
|
||||
)
|
||||
|
||||
func TestMaskedWebhookURL(t *testing.T) {
|
||||
// The archive rotations a database target's settings can name.
|
||||
const (
|
||||
rotationNone = "none"
|
||||
rotationMonthly = "monthly"
|
||||
rotationDaily = "daily"
|
||||
rotationHourly = "hourly"
|
||||
)
|
||||
|
||||
// TestNewTargetViews_SlackMasksWebhookURL shows what a Slack target's
|
||||
// URL renders as on the target list.
|
||||
func TestNewTargetViews_SlackMasksWebhookURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := map[string]struct {
|
||||
@@ -79,28 +89,24 @@ func TestMaskedWebhookURL(t *testing.T) {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := &delivery.SlackTargetConfig{
|
||||
WebhookURL: tc.url,
|
||||
}
|
||||
target := slackTarget()
|
||||
target.URL = tc.url
|
||||
|
||||
assert.Equal(
|
||||
t, tc.want, cfg.MaskedWebhookURL(),
|
||||
t, tc.want,
|
||||
fieldMap(viewFor(t, target).Config)["Webhook URL"],
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestMaskedWebhookURL_NeverLeaksPath is the direct
|
||||
// TestNewTargetViews_SlackNeverLeaksPath is the direct
|
||||
// expression of the rule: whatever the input, the masked
|
||||
// value never contains a path segment of it.
|
||||
func TestMaskedWebhookURL_NeverLeaksPath(t *testing.T) {
|
||||
func TestNewTargetViews_SlackNeverLeaksPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := &delivery.SlackTargetConfig{
|
||||
WebhookURL: slackWebhookURL,
|
||||
}
|
||||
|
||||
masked := cfg.MaskedWebhookURL()
|
||||
masked := fieldMap(viewFor(t, slackTarget()).Config)["Webhook URL"]
|
||||
|
||||
assert.NotContains(t, masked, "T00000000")
|
||||
assert.NotContains(t, masked, "B00000000")
|
||||
@@ -166,11 +172,10 @@ func TestNewTargetViews_DeletedTarget(t *testing.T) {
|
||||
// share.
|
||||
func slackTarget() database.Target {
|
||||
return database.Target{
|
||||
Name: slackTargetName,
|
||||
Type: database.TargetTypeSlack,
|
||||
Active: true,
|
||||
Config: `{"webhookUrl":"` +
|
||||
slackWebhookURL + `"}`,
|
||||
Name: slackTargetName,
|
||||
Type: database.TargetTypeSlack,
|
||||
Active: true,
|
||||
URL: slackWebhookURL,
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
}
|
||||
@@ -222,12 +227,12 @@ func TestNewTargetViews_HTTP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"` + viewExampleHook + `",` +
|
||||
`"timeout":30,` +
|
||||
`"headers":{"Authorization":"Bearer sekrit"},` +
|
||||
`"forwardQuery":true}`,
|
||||
MaxAttempts: 5,
|
||||
Type: database.TargetTypeHTTP,
|
||||
URL: viewExampleHook,
|
||||
TimeoutSeconds: 30,
|
||||
Headers: map[string]string{headerAuthorization: "Bearer sekrit"},
|
||||
ForwardQuery: true,
|
||||
MaxAttempts: 5,
|
||||
})
|
||||
|
||||
fields := fieldMap(view.Config)
|
||||
@@ -257,7 +262,7 @@ func TestNewTargetViews_HTTPSingleAttempt(t *testing.T) {
|
||||
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"` + viewExampleHook + `"}`,
|
||||
URL: viewExampleHook,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -280,7 +285,7 @@ func TestNewTargetViews_HTTPMasksDestinationURL(t *testing.T) {
|
||||
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"` + slackWebhookURL + `"}`,
|
||||
URL: slackWebhookURL,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -304,25 +309,20 @@ func TestNewTargetViews_Database(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := map[string]struct {
|
||||
config string
|
||||
expiry time.Duration
|
||||
want string
|
||||
}{
|
||||
"empty config": {config: "", want: viewExpiryNever},
|
||||
"empty expiry": {config: `{}`, want: viewExpiryNever},
|
||||
"never literal": {
|
||||
config: `{"expiry":"` + viewExpiryNever + `"}`,
|
||||
want: viewExpiryNever,
|
||||
},
|
||||
"1h": {config: `{"expiry":"1h"}`, want: "1 hour"},
|
||||
"12h": {config: `{"expiry":"12h"}`, want: "12 hours"},
|
||||
"24h": {config: `{"expiry":"24h"}`, want: "1 day"},
|
||||
"720h": {config: `{"expiry":"720h"}`, want: "30 days"},
|
||||
"2160h": {config: `{"expiry":"2160h"}`, want: "90 days"},
|
||||
"8760h": {config: `{"expiry":"8760h"}`, want: "365 days"},
|
||||
"36h": {config: `{"expiry":"36h"}`, want: "36 hours"},
|
||||
"1h30m": {config: `{"expiry":"1h30m"}`, want: "90 minutes"},
|
||||
"45s": {config: `{"expiry":"45s"}`, want: "45 seconds"},
|
||||
"1.5s": {config: `{"expiry":"1.5s"}`, want: "1.5s"},
|
||||
"never": {expiry: 0, want: viewExpiryNever},
|
||||
"1h": {expiry: time.Hour, want: "1 hour"},
|
||||
"12h": {expiry: 12 * time.Hour, want: "12 hours"},
|
||||
"24h": {expiry: 24 * time.Hour, want: "1 day"},
|
||||
"720h": {expiry: 720 * time.Hour, want: "30 days"},
|
||||
"2160h": {expiry: 2160 * time.Hour, want: "90 days"},
|
||||
"8760h": {expiry: 8760 * time.Hour, want: "365 days"},
|
||||
"36h": {expiry: 36 * time.Hour, want: "36 hours"},
|
||||
"1h30m": {expiry: 90 * time.Minute, want: "90 minutes"},
|
||||
"45s": {expiry: 45 * time.Second, want: "45 seconds"},
|
||||
"1.5s": {expiry: 1500 * time.Millisecond, want: "1.5s"},
|
||||
}
|
||||
|
||||
for name, tc := range tests {
|
||||
@@ -330,8 +330,9 @@ func TestNewTargetViews_Database(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: tc.config,
|
||||
Type: database.TargetTypeDatabase,
|
||||
ArchiveExpiry: tc.expiry,
|
||||
ArchiveRotation: rotationNone,
|
||||
})
|
||||
|
||||
assert.Equal(
|
||||
@@ -347,33 +348,23 @@ func TestNewTargetViews_Database(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestNewTargetViews_DatabaseRotation proves the target list shows a
|
||||
// database target's rotation, none when it has none stored.
|
||||
// database target's rotation.
|
||||
func TestNewTargetViews_DatabaseRotation(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Each stored config, and the rotation the list shows for it.
|
||||
tests := map[string]string{
|
||||
"": rotationNone,
|
||||
`{"rotation":""}`: rotationNone,
|
||||
}
|
||||
|
||||
for _, rotation := range []string{
|
||||
rotationNone, rotationMonthly, rotationDaily, rotationHourly,
|
||||
} {
|
||||
tests[`{"rotation":"`+rotation+`"}`] = rotation
|
||||
}
|
||||
|
||||
for config, want := range tests {
|
||||
t.Run(config, func(t *testing.T) {
|
||||
t.Run(rotation, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: config,
|
||||
Type: database.TargetTypeDatabase,
|
||||
ArchiveRotation: rotation,
|
||||
})
|
||||
|
||||
assert.Equal(
|
||||
t, want, fieldMap(view.Config)["Archive rotation"],
|
||||
t, rotation, fieldMap(view.Config)["Archive rotation"],
|
||||
)
|
||||
})
|
||||
}
|
||||
@@ -383,72 +374,28 @@ func TestNewTargetViews_Log(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetTypeLog,
|
||||
Config: "",
|
||||
Type: database.TargetTypeLog,
|
||||
})
|
||||
|
||||
assert.Empty(t, view.Config)
|
||||
}
|
||||
|
||||
// TestNewTargetViews_Unpresentable proves that no config the
|
||||
// view cannot present falls back to the stored blob.
|
||||
func TestNewTargetViews_Unpresentable(t *testing.T) {
|
||||
// TestNewTargetViews_UnknownType proves that a target of a type the
|
||||
// view does not know shows the neutral placeholder, and never its
|
||||
// URL.
|
||||
func TestNewTargetViews_UnknownType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const blob = `{"webhookUrl":"https://hooks.slack.com` +
|
||||
slackSecretPath + `"`
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetType("carrier-pigeon"),
|
||||
URL: slackWebhookURL,
|
||||
})
|
||||
|
||||
tests := map[string]database.Target{
|
||||
"unknown target type": {
|
||||
Type: database.TargetType("carrier-pigeon"),
|
||||
Config: blob,
|
||||
assert.Equal(
|
||||
t,
|
||||
map[string]string{
|
||||
"Configuration": viewUnavailable,
|
||||
},
|
||||
"unparseable json": {
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: blob,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
"empty slack config": {
|
||||
Type: database.TargetTypeSlack,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
"slack config without url": {
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{}`,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
"unparseable http json": {
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":`,
|
||||
MaxAttempts: 1,
|
||||
},
|
||||
"unparseable archive json": {
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: `{"expiry":`,
|
||||
},
|
||||
"invalid archive expiry": {
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: `{"expiry":"a fortnight"}`,
|
||||
},
|
||||
"invalid archive rotation": {
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: weeklyConfig,
|
||||
},
|
||||
}
|
||||
|
||||
for name, target := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
view := viewFor(t, target)
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
map[string]string{
|
||||
"Configuration": viewUnavailable,
|
||||
},
|
||||
fieldMap(view.Config),
|
||||
)
|
||||
})
|
||||
}
|
||||
fieldMap(view.Config),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -2,83 +2,23 @@ package delivery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// archiveNameMaxLen is how many characters of a webhook or target
|
||||
// name an archive file name keeps.
|
||||
const archiveNameMaxLen = 40
|
||||
|
||||
// databaseTarget is a no-retry target that archives the full
|
||||
// inbound event into the target's own archive SQLite file, separate
|
||||
// from the per-webhook event database. The event is already
|
||||
// persisted in the per-webhook event DB by the time delivery runs;
|
||||
// the database target additionally writes a durable long-term copy
|
||||
// into the file ArchiveFileName names, with a period added when the
|
||||
// target rotates (see archivePeriodPath), and then records a single
|
||||
// through the engine's archive.Writers, and then records a single
|
||||
// attempt whose outcome reflects whether the archive write
|
||||
// succeeded. See archiveWriter for the close/reopen, auto-recreate,
|
||||
// and expiry semantics.
|
||||
// succeeded. Package archive holds everything about the files
|
||||
// themselves.
|
||||
type databaseTarget struct {
|
||||
eng *Engine
|
||||
|
||||
// writers holds one archive writer per database target, keyed
|
||||
// by target ID.
|
||||
mu sync.Mutex
|
||||
writers map[string]*archiveWriter
|
||||
}
|
||||
|
||||
// ArchiveFileName returns the file name of a database target's
|
||||
// archive: archive-WEBHOOKNAME-TARGETNAME-TARGETID.db, with both
|
||||
// names passed through archiveNamePart. The target ID keeps the
|
||||
// name unique when two targets' names come out the same.
|
||||
func ArchiveFileName(webhookName, targetName, targetID string) string {
|
||||
return "archive-" + archiveNamePart(webhookName) + "-" +
|
||||
archiveNamePart(targetName) + "-" + targetID + ".db"
|
||||
}
|
||||
|
||||
// archiveNamePart makes a webhook or target name safe to put in a
|
||||
// file name. It is lowercased; ASCII letters and digits are kept,
|
||||
// every other run of characters becomes a single "-", and no "-" is
|
||||
// left at either end. It is cut to archiveNameMaxLen characters, and
|
||||
// a name with nothing left is "unnamed".
|
||||
func archiveNamePart(name string) string {
|
||||
var b strings.Builder
|
||||
|
||||
dash := false
|
||||
|
||||
for _, r := range strings.ToLower(name) {
|
||||
if (r < 'a' || r > 'z') && (r < '0' || r > '9') {
|
||||
dash = b.Len() > 0
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if dash {
|
||||
b.WriteByte('-')
|
||||
|
||||
dash = false
|
||||
}
|
||||
|
||||
b.WriteRune(r)
|
||||
}
|
||||
|
||||
part := b.String()
|
||||
if len(part) > archiveNameMaxLen {
|
||||
part = strings.TrimRight(part[:archiveNameMaxLen], "-")
|
||||
}
|
||||
|
||||
if part == "" {
|
||||
return "unnamed"
|
||||
}
|
||||
|
||||
return part
|
||||
}
|
||||
|
||||
// Deliver implements Target. It archives the event, then
|
||||
@@ -98,7 +38,7 @@ func (t *databaseTarget) Deliver(
|
||||
) {
|
||||
start := time.Now()
|
||||
|
||||
err := t.archive(d)
|
||||
err := t.eng.archives.Write(d)
|
||||
|
||||
elapsed := time.Since(start)
|
||||
|
||||
@@ -145,308 +85,3 @@ func (t *databaseTarget) Deliver(
|
||||
database.DeliveryStatusDelivered,
|
||||
)
|
||||
}
|
||||
|
||||
// archive writes the full event as a row into the target's
|
||||
// archive database, honouring the optional per-target expiry and
|
||||
// rotation parsed from the target config JSON. With rotation, the
|
||||
// event goes to the file for the period of its receive time.
|
||||
func (t *databaseTarget) archive(d *database.Delivery) error {
|
||||
webhookID := d.Event.WebhookID
|
||||
if webhookID == "" {
|
||||
return errArchiveMissingWebhookID
|
||||
}
|
||||
|
||||
expiry, err := parseArchiveExpiry(d.Target.Config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
rotation, err := parseArchiveRotation(d.Target.Config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// An event whose stored row was gone before its delivery ran has
|
||||
// no receive time (see Engine.hydrateEvent), and goes to the
|
||||
// file for now.
|
||||
receivedAt := d.Event.CreatedAt
|
||||
if receivedAt.IsZero() {
|
||||
receivedAt = time.Now()
|
||||
}
|
||||
|
||||
w, err := t.writerFor(d.TargetID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
row := archivedEvent{
|
||||
EventID: d.Event.ID,
|
||||
WebhookID: webhookID,
|
||||
EntrypointID: d.Event.EntrypointID,
|
||||
Method: d.Event.Method,
|
||||
RawQuery: d.Event.RawQuery,
|
||||
Headers: d.Event.Headers,
|
||||
Body: d.Event.Body,
|
||||
ContentType: d.Event.ContentType,
|
||||
}
|
||||
|
||||
return w.write(row, expiry, archivePeriod(rotation, receivedAt))
|
||||
}
|
||||
|
||||
// writerFor returns the archive writer for a database target,
|
||||
// creating and caching it on first use. Each target has one writer
|
||||
// so its close/reopen debounce state is shared across concurrent
|
||||
// deliveries, and so a rename and the idle sweep take the same lock
|
||||
// as its writes.
|
||||
func (t *databaseTarget) writerFor(
|
||||
targetID string,
|
||||
) (*archiveWriter, error) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
|
||||
w, ok := t.writers[targetID]
|
||||
if !ok {
|
||||
var err error
|
||||
|
||||
w, err = t.newWriter(targetID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if t.writers == nil {
|
||||
t.writers = make(map[string]*archiveWriter)
|
||||
}
|
||||
|
||||
t.writers[targetID] = w
|
||||
}
|
||||
|
||||
// A delivery claims the entry: even if the idle sweep created
|
||||
// it moments ago, it now belongs to the registry proper and
|
||||
// the sweep must leave it in place when it finishes.
|
||||
w.sweepOwned = false
|
||||
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// sweepWriterFor returns the archive writer the idle sweep should
|
||||
// prune a target's archive through, together with whether the sweep
|
||||
// itself created the registry entry.
|
||||
//
|
||||
// The sweep must route its prune through the registered writer so
|
||||
// the writer's mutex orders it against concurrent writes, but it
|
||||
// must never leave a registry entry behind: a sweep that ran
|
||||
// concurrently with the target's deletion would otherwise
|
||||
// re-create an entry that nothing will ever evict again, which is
|
||||
// exactly the leak eviction exists to prevent. An entry the sweep
|
||||
// creates is therefore marked sweep-owned and handed back to
|
||||
// releaseSweepWriter when the sweep is done.
|
||||
func (t *databaseTarget) sweepWriterFor(
|
||||
targetID string,
|
||||
) (*archiveWriter, bool, error) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
|
||||
w, ok := t.writers[targetID]
|
||||
if ok {
|
||||
return w, false, nil
|
||||
}
|
||||
|
||||
w, err := t.newWriter(targetID)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
|
||||
if t.writers == nil {
|
||||
t.writers = make(map[string]*archiveWriter)
|
||||
}
|
||||
|
||||
w.sweepOwned = true
|
||||
t.writers[targetID] = w
|
||||
|
||||
return w, true, nil
|
||||
}
|
||||
|
||||
// releaseSweepWriter drops a registry entry that the idle sweep
|
||||
// created, so a sweep leaves the registry exactly as it found it.
|
||||
//
|
||||
// The entry is removed only if it is still the very writer the
|
||||
// sweep installed and no delivery has claimed it in the meantime
|
||||
// (writerFor clears sweepOwned when it hands a writer to the
|
||||
// write path). Both conditions are evaluated under the registry
|
||||
// lock, so an eviction that raced the sweep — which removes the
|
||||
// entry outright — simply finds nothing left to do here, and a
|
||||
// delivery that adopted the writer keeps a registered, evictable
|
||||
// one.
|
||||
func (t *databaseTarget) releaseSweepWriter(
|
||||
targetID string, w *archiveWriter,
|
||||
) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
|
||||
cur, ok := t.writers[targetID]
|
||||
if !ok || cur != w || !cur.sweepOwned {
|
||||
return
|
||||
}
|
||||
|
||||
delete(t.writers, targetID)
|
||||
}
|
||||
|
||||
// newWriter builds the writer for a database target's archive. Its
|
||||
// path is the one ArchivePath gives for the webhook and the target as
|
||||
// the main database names them now; from then on only rename changes
|
||||
// the name the writer uses. It does not touch the archive files.
|
||||
func (t *databaseTarget) newWriter(
|
||||
targetID string,
|
||||
) (*archiveWriter, error) {
|
||||
if t.eng.dbManager == nil {
|
||||
return nil, errArchiveNoDataDir
|
||||
}
|
||||
|
||||
var target database.Target
|
||||
|
||||
err := t.eng.database.DB().
|
||||
Preload("Webhook").
|
||||
First(&target, "id = ?", targetID).Error
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"loading database target %s: %w", targetID, err,
|
||||
)
|
||||
}
|
||||
|
||||
w := newArchiveWriter(
|
||||
ArchivePath(t.eng.dbManager, &target.Webhook, &target),
|
||||
t.eng.log,
|
||||
)
|
||||
w.webhookID = target.WebhookID
|
||||
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// rename moves every one of a database target's archive files to the
|
||||
// name for webhookName and targetName. It goes through the target's
|
||||
// writer, so the move holds the lock that writes and the idle sweep
|
||||
// take, and later writes use the new name.
|
||||
//
|
||||
// The writer is created if there is none, and it stays cached. The
|
||||
// handlers rename before they save the new name, so until the save
|
||||
// the main database still has the old one; a delivery in that window
|
||||
// must find this writer rather than build one from the old name.
|
||||
func (t *databaseTarget) rename(
|
||||
targetID, webhookName, targetName string,
|
||||
) error {
|
||||
w, err := t.writerFor(targetID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return w.rename(ArchiveFileName(webhookName, targetName, targetID))
|
||||
}
|
||||
|
||||
// evict drops a database target's archive writer from the registry
|
||||
// and closes its handle, so a deleted target does not leave a
|
||||
// writer (and an open archive handle within its debounce window)
|
||||
// alive for the process lifetime.
|
||||
//
|
||||
// The map entry is removed under the registry lock, which is
|
||||
// then released before the handle is closed under the writer's
|
||||
// own lock: that ordering keeps the registry available to other
|
||||
// targets while an in-flight write on this one drains, and
|
||||
// closing under the writer's lock means eviction can never race
|
||||
// a write.
|
||||
//
|
||||
// Eviction is idempotent and silent for a target with no writer,
|
||||
// which is the common case: only a database target that has
|
||||
// received an event or been renamed has one. It never deletes the
|
||||
// archive file.
|
||||
func (t *databaseTarget) evict(targetID string) {
|
||||
t.mu.Lock()
|
||||
|
||||
w, ok := t.writers[targetID]
|
||||
if ok {
|
||||
delete(t.writers, targetID)
|
||||
}
|
||||
|
||||
t.mu.Unlock()
|
||||
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
w.evict()
|
||||
|
||||
t.eng.log.Info(
|
||||
"evicted archive writer",
|
||||
"target_id", targetID,
|
||||
"path", w.path,
|
||||
)
|
||||
}
|
||||
|
||||
// evictWebhook evicts, exactly as evict does, the writer of every
|
||||
// database target of a webhook.
|
||||
func (t *databaseTarget) evictWebhook(webhookID string) {
|
||||
t.mu.Lock()
|
||||
|
||||
var gone []*archiveWriter
|
||||
|
||||
for targetID, w := range t.writers {
|
||||
if w.webhookID == webhookID {
|
||||
delete(t.writers, targetID)
|
||||
|
||||
gone = append(gone, w)
|
||||
}
|
||||
}
|
||||
|
||||
t.mu.Unlock()
|
||||
|
||||
for _, w := range gone {
|
||||
w.evict()
|
||||
|
||||
t.eng.log.Info(
|
||||
"evicted archive writer",
|
||||
"webhook_id", webhookID,
|
||||
"path", w.path,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// evictAll evicts every cached archive writer, exactly as evict
|
||||
// does for one target. The engine calls it at shutdown, once its
|
||||
// workers have returned. Closing the last handle on an archive
|
||||
// moves the contents of its -wal into the .db and removes the
|
||||
// -wal, so a clean stop leaves each archive as a single file.
|
||||
func (t *databaseTarget) evictAll() {
|
||||
t.mu.Lock()
|
||||
|
||||
writers := t.writers
|
||||
t.writers = nil
|
||||
|
||||
t.mu.Unlock()
|
||||
|
||||
for _, w := range writers {
|
||||
w.evict()
|
||||
}
|
||||
}
|
||||
|
||||
// sweepArchive prunes one database target's archive of rows older
|
||||
// than expiry, without requiring a write. A missing archive file is
|
||||
// left missing (see sweepExpired), so a sweep never creates an
|
||||
// archive for a target that has never received an event.
|
||||
//
|
||||
// It also never leaves a registry entry behind: an entry it had
|
||||
// to create to reach the writer's mutex is released again once
|
||||
// the prune is done, so a sweep racing a target deletion cannot
|
||||
// resurrect the writer the eviction just dropped.
|
||||
func (t *databaseTarget) sweepArchive(
|
||||
targetID string, expiry time.Duration,
|
||||
) error {
|
||||
w, created, err := t.sweepWriterFor(targetID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if created {
|
||||
defer t.releaseSweepWriter(targetID, w)
|
||||
}
|
||||
|
||||
return w.sweepExpired(expiry)
|
||||
}
|
||||
|
||||
@@ -1,26 +1,69 @@
|
||||
package delivery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
// archiveTestWebhookName is the name of every webhook
|
||||
// seedDatabaseTarget creates. It is not safe in a file name as it
|
||||
// stands, so every archive file name made from it is made safe first.
|
||||
const archiveTestWebhookName = "Sweep Test!"
|
||||
|
||||
// archiveEnv bundles the pieces a database target test drives: a main
|
||||
// configuration database holding webhooks and targets, a delivery
|
||||
// engine whose archive writers write into the data directory, and that
|
||||
// directory.
|
||||
type archiveEnv struct {
|
||||
eng *delivery.Engine
|
||||
mainDB *database.Database
|
||||
dataDir string
|
||||
}
|
||||
|
||||
func setupArchiveTest(t *testing.T) *archiveEnv {
|
||||
t.Helper()
|
||||
|
||||
dataDir := t.TempDir()
|
||||
|
||||
mainDB, err := database.Open(dataDir, slog.New(slog.DiscardHandler))
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = mainDB.Close() })
|
||||
|
||||
return &archiveEnv{
|
||||
eng: delivery.NewTestEngineWithDB(
|
||||
mainDB,
|
||||
databasetest.NewWebhookDBManager(t, dataDir),
|
||||
archiveTestLogger(),
|
||||
&http.Client{Timeout: 5 * time.Second},
|
||||
1,
|
||||
),
|
||||
mainDB: mainDB,
|
||||
dataDir: dataDir,
|
||||
}
|
||||
}
|
||||
|
||||
func archiveTestLogger() *slog.Logger {
|
||||
return slog.New(slog.NewTextHandler(
|
||||
os.Stderr,
|
||||
@@ -28,8 +71,87 @@ func archiveTestLogger() *slog.Logger {
|
||||
))
|
||||
}
|
||||
|
||||
// archivePath returns where the engine keeps a database target's
|
||||
// archive file, for the names seedDatabaseTarget gave it.
|
||||
func (env *archiveEnv) archivePath(tgt *database.Target) string {
|
||||
return filepath.Join(
|
||||
env.dataDir,
|
||||
archive.FileName(archiveTestWebhookName, tgt.Name, tgt.ID),
|
||||
)
|
||||
}
|
||||
|
||||
// seedDatabaseTarget creates a webhook with one database target,
|
||||
// which keeps its archive forever in one file, and returns the
|
||||
// target.
|
||||
func (env *archiveEnv) seedDatabaseTarget(
|
||||
t *testing.T,
|
||||
) *database.Target {
|
||||
t.Helper()
|
||||
|
||||
wh := &database.Webhook{
|
||||
UserID: uuid.New().String(),
|
||||
Name: archiveTestWebhookName,
|
||||
}
|
||||
require.NoError(
|
||||
t,
|
||||
env.mainDB.DB().
|
||||
Omit(clause.Associations).
|
||||
Create(wh).Error,
|
||||
)
|
||||
|
||||
tgt := &database.Target{
|
||||
WebhookID: wh.ID,
|
||||
Name: "Archive",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Active: true,
|
||||
ArchiveRotation: "none",
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
require.NoError(
|
||||
t,
|
||||
env.mainDB.DB().
|
||||
Omit(clause.Associations).
|
||||
Create(tgt).Error,
|
||||
)
|
||||
|
||||
return tgt
|
||||
}
|
||||
|
||||
// seedDatabaseTargetDelivery seeds a pending delivery of an event
|
||||
// to a database target and returns the in-memory delivery the
|
||||
// target handler is invoked with.
|
||||
func seedDatabaseTargetDelivery(
|
||||
t *testing.T,
|
||||
webhookDB *gorm.DB,
|
||||
event database.Event,
|
||||
tgt *database.Target,
|
||||
) *database.Delivery {
|
||||
t.Helper()
|
||||
|
||||
dlv := seedDelivery(
|
||||
t, webhookDB, event.ID, tgt.ID,
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
d := &database.Delivery{
|
||||
EventID: event.ID,
|
||||
TargetID: tgt.ID,
|
||||
Status: database.DeliveryStatusPending,
|
||||
Event: event,
|
||||
Target: database.Target{
|
||||
Name: tgt.Name,
|
||||
Type: database.TargetTypeDatabase,
|
||||
ArchiveExpiry: tgt.ArchiveExpiry,
|
||||
ArchiveRotation: tgt.ArchiveRotation,
|
||||
},
|
||||
}
|
||||
d.ID = dlv.ID
|
||||
|
||||
return d
|
||||
}
|
||||
|
||||
// openArchiveDBForRead opens an archive file read-only so a
|
||||
// test can inspect the rows the writer persisted.
|
||||
// test can inspect the rows the database target archived.
|
||||
func openArchiveDBForRead(
|
||||
t *testing.T, path string,
|
||||
) *gorm.DB {
|
||||
@@ -52,26 +174,45 @@ func openArchiveDBForRead(
|
||||
return gdb
|
||||
}
|
||||
|
||||
// archiveFileSuffixes returns the archive file itself and the
|
||||
// SQLite sidecars that accompany an open database. A test that
|
||||
// asserts no archive was created has to check all of them.
|
||||
func archiveFileSuffixes() []string {
|
||||
return []string{"", "-wal", "-shm"}
|
||||
// countArchivedRows counts the rows in an archive file without
|
||||
// asserting anything, so it is safe to poll from an
|
||||
// assert.Eventually condition (which runs off the test
|
||||
// goroutine, where testify assertions must not be used).
|
||||
func countArchivedRows(path string) (int64, error) {
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite", fmt.Sprintf("file:%s?mode=ro", path),
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
defer func() { _ = sqlDB.Close() }()
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
var count int64
|
||||
|
||||
err = gdb.Table("archived_events").Count(&count).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return count, nil
|
||||
}
|
||||
|
||||
// removeArchiveFiles simulates an operator moving the archive
|
||||
// away by deleting the SQLite file and its sidecar files.
|
||||
func removeArchiveFiles(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
|
||||
for _, suffix := range []string{
|
||||
"", "-wal", "-shm", "-journal",
|
||||
} {
|
||||
err := os.Remove(path + suffix)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
t.Fatalf("removing %s%s: %v", path, suffix, err)
|
||||
}
|
||||
}
|
||||
// archivedRow is the columns of an archive row these tests read.
|
||||
type archivedRow struct {
|
||||
EventID string
|
||||
WebhookID string
|
||||
Method string
|
||||
RawQuery string
|
||||
Body string
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_ArchivesEvent verifies that delivering to
|
||||
@@ -81,7 +222,7 @@ func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
tgt := env.seedDatabaseTarget(t)
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"archived":true}`)
|
||||
@@ -107,9 +248,9 @@ func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
|
||||
|
||||
rdb := openArchiveDBForRead(t, archivePath)
|
||||
|
||||
var rows []delivery.ExportArchivedEvent
|
||||
var rows []archivedRow
|
||||
|
||||
require.NoError(t, rdb.Find(&rows).Error)
|
||||
require.NoError(t, rdb.Table("archived_events").Find(&rows).Error)
|
||||
require.Len(t, rows, 1)
|
||||
assert.Equal(t, event.ID, rows[0].EventID)
|
||||
assert.Equal(t, event.WebhookID, rows[0].WebhookID)
|
||||
@@ -118,266 +259,10 @@ func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
|
||||
assert.JSONEq(t, `{"archived":true}`, rows[0].Body)
|
||||
}
|
||||
|
||||
func TestArchiveWriter_WritesRow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
row := delivery.ExportArchivedEvent{
|
||||
EventID: "ev-1",
|
||||
WebhookID: "wh-1",
|
||||
EntrypointID: "ep-1",
|
||||
Method: "POST",
|
||||
Headers: `{"X":"Y"}`,
|
||||
Body: `{"hello":"world"}`,
|
||||
ContentType: "application/json",
|
||||
}
|
||||
|
||||
require.NoError(t, w.Write(row, 0))
|
||||
assert.FileExists(t, path)
|
||||
|
||||
var got []delivery.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, w.DB().Find(&got).Error)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "ev-1", got[0].EventID)
|
||||
assert.Equal(t, "wh-1", got[0].WebhookID)
|
||||
assert.Equal(t, "ep-1", got[0].EntrypointID)
|
||||
assert.Equal(t, row.Method, got[0].Method)
|
||||
assert.Equal(t, row.ContentType, got[0].ContentType)
|
||||
assert.JSONEq(t, `{"hello":"world"}`, got[0].Body)
|
||||
assert.False(t, got[0].ArchivedAt.IsZero())
|
||||
|
||||
var archivedAt string
|
||||
|
||||
require.NoError(t, w.DB().Raw(
|
||||
"SELECT CAST(archived_at AS TEXT) FROM archived_events",
|
||||
).Row().Scan(&archivedAt))
|
||||
assertStoredTime(t, archivedAt)
|
||||
}
|
||||
|
||||
func TestArchiveWriter_RecreatesAfterRemoval(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Write(
|
||||
delivery.ExportArchivedEvent{EventID: "a"}, 0,
|
||||
))
|
||||
assert.FileExists(t, path)
|
||||
|
||||
// The operator moves the archive away while the handle is
|
||||
// still open.
|
||||
removeArchiveFiles(t, path)
|
||||
require.NoFileExists(t, path)
|
||||
|
||||
// The next write recreates the file with a fresh schema and
|
||||
// only the new row.
|
||||
require.NoError(t, w.Write(
|
||||
delivery.ExportArchivedEvent{EventID: "b"}, 0,
|
||||
))
|
||||
assert.FileExists(t, path)
|
||||
|
||||
var got []delivery.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, w.DB().Find(&got).Error)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "b", got[0].EventID)
|
||||
}
|
||||
|
||||
// TestStatArchive proves StatArchive finds no file before the first
|
||||
// write; after a write still held in the -wal, counts the -wal in the
|
||||
// size and takes its later time as the last write; and finds no file
|
||||
// again once the file has been moved away.
|
||||
func TestStatArchive(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
|
||||
_, err := delivery.StatArchive(path)
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
|
||||
// With the clock stopped, the reopen debounce never passes, so
|
||||
// the handle stays open after the write.
|
||||
stopped := time.Now()
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
w.SetNow(func() time.Time { return stopped })
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "a"}, 0))
|
||||
|
||||
written := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
earlier := written.Add(-time.Hour)
|
||||
require.NoError(t, os.Chtimes(path, earlier, earlier))
|
||||
require.NoError(t, os.Chtimes(path+"-wal", written, written))
|
||||
|
||||
file, err := os.Stat(path)
|
||||
require.NoError(t, err)
|
||||
|
||||
wal, err := os.Stat(path + "-wal")
|
||||
require.NoError(t, err)
|
||||
require.Positive(t, wal.Size())
|
||||
|
||||
got, err := delivery.StatArchive(path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, got.Files)
|
||||
assert.Equal(t, file.Size()+wal.Size(), got.Size)
|
||||
assert.True(t, written.Equal(got.Written), got.Written)
|
||||
|
||||
removeArchiveFiles(t, path)
|
||||
|
||||
_, err = delivery.StatArchive(path)
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
}
|
||||
|
||||
func TestArchiveWriter_ReopenDebounce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const debounce = 2 * time.Second
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), debounce,
|
||||
)
|
||||
|
||||
// The writer measures its reopen debounce on this clock, which
|
||||
// only the test moves, so how long the host takes between
|
||||
// writes cannot change the result.
|
||||
now := time.Now()
|
||||
|
||||
w.SetNow(func() time.Time { return now })
|
||||
|
||||
require.NoError(t, w.Write(
|
||||
delivery.ExportArchivedEvent{EventID: "a"}, 0,
|
||||
))
|
||||
require.NoError(t, w.Write(
|
||||
delivery.ExportArchivedEvent{EventID: "b"}, 0,
|
||||
))
|
||||
|
||||
// Two writes inside the debounce window trigger only the
|
||||
// initial open — no extra close/reopen.
|
||||
assert.Equal(t, 1, w.Reopens())
|
||||
|
||||
now = now.Add(debounce)
|
||||
|
||||
require.NoError(t, w.Write(
|
||||
delivery.ExportArchivedEvent{EventID: "c"}, 0,
|
||||
))
|
||||
|
||||
// A write after the window elapses closes and reopens once.
|
||||
assert.Equal(t, 2, w.Reopens())
|
||||
}
|
||||
|
||||
func TestArchiveWriter_ExpiryPrune(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
path, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Open(0))
|
||||
|
||||
old := delivery.ExportArchivedEvent{
|
||||
EventID: "old",
|
||||
ArchivedAt: time.Now().Add(-2 * time.Hour),
|
||||
}
|
||||
fresh := delivery.ExportArchivedEvent{
|
||||
EventID: "fresh",
|
||||
ArchivedAt: time.Now(),
|
||||
}
|
||||
|
||||
require.NoError(t, w.DB().Create(&old).Error)
|
||||
require.NoError(t, w.DB().Create(&fresh).Error)
|
||||
|
||||
// Reopening with a one-hour expiry prunes the old row.
|
||||
require.NoError(t, w.Reopen(time.Hour))
|
||||
|
||||
var got []delivery.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, w.DB().Find(&got).Error)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "fresh", got[0].EventID)
|
||||
}
|
||||
|
||||
func TestParseArchiveExpiry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
in string
|
||||
want time.Duration
|
||||
wantErr bool
|
||||
}{
|
||||
{"empty config", "", 0, false},
|
||||
{"explicit never", `{"expiry":"never"}`, 0, false},
|
||||
{"empty expiry", `{"expiry":""}`, 0, false},
|
||||
{"duration", `{"expiry":"1h"}`, time.Hour, false},
|
||||
{"unparseable", `{"expiry":"nonsense"}`, 0, true},
|
||||
{"zero duration", `{"expiry":"0s"}`, 0, true},
|
||||
{"negative duration", `{"expiry":"-5h"}`, 0, true},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := delivery.ExportParseArchiveExpiry(tc.in)
|
||||
if tc.wantErr {
|
||||
require.Error(t, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tc.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// seedDatabaseTargetDelivery seeds a pending delivery of an event
|
||||
// to a database target and returns the in-memory delivery the
|
||||
// target handler is invoked with.
|
||||
func seedDatabaseTargetDelivery(
|
||||
t *testing.T,
|
||||
webhookDB *gorm.DB,
|
||||
event database.Event,
|
||||
tgt *database.Target,
|
||||
) *database.Delivery {
|
||||
t.Helper()
|
||||
|
||||
dlv := seedDelivery(
|
||||
t, webhookDB, event.ID, tgt.ID,
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
d := &database.Delivery{
|
||||
EventID: event.ID,
|
||||
TargetID: tgt.ID,
|
||||
Status: database.DeliveryStatusPending,
|
||||
Event: event,
|
||||
Target: database.Target{
|
||||
Name: tgt.Name,
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: tgt.Config,
|
||||
},
|
||||
}
|
||||
d.ID = dlv.ID
|
||||
|
||||
return d
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_ArchiveFailureFailsDelivery verifies that
|
||||
// an archive error (here: an unparseable expiry in the target
|
||||
// config) fails the delivery loudly: the attempt is recorded as
|
||||
// failed with the error and the delivery is marked failed, not
|
||||
// an archive error (here: an event with no webhook id to record)
|
||||
// fails the delivery loudly: the attempt is recorded as failed
|
||||
// with the error and the delivery is marked failed, not
|
||||
// delivered.
|
||||
func TestDeliverDatabase_ArchiveFailureFailsDelivery(
|
||||
t *testing.T,
|
||||
@@ -385,11 +270,12 @@ func TestDeliverDatabase_ArchiveFailureFailsDelivery(
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, `{"expiry":"nonsense"}`)
|
||||
tgt := env.seedDatabaseTarget(t)
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"archived":false}`)
|
||||
d := seedDatabaseTargetDelivery(t, webhookDB, event, tgt)
|
||||
d.Event.WebhookID = ""
|
||||
|
||||
env.eng.ExportDeliverDatabase(webhookDB, d)
|
||||
|
||||
@@ -414,357 +300,96 @@ func TestDeliverDatabase_ArchiveFailureFailsDelivery(
|
||||
"the attempt must be recorded as failed",
|
||||
)
|
||||
assert.Contains(t,
|
||||
results[0].Error, "nonsense",
|
||||
results[0].Error, "without a webhook id",
|
||||
"the archive error must be recorded on the attempt",
|
||||
)
|
||||
|
||||
assert.NoFileExists(t,
|
||||
env.archivePath(tgt),
|
||||
"no archive file should exist for a failed config",
|
||||
"no archive file should exist for a failed write",
|
||||
)
|
||||
}
|
||||
|
||||
func TestValidateArchiveExpiry(t *testing.T) {
|
||||
t.Parallel()
|
||||
// seedArchivedRow creates the archive file at path holding one row, for
|
||||
// eventID, archived at archivedAt.
|
||||
func seedArchivedRow(
|
||||
t *testing.T, path, eventID string, archivedAt time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
valid := []string{"", "never", "1h", "720h", "30m"}
|
||||
for _, in := range valid {
|
||||
require.NoError(t,
|
||||
delivery.ValidateArchiveExpiry(in),
|
||||
"expiry %q should be accepted", in,
|
||||
)
|
||||
}
|
||||
|
||||
invalid := []string{"nonsense", "7d", "-5h", "0s", "0"}
|
||||
for _, in := range invalid {
|
||||
require.Error(t,
|
||||
delivery.ValidateArchiveExpiry(in),
|
||||
"expiry %q should be rejected", in,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestArchiveFileName pins the archive file name and the rules
|
||||
// that make a webhook or target name safe to put in it.
|
||||
func TestArchiveFileName(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const id = "3f2a1c9e-8d4b-4c1a-9e2f-0a1b2c3d4e5f"
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
webhook string
|
||||
target string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"plain names", "orders", "archive",
|
||||
"archive-orders-archive-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"lowercased", "Orders", "Main Archive",
|
||||
"archive-orders-main-archive-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"a run of other characters is one dash",
|
||||
`a /\..b`, "c__--d",
|
||||
"archive-a-b-c-d-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"no dash at either end", " --orders!! ", "(archive)",
|
||||
"archive-orders-archive-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"path separators", "../../etc/passwd", "a/b",
|
||||
"archive-etc-passwd-a-b-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"letters outside ASCII are dropped",
|
||||
"Bestellungen Größe", "café",
|
||||
"archive-bestellungen-gr-e-caf-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"nothing left is unnamed", "", "!!!",
|
||||
"archive-unnamed-unnamed-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"cut to 40 characters", strings.Repeat("a", 50), "x",
|
||||
"archive-" + strings.Repeat("a", 40) + "-x-" + id + ".db",
|
||||
},
|
||||
{
|
||||
"no dash left by the cut",
|
||||
strings.Repeat("a", 39) + " b", "x",
|
||||
"archive-" + strings.Repeat("a", 39) + "-x-" + id + ".db",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assert.Equal(
|
||||
t, tc.want,
|
||||
delivery.ArchiveFileName(tc.webhook, tc.target, id),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_EachTargetHasItsOwnArchive proves two
|
||||
// database targets of one webhook archive into separate files.
|
||||
func TestDeliverDatabase_EachTargetHasItsOwnArchive(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
first := env.seedDatabaseTarget(t, "")
|
||||
second := env.addDatabaseTarget(t, first.WebhookID, "")
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
|
||||
for _, tgt := range []*database.Target{first, second} {
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB,
|
||||
seedDatabaseTargetDelivery(t, webhookDB, event, tgt),
|
||||
)
|
||||
}
|
||||
|
||||
require.NotEqual(
|
||||
t, env.archivePath(first), env.archivePath(second),
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
|
||||
)
|
||||
assert.Equal(
|
||||
t, []string{event.ID},
|
||||
archivedEventIDs(t, env.archivePath(first)),
|
||||
)
|
||||
assert.Equal(
|
||||
t, []string{event.ID},
|
||||
archivedEventIDs(t, env.archivePath(second)),
|
||||
)
|
||||
}
|
||||
|
||||
// TestRename_MovesTheFile proves a rename moves the archive, rows
|
||||
// and all, and that later writes go to the new name.
|
||||
func TestRename_MovesTheFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
oldPath := env.archivePath(tgt)
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
first := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB, seedDatabaseTargetDelivery(t, webhookDB, first, tgt),
|
||||
)
|
||||
require.FileExists(t, oldPath)
|
||||
|
||||
require.NoError(
|
||||
t, env.eng.Rename(tgt.ID, "Orders", "Long Term"),
|
||||
)
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
)
|
||||
|
||||
assert.NoFileExists(t, oldPath)
|
||||
assert.Equal(t, []string{first.ID}, archivedEventIDs(t, newPath))
|
||||
|
||||
second := seedEvent(t, webhookDB, `{"n":2}`)
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB,
|
||||
seedDatabaseTargetDelivery(t, webhookDB, second, tgt),
|
||||
)
|
||||
|
||||
assert.ElementsMatch(
|
||||
t, []string{first.ID, second.ID},
|
||||
archivedEventIDs(t, newPath),
|
||||
)
|
||||
assert.NoFileExists(
|
||||
t, oldPath, "a write after the rename must use the new name",
|
||||
)
|
||||
}
|
||||
|
||||
// TestRename_NeverReplacesAFile plants a file at the new name, once
|
||||
// the .db alone, once a lone -wal and once a lone -shm, and proves
|
||||
// each time that the rename is refused, the planted file survives,
|
||||
// and the archive keeps its name and its rows.
|
||||
func TestRename_NeverReplacesAFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
t.Run("planted .db"+suffix, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
oldPath := env.archivePath(tgt)
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
first := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB,
|
||||
seedDatabaseTargetDelivery(t, webhookDB, first, tgt),
|
||||
)
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
)
|
||||
plantedPath := newPath + suffix
|
||||
require.NoError(
|
||||
t, os.WriteFile(plantedPath, []byte("planted"), 0o600),
|
||||
)
|
||||
|
||||
require.ErrorIs(
|
||||
t, env.eng.Rename(tgt.ID, "Orders", "Long Term"),
|
||||
delivery.ErrArchiveNameTaken,
|
||||
)
|
||||
|
||||
//nolint:gosec // reads the file the test planted under t.TempDir()
|
||||
planted, err := os.ReadFile(plantedPath)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "planted", string(planted))
|
||||
|
||||
second := seedEvent(t, webhookDB, `{"n":2}`)
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB,
|
||||
seedDatabaseTargetDelivery(t, webhookDB, second, tgt),
|
||||
)
|
||||
|
||||
assert.ElementsMatch(
|
||||
t, []string{first.ID, second.ID},
|
||||
archivedEventIDs(t, oldPath),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRename_BeforeTheNameIsSaved covers the order the handlers
|
||||
// use: they rename before they save the new name, so a delivery in
|
||||
// between must write under the new name although the main database
|
||||
// still has the old one. It also shows that renaming an archive that
|
||||
// does not exist yet is not an error.
|
||||
func TestRename_BeforeTheNameIsSaved(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
|
||||
require.NoError(
|
||||
t, env.eng.Rename(tgt.ID, "Orders", "Archive"),
|
||||
)
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB, seedDatabaseTargetDelivery(t, webhookDB, event, tgt),
|
||||
)
|
||||
|
||||
assert.FileExists(
|
||||
t,
|
||||
filepath.Join(
|
||||
env.dataDir, "archive-orders-archive-"+tgt.ID+".db",
|
||||
),
|
||||
)
|
||||
assert.NoFileExists(t, env.archivePath(tgt))
|
||||
}
|
||||
|
||||
// TestArchiveWriter_RenameMovesSidecars proves a rename carries
|
||||
// the -wal and -shm a crash can leave beside an archive no handle
|
||||
// has opened since. SQLite finds them by name, so a -wal left
|
||||
// behind would lose the transactions it holds.
|
||||
func TestArchiveWriter_RenameMovesSidecars(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
oldPath := filepath.Join(dir, "archive-old.db")
|
||||
newPath := filepath.Join(dir, "archive-new.db")
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
require.NoError(
|
||||
t, os.WriteFile(oldPath+suffix, []byte(suffix), 0o600),
|
||||
)
|
||||
}
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
oldPath, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.NoError(t, w.Rename("archive-new.db"))
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
assert.NoFileExists(t, oldPath+suffix)
|
||||
assert.FileExists(t, newPath+suffix)
|
||||
}
|
||||
|
||||
assert.Equal(t, newPath, w.Path())
|
||||
}
|
||||
|
||||
// TestArchiveWriter_RenameMovesBackOnFailure makes the -wal fail to
|
||||
// move after the .db has moved, and proves the .db is moved back, so
|
||||
// the archive is never split across two names. The new name is 255
|
||||
// bytes, the longest a file name may be, so the .db can take it but
|
||||
// the -wal, four bytes longer, cannot.
|
||||
func TestArchiveWriter_RenameMovesBackOnFailure(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
oldPath := filepath.Join(dir, "archive-old.db")
|
||||
newName := strings.Repeat("a", 252) + ".db"
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
require.NoError(
|
||||
t, os.WriteFile(oldPath+suffix, []byte(suffix), 0o600),
|
||||
)
|
||||
}
|
||||
|
||||
w := delivery.NewExportArchiveWriter(
|
||||
oldPath, archiveTestLogger(), 0,
|
||||
)
|
||||
|
||||
require.Error(t, w.Rename(newName))
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
assert.FileExists(t, oldPath+suffix)
|
||||
}
|
||||
|
||||
assert.NoFileExists(t, filepath.Join(dir, newName))
|
||||
assert.Equal(t, oldPath, w.Path())
|
||||
}
|
||||
|
||||
// TestArchiveWriter_RenameMovesBackEveryFile renames a target with a
|
||||
// file without a period and a file for a month, and proves that when
|
||||
// the month's file fails to move, the file already moved is moved back:
|
||||
// both files are under the old name with their rows, and nothing is
|
||||
// under the new name. The new name is 251 bytes, so the file without a
|
||||
// period, with its -wal and -shm, can take it, but the month's file,
|
||||
// eight bytes longer, cannot.
|
||||
func TestArchiveWriter_RenameMovesBackEveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
oldPath := filepath.Join(dir, "archive-old.db")
|
||||
monthPath := filepath.Join(dir, "archive-old-2026-03.db")
|
||||
w := delivery.NewExportArchiveWriter(oldPath, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-none"}, 0, "",
|
||||
))
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-month"}, 0, "2026-03",
|
||||
))
|
||||
|
||||
require.Error(t, w.Rename(strings.Repeat("a", 248)+".db"))
|
||||
|
||||
assert.Equal(t, []string{"in-none"}, archivedEventIDs(t, oldPath))
|
||||
assert.Equal(t, []string{"in-month"}, archivedEventIDs(t, monthPath))
|
||||
|
||||
entries, err := os.ReadDir(dir)
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, entry := range entries {
|
||||
assert.True(t, strings.HasPrefix(entry.Name(), "archive-old"),
|
||||
"%s is not under the old name", entry.Name())
|
||||
}
|
||||
defer func() { _ = sqlDB.Close() }()
|
||||
|
||||
assert.Equal(t, oldPath, w.Path())
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, migrations.Apply(gdb, migrations.Archive))
|
||||
require.NoError(t, gdb.Exec(
|
||||
"INSERT INTO archived_events (event_id, archived_at) VALUES (?, ?)",
|
||||
eventID, archivedAt,
|
||||
).Error)
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_TaskArchiveSettings proves a task's ArchiveExpiry
|
||||
// and ArchiveRotation reach the archive through the engine. The
|
||||
// target's row has neither, so only the task can bring them: with daily
|
||||
// rotation the event goes to the file for its day, and opening that
|
||||
// file with a one-hour expiry prunes the row it already held, archived
|
||||
// two hours ago.
|
||||
func TestDeliverDatabase_TaskArchiveSettings(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
|
||||
webhook := database.Webhook{UserID: uuid.NewString(), Name: "settings"}
|
||||
webhook.ID = s.WebhookID
|
||||
require.NoError(t, s.MainDB.Create(&webhook).Error)
|
||||
|
||||
target := database.Target{
|
||||
WebhookID: s.WebhookID,
|
||||
Name: "archive",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Active: true,
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
require.NoError(t, s.MainDB.Create(&target).Error)
|
||||
|
||||
event := iSeedEvent(t, s.WebhookDB, s.WebhookID, `{"archived":true}`)
|
||||
d := iSeedDelivery(
|
||||
t, s.WebhookDB, event.ID, target.ID,
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, target.ID, target.Name, "", 1, 1,
|
||||
&event.Body,
|
||||
)
|
||||
task.TargetType = database.TargetTypeDatabase
|
||||
task.ArchiveExpiry = time.Hour
|
||||
task.ArchiveRotation = "daily"
|
||||
|
||||
daily := target
|
||||
daily.ArchiveRotation = task.ArchiveRotation
|
||||
dailyFile := archive.PathAt(s.DBMgr, &webhook, &daily, event.CreatedAt)
|
||||
seedArchivedRow(t, dailyFile, "expired", time.Now().Add(-2*time.Hour))
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
|
||||
iAssertStatus(t, s.WebhookDB, d.ID, database.DeliveryStatusDelivered)
|
||||
|
||||
var rows []archivedRow
|
||||
|
||||
require.NoError(t, openArchiveDBForRead(t, dailyFile).
|
||||
Table("archived_events").Find(&rows).Error)
|
||||
require.Len(t, rows, 1)
|
||||
assert.Equal(t, event.ID, rows[0].EventID,
|
||||
"the event goes to its day's file and the expired row is pruned",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
const MaxTargetTimeoutSeconds = 300
|
||||
|
||||
// Errors returned when a target's header or timeout form input
|
||||
// cannot be turned into a configuration.
|
||||
// cannot be turned into the target's setting.
|
||||
//
|
||||
// None of these ever quotes a header VALUE. A target header value
|
||||
// is routinely an authorization token, and these messages are shown
|
||||
@@ -48,8 +48,8 @@ var (
|
||||
)
|
||||
|
||||
// isReservedTargetHeader reports whether name (canonicalised) is a
|
||||
// header a target configuration may not set, because the delivery
|
||||
// path or net/http itself writes it regardless.
|
||||
// header a target may not set, because the delivery path or
|
||||
// net/http itself writes it regardless.
|
||||
//
|
||||
// These are rejected rather than accepted-and-ignored. Storing a
|
||||
// header that provably never reaches the wire tells the operator
|
||||
@@ -77,13 +77,11 @@ func isReservedTargetHeader(name string) bool {
|
||||
|
||||
// ParseTargetHeaders turns the target form's headers field — one
|
||||
// "Name: value" pair per line, blank lines ignored — into the map
|
||||
// stored in HTTPTargetConfig.Headers. Names are canonicalised, so a
|
||||
// stored in database.Target.Headers. Names are canonicalised, so a
|
||||
// name repeated in a different case is still a duplicate rather than
|
||||
// one pair silently overwriting the other.
|
||||
//
|
||||
// An input with no pairs yields an empty map, which omitempty drops
|
||||
// from the stored config, so a target configured with no headers
|
||||
// stores no headers key at all.
|
||||
// An input with no pairs yields an empty map.
|
||||
func ParseTargetHeaders(raw string) (map[string]string, error) {
|
||||
headers := make(map[string]string)
|
||||
|
||||
@@ -223,8 +221,8 @@ func FormatTargetHeaders(headers map[string]string) string {
|
||||
|
||||
// ParseTargetTimeout interprets the target form's timeout field as
|
||||
// a whole number of seconds. An empty field means "unset" and yields
|
||||
// 0, which omitempty drops from the stored config and which the
|
||||
// delivery path reads as "use the shared client's timeout".
|
||||
// 0, which the delivery path reads as "use the shared client's
|
||||
// timeout".
|
||||
//
|
||||
// Anything else that is not a whole number in range is an error, not
|
||||
// a silently substituted default: a target whose timeout was typed
|
||||
|
||||
@@ -1,23 +1,21 @@
|
||||
package delivery_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strconv"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// Literals these tests repeat, named so that the header names and the
|
||||
// keep-forever archive config each have one definition.
|
||||
// Literals these tests repeat, named so that each header name has one
|
||||
// definition.
|
||||
const (
|
||||
headerAuthorization = "Authorization"
|
||||
headerContentType = "Content-Type"
|
||||
headerUserAgent = "User-Agent"
|
||||
bearerValue = "Bearer abc"
|
||||
archiveConfigNever = "{\"expiry\":\"never\"}"
|
||||
)
|
||||
|
||||
func TestParseTargetHeaders_AcceptsPairs(t *testing.T) {
|
||||
@@ -41,23 +39,13 @@ func TestParseTargetHeaders_AcceptsPairs(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// A configuration with no headers must stay indistinguishable from
|
||||
// one written before the field existed, so omitempty drops the key.
|
||||
// An input of blank lines is no headers, not an error.
|
||||
func TestParseTargetHeaders_EmptyInputYieldsNoHeaders(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := delivery.ParseTargetHeaders("\n \n")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, got)
|
||||
|
||||
encoded, err := json.Marshal(delivery.HTTPTargetConfig{
|
||||
URL: "https://example.com/h",
|
||||
Headers: got,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.JSONEq(
|
||||
t, `{"url":"https://example.com/h"}`, string(encoded),
|
||||
)
|
||||
}
|
||||
|
||||
func TestParseTargetHeaders_Rejects(t *testing.T) {
|
||||
@@ -181,89 +169,6 @@ func TestFormatTargetTimeout(t *testing.T) {
|
||||
assert.Empty(t, delivery.FormatTargetTimeout(-1))
|
||||
}
|
||||
|
||||
func TestNewTargetConfigForm(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
form, err := delivery.NewTargetConfigForm(&database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"https://example.com/h",` +
|
||||
`"headers":{"Authorization":"Bearer abc"},` +
|
||||
`"timeout":9}`,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "https://example.com/h", form.URL)
|
||||
assert.Equal(t, "Authorization: Bearer abc\n", form.Headers)
|
||||
assert.Equal(t, "9", form.Timeout)
|
||||
|
||||
form, err = delivery.NewTargetConfigForm(&database.Target{
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"webhookUrl":"https://hooks.example/s"}`,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "https://hooks.example/s", form.URL)
|
||||
|
||||
form, err = delivery.NewTargetConfigForm(&database.Target{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: `{"expiry":"720h","rotation":"daily"}`,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "720h", form.Expiry)
|
||||
assert.Equal(t, rotationDaily, form.Rotation)
|
||||
|
||||
form, err = delivery.NewTargetConfigForm(&database.Target{
|
||||
Type: database.TargetTypeLog,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, form.URL)
|
||||
}
|
||||
|
||||
// A keep-forever archive target yields an empty expiry, so the edit
|
||||
// form starts on never; saving it unchanged stores never, which means
|
||||
// the same as an empty expiry.
|
||||
func TestNewTargetConfigForm_DatabaseNeverIsBlank(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, cfg := range []string{"", `{}`, archiveConfigNever} {
|
||||
form, err := delivery.NewTargetConfigForm(
|
||||
&database.Target{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: cfg,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err, cfg)
|
||||
assert.Empty(t, form.Expiry, cfg)
|
||||
}
|
||||
}
|
||||
|
||||
// An unreadable stored config is an error rather than a blank form
|
||||
// that looks like a target with no settings, so the caller can tell
|
||||
// the operator that saving replaces the stored value.
|
||||
func TestNewTargetConfigForm_UnreadableConfigErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []*database.Target{
|
||||
{Type: database.TargetTypeHTTP, Config: "not json", MaxAttempts: 1},
|
||||
{Type: database.TargetTypeHTTP, Config: `{}`, MaxAttempts: 1},
|
||||
{Type: database.TargetTypeSlack, Config: "", MaxAttempts: 1},
|
||||
{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: `{"expiry":"soon"}`,
|
||||
},
|
||||
{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: weeklyConfig,
|
||||
},
|
||||
{Type: database.TargetType("nope")},
|
||||
}
|
||||
|
||||
for _, target := range cases {
|
||||
_, err := delivery.NewTargetConfigForm(target)
|
||||
require.Error(t, err, target.Type)
|
||||
}
|
||||
}
|
||||
|
||||
// The ceiling exists so one misconfigured target cannot hold a
|
||||
// delivery worker indefinitely, and it is inclusive.
|
||||
func TestParseTargetTimeout_CeilingIsInclusive(t *testing.T) {
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
@@ -17,29 +16,6 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// Sentinel errors returned by the config parsers.
|
||||
var (
|
||||
errEmptyTargetConfig = errors.New(
|
||||
"empty target config",
|
||||
)
|
||||
errMissingTargetURL = errors.New(
|
||||
"target URL is required",
|
||||
)
|
||||
)
|
||||
|
||||
// HTTPTargetConfig holds configuration for http target
|
||||
// types.
|
||||
type HTTPTargetConfig struct {
|
||||
URL string `json:"url"`
|
||||
Headers map[string]string `json:"headers,omitempty"`
|
||||
Timeout int `json:"timeout,omitempty"`
|
||||
|
||||
// ForwardQuery passes each event's query string on to the target,
|
||||
// appended to URL. Off, the target URL is sent exactly as
|
||||
// configured.
|
||||
ForwardQuery bool `json:"forwardQuery,omitempty"`
|
||||
}
|
||||
|
||||
// httpCore holds the retry, backoff, and circuit-breaker
|
||||
// machinery shared by the HTTP and Slack targets. Each of
|
||||
// those targets owns its own httpCore instance (and thus its
|
||||
@@ -84,8 +60,8 @@ func (c *httpCore) deliver(
|
||||
// engine is stopping, which cancels the context its request is made
|
||||
// with. Such an attempt is not recorded: no DeliveryResult, no circuit
|
||||
// breaker outcome, no status change, no count in the attempt metrics.
|
||||
// The delivery stays pending or retrying, and startup recovery sends
|
||||
// it again on the next start.
|
||||
// The delivery stays pending or retrying, and the recovery routine
|
||||
// sends it again on the next start.
|
||||
//
|
||||
// A target's own timeout leaves that context alone, so its attempt is
|
||||
// still recorded as the failure it is.
|
||||
@@ -334,7 +310,8 @@ func (c *httpCore) publishCircuitState(
|
||||
|
||||
// remainingBackoff returns how long remains of the backoff
|
||||
// window for the last attempt of a recovered retrying
|
||||
// delivery. It implements rescheduler.
|
||||
// delivery: zero once it has passed, or when the last attempt
|
||||
// cannot be read. It implements rescheduler.
|
||||
func (c *httpCore) remainingBackoff(
|
||||
webhookDB *gorm.DB,
|
||||
deliveryID string,
|
||||
@@ -357,29 +334,6 @@ func (c *httpCore) remainingBackoff(
|
||||
return max(remaining, 0)
|
||||
}
|
||||
|
||||
// backoffElapsed reports whether the backoff window for the
|
||||
// last attempt of a retrying delivery has passed. It
|
||||
// implements rescheduler.
|
||||
func (c *httpCore) backoffElapsed(
|
||||
webhookDB *gorm.DB,
|
||||
deliveryID string,
|
||||
attemptNum int,
|
||||
) bool {
|
||||
var lastResult database.DeliveryResult
|
||||
|
||||
err := webhookDB.
|
||||
Where("delivery_id = ?", deliveryID).
|
||||
Order("created_at DESC").
|
||||
First(&lastResult).Error
|
||||
if err != nil {
|
||||
return true
|
||||
}
|
||||
|
||||
backoff := Backoff(attemptNum)
|
||||
|
||||
return time.Since(lastResult.CreatedAt) >= backoff
|
||||
}
|
||||
|
||||
// Backoff is how long an http or slack target with retries waits after
|
||||
// a delivery's failed attempt attemptNum before trying it again.
|
||||
func Backoff(attemptNum int) time.Duration {
|
||||
@@ -407,34 +361,8 @@ func (t *httpTarget) Deliver(
|
||||
task *Task,
|
||||
sched Scheduler,
|
||||
) {
|
||||
cfg, err := parseHTTPConfig(d.Target.Config)
|
||||
if err != nil {
|
||||
t.eng.log.Error(
|
||||
"invalid HTTP target config",
|
||||
"target_id", d.TargetID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
recErr := t.eng.recordResult(
|
||||
webhookDB, d, task.AttemptNum,
|
||||
false, 0, "", err.Error(), 0,
|
||||
)
|
||||
if recErr != nil {
|
||||
t.eng.bookkeepingFailed(d, recErr)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
t.eng.settleStatus(
|
||||
webhookDB, d, d.Target.Type,
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
attempt := func() attemptResult {
|
||||
return t.attempt(ctx, cfg, &d.Event)
|
||||
return t.attempt(ctx, &d.Target, &d.Event)
|
||||
}
|
||||
|
||||
t.deliver(
|
||||
@@ -449,11 +377,11 @@ func (t *httpTarget) Deliver(
|
||||
// string; only a transport-level error does.
|
||||
func (t *httpTarget) attempt(
|
||||
ctx context.Context,
|
||||
cfg *HTTPTargetConfig,
|
||||
target *database.Target,
|
||||
event *database.Event,
|
||||
) attemptResult {
|
||||
statusCode, respBody, duration, reqErr :=
|
||||
t.doHTTPRequest(ctx, cfg, event)
|
||||
t.doHTTPRequest(ctx, target, event)
|
||||
|
||||
success := reqErr == nil &&
|
||||
statusCode >= httpSuccessMin &&
|
||||
@@ -475,7 +403,7 @@ func (t *httpTarget) attempt(
|
||||
|
||||
func (t *httpTarget) doHTTPRequest(
|
||||
ctx context.Context,
|
||||
cfg *HTTPTargetConfig,
|
||||
target *database.Target,
|
||||
event *database.Event,
|
||||
) (int, string, int64, error) {
|
||||
start := time.Now()
|
||||
@@ -483,7 +411,7 @@ func (t *httpTarget) doHTTPRequest(
|
||||
req, reqErr := http.NewRequestWithContext(
|
||||
ctx,
|
||||
http.MethodPost,
|
||||
cfg.URL,
|
||||
target.URL,
|
||||
bytes.NewReader([]byte(event.Body)),
|
||||
)
|
||||
if reqErr != nil {
|
||||
@@ -493,15 +421,15 @@ func (t *httpTarget) doHTTPRequest(
|
||||
)
|
||||
}
|
||||
|
||||
if cfg.ForwardQuery {
|
||||
if target.ForwardQuery {
|
||||
appendQuery(req.URL, event.RawQuery)
|
||||
}
|
||||
|
||||
originScoped := applyRequestHeaders(
|
||||
req, event, cfg, t.eng.userAgent(),
|
||||
req, event, target.Headers, t.eng.userAgent(),
|
||||
)
|
||||
|
||||
client := t.clientForRequest(cfg, originScoped)
|
||||
client := t.clientForRequest(target.TimeoutSeconds, originScoped)
|
||||
|
||||
resp, doErr := executeHTTPRequest(client, req)
|
||||
|
||||
@@ -541,17 +469,17 @@ func appendQuery(u *url.URL, rawQuery string) {
|
||||
}
|
||||
|
||||
// clientForRequest returns the client for one delivery attempt.
|
||||
// originScoped is the header set applyRequestHeaders built for that
|
||||
// attempt; a request with neither a per-target timeout nor an
|
||||
// origin-scoped header gets the shared client, because there is
|
||||
// then nothing for the redirect policy to strip and net/http's
|
||||
// default policy already withholds Authorization and Cookie across
|
||||
// hosts.
|
||||
// timeoutSeconds is the target's TimeoutSeconds, and originScoped is
|
||||
// the header set applyRequestHeaders built for that attempt; a
|
||||
// request with neither a per-target timeout nor an origin-scoped
|
||||
// header gets the shared client, because there is then nothing for
|
||||
// the redirect policy to strip and net/http's default policy already
|
||||
// withholds Authorization and Cookie across hosts.
|
||||
func (t *httpTarget) clientForRequest(
|
||||
cfg *HTTPTargetConfig,
|
||||
timeoutSeconds int,
|
||||
originScoped []string,
|
||||
) *http.Client {
|
||||
if cfg.Timeout <= 0 && len(originScoped) == 0 {
|
||||
if timeoutSeconds <= 0 && len(originScoped) == 0 {
|
||||
return t.client
|
||||
}
|
||||
|
||||
@@ -564,9 +492,9 @@ func (t *httpTarget) clientForRequest(
|
||||
Transport: t.client.Transport,
|
||||
}
|
||||
|
||||
if cfg.Timeout > 0 {
|
||||
if timeoutSeconds > 0 {
|
||||
client.Timeout = time.Duration(
|
||||
cfg.Timeout,
|
||||
timeoutSeconds,
|
||||
) * time.Second
|
||||
}
|
||||
|
||||
@@ -577,31 +505,6 @@ func (t *httpTarget) clientForRequest(
|
||||
return client
|
||||
}
|
||||
|
||||
func parseHTTPConfig(
|
||||
configJSON string,
|
||||
) (*HTTPTargetConfig, error) {
|
||||
if configJSON == "" {
|
||||
return nil, errEmptyTargetConfig
|
||||
}
|
||||
|
||||
var cfg HTTPTargetConfig
|
||||
|
||||
err := json.Unmarshal(
|
||||
[]byte(configJSON), &cfg,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"parsing config JSON: %w", err,
|
||||
)
|
||||
}
|
||||
|
||||
if cfg.URL == "" {
|
||||
return nil, errMissingTargetURL
|
||||
}
|
||||
|
||||
return &cfg, nil
|
||||
}
|
||||
|
||||
// isForwardableHeader returns true if the header should
|
||||
// be forwarded to targets.
|
||||
func isForwardableHeader(name string) bool {
|
||||
@@ -633,11 +536,12 @@ func isForwardableHeader(name string) bool {
|
||||
// wins, otherwise the event's ContentType, otherwise none. The inbound
|
||||
// Content-Type in the event's headers is never forwarded.
|
||||
//
|
||||
// userAgent is set last, over any configured or inbound User-Agent.
|
||||
// configured is the target's Headers. userAgent is set last, over
|
||||
// any configured or inbound User-Agent.
|
||||
func applyRequestHeaders(
|
||||
req *http.Request,
|
||||
event *database.Event,
|
||||
cfg *HTTPTargetConfig,
|
||||
configured map[string]string,
|
||||
userAgent string,
|
||||
) []string {
|
||||
if event.ContentType != "" {
|
||||
@@ -648,7 +552,7 @@ func applyRequestHeaders(
|
||||
|
||||
originScoped := forwardEventHeaders(req, event)
|
||||
|
||||
for k, v := range cfg.Headers {
|
||||
for k, v := range configured {
|
||||
req.Header.Set(k, v)
|
||||
originScoped[http.CanonicalHeaderKey(k)] = struct{}{}
|
||||
}
|
||||
@@ -713,7 +617,7 @@ func forwardEventHeaders(
|
||||
}
|
||||
|
||||
// executeHTTPRequest sends an HTTP request using the provided
|
||||
// client. URLs are validated by the config parsers and the
|
||||
// client. URLs are validated by the target forms and the
|
||||
// SSRF-safe transport before reaching here.
|
||||
//
|
||||
// Transport failures are masked here, at the single point
|
||||
|
||||
@@ -2,7 +2,6 @@ package delivery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
@@ -18,25 +17,13 @@ import (
|
||||
// with.
|
||||
const eventQuery = "a=1&b=2"
|
||||
|
||||
// httpTargetConfig is the stored configuration of an HTTP target at
|
||||
// targetURL.
|
||||
func httpTargetConfig(
|
||||
t *testing.T, targetURL string, forwardQuery bool,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
cfg, err := json.Marshal(delivery.HTTPTargetConfig{
|
||||
URL: targetURL, ForwardQuery: forwardQuery,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
return string(cfg)
|
||||
}
|
||||
|
||||
// deliverWithQuery sends one event that arrived with eventQuery to an
|
||||
// HTTP target configured with cfg, through the path a received event's
|
||||
// delivery takes, and returns the attempt it recorded.
|
||||
func deliverWithQuery(t *testing.T, cfg string) database.DeliveryResult {
|
||||
// HTTP target at targetURL, with its ForwardQuery setting as given,
|
||||
// through the path a received event's delivery takes, and returns the
|
||||
// attempt it recorded.
|
||||
func deliverWithQuery(
|
||||
t *testing.T, targetURL string, forwardQuery bool,
|
||||
) database.DeliveryResult {
|
||||
t.Helper()
|
||||
|
||||
s := newISetup(t)
|
||||
@@ -47,8 +34,10 @@ func deliverWithQuery(t *testing.T, cfg string) database.DeliveryResult {
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
task := iTask(
|
||||
d, event, s.WebhookID, d.TargetID, "query", cfg, 1, 1, &event.Body,
|
||||
d, event, s.WebhookID, d.TargetID, "query", targetURL, 1, 1,
|
||||
&event.Body,
|
||||
)
|
||||
task.ForwardQuery = forwardQuery
|
||||
task.RawQuery = eventQuery
|
||||
|
||||
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||
@@ -111,9 +100,9 @@ func TestDeliverHTTP_ForwardQuery(t *testing.T) {
|
||||
))
|
||||
t.Cleanup(ts.Close)
|
||||
|
||||
result := deliverWithQuery(t, httpTargetConfig(
|
||||
result := deliverWithQuery(
|
||||
t, ts.URL+tc.path, tc.forwardQuery,
|
||||
))
|
||||
)
|
||||
|
||||
assert.True(t, result.Success)
|
||||
require.Len(t, received, 1)
|
||||
@@ -144,12 +133,13 @@ func TestDeliverHTTP_ForwardedQueryKeepsTheTargetURLMasked(t *testing.T) {
|
||||
t.Cleanup(ts.Close)
|
||||
|
||||
target := &database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: httpTargetConfig(t, ts.URL+"/in?token="+secret, true),
|
||||
MaxAttempts: 1,
|
||||
Type: database.TargetTypeHTTP,
|
||||
URL: ts.URL + "/in?token=" + secret,
|
||||
ForwardQuery: true,
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
|
||||
deliverWithQuery(t, target.Config)
|
||||
deliverWithQuery(t, target.URL, target.ForwardQuery)
|
||||
require.Len(t, received, 1)
|
||||
|
||||
sent := <-received
|
||||
@@ -164,9 +154,9 @@ func TestDeliverHTTP_ForwardedQueryKeepsTheTargetURLMasked(t *testing.T) {
|
||||
}
|
||||
|
||||
// Nothing listens on port 1.
|
||||
failed := deliverWithQuery(t, httpTargetConfig(
|
||||
failed := deliverWithQuery(
|
||||
t, "http://127.0.0.1:1/in?token="+secret, true,
|
||||
))
|
||||
)
|
||||
require.NotEmpty(t, failed.Error)
|
||||
assert.NotContains(t, failed.Error, secret)
|
||||
assert.NotContains(t, failed.Error, eventQuery)
|
||||
|
||||
@@ -18,7 +18,7 @@ const RedactionMarker = "(redacted)"
|
||||
// line behind the delivery path's own URL masking.
|
||||
//
|
||||
// It removes byte-identical echoes of strings taken from the
|
||||
// target's stored configuration, and nothing else. Anything
|
||||
// target's URL and headers, and nothing else. Anything
|
||||
// the remote re-encodes survives: JSON "\/" escaping (what
|
||||
// PHP's json_encode emits by default), percent-encoding, HTML
|
||||
// entities, and an echo of only part of a path. It cannot
|
||||
@@ -115,7 +115,7 @@ func (r Redactor) secretPrefixSuffix(s string) int {
|
||||
}
|
||||
|
||||
// targetSecrets returns the credential-bearing strings a
|
||||
// target's configuration carries.
|
||||
// target's settings carry.
|
||||
//
|
||||
// The destination URL contributes. Its path, query and
|
||||
// userinfo are the credential for both target types that have
|
||||
@@ -129,36 +129,15 @@ func (r Redactor) secretPrefixSuffix(s string) int {
|
||||
// picks out. That is the same class-based rule applied to
|
||||
// URLs: an echoed Accept or User-Agent still renders, an
|
||||
// echoed Authorization does not.
|
||||
//
|
||||
// A setting a type does not use is empty and contributes
|
||||
// nothing, so a database or log target has nothing to redact.
|
||||
func targetSecrets(t *database.Target) []string {
|
||||
if t == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
switch t.Type {
|
||||
case database.TargetTypeSlack:
|
||||
cfg, err := parseSlackConfig(t.Config)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return urlSecrets(cfg.WebhookURL)
|
||||
case database.TargetTypeHTTP:
|
||||
cfg, err := parseHTTPConfig(t.Config)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return append(
|
||||
urlSecrets(cfg.URL),
|
||||
headerSecrets(cfg.Headers)...,
|
||||
)
|
||||
case database.TargetTypeDatabase, database.TargetTypeLog:
|
||||
// Neither has a destination URL, so neither has
|
||||
// anything to redact.
|
||||
return nil
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
return append(urlSecrets(t.URL), headerSecrets(t.Headers)...)
|
||||
}
|
||||
|
||||
// urlSecrets returns the substrings of a destination URL that
|
||||
|
||||
@@ -23,7 +23,7 @@ func TestRedactor_RemovesSlackWebhookURL(t *testing.T) {
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||
URL: redactWebhookURL,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -54,7 +54,7 @@ func TestRedactor_RemovesSecretSeveredByACut(t *testing.T) {
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||
URL: redactWebhookURL,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -86,18 +86,19 @@ func TestRedactor_RedactsCredentialShapedHeaderValues(
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"https://example.com/in",` +
|
||||
`"headers":{` +
|
||||
`"Authorization":"Bearer AAAAAAAAAAAA",` +
|
||||
`"Cookie":"session=BBBBBBBBBBBB",` +
|
||||
`"X-Api-Key":"CCCCCCCCCCCC",` +
|
||||
`"X-Hub-Signature":"sha256=DDDDDDDDDDDD",` +
|
||||
`"X-Sig":"EEEEEEEEEEEE",` +
|
||||
`"X-Pass":"FFFFFFFFFFFF",` +
|
||||
`"X-HMAC":"GGGGGGGGGGGG",` +
|
||||
`"X-Credential":"HHHHHHHHHHHH",` +
|
||||
`"Accept":"application/json",` +
|
||||
`"User-Agent":"webhooker/1.0"}}`,
|
||||
URL: "https://example.com/in",
|
||||
Headers: map[string]string{
|
||||
headerAuthorization: "Bearer AAAAAAAAAAAA",
|
||||
"Cookie": "session=BBBBBBBBBBBB",
|
||||
probeHeaderName: "CCCCCCCCCCCC",
|
||||
"X-Hub-Signature": "sha256=DDDDDDDDDDDD",
|
||||
"X-Sig": "EEEEEEEEEEEE",
|
||||
"X-Pass": "FFFFFFFFFFFF",
|
||||
"X-HMAC": "GGGGGGGGGGGG",
|
||||
"X-Credential": "HHHHHHHHHHHH",
|
||||
"Accept": "application/json",
|
||||
headerUserAgent: "webhooker/1.0",
|
||||
},
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -136,9 +137,9 @@ func TestRedactor_IgnoresVeryShortHeaderValues(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"https://example.com/in",` +
|
||||
`"headers":{"X-Api-Key":"ab"}}`,
|
||||
Type: database.TargetTypeHTTP,
|
||||
URL: "https://example.com/in",
|
||||
Headers: map[string]string{probeHeaderName: "ab"},
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -155,7 +156,7 @@ func TestRedactor_RemovesBarePath(t *testing.T) {
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||
URL: redactWebhookURL,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -189,7 +190,7 @@ func TestRedactor_RemovesHTTPURLQueryAndUserinfo(t *testing.T) {
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"` + raw + `"}`,
|
||||
URL: raw,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -239,7 +240,7 @@ func TestRedactor_RemovesEchoedQueryOfURLWithoutPath(t *testing.T) {
|
||||
} {
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"` + dest + `"}`,
|
||||
URL: dest,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -261,7 +262,7 @@ func TestRedactor_LeavesUnrelatedTextAlone(t *testing.T) {
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeSlack,
|
||||
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||
URL: redactWebhookURL,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -288,7 +289,7 @@ func TestRedactor_EmptyUserinfoDoesNotShredTheBody(t *testing.T) {
|
||||
|
||||
r := delivery.NewRedactor(&database.Target{
|
||||
Type: database.TargetTypeHTTP,
|
||||
Config: `{"url":"` + dest + `"}`,
|
||||
URL: dest,
|
||||
MaxAttempts: 1,
|
||||
})
|
||||
|
||||
@@ -306,11 +307,10 @@ func TestRedactor_EmptyUserinfoDoesNotShredTheBody(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestRedactor_ZeroValueAndConfiglessTargets pins that a
|
||||
// caller with no target, an unparseable config, or a target
|
||||
// type with no destination URL gets a redactor that changes
|
||||
// nothing rather than one that panics.
|
||||
func TestRedactor_ZeroValueAndConfiglessTargets(t *testing.T) {
|
||||
// TestRedactor_ZeroValueAndURLlessTargets pins that a caller
|
||||
// with no target, or a target with no destination URL, gets a
|
||||
// redactor that changes nothing rather than one that panics.
|
||||
func TestRedactor_ZeroValueAndURLlessTargets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const text = "some response body"
|
||||
@@ -323,8 +323,8 @@ func TestRedactor_ZeroValueAndConfiglessTargets(t *testing.T) {
|
||||
for _, tgt := range []database.Target{
|
||||
{Type: database.TargetTypeLog},
|
||||
{Type: database.TargetTypeDatabase},
|
||||
{Type: database.TargetTypeSlack, Config: "not json", MaxAttempts: 1},
|
||||
{Type: database.TargetTypeHTTP, Config: "", MaxAttempts: 1},
|
||||
{Type: database.TargetTypeSlack, MaxAttempts: 1},
|
||||
{Type: database.TargetTypeHTTP, MaxAttempts: 1},
|
||||
} {
|
||||
assert.Equal(
|
||||
t, text,
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
@@ -15,18 +14,6 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// errMissingWebhookURL is returned when a Slack target config
|
||||
// omits its webhook URL.
|
||||
var errMissingWebhookURL = errors.New(
|
||||
"webhook_url is required",
|
||||
)
|
||||
|
||||
// SlackTargetConfig holds configuration for slack target
|
||||
// types.
|
||||
type SlackTargetConfig struct {
|
||||
WebhookURL string `json:"webhookUrl"`
|
||||
}
|
||||
|
||||
// slackTarget delivers events to Slack incoming webhooks. It
|
||||
// formats the event into a Slack message and posts it as
|
||||
// JSON. It shares the retry core with the HTTP target: a
|
||||
@@ -46,19 +33,6 @@ func (t *slackTarget) Deliver(
|
||||
task *Task,
|
||||
sched Scheduler,
|
||||
) {
|
||||
cfg, err := parseSlackConfig(d.Target.Config)
|
||||
if err != nil {
|
||||
t.eng.log.Error(
|
||||
"invalid Slack target config",
|
||||
"target_id", d.TargetID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
t.failConfig(webhookDB, d, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
msg := FormatSlackMessage(&d.Event)
|
||||
|
||||
payload, err := json.Marshal(
|
||||
@@ -71,13 +45,26 @@ func (t *slackTarget) Deliver(
|
||||
"error", err,
|
||||
)
|
||||
|
||||
t.failConfig(webhookDB, d, err)
|
||||
recErr := t.eng.recordResult(
|
||||
webhookDB, d, 1,
|
||||
false, 0, "", err.Error(), 0,
|
||||
)
|
||||
if recErr != nil {
|
||||
t.eng.bookkeepingFailed(d, recErr)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
t.eng.settleStatus(
|
||||
webhookDB, d, d.Target.Type,
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
attempt := func() attemptResult {
|
||||
return t.attempt(ctx, cfg, payload)
|
||||
return t.attempt(ctx, d.Target.URL, payload)
|
||||
}
|
||||
|
||||
t.deliver(
|
||||
@@ -86,37 +73,13 @@ func (t *slackTarget) Deliver(
|
||||
)
|
||||
}
|
||||
|
||||
// failConfig records a first-attempt failure for a delivery
|
||||
// that could not be prepared (bad config or unmarshalable
|
||||
// payload) and marks it failed.
|
||||
func (t *slackTarget) failConfig(
|
||||
webhookDB *gorm.DB,
|
||||
d *database.Delivery,
|
||||
err error,
|
||||
) {
|
||||
recErr := t.eng.recordResult(
|
||||
webhookDB, d, 1,
|
||||
false, 0, "", err.Error(), 0,
|
||||
)
|
||||
if recErr != nil {
|
||||
t.eng.bookkeepingFailed(d, recErr)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
t.eng.settleStatus(
|
||||
webhookDB, d, d.Target.Type,
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
}
|
||||
|
||||
// attempt performs a single Slack POST and derives its
|
||||
// outcome, preserving the engine's original semantics: a
|
||||
// attempt performs a single Slack POST to targetURL and derives
|
||||
// its outcome, preserving the engine's original semantics: a
|
||||
// non-2xx response records an "HTTP <code>" error string and
|
||||
// a transport error records a "sending request" error.
|
||||
func (t *slackTarget) attempt(
|
||||
ctx context.Context,
|
||||
cfg *SlackTargetConfig,
|
||||
targetURL string,
|
||||
payload []byte,
|
||||
) attemptResult {
|
||||
start := time.Now()
|
||||
@@ -124,7 +87,7 @@ func (t *slackTarget) attempt(
|
||||
req, err := http.NewRequestWithContext(
|
||||
ctx,
|
||||
http.MethodPost,
|
||||
cfg.WebhookURL,
|
||||
targetURL,
|
||||
bytes.NewReader(payload),
|
||||
)
|
||||
if err != nil {
|
||||
@@ -186,31 +149,6 @@ func (t *slackTarget) readSlackResponse(
|
||||
}
|
||||
}
|
||||
|
||||
func parseSlackConfig(
|
||||
configJSON string,
|
||||
) (*SlackTargetConfig, error) {
|
||||
if configJSON == "" {
|
||||
return nil, errEmptyTargetConfig
|
||||
}
|
||||
|
||||
var cfg SlackTargetConfig
|
||||
|
||||
err := json.Unmarshal(
|
||||
[]byte(configJSON), &cfg,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"parsing config JSON: %w", err,
|
||||
)
|
||||
}
|
||||
|
||||
if cfg.WebhookURL == "" {
|
||||
return nil, errMissingWebhookURL
|
||||
}
|
||||
|
||||
return &cfg, nil
|
||||
}
|
||||
|
||||
// FormatSlackMessage builds a Slack-compatible message
|
||||
// string from a webhook event.
|
||||
func FormatSlackMessage(
|
||||
|
||||
@@ -45,7 +45,7 @@ func tSeedDeletedTarget(
|
||||
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, name,
|
||||
database.TargetTypeHTTP, iHTTPConfig(url), 5,
|
||||
database.TargetTypeHTTP, url, 5,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
@@ -122,9 +122,9 @@ func TestProcessDelivery_UnknownTargetType_RecordsWhy(
|
||||
)
|
||||
|
||||
target := database.Target{
|
||||
Name: "mystery",
|
||||
Type: tUnknownType,
|
||||
Config: iHTTPConfig("http://example.com/hook"),
|
||||
Name: "mystery",
|
||||
Type: tUnknownType,
|
||||
URL: "http://example.com/hook",
|
||||
}
|
||||
target.ID = targetID
|
||||
|
||||
@@ -140,7 +140,7 @@ func TestProcessDelivery_UnknownTargetType_RecordsWhy(
|
||||
body := event.Body
|
||||
task := iTask(
|
||||
seeded, event, s.WebhookID, targetID, "mystery",
|
||||
target.Config, 0, 1, &body,
|
||||
target.URL, 0, 1, &body,
|
||||
)
|
||||
task.TargetType = tUnknownType
|
||||
|
||||
@@ -178,7 +178,7 @@ func TestRecoverSingleRetry_TargetDeleted(t *testing.T) {
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID,
|
||||
context.Background(), s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -201,7 +201,7 @@ func TestRecoverSingleRetry_TargetDeleted(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
func TestSweepSingleRetry_TargetDeleted(t *testing.T) {
|
||||
func TestRecoverSingleRetry_OnTick_TargetDeleted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
@@ -218,11 +218,11 @@ func TestSweepSingleRetry_TargetDeleted(t *testing.T) {
|
||||
// Twice, because the bug was an error the sweep repeated every
|
||||
// minute for the life of the database: the second sweep must
|
||||
// find nothing left to do.
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -239,10 +239,11 @@ func TestSweepSingleRetry_TargetDeleted(t *testing.T) {
|
||||
assert.Zero(t, s.Engine.ExportInflightHeld())
|
||||
}
|
||||
|
||||
// TestSweepSingleRetry_TargetNeverExisted covers the other half of the
|
||||
// soft-delete distinction: an id with no row at all, deleted or
|
||||
// otherwise, must not be reported as something the operator deleted.
|
||||
func TestSweepSingleRetry_TargetNeverExisted(t *testing.T) {
|
||||
// TestRecoverSingleRetry_OnTick_TargetNeverExisted covers the other
|
||||
// half of the soft-delete distinction: an id with no row at all,
|
||||
// deleted or otherwise, must not be reported as something the operator
|
||||
// deleted.
|
||||
func TestRecoverSingleRetry_OnTick_TargetNeverExisted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
@@ -264,8 +265,8 @@ func TestSweepSingleRetry_TargetNeverExisted(t *testing.T) {
|
||||
|
||||
iSeedFailedResult(t, s.WebhookDB, d.ID)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -303,8 +304,8 @@ func TestFailMissingTarget_WritesNoTargetRow(
|
||||
database.DeliveryStatusRetrying,
|
||||
)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(
|
||||
context.Background(), s.WebhookID,
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID, onTick,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -345,11 +346,10 @@ func tRetryChainSetup(
|
||||
iCreateWebhook(t, s.MainDB, s.WebhookID, name)
|
||||
|
||||
targetID := uuid.New().String()
|
||||
cfg := iHTTPConfig(ts.URL)
|
||||
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, name,
|
||||
database.TargetTypeHTTP, cfg, 5,
|
||||
database.TargetTypeHTTP, ts.URL, 5,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
@@ -366,7 +366,7 @@ func tRetryChainSetup(
|
||||
body := event.Body
|
||||
|
||||
return iTask(
|
||||
d, event, s.WebhookID, targetID, name, cfg, 5, 2, &body,
|
||||
d, event, s.WebhookID, targetID, name, ts.URL, 5, 2, &body,
|
||||
), targetID
|
||||
}
|
||||
|
||||
@@ -527,7 +527,7 @@ func TestRecoverSingleRetry_TargetUnreadable_LeavesDeliveryAlone(
|
||||
iCreateTarget(
|
||||
t, s.MainDB, targetID, s.WebhookID, "healthy",
|
||||
database.TargetTypeHTTP,
|
||||
iHTTPConfig("http://example.com/hook"), 5,
|
||||
"http://example.com/hook", 5,
|
||||
)
|
||||
|
||||
event := iSeedEvent(
|
||||
@@ -546,7 +546,7 @@ func TestRecoverSingleRetry_TargetUnreadable_LeavesDeliveryAlone(
|
||||
require.NoError(t, sqlDB.Close())
|
||||
|
||||
s.Engine.ExportRecoverRetryingDeliveries(
|
||||
s.WebhookDB, s.WebhookID,
|
||||
context.Background(), s.WebhookDB, s.WebhookID,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -564,7 +564,7 @@ func TestRecoverSingleRetry_TargetUnreadable_LeavesDeliveryAlone(
|
||||
|
||||
// --- 4. A pending delivery whose target is gone ---
|
||||
|
||||
func TestRecoverPending_TargetDeleted(t *testing.T) {
|
||||
func TestRecoverPendingDeliveries_TargetDeleted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
@@ -577,7 +577,7 @@ func TestRecoverPending_TargetDeleted(t *testing.T) {
|
||||
)
|
||||
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID,
|
||||
context.Background(), s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -600,10 +600,10 @@ func TestRecoverPending_TargetDeleted(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestRecoverPending_TargetDeleted_LeavesAnOwnedDeliveryAlone: the
|
||||
// terminal write takes ownership like every other recovery write, so a
|
||||
// delivery the engine still holds is not failed underneath its worker.
|
||||
func TestRecoverPending_TargetDeleted_LeavesAnOwnedDeliveryAlone(
|
||||
// TestRecoverPendingDeliveries_TargetDeleted_LeavesAnOwnedDeliveryAlone:
|
||||
// the terminal write takes ownership like every other recovery write, so
|
||||
// a delivery the engine still holds is not failed underneath its worker.
|
||||
func TestRecoverPendingDeliveries_TargetDeleted_LeavesAnOwnedDeliveryAlone(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
@@ -620,7 +620,7 @@ func TestRecoverPending_TargetDeleted_LeavesAnOwnedDeliveryAlone(
|
||||
require.True(t, s.Engine.ExportRetainDelivery(deliveryID))
|
||||
|
||||
s.Engine.ExportRecoverWebhookDeliveries(
|
||||
context.Background(), s.WebhookID,
|
||||
context.Background(), s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
@@ -675,18 +675,18 @@ func TestFailMissingTarget_LeavesASettledDeliveryAlone(
|
||||
assert.Zero(t, s.Engine.ExportInflightHeld())
|
||||
}
|
||||
|
||||
// TestSweepPending_TargetDeleted sweeps twice over a batch that also
|
||||
// holds a healthy stranded delivery. The one whose target is gone is
|
||||
// failed once and then left alone; the healthy one is queued by the
|
||||
// first sweep and not again by the second.
|
||||
func TestSweepPending_TargetDeleted(t *testing.T) {
|
||||
// TestRecoverPendingDeliveries_OnTick_TargetDeleted sweeps twice over a
|
||||
// batch that also holds a healthy stranded delivery. The one whose
|
||||
// target is gone is failed once and then left alone; the healthy one is
|
||||
// queued by the first sweep and not again by the second.
|
||||
func TestRecoverPendingDeliveries_OnTick_TargetDeleted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
liveTargetID := uuid.New().String()
|
||||
s := fSweepSetup(t, liveTargetID, "still-there")
|
||||
|
||||
deliveryID := tSeedDeletedTarget(
|
||||
t, s, "gone-on-pending-sweep", "http://example.com/hook",
|
||||
t, s, "gone-while-pending-on-tick", "http://example.com/hook",
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
rAgePending(t, s.WebhookDB, deliveryID)
|
||||
@@ -703,7 +703,7 @@ func TestSweepPending_TargetDeleted(t *testing.T) {
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
|
||||
s.Engine.ExportRecoverWebhookDeliveries(ctx, s.WebhookID, onTick)
|
||||
|
||||
tasks := fDrain(s.Engine)
|
||||
require.Len(t, tasks, 1,
|
||||
@@ -711,7 +711,7 @@ func TestSweepPending_TargetDeleted(t *testing.T) {
|
||||
)
|
||||
assert.Equal(t, healthy.ID, tasks[0].DeliveryID)
|
||||
|
||||
s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
|
||||
s.Engine.ExportRecoverWebhookDeliveries(ctx, s.WebhookID, onTick)
|
||||
|
||||
assert.Empty(t, fDrain(s.Engine),
|
||||
"the second sweep queued a delivery again",
|
||||
@@ -724,7 +724,7 @@ func TestSweepPending_TargetDeleted(t *testing.T) {
|
||||
|
||||
last := tLastResult(t, s, deliveryID, 2)
|
||||
|
||||
assert.Contains(t, last.Error, "gone-on-pending-sweep")
|
||||
assert.Contains(t, last.Error, "gone-while-pending-on-tick")
|
||||
assert.Contains(t, last.Error, "was deleted")
|
||||
}
|
||||
|
||||
@@ -775,11 +775,11 @@ func TestSendRecoveredDeliveries_TargetMissingFromMap(
|
||||
)
|
||||
}
|
||||
|
||||
// TestRecoverPending_TargetUnreadable_LeavesDeliveryAlone: a failed
|
||||
// read of the main database is not a deleted target. Restart recovery
|
||||
// holds every pending delivery of the webhook in one batch, so failing
|
||||
// on this would fail all of them.
|
||||
func TestRecoverPending_TargetUnreadable_LeavesDeliveryAlone(
|
||||
// TestRecoverPendingDeliveries_TargetUnreadable_LeavesDeliveryAlone: a
|
||||
// failed read of the main database is not a deleted target. At start the
|
||||
// recovery routine holds every pending delivery of the webhook in one
|
||||
// batch, so failing on this would fail all of them.
|
||||
func TestRecoverPendingDeliveries_TargetUnreadable_LeavesDeliveryAlone(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
@@ -807,7 +807,7 @@ func TestRecoverPending_TargetUnreadable_LeavesDeliveryAlone(
|
||||
require.NoError(t, sqlDB.Close())
|
||||
|
||||
s.Engine.ExportRecoverPendingDeliveries(
|
||||
context.Background(), s.WebhookDB, s.WebhookID,
|
||||
context.Background(), s.WebhookDB, s.WebhookID, atStart,
|
||||
)
|
||||
|
||||
iAssertStatus(
|
||||
|
||||
@@ -2,7 +2,6 @@ package delivery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
@@ -71,13 +70,6 @@ func deliverSlackTo(
|
||||
e := testEngine(t, 1)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
slackCfg, err := json.Marshal(
|
||||
delivery.SlackTargetConfig{
|
||||
WebhookURL: webhookURL,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
event := seedEvent(t, db, `{"test":true}`)
|
||||
|
||||
dlv := seedDelivery(
|
||||
@@ -87,7 +79,7 @@ func deliverSlackTo(
|
||||
|
||||
d := buildSlackDelivery(
|
||||
dlv, event, targetID,
|
||||
"test-slack-mask", string(slackCfg),
|
||||
"test-slack-mask", webhookURL,
|
||||
)
|
||||
|
||||
e.ExportDeliverSlack(context.TODO(), db, d)
|
||||
@@ -160,13 +152,9 @@ func TestDoHTTPRequest_TransportErrorMasksURL(t *testing.T) {
|
||||
|
||||
e := testEngine(t, 1)
|
||||
|
||||
cfg, err := e.ExportParseHTTPConfig(
|
||||
newHTTPTargetConfig(host + maskSecretPath),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
statusCode, _, _, reqErr := e.ExportDoHTTPRequest(
|
||||
context.TODO(), cfg,
|
||||
context.TODO(),
|
||||
&database.Target{URL: host + maskSecretPath},
|
||||
&database.Event{Body: `{"test":true}`},
|
||||
)
|
||||
require.Error(t, reqErr)
|
||||
@@ -188,7 +176,7 @@ func TestDoHTTPRequest_UnparsableURLIsMasked(t *testing.T) {
|
||||
|
||||
statusCode, _, _, reqErr := e.ExportDoHTTPRequest(
|
||||
context.TODO(),
|
||||
&delivery.HTTPTargetConfig{
|
||||
&database.Target{
|
||||
URL: "https://hooks.example.com" + maskSecretPath + "\n",
|
||||
},
|
||||
&database.Event{},
|
||||
|
||||
@@ -2,7 +2,6 @@ package delivery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
@@ -59,9 +58,9 @@ func TestUserAgent_IsTheBuildVersion(t *testing.T) {
|
||||
|
||||
statusCode, _, _, err := e.ExportDoHTTPRequest(
|
||||
context.Background(),
|
||||
&delivery.HTTPTargetConfig{
|
||||
&database.Target{
|
||||
URL: ts.URL,
|
||||
Headers: map[string]string{"User-Agent": "configured/1"},
|
||||
Headers: map[string]string{headerUserAgent: "configured/1"},
|
||||
},
|
||||
&database.Event{Headers: `{"User-Agent":["curl/8"]}`},
|
||||
)
|
||||
@@ -73,18 +72,13 @@ func TestUserAgent_IsTheBuildVersion(t *testing.T) {
|
||||
db := testWebhookDB(t)
|
||||
targetID := uuid.New().String()
|
||||
|
||||
slackCfg, err := json.Marshal(
|
||||
delivery.SlackTargetConfig{WebhookURL: ts.URL},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
event := seedEvent(t, db, `{"action":"test"}`)
|
||||
dlv := seedDelivery(
|
||||
t, db, event.ID, targetID, database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
e.ExportDeliverSlack(context.Background(), db, buildSlackDelivery(
|
||||
dlv, event, targetID, "test-slack", string(slackCfg),
|
||||
dlv, event, targetID, "test-slack", ts.URL,
|
||||
))
|
||||
require.Len(t, userAgents, 1, "the slack target sent no request")
|
||||
assert.Equal(t, want, <-userAgents, "slack target")
|
||||
|
||||
@@ -1,8 +1,14 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// archiveExpiryNever is the archive expiry that keeps archived
|
||||
// events forever. A stored empty expiry means the same.
|
||||
// events forever. It is stored as an ArchiveExpiry of 0, and an
|
||||
// empty expiry on a form means the same.
|
||||
archiveExpiryNever = "never"
|
||||
|
||||
// tmplKeyArchiveExpiryChoices is the template data key for the
|
||||
@@ -56,3 +62,19 @@ func archiveExpiryOptions(expiry string) []archiveChoice {
|
||||
|
||||
return append([]archiveChoice{own}, options...)
|
||||
}
|
||||
|
||||
// formatArchiveExpiry writes a stored ArchiveExpiry the way the
|
||||
// expiry select spells its values, so the target edit form starts on
|
||||
// the stored choice: never for 0, a whole number of hours as "720h",
|
||||
// and anything else as Go writes a duration, which the form accepts
|
||||
// back unchanged.
|
||||
func formatArchiveExpiry(expiry time.Duration) string {
|
||||
switch {
|
||||
case expiry == 0:
|
||||
return archiveExpiryNever
|
||||
case expiry%time.Hour == 0:
|
||||
return strconv.FormatInt(int64(expiry/time.Hour), 10) + "h"
|
||||
default:
|
||||
return expiry.String()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -94,14 +95,18 @@ func TestArchiveExpiryChoices(t *testing.T) {
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
choices := []struct{ value, shown string }{
|
||||
{expiryNever, expiryNever},
|
||||
{"1h", "1 hour"},
|
||||
{"12h", "12 hours"},
|
||||
{"24h", "1 day"},
|
||||
{"720h", "30 days"},
|
||||
{"2160h", "90 days"},
|
||||
{"8760h", "365 days"},
|
||||
choices := []struct {
|
||||
value string
|
||||
stored time.Duration
|
||||
shown string
|
||||
}{
|
||||
{expiryNever, 0, expiryNever},
|
||||
{"1h", time.Hour, "1 hour"},
|
||||
{"12h", 12 * time.Hour, "12 hours"},
|
||||
{"24h", 24 * time.Hour, "1 day"},
|
||||
{"720h", 720 * time.Hour, "30 days"},
|
||||
{"2160h", 2160 * time.Hour, "90 days"},
|
||||
{"8760h", 8760 * time.Hour, "365 days"},
|
||||
}
|
||||
|
||||
for _, choice := range choices {
|
||||
@@ -122,9 +127,7 @@ func TestArchiveExpiryChoices(t *testing.T) {
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.JSONEq(
|
||||
t, `{"expiry":"`+choice.value+`"}`, targets[0].Config,
|
||||
)
|
||||
assert.Equal(t, choice.stored, targets[0].ArchiveExpiry)
|
||||
|
||||
assert.Equal(
|
||||
t, []string{choice.shown},
|
||||
@@ -138,7 +141,7 @@ func TestArchiveExpiryChoices(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestArchiveExpiryEditStartsOnStoredValue checks the edit form of a
|
||||
// database target whose stored expiry is empty, which selects never,
|
||||
// database target whose stored expiry is 0, which selects never,
|
||||
// and of one whose expiry is not one of the choices, which is listed
|
||||
// first as its own selected entry and saved unchanged.
|
||||
func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
|
||||
@@ -148,7 +151,8 @@ func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
|
||||
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
empty := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase, "",
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase,
|
||||
database.Target{},
|
||||
)
|
||||
|
||||
_, selected := expirySelected(t, env, webhook.ID, empty.ID)
|
||||
@@ -157,7 +161,7 @@ func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
|
||||
webhook = seedWebhookWithRetention(t, env.db, 30)
|
||||
unlisted := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase,
|
||||
`{"expiry":"36h"}`,
|
||||
database.Target{ArchiveExpiry: 36 * time.Hour},
|
||||
)
|
||||
|
||||
assert.Equal(t, []string{"36 hours"}, expiryShown(t, env, webhook.ID))
|
||||
@@ -179,7 +183,7 @@ func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
|
||||
|
||||
w := submitTargetEdit(env, webhook.ID, unlisted.ID, form)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
assert.JSONEq(
|
||||
t, `{"expiry":"36h"}`, storedTarget(t, env, unlisted.ID).Config,
|
||||
assert.Equal(
|
||||
t, 36*time.Hour, storedTarget(t, env, unlisted.ID).ArchiveExpiry,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -12,12 +12,16 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// rotationNone is the archive rotation that keeps one file.
|
||||
const rotationNone = "none"
|
||||
// The archive rotations these tests store: none keeps one file.
|
||||
const (
|
||||
rotationNone = "none"
|
||||
rotationDaily = "daily"
|
||||
rotationHourly = "hourly"
|
||||
)
|
||||
|
||||
// rotationShown returns the archive rotations the webhook page's
|
||||
// target list shows.
|
||||
@@ -41,7 +45,9 @@ func TestArchiveRotationChoices(t *testing.T) {
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
for _, rotation := range []string{rotationNone, "monthly", "daily", "hourly"} {
|
||||
for _, rotation := range []string{
|
||||
rotationNone, "monthly", rotationDaily, rotationHourly,
|
||||
} {
|
||||
t.Run(rotation, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -60,10 +66,8 @@ func TestArchiveRotationChoices(t *testing.T) {
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.JSONEq(t,
|
||||
`{"expiry":"720h","rotation":"`+rotation+`"}`,
|
||||
targets[0].Config,
|
||||
)
|
||||
assert.Equal(t, 720*time.Hour, targets[0].ArchiveExpiry)
|
||||
assert.Equal(t, rotation, targets[0].ArchiveRotation)
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{rotation}, rotationShown(t, env, webhook.ID))
|
||||
@@ -78,26 +82,37 @@ func TestArchiveRotationChoices(t *testing.T) {
|
||||
|
||||
w = submitTargetEdit(env, webhook.ID, targets[0].ID, form)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
assert.JSONEq(t,
|
||||
`{"expiry":"720h","rotation":"hourly"}`,
|
||||
storedTarget(t, env, targets[0].ID).Config,
|
||||
)
|
||||
|
||||
stored := storedTarget(t, env, targets[0].ID)
|
||||
assert.Equal(t, 720*time.Hour, stored.ArchiveExpiry)
|
||||
assert.Equal(t, rotationHourly, stored.ArchiveRotation)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestArchiveRotationEditStartsOnNone checks the edit form of a
|
||||
// database target with no rotation stored starts on none.
|
||||
// TestArchiveRotationEditStartsOnNone checks that a database target
|
||||
// added with no rotation is stored with none and shown with none, and
|
||||
// that its edit form starts on none.
|
||||
func TestArchiveRotationEditStartsOnNone(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
target := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase, "",
|
||||
)
|
||||
|
||||
page := targetEditPage(t, env, webhook.ID, target.ID)
|
||||
form := url.Values{}
|
||||
form.Set("name", "archive")
|
||||
form.Set("type", string(database.TargetTypeDatabase))
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost, "/hook/"+webhook.ID+"/targets", form,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.Equal(t, rotationNone, targets[0].ArchiveRotation)
|
||||
|
||||
page := targetEditPage(t, env, webhook.ID, targets[0].ID)
|
||||
assert.Equal(t, []string{rotationNone}, selectedIn(page, "rotation"))
|
||||
assert.Equal(t, []string{rotationNone}, rotationShown(t, env, webhook.ID))
|
||||
}
|
||||
@@ -125,7 +140,7 @@ func TestArchiveRotationRefused(t *testing.T) {
|
||||
|
||||
target := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase,
|
||||
`{"rotation":"daily"}`,
|
||||
database.Target{ArchiveRotation: rotationDaily},
|
||||
)
|
||||
|
||||
form.Del("type")
|
||||
@@ -133,8 +148,8 @@ func TestArchiveRotationRefused(t *testing.T) {
|
||||
w = submitTargetEdit(env, webhook.ID, target.ID, form)
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "Invalid archive rotation")
|
||||
assert.JSONEq(t,
|
||||
`{"rotation":"daily"}`, storedTarget(t, env, target.ID).Config,
|
||||
assert.Equal(
|
||||
t, rotationDaily, storedTarget(t, env, target.ID).ArchiveRotation,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -161,9 +176,8 @@ func TestHandleSourceCreateSubmit_ArchiveRotation(t *testing.T) {
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.JSONEq(t,
|
||||
`{"expiry":"720h","rotation":"daily"}`, targets[0].Config,
|
||||
)
|
||||
assert.Equal(t, 720*time.Hour, targets[0].ArchiveExpiry)
|
||||
assert.Equal(t, rotationDaily, targets[0].ArchiveRotation)
|
||||
}
|
||||
|
||||
// TestArchiveFileView_Rotated describes a daily target's archive files
|
||||
@@ -178,10 +192,10 @@ func TestArchiveFileView_Rotated(t *testing.T) {
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
target := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase,
|
||||
`{"rotation":"daily"}`,
|
||||
database.Target{ArchiveRotation: rotationDaily},
|
||||
)
|
||||
|
||||
path := delivery.ArchivePath(env.dbMgr, &webhook, target)
|
||||
path := archive.Path(env.dbMgr, &webhook, target)
|
||||
stem := strings.TrimSuffix(path, ".db")
|
||||
written := time.Date(2026, 10, 2, 9, 0, 0, 0, time.UTC)
|
||||
|
||||
|
||||
@@ -5,9 +5,7 @@ import (
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// The outcomes of a replay POST, as the notice codes its redirect
|
||||
@@ -46,12 +44,12 @@ const (
|
||||
// HandleDeliveryReplay re-sends a finished delivery's event to its
|
||||
// target.
|
||||
//
|
||||
// A replay never touches the delivery it repeats. It creates a NEW
|
||||
// pending delivery row for the same event and target and hands it to
|
||||
// the delivery engine through the same Notifier the receiver uses, so
|
||||
// the original's status, attempts and timestamps stand as the record
|
||||
// of what actually happened, and the replay is retried, SSRF-guarded
|
||||
// and circuit-broken exactly as a first attempt is.
|
||||
// A replay never touches the delivery it repeats. The delivery engine
|
||||
// creates a NEW pending delivery row for the same event and target and
|
||||
// queues it as it queues the receiver's, so the original's status,
|
||||
// attempts and timestamps stand as the record of what actually
|
||||
// happened, and the replay is retried, SSRF-guarded and circuit-broken
|
||||
// exactly as a first attempt is.
|
||||
//
|
||||
// What is re-sent is the stored EVENT body, never the response the
|
||||
// original delivery received.
|
||||
@@ -145,7 +143,8 @@ func (h *Handlers) loadReplaySource(
|
||||
return &original, true
|
||||
}
|
||||
|
||||
// queueReplay writes the new delivery and hands it to the engine.
|
||||
// queueReplay has the delivery engine create and queue the new
|
||||
// delivery, unless an earlier replay is still in flight.
|
||||
func (h *Handlers) queueReplay(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
@@ -171,18 +170,8 @@ func (h *Handlers) queueReplay(
|
||||
return
|
||||
}
|
||||
|
||||
var event database.Event
|
||||
|
||||
err = webhookDB.
|
||||
First(&event, "id = ?", original.EventID).Error
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to load event for replay", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
task, err := createReplayDelivery(
|
||||
webhookDB, webhook.ID, &event, target,
|
||||
deliveryID, err := h.engine.CreateReplay(
|
||||
webhook.ID, original.EventID, target,
|
||||
)
|
||||
if err != nil {
|
||||
h.serverError(
|
||||
@@ -193,15 +182,14 @@ func (h *Handlers) queueReplay(
|
||||
}
|
||||
|
||||
h.mtr.DeliveryReplayed(target.Type)
|
||||
h.notifier.Notify([]delivery.Task{task})
|
||||
|
||||
h.log.Info(
|
||||
"delivery replay queued",
|
||||
"webhook_id", webhook.ID,
|
||||
"event_id", event.ID,
|
||||
"event_id", original.EventID,
|
||||
"target_id", target.ID,
|
||||
"replayed_delivery_id", original.ID,
|
||||
"delivery_id", task.DeliveryID,
|
||||
"delivery_id", deliveryID,
|
||||
)
|
||||
|
||||
redirectToEventLog(w, r, webhook, replayQueued)
|
||||
@@ -263,72 +251,6 @@ func countInFlightDeliveries(
|
||||
return count, err
|
||||
}
|
||||
|
||||
// createReplayDelivery writes the new pending delivery row, adds it to
|
||||
// its target's totals in the same transaction, and returns the task
|
||||
// that carries it to the delivery engine.
|
||||
//
|
||||
// The row is written with associations omitted, and neither Event nor
|
||||
// Target is populated on it: GORM's SaveBeforeAssociations would
|
||||
// otherwise upsert the whole target row — plaintext config, which for a
|
||||
// Slack target is the credential — into the per-webhook event database.
|
||||
// See https://git.eeqj.de/sneak/webhooker/issues/206.
|
||||
func createReplayDelivery(
|
||||
webhookDB *gorm.DB,
|
||||
webhookID string,
|
||||
event *database.Event,
|
||||
target *database.Target,
|
||||
) (delivery.Task, error) {
|
||||
dlv := &database.Delivery{
|
||||
EventID: event.ID,
|
||||
TargetID: target.ID,
|
||||
Status: database.DeliveryStatusPending,
|
||||
Replay: true,
|
||||
}
|
||||
|
||||
err := webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||
err := tx.Omit(clause.Associations).Create(dlv).Error
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return database.AddTargetTotals(tx, database.TargetTotals{
|
||||
TargetID: dlv.TargetID, Deliveries: 1,
|
||||
})
|
||||
})
|
||||
if err != nil {
|
||||
return delivery.Task{}, err
|
||||
}
|
||||
|
||||
return delivery.Task{
|
||||
DeliveryID: dlv.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: webhookID,
|
||||
EntrypointID: event.EntrypointID,
|
||||
TargetID: target.ID,
|
||||
TargetName: target.Name,
|
||||
TargetType: target.Type,
|
||||
TargetConfig: target.Config,
|
||||
MaxAttempts: target.MaxAttempts,
|
||||
Method: event.Method,
|
||||
RawQuery: event.RawQuery,
|
||||
Headers: event.Headers,
|
||||
ContentType: event.ContentType,
|
||||
Body: replayBody(event.Body),
|
||||
AttemptNum: 1,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// replayBody returns the stored event body for a replay task to carry
|
||||
// inline, or nil when it is large enough that the engine should fetch
|
||||
// it from the per-webhook database instead.
|
||||
func replayBody(body string) *string {
|
||||
if len(body) >= delivery.MaxInlineBodySize {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &body
|
||||
}
|
||||
|
||||
// redirectToEventLog redirects a replay or resubmit back to the event
|
||||
// log it was triggered from, carrying the outcome as its notice. A
|
||||
// Replay form carries the list it was pressed in as show, so a replay
|
||||
|
||||
@@ -12,7 +12,6 @@ import (
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
@@ -23,7 +22,7 @@ const paramDeliveryID = "deliveryID"
|
||||
|
||||
// replayTargetURL is a public destination, so a target configured with
|
||||
// it is one the SSRF guard would accept. Nothing in these tests
|
||||
// dispatches to it: the notifier is recorded, not run.
|
||||
// dispatches to it: the delivery engine is never started.
|
||||
const replayTargetURL = "http://93.184.216.34/hook"
|
||||
|
||||
// replayEventQuery is the query string a seeded event arrived with.
|
||||
@@ -163,14 +162,8 @@ func postReplay(
|
||||
|
||||
// TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal is the
|
||||
// core requirement: replaying a failed delivery succeeds, appends a
|
||||
// new delivery, and leaves the original row and its recorded attempt
|
||||
// exactly as they were.
|
||||
//
|
||||
// It also pins the two things a replay would be wrong to get from the
|
||||
// original: the task carries the target's CURRENT configuration, which
|
||||
// this test changes between the failure and the replay, and it carries
|
||||
// the stored EVENT body rather than anything the failed attempt
|
||||
// received back.
|
||||
// new delivery marked as a replay, and leaves the original row and its
|
||||
// recorded attempt exactly as they were.
|
||||
func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -181,10 +174,9 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -192,7 +184,7 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
event, original := seedFailedDelivery(
|
||||
@@ -204,15 +196,6 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
|
||||
|
||||
before := loadDelivery(t, webhookDB, original.ID)
|
||||
|
||||
// The operator fixes the destination, which is the whole reason
|
||||
// to replay. The replay must use this, not the config the
|
||||
// original delivery ran against.
|
||||
const fixedConfig = `{"url":"http://93.184.216.34/fixed"}`
|
||||
|
||||
require.NoError(t, db.DB().Model(&database.Target{}).
|
||||
Where("id = ?", tgt.ID).
|
||||
Update("config", fixedConfig).Error)
|
||||
|
||||
w := postReplay(t, h, sess, wh.ID, original.ID)
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
@@ -234,14 +217,9 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
|
||||
assert.Equal(
|
||||
t, database.DeliveryStatusPending, replayed.Status,
|
||||
)
|
||||
assert.True(t, replayed.Replay)
|
||||
|
||||
assertDeliveryUntouched(t, webhookDB, before)
|
||||
|
||||
tasks := notif.Tasks()
|
||||
require.Len(t, tasks, 1)
|
||||
assertReplayTask(
|
||||
t, tasks[0], wh.ID, event, tgt, replayed.ID, fixedConfig,
|
||||
)
|
||||
assertNoLeakedTarget(t, webhookDB)
|
||||
}
|
||||
|
||||
@@ -275,45 +253,6 @@ func assertDeliveryUntouched(
|
||||
)
|
||||
}
|
||||
|
||||
// assertReplayTask proves the task handed to the delivery engine is
|
||||
// the one the receiver would build for this event and this target, and
|
||||
// that it carries wantConfig — the target's configuration as it stands
|
||||
// now rather than as the original delivery ran against it.
|
||||
func assertReplayTask(
|
||||
t *testing.T,
|
||||
task delivery.Task,
|
||||
webhookID string,
|
||||
event *database.Event,
|
||||
target *database.Target,
|
||||
wantDeliveryID, wantConfig string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
assert.Equal(t, wantDeliveryID, task.DeliveryID)
|
||||
assert.Equal(t, event.ID, task.EventID)
|
||||
assert.Equal(t, webhookID, task.WebhookID)
|
||||
assert.Equal(t, event.EntrypointID, task.EntrypointID)
|
||||
assert.Equal(t, target.ID, task.TargetID)
|
||||
assert.Equal(t, target.Type, task.TargetType)
|
||||
assert.JSONEq(
|
||||
t, wantConfig, task.TargetConfig,
|
||||
"replay must use the target's current configuration",
|
||||
)
|
||||
assert.Equal(t, event.Method, task.Method)
|
||||
assert.Equal(
|
||||
t, replayEventQuery, task.RawQuery,
|
||||
"replay re-sends the stored query string",
|
||||
)
|
||||
assert.Equal(t, event.Headers, task.Headers)
|
||||
assert.Equal(t, event.ContentType, task.ContentType)
|
||||
assert.Equal(t, 1, task.AttemptNum)
|
||||
require.NotNil(t, task.Body)
|
||||
assert.Equal(
|
||||
t, event.Body, *task.Body,
|
||||
"replay re-sends the stored event body",
|
||||
)
|
||||
}
|
||||
|
||||
// assertNoLeakedTarget proves the per-webhook database holds no
|
||||
// target: an association write would have upserted the whole target,
|
||||
// plaintext config and all, into the event database. See
|
||||
@@ -327,9 +266,49 @@ func assertNoLeakedTarget(t *testing.T, webhookDB *gorm.DB) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestHandleDeliveryReplay_FailedStoreIsAnError proves a replay the
|
||||
// engine cannot store answers with the 500 error page and adds no
|
||||
// delivery.
|
||||
func TestHandleDeliveryReplay_FailedStoreIsAnError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
event, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
|
||||
|
||||
// Without its target totals table the webhook's database refuses
|
||||
// the totals write, which comes after the delivery row.
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t,
|
||||
webhookDB.Migrator().DropTable(&database.TargetTotals{}))
|
||||
|
||||
w := postReplay(t, h, sess, wh.ID, original.ID)
|
||||
|
||||
assert.Equal(t, http.StatusInternalServerError, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "500 Internal Server Error")
|
||||
assert.Len(t, listDeliveries(t, webhookDB, event.ID), 1)
|
||||
}
|
||||
|
||||
// TestHandleDeliveryReplay_RefusesDeletedTarget proves the required
|
||||
// refusal: a target deleted since the delivery ran is reported as
|
||||
// deleted rather than erroring, and nothing is created or queued.
|
||||
// deleted rather than erroring, and nothing is created.
|
||||
func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -338,10 +317,9 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -349,7 +327,7 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
event, original := seedFailedDelivery(
|
||||
@@ -377,10 +355,6 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
|
||||
t, listDeliveries(t, webhookDB, event.ID), 1,
|
||||
"a refused replay must create no delivery",
|
||||
)
|
||||
assert.Empty(
|
||||
t, notif.Tasks(),
|
||||
"a refused replay must queue nothing",
|
||||
)
|
||||
|
||||
// The refusal is specific, which is why the target is looked up
|
||||
// including soft-deleted rows: an id that never named a target
|
||||
@@ -413,10 +387,9 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -424,7 +397,7 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
event, original := seedFailedDelivery(
|
||||
@@ -454,10 +427,6 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
|
||||
t, listDeliveries(t, webhookDB, event.ID), 2,
|
||||
"the refused second replay must add nothing",
|
||||
)
|
||||
assert.Len(
|
||||
t, notif.Tasks(), 1,
|
||||
"only the first replay reaches the delivery engine",
|
||||
)
|
||||
|
||||
// A delivery the engine has not finished is not replayable
|
||||
// either, which is the same rule seen from the other side.
|
||||
@@ -498,7 +467,7 @@ func TestHandleDeliveryReplay_ReturnsToTheListItWasPressedIn(
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
_, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
|
||||
@@ -556,7 +525,7 @@ func TestHandleSourceLogs_RendersReplayControlAndBanner(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
_, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
|
||||
@@ -615,7 +584,7 @@ func TestHandleDeliveryReplay_LabelsTheReplay(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
event, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
|
||||
|
||||
@@ -86,12 +86,12 @@ func seedFailedDeliveryWithResponse(
|
||||
}
|
||||
|
||||
// seedFailureAndRender seeds a failed delivery against a
|
||||
// target of the given type and config, and returns the
|
||||
// target of the given type and URL, and returns the
|
||||
// rendered event log page.
|
||||
func seedFailureAndRender(
|
||||
t *testing.T,
|
||||
targetType database.TargetType,
|
||||
config, responseBody string,
|
||||
targetURL, responseBody string,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
@@ -109,7 +109,7 @@ func seedFailureAndRender(
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, targetType, config,
|
||||
t, db, wh.ID, targetType, database.Target{URL: targetURL},
|
||||
)
|
||||
|
||||
seedFailedDeliveryWithResponse(
|
||||
@@ -129,7 +129,7 @@ func TestHandleSourceLogs_RendersFailedAttempt(t *testing.T) {
|
||||
body := seedFailureAndRender(
|
||||
t,
|
||||
database.TargetTypeHTTP,
|
||||
`{"url":"https://example.com/hook/abc"}`,
|
||||
"https://example.com/hook/abc",
|
||||
"upstream exploded",
|
||||
)
|
||||
|
||||
@@ -167,7 +167,7 @@ func TestHandleSourceLogs_EscapesResponseBody(t *testing.T) {
|
||||
body := seedFailureAndRender(
|
||||
t,
|
||||
database.TargetTypeHTTP,
|
||||
`{"url":"https://example.com/hook/abc"}`,
|
||||
"https://example.com/hook/abc",
|
||||
payload,
|
||||
)
|
||||
|
||||
@@ -189,7 +189,7 @@ func TestHandleSourceLogs_RedactsCredentialEchoedInResponse(
|
||||
body := seedFailureAndRender(
|
||||
t,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
slackWebhookURL,
|
||||
"no_service: "+slackWebhookURL,
|
||||
)
|
||||
|
||||
@@ -228,7 +228,7 @@ func TestHandleSourceLogs_RedactsCredentialEchoedInError(
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
database.Target{URL: slackWebhookURL},
|
||||
)
|
||||
|
||||
dlv := seedFailedDeliveryWithResponse(t, dbMgr, wh.ID, tgt.ID, "")
|
||||
@@ -300,7 +300,7 @@ func TestHandleSourceLogs_RedactsCredentialSeveredByTheEngineCut(
|
||||
body := seedFailureAndRender(
|
||||
t,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
slackWebhookURL,
|
||||
stored,
|
||||
)
|
||||
|
||||
@@ -335,7 +335,7 @@ func TestHandleSourceLogs_RedactsCredentialSeveredBySQLCut(
|
||||
body := seedFailureAndRender(
|
||||
t,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
slackWebhookURL,
|
||||
stored,
|
||||
)
|
||||
|
||||
@@ -373,7 +373,7 @@ func TestHandleSourceLogs_RedactsForSoftDeletedTarget(
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
database.Target{URL: slackWebhookURL},
|
||||
)
|
||||
|
||||
seedFailedDeliveryWithResponse(
|
||||
@@ -413,7 +413,7 @@ func TestHandleSourceLogs_BoundsRenderedAttempts(t *testing.T) {
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeLog, "",
|
||||
t, db, wh.ID, database.TargetTypeLog, database.Target{},
|
||||
)
|
||||
|
||||
dlv := seedFailedDeliveryWithResponse(t, dbMgr, wh.ID, tgt.ID, "")
|
||||
@@ -480,7 +480,7 @@ func TestHandleSourceLogs_BoundsOversizeResponse(t *testing.T) {
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeLog, "",
|
||||
t, db, wh.ID, database.TargetTypeLog, database.Target{},
|
||||
)
|
||||
|
||||
stored := strings.Repeat("A", responseCap*4) + tail
|
||||
|
||||
@@ -13,8 +13,8 @@ import (
|
||||
|
||||
// DeliveryView is the display-safe projection of a delivery
|
||||
// for the event log page. Its target is a TargetView, so the
|
||||
// stored configuration blob — which holds the target's
|
||||
// credential — has no path to the template.
|
||||
// target's URL and headers — which hold the target's
|
||||
// credential — have no path to the template.
|
||||
type DeliveryView struct {
|
||||
ID string
|
||||
Status database.DeliveryStatus
|
||||
@@ -126,7 +126,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
|
||||
// loadTargetMap loads targets into a map of display-safe
|
||||
// views keyed by target ID, each paired with its redactor.
|
||||
// The projection happens here so that no caller can hand a
|
||||
// raw target, configuration blob and all, to a template: the
|
||||
// raw target, URL and headers and all, to a template: the
|
||||
// raw rows do not leave this function.
|
||||
//
|
||||
// The load is Unscoped because deleting a target only soft
|
||||
|
||||
@@ -171,37 +171,29 @@ func loadResubmitSource(
|
||||
return src, true, nil
|
||||
}
|
||||
|
||||
// queueResubmit stores the copy and fans it out to the webhook's
|
||||
// active targets.
|
||||
// queueResubmit has the delivery engine store the copy, with a delivery
|
||||
// for each of the webhook's active targets as they stand now, through
|
||||
// the same StoreEvent the receiver uses. The one difference is that the
|
||||
// copy leaves its entrypoint's last event as it is, since it did not
|
||||
// arrive on that URL.
|
||||
func (h *Handlers) queueResubmit(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
webhook database.Webhook,
|
||||
src resubmitSource,
|
||||
) {
|
||||
// The receiver's own query, run now: an active target created
|
||||
// after the original event arrived is included, and an
|
||||
// inactive one is skipped rather than refused.
|
||||
targets, err := h.loadActiveTargets(webhook.ID)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to query targets", err)
|
||||
|
||||
return
|
||||
event := &database.Event{
|
||||
WebhookID: webhook.ID,
|
||||
EntrypointID: src.EntrypointID,
|
||||
Method: src.Method,
|
||||
RawQuery: src.RawQuery,
|
||||
Headers: src.Headers,
|
||||
ContentType: src.ContentType,
|
||||
Body: string(src.Body),
|
||||
ResubmittedFromID: &src.ID,
|
||||
}
|
||||
|
||||
event, tasks, err := h.createAndFanOut(
|
||||
eventSource{
|
||||
WebhookID: webhook.ID,
|
||||
EntrypointID: src.EntrypointID,
|
||||
Method: src.Method,
|
||||
RawQuery: src.RawQuery,
|
||||
HeadersJSON: src.Headers,
|
||||
ContentType: src.ContentType,
|
||||
Body: src.Body,
|
||||
ResubmittedFromID: &src.ID,
|
||||
},
|
||||
targets,
|
||||
)
|
||||
deliveries, err := h.engine.StoreEvent(event)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to store resubmitted event", err)
|
||||
|
||||
@@ -215,11 +207,11 @@ func (h *Handlers) queueResubmit(
|
||||
"webhook_id", webhook.ID,
|
||||
"event_id", event.ID,
|
||||
"resubmitted_from_id", src.ID,
|
||||
"target_count", len(tasks),
|
||||
"target_count", deliveries,
|
||||
)
|
||||
|
||||
code := resubmitQueued
|
||||
if len(tasks) == 0 {
|
||||
if deliveries == 0 {
|
||||
code = resubmitNoTargets
|
||||
}
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ import (
|
||||
|
||||
// resubmitTargetURL is a public destination, so a target configured
|
||||
// with it is one the SSRF guard would accept. Nothing in these tests
|
||||
// dispatches to it: the notifier is recorded, not run.
|
||||
// dispatches to it: the delivery engine is never started.
|
||||
const resubmitTargetURL = "http://93.184.216.34/hook"
|
||||
|
||||
// resubmitEventHeaders and resubmitEventQuery are the stored header
|
||||
@@ -134,10 +134,9 @@ func TestHandleEventResubmit_DeliversToTargetCreatedAfterTheEvent(
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -151,7 +150,7 @@ func TestHandleEventResubmit_DeliversToTargetCreatedAfterTheEvent(
|
||||
// captured. It has no prior delivery of anything.
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+resubmitTargetURL+`"}`,
|
||||
database.Target{URL: resubmitTargetURL},
|
||||
)
|
||||
|
||||
w := postResubmit(t, h, sess, wh.ID, original.ID)
|
||||
@@ -189,11 +188,6 @@ func TestHandleEventResubmit_DeliversToTargetCreatedAfterTheEvent(
|
||||
t, database.DeliveryStatusPending, deliveries[0].Status,
|
||||
)
|
||||
|
||||
tasks := notif.Tasks()
|
||||
require.Len(t, tasks, 1)
|
||||
assert.Equal(t, deliveries[0].ID, tasks[0].DeliveryID)
|
||||
assertResubmitTask(t, tasks[0], wh.ID, &fresh, tgt)
|
||||
|
||||
assertNoLeakedTarget(t, webhookDB)
|
||||
}
|
||||
|
||||
@@ -225,32 +219,6 @@ func assertEventCopy(
|
||||
)
|
||||
}
|
||||
|
||||
// assertResubmitTask proves the task handed to the delivery engine is
|
||||
// the one the receiver would build for the NEW event and this target.
|
||||
func assertResubmitTask(
|
||||
t *testing.T,
|
||||
task delivery.Task,
|
||||
webhookID string,
|
||||
fresh *database.Event,
|
||||
target *database.Target,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
assert.Equal(t, fresh.ID, task.EventID)
|
||||
assert.Equal(t, webhookID, task.WebhookID)
|
||||
assert.Equal(t, fresh.EntrypointID, task.EntrypointID)
|
||||
assert.Equal(t, target.ID, task.TargetID)
|
||||
assert.Equal(t, target.Type, task.TargetType)
|
||||
assert.Equal(t, fresh.Method, task.Method)
|
||||
assert.Equal(t, fresh.RawQuery, task.RawQuery)
|
||||
assert.Equal(t, fresh.Headers, task.Headers)
|
||||
assert.Equal(t, fresh.ContentType, task.ContentType)
|
||||
assert.Equal(t, 1, task.AttemptNum)
|
||||
|
||||
require.NotNil(t, task.Body)
|
||||
assert.Equal(t, fresh.Body, *task.Body)
|
||||
}
|
||||
|
||||
// TestHandleEventResubmit_IsRepeatable proves the requirement replay
|
||||
// deliberately does not meet: firing the same captured event at a
|
||||
// backend over and over must work, with no in-flight refusal, and each
|
||||
@@ -263,10 +231,9 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -274,7 +241,7 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+resubmitTargetURL+`"}`,
|
||||
database.Target{URL: resubmitTargetURL},
|
||||
)
|
||||
original := seedStoredEvent(t, dbMgr, wh.ID, `{"fire":"again"}`)
|
||||
|
||||
@@ -300,30 +267,18 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
events := listEvents(t, webhookDB)
|
||||
require.Len(t, events, presses+1)
|
||||
require.Len(
|
||||
t, events, presses+1, "each resubmit creates its own event",
|
||||
)
|
||||
|
||||
tasks := notif.Tasks()
|
||||
require.Len(t, tasks, presses)
|
||||
assert.Empty(
|
||||
t, listDeliveries(t, webhookDB, original.ID),
|
||||
"each resubmit delivers its own new event",
|
||||
)
|
||||
|
||||
seen := make(map[string]struct{}, presses)
|
||||
|
||||
for _, task := range tasks {
|
||||
assert.Equal(t, tgt.ID, task.TargetID)
|
||||
assert.NotEqual(
|
||||
t, original.ID, task.EventID,
|
||||
"each resubmit delivers its own new event",
|
||||
)
|
||||
|
||||
_, dup := seen[task.EventID]
|
||||
assert.False(t, dup, "each resubmit creates its own event")
|
||||
|
||||
seen[task.EventID] = struct{}{}
|
||||
|
||||
require.Len(t, listDeliveries(t, webhookDB, task.EventID), 1)
|
||||
}
|
||||
|
||||
// Every copy names the same source event, so twenty presses stay
|
||||
// traceable to the one captured request.
|
||||
// Every copy has its own delivery, and names the same source
|
||||
// event, so twenty presses stay traceable to the one captured
|
||||
// request.
|
||||
for _, e := range events {
|
||||
if e.ID == original.ID {
|
||||
continue
|
||||
@@ -331,6 +286,10 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
|
||||
|
||||
require.NotNil(t, e.ResubmittedFromID)
|
||||
assert.Equal(t, original.ID, *e.ResubmittedFromID)
|
||||
|
||||
deliveries := listDeliveries(t, webhookDB, e.ID)
|
||||
require.Len(t, deliveries, 1)
|
||||
assert.Equal(t, tgt.ID, deliveries[0].TargetID)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -346,10 +305,9 @@ func TestHandleEventResubmit_OversizeBodySurvivesIntact(t *testing.T) {
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -368,7 +326,7 @@ func TestHandleEventResubmit_OversizeBodySurvivesIntact(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+resubmitTargetURL+`"}`,
|
||||
database.Target{URL: resubmitTargetURL},
|
||||
)
|
||||
original := seedStoredEvent(t, dbMgr, wh.ID, stored)
|
||||
|
||||
@@ -385,22 +343,7 @@ func TestHandleEventResubmit_OversizeBodySurvivesIntact(t *testing.T) {
|
||||
t, stored, fresh.Body,
|
||||
"the stored body must be copied byte for byte",
|
||||
)
|
||||
|
||||
tasks := notif.Tasks()
|
||||
require.Len(t, tasks, 1)
|
||||
assert.Nil(
|
||||
t, tasks[0].Body,
|
||||
"a body over the inline limit is fetched from the new "+
|
||||
"event row rather than carried on the task",
|
||||
)
|
||||
|
||||
// The engine's own read of the body, against the new event id:
|
||||
// what it would send is what was stored.
|
||||
var loaded database.Event
|
||||
|
||||
require.NoError(t, webhookDB.Select("body").
|
||||
First(&loaded, "id = ?", tasks[0].EventID).Error)
|
||||
assert.Equal(t, stored, loaded.Body)
|
||||
assert.Len(t, listDeliveries(t, webhookDB, fresh.ID), 1)
|
||||
}
|
||||
|
||||
// TestHandleEventResubmit_SkipsInactiveTarget proves a deactivated
|
||||
@@ -414,10 +357,9 @@ func TestHandleEventResubmit_SkipsInactiveTarget(t *testing.T) {
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -425,11 +367,11 @@ func TestHandleEventResubmit_SkipsInactiveTarget(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
active := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+resubmitTargetURL+`"}`,
|
||||
database.Target{URL: resubmitTargetURL},
|
||||
)
|
||||
off := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+resubmitTargetURL+`/off"}`,
|
||||
database.Target{URL: resubmitTargetURL + "/off"},
|
||||
)
|
||||
|
||||
require.NoError(t, db.DB().Model(&database.Target{}).
|
||||
@@ -448,21 +390,19 @@ func TestHandleEventResubmit_SkipsInactiveTarget(t *testing.T) {
|
||||
"an inactive target is skipped, not an error",
|
||||
)
|
||||
|
||||
tasks := notif.Tasks()
|
||||
require.Len(t, tasks, 1)
|
||||
assert.Equal(t, active.ID, tasks[0].TargetID)
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
fresh := theOtherEvent(t, listEvents(t, webhookDB), original.ID)
|
||||
require.Len(t, listDeliveries(t, webhookDB, fresh.ID), 1)
|
||||
deliveries := listDeliveries(t, webhookDB, fresh.ID)
|
||||
require.Len(t, deliveries, 1)
|
||||
assert.Equal(t, active.ID, deliveries[0].TargetID)
|
||||
}
|
||||
|
||||
// TestHandleEventResubmit_NoActiveTargetsStillStoresEvent proves a
|
||||
// source with nothing to deliver to behaves as the receiver does: the
|
||||
// event is stored, nothing is queued, and the operator is told so
|
||||
// rather than being shown an error.
|
||||
// event is stored with no delivery, and the operator is told so rather
|
||||
// than being shown an error.
|
||||
func TestHandleEventResubmit_NoActiveTargetsStillStoresEvent(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -473,10 +413,9 @@ func TestHandleEventResubmit_NoActiveTargetsStillStoresEvent(
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -493,12 +432,14 @@ func TestHandleEventResubmit_NoActiveTargetsStillStoresEvent(
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
assert.Empty(t, notif.Tasks())
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Len(t, listEvents(t, webhookDB), 2)
|
||||
events := listEvents(t, webhookDB)
|
||||
assert.Len(t, events, 2)
|
||||
|
||||
fresh := theOtherEvent(t, events, original.ID)
|
||||
assert.Empty(t, listDeliveries(t, webhookDB, fresh.ID))
|
||||
}
|
||||
|
||||
// TestHandleEventResubmit_RefusesEventOfAnotherWebhook proves the
|
||||
@@ -514,10 +455,9 @@ func TestHandleEventResubmit_RefusesEventOfAnotherWebhook(
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
notif *recordingNotifier
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, ¬if)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -528,7 +468,7 @@ func TestHandleEventResubmit_RefusesEventOfAnotherWebhook(
|
||||
mine := seedWebhook(t, db)
|
||||
seedConfiguredTarget(
|
||||
t, db, mine.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+resubmitTargetURL+`"}`,
|
||||
database.Target{URL: resubmitTargetURL},
|
||||
)
|
||||
seedStoredEvent(t, dbMgr, mine.ID, `{"mine":true}`)
|
||||
|
||||
@@ -548,10 +488,49 @@ func TestHandleEventResubmit_RefusesEventOfAnotherWebhook(
|
||||
w = postResubmit(t, h, sess, mine.ID, "not-a-uuid")
|
||||
assert.Equal(t, http.StatusNotFound, w.Code)
|
||||
|
||||
assert.Empty(
|
||||
t, notif.Tasks(),
|
||||
"a refused resubmit must queue nothing",
|
||||
for _, webhookID := range []string{theirs.ID, mine.ID} {
|
||||
webhookDB, err := dbMgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Len(
|
||||
t, listEvents(t, webhookDB), 1,
|
||||
"a refused resubmit must store nothing",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleEventResubmit_FailedStoreIsAnError proves a copy the engine
|
||||
// cannot store answers with the 500 error page and leaves no new event.
|
||||
func TestHandleEventResubmit_FailedStoreIsAnError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
original := seedStoredEvent(t, dbMgr, wh.ID, `{"not":"stored"}`)
|
||||
|
||||
// Without its event totals table the webhook's database refuses
|
||||
// the totals write, which comes after the event row.
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t,
|
||||
webhookDB.Migrator().DropTable(&database.EventTotals{}))
|
||||
|
||||
w := postResubmit(t, h, sess, wh.ID, original.ID)
|
||||
|
||||
assert.Equal(t, http.StatusInternalServerError, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "500 Internal Server Error")
|
||||
assert.Len(t, listEvents(t, webhookDB), 1)
|
||||
}
|
||||
|
||||
// TestHandleSourceLogs_ShowsResubmitProvenance proves the event log
|
||||
@@ -575,7 +554,7 @@ func TestHandleSourceLogs_ShowsResubmitProvenance(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+resubmitTargetURL+`"}`,
|
||||
database.Target{URL: resubmitTargetURL},
|
||||
)
|
||||
original := seedStoredEvent(t, dbMgr, wh.ID, `{"trace":"me"}`)
|
||||
|
||||
|
||||
@@ -133,36 +133,28 @@ func (s *Handlers) RenderTemplateForTest(
|
||||
s.renderTemplate(w, r, pageTemplate, data)
|
||||
}
|
||||
|
||||
// BuildSlackTargetConfigForTest exposes
|
||||
// buildSlackTargetConfig for use in the handlers_test package.
|
||||
func (s *Handlers) BuildSlackTargetConfigForTest(
|
||||
// SetSlackTargetSettingsForTest exposes setSlackTargetSettings for use
|
||||
// in the handlers_test package.
|
||||
func (s *Handlers) SetSlackTargetSettingsForTest(
|
||||
ctx context.Context,
|
||||
target *database.Target,
|
||||
targetURL string,
|
||||
) (string, string, error) {
|
||||
return s.buildSlackTargetConfig(ctx, targetURL)
|
||||
) string {
|
||||
return s.setSlackTargetSettings(ctx, target, targetURL)
|
||||
}
|
||||
|
||||
// BuildHTTPTargetConfigForTest exposes buildHTTPTargetConfig
|
||||
// for use in the handlers_test package, taking the form fields
|
||||
// an HTTP target's configuration is built from.
|
||||
func (s *Handlers) BuildHTTPTargetConfigForTest(
|
||||
ctx context.Context,
|
||||
targetURL, headers, timeout string,
|
||||
) (string, string, error) {
|
||||
return s.buildHTTPTargetConfig(ctx, targetFormInput{
|
||||
URL: targetURL,
|
||||
Headers: headers,
|
||||
Timeout: timeout,
|
||||
})
|
||||
}
|
||||
|
||||
// BuildDatabaseTargetConfigForTest exposes
|
||||
// buildDatabaseTargetConfig for use in the handlers_test
|
||||
// package.
|
||||
func BuildDatabaseTargetConfigForTest(
|
||||
// SetDatabaseTargetSettingsForTest exposes setDatabaseTargetSettings
|
||||
// for use in the handlers_test package.
|
||||
func SetDatabaseTargetSettingsForTest(
|
||||
target *database.Target,
|
||||
expiry, rotation string,
|
||||
) (string, string, error) {
|
||||
return buildDatabaseTargetConfig(expiry, rotation)
|
||||
) string {
|
||||
return setDatabaseTargetSettings(target, expiry, rotation)
|
||||
}
|
||||
|
||||
// FormatArchiveExpiryForTest exposes formatArchiveExpiry.
|
||||
func FormatArchiveExpiryForTest(expiry time.Duration) string {
|
||||
return formatArchiveExpiry(expiry)
|
||||
}
|
||||
|
||||
// ArchiveFileViewForTest exposes archiveFileView, which describes a
|
||||
|
||||
@@ -374,7 +374,7 @@ func assertFloodBounded(t *testing.T, label, out string) {
|
||||
// this service's three gorm.Open sites: the main database behind the
|
||||
// two unauthenticated lookups, and the per-webhook database the
|
||||
// WebhookDBManager opens. The third, the archive writer, is pinned in
|
||||
// internal/delivery, where its type lives.
|
||||
// internal/archive, where its type lives.
|
||||
//
|
||||
// What each assertion is worth, since two of the three would pass
|
||||
// against a service that had never been fixed if the capture were set
|
||||
|
||||
@@ -10,11 +10,11 @@ import (
|
||||
"html/template"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
@@ -64,8 +64,8 @@ type HandlersParams struct {
|
||||
Healthcheck *healthcheck.Healthcheck
|
||||
Session *session.Session
|
||||
Middleware *middleware.Middleware
|
||||
Notifier delivery.Notifier
|
||||
Archives delivery.Archives
|
||||
Engine *delivery.Engine
|
||||
Archives archive.Archives
|
||||
CircuitBreakers delivery.CircuitBreakers
|
||||
SSRFGuard *delivery.Guard
|
||||
Metrics *metrics.Set
|
||||
@@ -82,8 +82,8 @@ type Handlers struct {
|
||||
dbMgr *database.WebhookDBManager
|
||||
session *session.Session
|
||||
mw *middleware.Middleware
|
||||
notifier delivery.Notifier
|
||||
archives delivery.Archives
|
||||
engine *delivery.Engine
|
||||
archives archive.Archives
|
||||
breakers delivery.CircuitBreakers
|
||||
mtr *metrics.Set
|
||||
templates map[string]*template.Template
|
||||
@@ -93,17 +93,6 @@ type Handlers struct {
|
||||
// is one delivery will actually attempt.
|
||||
ssrf *delivery.Guard
|
||||
|
||||
// renameMu makes the webhook edit, the target edit and target
|
||||
// creation run one at a time, each held from loading the stored
|
||||
// names through the archive rename, the save and any move back.
|
||||
// Interleaved, one could rename an archive between another's
|
||||
// rename and save, leaving the file named for one edit and the
|
||||
// stored names from the other. An archive download holds it while
|
||||
// it reads the stored names and lists the files they give, and
|
||||
// again for each file while it finds the file under the names
|
||||
// stored then and opens it.
|
||||
renameMu sync.Mutex
|
||||
|
||||
// dummyVerifications counts the equivalent-cost verifications
|
||||
// charged for usernames that do not exist. It exists so a test
|
||||
// can prove that path runs without measuring wall-clock time.
|
||||
@@ -149,7 +138,7 @@ func New(
|
||||
s.dbMgr = params.WebhookDBMgr
|
||||
s.session = params.Session
|
||||
s.mw = params.Middleware
|
||||
s.notifier = params.Notifier
|
||||
s.engine = params.Engine
|
||||
s.archives = params.Archives
|
||||
s.breakers = params.CircuitBreakers
|
||||
s.mtr = params.Metrics
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -16,6 +17,7 @@ import (
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxtest"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
@@ -28,39 +30,14 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// recordingNotifier is a delivery.Notifier that records the tasks it
|
||||
// was handed, so a test can prove a handler queued the delivery it
|
||||
// claims to have queued — and, on the refusal paths, that it queued
|
||||
// nothing.
|
||||
type recordingNotifier struct {
|
||||
mu sync.Mutex
|
||||
tasks []delivery.Task
|
||||
}
|
||||
|
||||
func (n *recordingNotifier) Notify(tasks []delivery.Task) {
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
|
||||
n.tasks = append(n.tasks, tasks...)
|
||||
}
|
||||
|
||||
// Tasks returns a copy of the recorded tasks.
|
||||
func (n *recordingNotifier) Tasks() []delivery.Task {
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
|
||||
out := make([]delivery.Task, len(n.tasks))
|
||||
copy(out, n.tasks)
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// recordingArchives is a delivery.Archives that records what it
|
||||
// recordingArchives is an archive.Archives that records what it
|
||||
// was asked to do, so a test can prove that a deletion or rename
|
||||
// path reached the delivery engine. After FailRenames, every
|
||||
// path reached the archive writers. After FailRenames, every
|
||||
// rename of that target fails with the given error. After
|
||||
// BlockNextRename, the next rename is recorded and then waits.
|
||||
// RenameLock returns a plain mutex, as in archive.Writers.
|
||||
type recordingArchives struct {
|
||||
renameMu sync.Mutex
|
||||
mu sync.Mutex
|
||||
evicted []string
|
||||
evictedTargets []string
|
||||
@@ -73,10 +50,10 @@ type recordingArchives struct {
|
||||
// errInjectedRename is the failure a test hands FailRenames.
|
||||
var errInjectedRename = errors.New("injected rename failure")
|
||||
|
||||
// errNameTaken is what the delivery engine returns when a file
|
||||
// errNameTaken is what the archive writers return when a file
|
||||
// already has an archive's new name, here archive-taken.db.
|
||||
var errNameTaken = fmt.Errorf(
|
||||
"%w: archive-taken.db", delivery.ErrArchiveNameTaken,
|
||||
"%w: archive-taken.db", archive.ErrNameTaken,
|
||||
)
|
||||
|
||||
// archiveRename is one recorded Rename call.
|
||||
@@ -86,6 +63,10 @@ type archiveRename struct {
|
||||
TargetName string
|
||||
}
|
||||
|
||||
func (r *recordingArchives) RenameLock() *sync.Mutex {
|
||||
return &r.renameMu
|
||||
}
|
||||
|
||||
func (r *recordingArchives) EvictWebhook(webhookID string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
@@ -256,16 +237,17 @@ func newTestAppWithConfig(
|
||||
database.NewWebhookDBManager,
|
||||
healthcheck.New,
|
||||
session.New,
|
||||
func() *recordingNotifier {
|
||||
return &recordingNotifier{}
|
||||
},
|
||||
func(n *recordingNotifier) delivery.Notifier {
|
||||
return n
|
||||
archive.NewWriters,
|
||||
// The engine is built on a lifecycle that is never started,
|
||||
// so no worker runs: every delivery the engine creates for a
|
||||
// handler stays pending, where the test can read it.
|
||||
func(p delivery.EngineParams) *delivery.Engine {
|
||||
return delivery.New(fxtest.NewLifecycle(t), p)
|
||||
},
|
||||
func() *recordingArchives {
|
||||
return &recordingArchives{}
|
||||
},
|
||||
func(r *recordingArchives) delivery.Archives {
|
||||
func(r *recordingArchives) archive.Archives {
|
||||
return r
|
||||
},
|
||||
func() *testCircuitBreakers {
|
||||
@@ -347,7 +329,7 @@ func TestHandleIndex_Authenticated(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
func TestBuildSlackTargetConfig_AcceptsPublicURL(t *testing.T) {
|
||||
func TestSetSlackTargetSettings_AcceptsPublicURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var h *handlers.Handlers
|
||||
@@ -357,16 +339,17 @@ func TestBuildSlackTargetConfig_AcceptsPublicURL(t *testing.T) {
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
cfg, errMsg, err := h.BuildSlackTargetConfigForTest(
|
||||
t.Context(), "http://93.184.216.34/services/T00/B00/xxx",
|
||||
)
|
||||
const hookURL = "http://93.184.216.34/services/T00/B00/xxx"
|
||||
|
||||
var target database.Target
|
||||
|
||||
errMsg := h.SetSlackTargetSettingsForTest(t.Context(), &target, hookURL)
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.Contains(t, cfg, "webhookUrl")
|
||||
assert.Equal(t, hookURL, target.URL)
|
||||
}
|
||||
|
||||
func TestBuildSlackTargetConfig_RejectsReservedURL(t *testing.T) {
|
||||
func TestSetSlackTargetSettings_RejectsReservedURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var h *handlers.Handlers
|
||||
@@ -376,13 +359,14 @@ func TestBuildSlackTargetConfig_RejectsReservedURL(t *testing.T) {
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
cfg, errMsg, err := h.BuildSlackTargetConfigForTest(
|
||||
t.Context(), "http://169.254.169.254/latest/meta-data/",
|
||||
var target database.Target
|
||||
|
||||
errMsg := h.SetSlackTargetSettingsForTest(
|
||||
t.Context(), &target, "http://169.254.169.254/latest/meta-data/",
|
||||
)
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, errMsg, "Invalid target URL")
|
||||
assert.Empty(t, cfg)
|
||||
assert.Empty(t, target.URL)
|
||||
}
|
||||
|
||||
func TestRenderTemplate(t *testing.T) {
|
||||
@@ -476,72 +460,100 @@ func TestRenderTemplateMidRenderErrorSendsNoPartialBody(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {
|
||||
func TestSetDatabaseTargetSettings_Valid(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Empty expiry and rotation: the keep-forever, one-file default,
|
||||
// empty config.
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest("", "")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.Empty(t, cfg)
|
||||
cases := []struct {
|
||||
expiry, rotation string
|
||||
wantExpiry time.Duration
|
||||
wantRotation string
|
||||
}{
|
||||
// Empty expiry and rotation: the keep-forever, one-file
|
||||
// default.
|
||||
{"", "", 0, rotationNone},
|
||||
{expiryNever, "", 0, rotationNone},
|
||||
{" 36h ", "", 36 * time.Hour, rotationNone},
|
||||
{"", rotationDaily, 0, rotationDaily},
|
||||
{"90m", rotationHourly, 90 * time.Minute, rotationHourly},
|
||||
}
|
||||
|
||||
// Explicit never is stored as config.
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("never", "")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"expiry":"never"}`, cfg)
|
||||
for _, tc := range cases {
|
||||
var target database.Target
|
||||
|
||||
// A positive duration is stored as config.
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("720h", "")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"expiry":"720h"}`, cfg)
|
||||
errMsg := handlers.SetDatabaseTargetSettingsForTest(
|
||||
&target, tc.expiry, tc.rotation,
|
||||
)
|
||||
|
||||
// A rotation is stored as config, with or without an expiry.
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("", "daily")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"rotation":"daily"}`, cfg)
|
||||
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest(
|
||||
"720h", "hourly",
|
||||
)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"expiry":"720h","rotation":"hourly"}`, cfg)
|
||||
assert.Empty(t, errMsg, tc)
|
||||
assert.Equal(t, tc.wantExpiry, target.ArchiveExpiry, tc)
|
||||
assert.Equal(t, tc.wantRotation, target.ArchiveRotation, tc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
|
||||
// TestSetDatabaseTargetSettings_RejectsBadExpiry pins each refusal's
|
||||
// whole message, as the form shows it.
|
||||
func TestSetDatabaseTargetSettings_RejectsBadExpiry(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
for _, bad := range []string{"nonsense", "7d", "-5h"} {
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest(bad, "")
|
||||
cases := map[string]string{
|
||||
"nonsense": `Invalid archive expiry: expiry must be "never" or ` +
|
||||
`a Go duration such as "720h": time: invalid duration ` +
|
||||
`"nonsense"`,
|
||||
"7d": `Invalid archive expiry: expiry must be "never" or a Go ` +
|
||||
`duration such as "720h": time: unknown unit "d" in ` +
|
||||
`duration "7d"`,
|
||||
"-5h": `Invalid archive expiry: expiry must be a positive ` +
|
||||
`duration or "never": "-5h"`,
|
||||
}
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Contains(
|
||||
t, errMsg, "Invalid archive expiry",
|
||||
"expiry %q should be refused", bad,
|
||||
for bad, want := range cases {
|
||||
var target database.Target
|
||||
|
||||
errMsg := handlers.SetDatabaseTargetSettingsForTest(
|
||||
&target, bad, "",
|
||||
)
|
||||
assert.Empty(t, cfg)
|
||||
|
||||
assert.Equal(t, want, errMsg, "expiry %q", bad)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildDatabaseTargetConfig_RejectsBadRotation(
|
||||
func TestSetDatabaseTargetSettings_RejectsBadRotation(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
for _, bad := range []string{"weekly", "Daily", " none"} {
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest("", bad)
|
||||
var target database.Target
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Contains(
|
||||
t, errMsg, "Invalid archive rotation",
|
||||
"rotation %q should be refused", bad,
|
||||
errMsg := handlers.SetDatabaseTargetSettingsForTest(
|
||||
&target, "", bad,
|
||||
)
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
"Invalid archive rotation: rotation must be none, "+
|
||||
"monthly, daily or hourly: "+strconv.Quote(bad),
|
||||
errMsg,
|
||||
)
|
||||
assert.Empty(t, cfg)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFormatArchiveExpiry covers the expiries the expiry choices do
|
||||
// not: TestArchiveExpiryChoices shows the edit form starting on each
|
||||
// choice. An expiry in whole hours is written in hours, and any other
|
||||
// as Go writes it, which the form accepts back unchanged.
|
||||
func TestFormatArchiveExpiry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := map[time.Duration]string{
|
||||
0: expiryNever,
|
||||
36 * time.Hour: "36h",
|
||||
90 * time.Minute: "1h30m0s",
|
||||
}
|
||||
|
||||
for expiry, want := range cases {
|
||||
assert.Equal(t, want, handlers.FormatArchiveExpiryForTest(expiry))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -54,15 +55,26 @@ func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
// Each value the archive pruning choice submits, after an empty
|
||||
// one that stands for the archive checkbox left off.
|
||||
expiries := []string{
|
||||
"", "never", "1h", "12h", "24h", "720h", "2160h", "8760h",
|
||||
// Each value the archive pruning choice submits and the expiry it
|
||||
// stores, after an empty one that stands for the archive checkbox
|
||||
// left off.
|
||||
expiries := []struct {
|
||||
value string
|
||||
stored time.Duration
|
||||
}{
|
||||
{"", 0},
|
||||
{expiryNever, 0},
|
||||
{"1h", time.Hour},
|
||||
{"12h", 12 * time.Hour},
|
||||
{"24h", 24 * time.Hour},
|
||||
{"720h", 720 * time.Hour},
|
||||
{"2160h", 2160 * time.Hour},
|
||||
{"8760h", 8760 * time.Hour},
|
||||
}
|
||||
|
||||
for _, httpURL := range []string{"", editOriginalURL} {
|
||||
for _, expiry := range expiries {
|
||||
name := "url=" + httpURL + " archive=" + expiry
|
||||
name := "url=" + httpURL + " archive=" + expiry.value
|
||||
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -72,9 +84,9 @@ func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
|
||||
form.Set("http_url", httpURL)
|
||||
form.Set("archive_expiry", "720h")
|
||||
|
||||
if expiry != "" {
|
||||
if expiry.value != "" {
|
||||
form.Set("archive", "on")
|
||||
form.Set("archive_expiry", expiry)
|
||||
form.Set("archive_expiry", expiry.value)
|
||||
}
|
||||
|
||||
w := submitCreateForm(env, form)
|
||||
@@ -96,17 +108,16 @@ func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
|
||||
wantCount++
|
||||
|
||||
assert.Equal(t, "HTTP", byType[database.TargetTypeHTTP].Name)
|
||||
assert.JSONEq(t, `{"url":"`+httpURL+`"}`,
|
||||
byType[database.TargetTypeHTTP].Config)
|
||||
assert.Equal(t, httpURL, byType[database.TargetTypeHTTP].URL)
|
||||
}
|
||||
|
||||
if expiry != "" {
|
||||
if expiry.value != "" {
|
||||
wantCount++
|
||||
|
||||
assert.Equal(t, "Archive",
|
||||
byType[database.TargetTypeDatabase].Name)
|
||||
assert.JSONEq(t, `{"expiry":"`+expiry+`"}`,
|
||||
byType[database.TargetTypeDatabase].Config)
|
||||
assert.Equal(t, expiry.stored,
|
||||
byType[database.TargetTypeDatabase].ArchiveExpiry)
|
||||
}
|
||||
|
||||
assert.Len(t, byType, wantCount)
|
||||
|
||||
@@ -16,8 +16,8 @@ import (
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
@@ -180,7 +180,7 @@ func failSaveOnTable(
|
||||
}
|
||||
|
||||
// archivePathFor returns the archive database path the
|
||||
// delivery engine would use for a database target: beside the
|
||||
// archive writers would use for a database target: beside the
|
||||
// webhook's event database in the data directory.
|
||||
func archivePathFor(
|
||||
t *testing.T,
|
||||
@@ -192,7 +192,7 @@ func archivePathFor(
|
||||
|
||||
return filepath.Join(
|
||||
filepath.Dir(mgr.DBPath(wh.ID)),
|
||||
delivery.ArchiveFileName(wh.Name, tgt.Name, tgt.ID),
|
||||
archive.FileName(wh.Name, tgt.Name, tgt.ID),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -228,9 +228,9 @@ func postRequest(
|
||||
}
|
||||
|
||||
// TestHandleSourceDelete_EvictsArchiveWriter proves that
|
||||
// deleting a webhook reaches the delivery engine and releases
|
||||
// the webhook's archive writers, exercised through the real
|
||||
// deletion handler rather than by calling the engine directly.
|
||||
// deleting a webhook releases the webhook's archive writers,
|
||||
// exercised through the real deletion handler rather than by
|
||||
// calling the archive writers directly.
|
||||
func TestHandleSourceDelete_EvictsArchiveWriter(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -290,7 +290,7 @@ func TestHandleSourceDelete_KeepsArchiveFile(t *testing.T) {
|
||||
wh := seedWebhook(t, db)
|
||||
tgt := seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
|
||||
|
||||
// Place an archive file where the delivery engine would.
|
||||
// Place an archive file where the archive writers would.
|
||||
archivePath := archivePathFor(t, mgr, wh, tgt)
|
||||
require.NoError(
|
||||
t,
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -25,25 +26,24 @@ const (
|
||||
slackSecretPath
|
||||
)
|
||||
|
||||
// seedConfiguredTarget inserts a target with a stored config
|
||||
// blob and returns it.
|
||||
// seedConfiguredTarget inserts an active target of targetType with
|
||||
// the settings (URL to ArchiveRotation) that settings carries, and
|
||||
// returns it.
|
||||
func seedConfiguredTarget(
|
||||
t *testing.T,
|
||||
db *database.Database,
|
||||
webhookID string,
|
||||
targetType database.TargetType,
|
||||
config string,
|
||||
settings database.Target,
|
||||
) *database.Target {
|
||||
t.Helper()
|
||||
|
||||
tgt := &database.Target{
|
||||
WebhookID: webhookID,
|
||||
Name: "t-" + string(targetType),
|
||||
Type: targetType,
|
||||
Active: true,
|
||||
Config: config,
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
tgt := &settings
|
||||
tgt.WebhookID = webhookID
|
||||
tgt.Name = "t-" + string(targetType)
|
||||
tgt.Type = targetType
|
||||
tgt.Active = true
|
||||
tgt.MaxAttempts = 1
|
||||
|
||||
require.NoError(
|
||||
t,
|
||||
@@ -108,89 +108,59 @@ func serveSourceDetailPage(
|
||||
return w
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_MasksSlackWebhookURL is the
|
||||
// load-bearing regression test for the credential leak: the
|
||||
// rendered page must show the Slack target without any of the
|
||||
// secret path segments of its webhook URL.
|
||||
func TestHandleSourceDetail_MasksSlackWebhookURL(t *testing.T) {
|
||||
// TestHandleSourceDetail_MasksTargetURL is the load-bearing
|
||||
// regression test for the credential leak: the rendered page must
|
||||
// show a Slack target, and an http target, without any of the secret
|
||||
// path segments of its URL. An http target's destination is routinely
|
||||
// an incoming-webhook endpoint too, so the same Slack URL serves for
|
||||
// both.
|
||||
func TestHandleSourceDetail_MasksTargetURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
)
|
||||
labels := map[database.TargetType]string{
|
||||
database.TargetTypeSlack: "Webhook URL",
|
||||
database.TargetTypeHTTP: "Destination URL",
|
||||
}
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db)
|
||||
app.RequireStart()
|
||||
for targetType, label := range labels {
|
||||
t.Run(string(targetType), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
)
|
||||
app := newTestApp(t, &h, &sess, &db)
|
||||
app.RequireStart()
|
||||
|
||||
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
assert.NotContains(t, body, slackSecretPath)
|
||||
assert.NotContains(t, body, "T00000000")
|
||||
assert.NotContains(t, body, "B00000000")
|
||||
assert.NotContains(
|
||||
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
|
||||
)
|
||||
assert.NotContains(t, body, "webhookUrl")
|
||||
wh := seedWebhook(t, db)
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID, targetType,
|
||||
database.Target{URL: slackWebhookURL},
|
||||
)
|
||||
|
||||
assert.Contains(t, body, "Webhook URL")
|
||||
assert.Contains(t, body, "https://hooks.slack.com/...")
|
||||
}
|
||||
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
|
||||
// TestHandleSourceDetail_MasksHTTPDestinationURL is the
|
||||
// regression test for the same leak reached through the http
|
||||
// target: its destination is routinely an incoming-webhook
|
||||
// endpoint whose path segments are the credential, so the
|
||||
// rendered page must not contain them.
|
||||
func TestHandleSourceDetail_MasksHTTPDestinationURL(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
assert.NotContains(t, body, slackSecretPath)
|
||||
assert.NotContains(t, body, "T00000000")
|
||||
assert.NotContains(t, body, "B00000000")
|
||||
assert.NotContains(
|
||||
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
|
||||
)
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeHTTP,
|
||||
`{"url":"`+slackWebhookURL+`"}`,
|
||||
)
|
||||
|
||||
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
|
||||
assert.NotContains(t, body, slackSecretPath)
|
||||
assert.NotContains(t, body, "T00000000")
|
||||
assert.NotContains(t, body, "B00000000")
|
||||
assert.NotContains(
|
||||
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
|
||||
)
|
||||
|
||||
assert.Contains(t, body, "Destination URL")
|
||||
assert.Contains(t, body, "https://hooks.slack.com/...")
|
||||
assert.Contains(t, body, label)
|
||||
assert.Contains(t, body, "https://hooks.slack.com/...")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_RendersNamedTargetFields proves the
|
||||
// other target types render labelled fields rather than the
|
||||
// stored blob.
|
||||
// stored values.
|
||||
func TestHandleSourceDetail_RendersNamedTargetFields(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -212,18 +182,24 @@ func TestHandleSourceDetail_RendersNamedTargetFields(
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeHTTP,
|
||||
`{"url":"https://example.com/hook","timeout":30,`+
|
||||
`"headers":{"Authorization":"Bearer sekrit"}}`,
|
||||
database.Target{
|
||||
URL: "https://example.com/hook",
|
||||
TimeoutSeconds: 30,
|
||||
Headers: map[string]string{"X-Api-Key": "sekrit"},
|
||||
},
|
||||
)
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeDatabase,
|
||||
`{"expiry":"720h"}`,
|
||||
database.Target{
|
||||
ArchiveExpiry: 720 * time.Hour,
|
||||
ArchiveRotation: rotationNone,
|
||||
},
|
||||
)
|
||||
seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetType("carrier-pigeon"),
|
||||
`{"beak":"sharp"}`,
|
||||
database.Target{URL: "https://pigeon.example/beak"},
|
||||
)
|
||||
|
||||
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
@@ -242,8 +218,8 @@ func TestHandleSourceDetail_RendersNamedTargetFields(
|
||||
assert.Contains(t, list, "Archive rotation: none")
|
||||
assert.Contains(t, body, `<option value="database">Archive</option>`)
|
||||
|
||||
// An unknown type gets the neutral placeholder, never the
|
||||
// stored blob.
|
||||
// An unknown type gets the neutral placeholder, never its
|
||||
// stored URL.
|
||||
assert.Contains(t, body, "(unavailable)")
|
||||
assert.NotContains(t, body, "beak")
|
||||
}
|
||||
|
||||
@@ -161,7 +161,7 @@ func TestHandleSourceLogs_MasksDeletedTargetConfig(t *testing.T) {
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
database.Target{URL: slackWebhookURL},
|
||||
)
|
||||
|
||||
seedDeliveredEvent(t, dbMgr, wh.ID, tgt.ID)
|
||||
@@ -175,7 +175,6 @@ func TestHandleSourceLogs_MasksDeletedTargetConfig(t *testing.T) {
|
||||
assert.NotContains(
|
||||
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
|
||||
)
|
||||
assert.NotContains(t, body, "webhookUrl")
|
||||
|
||||
// The name is there; only the credential is not.
|
||||
assert.Contains(t, body, tgt.Name+deletedMarker)
|
||||
|
||||
@@ -138,7 +138,7 @@ func TestHandleSourceLogs_MasksSlackWebhookURL(t *testing.T) {
|
||||
tgt := seedConfiguredTarget(
|
||||
t, db, wh.ID,
|
||||
database.TargetTypeSlack,
|
||||
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||
database.Target{URL: slackWebhookURL},
|
||||
)
|
||||
|
||||
seedDeliveredEvent(t, dbMgr, wh.ID, tgt.ID)
|
||||
@@ -151,7 +151,6 @@ func TestHandleSourceLogs_MasksSlackWebhookURL(t *testing.T) {
|
||||
assert.NotContains(
|
||||
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
|
||||
)
|
||||
assert.NotContains(t, body, "webhookUrl")
|
||||
|
||||
// The page still identifies the delivery's target.
|
||||
assert.Contains(t, body, tgt.Name)
|
||||
|
||||
@@ -2,13 +2,13 @@ package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
@@ -27,8 +27,8 @@ func (h *Handlers) HandleTargetCreate() http.HandlerFunc {
|
||||
|
||||
sourceID := chi.URLParam(r, "sourceID")
|
||||
|
||||
h.renameMu.Lock()
|
||||
defer h.renameMu.Unlock()
|
||||
h.archives.RenameLock().Lock()
|
||||
defer h.archives.RenameLock().Unlock()
|
||||
|
||||
var webhook database.Webhook
|
||||
|
||||
@@ -64,20 +64,14 @@ func (h *Handlers) processTargetCreate(
|
||||
) {
|
||||
in := targetFormInputFrom(r)
|
||||
|
||||
target, errMsg, err := h.newTarget(r.Context(), webhook.ID, in)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to encode target config", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
target, errMsg := h.newTarget(r.Context(), webhook.ID, in)
|
||||
if errMsg != "" {
|
||||
h.renderSourceDetail(w, r, webhook, in, errMsg)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
err = h.db.DB().Create(target).Error
|
||||
err := h.db.DB().Create(target).Error
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to create target", err)
|
||||
|
||||
@@ -92,15 +86,13 @@ func (h *Handlers) processTargetCreate(
|
||||
|
||||
// newTarget validates a new target for a webhook and returns the row
|
||||
// to create, or, when it refuses the target, the message the form
|
||||
// shows. An error is the server's fault, not a refusal: the accepted
|
||||
// configuration could not be encoded. Every form that creates a
|
||||
// target goes through here, so they all accept and refuse the same
|
||||
// things.
|
||||
// shows. Every form that creates a target goes through here, so they
|
||||
// all accept and refuse the same things.
|
||||
func (h *Handlers) newTarget(
|
||||
ctx context.Context,
|
||||
webhookID string,
|
||||
in targetFormInput,
|
||||
) (*database.Target, string, error) {
|
||||
) (*database.Target, string) {
|
||||
target := &database.Target{
|
||||
WebhookID: webhookID,
|
||||
Type: in.Type,
|
||||
@@ -108,34 +100,33 @@ func (h *Handlers) newTarget(
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
|
||||
errMsg, err := h.setTargetFromForm(ctx, target, in)
|
||||
if err != nil || errMsg != "" {
|
||||
return nil, errMsg, err
|
||||
errMsg := h.setTargetFromForm(ctx, target, in)
|
||||
if errMsg != "" {
|
||||
return nil, errMsg
|
||||
}
|
||||
|
||||
return target, "", nil
|
||||
return target, ""
|
||||
}
|
||||
|
||||
// setTargetFromForm validates a target form against the target's type
|
||||
// and, when it accepts it, sets the target's name, configuration and
|
||||
// number of delivery attempts from it. It returns the message the form
|
||||
// shows for anything it refuses, an unknown type among them, and then
|
||||
// leaves the target unchanged; an error is the server's fault, as for
|
||||
// newTarget.
|
||||
// and sets the target's name, settings and number of delivery attempts
|
||||
// from it. It returns the message the form shows for anything it
|
||||
// refuses, an unknown type among them; the caller then discards the
|
||||
// target, which may be partly set.
|
||||
// The add target form and the target edit form both go through here,
|
||||
// so the two cannot come to disagree about what a target may be.
|
||||
func (h *Handlers) setTargetFromForm(
|
||||
ctx context.Context,
|
||||
target *database.Target,
|
||||
in targetFormInput,
|
||||
) (string, error) {
|
||||
) string {
|
||||
if in.Name == "" {
|
||||
return "Name is required", nil
|
||||
return "Name is required"
|
||||
}
|
||||
|
||||
configJSON, errMsg, err := h.buildTargetConfig(ctx, target.Type, in)
|
||||
if err != nil || errMsg != "" {
|
||||
return errMsg, err
|
||||
errMsg := h.setTargetSettings(ctx, target, in)
|
||||
if errMsg != "" {
|
||||
return errMsg
|
||||
}
|
||||
|
||||
// An empty max_attempts keeps the target's count: the default of
|
||||
@@ -146,14 +137,13 @@ func (h *Handlers) setTargetFromForm(
|
||||
// target is delivering with.
|
||||
maxAttempts, err := parseMaxAttempts(in.MaxAttempts, target.MaxAttempts)
|
||||
if err != nil {
|
||||
return "Invalid delivery attempts: " + attemptsErrorMessage(err), nil
|
||||
return "Invalid delivery attempts: " + attemptsErrorMessage(err)
|
||||
}
|
||||
|
||||
target.Name = in.Name
|
||||
target.Config = configJSON
|
||||
target.MaxAttempts = maxAttempts
|
||||
|
||||
return "", nil
|
||||
return ""
|
||||
}
|
||||
|
||||
// targetFormInput carries the raw values of a target form. Both the
|
||||
@@ -212,84 +202,81 @@ func targetFormInputFrom(r *http.Request) targetFormInput {
|
||||
}
|
||||
}
|
||||
|
||||
// buildTargetConfig builds the JSON config string for a target from
|
||||
// the submitted form values, or returns the message the form shows
|
||||
// for a value it refuses. An error is the server's fault, not a
|
||||
// refusal: the accepted configuration could not be encoded. Which
|
||||
// fields of in apply depends on the target type; a type without a URL
|
||||
// ignores any URL submitted.
|
||||
func (h *Handlers) buildTargetConfig(
|
||||
// setTargetSettings sets a target's settings from the submitted form
|
||||
// values, or returns the message the form shows for a value it
|
||||
// refuses. Which fields of in apply depends on the target type; a
|
||||
// type without a URL ignores any URL submitted, and the settings a
|
||||
// type does not use are left empty.
|
||||
func (h *Handlers) setTargetSettings(
|
||||
ctx context.Context,
|
||||
targetType database.TargetType,
|
||||
target *database.Target,
|
||||
in targetFormInput,
|
||||
) (string, string, error) {
|
||||
switch targetType {
|
||||
) string {
|
||||
switch target.Type {
|
||||
case database.TargetTypeHTTP:
|
||||
return h.buildHTTPTargetConfig(ctx, in)
|
||||
return h.setHTTPTargetSettings(ctx, target, in)
|
||||
case database.TargetTypeSlack:
|
||||
return h.buildSlackTargetConfig(ctx, in.URL)
|
||||
return h.setSlackTargetSettings(ctx, target, in.URL)
|
||||
case database.TargetTypeDatabase:
|
||||
return buildDatabaseTargetConfig(in.Expiry, in.Rotation)
|
||||
return setDatabaseTargetSettings(target, in.Expiry, in.Rotation)
|
||||
case database.TargetTypeLog:
|
||||
return "", "", nil
|
||||
return ""
|
||||
default:
|
||||
return "", "Invalid target type", nil
|
||||
return "Invalid target type"
|
||||
}
|
||||
}
|
||||
|
||||
// buildHTTPTargetConfig builds config JSON for an HTTP target: an
|
||||
// SSRF-validated destination plus the optional headers, timeout and
|
||||
// query string setting the delivery path honours.
|
||||
func (h *Handlers) buildHTTPTargetConfig(
|
||||
// setHTTPTargetSettings sets an HTTP target's SSRF-validated
|
||||
// destination plus the optional headers, timeout and query string
|
||||
// setting the delivery path honours.
|
||||
func (h *Handlers) setHTTPTargetSettings(
|
||||
ctx context.Context,
|
||||
target *database.Target,
|
||||
in targetFormInput,
|
||||
) (string, string, error) {
|
||||
) string {
|
||||
errMsg := h.validateTargetURL(
|
||||
ctx, in.URL, "URL is required for HTTP targets",
|
||||
)
|
||||
if errMsg != "" {
|
||||
return "", errMsg, nil
|
||||
return errMsg
|
||||
}
|
||||
|
||||
headers, err := delivery.ParseTargetHeaders(in.Headers)
|
||||
if err != nil {
|
||||
return "", fmt.Sprintf("Invalid headers: %v", err), nil
|
||||
return fmt.Sprintf("Invalid headers: %v", err)
|
||||
}
|
||||
|
||||
timeout, err := delivery.ParseTargetTimeout(in.Timeout)
|
||||
if err != nil {
|
||||
return "", fmt.Sprintf("Invalid timeout: %v", err), nil
|
||||
return fmt.Sprintf("Invalid timeout: %v", err)
|
||||
}
|
||||
|
||||
configJSON, err := marshalTargetConfig(delivery.HTTPTargetConfig{
|
||||
URL: in.URL,
|
||||
Headers: headers,
|
||||
Timeout: timeout,
|
||||
ForwardQuery: in.ForwardQuery,
|
||||
})
|
||||
target.URL = in.URL
|
||||
target.Headers = headers
|
||||
target.TimeoutSeconds = timeout
|
||||
target.ForwardQuery = in.ForwardQuery
|
||||
|
||||
return configJSON, "", err
|
||||
return ""
|
||||
}
|
||||
|
||||
// buildSlackTargetConfig builds config JSON for a Slack target,
|
||||
// whose whole configuration is one SSRF-validated webhook URL.
|
||||
func (h *Handlers) buildSlackTargetConfig(
|
||||
// setSlackTargetSettings sets a Slack target's one setting, its
|
||||
// SSRF-validated webhook URL.
|
||||
func (h *Handlers) setSlackTargetSettings(
|
||||
ctx context.Context,
|
||||
target *database.Target,
|
||||
targetURL string,
|
||||
) (string, string, error) {
|
||||
) string {
|
||||
errMsg := h.validateTargetURL(
|
||||
ctx, targetURL,
|
||||
"Webhook URL is required for Slack targets",
|
||||
)
|
||||
if errMsg != "" {
|
||||
return "", errMsg, nil
|
||||
return errMsg
|
||||
}
|
||||
|
||||
configJSON, err := marshalTargetConfig(delivery.SlackTargetConfig{
|
||||
WebhookURL: targetURL,
|
||||
})
|
||||
target.URL = targetURL
|
||||
|
||||
return configJSON, "", err
|
||||
return ""
|
||||
}
|
||||
|
||||
// validateTargetURL refuses an empty or SSRF-blocked destination,
|
||||
@@ -340,52 +327,31 @@ func (h *Handlers) validateTargetURL(
|
||||
return ""
|
||||
}
|
||||
|
||||
// marshalTargetConfig serialises a target configuration for storage.
|
||||
func marshalTargetConfig(cfg any) (string, error) {
|
||||
configBytes, err := json.Marshal(cfg)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return string(configBytes), nil
|
||||
}
|
||||
|
||||
// buildDatabaseTargetConfig builds config JSON for a database
|
||||
// (archive) target. The optional expiry and rotation are validated
|
||||
// here, at creation time, so a bad value is refused instead of
|
||||
// failing every subsequent delivery. Each is stored only when set,
|
||||
// and with neither the config is empty (the keep-forever, one-file
|
||||
// default).
|
||||
func buildDatabaseTargetConfig(
|
||||
// setDatabaseTargetSettings sets a database (archive) target's
|
||||
// expiry and rotation. Both are optional: an empty or never expiry
|
||||
// keeps the archive forever, and an empty rotation is stored as none,
|
||||
// one file. A bad value is refused here, at the form, instead of
|
||||
// being stored.
|
||||
func setDatabaseTargetSettings(
|
||||
target *database.Target,
|
||||
expiry, rotation string,
|
||||
) (string, string, error) {
|
||||
expiry = strings.TrimSpace(expiry)
|
||||
|
||||
err := delivery.ValidateArchiveExpiry(expiry)
|
||||
) string {
|
||||
archiveExpiry, err := archive.ParseExpiry(strings.TrimSpace(expiry))
|
||||
if err != nil {
|
||||
return "", fmt.Sprintf("Invalid archive expiry: %v", err), nil
|
||||
return fmt.Sprintf("Invalid archive expiry: %v", err)
|
||||
}
|
||||
|
||||
err = delivery.ValidateArchiveRotation(rotation)
|
||||
err = archive.ValidateRotation(rotation)
|
||||
if err != nil {
|
||||
return "", fmt.Sprintf("Invalid archive rotation: %v", err), nil
|
||||
return fmt.Sprintf("Invalid archive rotation: %v", err)
|
||||
}
|
||||
|
||||
cfg := map[string]any{}
|
||||
|
||||
if expiry != "" {
|
||||
cfg["expiry"] = expiry
|
||||
if rotation == "" {
|
||||
rotation = archiveRotationNone
|
||||
}
|
||||
|
||||
if rotation != "" {
|
||||
cfg["rotation"] = rotation
|
||||
}
|
||||
target.ArchiveExpiry = archiveExpiry
|
||||
target.ArchiveRotation = rotation
|
||||
|
||||
if len(cfg) == 0 {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
configJSON, err := marshalTargetConfig(cfg)
|
||||
|
||||
return configJSON, "", err
|
||||
return ""
|
||||
}
|
||||
|
||||
@@ -176,7 +176,7 @@ func TestHandleTargetCreate_BodyURLStillCreatesTheTarget(t *testing.T) {
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.Contains(t, targets[0].Config, targetSecretSegments)
|
||||
assert.Equal(t, targetSecretURL, targets[0].URL)
|
||||
|
||||
// The body carried the credential, so the access log must still
|
||||
// not have it: the log records the request line only.
|
||||
|
||||
@@ -6,73 +6,114 @@ import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// TestHandleTargetCreate_EveryType adds a target of each type. Each
|
||||
// submission carries a url: only the http and slack types store one.
|
||||
// TestHandleTargetCreate_EveryType adds a target of each type and
|
||||
// checks the settings it stores. Each submission carries a url and an
|
||||
// expiry: only the http and slack types store the url, only the
|
||||
// database type the expiry, and every setting a type does not use
|
||||
// stays empty.
|
||||
func TestHandleTargetCreate_EveryType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
// fields is the rest of each submission, as a query string.
|
||||
// fields is the rest of each submission, as a query string, and
|
||||
// want holds the settings stored.
|
||||
cases := []struct {
|
||||
targetType database.TargetType
|
||||
fields string
|
||||
wantConfig string
|
||||
want database.Target
|
||||
wantAttempts int
|
||||
}{
|
||||
{
|
||||
database.TargetTypeHTTP, "timeout=12&max_attempts=3",
|
||||
`{"url":"` + editOriginalURL + `","timeout":12}`, 3,
|
||||
database.TargetTypeHTTP,
|
||||
"timeout=12&max_attempts=3&forward_query=on",
|
||||
database.Target{
|
||||
URL: editOriginalURL,
|
||||
Headers: map[string]string{},
|
||||
TimeoutSeconds: 12,
|
||||
ForwardQuery: true,
|
||||
}, 3,
|
||||
},
|
||||
{
|
||||
database.TargetTypeSlack, "max_attempts=4",
|
||||
`{"webhookUrl":"` + editOriginalURL + `"}`, 4,
|
||||
database.Target{URL: editOriginalURL}, 4,
|
||||
},
|
||||
{
|
||||
database.TargetTypeDatabase, "expiry=720h",
|
||||
`{"expiry":"720h"}`, 1,
|
||||
database.TargetTypeDatabase, "rotation=daily",
|
||||
database.Target{
|
||||
ArchiveExpiry: 720 * time.Hour,
|
||||
ArchiveRotation: rotationDaily,
|
||||
}, 1,
|
||||
},
|
||||
{database.TargetTypeLog, "", "", 1},
|
||||
{database.TargetTypeLog, "", database.Target{}, 1},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(string(tc.targetType), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
|
||||
form, err := url.ParseQuery(tc.fields)
|
||||
require.NoError(t, err)
|
||||
form.Set("name", "every-type")
|
||||
form.Set("type", string(tc.targetType))
|
||||
form.Set("url", editOriginalURL)
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost,
|
||||
"/hook/"+webhook.ID+"/targets", form,
|
||||
_, got := addTargetThroughForm(
|
||||
t, env, tc.targetType,
|
||||
"url="+editOriginalURL+"&expiry=720h&"+tc.fields,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.Equal(t, tc.targetType, targets[0].Type)
|
||||
assert.Equal(t, tc.wantAttempts, targets[0].MaxAttempts)
|
||||
|
||||
if tc.wantConfig == "" {
|
||||
assert.Empty(t, targets[0].Config)
|
||||
} else {
|
||||
assert.JSONEq(t, tc.wantConfig, targets[0].Config)
|
||||
}
|
||||
assert.Equal(t, tc.targetType, got.Type)
|
||||
assert.Equal(t, tc.wantAttempts, got.MaxAttempts)
|
||||
assert.Equal(t, tc.want, targetSettingsOf(got))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// addTargetThroughForm adds a target of targetType named every-type to
|
||||
// a new webhook through the add target form, with the rest of the form
|
||||
// given as a query string, and returns the webhook and the target the
|
||||
// form stored.
|
||||
func addTargetThroughForm(
|
||||
t *testing.T,
|
||||
env *sourceTestEnv,
|
||||
targetType database.TargetType,
|
||||
fields string,
|
||||
) (database.Webhook, database.Target) {
|
||||
t.Helper()
|
||||
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
|
||||
form, err := url.ParseQuery(fields)
|
||||
require.NoError(t, err)
|
||||
form.Set("name", "every-type")
|
||||
form.Set("type", string(targetType))
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost, "/hook/"+webhook.ID+"/targets", form,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
|
||||
return webhook, targets[0]
|
||||
}
|
||||
|
||||
// targetSettingsOf returns a target's settings, URL to
|
||||
// ArchiveRotation, and nothing else, for comparing with the settings a
|
||||
// test expects.
|
||||
func targetSettingsOf(target database.Target) database.Target {
|
||||
return database.Target{
|
||||
URL: target.URL,
|
||||
Headers: target.Headers,
|
||||
TimeoutSeconds: target.TimeoutSeconds,
|
||||
ForwardQuery: target.ForwardQuery,
|
||||
ArchiveExpiry: target.ArchiveExpiry,
|
||||
ArchiveRotation: target.ArchiveRotation,
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleTargetCreate_RefusedFormComesBack refuses a target of each
|
||||
// type and checks that the webhook page comes back with the add target
|
||||
// form open on that type, the values entered, and the reason.
|
||||
|
||||
@@ -6,8 +6,8 @@ import (
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// downloadWriteTimeout is how long one write of a download may wait
|
||||
@@ -16,7 +16,7 @@ const downloadWriteTimeout = 60 * time.Second
|
||||
|
||||
// HandleTargetDownload serves a database target's archive as one
|
||||
// gzipped JSON file, named for the webhook, the target and the time;
|
||||
// see delivery.ArchiveExport.WriteGzipJSON for what it holds. Other
|
||||
// see archive.Export.WriteGzipJSON for what it holds. Other
|
||||
// target types have no archive and are a 404.
|
||||
//
|
||||
// A download runs for as long as the client keeps reading: it reads
|
||||
@@ -37,7 +37,7 @@ func (h *Handlers) HandleTargetDownload() http.HandlerFunc {
|
||||
w.Header().Set("Content-Type", "application/gzip")
|
||||
w.Header().Set(
|
||||
"Content-Disposition",
|
||||
`attachment; filename="`+delivery.ArchiveExportFileName(
|
||||
`attachment; filename="`+archive.ExportFileName(
|
||||
webhook.Name, target.Name, now,
|
||||
)+`"`,
|
||||
)
|
||||
@@ -82,17 +82,17 @@ func (d downloadWriter) Write(b []byte) (int, error) {
|
||||
// listTargetArchive lists the archive files of the request's database
|
||||
// target for export. It reports false once it has written the response.
|
||||
//
|
||||
// It holds renameMu, which every archive rename runs under, while it
|
||||
// reads the stored names and lists the files, so the files it lists
|
||||
// are the ones those names give. It lets go before the export is
|
||||
// streamed, which takes renameMu again for each file only while it
|
||||
// It holds the rename lock, which every archive rename runs under,
|
||||
// while it reads the stored names and lists the files, so the files it
|
||||
// lists are the ones those names give. It lets go before the export is
|
||||
// streamed, which takes the lock again for each file only while it
|
||||
// finds the file under the names stored then and opens it.
|
||||
func (h *Handlers) listTargetArchive(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
) (database.Webhook, *database.Target, *delivery.ArchiveExport, bool) {
|
||||
h.renameMu.Lock()
|
||||
defer h.renameMu.Unlock()
|
||||
) (database.Webhook, *database.Target, *archive.Export, bool) {
|
||||
h.archives.RenameLock().Lock()
|
||||
defer h.archives.RenameLock().Unlock()
|
||||
|
||||
webhook, target, ok := h.ownedTarget(w, r)
|
||||
if !ok {
|
||||
@@ -105,9 +105,9 @@ func (h *Handlers) listTargetArchive(
|
||||
return database.Webhook{}, nil, nil, false
|
||||
}
|
||||
|
||||
export, err := delivery.NewArchiveExport(
|
||||
delivery.ArchivePath(h.dbMgr, &webhook, target),
|
||||
&h.renameMu,
|
||||
export, err := archive.NewExport(
|
||||
archive.Path(h.dbMgr, &webhook, target),
|
||||
h.archives.RenameLock(),
|
||||
func() (string, error) {
|
||||
return h.storedArchivePath(webhook.ID, target.ID)
|
||||
},
|
||||
@@ -122,9 +122,9 @@ func (h *Handlers) listTargetArchive(
|
||||
return webhook, target, export, true
|
||||
}
|
||||
|
||||
// storedArchivePath returns the path delivery.ArchivePath gives a
|
||||
// storedArchivePath returns the path archive.Path gives a
|
||||
// database target under the names stored for it and its webhook now.
|
||||
// Its caller holds renameMu. A webhook or target deleted since is still
|
||||
// Its caller holds the rename lock. A webhook or target deleted since is still
|
||||
// found, since deleting one leaves its archive files under their names.
|
||||
func (h *Handlers) storedArchivePath(
|
||||
webhookID, targetID string,
|
||||
@@ -143,5 +143,5 @@ func (h *Handlers) storedArchivePath(
|
||||
return "", err
|
||||
}
|
||||
|
||||
return delivery.ArchivePath(h.dbMgr, &webhook, &target), nil
|
||||
return archive.Path(h.dbMgr, &webhook, &target), nil
|
||||
}
|
||||
|
||||
@@ -21,9 +21,9 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||
)
|
||||
|
||||
@@ -223,8 +223,8 @@ func TestHandleTargetDownload_FindsFilesAfterRename(t *testing.T) {
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
oldPath := delivery.ArchivePath(env.dbMgr, &wh, archive)
|
||||
target := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
oldPath := archive.Path(env.dbMgr, &wh, target)
|
||||
|
||||
month := func(path string) string {
|
||||
return strings.TrimSuffix(path, ".db") + "-2026-10.db"
|
||||
@@ -233,19 +233,19 @@ func TestHandleTargetDownload_FindsFilesAfterRename(t *testing.T) {
|
||||
seedArchive(t, oldPath, 1, 16)
|
||||
seedArchive(t, month(oldPath), 1, 16)
|
||||
|
||||
sw, downloaded := startStalledDownload(t, env, wh.ID, archive.ID)
|
||||
sw, downloaded := startStalledDownload(t, env, wh.ID, target.ID)
|
||||
|
||||
require.Equal(t,
|
||||
http.StatusSeeOther, renameTarget(env, wh.ID, archive.ID).Code,
|
||||
http.StatusSeeOther, renameTarget(env, wh.ID, target.ID).Code,
|
||||
)
|
||||
|
||||
// The test's archives record a rename without moving any file, so
|
||||
// the files are moved here, as the delivery engine moves them.
|
||||
// the files are moved here, as the archive writers move them.
|
||||
var renamed database.Target
|
||||
|
||||
require.NoError(t, env.db.DB().First(&renamed, "id = ?", archive.ID).Error)
|
||||
require.NoError(t, env.db.DB().First(&renamed, "id = ?", target.ID).Error)
|
||||
|
||||
newPath := delivery.ArchivePath(env.dbMgr, &wh, &renamed)
|
||||
newPath := archive.Path(env.dbMgr, &wh, &renamed)
|
||||
|
||||
require.NoError(t, os.Rename(oldPath, newPath))
|
||||
require.NoError(t, os.Rename(month(oldPath), month(newPath)))
|
||||
@@ -355,16 +355,16 @@ func TestHandleTargetDownload_OutlastsTheRequestLimit(t *testing.T) {
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
target := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
seedArchive(
|
||||
t, delivery.ArchivePath(env.dbMgr, &wh, archive), rows, bodySize,
|
||||
t, archive.Path(env.dbMgr, &wh, target), rows, bodySize,
|
||||
)
|
||||
|
||||
srv, accessLog := limitedServer(t, env, limit)
|
||||
|
||||
req, err := http.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet,
|
||||
srv.URL+downloadPath(wh.ID, archive.ID), nil,
|
||||
srv.URL+downloadPath(wh.ID, target.ID), nil,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"strconv"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
@@ -24,22 +25,15 @@ const (
|
||||
tmplKeyMaxTimeout = "MaxTimeout"
|
||||
)
|
||||
|
||||
// configUnreadableMessage is shown when a target's stored
|
||||
// configuration does not parse. It says plainly that saving replaces
|
||||
// the stored value rather than preserving it, because the form
|
||||
// cannot pre-fill what it could not read.
|
||||
const configUnreadableMessage = "The stored configuration for this " +
|
||||
"target could not be read. Enter the values below; saving " +
|
||||
"replaces the stored configuration."
|
||||
|
||||
// targetEditView is the display model for the target edit page: the
|
||||
// target's row fields as stored. The values the form shows, the
|
||||
// UNMASKED configuration among them, come separately, as a
|
||||
// UNMASKED URL and headers among them, come separately, as a
|
||||
// targetFormInput.
|
||||
//
|
||||
// It deliberately omits database.Target's raw Config blob: the form
|
||||
// renders named fields, and giving the template the blob as well
|
||||
// would put an unreviewed second path to the credential on the page.
|
||||
// It deliberately omits database.Target's settings: the form renders
|
||||
// them from the targetFormInput, and giving the template the target's
|
||||
// URL and headers as well would put an unreviewed second path to the
|
||||
// credential on the page.
|
||||
type targetEditView struct {
|
||||
ID string
|
||||
Name string
|
||||
@@ -47,13 +41,16 @@ type targetEditView struct {
|
||||
Active bool
|
||||
}
|
||||
|
||||
// HandleTargetEdit shows the form to edit a target.
|
||||
// HandleTargetEdit shows the form to edit a target, filled with the
|
||||
// target's settings as stored.
|
||||
//
|
||||
// This page is the one place the full destination URL and header
|
||||
// values are shown. It is reachable only through the
|
||||
// /hook/{sourceID} route group, which supplies RequireAuth and
|
||||
// NoCache, and only for a target of a webhook the session's user
|
||||
// owns; masking (delivery.TargetView) is unchanged everywhere else.
|
||||
// values are shown: an operator cannot correct a value they cannot
|
||||
// see. It is reachable only through the /hook/{sourceID} route
|
||||
// group, which supplies RequireAuth and NoCache, and only for a
|
||||
// target of a webhook the session's user owns; masking
|
||||
// (delivery.TargetView) is unchanged everywhere else. Do not fill a
|
||||
// form like this one on any other page.
|
||||
func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
webhook, target, ok := h.ownedTarget(w, r)
|
||||
@@ -61,34 +58,19 @@ func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
cfg, err := delivery.NewTargetConfigForm(target)
|
||||
msg := ""
|
||||
|
||||
if err != nil {
|
||||
// The error carries the parse failure, never the
|
||||
// blob, so it is safe to log against the target id.
|
||||
h.log.Warn(
|
||||
"stored target config could not be read for editing",
|
||||
"target_id", target.ID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
msg = configUnreadableMessage
|
||||
}
|
||||
|
||||
form := targetFormInput{
|
||||
Name: target.Name,
|
||||
URL: cfg.URL,
|
||||
Headers: cfg.Headers,
|
||||
Timeout: cfg.Timeout,
|
||||
ForwardQuery: cfg.ForwardQuery,
|
||||
URL: target.URL,
|
||||
Headers: delivery.FormatTargetHeaders(target.Headers),
|
||||
Timeout: delivery.FormatTargetTimeout(target.TimeoutSeconds),
|
||||
ForwardQuery: target.ForwardQuery,
|
||||
MaxAttempts: strconv.Itoa(target.MaxAttempts),
|
||||
Expiry: cfg.Expiry,
|
||||
Rotation: cfg.Rotation,
|
||||
Expiry: formatArchiveExpiry(target.ArchiveExpiry),
|
||||
Rotation: target.ArchiveRotation,
|
||||
}
|
||||
|
||||
h.renderTargetEdit(
|
||||
w, r, webhook, target, form, msg, http.StatusOK,
|
||||
w, r, webhook, target, form, "", http.StatusOK,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -96,8 +78,8 @@ func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
|
||||
// HandleTargetEditSubmit handles the target edit form submission.
|
||||
func (h *Handlers) HandleTargetEditSubmit() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
h.renameMu.Lock()
|
||||
defer h.renameMu.Unlock()
|
||||
h.archives.RenameLock().Lock()
|
||||
defer h.archives.RenameLock().Unlock()
|
||||
|
||||
webhook, target, ok := h.ownedTarget(w, r)
|
||||
if !ok {
|
||||
@@ -124,11 +106,11 @@ func (h *Handlers) HandleTargetEditSubmit() http.HandlerFunc {
|
||||
// does, so an edited destination is SSRF-validated exactly as a new
|
||||
// one is.
|
||||
//
|
||||
// The target's type is not editable. Each type stores a different
|
||||
// configuration shape and its delivery history is recorded against
|
||||
// the target row, so changing the type of an existing target is
|
||||
// really the creation of a different one. The stored type decides
|
||||
// which fields the form offers and which builder runs.
|
||||
// The target's type is not editable. Each type uses different
|
||||
// settings and its delivery history is recorded against the target
|
||||
// row, so changing the type of an existing target is really the
|
||||
// creation of a different one. The stored type decides which fields
|
||||
// the form offers and which settings are set.
|
||||
func (h *Handlers) applyTargetEdit(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
@@ -141,13 +123,7 @@ func (h *Handlers) applyTargetEdit(
|
||||
// as stored, for the page shown again when the save is refused.
|
||||
edited := *target
|
||||
|
||||
errMsg, err := h.setTargetFromForm(r.Context(), &edited, in)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to encode target config", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
errMsg := h.setTargetFromForm(r.Context(), &edited, in)
|
||||
if errMsg != "" {
|
||||
h.renderTargetEdit(
|
||||
w, r, webhook, target, in, errMsg, http.StatusBadRequest,
|
||||
@@ -157,9 +133,9 @@ func (h *Handlers) applyTargetEdit(
|
||||
}
|
||||
|
||||
// A new name renames the archive file before it is saved (see
|
||||
// delivery.Engine.Rename). If either step fails, it goes back to
|
||||
// archive.Writers.Rename). If either step fails, it goes back to
|
||||
// the name that is still stored.
|
||||
err = h.renameTargetArchive(
|
||||
err := h.renameTargetArchive(
|
||||
target, webhook.Name, target.Name, edited.Name,
|
||||
)
|
||||
if err == nil {
|
||||
@@ -178,7 +154,7 @@ func (h *Handlers) applyTargetEdit(
|
||||
)
|
||||
}
|
||||
|
||||
if errors.Is(err, delivery.ErrArchiveNameTaken) {
|
||||
if errors.Is(err, archive.ErrNameTaken) {
|
||||
h.renderTargetEdit(
|
||||
w, r, webhook, target, in,
|
||||
"Not saved: "+err.Error()+
|
||||
|
||||
@@ -2,19 +2,18 @@ package handlers_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// The destinations the target edit tests configure. Both are literal
|
||||
@@ -150,27 +149,6 @@ func storedTarget(
|
||||
return target
|
||||
}
|
||||
|
||||
// storedHTTPConfig reloads a target and parses its stored HTTP
|
||||
// configuration.
|
||||
func storedHTTPConfig(
|
||||
t *testing.T,
|
||||
env *sourceTestEnv,
|
||||
targetID string,
|
||||
) delivery.HTTPTargetConfig {
|
||||
t.Helper()
|
||||
|
||||
var cfg delivery.HTTPTargetConfig
|
||||
|
||||
require.NoError(
|
||||
t,
|
||||
json.Unmarshal(
|
||||
[]byte(storedTarget(t, env, targetID).Config), &cfg,
|
||||
),
|
||||
)
|
||||
|
||||
return cfg
|
||||
}
|
||||
|
||||
// editForm is the fully populated edit submission for an HTTP
|
||||
// target.
|
||||
func editForm(targetURL, headers, timeout string) url.Values {
|
||||
@@ -197,12 +175,9 @@ func submitTargetEdit(
|
||||
)
|
||||
}
|
||||
|
||||
// TestHandleTargetCreate_Configuration covers the half of the gap
|
||||
// that is not about editing at all: HTTPTargetConfig has carried
|
||||
// Headers and Timeout, and the delivery path has honoured them, but
|
||||
// the create form wrote {"url":...} and nothing else, so a
|
||||
// destination needing an Authorization header could not be
|
||||
// configured through the UI at all.
|
||||
// TestHandleTargetCreate_Configuration proves the add target form
|
||||
// stores an HTTP target's headers and timeout, and leaves them empty
|
||||
// when the form leaves them empty.
|
||||
func TestHandleTargetCreate_Configuration(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -213,9 +188,9 @@ func TestHandleTargetCreate_Configuration(t *testing.T) {
|
||||
assertCreateStoresHeadersAndTimeout(t, env)
|
||||
})
|
||||
|
||||
t.Run("without them keeps a url-only config", func(t *testing.T) {
|
||||
t.Run("without them stores only the url", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
assertCreateKeepsURLOnlyConfig(t, env)
|
||||
assertCreateStoresOnlyTheURL(t, env)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -228,33 +203,31 @@ func assertCreateStoresHeadersAndTimeout(
|
||||
t, env, editAuthHeader+"\nX-Tenant: acme\n", "12",
|
||||
)
|
||||
|
||||
cfg := storedHTTPConfig(t, env, target.ID)
|
||||
stored := storedTarget(t, env, target.ID)
|
||||
|
||||
assert.Equal(t, editOriginalURL, cfg.URL)
|
||||
assert.Equal(t, 12, cfg.Timeout)
|
||||
assert.Equal(t, editOriginalURL, stored.URL)
|
||||
assert.Equal(t, 12, stored.TimeoutSeconds)
|
||||
assert.Equal(
|
||||
t,
|
||||
map[string]string{
|
||||
"Authorization": "Bearer " + editBearerSecret,
|
||||
"X-Tenant": "acme",
|
||||
},
|
||||
cfg.Headers,
|
||||
stored.Headers,
|
||||
)
|
||||
}
|
||||
|
||||
// Without the new fields the stored shape must be the same
|
||||
// {"url":...} the create form wrote before they existed, so no
|
||||
// existing target's configuration is rewritten by this change.
|
||||
func assertCreateKeepsURLOnlyConfig(
|
||||
func assertCreateStoresOnlyTheURL(
|
||||
t *testing.T, env *sourceTestEnv,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
_, target := seedHTTPTarget(t, env, "", "")
|
||||
|
||||
assert.JSONEq(
|
||||
t, `{"url":"`+editOriginalURL+`"}`, target.Config,
|
||||
)
|
||||
assert.Equal(t, editOriginalURL, target.URL)
|
||||
assert.Empty(t, target.Headers)
|
||||
assert.Zero(t, target.TimeoutSeconds)
|
||||
assert.False(t, target.ForwardQuery)
|
||||
}
|
||||
|
||||
// TestHandleTargetEditSubmit_Saves is the round trip the issue asks
|
||||
@@ -299,13 +272,8 @@ func assertEditChangesDestination(
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
editReplacedURL,
|
||||
storedHTTPConfig(t, env, target.ID).URL,
|
||||
)
|
||||
|
||||
reloaded := storedTarget(t, env, target.ID)
|
||||
assert.Equal(t, editReplacedURL, reloaded.URL)
|
||||
assert.Equal(t, "edited-name", reloaded.Name)
|
||||
assert.Equal(t, 5, reloaded.MaxAttempts)
|
||||
assert.Equal(
|
||||
@@ -333,16 +301,16 @@ func assertEditRoundTripsHeadersAndTimeout(
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
cfg := storedHTTPConfig(t, env, target.ID)
|
||||
stored := storedTarget(t, env, target.ID)
|
||||
|
||||
assert.Equal(t, 21, cfg.Timeout)
|
||||
assert.Equal(t, 21, stored.TimeoutSeconds)
|
||||
assert.Equal(
|
||||
t,
|
||||
map[string]string{
|
||||
"Authorization": "Bearer rotated",
|
||||
"X-Trace": "on",
|
||||
},
|
||||
cfg.Headers,
|
||||
stored.Headers,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -362,10 +330,10 @@ func assertEditClearingRemovesThem(
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
cfg := storedHTTPConfig(t, env, target.ID)
|
||||
stored := storedTarget(t, env, target.ID)
|
||||
|
||||
assert.Empty(t, cfg.Headers)
|
||||
assert.Zero(t, cfg.Timeout)
|
||||
assert.Empty(t, stored.Headers)
|
||||
assert.Zero(t, stored.TimeoutSeconds)
|
||||
}
|
||||
|
||||
// Delivery attempts are offered only by the forms for target types
|
||||
@@ -442,6 +410,90 @@ func TestHandleTargetEdit_CallsTheDatabaseTypeArchive(t *testing.T) {
|
||||
assert.Contains(t, page, `class="label">Archive rotation</label>`)
|
||||
}
|
||||
|
||||
// TestHandleTargetEdit_EveryType edits a target of each type: the edit
|
||||
// form starts on the settings the add target form stored, and saving
|
||||
// it stores the settings submitted.
|
||||
func TestHandleTargetEdit_EveryType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
// created and edited are the add target form's and the edit
|
||||
// form's fields, as query strings; shown is what the edit form
|
||||
// must show of created; want is the settings edited stores.
|
||||
cases := []struct {
|
||||
targetType database.TargetType
|
||||
created, edited string
|
||||
shown []string
|
||||
want database.Target
|
||||
}{
|
||||
{
|
||||
database.TargetTypeHTTP,
|
||||
"url=" + editOriginalURL +
|
||||
"&headers=X-Tenant:+acme&timeout=7&forward_query=on",
|
||||
"url=" + editReplacedURL + "&timeout=9",
|
||||
[]string{
|
||||
`value="` + editOriginalURL + `"`,
|
||||
"X-Tenant: acme",
|
||||
`value="7"`,
|
||||
`name="forward_query" value="on" checked`,
|
||||
},
|
||||
database.Target{
|
||||
URL: editReplacedURL,
|
||||
Headers: map[string]string{},
|
||||
TimeoutSeconds: 9,
|
||||
},
|
||||
},
|
||||
{
|
||||
database.TargetTypeSlack,
|
||||
"url=" + editOriginalURL,
|
||||
"url=" + editReplacedURL,
|
||||
[]string{`value="` + editOriginalURL + `"`},
|
||||
database.Target{URL: editReplacedURL},
|
||||
},
|
||||
{
|
||||
database.TargetTypeDatabase,
|
||||
"expiry=720h&rotation=daily",
|
||||
"expiry=24h&rotation=hourly",
|
||||
[]string{
|
||||
`<option value="720h" selected>`,
|
||||
`<option value="daily" selected>`,
|
||||
},
|
||||
database.Target{
|
||||
ArchiveExpiry: 24 * time.Hour,
|
||||
ArchiveRotation: rotationHourly,
|
||||
},
|
||||
},
|
||||
{database.TargetTypeLog, "", "", nil, database.Target{}},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(string(tc.targetType), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
webhook, target := addTargetThroughForm(
|
||||
t, env, tc.targetType, tc.created,
|
||||
)
|
||||
|
||||
page := targetEditPage(t, env, webhook.ID, target.ID)
|
||||
for _, shown := range tc.shown {
|
||||
assert.Contains(t, page, shown)
|
||||
}
|
||||
|
||||
form, err := url.ParseQuery(tc.edited)
|
||||
require.NoError(t, err)
|
||||
form.Set("name", "edited")
|
||||
|
||||
w := submitTargetEdit(env, webhook.ID, target.ID, form)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
got := storedTarget(t, env, target.ID)
|
||||
assert.Equal(t, "edited", got.Name)
|
||||
assert.Equal(t, tc.want, targetSettingsOf(got))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleTarget_ForwardQuery covers the HTTP target's setting that
|
||||
// passes each event's query string on to it: the add target form
|
||||
// stores it checked, the edit form starts with it checked and turns it
|
||||
@@ -467,7 +519,7 @@ func TestHandleTarget_ForwardQuery(t *testing.T) {
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.True(t, storedHTTPConfig(t, env, targets[0].ID).ForwardQuery)
|
||||
assert.True(t, storedTarget(t, env, targets[0].ID).ForwardQuery)
|
||||
|
||||
w = serveTarget(
|
||||
env, http.MethodGet, targetsPath+"/"+targets[0].ID+"/edit", nil,
|
||||
@@ -479,7 +531,7 @@ func TestHandleTarget_ForwardQuery(t *testing.T) {
|
||||
|
||||
w = submitTargetEdit(env, webhook.ID, targets[0].ID, edit)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
assert.False(t, storedHTTPConfig(t, env, targets[0].ID).ForwardQuery)
|
||||
assert.False(t, storedTarget(t, env, targets[0].ID).ForwardQuery)
|
||||
|
||||
edit.Set("url", editBlockedURL)
|
||||
edit.Set("forward_query", "on")
|
||||
@@ -568,7 +620,7 @@ func assertEditRejectsBlockedDestination(
|
||||
assert.Contains(t, w.Body.String(), "Invalid target URL")
|
||||
assert.Equal(
|
||||
t, editOriginalURL,
|
||||
storedHTTPConfig(t, env, target.ID).URL,
|
||||
storedTarget(t, env, target.ID).URL,
|
||||
"a rejected edit must leave the stored config alone",
|
||||
)
|
||||
}
|
||||
@@ -597,9 +649,9 @@ func assertEditIgnoresQueryString(
|
||||
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
|
||||
cfg := storedHTTPConfig(t, env, target.ID)
|
||||
assert.Equal(t, editOriginalURL, cfg.URL)
|
||||
assert.Empty(t, cfg.Headers)
|
||||
stored := storedTarget(t, env, target.ID)
|
||||
assert.Equal(t, editOriginalURL, stored.URL)
|
||||
assert.Empty(t, stored.Headers)
|
||||
}
|
||||
|
||||
func assertEditRejectsHeaders(
|
||||
@@ -617,7 +669,7 @@ func assertEditRejectsHeaders(
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "Invalid headers")
|
||||
assert.Empty(
|
||||
t, storedHTTPConfig(t, env, target.ID).Headers,
|
||||
t, storedTarget(t, env, target.ID).Headers,
|
||||
"a rejected header must not be stored",
|
||||
)
|
||||
}
|
||||
@@ -637,7 +689,7 @@ func assertEditRejectsTimeout(
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "Invalid timeout")
|
||||
assert.Equal(
|
||||
t, 9, storedHTTPConfig(t, env, target.ID).Timeout,
|
||||
t, 9, storedTarget(t, env, target.ID).TimeoutSeconds,
|
||||
"a rejected timeout must leave the stored one alone",
|
||||
)
|
||||
}
|
||||
@@ -757,7 +809,7 @@ func assertTargetOfAnotherWebhook404s(
|
||||
|
||||
assert.Equal(
|
||||
t, editOriginalURL,
|
||||
storedHTTPConfig(t, env, target.ID).URL,
|
||||
storedTarget(t, env, target.ID).URL,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -778,7 +830,7 @@ func assertWebhookOfAnotherUser404s(
|
||||
|
||||
target := seedConfiguredTarget(
|
||||
t, env.db, other.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+editOriginalURL+`"}`,
|
||||
database.Target{URL: editOriginalURL},
|
||||
)
|
||||
|
||||
w := serveTarget(
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
@@ -242,10 +243,7 @@ func readTargetDeliveries(
|
||||
func (h *Handlers) archiveFileView(
|
||||
webhook *database.Webhook, target *database.Target, now time.Time,
|
||||
) *ArchiveFileView {
|
||||
current, err := delivery.ArchivePathAt(h.dbMgr, webhook, target, now)
|
||||
if err != nil {
|
||||
return h.archiveUnreadable(&ArchiveFileView{}, target, err)
|
||||
}
|
||||
current := archive.PathAt(h.dbMgr, webhook, target, now)
|
||||
|
||||
view := &ArchiveFileView{Name: filepath.Base(current)}
|
||||
|
||||
@@ -254,8 +252,8 @@ func (h *Handlers) archiveFileView(
|
||||
view.Note = "not created yet"
|
||||
}
|
||||
|
||||
files, err := delivery.StatArchive(
|
||||
delivery.ArchivePath(h.dbMgr, webhook, target),
|
||||
files, err := archive.Stat(
|
||||
archive.Path(h.dbMgr, webhook, target),
|
||||
)
|
||||
|
||||
switch {
|
||||
|
||||
@@ -10,8 +10,8 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
@@ -38,10 +38,10 @@ func TestHandleSourceDetail_ShowsArchiveFile(t *testing.T) {
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
archive := seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
|
||||
target := seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
|
||||
seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
path := delivery.ArchivePath(dbMgr, wh, archive)
|
||||
path := archive.Path(dbMgr, wh, target)
|
||||
|
||||
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
assert.Equal(t, 1, strings.Count(body, "Archive file:"))
|
||||
|
||||
@@ -60,7 +60,5 @@ func TestHandleTargetToggle_DoesNotUndoAnEdit(t *testing.T) {
|
||||
assert.False(t, stored.Active)
|
||||
assert.Equal(t, "edited-name", stored.Name)
|
||||
assert.Equal(t, 5, stored.MaxAttempts)
|
||||
assert.Equal(
|
||||
t, editReplacedURL, storedHTTPConfig(t, env, tgt.ID).URL,
|
||||
)
|
||||
assert.Equal(t, editReplacedURL, stored.URL)
|
||||
}
|
||||
|
||||
+20
-259
@@ -2,14 +2,11 @@ package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
)
|
||||
|
||||
@@ -70,8 +67,9 @@ func (h *Handlers) HandleWebhook() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// processWebhookRequest reads the body, serializes headers, loads
|
||||
// targets, and delivers the event.
|
||||
// processWebhookRequest reads the request into an event and has the
|
||||
// delivery engine store it, with a delivery for each of the webhook's
|
||||
// active targets, and answers the sender.
|
||||
//
|
||||
// Nothing about the request itself is authenticated: the entrypoint
|
||||
// UUID in the path is the credential, and reaching here means it
|
||||
@@ -93,30 +91,24 @@ func (h *Handlers) processWebhookRequest(
|
||||
return
|
||||
}
|
||||
|
||||
targets, err := h.loadActiveTargets(entrypoint.WebhookID)
|
||||
event := &database.Event{
|
||||
WebhookID: entrypoint.WebhookID,
|
||||
EntrypointID: entrypoint.ID,
|
||||
Method: r.Method,
|
||||
RawQuery: r.URL.RawQuery,
|
||||
Headers: string(headersJSON),
|
||||
ContentType: r.Header.Get("Content-Type"),
|
||||
Body: string(body),
|
||||
}
|
||||
|
||||
deliveries, err := h.engine.StoreEvent(event)
|
||||
if err != nil {
|
||||
h.receiverError(w, "failed to query targets", err)
|
||||
h.receiverError(w, "failed to store webhook event", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
h.createAndDeliverEvent(
|
||||
w, r, entrypoint, body, headersJSON, targets,
|
||||
)
|
||||
}
|
||||
|
||||
// loadActiveTargets returns all active targets for a webhook.
|
||||
func (h *Handlers) loadActiveTargets(
|
||||
webhookID string,
|
||||
) ([]database.Target, error) {
|
||||
var targets []database.Target
|
||||
|
||||
err := h.db.DB().Where(
|
||||
"webhook_id = ? AND active = ?",
|
||||
webhookID, true,
|
||||
).Find(&targets).Error
|
||||
|
||||
return targets, err
|
||||
h.finishWebhookResponse(w, event, entrypoint, deliveries)
|
||||
}
|
||||
|
||||
// lookupEntrypoint finds an entrypoint by UUID path.
|
||||
@@ -177,28 +169,6 @@ func (h *Handlers) readWebhookBody(
|
||||
return body, true
|
||||
}
|
||||
|
||||
// createAndDeliverEvent stores the received event, fans it out to the
|
||||
// webhook's targets, and answers the sender.
|
||||
func (h *Handlers) createAndDeliverEvent(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
entrypoint database.Entrypoint,
|
||||
body, headersJSON []byte,
|
||||
targets []database.Target,
|
||||
) {
|
||||
event, tasks, err := h.createAndFanOut(
|
||||
requestEventSource(r, entrypoint, headersJSON, body),
|
||||
targets,
|
||||
)
|
||||
if err != nil {
|
||||
h.receiverError(w, "failed to store webhook event", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
h.finishWebhookResponse(w, event, entrypoint, tasks)
|
||||
}
|
||||
|
||||
// receiverError logs an error and answers the sender with a plain-text
|
||||
// 500. The receiver's answers are for programs, so it never sends the
|
||||
// error page the web UI uses.
|
||||
@@ -212,172 +182,19 @@ func (h *Handlers) receiverError(
|
||||
)
|
||||
}
|
||||
|
||||
// eventSource carries the fields a new event is built from. The
|
||||
// receiver fills it from the live request; the resubmit handler fills
|
||||
// it from a stored event. Both then go through createAndFanOut, so an
|
||||
// event is constructed and fanned out in one place however it entered
|
||||
// the system.
|
||||
type eventSource struct {
|
||||
WebhookID string
|
||||
EntrypointID string
|
||||
Method string
|
||||
RawQuery string
|
||||
HeadersJSON string
|
||||
ContentType string
|
||||
Body []byte
|
||||
|
||||
// ResubmittedFromID names the event this one copies. Only the
|
||||
// resubmit path sets it.
|
||||
ResubmittedFromID *string
|
||||
}
|
||||
|
||||
// event builds the row this source stores.
|
||||
func (s eventSource) event() *database.Event {
|
||||
return &database.Event{
|
||||
WebhookID: s.WebhookID,
|
||||
EntrypointID: s.EntrypointID,
|
||||
Method: s.Method,
|
||||
RawQuery: s.RawQuery,
|
||||
Headers: s.HeadersJSON,
|
||||
Body: string(s.Body),
|
||||
BodyBytes: int64(len(s.Body)),
|
||||
ContentType: s.ContentType,
|
||||
ResubmittedFromID: s.ResubmittedFromID,
|
||||
}
|
||||
}
|
||||
|
||||
// requestEventSource describes the event a live receiver request
|
||||
// stores.
|
||||
func requestEventSource(
|
||||
r *http.Request,
|
||||
entrypoint database.Entrypoint,
|
||||
headersJSON, body []byte,
|
||||
) eventSource {
|
||||
return eventSource{
|
||||
WebhookID: entrypoint.WebhookID,
|
||||
EntrypointID: entrypoint.ID,
|
||||
Method: r.Method,
|
||||
RawQuery: r.URL.RawQuery,
|
||||
HeadersJSON: string(headersJSON),
|
||||
ContentType: r.Header.Get("Content-Type"),
|
||||
Body: body,
|
||||
}
|
||||
}
|
||||
|
||||
// createAndFanOut writes the event and one pending delivery per target,
|
||||
// and adds them to the webhook's running totals, in a single
|
||||
// transaction, then hands the tasks to the delivery engine. Every
|
||||
// event is created here, received or resubmitted, so a resubmitted
|
||||
// event is retried, SSRF-guarded and circuit-broken exactly as a
|
||||
// received one is. Per-delivery replay is the one other path that
|
||||
// creates a delivery: it adds one to an existing event without
|
||||
// coming through here.
|
||||
//
|
||||
// The tasks are returned as well as queued, so a caller can report how
|
||||
// many targets the event went to.
|
||||
func (h *Handlers) createAndFanOut(
|
||||
src eventSource,
|
||||
targets []database.Target,
|
||||
) (*database.Event, []delivery.Task, error) {
|
||||
webhookDB, err := h.dbMgr.GetDB(src.WebhookID)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf(
|
||||
"getting webhook database: %w", err,
|
||||
)
|
||||
}
|
||||
|
||||
tx := webhookDB.Begin()
|
||||
if tx.Error != nil {
|
||||
return nil, nil, fmt.Errorf(
|
||||
"beginning transaction: %w", tx.Error,
|
||||
)
|
||||
}
|
||||
|
||||
event := src.event()
|
||||
|
||||
err = tx.Create(event).Error
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return nil, nil, fmt.Errorf("creating event: %w", err)
|
||||
}
|
||||
|
||||
tasks, err := buildDeliveryTasks(
|
||||
tx, event, targets, inlineBody(src.Body),
|
||||
)
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
err = database.AddEventTotals(tx, database.EventTotals{
|
||||
Events: 1, LastEventAt: &event.CreatedAt,
|
||||
})
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// A resubmitted copy did not arrive on its entrypoint's URL, so it
|
||||
// leaves the entrypoint's last event as it is.
|
||||
if src.ResubmittedFromID == nil {
|
||||
err = database.AddEntrypointTotals(tx, database.EntrypointTotals{
|
||||
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
|
||||
})
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return nil, nil, err
|
||||
}
|
||||
}
|
||||
|
||||
err = tx.Commit().Error
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf(
|
||||
"committing transaction: %w", err,
|
||||
)
|
||||
}
|
||||
|
||||
// Counted here, after the commit: an event exists once it is
|
||||
// durably stored, which is what the delivery counters are
|
||||
// compared against on a dashboard. A resubmitted event counts
|
||||
// too, because it produces deliveries that the delivery side
|
||||
// counts; the resubmit counter is what separates the two.
|
||||
h.mtr.EventReceived()
|
||||
|
||||
if len(tasks) > 0 {
|
||||
h.notifier.Notify(tasks)
|
||||
}
|
||||
|
||||
return event, tasks, nil
|
||||
}
|
||||
|
||||
// inlineBody returns a pointer to body as a string if it fits
|
||||
// within the inline size limit, or nil otherwise.
|
||||
func inlineBody(body []byte) *string {
|
||||
if len(body) < delivery.MaxInlineBodySize {
|
||||
s := string(body)
|
||||
|
||||
return &s
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// finishWebhookResponse logs the event and writes the HTTP response.
|
||||
// finishWebhookResponse logs the stored event, with how many
|
||||
// deliveries it has, and writes the HTTP response.
|
||||
func (h *Handlers) finishWebhookResponse(
|
||||
w http.ResponseWriter,
|
||||
event *database.Event,
|
||||
entrypoint database.Entrypoint,
|
||||
tasks []delivery.Task,
|
||||
deliveries int,
|
||||
) {
|
||||
h.log.Info("webhook event created",
|
||||
"event_id", event.ID,
|
||||
"webhook_id", entrypoint.WebhookID,
|
||||
"entrypoint_id", entrypoint.ID,
|
||||
"target_count", len(tasks),
|
||||
"target_count", deliveries,
|
||||
)
|
||||
|
||||
w.WriteHeader(http.StatusOK)
|
||||
@@ -389,59 +206,3 @@ func (h *Handlers) finishWebhookResponse(
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// buildDeliveryTasks creates one pending delivery per target in the
|
||||
// transaction, adds each to its target's totals, and returns the tasks
|
||||
// for the delivery engine. The caller owns the transaction and rolls
|
||||
// it back on error.
|
||||
func buildDeliveryTasks(
|
||||
tx *gorm.DB,
|
||||
event *database.Event,
|
||||
targets []database.Target,
|
||||
bodyPtr *string,
|
||||
) ([]delivery.Task, error) {
|
||||
tasks := make([]delivery.Task, 0, len(targets))
|
||||
|
||||
for i := range targets {
|
||||
dlv := &database.Delivery{
|
||||
EventID: event.ID,
|
||||
TargetID: targets[i].ID,
|
||||
Status: database.DeliveryStatusPending,
|
||||
}
|
||||
|
||||
err := tx.Create(dlv).Error
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"creating delivery for target %s: %w",
|
||||
targets[i].ID, err,
|
||||
)
|
||||
}
|
||||
|
||||
err = database.AddTargetTotals(tx, database.TargetTotals{
|
||||
TargetID: targets[i].ID, Deliveries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
tasks = append(tasks, delivery.Task{
|
||||
DeliveryID: dlv.ID,
|
||||
EventID: event.ID,
|
||||
WebhookID: event.WebhookID,
|
||||
EntrypointID: event.EntrypointID,
|
||||
TargetID: targets[i].ID,
|
||||
TargetName: targets[i].Name,
|
||||
TargetType: targets[i].Type,
|
||||
TargetConfig: targets[i].Config,
|
||||
MaxAttempts: targets[i].MaxAttempts,
|
||||
Method: event.Method,
|
||||
RawQuery: event.RawQuery,
|
||||
Headers: event.Headers,
|
||||
ContentType: event.ContentType,
|
||||
Body: bodyPtr,
|
||||
AttemptNum: 1,
|
||||
})
|
||||
}
|
||||
|
||||
return tasks, nil
|
||||
}
|
||||
|
||||
@@ -115,13 +115,7 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
targets, errMsg, err := h.newWebhookTargets(r.Context(), in)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to encode target config", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
targets, errMsg := h.newWebhookTargets(r.Context(), in)
|
||||
if errMsg != "" {
|
||||
refuse(errMsg)
|
||||
|
||||
@@ -147,7 +141,7 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
|
||||
func (h *Handlers) newWebhookTargets(
|
||||
ctx context.Context,
|
||||
in sourceFormInput,
|
||||
) ([]*database.Target, string, error) {
|
||||
) ([]*database.Target, string) {
|
||||
var requested []targetFormInput
|
||||
|
||||
if in.HTTPURL != "" {
|
||||
@@ -170,15 +164,15 @@ func (h *Handlers) newWebhookTargets(
|
||||
targets := make([]*database.Target, 0, len(requested))
|
||||
|
||||
for _, form := range requested {
|
||||
target, errMsg, err := h.newTarget(ctx, "", form)
|
||||
if err != nil || errMsg != "" {
|
||||
return nil, errMsg, err
|
||||
target, errMsg := h.newTarget(ctx, "", form)
|
||||
if errMsg != "" {
|
||||
return nil, errMsg
|
||||
}
|
||||
|
||||
targets = append(targets, target)
|
||||
}
|
||||
|
||||
return targets, "", nil
|
||||
return targets, ""
|
||||
}
|
||||
|
||||
// createWebhookWithEntrypoint creates a webhook, its default
|
||||
|
||||
@@ -77,8 +77,8 @@ func (h *Handlers) deleteWebhookResources(
|
||||
"user_id", userID,
|
||||
)
|
||||
|
||||
// Release the delivery engine's per-webhook archiving state
|
||||
// so a deleted webhook's archive writer (and any handle open
|
||||
// Release the webhook's archive writers so a deleted
|
||||
// webhook's archive writer (and any handle open
|
||||
// within its debounce window) does not linger for the
|
||||
// process lifetime. The archive file itself is deliberately
|
||||
// left on disk; see evictArchiveWriter.
|
||||
@@ -150,9 +150,9 @@ func (h *Handlers) commitWebhookDeletion(
|
||||
return tx.Commit().Error
|
||||
}
|
||||
|
||||
// evictArchiveWriter asks the delivery engine to drop the cached
|
||||
// archive writers of a webhook's database targets, closing their
|
||||
// archive file handles.
|
||||
// evictArchiveWriter asks the archive writers to drop the cached
|
||||
// writers of a webhook's database targets, closing their archive
|
||||
// file handles.
|
||||
//
|
||||
// The archive database files are NOT deleted. Unlike the event
|
||||
// database — which is per-webhook working storage and is
|
||||
|
||||
@@ -109,7 +109,7 @@ func (h *Handlers) renderSourceDetail(
|
||||
data := map[string]any{
|
||||
tmplKeyWebhook: &webhook,
|
||||
// Targets are projected to a display-safe view: a
|
||||
// target's stored config blob holds a credential, and it
|
||||
// target's URL and headers hold a credential, and they
|
||||
// must never reach a template.
|
||||
"Entrypoints": entrypointViews,
|
||||
"Targets": h.targetRows(&webhook, targets),
|
||||
|
||||
@@ -6,8 +6,8 @@ import (
|
||||
"strconv"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// HandleSourceEdit shows the form to edit a webhook.
|
||||
@@ -59,8 +59,8 @@ func (h *Handlers) HandleSourceEditSubmit() http.HandlerFunc {
|
||||
|
||||
sourceID := chi.URLParam(r, "sourceID")
|
||||
|
||||
h.renameMu.Lock()
|
||||
defer h.renameMu.Unlock()
|
||||
h.archives.RenameLock().Lock()
|
||||
defer h.archives.RenameLock().Unlock()
|
||||
|
||||
var webhook database.Webhook
|
||||
|
||||
@@ -130,7 +130,7 @@ func (h *Handlers) applyWebhookEdit(
|
||||
edited.RetentionDays = retentionDays
|
||||
|
||||
// A new name renames the archive files before it is saved (see
|
||||
// delivery.Engine.Rename). If either step fails, the same targets'
|
||||
// archive.Writers.Rename). If either step fails, the same targets'
|
||||
// archives go back to the name that is still stored, without
|
||||
// reading the main database again.
|
||||
targets, err := h.renameWebhookArchives(
|
||||
@@ -150,7 +150,7 @@ func (h *Handlers) applyWebhookEdit(
|
||||
)
|
||||
}
|
||||
|
||||
if errors.Is(err, delivery.ErrArchiveNameTaken) {
|
||||
if errors.Is(err, archive.ErrNameTaken) {
|
||||
h.renderWebhookEdit(
|
||||
w, r, webhook, name, description, retention,
|
||||
"Not saved: "+err.Error()+
|
||||
|
||||
@@ -153,7 +153,7 @@ func seedStatsHistory(
|
||||
|
||||
first := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
|
||||
@@ -206,3 +206,87 @@ func receivedLine(t *testing.T, buf *bytes.Buffer) map[string]any {
|
||||
|
||||
return found[0]
|
||||
}
|
||||
|
||||
// TestHandleWebhook_StoresTheEvent proves an accepted request is
|
||||
// answered with the receiver's 200 and body, and that the event is
|
||||
// stored with a pending delivery to the webhook's active target.
|
||||
func TestHandleWebhook_StoresTheEvent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
webhook := seedWebhook(t, db)
|
||||
entrypoint := seedEntrypoint(t, db, webhook.ID)
|
||||
target := seedConfiguredTarget(
|
||||
t, db, webhook.ID, database.TargetTypeHTTP,
|
||||
database.Target{URL: replayTargetURL},
|
||||
)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleWebhook().ServeHTTP(w, postRequest(
|
||||
"/h/"+entrypoint.Path, nil,
|
||||
map[string]string{"uuid": entrypoint.Path},
|
||||
))
|
||||
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
assert.JSONEq(t, `{"status":"ok"}`, w.Body.String())
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(webhook.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
events := listEvents(t, webhookDB)
|
||||
require.Len(t, events, 1)
|
||||
assert.Equal(t, entrypoint.ID, events[0].EntrypointID)
|
||||
|
||||
deliveries := listDeliveries(t, webhookDB, events[0].ID)
|
||||
require.Len(t, deliveries, 1)
|
||||
assert.Equal(t, target.ID, deliveries[0].TargetID)
|
||||
assert.Equal(t, database.DeliveryStatusPending, deliveries[0].Status)
|
||||
}
|
||||
|
||||
// TestHandleWebhook_FailedStoreAnswersPlainError proves that an event
|
||||
// the engine cannot store is answered with the receiver's plain-text
|
||||
// 500, and that nothing of it is left stored.
|
||||
func TestHandleWebhook_FailedStoreAnswersPlainError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
webhook := seedWebhook(t, db)
|
||||
entrypoint := seedEntrypoint(t, db, webhook.ID)
|
||||
|
||||
// Without its event totals table the webhook's database refuses
|
||||
// the totals write, which comes after the event row.
|
||||
webhookDB, err := dbMgr.GetDB(webhook.ID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t,
|
||||
webhookDB.Migrator().DropTable(&database.EventTotals{}))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleWebhook().ServeHTTP(w, postRequest(
|
||||
"/h/"+entrypoint.Path, nil,
|
||||
map[string]string{"uuid": entrypoint.Path},
|
||||
))
|
||||
|
||||
assert.Equal(t, http.StatusInternalServerError, w.Code)
|
||||
assert.Equal(t, "Internal server error\n", w.Body.String())
|
||||
assert.Empty(t, listEvents(t, webhookDB))
|
||||
}
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -17,6 +18,7 @@ import (
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxtest"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
@@ -129,11 +131,13 @@ func run(
|
||||
return code, stdout.String(), stderr.String()
|
||||
}
|
||||
|
||||
type noopNotifier struct{}
|
||||
type noopArchives struct {
|
||||
renameMu sync.Mutex
|
||||
}
|
||||
|
||||
func (n *noopNotifier) Notify([]delivery.Task) {}
|
||||
|
||||
type noopArchives struct{}
|
||||
func (n *noopArchives) RenameLock() *sync.Mutex {
|
||||
return &n.renameMu
|
||||
}
|
||||
|
||||
func (n *noopArchives) EvictWebhook(string) {}
|
||||
|
||||
@@ -182,8 +186,13 @@ func newServerApp(
|
||||
database.NewWebhookDBManager,
|
||||
healthcheck.New,
|
||||
session.New,
|
||||
func() delivery.Notifier { return &noopNotifier{} },
|
||||
func() delivery.Archives { return &noopArchives{} },
|
||||
archive.NewWriters,
|
||||
// The engine is built on a lifecycle that is never started,
|
||||
// so no worker runs.
|
||||
func(p delivery.EngineParams) *delivery.Engine {
|
||||
return delivery.New(fxtest.NewLifecycle(t), p)
|
||||
},
|
||||
func() archive.Archives { return &noopArchives{} },
|
||||
func() delivery.CircuitBreakers {
|
||||
return &noopCircuitBreakers{}
|
||||
},
|
||||
|
||||
@@ -1183,16 +1183,26 @@ func checkNewWebhookTargets(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
// Each value the pruning choice submits, after an empty one that
|
||||
// stands for archive left off.
|
||||
expiries := []string{
|
||||
"", "never", "1h", "12h", "24h", "720h", "2160h", "8760h",
|
||||
// Each value the pruning choice submits and the expiry it stores,
|
||||
// after an empty one that stands for archive left off.
|
||||
expiries := []struct {
|
||||
value string
|
||||
stored time.Duration
|
||||
}{
|
||||
{"", 0},
|
||||
{"never", 0},
|
||||
{"1h", time.Hour},
|
||||
{"12h", 12 * time.Hour},
|
||||
{"24h", 24 * time.Hour},
|
||||
{"720h", 720 * time.Hour},
|
||||
{"2160h", 2160 * time.Hour},
|
||||
{"8760h", 8760 * time.Hour},
|
||||
}
|
||||
|
||||
for _, httpURL := range []string{"", publicTargetURL} {
|
||||
for _, expiry := range expiries {
|
||||
name := "url=" + httpURL + " archive=" + expiry
|
||||
want := map[database.TargetType]string{}
|
||||
name := "url=" + httpURL + " archive=" + expiry.value
|
||||
want := map[database.TargetType]database.Target{}
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx,
|
||||
@@ -1205,10 +1215,12 @@ func checkNewWebhookTargets(
|
||||
"#http_url", httpURL, chromedp.ByQuery,
|
||||
)))
|
||||
|
||||
want[database.TargetTypeHTTP] = `{"url":"` + httpURL + `"}`
|
||||
want[database.TargetTypeHTTP] = database.Target{
|
||||
URL: httpURL, Headers: map[string]string{},
|
||||
}
|
||||
}
|
||||
|
||||
if expiry != "" {
|
||||
if expiry.value != "" {
|
||||
// The choice showing moves Create down, so it is
|
||||
// waited for before Create is clicked.
|
||||
click(ctx, t, archiveBox)
|
||||
@@ -1216,29 +1228,30 @@ func checkNewWebhookTargets(
|
||||
"%s: checking archive does not show the pruning choice",
|
||||
name)
|
||||
require.NoError(t, chromedp.Run(ctx, chromedp.SetValue(
|
||||
pruningChoice, expiry, chromedp.BySearch,
|
||||
pruningChoice, expiry.value, chromedp.BySearch,
|
||||
)))
|
||||
|
||||
// The rotation choice is left on none.
|
||||
want[database.TargetTypeDatabase] = `{"expiry":"` + expiry +
|
||||
`","rotation":"none"}`
|
||||
want[database.TargetTypeDatabase] = database.Target{
|
||||
ArchiveExpiry: expiry.stored, ArchiveRotation: "none",
|
||||
}
|
||||
}
|
||||
|
||||
clickAndLoad(ctx, t, createButton)
|
||||
require.Truef(t, shown(ctx, `//h1[text()="`+name+`"]`),
|
||||
"%s: the new webhook's page does not open", name)
|
||||
|
||||
assert.Equalf(t, want, targetConfigs(t, env, name),
|
||||
assert.Equalf(t, want, targetSettings(t, env, name),
|
||||
"%s: the webhook does not have the targets asked for", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// targetConfigs reads the targets of the webhook named name, and
|
||||
// returns each one's stored configuration by its type.
|
||||
func targetConfigs(
|
||||
// targetSettings reads the targets of the webhook named name, and
|
||||
// returns each one's stored settings by its type.
|
||||
func targetSettings(
|
||||
t *testing.T, env *testEnv, name string,
|
||||
) map[database.TargetType]string {
|
||||
) map[database.TargetType]database.Target {
|
||||
t.Helper()
|
||||
|
||||
var (
|
||||
@@ -1251,12 +1264,19 @@ func targetConfigs(
|
||||
require.NoError(t, env.db.DB().
|
||||
Where("webhook_id = ?", webhook.ID).Find(&targets).Error)
|
||||
|
||||
configs := map[database.TargetType]string{}
|
||||
settings := map[database.TargetType]database.Target{}
|
||||
for _, target := range targets {
|
||||
configs[target.Type] = target.Config
|
||||
settings[target.Type] = database.Target{
|
||||
URL: target.URL,
|
||||
Headers: target.Headers,
|
||||
TimeoutSeconds: target.TimeoutSeconds,
|
||||
ForwardQuery: target.ForwardQuery,
|
||||
ArchiveExpiry: target.ArchiveExpiry,
|
||||
ArchiveRotation: target.ArchiveRotation,
|
||||
}
|
||||
}
|
||||
|
||||
return configs
|
||||
return settings
|
||||
}
|
||||
|
||||
// checkRefusedNewWebhook submits the new webhook page with archive
|
||||
|
||||
@@ -311,7 +311,7 @@ func (s *Server) setupSourceRoutes() {
|
||||
r.Post("/targets", s.h.HandleTargetCreate())
|
||||
// The edit form is the one page that renders a target's
|
||||
// destination URL and header values in full; see
|
||||
// delivery.TargetConfigForm. It belongs to this group for
|
||||
// Handlers.HandleTargetEdit. It belongs to this group for
|
||||
// its RequireAuth and NoCache, which are what keep that
|
||||
// exception from reaching an unauthenticated request or a
|
||||
// shared cache.
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -20,6 +21,7 @@ import (
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxtest"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/archive"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
@@ -46,14 +48,16 @@ const (
|
||||
metricsAuthValue = "s3cret"
|
||||
)
|
||||
|
||||
type noopNotifier struct{}
|
||||
|
||||
func (n *noopNotifier) Notify([]delivery.Task) {}
|
||||
|
||||
// noopArchives satisfies handlers.New's delivery.Archives
|
||||
// noopArchives satisfies handlers.New's archive.Archives
|
||||
// dependency. No test here checks what gets evicted or renamed, so
|
||||
// it records nothing.
|
||||
type noopArchives struct{}
|
||||
// it records nothing. Its rename lock is a plain mutex.
|
||||
type noopArchives struct {
|
||||
renameMu sync.Mutex
|
||||
}
|
||||
|
||||
func (e *noopArchives) RenameLock() *sync.Mutex {
|
||||
return &e.renameMu
|
||||
}
|
||||
|
||||
func (e *noopArchives) EvictWebhook(string) {}
|
||||
|
||||
@@ -137,8 +141,13 @@ func newTestEnvWithConfig(
|
||||
database.NewWebhookDBManager,
|
||||
healthcheck.New,
|
||||
session.New,
|
||||
func() delivery.Notifier { return &noopNotifier{} },
|
||||
func() delivery.Archives { return &noopArchives{} },
|
||||
archive.NewWriters,
|
||||
// The engine is built on a lifecycle that is never started,
|
||||
// so no worker runs and nothing is sent anywhere.
|
||||
func(p delivery.EngineParams) *delivery.Engine {
|
||||
return delivery.New(fxtest.NewLifecycle(t), p)
|
||||
},
|
||||
func() archive.Archives { return &noopArchives{} },
|
||||
func() delivery.CircuitBreakers {
|
||||
return &noopCircuitBreakers{}
|
||||
},
|
||||
@@ -406,7 +415,7 @@ func (e *testEnv) seedTarget(
|
||||
Name: "routed-target",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Active: true,
|
||||
Config: `{"url":"http://93.184.216.34/hook"}`,
|
||||
URL: "http://93.184.216.34/hook",
|
||||
MaxAttempts: 1,
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user