Commit Graph
159 Commits
Author SHA1 Message Date
clawbot f755c03110 Default-block Azure WireServer's public address (closes #245)
check / check (push) Successful in 4m34s
Add 168.63.129.16 (Azure WireServer) to blockedNetworks, the default
blocklist, not alwaysBlockedNetworks: it is public unicast, so an
operator who lists it in ALLOWED_EGRESS_CIDRS can reach it again. The
refusal message, the allowlist startup warning, the README and the
comments no longer call every blocked address private/reserved, and
no longer claim the allowlist cannot open any metadata endpoint.

Sources:
- https://learn.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16
- https://learn.microsoft.com/en-us/azure/virtual-machines/metadata-security-protocol/overview

Deviation: 147.75.207.243 (Equinix Metal) is not added; Equinix
documents only a hostname, and the service was sunset on 2026-06-30.

Model: opus-5-5
2026-09-29 10:22:07 +02:00
clawbot 4a724130ca Close archive writers when the delivery engine stops (closes #280)
check / check (push) Successful in 3m45s
The engine cached archive writers and never closed them at shutdown,
so after a clean stop an archive's rows could sit in its -wal while
the .db held no table. The engine's stop hook now evicts every cached
writer once its workers have returned, the same way deleting a webhook
does, so a clean stop leaves each archive as one file and a late write
is refused. If the workers do not return within the stop budget, the
writers are left open as a kill would leave them: closing would wait
on a write in progress, and a still-running worker would open new
ones.

The README no longer says archives keep their sidecars across a clean
stop.

Model: opus-5-5
2026-09-29 08:30:22 +02:00
clawbot d4f4ddf51f Send Content-Type once on a delivery (closes #246)
check / check (push) Successful in 3m20s
A delivery set Content-Type from the event's ContentType and then
added the inbound Content-Type from the event's stored headers, so a
target could receive two values. The inbound Content-Type is no
longer forwarded from the stored headers; the receiver already saves
it as the event's ContentType.

Which value is sent is now stated at applyRequestHeaders: a
Content-Type configured on the target, otherwise the event's
ContentType, otherwise none. A configured one still survives a
cross-origin 307/308 with its body.

Model: opus-5-5
2026-09-29 07:11:55 +02:00
clawbot 51580a2bc6 Fail a pending delivery whose target was deleted (closes #293)
check / check (push) Successful in 3m38s
Restart recovery and the pending sweep skipped a pending delivery
whose target was missing from the batch's target map, every minute,
for the life of the database. A miss now asks loadTarget: no row
fails the delivery terminally with a recorded reason; any other error
leaves it pending, since the map is also empty when its query failed;
a target found there is used.

The failure goes through the ownership-gated function the retrying
paths already used, now failMissingTarget. Once it owns the delivery
it re-reads the row and fails it only if the status is unchanged, so
a delivery sent and settled in between is left alone.

Model: opus-5-5
2026-09-29 06:48:19 +02:00
clawbot e0b211f960 Make deliveries refused while half-open wait a cooldown (closes #306)
check / check (push) Successful in 5m0s
While the breaker was half-open, Allow refused every delivery but the
probe and CooldownRemaining returned zero, so each queued task for the
target went straight back onto the retry channel and rewrote its status
on every pass until the probe finished.

CooldownRemaining now returns the whole cooldown while half-open, so a
refused delivery waits that long. A refused delivery already at
retrying is not written again, so the retry counter now moves only
when a refusal moves a delivery into retrying.

Model: opus-5-5
2026-09-29 05:48:20 +02:00
clawbot 978eb01b29 Open each event database once when callers race (closes #291)
check / check (push) Successful in 3m48s
GetDB opened the database on a cache miss and then tried to cache it,
so callers racing on a webhook's first use could each open the file,
and the losers closed their copies. On a new file the parallel opens
also create its tables at the same time, and one caller can fail with
"table already exists".

A mutex now covers the open: GetDB looks in the cache again under it,
then opens and caches. DeleteDB and CloseAll take the same mutex, so
neither runs while an open is under way. Reading an already cached
database takes no lock.

The new test starts many callers on one webhook at once and checks
that exactly one open happened.

Model: opus-5-5
2026-09-29 04:37:30 +02:00
clawbot 3cdab97930 Say make check needs make bootstrap on a fresh clone (closes #282)
check / check (push) Successful in 11s
The third-party browser assets are not committed, so on a fresh clone
make check fails in the tests until make bootstrap (or make assets) has
fetched them. The Entrypoints section now says so up front, and why the
check does not fetch them itself: it must not change files in the repo.

Model: opus-5-5
2026-09-29 04:30:14 +02:00
clawbot 83740b1de1 Do not resend a delivery that restart recovery already sent (closes #299)
check / check (push) Successful in 3m35s
Restart recovery could find a just-written delivery pending, send it
and release it before the receiver's Notify queued the same delivery.
Notify's claim then succeeded on the released id, and the worker sent
it again because the new-task path never read the delivery's row.

Before sending a new task the worker now reads the delivery's status
by primary key and skips the task unless the row still says pending,
as the retry path already does for retrying. Nothing else can change
the row while the worker owns the delivery. A row left pending by a
failed bookkeeping write is still sent again.

loadRetryDelivery is renamed loadDelivery now that both paths use it.

Model: opus-5-5
2026-09-29 04:12:14 +02:00
sneak aeeeca5ea1 Merge branch 'main' into next
check / check (push) Successful in 9s
2026-09-29 03:13:57 +02:00
clawbot 6ebac4fa71 Index the event-tier columns the sweeps, event log and retention scan (closes #314)
check / check (push) Successful in 3m4s
The per-webhook event databases had no secondary indexes, so startup
recovery, the retry and pending sweeps, the queue-depth sampler, the
event log and retention each read whole tables. Indexes declared in
the GORM model tags now serve them, and AutoMigrate adds them to new
and existing databases alike.

Each index also covers deleted_at: GORM adds deleted_at IS NULL to
these queries, and SQLite, with no table statistics, otherwise
prefers the existing deleted_at index. A test checks SQLite's plan
for each statement as GORM builds it.

Rule suppressed: lll on the three event-tier model structs, whose
struct tags cannot wrap.
The resubmitted_from_id scan is left to
#325.

Model: opus-4-8 (implementation); opus-5-5 (rework)
2026-09-28 14:13:22 +02:00
clawbot 237f131367 Default WEBHOOKER_ENVIRONMENT to prod (closes #307)
check / check (push) Successful in 3m14s
An unset WEBHOOKER_ENVIRONMENT now means prod, not dev. The only
thing dev still changes is CORS, which then answers every origin with
Access-Control-Allow-Origin: *, so an operator who forgets the
variable is no longer silently permissive; dev must be set
explicitly. Cookie Secure and CSRF strictness follow each request's
transport and are unaffected.

The README, comments and tests no longer describe dev as the default:
the deployment checklist asks only that the environment is not dev,
the Docker and nginx examples drop the now-redundant setting, and the
TRUSTED_PROXIES warning gives its real reason for firing in every
environment.

Model: opus-4-8 (implementation); opus-5-5 (rework)
2026-09-28 12:47:31 +02:00
clawbot 7ed1588443 Document running webhooker under upaas (closes #323)
check / check (push) Successful in 8s
Adds a "Running under upaas" section to the README: add no port
mapping, since upaas publishes mapped ports on every host interface,
and put the app on the reverse proxy's Docker network instead; one
data volume at /var/lib/webhooker, created owned by UID 1000 before
the first deploy; WEBHOOKER_ENVIRONMENT and TRUSTED_PROXIES; the
health check upaas reads 60 seconds after a deploy; and where the
first-run admin password appears and how to reset it.

upaas bind-mounts a host directory it never creates, and one made by
root stops the container at its data directory lock. The documented
creation step removes that; the image is unchanged.

Model: opus-5-5
2026-09-28 12:30:32 +02:00
clawbot b051821370 Say max_retries is the total attempt count, not a retry count (closes #316)
check / check (push) Successful in 3m54s
The help text under the field on both target forms and the max_retries rows in the README now say the number is the total number of delivery attempts: 0 is a single attempt with no retries and no circuit breaker, and N is N attempts in total. The delivery code already worked this way; only the wording was wrong, so an operator wanting one try plus two retries would have entered 2 instead of 3. A UI copy test renders both forms and pins the wording. Delivery behaviour is unchanged.

Model: opus-4-8 (implementation); fable-5-1 (merge)
2026-09-21 18:33:09 +02:00
clawbot 39afa69bfc Consolidate the data directory mode into one owner (closes #288)
check / check (push) Successful in 6m34s
Two packages each declared the 0o750 mode for DATA_DIR and both created the directory. internal/datadir now exports DirPerm as the single definition, and internal/database uses it in both places it creates the directory. The value is unchanged, so existing deployments see no permission change. datadir owns it because guarding and creating DATA_DIR is that package's whole purpose and it imports nothing that would form a cycle.

Model: opus-4-8 (implementation and review); fable-5-1 (merge)
2026-09-21 10:01:51 +02:00
clawbotandsneak 888eaf526b State the UUID-is-the-credential rule as a rule (closes #301) (#302)
check / check (push) Successful in 4m28s
Closes #301. Docs-only apart from one test comment; no behaviour change.

The receiver has authenticated on the entrypoint UUID alone since inbound signature verification was removed in #279. The README described that as the current state. It did not say it is the decision, which leaves a future contributor free to propose HMAC as an improvement rather than as a reversal.

What changed:

- `## The entrypoint URL is the authentication secret` now states the rule: the v4 UUID at `/webhook/{uuid}` is the credential and the only one; no shared secret, HMAC signature, bearer token or second factor will be added, including as defence in depth. It names the removal that settled it, and it says explicitly that signature headers a sender sends anyway are stored and forwarded but never checked — the previous text left that ambiguous.
- The same section carries the two consequences an operator has to act on: the URL is a capability, so keep it out of logs, tickets and screenshots; and rotation means minting a new entrypoint, not changing a key.
- It also handles the case the rule will next be argued from: a sender that only supports signed payloads to a well-known URL is a constraint on that integration, to be raised on its own terms, not grounds to reintroduce shared secrets.
- The rule is reachable without scrolling 1,100 lines: a pointer in the intro, a new first bullet under Authentication (which previously listed the web UI, the API and `/metrics` and said nothing about the receiver at all), and a sharpened bullet under Security.

Stale language found and corrected: one, in `internal/delivery/redirect_test.go`. Its comment justified same-origin header retention partly by "the inbound signature the receiver verifies" — in this repo's vocabulary "the receiver" is `/webhook/{uuid}`, which verifies nothing. The endpoint that verifies it is the delivery target's, and the comment now says so.

Two places that read like stale signing language were checked and left alone as accurate: `internal/delivery/redirect.go` and `internal/server/sentry.go` describe signature headers senders put on the receiver route, which do arrive and are forwarded — neither claims webhooker checks them.

`REPO_POLICIES.md` was deliberately not touched. It is the cross-project policy document synced from `sneak/prompts` and carries `last_modified` front matter for that purpose, so a webhooker-specific carve-out does not belong in it. Worth knowing: its hardening section ends "if a standard security hardening measure exists for HTTP services and is not listed here, it is still expected. When in doubt, harden" — that is the sentence a future HMAC proposal will cite, and only the README now answers it.

`TODO.md` is untouched per its own Workflow section (issue branches do not touch it).

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #302
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-30 04:05:38 +02:00
sneak 251cb3d3d3 1.0.0b1 - testing remains
check / check (push) Successful in 8s
Reviewed-on: #111
2026-08-30 04:04:59 +02:00
clawbot d61d9dc1c1 Drop the stale open-work claim from the TODO status section
check / check (push) Successful in 2m56s
The milestone is the authoritative count and the section already says
so; the lead-in asserted work remaining independently of it.
2026-08-24 15:52:44 +00:00
clawbot b0a011f6b4 Render unknown for a zero CreatedAt in Slack/Mattermost messages (closes #298)
check / check (push) Superseded by a newer commit; never tested
2026-08-24 17:52:25 +02:00
clawbot 5976a4a98f Carry the event's receipt time into every delivery (closes #257)
check / check (push) Successful in 3m4s
2026-08-24 06:44:24 +02:00
clawbot b2c9acdaa6 Correct four documentation claims ahead of the 1.0.0 tag
check / check (push) Successful in 7s
2026-08-24 06:25:08 +02:00
clawbot af3703d748 Close the two remaining delivery terminal-state gaps (closes #107)
check / check (push) Successful in 3m16s
2026-08-24 05:12:02 +02:00
clawbot 322d9a6d6b Create every SQLite file 0600 (closes #255)
check / check (push) Successful in 3m17s
2026-08-24 04:33:40 +02:00
clawbot b9f7db6901 Fail loudly on an unparseable SENTRY_DSN and a malformed .env (closes #283)
check / check (push) Successful in 2m54s
2026-08-24 04:25:29 +02:00
clawbot 48cf93ec7e Derive the entrypoint URL scheme from the shared TLS predicate (closes #272)
check / check (push) Successful in 3m44s
2026-08-24 04:04:04 +02:00
clawbot 8d64259283 Open SQLite with WAL and bound delivery re-dispatch by ownership (closes #256)
check / check (push) Successful in 3m1s
2026-08-24 03:49:17 +02:00
clawbot bde32d3ee6 Record landed 1.0.0 work and drop the removed signing roadmap in TODO.md
check / check (push) Successful in 8s
2026-08-24 03:43:16 +02:00
clawbot 62576f6fc6 Bind the app port deliberately and document the proxy deployment (closes #268) (closes #226)
check / check (push) Successful in 3m4s
2026-08-24 03:38:44 +02:00
clawbot 37b59f8822 Remove inbound request signature verification (closes #279)
check / check (push) Successful in 3m14s
2026-08-24 03:25:09 +02:00
clawbot ee2276a912 Stamp the build version into the binary (closes #253)
check / check (push) Successful in 3m12s
2026-08-24 03:15:20 +02:00
clawbot 032f265d69 Derive cookie Secure and CSRF strictness from the request transport (closes #269)
check / check (push) Successful in 2m56s
2026-08-24 03:01:37 +02:00
clawbot 65ace2d856 Roll back a failed webhook deletion instead of committing it (closes #262)
check / check (push) Superseded by a newer commit; never tested
2026-08-24 03:01:33 +02:00
clawbot 5fda446c71 Name a deleted target on its historical deliveries (closes #211)
check / check (push) Successful in 3m13s
2026-08-24 02:03:23 +02:00
clawbot 763d8f8058 Bound the /metrics method label (closes #261)
check / check (push) Superseded by a newer commit; never tested
2026-08-24 02:03:18 +02:00
clawbot fd5966f807 Validate max_retries on both target forms (closes #221)
check / check (push) Successful in 3m36s
2026-08-24 01:32:48 +02:00
clawbot 0082f216fa Label HTTP metrics with the chi route pattern (closes #254)
check / check (push) Superseded by a newer commit; never tested
2026-08-24 01:32:44 +02:00
clawbot 89f3b984d2 Resubmit a stored event as a new undelivered event (closes #250)
check / check (push) Successful in 3m1s
2026-08-24 00:53:37 +02:00
clawbot a83e8fe654 Record the completed 1.0.0 milestone in TODO.md (#249)
check / check (push) Successful in 7s
2026-08-20 11:16:51 +02:00
clawbot 687405993e Harden operator-set target headers (closes #233) (#242)
check / check (push) Successful in 2m58s
2026-08-20 10:54:43 +02:00
clawbot 03cd1859d7 Add an egress CIDR allowlist to the SSRF guard (closes #204) (#217)
check / check (push) Successful in 3m1s
2026-08-20 10:34:42 +02:00
clawbot f0512f1c3c Render delivery attempt detail in the event log (closes #202) (#219)
check / check (push) Successful in 2m55s
delivery_results stored status_code, response_body, error, duration and
attempt_num, and no template rendered any of it, so a failure read as
"target: failed" and diagnosing it meant opening the per-webhook SQLite
file by hand.

An expanded delivery now lists its attempts with attempt number, status
code, duration, error and response body. The body is bounded in the query
rather than read whole and truncated in Go (#135), and a body the engine
itself cut is no longer presented as complete.

The response body and error are untrusted remote content, so target
credentials are removed before rendering. Two cases needed care: a secret
severed by the 4096-byte cut matches nothing as a whole string, and the
engine's io.LimitReader cuts at the same constant the renderer uses, so
the guard keys on the body reaching the cap rather than on the stored size
exceeding it. Empty secrets are filtered where the secret list is built,
because an empty string passed to strings.ReplaceAll inserts the marker at
every byte boundary.

loadTargetMap builds the redactor half unscoped, so a soft-deleted
target's historical deliveries still render redacted.

Also regenerates static/css/tailwind.css, which had drifted from the
templates: hover:text-red-700, text-red-500, underline and w-28 were in
use but absent from the served stylesheet (#236).
2026-08-20 08:36:25 +02:00
clawbot 3b0ed826bc Add per-delivery replay to the event log (closes #203) (#240)
check / check (push) Successful in 3m21s
There was no redelivery path anywhere: once a delivery exhausted
max_retries it was failed permanently, even though the event body is
durably stored. Storing an event and being unable to re-send it defeats
the reason it is stored, and the ordinary case is a destination that was
down longer than the backoff ladder.

Adds POST /source/{sourceID}/deliveries/{deliveryID}/replay, inside the
authenticated group so it inherits MaxBodySize, CSRF, NoCache and
RequireAuth. Replay creates a NEW pending delivery against the target's
CURRENT config and hands it to the engine through the same notifier the
receiver uses, so it runs the normal path with the retry ladder, the
SSRF-guarded transport and the circuit breaker. The original delivery's
rows are never touched, and the stored event body is re-sent, never the
recorded response.

Replay is refused, with a distinct message, for a non-terminal delivery, a
deleted target, a deactivated target, and when an earlier replay of the
same event and target is still in flight. Bounded by a per-client rate
limit and by that in-flight check.

The new delivery row is written with Omit(clause.Associations) and with
neither Event nor Target populated, so it cannot upsert a targets row into
the per-webhook event database (#206).

Counted by webhooker_delivery_replays_total on the existing target_type
label. A replay also moves the ordinary attempt, outcome and duration
series, because it is a real delivery.
2026-08-20 08:11:35 +02:00
clawbot 9969694a47 Add a webhooker resetpw subcommand and a bootstrap banner (closes #208) (#239)
check / check (push) Successful in 3m25s
The admin bootstrap password was printed once, as one line among roughly
45 fx lines, and under docker run -d went to container logs subject to
rotation. There was no reset path at all -- no subcommand, no forgot-password
flow, no env override -- so recovery meant hand-deleting the users row from
webhooker.db, which was documented nowhere.

Adds webhooker resetpw [-generate] &lt;username&gt;. The password is read from
stdin or generated with the existing crypto/rand helper, never taken from
argv where /proc would publish it. It reuses the existing Argon2id hashing
rather than reimplementing the parameters, and writes a single UPDATE only
after the hash is complete, so no failure can leave an account with no
usable password. An unknown username is a hard error and never creates an
account.

It refuses to run against a DATA_DIR held by a live instance, via the
exclusive lock from #201. DATA_DIR and webhooker.db are checked to exist
before the lock is acquired, so a mistyped path creates nothing -- neither
a directory tree nor a stray lock file.

The bootstrap password now appears exactly once, in a distinct banner
written straight to a caller-named writer rather than as an fx log line.
2026-08-20 08:01:42 +02:00
clawbot fcead5d401 Add optional inbound webhook signature verification (closes #67) (#228)
check / check (push) Superseded by a newer commit; never tested
The receiver had no inbound authentication of any kind: /webhook/{uuid}
was mounted behind a rate limiter alone, so the only thing protecting an
entrypoint was the secrecy of a v4 UUID in a URL path. Inbound headers are
forwarded almost verbatim to the target, so anyone who learned the URL
also chose the headers the downstream service received.

Adds an optional per-entrypoint secret with two schemes: github
(X-Hub-Signature-256, HMAC-SHA256 hex over the raw body) and gitlab
(X-Gitlab-Token, a plain shared token). Comparison is constant-time, the
HMAC is computed over the raw body before any parsing, and rejection
happens before persistence -- an unauthenticated request creates no event
row. An entrypoint with no secret behaves exactly as before, including
every row that predates this change.

The scheme's credential header is stripped from the header map before it
is marshalled into Event.Headers, so the GitLab token reaches neither the
event store nor any delivery target. SchemeInfo.HeaderIsDigest defaults to
false meaning strip, so a scheme added later is protected unless its
header is positively declared a digest.
2026-08-20 08:01:32 +02:00
clawbot ac782f4c5a Stop target credentials leaking into event databases (closes #206) (#223)
check / check (push) Successful in 3m34s
GORM's association upsert copied whole targets rows -- plaintext
credential-bearing config -- into the per-webhook event databases with an
empty webhook_id. The leak was in updateDeliveryStatus, not the create
path: Update leaves Statement.Model pointing at a Delivery whose Target
the engine populated, so save_before_associations upserts it.

A connection-level callback now appends clause.Associations to
Statement.Omits on the create and update chains of every per-webhook
connection, so every write path is covered rather than one call site.

Existing files are swept on first open: the leaked rows are deleted and
the file is VACUUMed, because DELETE alone only unlinks the pages and
leaves the credential recoverable in the file's free space. The sweep is
recorded in PRAGMA user_version only after the VACUUM returns, so a sweep
that fails or is interrupted fails the open and is retried on the next
one, rather than being marked done.

Encryption of target config at rest is deliberately out of scope and
deferred to #212.
2026-08-20 07:55:34 +02:00
clawbot 89b2dadd48 Read queue depths with Find, not Scan (closes #234) (#237)
check / check (push) Superseded by a newer commit; never tested
Both queue-depth reads used (*gorm.DB).Scan, which swaps GORM's own trace
recorder in for the logging adapter. That recorder does not implement
gorm.ParamsFilter, so those statements logged their bound values
interpolated, bypassing the suppression added for #207.

The scan guard from #222 and the queue-depth sampler from #224 each gated
green against a next that lacked the other; both landed and next went red.

Converted to Find. The emitted SQL is identical apart from placeholders,
and both paths parse the anonymous dest schema the same way, so the
queue-depth gauges are unchanged.
2026-08-20 07:55:20 +02:00
clawbot aba02bc509 Add a target edit form with headers and timeout fields (closes #127) (#229)
check / check (push) Failing after 2m59s
2026-08-20 07:24:12 +02:00
clawbot c6a9884f86 Take an exclusive lock on DATA_DIR at startup (closes #201) (#220)
check / check (push) Superseded by a newer commit; never tested
2026-08-20 07:23:00 +02:00
clawbot 5af161ef60 Log SQL with placeholders, never bound values (closes #207) (#222)
check / check (push) Superseded by a newer commit; never tested
2026-08-20 07:20:59 +02:00
clawbot 4cc83b2326 Expose delivery metrics on /metrics (closes #209) (#224)
check / check (push) Superseded by a newer commit; never tested
2026-08-20 07:19:04 +02:00
clawbot a13e5b7ded Shut down the app when the listener fails (closes #200) (#218)
check / check (push) Successful in 3m48s
2026-08-20 06:42:36 +02:00