Compare commits
4 Commits
2a3d260ee9
...
next
| Author | SHA1 | Date | |
|---|---|---|---|
| a83e8fe654 | |||
| 687405993e | |||
| 03cd1859d7 | |||
| f0512f1c3c |
155
README.md
155
README.md
@@ -114,6 +114,116 @@ TTY detection, and security headers are always applied.
|
|||||||
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
||||||
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
||||||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
|
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
|
||||||
|
| `ALLOWED_EGRESS_CIDRS` | CIDRs that delivery targets may reach despite the SSRF blocklist. Read [Allowing egress to your own network](#allowing-egress-to-your-own-network) before setting it | `""` (none) |
|
||||||
|
|
||||||
|
#### Allowing egress to your own network
|
||||||
|
|
||||||
|
By default every delivery target must resolve to a public address. The
|
||||||
|
private and reserved ranges — RFC 1918, loopback, CGNAT, link-local and
|
||||||
|
the rest — are refused, which stops a target from being used to make
|
||||||
|
webhooker probe the network it sits in.
|
||||||
|
|
||||||
|
That default is also inconvenient for the thing webhooker is mostly
|
||||||
|
for: taking a public webhook and forwarding it to something on your own
|
||||||
|
network. A container on the same Docker network, a box on `10.x`, a
|
||||||
|
service on `127.0.0.1` — all refused, until you name them.
|
||||||
|
|
||||||
|
`ALLOWED_EGRESS_CIDRS` is a comma-separated list of CIDR blocks (a bare
|
||||||
|
address such as `10.0.0.7` is accepted and treated as a single host),
|
||||||
|
for example `10.0.0.0/8, 172.17.0.0/16`. Addresses inside those blocks
|
||||||
|
become valid delivery destinations. Everything outside them keeps the
|
||||||
|
default answer, so this only ever adds destinations — it never removes
|
||||||
|
any, and it cannot narrow what was already reachable.
|
||||||
|
|
||||||
|
**The risk, plainly.** Each block you list is a network that anyone who
|
||||||
|
can create a delivery target can now make this process issue requests
|
||||||
|
into, and read the response body back out of via the delivery log. That
|
||||||
|
is server-side request forgery, deliberately enabled and scoped by you.
|
||||||
|
A webhooker admin account is therefore as trusted as the narrowest
|
||||||
|
thing on those networks: an unauthenticated admin panel, a database
|
||||||
|
listening without a password, or an internal API that trusts its
|
||||||
|
network position is reachable through it. List the smallest blocks that
|
||||||
|
cover the destinations you actually deliver to — prefer
|
||||||
|
`10.1.2.3/32` over `10.0.0.0/8` — and never list a block wider than the
|
||||||
|
network you are willing to expose.
|
||||||
|
|
||||||
|
Listing `0.0.0.0/0` or `::/0` opens **every** other private and
|
||||||
|
reserved range at once — loopback, RFC 1918, CGNAT, ULA, the lot. It is
|
||||||
|
a functional off switch for everything except the addresses listed as
|
||||||
|
unconditionally blocked below, and it makes any delivery target a probe
|
||||||
|
into your entire network and this host's own loopback services. Do not
|
||||||
|
list it.
|
||||||
|
|
||||||
|
Two things this setting cannot do:
|
||||||
|
|
||||||
|
- **It cannot turn the guard off.** There is no boolean, and no value
|
||||||
|
that disables SSRF protection wholesale. The guard is always on and
|
||||||
|
the list is always an allowlist; an empty list (the default) means
|
||||||
|
every private and reserved range stays refused. Note that
|
||||||
|
`0.0.0.0/0` gets you most of the way there anyway, per above.
|
||||||
|
- **It cannot open link-local, or a cloud metadata endpoint that
|
||||||
|
discloses credentials or user data.** An address is on the list below
|
||||||
|
when both of these hold: the provider fixes it, so it cannot collide
|
||||||
|
with anything you run; and reaching it hands out credentials, user
|
||||||
|
data or bootstrap material. Those stay blocked no matter what you
|
||||||
|
list, including when you list them outright or list a supernet such
|
||||||
|
as `0.0.0.0/0`, `::/0`, `fd00::/8` or `100.64.0.0/10`. Treat this as
|
||||||
|
best effort rather than a guarantee — it is a hand-maintained list
|
||||||
|
and the caveat below the table applies:
|
||||||
|
|
||||||
|
| Blocked unconditionally | What it is |
|
||||||
|
| ----------------------- | ---------- |
|
||||||
|
| `169.254.0.0/16` | IPv4 link-local, carrying `169.254.169.254` (AWS, Azure, DigitalOcean, Hetzner, OpenStack and others — not Alibaba, which uses `100.100.100.200` below) |
|
||||||
|
| `fe80::/10` | IPv6 link-local |
|
||||||
|
| `fd00:ec2::254/128` | AWS IPv6 IMDS |
|
||||||
|
| `fd00:ec2::23/128` | AWS EKS Pod Identity Agent |
|
||||||
|
| `fd20:ce::254/128` | GCP metadata for IPv6-only instances |
|
||||||
|
| `fd00:c1::a9fe:a9fe/128` | Oracle OCI IMDS over IPv6 |
|
||||||
|
| `fd00:42::42/128` | Scaleway metadata over IPv6 |
|
||||||
|
| `fd00:a9fe:a9fe::1/128` | Linode/Akamai metadata over IPv6 |
|
||||||
|
| `100.100.100.200/32` | Alibaba Cloud metadata, inside CGNAT |
|
||||||
|
| `192.0.0.192/32` | Oracle Cloud Classic metadata |
|
||||||
|
| `::a9fe:a9fe/128` | `169.254.169.254` as an IPv4-compatible IPv6 address |
|
||||||
|
| `64:ff9b::a9fe:a9fe/128` | `169.254.169.254` behind the NAT64 well-known prefix |
|
||||||
|
|
||||||
|
The IPv4-mapped form `::ffff:169.254.169.254` is covered by the
|
||||||
|
`169.254.0.0/16` entry. Reaching any of these is credential or
|
||||||
|
user-data theft rather than delivery to an internal service. Every
|
||||||
|
entry outside the two link-local blocks is a single address, so
|
||||||
|
blocking it costs you nothing else on the network around it.
|
||||||
|
|
||||||
|
The six ULA entries, all inside `fd00::/8`, are why this matters in
|
||||||
|
practice: `fd00::/8` is an ordinary block to allowlist for your own
|
||||||
|
IPv6 network, and without those host routes that one line would hand
|
||||||
|
out cloud credentials on five providers at once. There is only one
|
||||||
|
`/8` involved — `fd20:ce::254` masks into `fd00::/8` as well — and
|
||||||
|
the six endpoints are five providers because AWS appears twice, IMDS
|
||||||
|
and EKS Pod Identity. Several of them are described as "link-local" —
|
||||||
|
or even "localhost" — in their own vendor's documentation, but they
|
||||||
|
are ULAs and `fe80::/10` does not cover them.
|
||||||
|
|
||||||
|
Every entry above is reserved space. All but the last two are already
|
||||||
|
refused with no allowlist set, and listing them here is only what
|
||||||
|
stops an allowlist from reopening them; the last two are the alternate
|
||||||
|
encodings, which the default blocklist does not match. A publicly
|
||||||
|
routable metadata address is not listed here, because nothing on this
|
||||||
|
list can be reopened and blocking one that way would leave you no
|
||||||
|
escape hatch at all.
|
||||||
|
|
||||||
|
This list is not exhaustive of every cloud's metadata address — if
|
||||||
|
yours is not here, do not allowlist the block that contains it.
|
||||||
|
|
||||||
|
The list is applied at one place in the code, which both target
|
||||||
|
creation and delivery consult, so a URL that the target form accepts is
|
||||||
|
one that delivery will actually attempt — the two cannot disagree.
|
||||||
|
Delivery re-resolves and re-checks the destination at dial time, so a
|
||||||
|
hostname that resolves to an allowed address during validation and a
|
||||||
|
different one later (DNS rebinding) is still refused unless the new
|
||||||
|
address is also allowed.
|
||||||
|
|
||||||
|
A set but unparseable value aborts startup. When the list is non-empty
|
||||||
|
webhooker logs it at startup, blocks and all, so the hole is visible in
|
||||||
|
the log of any deployment that has one.
|
||||||
|
|
||||||
#### Metrics credentials
|
#### Metrics credentials
|
||||||
|
|
||||||
@@ -272,8 +382,9 @@ additionally be a number in the range 1–65535,
|
|||||||
`RECEIVER_RATE_LIMIT` must be at least 1,
|
`RECEIVER_RATE_LIMIT` must be at least 1,
|
||||||
`RETENTION_SWEEP_INTERVAL` must be greater than zero (it is a ticker
|
`RETENTION_SWEEP_INTERVAL` must be greater than zero (it is a ticker
|
||||||
period, so `0s` or a negative value would crash the reaper after
|
period, so `0s` or a negative value would crash the reaper after
|
||||||
startup), and every entry in `TRUSTED_PROXIES` must be a CIDR block or
|
startup), and every entry in `TRUSTED_PROXIES` and
|
||||||
a bare IP address. `SESSION_IDLE_TIMEOUT` is the exception: a
|
`ALLOWED_EGRESS_CIDRS` must be a CIDR block or a bare IP address.
|
||||||
|
`SESSION_IDLE_TIMEOUT` is the exception: a
|
||||||
non-positive value there means idle expiry is disabled, not invalid.
|
non-positive value there means idle expiry is disabled, not invalid.
|
||||||
|
|
||||||
Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the
|
Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the
|
||||||
@@ -1072,17 +1183,29 @@ reserved: a configured one deliberately overrides the event's.
|
|||||||
|
|
||||||
**Redirects.** A redirect from an `http` target's destination is
|
**Redirects.** A redirect from an `http` target's destination is
|
||||||
followed, up to ten hops, and the delivery's recorded status and body
|
followed, up to ten hops, and the delivery's recorded status and body
|
||||||
come from the final hop. Configured `headers` are dropped as soon as a
|
come from the final hop. One rule governs every header the delivery
|
||||||
hop leaves the origin the target names — a different host, a different
|
carries for someone else — the configured `headers` and the inbound
|
||||||
port, or a step down from `https` to `http` — because a configured
|
event headers forwarded from the sender alike: **a hop that leaves the
|
||||||
header is routinely a credential (`X-Api-Key`, `PRIVATE-TOKEN`) and an
|
origin the target names carries none of them.** Leaving the origin
|
||||||
|
means a different host, a different port, or a step down from `https`
|
||||||
|
to `http`. Both classes routinely carry a secret — a configured
|
||||||
|
`X-Api-Key` or `PRIVATE-TOKEN`, an inbound `X-Hub-Signature` — and an
|
||||||
open redirect at the destination would otherwise hand it to a host the
|
open redirect at the destination would otherwise hand it to a host the
|
||||||
operator never chose. `net/http` already does this for `Authorization`
|
operator never chose. `net/http` already does this for `Authorization`
|
||||||
and `Cookie`. Redirects within the target's own origin keep every
|
and `Cookie`. The delivery path's own headers (`Content-Type`,
|
||||||
configured header, so a destination that redirects its own paths is
|
`User-Agent`) are not origin-scoped and always travel, so a body
|
||||||
unaffected. Each hop is dialled through the same SSRF guard as the
|
preserved across a `307` is still typed. A `301`, `302` or `303` is a
|
||||||
first, so a redirect aimed at a private or reserved address is refused
|
different matter, and this is `net/http`'s behaviour rather than
|
||||||
at connect time.
|
webhooker's: the POST becomes a GET and the event body and its
|
||||||
|
`Content-Type` are dropped, so the destination the chain ends at
|
||||||
|
receives no event at all — and the delivery is still recorded
|
||||||
|
`Delivered` on that hop's `2xx`. Redirects within the target's own
|
||||||
|
origin keep everything, so a destination that redirects its own paths
|
||||||
|
is unaffected; the drop is per hop rather than permanent, so a chain
|
||||||
|
that returns to the configured origin carries the headers again,
|
||||||
|
exactly as `net/http` treats `Authorization`. Each hop is dialled
|
||||||
|
through the same SSRF guard as the first, so a redirect aimed at a
|
||||||
|
private or reserved address is refused at connect time.
|
||||||
|
|
||||||
#### APIKey
|
#### APIKey
|
||||||
|
|
||||||
@@ -2367,7 +2490,15 @@ check, see [The login endpoint](#the-login-endpoint).
|
|||||||
ranges (RFC 1918, loopback, link-local, cloud metadata) are blocked
|
ranges (RFC 1918, loopback, link-local, cloud metadata) are blocked
|
||||||
both at target creation time (URL validation) and at delivery time
|
both at target creation time (URL validation) and at delivery time
|
||||||
(custom HTTP transport with SSRF-safe dialer that validates resolved
|
(custom HTTP transport with SSRF-safe dialer that validates resolved
|
||||||
IPs before connecting, preventing DNS rebinding attacks)
|
IPs before connecting, preventing DNS rebinding attacks). Both paths
|
||||||
|
route through a single decision function, so they cannot disagree
|
||||||
|
about a destination. An operator can permit specific blocks with
|
||||||
|
[`ALLOWED_EGRESS_CIDRS`](#allowing-egress-to-your-own-network); the
|
||||||
|
guard cannot be switched off, and link-local plus a
|
||||||
|
[pinned set](#allowing-egress-to-your-own-network) of known cloud
|
||||||
|
metadata endpoints — several of which are ULAs outside link-local —
|
||||||
|
stay blocked whatever is listed, though listing `0.0.0.0/0` or
|
||||||
|
`::/0` does open every other private range
|
||||||
- **Login limiting is inverted, deliberately.** The login `POST` has
|
- **Login limiting is inverted, deliberately.** The login `POST` has
|
||||||
no pre-emptive rate limiter in front of it. Credentials are
|
no pre-emptive rate limiter in front of it. Credentials are
|
||||||
verified first and only a _failed_ attempt spends budget, so a
|
verified first and only a _failed_ attempt spends budget, so a
|
||||||
|
|||||||
113
TODO.md
113
TODO.md
@@ -18,68 +18,52 @@ Issue branches do NOT touch this file — the manager maintains it on
|
|||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
pre-1.0. No git tags exist. `main` (4f5ecb1) is a working webhook proxy
|
1.0.0 is complete: 55 closed, 0 open. `next` (6874059) is 62 commits
|
||||||
with auth, CSRF/SSRF protections, login rate limiting, Slack target,
|
ahead of `main` and a strict fast-forward. No git tags exist yet.
|
||||||
event retention (#63), the database archiving target (#43), the admin
|
|
||||||
password change flow (#65), policy compliance (#6), pinned lint tooling
|
|
||||||
(#55), and fail-loud configuration parsing (#80).
|
|
||||||
|
|
||||||
`next` is green — verified both by CI and by cache-defeated container
|
The bar was not "the milestone is empty" but "sneak can deploy this and
|
||||||
runs (`docker build --no-cache-filter=lint --no-cache-filter=builder`) —
|
use it in low-volume production". Every gap the deployability audit
|
||||||
but the **1.0.0 milestone is no longer complete**. It was reopened on
|
named against that bar is now closed:
|
||||||
2026-08-20 by a code-level deployability audit that ran the service end
|
|
||||||
to end (verdict:
|
|
||||||
https://git.eeqj.de/sneak/webhooker/issues/33#issuecomment-66686).
|
|
||||||
|
|
||||||
The bar for 1.0 is not "the milestone is empty" but "sneak can deploy
|
- `DATA_DIR` locking, so two instances cannot both deliver
|
||||||
this and use it in low-volume production". The audit found the gap
|
(https://git.eeqj.de/sneak/webhooker/issues/201)
|
||||||
between those two: two instances on one `DATA_DIR` both deliver
|
- shutdown on listener failure, rather than a live non-serving process
|
||||||
(reproduced), a failed listen leaves a live non-serving process that
|
(https://git.eeqj.de/sneak/webhooker/issues/200)
|
||||||
restart policies never fire on, there is no inbound authentication of
|
- inbound signature verification
|
||||||
any kind, delivery failures render as a bare word with no status code or
|
(https://git.eeqj.de/sneak/webhooker/issues/67)
|
||||||
error, a terminally failed delivery can never be replayed, the SSRF
|
- per-attempt delivery detail in the event log
|
||||||
blocklist has no escape hatch so the proxy cannot forward to your own
|
(https://git.eeqj.de/sneak/webhooker/issues/202)
|
||||||
network at all, and target credentials leak into the per-webhook event
|
- replay of a terminally failed delivery
|
||||||
databases.
|
(https://git.eeqj.de/sneak/webhooker/issues/203)
|
||||||
|
- `ALLOWED_EGRESS_CIDRS`, an allowlist escape hatch for the SSRF guard
|
||||||
|
(https://git.eeqj.de/sneak/webhooker/issues/204)
|
||||||
|
- the three credential exposures
|
||||||
|
(https://git.eeqj.de/sneak/webhooker/issues/205,
|
||||||
|
https://git.eeqj.de/sneak/webhooker/issues/206,
|
||||||
|
https://git.eeqj.de/sneak/webhooker/issues/207)
|
||||||
|
|
||||||
One caveat on reading a green check, narrower than it used to be. A
|
One caveat on reading a green check: a docs-only commit deliberately
|
||||||
docs-only commit deliberately replays from the layer cache (#119), so a
|
replays from the layer cache
|
||||||
green status on such a commit evidences a replay rather than an executed
|
(https://git.eeqj.de/sneak/webhooker/issues/119), so a green status on
|
||||||
run; a code commit invalidates the `COPY` layer and genuinely executes.
|
such a commit evidences a replay rather than an executed run. A code
|
||||||
Superseded runs are no longer the hazard they were: before #152 they
|
commit invalidates the `COPY` layer and genuinely executes.
|
||||||
were recorded as `skipped` and rolled up green, and before #119 a warm
|
|
||||||
layer cache let the gate report success without executing anything,
|
|
||||||
replaying the previous build's console log so the lie looked like a real
|
|
||||||
run. Both are fixed. Note: `TODO.md` was deliberately
|
|
||||||
deleted from this repo in f9a9569 (2026-03-01, #6); its content was
|
|
||||||
folded into the README TODO section, which this draft reconstructs as
|
|
||||||
of 2026-07-06.
|
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
Clear the reopened 1.0.0 milestone. The milestone PR
|
Merge the milestone PR (https://git.eeqj.de/sneak/webhooker/pulls/111)
|
||||||
(https://git.eeqj.de/sneak/webhooker/pulls/111) is held: it carries a
|
and tag `v1.0.0`. It is `merge-ready` and assigned to sneak; nothing
|
||||||
`WIP: ` prefix, no labels and is assigned to `clawbot`, and it stays
|
else gates it.
|
||||||
that way until the milestone is empty. Correctness first — the
|
|
||||||
duplicate-delivery lock and the listen-failure shutdown — then the
|
|
||||||
operability gaps that make the service usable in production, then the
|
|
||||||
three credential exposures.
|
|
||||||
|
|
||||||
Three items belong to the owner, none of them blocking. #150 was decided
|
Post-1.0 follow-ups are open, none blocking the tag:
|
||||||
by the manager rather than left to stall the queue and is flagged on the
|
https://git.eeqj.de/sneak/webhooker/issues/245,
|
||||||
issue for reversal if that call was wrong. #112 (whether `Completed
|
https://git.eeqj.de/sneak/webhooker/issues/246,
|
||||||
Steps` should exist at all, given it once conflicted on every unit) is
|
https://git.eeqj.de/sneak/webhooker/issues/247 and
|
||||||
unanswered; the provisional ruling in force is that issue branches do
|
https://git.eeqj.de/sneak/webhooker/issues/248. Also still open and
|
||||||
not touch this file. #198 records that `make test` is past the org 20s
|
unmilestoned: https://git.eeqj.de/sneak/webhooker/issues/193 (a design
|
||||||
target — 46s of test execution inside a 62.8s CI layer — and turns on
|
question, not a defect), https://git.eeqj.de/sneak/webhooker/issues/198
|
||||||
which quantity the 60s hard cap governs; it is scoped as the improvement
|
(`make test` is past the org 20s target) and
|
||||||
bug the 20-60s band requires, and should be milestoned instead if the
|
https://git.eeqj.de/sneak/webhooker/issues/212 (encrypting target config
|
||||||
cap is read as covering the whole invocation.
|
at rest).
|
||||||
|
|
||||||
After the tag, the largest open cluster is the unmilestoned follow-up
|
|
||||||
backlog these units generated: #183, #184, #185, #190, #191, #193, #198,
|
|
||||||
#211 and #212 (encrypting target config at rest, split out of the
|
|
||||||
credential-leak fix because it needs a key-rotation and re-wrap story).
|
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
@@ -308,14 +292,16 @@ credential-leak fix because it needs a key-rotation and re-wrap story).
|
|||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- Manual event redelivery from the web UI — the "Replay" capability the
|
- Delivery status and retry management UI. Replay of a terminally
|
||||||
README describes as planned. No redelivery code exists anywhere in the
|
failed delivery and per-attempt detail already landed
|
||||||
tree; events are stored in full, which is all it would be built on
|
(https://git.eeqj.de/sneak/webhooker/issues/203,
|
||||||
- Delivery status and retry management UI
|
https://git.eeqj.de/sneak/webhooker/issues/202)
|
||||||
- Per-webhook rate limiting in the receiver handler (per-webhook config
|
- Per-webhook rate limiting in the receiver handler (per-webhook config
|
||||||
plus handler enforcement; global limits must not apply to receiver
|
plus handler enforcement; global limits must not apply to receiver
|
||||||
endpoints)
|
endpoints)
|
||||||
- Webhook signature verification for GitHub and Stripe HMAC formats
|
- Stripe HMAC signature verification. The GitHub and GitLab schemes
|
||||||
|
landed with inbound verification
|
||||||
|
(https://git.eeqj.de/sneak/webhooker/issues/67)
|
||||||
- API key authentication for programmatic access (APIKey model exists;
|
- API key authentication for programmatic access (APIKey model exists;
|
||||||
Bearer token middleware does not)
|
Bearer token middleware does not)
|
||||||
- REST API v1
|
- REST API v1
|
||||||
@@ -325,9 +311,10 @@ credential-leak fix because it needs a key-rotation and re-wrap story).
|
|||||||
- OpenAPI specification
|
- OpenAPI specification
|
||||||
- Analytics dashboard: success rates, response times, volume
|
- Analytics dashboard: success rates, response times, volume
|
||||||
- A remember-me option at login
|
- A remember-me option at login
|
||||||
- Password reset flow for a forgotten password. The authenticated
|
- Password reset flow for a forgotten password over the web. The
|
||||||
password *change* flow already landed on `main` (#65); reset does not
|
authenticated password *change* flow already landed, and a lost
|
||||||
exist
|
password is recoverable from the console with `webhooker resetpw`
|
||||||
|
(https://git.eeqj.de/sneak/webhooker/issues/208)
|
||||||
- Later, nice to have
|
- Later, nice to have
|
||||||
- email delivery target type
|
- email delivery target type
|
||||||
- SNS and S3 delivery targets
|
- SNS and S3 delivery targets
|
||||||
|
|||||||
@@ -157,6 +157,10 @@ func newApp() *fx.App {
|
|||||||
session.New,
|
session.New,
|
||||||
handlers.New,
|
handlers.New,
|
||||||
middleware.New,
|
middleware.New,
|
||||||
|
// The one SSRF guard both target-creation validation
|
||||||
|
// and the delivery dialer consult, so they cannot
|
||||||
|
// disagree about a destination.
|
||||||
|
delivery.NewGuard,
|
||||||
delivery.New,
|
delivery.New,
|
||||||
delivery.NewArchiveSweeper,
|
delivery.NewArchiveSweeper,
|
||||||
// Wire *delivery.Engine as delivery.Notifier so the
|
// Wire *delivery.Engine as delivery.Notifier so the
|
||||||
|
|||||||
@@ -128,6 +128,22 @@ type Config struct {
|
|||||||
// clients.
|
// clients.
|
||||||
TrustedProxies []netip.Prefix
|
TrustedProxies []netip.Prefix
|
||||||
|
|
||||||
|
// AllowedEgressCIDRs is the set of networks a delivery target
|
||||||
|
// may reach even though the SSRF guard's default blocklist
|
||||||
|
// covers them. It is empty unless ALLOWED_EGRESS_CIDRS is set,
|
||||||
|
// and empty means every private/reserved range stays refused.
|
||||||
|
//
|
||||||
|
// This only ever adds destinations to what the guard would
|
||||||
|
// otherwise refuse. The guard itself is always on: there is no
|
||||||
|
// setting that disables SSRF protection, and delivery's
|
||||||
|
// alwaysBlockedNetworks stays blocked no matter what is listed
|
||||||
|
// here. That set is link-local plus the cloud metadata
|
||||||
|
// endpoints outside it that disclose credentials or user data
|
||||||
|
// at a provider-fixed address; it is not exhaustive of every
|
||||||
|
// cloud's metadata address. See alwaysBlockedNetworks for the
|
||||||
|
// authoritative list and the criterion it is built from.
|
||||||
|
AllowedEgressCIDRs []netip.Prefix
|
||||||
|
|
||||||
params *ConfigParams
|
params *ConfigParams
|
||||||
log *slog.Logger
|
log *slog.Logger
|
||||||
}
|
}
|
||||||
@@ -472,6 +488,11 @@ func loadFromEnv() (*Config, error) {
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
allowedEgressCIDRs, err := envPrefixList("ALLOWED_EGRESS_CIDRS")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
metricsUsername, metricsPassword, err := resolveMetricsAuth()
|
metricsUsername, metricsPassword, err := resolveMetricsAuth()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -490,9 +511,49 @@ func loadFromEnv() (*Config, error) {
|
|||||||
SessionIdleTimeout: sessionIdleTimeout,
|
SessionIdleTimeout: sessionIdleTimeout,
|
||||||
ReceiverRateLimit: receiverRateLimit,
|
ReceiverRateLimit: receiverRateLimit,
|
||||||
TrustedProxies: trustedProxies,
|
TrustedProxies: trustedProxies,
|
||||||
|
AllowedEgressCIDRs: allowedEgressCIDRs,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// PrefixStrings renders a prefix list as its CIDR strings, for
|
||||||
|
// logging a list an operator has to be able to read back.
|
||||||
|
func PrefixStrings(prefixes []netip.Prefix) []string {
|
||||||
|
out := make([]string, 0, len(prefixes))
|
||||||
|
|
||||||
|
for _, prefix := range prefixes {
|
||||||
|
out = append(out, prefix.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// warnEgressAllowlist logs the effective ALLOWED_EGRESS_CIDRS
|
||||||
|
// whenever it is non-empty.
|
||||||
|
//
|
||||||
|
// It prints the blocks themselves rather than a count, because
|
||||||
|
// this is the one setting that lets a delivery target reach the
|
||||||
|
// host's own network: an operator reading the startup log has to
|
||||||
|
// be able to see exactly which hole is open. Silence means the
|
||||||
|
// list is empty and the SSRF guard is refusing every
|
||||||
|
// private/reserved range, which is the default.
|
||||||
|
func (c *Config) warnEgressAllowlist(log *slog.Logger) {
|
||||||
|
if len(c.AllowedEgressCIDRs) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Warn(
|
||||||
|
"ALLOWED_EGRESS_CIDRS lets delivery targets reach these "+
|
||||||
|
"otherwise-blocked private/reserved networks. Anyone "+
|
||||||
|
"who can create a delivery target can now make this "+
|
||||||
|
"process issue requests into them, and read back the "+
|
||||||
|
"response. Link-local and the known cloud instance "+
|
||||||
|
"metadata endpoints outside it stay blocked "+
|
||||||
|
"regardless of what is listed here.",
|
||||||
|
"allowedEgressCIDRs",
|
||||||
|
strings.Join(PrefixStrings(c.AllowedEgressCIDRs), ","),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
// warnSharedRateLimitBucket logs a startup warning whenever
|
// warnSharedRateLimitBucket logs a startup warning whenever
|
||||||
// TRUSTED_PROXIES is empty, in any environment.
|
// TRUSTED_PROXIES is empty, in any environment.
|
||||||
//
|
//
|
||||||
@@ -574,11 +635,13 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
|
|||||||
"sessionIdleTimeout", s.SessionIdleTimeout.String(),
|
"sessionIdleTimeout", s.SessionIdleTimeout.String(),
|
||||||
"receiverRateLimit", s.ReceiverRateLimit,
|
"receiverRateLimit", s.ReceiverRateLimit,
|
||||||
"trustedProxies", len(s.TrustedProxies),
|
"trustedProxies", len(s.TrustedProxies),
|
||||||
|
"allowedEgressCIDRs", len(s.AllowedEgressCIDRs),
|
||||||
"hasSentryDSN", s.SentryDSN != "",
|
"hasSentryDSN", s.SentryDSN != "",
|
||||||
"hasMetricsAuth", s.MetricsAuthEnabled(),
|
"hasMetricsAuth", s.MetricsAuthEnabled(),
|
||||||
)
|
)
|
||||||
|
|
||||||
s.warnSharedRateLimitBucket(log)
|
s.warnSharedRateLimitBucket(log)
|
||||||
|
s.warnEgressAllowlist(log)
|
||||||
|
|
||||||
return s, nil
|
return s, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -663,6 +663,187 @@ func testTrustedProxiesSuccess(
|
|||||||
assert.Equal(t, expected, got)
|
assert.Equal(t, expected, got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestAllowedEgressCIDRs covers ALLOWED_EGRESS_CIDRS, the escape
|
||||||
|
// hatch that lets a self-hosted deployment forward to its own
|
||||||
|
// network. Unset it must stay empty, so the SSRF guard keeps
|
||||||
|
// refusing every private/reserved range; a set-but-unparseable
|
||||||
|
// value must abort startup naming the variable rather than
|
||||||
|
// silently running with a list the operator did not write.
|
||||||
|
func TestAllowedEgressCIDRs(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
set bool
|
||||||
|
value string
|
||||||
|
expected []string
|
||||||
|
expectError bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: caseUnsetUsesDefault,
|
||||||
|
set: false,
|
||||||
|
expected: []string{},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty value yields empty list",
|
||||||
|
set: true,
|
||||||
|
value: "",
|
||||||
|
expected: []string{},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: caseValidValueParsed,
|
||||||
|
set: true,
|
||||||
|
value: cidrPrivateV4,
|
||||||
|
expected: []string{cidrPrivateV4},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "multiple blocks with whitespace",
|
||||||
|
set: true,
|
||||||
|
value: " 10.0.0.0/8 , 127.0.0.0/8 ",
|
||||||
|
expected: []string{cidrPrivateV4, "127.0.0.0/8"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "bare address becomes a single host",
|
||||||
|
set: true,
|
||||||
|
value: "172.17.0.5",
|
||||||
|
expected: []string{"172.17.0.5/32"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: caseUnparseableFails,
|
||||||
|
set: true,
|
||||||
|
value: cidrPrivateV4 + ",not-an-address",
|
||||||
|
expectError: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "out-of-range prefix length fails startup",
|
||||||
|
set: true,
|
||||||
|
value: "10.0.0.0/33",
|
||||||
|
expectError: true,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
|
// is incompatible with parallel subtests.
|
||||||
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
|
if tt.set {
|
||||||
|
t.Setenv("ALLOWED_EGRESS_CIDRS", tt.value)
|
||||||
|
} else {
|
||||||
|
require.NoError(
|
||||||
|
t, os.Unsetenv("ALLOWED_EGRESS_CIDRS"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
if tt.expectError {
|
||||||
|
expectStartupErrorFor(
|
||||||
|
t, "ALLOWED_EGRESS_CIDRS", config.ErrInvalidCIDR,
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
testAllowedEgressCIDRsSuccess(t, tt.expected)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testAllowedEgressCIDRsSuccess(
|
||||||
|
t *testing.T,
|
||||||
|
expected []string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var cfg *config.Config
|
||||||
|
|
||||||
|
app := fxtest.New(
|
||||||
|
t,
|
||||||
|
fx.Provide(
|
||||||
|
globals.New,
|
||||||
|
logger.New,
|
||||||
|
config.New,
|
||||||
|
),
|
||||||
|
fx.Populate(&cfg),
|
||||||
|
)
|
||||||
|
require.NoError(t, app.Err())
|
||||||
|
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
defer app.RequireStop()
|
||||||
|
|
||||||
|
assert.Equal(
|
||||||
|
t, expected, config.PrefixStrings(cfg.AllowedEgressCIDRs),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEgressAllowlistWarning covers the startup log that shows an
|
||||||
|
// operator the hole ALLOWED_EGRESS_CIDRS opened. It must stay
|
||||||
|
// silent on the default (empty) list and, when set, print the
|
||||||
|
// blocks themselves rather than a count.
|
||||||
|
func TestEgressAllowlistWarning(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
allowed string
|
||||||
|
expectWarning bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "empty allowlist is quiet",
|
||||||
|
expectWarning: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "non-empty allowlist warns",
|
||||||
|
allowed: "10.0.0.0/8,127.0.0.0/8",
|
||||||
|
expectWarning: true,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
|
// is incompatible with parallel subtests.
|
||||||
|
t.Setenv("WEBHOOKER_ENVIRONMENT", config.EnvironmentDev)
|
||||||
|
|
||||||
|
if tt.allowed == "" {
|
||||||
|
require.NoError(
|
||||||
|
t, os.Unsetenv("ALLOWED_EGRESS_CIDRS"),
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
t.Setenv("ALLOWED_EGRESS_CIDRS", tt.allowed)
|
||||||
|
}
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
log := slog.New(slog.NewJSONHandler(
|
||||||
|
&buf, &slog.HandlerOptions{
|
||||||
|
Level: slog.LevelDebug,
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
require.NoError(
|
||||||
|
t, config.WarnEgressAllowlistForTest(log),
|
||||||
|
)
|
||||||
|
|
||||||
|
if !tt.expectWarning {
|
||||||
|
assert.Empty(t, buf.String())
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
logged := buf.String()
|
||||||
|
|
||||||
|
assert.Contains(t, logged, `"level":"WARN"`)
|
||||||
|
assert.Contains(t, logged, "ALLOWED_EGRESS_CIDRS")
|
||||||
|
// The blocks themselves, not a count: the operator has
|
||||||
|
// to be able to read back which networks are open.
|
||||||
|
assert.Contains(t, logged, "10.0.0.0/8")
|
||||||
|
assert.Contains(t, logged, "127.0.0.0/8")
|
||||||
|
// What stays shut. Asserted on the clause naming the
|
||||||
|
// wider set rather than on "Link-local" alone, so the
|
||||||
|
// string cannot narrow back to link-local only while
|
||||||
|
// the always-blocked set covers ULA, CGNAT and two
|
||||||
|
// public metadata addresses as well.
|
||||||
|
assert.Contains(t, logged, "metadata endpoints outside it")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestSharedRateLimitBucketWarning covers the startup warning that
|
// TestSharedRateLimitBucketWarning covers the startup warning that
|
||||||
// tells an operator a deployment behind a reverse proxy shares one
|
// tells an operator a deployment behind a reverse proxy shares one
|
||||||
// rate-limit bucket between every client, which turns the receiver
|
// rate-limit bucket between every client, which turns the receiver
|
||||||
|
|||||||
@@ -21,6 +21,21 @@ func WarnSharedRateLimitBucketForTest(log *slog.Logger) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// WarnEgressAllowlistForTest loads a Config from the current
|
||||||
|
// environment and emits its egress-allowlist startup warning to
|
||||||
|
// log, so a test can assert both that the warning fires only when
|
||||||
|
// the list is non-empty and that it names the blocks it opened.
|
||||||
|
func WarnEgressAllowlistForTest(log *slog.Logger) error {
|
||||||
|
c, err := loadFromEnv()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
c.warnEgressAllowlist(log)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// EnvBoolForTest exposes envBool.
|
// EnvBoolForTest exposes envBool.
|
||||||
func EnvBoolForTest(key string, defaultValue bool) (bool, error) {
|
func EnvBoolForTest(key string, defaultValue bool) (bool, error) {
|
||||||
return envBool(key, defaultValue)
|
return envBool(key, defaultValue)
|
||||||
|
|||||||
@@ -19,25 +19,26 @@ func newSSRFTestEngine() *delivery.Engine {
|
|||||||
|
|
||||||
client := &http.Client{
|
client := &http.Client{
|
||||||
Timeout: 30 * time.Second,
|
Timeout: 30 * time.Second,
|
||||||
Transport: delivery.NewSSRFSafeTransport(),
|
Transport: delivery.NewTestGuard().
|
||||||
|
NewSSRFSafeTransport(),
|
||||||
}
|
}
|
||||||
|
|
||||||
return delivery.NewTestEngine(log, client, 1)
|
return delivery.NewTestEngine(log, client, 1)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestClientForConfig_TimeoutKeepsSSRFGuard asserts that a
|
// TestClientForRequest_TimeoutKeepsSSRFGuard asserts that a
|
||||||
// client returned by clientForConfig for a config with a
|
// client returned by clientForRequest for a config with a
|
||||||
// per-target timeout still refuses connections to
|
// per-target timeout still refuses connections to
|
||||||
// private/reserved addresses (the timeout must not drop the
|
// private/reserved addresses (the timeout must not drop the
|
||||||
// SSRF-safe transport).
|
// SSRF-safe transport).
|
||||||
func TestClientForConfig_TimeoutKeepsSSRFGuard(t *testing.T) {
|
func TestClientForRequest_TimeoutKeepsSSRFGuard(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
engine := newSSRFTestEngine()
|
engine := newSSRFTestEngine()
|
||||||
|
|
||||||
blocked := []string{
|
blocked := []string{
|
||||||
"http://127.0.0.1/hook",
|
loopbackHookURL,
|
||||||
"http://169.254.169.254/latest/meta-data/",
|
metadataURL,
|
||||||
"http://[fe80::1]/hook",
|
"http://[fe80::1]/hook",
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -50,7 +51,7 @@ func TestClientForConfig_TimeoutKeepsSSRFGuard(t *testing.T) {
|
|||||||
Timeout: 5,
|
Timeout: 5,
|
||||||
}
|
}
|
||||||
|
|
||||||
client := engine.ExportClientForConfig(cfg)
|
client := engine.ExportClientForRequest(cfg, nil)
|
||||||
|
|
||||||
require.NotSame(t, engine.ExportClient(), client,
|
require.NotSame(t, engine.ExportClient(), client,
|
||||||
"a per-target timeout must yield a "+
|
"a per-target timeout must yield a "+
|
||||||
@@ -91,11 +92,11 @@ func TestClientForConfig_TimeoutKeepsSSRFGuard(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestClientForConfig_NoTimeoutUnchanged asserts that a
|
// TestClientForRequest_NoTimeoutUnchanged asserts that a
|
||||||
// config overriding neither the timeout nor the headers gets
|
// request with neither a per-target timeout nor an origin-scoped
|
||||||
// the shared SSRF-safe client unchanged: with no configured
|
// header gets the shared SSRF-safe client unchanged: there is then
|
||||||
// headers there is nothing for a redirect policy to strip.
|
// nothing for a redirect policy to strip.
|
||||||
func TestClientForConfig_NoTimeoutUnchanged(t *testing.T) {
|
func TestClientForRequest_NoTimeoutUnchanged(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
engine := newSSRFTestEngine()
|
engine := newSSRFTestEngine()
|
||||||
@@ -104,7 +105,7 @@ func TestClientForConfig_NoTimeoutUnchanged(t *testing.T) {
|
|||||||
URL: "https://example.com/hook",
|
URL: "https://example.com/hook",
|
||||||
}
|
}
|
||||||
|
|
||||||
client := engine.ExportClientForConfig(cfg)
|
client := engine.ExportClientForRequest(cfg, nil)
|
||||||
|
|
||||||
assert.Same(t, engine.ExportClient(), client,
|
assert.Same(t, engine.ExportClient(), client,
|
||||||
"without a per-target timeout the shared client "+
|
"without a per-target timeout the shared client "+
|
||||||
@@ -112,12 +113,12 @@ func TestClientForConfig_NoTimeoutUnchanged(t *testing.T) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestClientForConfig_HeadersKeepSSRFGuard asserts that the
|
// TestClientForRequest_HeadersKeepSSRFGuard asserts that the
|
||||||
// redirect policy a target's configured headers install is
|
// redirect policy an origin-scoped header installs is added to a
|
||||||
// added to a client that still carries the SSRF-safe
|
// client that still carries the SSRF-safe transport. The guard is
|
||||||
// transport. The guard is a dial hook, so keeping it is what
|
// a dial hook, so keeping it is what makes each redirect hop pass
|
||||||
// makes each redirect hop pass the private-IP check too.
|
// the private-IP check too.
|
||||||
func TestClientForConfig_HeadersKeepSSRFGuard(t *testing.T) {
|
func TestClientForRequest_HeadersKeepSSRFGuard(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
engine := newSSRFTestEngine()
|
engine := newSSRFTestEngine()
|
||||||
@@ -129,10 +130,12 @@ func TestClientForConfig_HeadersKeepSSRFGuard(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
client := engine.ExportClientForConfig(cfg)
|
client := engine.ExportClientForRequest(
|
||||||
|
cfg, []string{"X-Api-Key"},
|
||||||
|
)
|
||||||
|
|
||||||
require.NotNil(t, client.CheckRedirect,
|
require.NotNil(t, client.CheckRedirect,
|
||||||
"configured headers must install a redirect policy",
|
"an origin-scoped header must install a redirect policy",
|
||||||
)
|
)
|
||||||
|
|
||||||
assert.Same(t,
|
assert.Same(t,
|
||||||
|
|||||||
@@ -121,6 +121,7 @@ type EngineParams struct {
|
|||||||
DB *database.Database
|
DB *database.Database
|
||||||
DBManager *database.WebhookDBManager
|
DBManager *database.WebhookDBManager
|
||||||
Logger *logger.Logger
|
Logger *logger.Logger
|
||||||
|
SSRFGuard *Guard
|
||||||
}
|
}
|
||||||
|
|
||||||
// Engine processes queued deliveries in the background
|
// Engine processes queued deliveries in the background
|
||||||
@@ -176,7 +177,7 @@ func New(
|
|||||||
|
|
||||||
e.initTargets(&http.Client{
|
e.initTargets(&http.Client{
|
||||||
Timeout: httpClientTimeout,
|
Timeout: httpClientTimeout,
|
||||||
Transport: NewSSRFSafeTransport(),
|
Transport: params.SSRFGuard.NewSSRFSafeTransport(),
|
||||||
})
|
})
|
||||||
|
|
||||||
e.registerHooks(lc)
|
e.registerHooks(lc)
|
||||||
|
|||||||
@@ -1201,6 +1201,83 @@ func TestDeliverHTTP_TargetTimeout(t *testing.T) {
|
|||||||
iAssertResultFailed(t, db, del.ID)
|
iAssertResultFailed(t, db, del.ID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog pins the size
|
||||||
|
// this engine stores for an oversized response, because the
|
||||||
|
// event log's redaction is written against it: the row holds
|
||||||
|
// exactly maxBodyLog bytes and records nothing about how much
|
||||||
|
// more the remote sent, so a credential echoed across that
|
||||||
|
// boundary reaches the database already severed and no reader
|
||||||
|
// of the row can tell the cut happened.
|
||||||
|
func TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// Padded so the cut falls five bytes before the end of the
|
||||||
|
// echoed webhook URL.
|
||||||
|
const (
|
||||||
|
severedTail = 5
|
||||||
|
overshoot = 100000
|
||||||
|
)
|
||||||
|
|
||||||
|
sent := strings.Repeat(
|
||||||
|
"A",
|
||||||
|
delivery.ExportMaxBodyLog-len(slackWebhookURL)+
|
||||||
|
severedTail,
|
||||||
|
) + slackWebhookURL + strings.Repeat("Z", overshoot)
|
||||||
|
|
||||||
|
s := newISetup(t)
|
||||||
|
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusBadGateway)
|
||||||
|
_, _ = io.WriteString(w, sent)
|
||||||
|
},
|
||||||
|
))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
cfgJSON := iHTTPConfig(ts.URL)
|
||||||
|
|
||||||
|
event := iSeedEvent(
|
||||||
|
t, s.WebhookDB, s.WebhookID, `{"cut":"test"}`,
|
||||||
|
)
|
||||||
|
targetID := uuid.New().String()
|
||||||
|
|
||||||
|
del := iSeedDelivery(
|
||||||
|
t, s.WebhookDB, event.ID, targetID,
|
||||||
|
database.DeliveryStatusPending,
|
||||||
|
)
|
||||||
|
|
||||||
|
bodyStr := event.Body
|
||||||
|
task := iTask(
|
||||||
|
del, event, s.WebhookID, targetID,
|
||||||
|
"cut-target", cfgJSON, 0, 1, &bodyStr,
|
||||||
|
)
|
||||||
|
|
||||||
|
s.Engine.ExportProcessNewTask(context.TODO(), &task)
|
||||||
|
|
||||||
|
results := iResults(t, s.WebhookDB, del.ID)
|
||||||
|
require.Len(t, results, 1)
|
||||||
|
|
||||||
|
stored := results[0].ResponseBody
|
||||||
|
|
||||||
|
assert.Len(
|
||||||
|
t, stored, delivery.ExportMaxBodyLog,
|
||||||
|
"an oversized response is stored at exactly the cap",
|
||||||
|
)
|
||||||
|
assert.Equal(
|
||||||
|
t, sent[:delivery.ExportMaxBodyLog], stored,
|
||||||
|
)
|
||||||
|
assert.NotContains(
|
||||||
|
t, stored, slackWebhookURL,
|
||||||
|
"the echoed URL is severed by the cut",
|
||||||
|
)
|
||||||
|
assert.Contains(
|
||||||
|
t, stored, "T00000000",
|
||||||
|
"the severed prefix still carries the credential",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
// iSeedEventAndDelivery creates event + delivery
|
// iSeedEventAndDelivery creates event + delivery
|
||||||
// for standalone tests.
|
// for standalone tests.
|
||||||
func iSeedEventAndDelivery(
|
func iSeedEventAndDelivery(
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -26,6 +27,12 @@ const (
|
|||||||
ExportRetryChannelSize = retryChannelSize
|
ExportRetryChannelSize = retryChannelSize
|
||||||
ExportDefaultFailureThreshold = defaultFailureThreshold
|
ExportDefaultFailureThreshold = defaultFailureThreshold
|
||||||
ExportDefaultCooldown = defaultCooldown
|
ExportDefaultCooldown = defaultCooldown
|
||||||
|
|
||||||
|
// ExportMaxBodyLog is the cap the engine applies to a
|
||||||
|
// recorded response body. The event log's handling of a cut
|
||||||
|
// response is written against this number, so a test has to
|
||||||
|
// be able to name it.
|
||||||
|
ExportMaxBodyLog = maxBodyLog
|
||||||
)
|
)
|
||||||
|
|
||||||
// ExportIsBlockedIP exposes isBlockedIP for testing.
|
// ExportIsBlockedIP exposes isBlockedIP for testing.
|
||||||
@@ -33,6 +40,26 @@ func ExportIsBlockedIP(ip net.IP) bool {
|
|||||||
return isBlockedIP(ip)
|
return isBlockedIP(ip)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// NewTestGuard builds an SSRF Guard from an explicit egress
|
||||||
|
// allowlist, without going through config. Passing no prefixes
|
||||||
|
// yields the default guard, which blocks every private/reserved
|
||||||
|
// range.
|
||||||
|
func NewTestGuard(allowed ...netip.Prefix) *Guard {
|
||||||
|
return &Guard{allowed: allowed}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExportCheckIP exposes the guard's single decision point, so a
|
||||||
|
// test can assert the policy both the validator and the dialer
|
||||||
|
// inherit without needing a live destination.
|
||||||
|
func (g *Guard) ExportCheckIP(ip net.IP) error {
|
||||||
|
return g.checkIP(ip)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExportAlwaysBlockedNetworks exposes alwaysBlockedNetworks.
|
||||||
|
func ExportAlwaysBlockedNetworks() []*net.IPNet {
|
||||||
|
return alwaysBlockedNetworks
|
||||||
|
}
|
||||||
|
|
||||||
// ExportBlockedNetworks exposes blockedNetworks.
|
// ExportBlockedNetworks exposes blockedNetworks.
|
||||||
func ExportBlockedNetworks() []*net.IPNet {
|
func ExportBlockedNetworks() []*net.IPNet {
|
||||||
return blockedNetworks
|
return blockedNetworks
|
||||||
@@ -44,13 +71,14 @@ func ExportIsForwardableHeader(name string) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ExportApplyRequestHeaders exposes applyRequestHeaders, so a test
|
// ExportApplyRequestHeaders exposes applyRequestHeaders, so a test
|
||||||
// can inspect the header set an outbound delivery actually carries.
|
// can inspect the header set an outbound delivery actually carries
|
||||||
|
// and the origin-scoped names it reports for the redirect policy.
|
||||||
func ExportApplyRequestHeaders(
|
func ExportApplyRequestHeaders(
|
||||||
req *http.Request,
|
req *http.Request,
|
||||||
event *database.Event,
|
event *database.Event,
|
||||||
cfg *HTTPTargetConfig,
|
cfg *HTTPTargetConfig,
|
||||||
) {
|
) []string {
|
||||||
applyRequestHeaders(req, event, cfg)
|
return applyRequestHeaders(req, event, cfg)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExportTruncate exposes truncate for testing.
|
// ExportTruncate exposes truncate for testing.
|
||||||
@@ -160,14 +188,24 @@ func (e *Engine) ExportDoHTTPRequest(
|
|||||||
return e.httpTarget.doHTTPRequest(ctx, cfg, event)
|
return e.httpTarget.doHTTPRequest(ctx, cfg, event)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExportClientForConfig exposes the http target's
|
// ExportClientForRequest exposes the http target's
|
||||||
// clientForConfig.
|
// clientForRequest.
|
||||||
func (e *Engine) ExportClientForConfig(
|
func (e *Engine) ExportClientForRequest(
|
||||||
cfg *HTTPTargetConfig,
|
cfg *HTTPTargetConfig,
|
||||||
|
originScoped []string,
|
||||||
) *http.Client {
|
) *http.Client {
|
||||||
return e.httpTarget.clientForConfig(cfg)
|
return e.httpTarget.clientForRequest(cfg, originScoped)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ErrExportTooManyRedirects exposes the sentinel the redirect
|
||||||
|
// policy returns once a chain exceeds the hop cap. It carries the
|
||||||
|
// Err prefix rather than this file's usual Export one because it
|
||||||
|
// is a sentinel error.
|
||||||
|
var ErrExportTooManyRedirects = errTooManyRedirects
|
||||||
|
|
||||||
|
// ExportMaxDeliveryRedirects exposes the redirect hop cap.
|
||||||
|
const ExportMaxDeliveryRedirects = maxDeliveryRedirects
|
||||||
|
|
||||||
// ExportSameDeliveryOrigin exposes sameDeliveryOrigin.
|
// ExportSameDeliveryOrigin exposes sameDeliveryOrigin.
|
||||||
func ExportSameDeliveryOrigin(origin, dest *url.URL) bool {
|
func ExportSameDeliveryOrigin(origin, dest *url.URL) bool {
|
||||||
return sameDeliveryOrigin(origin, dest)
|
return sameDeliveryOrigin(origin, dest)
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package delivery
|
|||||||
import (
|
import (
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -19,30 +20,33 @@ const schemeHTTPS = "https"
|
|||||||
|
|
||||||
var errTooManyRedirects = errors.New("too many redirects")
|
var errTooManyRedirects = errors.New("too many redirects")
|
||||||
|
|
||||||
// configuredHeaderRedirectPolicy returns a CheckRedirect that
|
// offOriginHeaderPolicy returns a CheckRedirect that drops every
|
||||||
// drops a target's configured headers once a redirect leaves the
|
// origin-scoped header once a redirect leaves the origin the
|
||||||
// origin the operator configured.
|
// operator configured. names is the set applyRequestHeaders
|
||||||
|
// reports: the operator's configured headers and the inbound event
|
||||||
|
// headers this delivery forwarded, under one rule rather than two.
|
||||||
//
|
//
|
||||||
// net/http withholds Authorization and Cookie across a host change
|
// net/http withholds Authorization and Cookie across a host change
|
||||||
// and forwards everything else. A target header is routinely a
|
// and forwards everything else. A target header is routinely a
|
||||||
// credential under another name — X-Api-Key, PRIVATE-TOKEN,
|
// credential under another name — X-Api-Key, PRIVATE-TOKEN,
|
||||||
// X-Auth-Token — so an open redirect at an otherwise trusted
|
// X-Auth-Token — and a forwarded inbound header is routinely a
|
||||||
// destination would hand that credential to a host the operator
|
// sender's signature — X-Hub-Signature — so an open redirect at an
|
||||||
// never named. Redirects are still followed: refusing them would
|
// otherwise trusted destination would hand either to a host the
|
||||||
// break every destination that legitimately redirects and would
|
// operator never named. Redirects are still followed: refusing them
|
||||||
// record the 3xx as the delivery's result.
|
// would break every destination that legitimately redirects and
|
||||||
|
// would record the 3xx as the delivery's result.
|
||||||
|
//
|
||||||
|
// The strip is per hop, not permanent: net/http re-copies the
|
||||||
|
// initial request's headers for every hop, so a chain that returns
|
||||||
|
// to the configured origin carries them again, exactly as net/http
|
||||||
|
// treats Authorization.
|
||||||
//
|
//
|
||||||
// Each hop is dialled through the same SSRF-safe transport, whose
|
// Each hop is dialled through the same SSRF-safe transport, whose
|
||||||
// guard runs per connection, so a redirect aimed at a private or
|
// guard runs per connection, so a redirect aimed at a private or
|
||||||
// reserved address is still refused at connect time.
|
// reserved address is still refused at connect time.
|
||||||
func configuredHeaderRedirectPolicy(
|
func offOriginHeaderPolicy(
|
||||||
headers map[string]string,
|
names []string,
|
||||||
) func(*http.Request, []*http.Request) error {
|
) func(*http.Request, []*http.Request) error {
|
||||||
names := make([]string, 0, len(headers))
|
|
||||||
for name := range headers {
|
|
||||||
names = append(names, http.CanonicalHeaderKey(name))
|
|
||||||
}
|
|
||||||
|
|
||||||
return func(req *http.Request, via []*http.Request) error {
|
return func(req *http.Request, via []*http.Request) error {
|
||||||
if len(via) >= maxDeliveryRedirects {
|
if len(via) >= maxDeliveryRedirects {
|
||||||
return fmt.Errorf(
|
return fmt.Errorf(
|
||||||
@@ -64,7 +68,7 @@ func configuredHeaderRedirectPolicy(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// sameDeliveryOrigin reports whether dest is close enough to the
|
// sameDeliveryOrigin reports whether dest is close enough to the
|
||||||
// configured target URL to keep carrying its configured headers.
|
// configured target URL to keep carrying its origin-scoped headers.
|
||||||
//
|
//
|
||||||
// This is stricter than the rule net/http applies to Authorization:
|
// This is stricter than the rule net/http applies to Authorization:
|
||||||
// the port is part of the comparison (a different port is a
|
// the port is part of the comparison (a different port is a
|
||||||
@@ -83,6 +87,12 @@ func sameDeliveryOrigin(origin, dest *url.URL) bool {
|
|||||||
// originHostPort renders a URL's host for comparison, lowercased
|
// originHostPort renders a URL's host for comparison, lowercased
|
||||||
// and with the scheme's default port normalised away so that
|
// and with the scheme's default port normalised away so that
|
||||||
// "https://h" and "https://h:443" are one origin.
|
// "https://h" and "https://h:443" are one origin.
|
||||||
|
//
|
||||||
|
// The port is joined with net.JoinHostPort rather than a bare
|
||||||
|
// colon: Hostname() unwraps an IPv6 literal's brackets, so
|
||||||
|
// "[2001:db8::1]:8080" and "[2001:db8::1:8080]" — a different
|
||||||
|
// address on a different port — would otherwise render the same
|
||||||
|
// string and pass as one origin.
|
||||||
func originHostPort(u *url.URL) string {
|
func originHostPort(u *url.URL) string {
|
||||||
host := strings.ToLower(u.Hostname())
|
host := strings.ToLower(u.Hostname())
|
||||||
|
|
||||||
@@ -93,5 +103,5 @@ func originHostPort(u *url.URL) string {
|
|||||||
return host
|
return host
|
||||||
}
|
}
|
||||||
|
|
||||||
return host + ":" + port
|
return net.JoinHostPort(host, port)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,11 +3,14 @@ package delivery_test
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/url"
|
"net/url"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
@@ -16,19 +19,22 @@ import (
|
|||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The header these tests configure stands in for the credential
|
// The headers these tests drive stand in for the two classes the
|
||||||
// headers net/http forwards across a host change: it withholds
|
// off-origin rule covers: an operator-configured credential and an
|
||||||
// Authorization and Cookie, and nothing else.
|
// inbound header the delivery path forwards. net/http withholds
|
||||||
|
// Authorization and Cookie across a host change, and nothing else.
|
||||||
const (
|
const (
|
||||||
probeHeaderName = "X-Api-Key"
|
probeHeaderName = "X-Api-Key"
|
||||||
probeHeaderValue = "QQNEVERONTHEWIREQQ"
|
probeHeaderValue = "QQNEVERONTHEWIREQQ"
|
||||||
|
inboundHeaderName = "X-Hub-Signature"
|
||||||
|
inboundHeaderValue = "sha1=QQINBOUNDQQ"
|
||||||
)
|
)
|
||||||
|
|
||||||
// redirectProbe records what the last hop of a redirect chain
|
// redirectProbe records what the last hop of a redirect chain
|
||||||
// actually received.
|
// actually received.
|
||||||
type redirectProbe struct {
|
type redirectProbe struct {
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
seen string
|
seen http.Header
|
||||||
hits int
|
hits int
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -36,24 +42,26 @@ func (p *redirectProbe) serve(
|
|||||||
w http.ResponseWriter, r *http.Request,
|
w http.ResponseWriter, r *http.Request,
|
||||||
) {
|
) {
|
||||||
p.mu.Lock()
|
p.mu.Lock()
|
||||||
p.seen = r.Header.Get(probeHeaderName)
|
p.seen = r.Header.Clone()
|
||||||
p.hits++
|
p.hits++
|
||||||
p.mu.Unlock()
|
p.mu.Unlock()
|
||||||
|
|
||||||
w.WriteHeader(http.StatusOK)
|
w.WriteHeader(http.StatusOK)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *redirectProbe) result() (string, int) {
|
func (p *redirectProbe) result() (http.Header, int) {
|
||||||
p.mu.Lock()
|
p.mu.Lock()
|
||||||
defer p.mu.Unlock()
|
defer p.mu.Unlock()
|
||||||
|
|
||||||
return p.seen, p.hits
|
return p.seen, p.hits
|
||||||
}
|
}
|
||||||
|
|
||||||
// deliverWithConfiguredHeader runs one real delivery of a new task
|
// deliverWithProbeHeaders runs one real delivery of a new task
|
||||||
// through the engine to targetURL, with probeHeaderName set on the
|
// through the engine to targetURL, carrying both probe headers —
|
||||||
// target, and returns the delivery status the engine recorded.
|
// probeHeaderName configured on the target, inboundHeaderName
|
||||||
func deliverWithConfiguredHeader(
|
// forwarded from the event — and returns the delivery status the
|
||||||
|
// engine recorded.
|
||||||
|
func deliverWithProbeHeaders(
|
||||||
t *testing.T, targetURL string,
|
t *testing.T, targetURL string,
|
||||||
) database.DeliveryStatus {
|
) database.DeliveryStatus {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -65,6 +73,13 @@ func deliverWithConfiguredHeader(
|
|||||||
)
|
)
|
||||||
targetID := uuid.New().String()
|
targetID := uuid.New().String()
|
||||||
|
|
||||||
|
inbound, err := json.Marshal(map[string][]string{
|
||||||
|
inboundHeaderName: {inboundHeaderValue},
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
event.Headers = string(inbound)
|
||||||
|
|
||||||
d := iSeedDelivery(
|
d := iSeedDelivery(
|
||||||
t, s.WebhookDB, event.ID, targetID,
|
t, s.WebhookDB, event.ID, targetID,
|
||||||
database.DeliveryStatusPending,
|
database.DeliveryStatusPending,
|
||||||
@@ -96,9 +111,11 @@ func deliverWithConfiguredHeader(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// A 302 to an origin the operator never configured must not carry
|
// A 302 to an origin the operator never configured must not carry
|
||||||
// the credential they configured for the one they did. The chain is
|
// the credential they configured for the one they did, nor the
|
||||||
// still followed, so the delivery is recorded from the final hop.
|
// inbound header this delivery forwarded — one rule for both
|
||||||
func TestDelivery_CrossOriginRedirectDropsConfiguredHeader(
|
// classes. The chain is still followed, so the delivery is recorded
|
||||||
|
// from the final hop.
|
||||||
|
func TestDelivery_CrossOriginRedirectDropsOriginScopedHeaders(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
) {
|
) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -129,17 +146,21 @@ func TestDelivery_CrossOriginRedirectDropsConfiguredHeader(
|
|||||||
))
|
))
|
||||||
defer origin.Close()
|
defer origin.Close()
|
||||||
|
|
||||||
status := deliverWithConfiguredHeader(t, origin.URL)
|
status := deliverWithProbeHeaders(t, origin.URL)
|
||||||
|
|
||||||
seen, hits := probe.result()
|
seen, hits := probe.result()
|
||||||
|
|
||||||
assert.Equal(t, 1, hits,
|
assert.Equal(t, 1, hits,
|
||||||
"the redirect must still be followed",
|
"the redirect must still be followed",
|
||||||
)
|
)
|
||||||
assert.Empty(t, seen,
|
assert.Empty(t, seen.Get(probeHeaderName),
|
||||||
"a configured credential header must not reach an "+
|
"a configured credential header must not reach an "+
|
||||||
"origin the operator did not configure",
|
"origin the operator did not configure",
|
||||||
)
|
)
|
||||||
|
assert.Empty(t, seen.Get(inboundHeaderName),
|
||||||
|
"a forwarded inbound header must not reach an origin "+
|
||||||
|
"the operator did not configure",
|
||||||
|
)
|
||||||
assert.Equal(t,
|
assert.Equal(t,
|
||||||
database.DeliveryStatusDelivered, status,
|
database.DeliveryStatusDelivered, status,
|
||||||
"the final hop's 200 is the delivery's result",
|
"the final hop's 200 is the delivery's result",
|
||||||
@@ -148,8 +169,9 @@ func TestDelivery_CrossOriginRedirectDropsConfiguredHeader(
|
|||||||
|
|
||||||
// Stripping must not fire within the configured origin, or every
|
// Stripping must not fire within the configured origin, or every
|
||||||
// destination that redirects its own path would lose its
|
// destination that redirects its own path would lose its
|
||||||
// credential and start answering 401.
|
// credential and start answering 401 — and would lose the inbound
|
||||||
func TestDelivery_SameOriginRedirectKeepsConfiguredHeader(
|
// signature the receiver verifies.
|
||||||
|
func TestDelivery_SameOriginRedirectKeepsOriginScopedHeaders(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
) {
|
) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -171,15 +193,20 @@ func TestDelivery_SameOriginRedirectKeepsConfiguredHeader(
|
|||||||
))
|
))
|
||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
|
|
||||||
status := deliverWithConfiguredHeader(t, srv.URL+"/hook")
|
status := deliverWithProbeHeaders(t, srv.URL+"/hook")
|
||||||
|
|
||||||
seen, hits := probe.result()
|
seen, hits := probe.result()
|
||||||
|
|
||||||
assert.Equal(t, 1, hits)
|
assert.Equal(t, 1, hits)
|
||||||
assert.Equal(t, probeHeaderValue, seen,
|
assert.Equal(t, probeHeaderValue, seen.Get(probeHeaderName),
|
||||||
"a redirect within the configured origin must keep "+
|
"a redirect within the configured origin must keep "+
|
||||||
"the configured header",
|
"the configured header",
|
||||||
)
|
)
|
||||||
|
assert.Equal(t,
|
||||||
|
inboundHeaderValue, seen.Get(inboundHeaderName),
|
||||||
|
"a redirect within the configured origin must keep "+
|
||||||
|
"the forwarded inbound header",
|
||||||
|
)
|
||||||
assert.Equal(t,
|
assert.Equal(t,
|
||||||
database.DeliveryStatusDelivered, status,
|
database.DeliveryStatusDelivered, status,
|
||||||
)
|
)
|
||||||
@@ -211,6 +238,27 @@ func TestSameDeliveryOrigin(t *testing.T) {
|
|||||||
"a subdomain": {configured, "https://x.h/i", false},
|
"a subdomain": {configured, "https://x.h/i", false},
|
||||||
"the parent domain": {"https://x.h/a", "https://h/j", false},
|
"the parent domain": {"https://x.h/a", "https://h/j", false},
|
||||||
"another port": {configured, "https://h:8443/k", false},
|
"another port": {configured, "https://h:8443/k", false},
|
||||||
|
|
||||||
|
// Hostname() unwraps an IPv6 literal's brackets, so a
|
||||||
|
// bracketed host whose last group is the origin's port
|
||||||
|
// renders identically to the origin unless the port is
|
||||||
|
// re-joined with brackets. Each dest below differs from
|
||||||
|
// its origin in address AND in port.
|
||||||
|
"ipv6 port as final group": {
|
||||||
|
"https://[2001:db8::1]:8080/a",
|
||||||
|
"https://[2001:db8::1:8080]/l",
|
||||||
|
false,
|
||||||
|
},
|
||||||
|
"ipv6 loopback port as final group": {
|
||||||
|
"https://[::1]:8080/a",
|
||||||
|
"https://[::1:8080]/m",
|
||||||
|
false,
|
||||||
|
},
|
||||||
|
"ipv6 same origin": {
|
||||||
|
"https://[2001:db8::1]:8080/a",
|
||||||
|
"https://[2001:DB8::1]:8080/n",
|
||||||
|
true,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, tc := range cases {
|
for name, tc := range cases {
|
||||||
@@ -231,3 +279,105 @@ func TestSameDeliveryOrigin(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Installing a CheckRedirect discards net/http's own redirect
|
||||||
|
// limit, so the cap this policy restates is the only thing between
|
||||||
|
// a self-redirecting destination and an unbounded chain. A
|
||||||
|
// destination that always redirects must be cut off after exactly
|
||||||
|
// maxDeliveryRedirects requests, with the sentinel surfacing to the
|
||||||
|
// caller rather than a generic net/http error.
|
||||||
|
func TestRedirectPolicy_StopsAtHopCap(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var hits atomic.Int64
|
||||||
|
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
hits.Add(1)
|
||||||
|
http.Redirect(
|
||||||
|
w, r, "/loop", http.StatusFound,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
engine := delivery.NewTestEngine(
|
||||||
|
slog.New(slog.DiscardHandler),
|
||||||
|
&http.Client{Timeout: 10 * time.Second},
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
|
||||||
|
client := engine.ExportClientForRequest(
|
||||||
|
&delivery.HTTPTargetConfig{URL: srv.URL},
|
||||||
|
[]string{probeHeaderName},
|
||||||
|
)
|
||||||
|
require.NotNil(t, client.CheckRedirect)
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(
|
||||||
|
context.Background(), http.MethodGet, srv.URL, http.NoBody,
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
resp, doErr := client.Do(req)
|
||||||
|
if resp != nil {
|
||||||
|
_ = resp.Body.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
require.Error(t, doErr,
|
||||||
|
"an endless redirect chain must not be followed forever",
|
||||||
|
)
|
||||||
|
require.ErrorIs(t, doErr, delivery.ErrExportTooManyRedirects)
|
||||||
|
|
||||||
|
assert.Equal(t,
|
||||||
|
int64(delivery.ExportMaxDeliveryRedirects), hits.Load(),
|
||||||
|
"the chain must stop after exactly %d hops",
|
||||||
|
delivery.ExportMaxDeliveryRedirects,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The set the redirect policy strips is whatever the delivery path
|
||||||
|
// actually put on the wire, so a header added to the forward set is
|
||||||
|
// covered without a second edit. A header the event never carried
|
||||||
|
// is not in the set, and the delivery path's own two are deliberately
|
||||||
|
// excluded: Content-Type describes the body, which a 307 carries
|
||||||
|
// across hosts, and the inbound User-Agent every real sender supplies
|
||||||
|
// is overwritten before the request goes out.
|
||||||
|
func TestApplyRequestHeaders_ReportsOriginScopedNames(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
inbound, err := json.Marshal(map[string][]string{
|
||||||
|
inboundHeaderName: {inboundHeaderValue},
|
||||||
|
"Content-Type": {testContentType},
|
||||||
|
"User-Agent": {"curl/8.7.1"},
|
||||||
|
"Host": {"inbound.example.com"},
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(
|
||||||
|
context.Background(),
|
||||||
|
http.MethodPost,
|
||||||
|
"https://target.example.com/hook",
|
||||||
|
http.NoBody,
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
names := delivery.ExportApplyRequestHeaders(
|
||||||
|
req,
|
||||||
|
&database.Event{
|
||||||
|
Headers: string(inbound),
|
||||||
|
ContentType: testContentType,
|
||||||
|
},
|
||||||
|
&delivery.HTTPTargetConfig{
|
||||||
|
Headers: map[string]string{
|
||||||
|
probeHeaderName: probeHeaderValue,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert.Equal(t,
|
||||||
|
[]string{probeHeaderName, inboundHeaderName}, names,
|
||||||
|
"both header classes are reported, and only those: "+
|
||||||
|
"Host is never forwarded, Content-Type and "+
|
||||||
|
"User-Agent are the delivery path's own",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|||||||
@@ -6,8 +6,11 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -25,20 +28,83 @@ var (
|
|||||||
errBlockedIP = errors.New(
|
errBlockedIP = errors.New(
|
||||||
"blocked private/reserved IP range",
|
"blocked private/reserved IP range",
|
||||||
)
|
)
|
||||||
|
errBlockedMetadata = errors.New(
|
||||||
|
"blocked link-local or cloud instance metadata " +
|
||||||
|
"address: ALLOWED_EGRESS_CIDRS cannot open it",
|
||||||
|
)
|
||||||
errInvalidScheme = errors.New(
|
errInvalidScheme = errors.New(
|
||||||
"only http and https are allowed",
|
"only http and https are allowed",
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
// blockedNetworks contains all private/reserved IP ranges
|
// blockedNetworks contains all private/reserved IP ranges
|
||||||
// that should be blocked to prevent SSRF attacks.
|
// that should be blocked to prevent SSRF attacks. An operator
|
||||||
|
// can permit specific blocks out of this set with
|
||||||
|
// ALLOWED_EGRESS_CIDRS; see Guard.
|
||||||
//
|
//
|
||||||
//nolint:gochecknoglobals // package-level network list is appropriate here
|
//nolint:gochecknoglobals // package-level network list is appropriate here
|
||||||
var blockedNetworks []*net.IPNet
|
var blockedNetworks []*net.IPNet
|
||||||
|
|
||||||
|
// alwaysBlockedNetworks are the ranges no configuration can
|
||||||
|
// open: the link-local blocks and the cloud instance metadata
|
||||||
|
// endpoints that live outside them. Reaching one is credential
|
||||||
|
// or user-data theft rather than delivery to an internal
|
||||||
|
// service, so a supplied CIDR that covers such an address still
|
||||||
|
// leaves it blocked.
|
||||||
|
//
|
||||||
|
// Inclusion criterion — an address belongs here only if BOTH
|
||||||
|
// hold, and every entry below satisfies both:
|
||||||
|
//
|
||||||
|
// 1. It is a fixed address assigned by the provider, or a
|
||||||
|
// range reserved by IANA — never one the operator chose.
|
||||||
|
// That is what makes a host route free: it cannot collide
|
||||||
|
// with anything the operator runs.
|
||||||
|
// 2. Reaching it discloses credentials, or user data or
|
||||||
|
// bootstrap material — something granting onward access, or
|
||||||
|
// not cheaply rotated.
|
||||||
|
//
|
||||||
|
// Both halves are load-bearing, so use them to refuse a
|
||||||
|
// candidate and say why. An endpoint disclosing only the
|
||||||
|
// operator's own inventory (instance id, region, disks, NICs)
|
||||||
|
// fails (2): letting a delivery target reach the operator's own
|
||||||
|
// infrastructure is the feature ALLOWED_EGRESS_CIDRS exists to
|
||||||
|
// provide. But (2) is not "IAM credentials only" either —
|
||||||
|
// fd00:42::42 serves /user_data and /conf rather than tokens,
|
||||||
|
// and user data routinely carries bootstrap secrets. An address
|
||||||
|
// stays out if it fails (1) however well it clears (2): a host
|
||||||
|
// route inside a block operators really assign from, such as
|
||||||
|
// 10.0.0.0/8, can collide with a real internal service and
|
||||||
|
// forfeits the justification in (1).
|
||||||
|
//
|
||||||
|
// A publicly routable unicast address does not belong here even
|
||||||
|
// when it clears both halves. Nothing in this list can be
|
||||||
|
// reopened, so putting a public address here leaves the operator
|
||||||
|
// no escape hatch at all — the condition ALLOWED_EGRESS_CIDRS
|
||||||
|
// exists to remove. Default-block it in blockedNetworks instead,
|
||||||
|
// which an allowlist can override.
|
||||||
|
//
|
||||||
|
// This is a criterion, not an enumeration of every metadata
|
||||||
|
// address in existence.
|
||||||
|
//
|
||||||
|
// Every entry is either already in blockedNetworks — this list is
|
||||||
|
// what makes it unconditional — or an alternate encoding of
|
||||||
|
// 169.254.169.254 that Contains does not match against
|
||||||
|
// 169.254.0.0/16. Every entry outside the link-local blocks is a
|
||||||
|
// /32 or /128 host route, so blocking it costs an operator
|
||||||
|
// nothing else on the surrounding network.
|
||||||
|
//
|
||||||
|
// Derive membership from the address, never from the vendor's
|
||||||
|
// prose. Several providers call these endpoints "link-local" or
|
||||||
|
// even "localhost" in their own documentation while the address
|
||||||
|
// is a ULA outside fe80::/10, so a set derived from the docs
|
||||||
|
// comes out wrong.
|
||||||
|
//
|
||||||
|
//nolint:gochecknoglobals // package-level network list is appropriate here
|
||||||
|
var alwaysBlockedNetworks []*net.IPNet
|
||||||
|
|
||||||
//nolint:gochecknoinits // init is the idiomatic way to parse CIDRs once at startup
|
//nolint:gochecknoinits // init is the idiomatic way to parse CIDRs once at startup
|
||||||
func init() {
|
func init() {
|
||||||
cidrs := []string{
|
blockedNetworks = mustParseCIDRs([]string{
|
||||||
"127.0.0.0/8",
|
"127.0.0.0/8",
|
||||||
"10.0.0.0/8",
|
"10.0.0.0/8",
|
||||||
"172.16.0.0/12",
|
"172.16.0.0/12",
|
||||||
@@ -56,7 +122,72 @@ func init() {
|
|||||||
"::1/128",
|
"::1/128",
|
||||||
"fc00::/7",
|
"fc00::/7",
|
||||||
"fe80::/10",
|
"fe80::/10",
|
||||||
}
|
})
|
||||||
|
|
||||||
|
// Every entry is named. The set must not grow or shrink
|
||||||
|
// without a matching change to
|
||||||
|
// TestAlwaysBlockedNetworks_PinnedSet.
|
||||||
|
//
|
||||||
|
// The IPv4-mapped form ::ffff:169.254.169.254 needs no
|
||||||
|
// entry: net.IPNet.Contains normalises it via To4() before
|
||||||
|
// comparing, so 169.254.0.0/16 already matches it. To4()
|
||||||
|
// does not normalise the IPv4-compatible or NAT64 forms,
|
||||||
|
// which is why those are listed separately.
|
||||||
|
alwaysBlockedNetworks = mustParseCIDRs([]string{
|
||||||
|
// IPv4 link-local, carrying the 169.254.169.254
|
||||||
|
// metadata service used by AWS, Azure, DigitalOcean,
|
||||||
|
// Hetzner, OpenStack and others. Not Alibaba, which uses
|
||||||
|
// 100.100.100.200 below exclusively.
|
||||||
|
"169.254.0.0/16",
|
||||||
|
// IPv6 link-local, its IPv6 counterpart.
|
||||||
|
"fe80::/10",
|
||||||
|
|
||||||
|
// IPv6 metadata endpoints in ULA space. Each is a host
|
||||||
|
// route, and fd00::/8 is an ordinary block for an
|
||||||
|
// operator to allowlist, so without these entries that
|
||||||
|
// one allowlist line hands out cloud credentials on
|
||||||
|
// every provider below.
|
||||||
|
//
|
||||||
|
// AWS IPv6 IMDS.
|
||||||
|
"fd00:ec2::254/128",
|
||||||
|
// AWS EKS Pod Identity Agent, which issues pod identity
|
||||||
|
// credentials. A second AWS endpoint, distinct from
|
||||||
|
// IMDS above. AWS's own docs call it "localhost".
|
||||||
|
"fd00:ec2::23/128",
|
||||||
|
// GCP metadata server for IPv6-only instances.
|
||||||
|
"fd20:ce::254/128",
|
||||||
|
// Oracle OCI IMDS, serving /opc/v2 instance principals.
|
||||||
|
"fd00:c1::a9fe:a9fe/128",
|
||||||
|
// Scaleway metadata, serving /user_data and /conf.
|
||||||
|
"fd00:42::42/128",
|
||||||
|
// Linode/Akamai metadata. Akamai's docs call it
|
||||||
|
// "link-local"; it is not.
|
||||||
|
"fd00:a9fe:a9fe::1/128",
|
||||||
|
|
||||||
|
// IPv4 metadata endpoints outside link-local.
|
||||||
|
//
|
||||||
|
// Alibaba Cloud metadata. It sits in CGNAT
|
||||||
|
// 100.64.0.0/10, which Tailscale also uses, so an
|
||||||
|
// operator allowlisting a Tailscale peer's range would
|
||||||
|
// otherwise reopen it.
|
||||||
|
"100.100.100.200/32",
|
||||||
|
// Oracle Cloud Classic metadata. Inside the blocked
|
||||||
|
// 192.0.0.0/24, so this entry is what stops an
|
||||||
|
// allowlist from opening it.
|
||||||
|
"192.0.0.192/32",
|
||||||
|
|
||||||
|
// 169.254.169.254 as an IPv4-compatible IPv6 address.
|
||||||
|
"::a9fe:a9fe/128",
|
||||||
|
// 169.254.169.254 behind the NAT64 well-known prefix.
|
||||||
|
"64:ff9b::a9fe:a9fe/128",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// mustParseCIDRs parses a list of CIDR literals, panicking on a
|
||||||
|
// bad one. The inputs are compile-time constants, so a failure
|
||||||
|
// is a programming error rather than a runtime condition.
|
||||||
|
func mustParseCIDRs(cidrs []string) []*net.IPNet {
|
||||||
|
networks := make([]*net.IPNet, 0, len(cidrs))
|
||||||
|
|
||||||
for _, cidr := range cidrs {
|
for _, cidr := range cidrs {
|
||||||
_, network, err := net.ParseCIDR(cidr)
|
_, network, err := net.ParseCIDR(cidr)
|
||||||
@@ -67,16 +198,15 @@ func init() {
|
|||||||
))
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
blockedNetworks = append(
|
networks = append(networks, network)
|
||||||
blockedNetworks, network,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return networks
|
||||||
}
|
}
|
||||||
|
|
||||||
// isBlockedIP checks whether an IP address falls within
|
// matchesAny reports whether ip falls inside any of networks.
|
||||||
// any blocked private/reserved network range.
|
func matchesAny(networks []*net.IPNet, ip net.IP) bool {
|
||||||
func isBlockedIP(ip net.IP) bool {
|
for _, network := range networks {
|
||||||
for _, network := range blockedNetworks {
|
|
||||||
if network.Contains(ip) {
|
if network.Contains(ip) {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
@@ -85,9 +215,40 @@ func isBlockedIP(ip net.IP) bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// isBlockedIP checks whether an IP address falls within
|
||||||
|
// any blocked private/reserved network range, before any
|
||||||
|
// operator allowlist is considered.
|
||||||
|
func isBlockedIP(ip net.IP) bool {
|
||||||
|
return matchesAny(blockedNetworks, ip)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Guard makes every SSRF decision in the process.
|
||||||
|
//
|
||||||
|
// It holds the operator's ALLOWED_EGRESS_CIDRS allowlist and
|
||||||
|
// applies it in exactly one place, checkIP, which both the
|
||||||
|
// target-creation validator (ValidateTargetURL) and the delivery
|
||||||
|
// dialer call. Routing both through the same function is the
|
||||||
|
// point: when the two paths decided separately they drifted and
|
||||||
|
// disagreed, which is what made a target creatable but
|
||||||
|
// undeliverable.
|
||||||
|
//
|
||||||
|
// The guard is always on. The allowlist only ever adds specific
|
||||||
|
// networks to what the default blocklist refuses, and no
|
||||||
|
// configuration turns the guard off wholesale.
|
||||||
|
type Guard struct {
|
||||||
|
// allowed is the operator's ALLOWED_EGRESS_CIDRS. Empty
|
||||||
|
// (the default) means the default blocklist stands as-is.
|
||||||
|
allowed []netip.Prefix
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewGuard builds the process-wide SSRF guard from configuration.
|
||||||
|
func NewGuard(cfg *config.Config) *Guard {
|
||||||
|
return &Guard{allowed: cfg.AllowedEgressCIDRs}
|
||||||
|
}
|
||||||
|
|
||||||
// ValidateTargetURL checks that an HTTP delivery target
|
// ValidateTargetURL checks that an HTTP delivery target
|
||||||
// URL is safe from SSRF attacks.
|
// URL is safe from SSRF attacks.
|
||||||
func ValidateTargetURL(
|
func (g *Guard) ValidateTargetURL(
|
||||||
ctx context.Context, targetURL string,
|
ctx context.Context, targetURL string,
|
||||||
) error {
|
) error {
|
||||||
parsed, err := url.Parse(targetURL)
|
parsed, err := url.Parse(targetURL)
|
||||||
@@ -111,36 +272,79 @@ func ValidateTargetURL(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if ip := net.ParseIP(host); ip != nil {
|
if ip := net.ParseIP(host); ip != nil {
|
||||||
return checkBlockedIP(ip)
|
return g.checkIP(ip)
|
||||||
}
|
}
|
||||||
|
|
||||||
return validateHostname(ctx, host)
|
return g.validateHostname(ctx, host)
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateScheme(scheme string) error {
|
// NewSSRFSafeTransport creates an http.Transport with a
|
||||||
if scheme != "http" && scheme != "https" {
|
// custom DialContext that refuses connections to any address
|
||||||
|
// this guard blocks. It resolves and checks at dial time, so a
|
||||||
|
// name that passed validation but now answers with a blocked
|
||||||
|
// address (DNS rebinding) is still refused.
|
||||||
|
func (g *Guard) NewSSRFSafeTransport() *http.Transport {
|
||||||
|
return &http.Transport{
|
||||||
|
DialContext: g.ssrfDialContext,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// allows reports whether ip falls inside the operator's
|
||||||
|
// configured egress allowlist.
|
||||||
|
func (g *Guard) allows(ip net.IP) bool {
|
||||||
|
if len(g.allowed) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
addr, ok := netip.AddrFromSlice(ip)
|
||||||
|
if !ok {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// Config unmaps every parsed prefix, so an IPv4-mapped
|
||||||
|
// address has to be unmapped too or it would never match.
|
||||||
|
addr = addr.Unmap()
|
||||||
|
|
||||||
|
for _, prefix := range g.allowed {
|
||||||
|
if prefix.Contains(addr) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkIP is the single point at which SSRF policy is decided.
|
||||||
|
//
|
||||||
|
// The order is the policy:
|
||||||
|
//
|
||||||
|
// 1. alwaysBlockedNetworks is refused before the allowlist is
|
||||||
|
// consulted, so no configured CIDR reaches link-local or a
|
||||||
|
// cloud instance metadata endpoint.
|
||||||
|
// 2. The allowlist is consulted next, so a listed private
|
||||||
|
// network becomes reachable.
|
||||||
|
// 3. Everything else keeps the default blocklist's answer.
|
||||||
|
func (g *Guard) checkIP(ip net.IP) error {
|
||||||
|
if matchesAny(alwaysBlockedNetworks, ip) {
|
||||||
return fmt.Errorf(
|
return fmt.Errorf(
|
||||||
"unsupported URL scheme %q: %w",
|
"target IP %s: %w", ip, errBlockedMetadata,
|
||||||
scheme, errInvalidScheme,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if g.allows(ip) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func checkBlockedIP(ip net.IP) error {
|
|
||||||
if isBlockedIP(ip) {
|
if isBlockedIP(ip) {
|
||||||
return fmt.Errorf(
|
return fmt.Errorf(
|
||||||
"target IP %s is in a blocked "+
|
"target IP %s: %w", ip, errBlockedIP,
|
||||||
"private/reserved range: %w",
|
|
||||||
ip, errBlockedIP,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateHostname(
|
func (g *Guard) validateHostname(
|
||||||
ctx context.Context, host string,
|
ctx context.Context, host string,
|
||||||
) error {
|
) error {
|
||||||
dnsCtx, cancel := context.WithTimeout(
|
dnsCtx, cancel := context.WithTimeout(
|
||||||
@@ -165,11 +369,11 @@ func validateHostname(
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, ipAddr := range ips {
|
for _, ipAddr := range ips {
|
||||||
if isBlockedIP(ipAddr.IP) {
|
err = g.checkIP(ipAddr.IP)
|
||||||
|
if err != nil {
|
||||||
return fmt.Errorf(
|
return fmt.Errorf(
|
||||||
"hostname %q resolves to blocked "+
|
"hostname %q resolves to a blocked address: %w",
|
||||||
"IP %s: %w",
|
host, err,
|
||||||
host, ipAddr.IP, errBlockedIP,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -177,16 +381,7 @@ func validateHostname(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewSSRFSafeTransport creates an http.Transport with a
|
func (g *Guard) ssrfDialContext(
|
||||||
// custom DialContext that blocks connections to
|
|
||||||
// private/reserved IP addresses.
|
|
||||||
func NewSSRFSafeTransport() *http.Transport {
|
|
||||||
return &http.Transport{
|
|
||||||
DialContext: ssrfDialContext,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func ssrfDialContext(
|
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
network, addr string,
|
network, addr string,
|
||||||
) (net.Conn, error) {
|
) (net.Conn, error) {
|
||||||
@@ -209,11 +404,11 @@ func ssrfDialContext(
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, ipAddr := range ips {
|
for _, ipAddr := range ips {
|
||||||
if isBlockedIP(ipAddr.IP) {
|
err = g.checkIP(ipAddr.IP)
|
||||||
|
if err != nil {
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"ssrf: connection to %s (%s) "+
|
"ssrf: connection to %s blocked: %w",
|
||||||
"blocked: %w",
|
host, err,
|
||||||
host, ipAddr.IP, errBlockedIP,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -225,3 +420,14 @@ func ssrfDialContext(
|
|||||||
net.JoinHostPort(ips[0].IP.String(), port),
|
net.JoinHostPort(ips[0].IP.String(), port),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateScheme(scheme string) error {
|
||||||
|
if scheme != "http" && scheme != "https" {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"unsupported URL scheme %q: %w",
|
||||||
|
scheme, errInvalidScheme,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
562
internal/delivery/ssrf_allowlist_test.go
Normal file
562
internal/delivery/ssrf_allowlist_test.go
Normal file
@@ -0,0 +1,562 @@
|
|||||||
|
package delivery_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"net/url"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Addresses the SSRF tests in this package share.
|
||||||
|
const (
|
||||||
|
// metadataIP is the cloud instance metadata address, and
|
||||||
|
// metadataURL an endpoint on it. The guard must never reach
|
||||||
|
// either, whatever an operator lists.
|
||||||
|
metadataIP = "169.254.169.254"
|
||||||
|
metadataURL = "http://" + metadataIP + "/latest/meta-data/"
|
||||||
|
|
||||||
|
// loopbackHookURL is a target on this host: blocked by
|
||||||
|
// default, reachable only once an operator allowlists
|
||||||
|
// loopback.
|
||||||
|
loopbackHookURL = "http://127.0.0.1/hook"
|
||||||
|
|
||||||
|
// publicIP is an ordinary public address, which the guard
|
||||||
|
// permits with or without an allowlist.
|
||||||
|
publicIP = "93.184.216.34"
|
||||||
|
|
||||||
|
// allowAllIPv4 and allowAllIPv6 are the widest allowlist
|
||||||
|
// entries expressible: the whole internet, in each family.
|
||||||
|
// Nothing unconditionally blocked may be reachable under
|
||||||
|
// them.
|
||||||
|
allowAllIPv4 = "0.0.0.0/0"
|
||||||
|
allowAllIPv6 = "::/0"
|
||||||
|
|
||||||
|
// allowAllULA is the ordinary ULA block an operator lists to
|
||||||
|
// reach their own IPv6 network. Several providers park a
|
||||||
|
// metadata endpoint inside it.
|
||||||
|
allowAllULA = "fd00::/8"
|
||||||
|
|
||||||
|
// metadataRefusalClause is the part of the refusal that only
|
||||||
|
// alwaysBlockedNetworks produces. Asserting it, rather than
|
||||||
|
// the bare word "blocked", is what proves the unconditional
|
||||||
|
// set did the refusing and not the default blocklist.
|
||||||
|
metadataRefusalClause = "ALLOWED_EGRESS_CIDRS cannot open it"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestGuardAllowlist_PermittedCIDRDelivers proves the escape
|
||||||
|
// hatch actually works end to end: with 127.0.0.0/8 allowed, the
|
||||||
|
// guard's own transport connects to a loopback server and gets a
|
||||||
|
// response back. The default guard, given the identical URL,
|
||||||
|
// refuses it — so the delivery succeeds because of the allowlist
|
||||||
|
// and nothing else.
|
||||||
|
func TestGuardAllowlist_PermittedCIDRDelivers(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
},
|
||||||
|
))
|
||||||
|
t.Cleanup(srv.Close)
|
||||||
|
|
||||||
|
// httptest listens on loopback, which the default blocklist
|
||||||
|
// covers: exactly the "forward to a service on this host"
|
||||||
|
// case the allowlist exists for.
|
||||||
|
requireLoopback(t, srv.URL)
|
||||||
|
|
||||||
|
guard := delivery.NewTestGuard(
|
||||||
|
netip.MustParsePrefix("127.0.0.0/8"),
|
||||||
|
)
|
||||||
|
|
||||||
|
require.NoError(t,
|
||||||
|
guard.ValidateTargetURL(context.Background(), srv.URL),
|
||||||
|
"an allowlisted loopback target must pass validation",
|
||||||
|
)
|
||||||
|
|
||||||
|
client := &http.Client{
|
||||||
|
Timeout: 5 * time.Second,
|
||||||
|
Transport: guard.NewSSRFSafeTransport(),
|
||||||
|
}
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(
|
||||||
|
context.Background(), http.MethodPost, srv.URL, nil,
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
resp, err := client.Do(req)
|
||||||
|
require.NoError(t,
|
||||||
|
err, "an allowlisted loopback target must be deliverable",
|
||||||
|
)
|
||||||
|
|
||||||
|
defer func() { _ = resp.Body.Close() }()
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusNoContent, resp.StatusCode)
|
||||||
|
|
||||||
|
// The same URL through the default guard must still fail, or
|
||||||
|
// this test would pass without the allowlist doing anything.
|
||||||
|
assert.Error(t,
|
||||||
|
delivery.NewTestGuard().ValidateTargetURL(
|
||||||
|
context.Background(), srv.URL,
|
||||||
|
),
|
||||||
|
"without the allowlist the same target must be refused",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGuardAllowlist_UnlistedPrivateStillRefused proves the
|
||||||
|
// allowlist grants only what it names. A guard that opens one
|
||||||
|
// private block must keep refusing every other one, at both the
|
||||||
|
// validation and the delivery entry point.
|
||||||
|
func TestGuardAllowlist_UnlistedPrivateStillRefused(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// Only 10.1.0.0/16 is open — a narrow block inside a much
|
||||||
|
// wider private range, so the test can tell "permits the
|
||||||
|
// listed block" from "permits anything private".
|
||||||
|
guard := delivery.NewTestGuard(
|
||||||
|
netip.MustParsePrefix("10.1.0.0/16"),
|
||||||
|
)
|
||||||
|
|
||||||
|
refused := []string{
|
||||||
|
"http://192.168.1.10/hook",
|
||||||
|
"http://172.16.0.1/hook",
|
||||||
|
loopbackHookURL,
|
||||||
|
"http://[fc00::1]/hook",
|
||||||
|
"http://100.64.0.1/hook",
|
||||||
|
// Private, adjacent to the allowed block, outside it.
|
||||||
|
"http://10.2.0.1/hook",
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, target := range refused {
|
||||||
|
t.Run(target, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
err := guard.ValidateTargetURL(
|
||||||
|
context.Background(), target,
|
||||||
|
)
|
||||||
|
require.Error(t,
|
||||||
|
err, "%s is not allowlisted and must be refused",
|
||||||
|
target,
|
||||||
|
)
|
||||||
|
assert.Contains(t, err.Error(), "blocked")
|
||||||
|
|
||||||
|
assertDialRefused(t, guard, target)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// The block that is listed must in fact be permitted, so the
|
||||||
|
// refusals above are selective rather than a guard that
|
||||||
|
// ignores its allowlist entirely.
|
||||||
|
assert.NoError(t,
|
||||||
|
guard.ValidateTargetURL(
|
||||||
|
context.Background(), "http://10.1.2.3/hook",
|
||||||
|
),
|
||||||
|
"the allowlisted block must be permitted",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGuardAllowlist_MetadataAlwaysRefused is the load-bearing
|
||||||
|
// case: cloud instance metadata endpoints are credential theft
|
||||||
|
// rather than delivery to an internal service, so no allowlist
|
||||||
|
// reaches one. Every guard below names a CIDR that covers its
|
||||||
|
// target — including 0.0.0.0/0, ::/0, and the ordinary ULA and
|
||||||
|
// CGNAT blocks an operator would really list — and the address
|
||||||
|
// must stay refused anyway, on both the validation and the
|
||||||
|
// delivery path.
|
||||||
|
func TestGuardAllowlist_MetadataAlwaysRefused(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tt := range metadataAlwaysRefusedCases() {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
guard := delivery.NewTestGuard(
|
||||||
|
netip.MustParsePrefix(tt.allow),
|
||||||
|
)
|
||||||
|
|
||||||
|
err := guard.ValidateTargetURL(
|
||||||
|
context.Background(), tt.target,
|
||||||
|
)
|
||||||
|
require.Error(t,
|
||||||
|
err,
|
||||||
|
"%s must stay blocked even though %s covers it",
|
||||||
|
tt.target, tt.allow,
|
||||||
|
)
|
||||||
|
assert.Contains(t,
|
||||||
|
err.Error(),
|
||||||
|
metadataRefusalClause,
|
||||||
|
"the refusal must say why it cannot be opened",
|
||||||
|
)
|
||||||
|
|
||||||
|
// The metadata clause, not just "blocked": that is
|
||||||
|
// what distinguishes the unconditional set from the
|
||||||
|
// ordinary blocklist.
|
||||||
|
assertDialRefusedWith(
|
||||||
|
t, guard, tt.target, metadataRefusalClause,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// metadataAlwaysRefusedCase is one (allowlist, target) pair that
|
||||||
|
// must be refused: allow covers target, and target must stay
|
||||||
|
// blocked regardless.
|
||||||
|
type metadataAlwaysRefusedCase struct {
|
||||||
|
name string
|
||||||
|
allow string
|
||||||
|
target string
|
||||||
|
}
|
||||||
|
|
||||||
|
// metadataAlwaysRefusedCases enumerates every unconditionally
|
||||||
|
// blocked address together with an allowlist entry that would
|
||||||
|
// otherwise reach it. Split by family of address only to stay
|
||||||
|
// under the function-length limit.
|
||||||
|
func metadataAlwaysRefusedCases() []metadataAlwaysRefusedCase {
|
||||||
|
cases := linkLocalRefusedCases()
|
||||||
|
cases = append(cases, ulaMetadataRefusedCases()...)
|
||||||
|
cases = append(cases, ipv4MetadataRefusedCases()...)
|
||||||
|
|
||||||
|
return append(cases, encodedMetadataRefusedCases()...)
|
||||||
|
}
|
||||||
|
|
||||||
|
// linkLocalRefusedCases covers the link-local blocks, including
|
||||||
|
// an operator naming the metadata address outright.
|
||||||
|
func linkLocalRefusedCases() []metadataAlwaysRefusedCase {
|
||||||
|
return []metadataAlwaysRefusedCase{
|
||||||
|
{
|
||||||
|
name: "exact metadata host",
|
||||||
|
allow: "169.254.169.254/32",
|
||||||
|
target: metadataURL,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "whole link-local block",
|
||||||
|
allow: "169.254.0.0/16",
|
||||||
|
target: metadataURL,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "supernet covering link-local",
|
||||||
|
allow: "169.0.0.0/8",
|
||||||
|
target: metadataURL,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the entire IPv4 internet",
|
||||||
|
allow: allowAllIPv4,
|
||||||
|
target: metadataURL,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "other link-local address",
|
||||||
|
allow: allowAllIPv4,
|
||||||
|
target: "http://169.254.1.1/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "IPv6 link-local",
|
||||||
|
allow: allowAllIPv6,
|
||||||
|
target: "http://[fe80::1]/",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ulaMetadataRefusedCases covers the metadata endpoints parked
|
||||||
|
// in ULA space. Every one is opened by the single ordinary
|
||||||
|
// allowlist entry fd00::/8, which is the whole reason they need
|
||||||
|
// their own /128 host routes: fe80::/10 does not cover a ULA,
|
||||||
|
// whatever the vendor's documentation calls the address.
|
||||||
|
func ulaMetadataRefusedCases() []metadataAlwaysRefusedCase {
|
||||||
|
return []metadataAlwaysRefusedCase{
|
||||||
|
{
|
||||||
|
name: "AWS IPv6 IMDS under an allowlisted ULA block",
|
||||||
|
allow: allowAllULA,
|
||||||
|
target: "http://[fd00:ec2::254]/latest/meta-data/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// A second AWS credential endpoint, distinct from
|
||||||
|
// IMDS. AWS's own docs call this one "localhost".
|
||||||
|
name: "AWS EKS Pod Identity under an allowlisted ULA block",
|
||||||
|
allow: allowAllULA,
|
||||||
|
target: "http://[fd00:ec2::23]/v1/credentials",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "GCP IPv6 metadata under an allowlisted ULA block",
|
||||||
|
allow: allowAllULA,
|
||||||
|
target: "http://[fd20:ce::254]/computeMetadata/v1/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "Oracle OCI IPv6 IMDS under an allowlisted ULA block",
|
||||||
|
allow: allowAllULA,
|
||||||
|
target: "http://[fd00:c1::a9fe:a9fe]/opc/v2/instance/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "Scaleway IPv6 metadata under an allowlisted ULA block",
|
||||||
|
allow: allowAllULA,
|
||||||
|
target: "http://[fd00:42::42]/conf",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Akamai's docs call this "link-local"; it is a ULA,
|
||||||
|
// so fe80::/10 does not cover it.
|
||||||
|
name: "Linode IPv6 metadata under an allowlisted ULA block",
|
||||||
|
allow: allowAllULA,
|
||||||
|
target: "http://[fd00:a9fe:a9fe::1]/v1/instance",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ipv4MetadataRefusedCases covers the IPv4 metadata endpoints
|
||||||
|
// that sit outside link-local: one in CGNAT and one in the
|
||||||
|
// blocked 192.0.0.0/24, each reachable only through an allowlist
|
||||||
|
// that this set overrides.
|
||||||
|
func ipv4MetadataRefusedCases() []metadataAlwaysRefusedCase {
|
||||||
|
return []metadataAlwaysRefusedCase{
|
||||||
|
{
|
||||||
|
// Tailscale uses 100.64.0.0/10, so an operator
|
||||||
|
// forwarding to a Tailscale peer lists exactly this.
|
||||||
|
name: "Alibaba metadata under allowlisted CGNAT",
|
||||||
|
allow: "100.64.0.0/10",
|
||||||
|
target: "http://100.100.100.200/latest/meta-data/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Inside the already-blocked 192.0.0.0/24, so only
|
||||||
|
// an allowlist can reach it — and must not.
|
||||||
|
name: "Oracle Cloud Classic metadata under 0.0.0.0/0",
|
||||||
|
allow: allowAllIPv4,
|
||||||
|
target: "http://192.0.0.192/latest/meta-data/",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// encodedMetadataRefusedCases covers the alternate IPv6
|
||||||
|
// encodings of 169.254.169.254.
|
||||||
|
func encodedMetadataRefusedCases() []metadataAlwaysRefusedCase {
|
||||||
|
return []metadataAlwaysRefusedCase{
|
||||||
|
{
|
||||||
|
// To4() does not normalise the IPv4-compatible form,
|
||||||
|
// so this needs its own always-blocked entry.
|
||||||
|
name: "IPv4-compatible IPv6 form of the metadata IP",
|
||||||
|
allow: allowAllIPv6,
|
||||||
|
target: "http://[::a9fe:a9fe]/latest/meta-data/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Nor the NAT64 well-known prefix form.
|
||||||
|
name: "NAT64 form of the metadata IP",
|
||||||
|
allow: allowAllIPv6,
|
||||||
|
target: "http://[64:ff9b::a9fe:a9fe]/latest/meta-data/",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Already refused before this change: IPNet.Contains
|
||||||
|
// calls To4() first, so the mapped form matches
|
||||||
|
// 169.254.0.0/16. Pinned so it cannot regress.
|
||||||
|
//
|
||||||
|
// Allowed under 0.0.0.0/0 rather than ::/0: allows()
|
||||||
|
// unmaps before matching, so ::/0 would not cover the
|
||||||
|
// unmapped v4 address and the case would not prove
|
||||||
|
// the allowlist was overridden.
|
||||||
|
name: "IPv4-mapped IPv6 form of the metadata IP",
|
||||||
|
allow: allowAllIPv4,
|
||||||
|
target: "http://[::ffff:169.254.169.254]/latest/meta-data/",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGuardAllowlist_PublicUnaffected asserts the allowlist does
|
||||||
|
// not narrow anything: public addresses were reachable before it
|
||||||
|
// existed and stay reachable, whether or not a list is set.
|
||||||
|
func TestGuardAllowlist_PublicUnaffected(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
guards := map[string]*delivery.Guard{
|
||||||
|
"default": delivery.NewTestGuard(),
|
||||||
|
"with allowlist": delivery.NewTestGuard(
|
||||||
|
netip.MustParsePrefix("10.0.0.0/8"),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, guard := range guards {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
assert.NoError(t,
|
||||||
|
guard.ValidateTargetURL(
|
||||||
|
context.Background(),
|
||||||
|
"http://"+publicIP+"/webhook",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGuardCheckIP_BothPathsShareOneDecision asserts that the
|
||||||
|
// validator and the dialer are not two policies that happen to
|
||||||
|
// agree: both are defined in terms of checkIP, so the exported
|
||||||
|
// decision function is the whole answer for a given address.
|
||||||
|
func TestGuardCheckIP_BothPathsShareOneDecision(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
guard := delivery.NewTestGuard(
|
||||||
|
netip.MustParsePrefix("10.0.0.0/8"),
|
||||||
|
)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
ip string
|
||||||
|
allowed bool
|
||||||
|
}{
|
||||||
|
{"10.1.2.3", true},
|
||||||
|
{publicIP, true},
|
||||||
|
{"192.168.1.1", false},
|
||||||
|
{"127.0.0.1", false},
|
||||||
|
{metadataIP, false},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.ip, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ip := net.ParseIP(tt.ip)
|
||||||
|
require.NotNil(t, ip)
|
||||||
|
|
||||||
|
decision := guard.ExportCheckIP(ip)
|
||||||
|
|
||||||
|
validation := guard.ValidateTargetURL(
|
||||||
|
context.Background(), "http://"+hostFor(tt.ip)+"/x",
|
||||||
|
)
|
||||||
|
|
||||||
|
if tt.allowed {
|
||||||
|
require.NoError(t, decision)
|
||||||
|
require.NoError(t, validation)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
require.Error(t, decision)
|
||||||
|
require.Error(t, validation,
|
||||||
|
"validation must refuse what checkIP refuses",
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestAlwaysBlockedNetworks_PinnedSet pins the unconditional set
|
||||||
|
// exactly, so it cannot quietly grow or shrink.
|
||||||
|
//
|
||||||
|
// It stays deliberately small. Everything else in the default
|
||||||
|
// blocklist is an operator's own network and must remain
|
||||||
|
// openable, or the escape hatch would not work — which is why
|
||||||
|
// the metadata endpoints outside the link-local range are host
|
||||||
|
// routes rather than the blocks that contain them.
|
||||||
|
func TestAlwaysBlockedNetworks_PinnedSet(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
nets := delivery.ExportAlwaysBlockedNetworks()
|
||||||
|
|
||||||
|
got := make([]string, 0, len(nets))
|
||||||
|
for _, n := range nets {
|
||||||
|
got = append(got, n.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{
|
||||||
|
// IPv4 link-local: the 169.254.169.254 metadata
|
||||||
|
// service on AWS, Azure and others.
|
||||||
|
"169.254.0.0/16",
|
||||||
|
// IPv6 link-local.
|
||||||
|
"fe80::/10",
|
||||||
|
// AWS IPv6 IMDS, inside the ULA space an operator may
|
||||||
|
// legitimately allowlist.
|
||||||
|
"fd00:ec2::254/128",
|
||||||
|
// AWS EKS Pod Identity Agent, likewise ULA.
|
||||||
|
"fd00:ec2::23/128",
|
||||||
|
// GCP metadata for IPv6-only instances, likewise ULA.
|
||||||
|
"fd20:ce::254/128",
|
||||||
|
// Oracle OCI IMDS over IPv6, likewise ULA.
|
||||||
|
"fd00:c1::a9fe:a9fe/128",
|
||||||
|
// Scaleway metadata over IPv6, likewise ULA.
|
||||||
|
"fd00:42::42/128",
|
||||||
|
// Linode/Akamai metadata over IPv6, likewise ULA.
|
||||||
|
"fd00:a9fe:a9fe::1/128",
|
||||||
|
// Alibaba Cloud metadata, inside CGNAT.
|
||||||
|
"100.100.100.200/32",
|
||||||
|
// Oracle Cloud Classic metadata, inside the blocked
|
||||||
|
// 192.0.0.0/24.
|
||||||
|
"192.0.0.192/32",
|
||||||
|
// 169.254.169.254 as an IPv4-compatible IPv6 address.
|
||||||
|
"::a9fe:a9fe/128",
|
||||||
|
// 169.254.169.254 behind the NAT64 well-known prefix.
|
||||||
|
"64:ff9b::a9fe:a9fe/128",
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Equal(t, want, got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// requireLoopback fails the test unless rawURL's host is a
|
||||||
|
// loopback address, so the allowlist test cannot silently stop
|
||||||
|
// exercising a blocked range.
|
||||||
|
func requireLoopback(t *testing.T, rawURL string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
parsed, err := url.Parse(rawURL)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
ip := net.ParseIP(parsed.Hostname())
|
||||||
|
require.NotNil(t, ip, "test server host must be an IP literal")
|
||||||
|
require.True(t, ip.IsLoopback(),
|
||||||
|
"test server must listen on loopback, got %s", ip,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertDialRefused asserts the guard's transport refuses to
|
||||||
|
// connect to target, which is the delivery-time half of the
|
||||||
|
// policy. It never reaches the network: the guard checks the
|
||||||
|
// resolved address before dialling.
|
||||||
|
func assertDialRefused(
|
||||||
|
t *testing.T, guard *delivery.Guard, target string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
assertDialRefusedWith(t, guard, target, "blocked")
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertDialRefusedWith is assertDialRefused with the refusal
|
||||||
|
// text pinned. Callers testing the unconditional set pass
|
||||||
|
// metadataRefusalClause so the subtest cannot pass on an
|
||||||
|
// ordinary blocklist refusal instead.
|
||||||
|
func assertDialRefusedWith(
|
||||||
|
t *testing.T, guard *delivery.Guard, target, clause string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
client := &http.Client{
|
||||||
|
Timeout: 5 * time.Second,
|
||||||
|
Transport: guard.NewSSRFSafeTransport(),
|
||||||
|
}
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(
|
||||||
|
context.Background(), http.MethodPost, target, nil,
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
resp, err := client.Do(req)
|
||||||
|
if resp != nil {
|
||||||
|
_ = resp.Body.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
require.Error(t, err,
|
||||||
|
"delivery to %s must be refused by the dialer", target,
|
||||||
|
)
|
||||||
|
assert.Contains(t, err.Error(), clause,
|
||||||
|
"the refusal must come from the SSRF guard",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostFor renders an IP as it appears in a URL host, bracketing
|
||||||
|
// IPv6 literals.
|
||||||
|
func hostFor(ip string) string {
|
||||||
|
if net.ParseIP(ip).To4() == nil {
|
||||||
|
return "[" + ip + "]"
|
||||||
|
}
|
||||||
|
|
||||||
|
return ip
|
||||||
|
}
|
||||||
@@ -31,10 +31,10 @@ func TestIsBlockedIP_PrivateRanges(t *testing.T) {
|
|||||||
{"192.168.0.1", "192.168.0.1", true},
|
{"192.168.0.1", "192.168.0.1", true},
|
||||||
{"192.168.255.255", "192.168.255.255", true},
|
{"192.168.255.255", "192.168.255.255", true},
|
||||||
{"169.254.0.1", "169.254.0.1", true},
|
{"169.254.0.1", "169.254.0.1", true},
|
||||||
{"169.254.169.254", "169.254.169.254", true},
|
{metadataIP, metadataIP, true},
|
||||||
{"8.8.8.8", "8.8.8.8", false},
|
{"8.8.8.8", "8.8.8.8", false},
|
||||||
{"1.1.1.1", "1.1.1.1", false},
|
{"1.1.1.1", "1.1.1.1", false},
|
||||||
{"93.184.216.34", "93.184.216.34", false},
|
{publicIP, publicIP, false},
|
||||||
{"::1", "::1", true},
|
{"::1", "::1", true},
|
||||||
{"fd00::1", "fd00::1", true},
|
{"fd00::1", "fd00::1", true},
|
||||||
{"fc00::1", "fc00::1", true},
|
{"fc00::1", "fc00::1", true},
|
||||||
@@ -72,12 +72,12 @@ func TestValidateTargetURL_Blocked(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
blockedURLs := []string{
|
blockedURLs := []string{
|
||||||
"http://127.0.0.1/hook",
|
loopbackHookURL,
|
||||||
"http://127.0.0.1:8080/hook",
|
"http://127.0.0.1:8080/hook",
|
||||||
"https://10.0.0.1/hook",
|
"https://10.0.0.1/hook",
|
||||||
"http://192.168.1.1/webhook",
|
"http://192.168.1.1/webhook",
|
||||||
"http://172.16.0.1/api",
|
"http://172.16.0.1/api",
|
||||||
"http://169.254.169.254/latest/meta-data/",
|
metadataURL,
|
||||||
"http://[::1]/hook",
|
"http://[::1]/hook",
|
||||||
"http://[fc00::1]/hook",
|
"http://[fc00::1]/hook",
|
||||||
"http://[fe80::1]/hook",
|
"http://[fe80::1]/hook",
|
||||||
@@ -88,7 +88,7 @@ func TestValidateTargetURL_Blocked(t *testing.T) {
|
|||||||
t.Run(u, func(t *testing.T) {
|
t.Run(u, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
err := delivery.ValidateTargetURL(
|
err := delivery.NewTestGuard().ValidateTargetURL(
|
||||||
context.Background(), u,
|
context.Background(), u,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -112,7 +112,7 @@ func TestValidateTargetURL_Allowed(t *testing.T) {
|
|||||||
t.Run(u, func(t *testing.T) {
|
t.Run(u, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
err := delivery.ValidateTargetURL(
|
err := delivery.NewTestGuard().ValidateTargetURL(
|
||||||
context.Background(), u,
|
context.Background(), u,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -126,7 +126,7 @@ func TestValidateTargetURL_Allowed(t *testing.T) {
|
|||||||
func TestValidateTargetURL_InvalidScheme(t *testing.T) {
|
func TestValidateTargetURL_InvalidScheme(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
err := delivery.ValidateTargetURL(
|
err := delivery.NewTestGuard().ValidateTargetURL(
|
||||||
context.Background(), "ftp://example.com/hook",
|
context.Background(), "ftp://example.com/hook",
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -140,7 +140,7 @@ func TestValidateTargetURL_InvalidScheme(t *testing.T) {
|
|||||||
func TestValidateTargetURL_EmptyHost(t *testing.T) {
|
func TestValidateTargetURL_EmptyHost(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
err := delivery.ValidateTargetURL(
|
err := delivery.NewTestGuard().ValidateTargetURL(
|
||||||
context.Background(), "http:///path",
|
context.Background(), "http:///path",
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -150,7 +150,7 @@ func TestValidateTargetURL_EmptyHost(t *testing.T) {
|
|||||||
func TestValidateTargetURL_InvalidURL(t *testing.T) {
|
func TestValidateTargetURL_InvalidURL(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
err := delivery.ValidateTargetURL(
|
err := delivery.NewTestGuard().ValidateTargetURL(
|
||||||
context.Background(), "://invalid",
|
context.Background(), "://invalid",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"sort"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -404,9 +405,9 @@ func (t *httpTarget) doHTTPRequest(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
applyRequestHeaders(req, event, cfg)
|
originScoped := applyRequestHeaders(req, event, cfg)
|
||||||
|
|
||||||
client := t.clientForConfig(cfg)
|
client := t.clientForRequest(cfg, originScoped)
|
||||||
|
|
||||||
resp, doErr := executeHTTPRequest(client, req)
|
resp, doErr := executeHTTPRequest(client, req)
|
||||||
|
|
||||||
@@ -432,15 +433,18 @@ func (t *httpTarget) doHTTPRequest(
|
|||||||
return resp.StatusCode, string(body), dur, nil
|
return resp.StatusCode, string(body), dur, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// clientForConfig returns the client for one target's requests.
|
// clientForRequest returns the client for one delivery attempt.
|
||||||
// A config that overrides neither the timeout nor the headers gets
|
// originScoped is the header set applyRequestHeaders built for that
|
||||||
// the shared client: with no configured headers there is nothing
|
// attempt; a request with neither a per-target timeout nor an
|
||||||
// for the redirect policy to strip, and net/http's default policy
|
// origin-scoped header gets the shared client, because there is
|
||||||
// already withholds Authorization and Cookie across hosts.
|
// then nothing for the redirect policy to strip and net/http's
|
||||||
func (t *httpTarget) clientForConfig(
|
// default policy already withholds Authorization and Cookie across
|
||||||
|
// hosts.
|
||||||
|
func (t *httpTarget) clientForRequest(
|
||||||
cfg *HTTPTargetConfig,
|
cfg *HTTPTargetConfig,
|
||||||
|
originScoped []string,
|
||||||
) *http.Client {
|
) *http.Client {
|
||||||
if cfg.Timeout <= 0 && len(cfg.Headers) == 0 {
|
if cfg.Timeout <= 0 && len(originScoped) == 0 {
|
||||||
return t.client
|
return t.client
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -459,10 +463,8 @@ func (t *httpTarget) clientForConfig(
|
|||||||
) * time.Second
|
) * time.Second
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(cfg.Headers) > 0 {
|
if len(originScoped) > 0 {
|
||||||
client.CheckRedirect = configuredHeaderRedirectPolicy(
|
client.CheckRedirect = offOriginHeaderPolicy(originScoped)
|
||||||
cfg.Headers,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return client
|
return client
|
||||||
@@ -507,40 +509,88 @@ func isForwardableHeader(name string) bool {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// applyRequestHeaders builds one outbound delivery's header set and
|
||||||
|
// returns the canonical names of every header in it that is scoped
|
||||||
|
// to the configured origin: the inbound event headers this delivery
|
||||||
|
// forwarded, plus the operator's configured headers. The redirect
|
||||||
|
// policy strips exactly that set on a hop that leaves the origin,
|
||||||
|
// so the forward set is decided here and only here — a header added
|
||||||
|
// to it is covered off-origin without a second edit elsewhere.
|
||||||
func applyRequestHeaders(
|
func applyRequestHeaders(
|
||||||
req *http.Request,
|
req *http.Request,
|
||||||
event *database.Event,
|
event *database.Event,
|
||||||
cfg *HTTPTargetConfig,
|
cfg *HTTPTargetConfig,
|
||||||
) {
|
) []string {
|
||||||
if event.ContentType != "" {
|
if event.ContentType != "" {
|
||||||
req.Header.Set(
|
req.Header.Set(
|
||||||
"Content-Type", event.ContentType,
|
"Content-Type", event.ContentType,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
var originalHeaders map[string][]string
|
originScoped := forwardEventHeaders(req, event)
|
||||||
|
|
||||||
if event.Headers != "" {
|
|
||||||
jsonErr := json.Unmarshal(
|
|
||||||
[]byte(event.Headers),
|
|
||||||
&originalHeaders,
|
|
||||||
)
|
|
||||||
if jsonErr == nil {
|
|
||||||
for k, vals := range originalHeaders {
|
|
||||||
if isForwardableHeader(k) {
|
|
||||||
for _, v := range vals {
|
|
||||||
req.Header.Add(k, v)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for k, v := range cfg.Headers {
|
for k, v := range cfg.Headers {
|
||||||
req.Header.Set(k, v)
|
req.Header.Set(k, v)
|
||||||
|
originScoped[http.CanonicalHeaderKey(k)] = struct{}{}
|
||||||
}
|
}
|
||||||
|
|
||||||
req.Header.Set("User-Agent", "webhooker/1.0")
|
req.Header.Set("User-Agent", "webhooker/1.0")
|
||||||
|
|
||||||
|
// Content-Type describes the body being sent rather than the
|
||||||
|
// sender, and the delivery path sets it from the event itself.
|
||||||
|
// A 307/308 preserves the body across hosts, so stripping it
|
||||||
|
// would send that body untyped.
|
||||||
|
delete(originScoped, "Content-Type")
|
||||||
|
|
||||||
|
// User-Agent is overwritten just above, so an inbound one never
|
||||||
|
// reaches the wire and the value that does identifies this
|
||||||
|
// delivery path rather than the sender. Reporting it would strip
|
||||||
|
// it off-origin and leave net/http's own default in its place.
|
||||||
|
delete(originScoped, "User-Agent")
|
||||||
|
|
||||||
|
names := make([]string, 0, len(originScoped))
|
||||||
|
for name := range originScoped {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
|
||||||
|
sort.Strings(names)
|
||||||
|
|
||||||
|
return names
|
||||||
|
}
|
||||||
|
|
||||||
|
// forwardEventHeaders copies the inbound event's forwardable
|
||||||
|
// headers onto the outbound request and returns the canonical names
|
||||||
|
// it forwarded. Headers the event never carried are absent from the
|
||||||
|
// result, so the redirect policy strips what was actually sent.
|
||||||
|
func forwardEventHeaders(
|
||||||
|
req *http.Request,
|
||||||
|
event *database.Event,
|
||||||
|
) map[string]struct{} {
|
||||||
|
forwarded := make(map[string]struct{})
|
||||||
|
|
||||||
|
if event.Headers == "" {
|
||||||
|
return forwarded
|
||||||
|
}
|
||||||
|
|
||||||
|
var inbound map[string][]string
|
||||||
|
|
||||||
|
if json.Unmarshal([]byte(event.Headers), &inbound) != nil {
|
||||||
|
return forwarded
|
||||||
|
}
|
||||||
|
|
||||||
|
for k, vals := range inbound {
|
||||||
|
if !isForwardableHeader(k) || len(vals) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, v := range vals {
|
||||||
|
req.Header.Add(k, v)
|
||||||
|
}
|
||||||
|
|
||||||
|
forwarded[http.CanonicalHeaderKey(k)] = struct{}{}
|
||||||
|
}
|
||||||
|
|
||||||
|
return forwarded
|
||||||
}
|
}
|
||||||
|
|
||||||
// executeHTTPRequest sends an HTTP request using the provided
|
// executeHTTPRequest sends an HTTP request using the provided
|
||||||
|
|||||||
272
internal/delivery/target_redact.go
Normal file
272
internal/delivery/target_redact.go
Normal file
@@ -0,0 +1,272 @@
|
|||||||
|
package delivery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/url"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RedactionMarker stands in for a target credential found in
|
||||||
|
// text the target's remote peer chose.
|
||||||
|
const RedactionMarker = "(redacted)"
|
||||||
|
|
||||||
|
// Redactor removes one target's own credential material from
|
||||||
|
// text that target's remote peer chose: a delivery response
|
||||||
|
// body, or a delivery error stored before the delivery path
|
||||||
|
// learned to mask the URLs it embeds.
|
||||||
|
//
|
||||||
|
// It removes byte-identical echoes of strings taken from the
|
||||||
|
// target's stored configuration, and nothing else. Anything
|
||||||
|
// the remote re-encodes survives: JSON "\/" escaping (what
|
||||||
|
// PHP's json_encode emits by default), percent-encoding, HTML
|
||||||
|
// entities, and an echo of only part of a path. It cannot
|
||||||
|
// remove a secret the remote invented.
|
||||||
|
//
|
||||||
|
// The zero Redactor removes nothing, which is what a caller
|
||||||
|
// holding no target for a delivery gets.
|
||||||
|
type Redactor struct {
|
||||||
|
secrets []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewRedactor builds the redactor for one target.
|
||||||
|
func NewRedactor(t *database.Target) Redactor {
|
||||||
|
// Drop empty strings here rather than at the site that
|
||||||
|
// produced one. strings.ReplaceAll with an empty old string
|
||||||
|
// inserts the marker at every byte boundary, so a single
|
||||||
|
// empty secret destroys every body and error the target
|
||||||
|
// renders; filtering at the collection point means no field
|
||||||
|
// added to targetSecrets later can reintroduce that.
|
||||||
|
// url.Parse("https://@example.com/in") is the known
|
||||||
|
// producer: a non-nil User whose String is "".
|
||||||
|
secrets := slices.DeleteFunc(
|
||||||
|
targetSecrets(t),
|
||||||
|
func(s string) bool { return s == "" },
|
||||||
|
)
|
||||||
|
|
||||||
|
// Longest first, so replacing a secret that is contained
|
||||||
|
// in a longer one cannot leave a fragment of the longer
|
||||||
|
// one behind. Configured headers arrive in map order, so
|
||||||
|
// the sort is also what makes the result deterministic.
|
||||||
|
slices.SortFunc(secrets, func(a, b string) int {
|
||||||
|
if d := len(b) - len(a); d != 0 {
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.Compare(a, b)
|
||||||
|
})
|
||||||
|
|
||||||
|
return Redactor{secrets: secrets}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Redact replaces every occurrence of the target's credential
|
||||||
|
// material in s.
|
||||||
|
func (r Redactor) Redact(s string) string {
|
||||||
|
if s == "" {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, secret := range r.secrets {
|
||||||
|
s = strings.ReplaceAll(s, secret, RedactionMarker)
|
||||||
|
}
|
||||||
|
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// RedactCut redacts s, which its caller has already cut to a
|
||||||
|
// byte budget, and additionally drops any tail of s that is a
|
||||||
|
// proper prefix of a secret.
|
||||||
|
//
|
||||||
|
// The cut lands wherever the remote's padding puts it, so the
|
||||||
|
// remote chooses where inside the credential it falls. The
|
||||||
|
// severed prefix left behind equals no secret, so plain
|
||||||
|
// Redact would render it verbatim.
|
||||||
|
func (r Redactor) RedactCut(s string) string {
|
||||||
|
s = r.Redact(s)
|
||||||
|
|
||||||
|
if n := r.secretPrefixSuffix(s); n > 0 {
|
||||||
|
return s[:len(s)-n] + RedactionMarker
|
||||||
|
}
|
||||||
|
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// secretPrefixSuffix returns the length of the longest suffix
|
||||||
|
// of s that is a proper prefix of one of the secrets, or 0
|
||||||
|
// when there is none.
|
||||||
|
func (r Redactor) secretPrefixSuffix(s string) int {
|
||||||
|
longest := 0
|
||||||
|
|
||||||
|
for _, secret := range r.secrets {
|
||||||
|
// Proper prefixes only: a whole secret at the tail was
|
||||||
|
// already replaced by Redact.
|
||||||
|
n := min(len(secret)-1, len(s))
|
||||||
|
for ; n > longest; n-- {
|
||||||
|
if strings.HasSuffix(s, secret[:n]) {
|
||||||
|
longest = n
|
||||||
|
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return longest
|
||||||
|
}
|
||||||
|
|
||||||
|
// targetSecrets returns the credential-bearing strings a
|
||||||
|
// target's configuration carries.
|
||||||
|
//
|
||||||
|
// The destination URL contributes. Its path, query and
|
||||||
|
// userinfo are the credential for both target types that have
|
||||||
|
// one — an incoming-webhook URL is a bearer token, which is
|
||||||
|
// why MaskURL elides exactly those parts — and they are the
|
||||||
|
// material this service actually sends, so a remote that
|
||||||
|
// echoes the request back echoes them.
|
||||||
|
//
|
||||||
|
// Configured request headers contribute their values, but
|
||||||
|
// only for the credential-shaped names isCredentialHeaderName
|
||||||
|
// picks out. That is the same class-based rule applied to
|
||||||
|
// URLs: an echoed Accept or User-Agent still renders, an
|
||||||
|
// echoed Authorization does not.
|
||||||
|
func targetSecrets(t *database.Target) []string {
|
||||||
|
if t == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
switch t.Type {
|
||||||
|
case database.TargetTypeSlack:
|
||||||
|
cfg, err := parseSlackConfig(t.Config)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return urlSecrets(cfg.WebhookURL)
|
||||||
|
case database.TargetTypeHTTP:
|
||||||
|
cfg, err := parseHTTPConfig(t.Config)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return append(
|
||||||
|
urlSecrets(cfg.URL),
|
||||||
|
headerSecrets(cfg.Headers)...,
|
||||||
|
)
|
||||||
|
case database.TargetTypeDatabase, database.TargetTypeLog:
|
||||||
|
// Neither has a destination URL, so neither has
|
||||||
|
// anything to redact.
|
||||||
|
return nil
|
||||||
|
default:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// urlSecrets returns the substrings of a destination URL that
|
||||||
|
// must not survive into a rendered page: the whole URL, the
|
||||||
|
// parts of it MaskURL elides, and any userinfo.
|
||||||
|
//
|
||||||
|
// No length floor is applied to the path, and none to the
|
||||||
|
// userinfo. A short path or a four-byte username is treated as
|
||||||
|
// a credential exactly like a long one, because the field takes
|
||||||
|
// an arbitrary URL and no part of it can be assumed non-secret —
|
||||||
|
// the same rule MaskURL applies. headerSecrets does carry a
|
||||||
|
// floor, and the difference is deliberate: a header is picked
|
||||||
|
// out by a name-shaped guess and its value may be ordinary
|
||||||
|
// text, whereas a URL's path and userinfo are credential
|
||||||
|
// material by position.
|
||||||
|
func urlSecrets(raw string) []string {
|
||||||
|
raw = strings.TrimSpace(raw)
|
||||||
|
if raw == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
secrets := []string{raw}
|
||||||
|
|
||||||
|
parsed, err := url.Parse(raw)
|
||||||
|
if err != nil {
|
||||||
|
return secrets
|
||||||
|
}
|
||||||
|
|
||||||
|
if parsed.Path != "" && parsed.Path != "/" {
|
||||||
|
requestURI := parsed.RequestURI()
|
||||||
|
secrets = append(secrets, requestURI)
|
||||||
|
|
||||||
|
if escaped := parsed.EscapedPath(); escaped != requestURI {
|
||||||
|
secrets = append(secrets, escaped)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if parsed.User != nil {
|
||||||
|
secrets = append(secrets, parsed.User.String())
|
||||||
|
|
||||||
|
if pw, ok := parsed.User.Password(); ok && pw != "" {
|
||||||
|
secrets = append(secrets, pw)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return secrets
|
||||||
|
}
|
||||||
|
|
||||||
|
// minHeaderSecretBytes is the shortest header value treated as
|
||||||
|
// a credential. Unlike a URL path, a header value can be a
|
||||||
|
// couple of bytes long, and redacting those would scatter the
|
||||||
|
// marker through ordinary response text for no gain.
|
||||||
|
const minHeaderSecretBytes = 4
|
||||||
|
|
||||||
|
// headerSecrets returns the values of the configured headers
|
||||||
|
// whose names are credential-shaped.
|
||||||
|
func headerSecrets(headers map[string]string) []string {
|
||||||
|
var secrets []string
|
||||||
|
|
||||||
|
for name, value := range headers {
|
||||||
|
value = strings.TrimSpace(value)
|
||||||
|
if len(value) < minHeaderSecretBytes {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if isCredentialHeaderName(name) {
|
||||||
|
secrets = append(secrets, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return secrets
|
||||||
|
}
|
||||||
|
|
||||||
|
// isCredentialHeaderName classifies a header by its name. The
|
||||||
|
// value is never inspected, so the rule is the same
|
||||||
|
// class-based one MaskURL applies to a destination URL.
|
||||||
|
//
|
||||||
|
// The fragments are short on purpose, and match anywhere in
|
||||||
|
// the name, so abbreviations an operator might use are covered
|
||||||
|
// too: X-Sig, X-Pass, X-HMAC. That over-matches — a header
|
||||||
|
// named X-Design contains "sig" — and over-matching is the
|
||||||
|
// safe direction here: the cost is a marker where an echoed
|
||||||
|
// header value would have rendered.
|
||||||
|
func isCredentialHeaderName(name string) bool {
|
||||||
|
name = strings.ToLower(strings.TrimSpace(name))
|
||||||
|
|
||||||
|
// Names that carry a credential by definition.
|
||||||
|
switch name {
|
||||||
|
case "authorization", "proxy-authorization", "cookie":
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// What operators call their own credential headers:
|
||||||
|
// X-Api-Key, X-Hub-Signature, X-Auth-Token.
|
||||||
|
for _, fragment := range []string{
|
||||||
|
"auth",
|
||||||
|
"credential",
|
||||||
|
"hmac",
|
||||||
|
"key",
|
||||||
|
"pass",
|
||||||
|
"secret",
|
||||||
|
"sig",
|
||||||
|
"token",
|
||||||
|
} {
|
||||||
|
if strings.Contains(name, fragment) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
284
internal/delivery/target_redact_test.go
Normal file
284
internal/delivery/target_redact_test.go
Normal file
@@ -0,0 +1,284 @@
|
|||||||
|
package delivery_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/url"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The secret path segments of a Slack incoming webhook URL.
|
||||||
|
const (
|
||||||
|
redactSecretPath = "/services/T11111111/B11111111/" +
|
||||||
|
"YYYYYYYYYYYYYYYYYYYYYYYY"
|
||||||
|
redactWebhookURL = "https://hooks.slack.com" +
|
||||||
|
redactSecretPath
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRedactor_RemovesSlackWebhookURL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeSlack,
|
||||||
|
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
got := r.Redact("no_service for " + redactWebhookURL)
|
||||||
|
|
||||||
|
assert.NotContains(t, got, redactSecretPath)
|
||||||
|
assert.NotContains(t, got, "T11111111")
|
||||||
|
|
||||||
|
// One marker, not a marker with the host left in front of
|
||||||
|
// it: the whole URL is replaced before the path it
|
||||||
|
// contains, which is what sorting the secrets longest
|
||||||
|
// first buys.
|
||||||
|
assert.Equal(
|
||||||
|
t,
|
||||||
|
"no_service for "+delivery.RedactionMarker,
|
||||||
|
got,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_RemovesSecretSeveredByACut covers the input the
|
||||||
|
// redactor exists for: text cut to a byte budget with the
|
||||||
|
// credential straddling the cut. The remote chooses the
|
||||||
|
// padding, so it chooses where the cut lands inside the
|
||||||
|
// credential, and the severed prefix that remains equals no
|
||||||
|
// secret.
|
||||||
|
func TestRedactor_RemovesSecretSeveredByACut(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeSlack,
|
||||||
|
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
// Every cut position inside the credential, not just a
|
||||||
|
// convenient one.
|
||||||
|
for n := 1; n < len(redactWebhookURL); n++ {
|
||||||
|
severed := redactWebhookURL[:n]
|
||||||
|
cut := "padding " + severed
|
||||||
|
|
||||||
|
got := r.RedactCut(cut)
|
||||||
|
|
||||||
|
assert.Equal(
|
||||||
|
t,
|
||||||
|
"padding "+delivery.RedactionMarker,
|
||||||
|
got,
|
||||||
|
"cut after %d bytes of the credential", n,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_RedactsCredentialShapedHeaderValues pins the
|
||||||
|
// class-based header rule: a header whose name says credential
|
||||||
|
// has its value redacted, and a routine header does not, so
|
||||||
|
// ordinary response content survives.
|
||||||
|
func TestRedactor_RedactsCredentialShapedHeaderValues(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeHTTP,
|
||||||
|
Config: `{"url":"https://example.com/in",` +
|
||||||
|
`"headers":{` +
|
||||||
|
`"Authorization":"Bearer AAAAAAAAAAAA",` +
|
||||||
|
`"Cookie":"session=BBBBBBBBBBBB",` +
|
||||||
|
`"X-Api-Key":"CCCCCCCCCCCC",` +
|
||||||
|
`"X-Hub-Signature":"sha256=DDDDDDDDDDDD",` +
|
||||||
|
`"X-Sig":"EEEEEEEEEEEE",` +
|
||||||
|
`"X-Pass":"FFFFFFFFFFFF",` +
|
||||||
|
`"X-HMAC":"GGGGGGGGGGGG",` +
|
||||||
|
`"X-Credential":"HHHHHHHHHHHH",` +
|
||||||
|
`"Accept":"application/json",` +
|
||||||
|
`"User-Agent":"webhooker/1.0"}}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, secret := range []string{
|
||||||
|
"Bearer AAAAAAAAAAAA",
|
||||||
|
"session=BBBBBBBBBBBB",
|
||||||
|
"CCCCCCCCCCCC",
|
||||||
|
"sha256=DDDDDDDDDDDD",
|
||||||
|
// Abbreviated names an operator might use.
|
||||||
|
"EEEEEEEEEEEE",
|
||||||
|
"FFFFFFFFFFFF",
|
||||||
|
"GGGGGGGGGGGG",
|
||||||
|
"HHHHHHHHHHHH",
|
||||||
|
} {
|
||||||
|
got := r.Redact("echo: " + secret)
|
||||||
|
|
||||||
|
assert.Equal(
|
||||||
|
t,
|
||||||
|
"echo: "+delivery.RedactionMarker,
|
||||||
|
got,
|
||||||
|
secret,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
const routine = "Accept: application/json, " +
|
||||||
|
"User-Agent: webhooker/1.0"
|
||||||
|
|
||||||
|
assert.Equal(t, routine, r.Redact(routine))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_IgnoresVeryShortHeaderValues pins the floor
|
||||||
|
// under a header value. Redacting a two-byte value would put
|
||||||
|
// the marker through every response that happens to contain
|
||||||
|
// those bytes.
|
||||||
|
func TestRedactor_IgnoresVeryShortHeaderValues(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeHTTP,
|
||||||
|
Config: `{"url":"https://example.com/in",` +
|
||||||
|
`"headers":{"X-Api-Key":"ab"}}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
const response = "rabbit"
|
||||||
|
|
||||||
|
assert.Equal(t, response, r.Redact(response))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_RemovesBarePath covers a remote that echoes
|
||||||
|
// only the request path rather than the whole URL. The path
|
||||||
|
// segments are the credential on their own.
|
||||||
|
func TestRedactor_RemovesBarePath(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeSlack,
|
||||||
|
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
got := r.Redact("POST " + redactSecretPath + " 404")
|
||||||
|
|
||||||
|
assert.NotContains(t, got, redactSecretPath)
|
||||||
|
assert.Equal(
|
||||||
|
t,
|
||||||
|
"POST "+delivery.RedactionMarker+" 404",
|
||||||
|
got,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_RemovesHTTPURLQueryAndUserinfo covers the HTTP
|
||||||
|
// target, whose destination is an arbitrary URL: the query
|
||||||
|
// string and the userinfo carry credentials as readily as the
|
||||||
|
// path does.
|
||||||
|
func TestRedactor_RemovesHTTPURLQueryAndUserinfo(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// Assembled rather than written out, so the literal is
|
||||||
|
// not itself a credential-shaped string.
|
||||||
|
dest := url.URL{
|
||||||
|
Scheme: "https",
|
||||||
|
User: url.UserPassword("user", "hunter2"),
|
||||||
|
Host: "example.com",
|
||||||
|
Path: "/in",
|
||||||
|
RawQuery: "token=s3cr3t",
|
||||||
|
}
|
||||||
|
raw := dest.String()
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeHTTP,
|
||||||
|
Config: `{"url":"` + raw + `"}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, echoed := range []string{
|
||||||
|
raw,
|
||||||
|
"/in?token=s3cr3t",
|
||||||
|
"hunter2",
|
||||||
|
} {
|
||||||
|
got := r.Redact("rejected: " + echoed)
|
||||||
|
|
||||||
|
assert.NotContains(t, got, "s3cr3t", echoed)
|
||||||
|
assert.NotContains(t, got, "hunter2", echoed)
|
||||||
|
assert.Contains(
|
||||||
|
t, got, delivery.RedactionMarker, echoed,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_LeavesUnrelatedTextAlone pins that the
|
||||||
|
// redactor matches literally: it does not guess at what a
|
||||||
|
// secret looks like, so ordinary response content survives.
|
||||||
|
func TestRedactor_LeavesUnrelatedTextAlone(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const response = "ok=false error=channel_not_found"
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeSlack,
|
||||||
|
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
assert.Equal(t, response, r.Redact(response))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_EmptyUserinfoDoesNotShredTheBody covers a
|
||||||
|
// destination URL written with a bare "@" and no userinfo:
|
||||||
|
// url.Parse returns a non-nil User whose String is empty. An
|
||||||
|
// empty secret in the list would make strings.ReplaceAll
|
||||||
|
// insert the marker at every byte boundary, destroying every
|
||||||
|
// body and error string the target renders.
|
||||||
|
func TestRedactor_EmptyUserinfoDoesNotShredTheBody(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const dest = "https://@example.com/in"
|
||||||
|
|
||||||
|
// The premise: this URL really does parse to a non-nil
|
||||||
|
// User contributing an empty string.
|
||||||
|
parsed, err := url.Parse(dest)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NotNil(t, parsed.User)
|
||||||
|
require.Empty(t, parsed.User.String())
|
||||||
|
|
||||||
|
r := delivery.NewRedactor(&database.Target{
|
||||||
|
Type: database.TargetTypeHTTP,
|
||||||
|
Config: `{"url":"` + dest + `"}`,
|
||||||
|
})
|
||||||
|
|
||||||
|
const body = "ok=false error=channel_not_found"
|
||||||
|
|
||||||
|
assert.Equal(t, body, r.Redact(body))
|
||||||
|
assert.Equal(t, body, r.RedactCut(body))
|
||||||
|
|
||||||
|
// The real credential material still goes, so filtering the
|
||||||
|
// empty string out did not disarm the redactor.
|
||||||
|
assert.Equal(
|
||||||
|
t,
|
||||||
|
"POST "+delivery.RedactionMarker,
|
||||||
|
r.Redact("POST "+dest),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRedactor_ZeroValueAndConfiglessTargets pins that a
|
||||||
|
// caller with no target, an unparseable config, or a target
|
||||||
|
// type with no destination URL gets a redactor that changes
|
||||||
|
// nothing rather than one that panics.
|
||||||
|
func TestRedactor_ZeroValueAndConfiglessTargets(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const text = "some response body"
|
||||||
|
|
||||||
|
var zero delivery.Redactor
|
||||||
|
|
||||||
|
assert.Equal(t, text, zero.Redact(text))
|
||||||
|
assert.Equal(t, text, delivery.NewRedactor(nil).Redact(text))
|
||||||
|
|
||||||
|
for _, tgt := range []database.Target{
|
||||||
|
{Type: database.TargetTypeLog},
|
||||||
|
{Type: database.TargetTypeDatabase},
|
||||||
|
{Type: database.TargetTypeSlack, Config: "not json"},
|
||||||
|
{Type: database.TargetTypeHTTP, Config: ""},
|
||||||
|
} {
|
||||||
|
assert.Equal(
|
||||||
|
t, text,
|
||||||
|
delivery.NewRedactor(&tgt).Redact(text),
|
||||||
|
tgt.Type,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -185,7 +185,7 @@ func TestDoHTTPRequest_TransportErrorMasksURL(t *testing.T) {
|
|||||||
func TestValidateTargetURL_UnparsableURLIsMasked(t *testing.T) {
|
func TestValidateTargetURL_UnparsableURLIsMasked(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
err := delivery.ValidateTargetURL(
|
err := delivery.NewTestGuard().ValidateTargetURL(
|
||||||
context.TODO(),
|
context.TODO(),
|
||||||
"https://hooks.slack.com"+maskSecretPath+"\n",
|
"https://hooks.slack.com"+maskSecretPath+"\n",
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ func seedFailedDelivery(
|
|||||||
Method: http.MethodPost,
|
Method: http.MethodPost,
|
||||||
Headers: `{"X-Test":["yes"]}`,
|
Headers: `{"X-Test":["yes"]}`,
|
||||||
Body: `{"replay":"me"}`,
|
Body: `{"replay":"me"}`,
|
||||||
ContentType: "application/json",
|
ContentType: contentTypeJSON,
|
||||||
}
|
}
|
||||||
|
|
||||||
require.NoError(t, webhookDB.Omit(
|
require.NoError(t, webhookDB.Omit(
|
||||||
|
|||||||
169
internal/handlers/delivery_result_view.go
Normal file
169
internal/handlers/delivery_result_view.go
Normal file
@@ -0,0 +1,169 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// maxRenderedResponseBytes caps how many bytes of one stored
|
||||||
|
// delivery response body reach the event log page.
|
||||||
|
//
|
||||||
|
// The bound is enforced here and in SQL, because this page's
|
||||||
|
// memory profile must not depend on a constant in another
|
||||||
|
// package staying where it is, and because rows predating the
|
||||||
|
// delivery engine's own cap or restored from an archive are
|
||||||
|
// not covered by it at all.
|
||||||
|
//
|
||||||
|
// It happens to equal that engine cap, so a row written by the
|
||||||
|
// current engine reaches this one exactly and is never cut
|
||||||
|
// twice. Nothing here may assume the two differ: see view.
|
||||||
|
const maxRenderedResponseBytes = 4096
|
||||||
|
|
||||||
|
// deliveryResultColumns is the delivery attempt projection.
|
||||||
|
// The casts to blob are load-bearing for the same reason they
|
||||||
|
// are in eventLogColumns: they make substr and length count
|
||||||
|
// bytes rather than characters, and they make SQLite do the
|
||||||
|
// cut, so an oversized stored response never becomes a Go
|
||||||
|
// string at all.
|
||||||
|
const deliveryResultColumns = "delivery_id, attempt_num, success, " +
|
||||||
|
"status_code, error, duration, " +
|
||||||
|
"substr(cast(response_body as blob), 1, ?) AS response_body, " +
|
||||||
|
"length(cast(response_body as blob)) AS response_bytes"
|
||||||
|
|
||||||
|
// DeliveryResultView is the display-safe projection of one
|
||||||
|
// delivery attempt for the event log page. It carries a
|
||||||
|
// capped response body plus the true stored size, so the page
|
||||||
|
// can mark a response as truncated without holding the whole
|
||||||
|
// thing.
|
||||||
|
//
|
||||||
|
// Both Error and ResponseBody have been through the target's
|
||||||
|
// Redactor. The engine already masks the URL out of the
|
||||||
|
// errors it stores, so for errors this is a second line
|
||||||
|
// covering rows written before it did; for response bodies it
|
||||||
|
// is the only line, and its reach is what
|
||||||
|
// delivery.Redactor documents.
|
||||||
|
type DeliveryResultView struct {
|
||||||
|
AttemptNum int
|
||||||
|
Success bool
|
||||||
|
|
||||||
|
// StatusCode is 0 when the attempt never got a response,
|
||||||
|
// which is why the page asks HasStatusCode rather than
|
||||||
|
// printing the number.
|
||||||
|
StatusCode int
|
||||||
|
|
||||||
|
// Error is the stored failure message, redacted.
|
||||||
|
Error string
|
||||||
|
|
||||||
|
// DurationMS is how long the attempt took.
|
||||||
|
DurationMS int64
|
||||||
|
|
||||||
|
// ResponseBody holds at most maxRenderedResponseBytes
|
||||||
|
// bytes of the stored response, redacted. It is remote
|
||||||
|
// content and must only ever be rendered escaped.
|
||||||
|
ResponseBody string
|
||||||
|
|
||||||
|
// ResponseBytes is the size of the stored response body,
|
||||||
|
// before the cut and before redaction. It is what the
|
||||||
|
// remote sent only when ResponseSizeKnown is set.
|
||||||
|
ResponseBytes int64
|
||||||
|
|
||||||
|
// ResponseShownBytes is how much of that the page is
|
||||||
|
// showing. It is the size of the cut, taken before
|
||||||
|
// redaction, so the truncation marker reports what SQLite
|
||||||
|
// returned rather than how much the marker substitution
|
||||||
|
// then changed the length.
|
||||||
|
ResponseShownBytes int
|
||||||
|
|
||||||
|
// ResponseTruncated reports that the body shown may be
|
||||||
|
// incomplete, so the page owes the reader a marker. Every
|
||||||
|
// body that reaches the cap counts, because one the
|
||||||
|
// delivery engine cut at its own equal cap is
|
||||||
|
// indistinguishable from a complete one.
|
||||||
|
ResponseTruncated bool
|
||||||
|
|
||||||
|
// ResponseSizeKnown reports that ResponseBytes is the whole
|
||||||
|
// response the remote sent, so the marker may quote it. It
|
||||||
|
// is false for a body that only reaches the cap, where how
|
||||||
|
// much came after it was never recorded.
|
||||||
|
ResponseSizeKnown bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// HasStatusCode reports whether the attempt got as far as an
|
||||||
|
// HTTP response. A transport failure stores no status code,
|
||||||
|
// and rendering that as "0" would read as a real status.
|
||||||
|
func (v DeliveryResultView) HasStatusCode() bool {
|
||||||
|
return v.StatusCode != 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// deliveryResultRow is one row of the delivery attempt
|
||||||
|
// projection. Its response body arrives already cut to the
|
||||||
|
// cap by SQLite, with the true size beside it.
|
||||||
|
type deliveryResultRow struct {
|
||||||
|
DeliveryID string
|
||||||
|
AttemptNum int
|
||||||
|
Success bool
|
||||||
|
StatusCode int
|
||||||
|
Error string
|
||||||
|
Duration int64
|
||||||
|
ResponseBody []byte
|
||||||
|
ResponseBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// view projects a loaded row for rendering, stripping the
|
||||||
|
// target's own credential out of the two fields a remote peer
|
||||||
|
// gets to influence.
|
||||||
|
func (r *deliveryResultRow) view(
|
||||||
|
redactor delivery.Redactor,
|
||||||
|
) DeliveryResultView {
|
||||||
|
body := r.ResponseBody
|
||||||
|
|
||||||
|
// Two different cuts can have shortened this body, and the
|
||||||
|
// row records only one of them. SQLite cuts here, whenever
|
||||||
|
// the stored value is larger than the cap. The delivery
|
||||||
|
// engine cut earlier, whenever the remote sent more than
|
||||||
|
// its own maxBodyLog — which is this same number, so such a
|
||||||
|
// row stores the cut length as its whole length and nothing
|
||||||
|
// in it separates a response that ended at the cap from one
|
||||||
|
// severed there.
|
||||||
|
//
|
||||||
|
// So a body that reaches the cap is treated as cut either
|
||||||
|
// way. Gating on ResponseBytes alone would assume the two
|
||||||
|
// caps differ, and they do not: under the current engine
|
||||||
|
// that gate never opens.
|
||||||
|
cut := r.ResponseBytes > int64(len(body)) ||
|
||||||
|
len(body) >= maxRenderedResponseBytes
|
||||||
|
|
||||||
|
// The row holds more than the page shows only in the first
|
||||||
|
// of those cases. In the second the stored row is all there
|
||||||
|
// is, and its size is a floor rather than the true one.
|
||||||
|
sizeKnown := r.ResponseBytes > int64(len(body))
|
||||||
|
|
||||||
|
// Only a cut response can have been left mid-sequence,
|
||||||
|
// exactly as with an event body.
|
||||||
|
if cut {
|
||||||
|
body = trimPartialRune(body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A cut body goes through RedactCut: the remote controls
|
||||||
|
// the padding ahead of a credential it echoes, so it
|
||||||
|
// controls where the cut falls inside that credential, and
|
||||||
|
// the severed prefix left behind matches no secret whole.
|
||||||
|
rendered := string(body)
|
||||||
|
if cut {
|
||||||
|
rendered = redactor.RedactCut(rendered)
|
||||||
|
} else {
|
||||||
|
rendered = redactor.Redact(rendered)
|
||||||
|
}
|
||||||
|
|
||||||
|
return DeliveryResultView{
|
||||||
|
AttemptNum: r.AttemptNum,
|
||||||
|
Success: r.Success,
|
||||||
|
StatusCode: r.StatusCode,
|
||||||
|
Error: redactor.Redact(r.Error),
|
||||||
|
DurationMS: r.Duration,
|
||||||
|
ResponseBody: rendered,
|
||||||
|
ResponseBytes: r.ResponseBytes,
|
||||||
|
ResponseShownBytes: len(body),
|
||||||
|
ResponseTruncated: cut,
|
||||||
|
ResponseSizeKnown: sizeKnown,
|
||||||
|
}
|
||||||
|
}
|
||||||
515
internal/handlers/delivery_result_view_test.go
Normal file
515
internal/handlers/delivery_result_view_test.go
Normal file
@@ -0,0 +1,515 @@
|
|||||||
|
package handlers_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
|
"sneak.berlin/go/webhooker/internal/handlers"
|
||||||
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// responseCap is the number of response bytes the event log
|
||||||
|
// page is allowed to render for one delivery attempt.
|
||||||
|
const responseCap = handlers.MaxRenderedResponseBytesForTest
|
||||||
|
|
||||||
|
// failedAttempt describes the failed delivery every test in
|
||||||
|
// this file seeds. The values are distinctive so that finding
|
||||||
|
// them in the rendered page cannot be a coincidence.
|
||||||
|
const (
|
||||||
|
attemptStatusCode = 502
|
||||||
|
attemptDurationMS = 1234
|
||||||
|
attemptNumber = 3
|
||||||
|
attemptError = "upstream returned 502 Bad Gateway"
|
||||||
|
)
|
||||||
|
|
||||||
|
// seedFailedDeliveryWithResponse records an event, a failed
|
||||||
|
// delivery against targetID, and one delivery result carrying
|
||||||
|
// the given response body. It returns the delivery.
|
||||||
|
//
|
||||||
|
// Distinct from seedFailedDelivery in delivery_replay_test.go,
|
||||||
|
// which seeds an attempt with no response body and returns the
|
||||||
|
// event as well; these tests need the recorded response.
|
||||||
|
func seedFailedDeliveryWithResponse(
|
||||||
|
t *testing.T,
|
||||||
|
dbMgr *database.WebhookDBManager,
|
||||||
|
webhookID, targetID, responseBody string,
|
||||||
|
) *database.Delivery {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
webhookDB, err := dbMgr.GetDB(webhookID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
event := &database.Event{
|
||||||
|
WebhookID: webhookID,
|
||||||
|
Method: http.MethodPost,
|
||||||
|
Body: `{"test":true}`,
|
||||||
|
ContentType: contentTypeJSON,
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Omit(
|
||||||
|
clause.Associations,
|
||||||
|
).Create(event).Error)
|
||||||
|
|
||||||
|
dlv := &database.Delivery{
|
||||||
|
EventID: event.ID,
|
||||||
|
TargetID: targetID,
|
||||||
|
Status: database.DeliveryStatusFailed,
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Omit(
|
||||||
|
clause.Associations,
|
||||||
|
).Create(dlv).Error)
|
||||||
|
|
||||||
|
result := &database.DeliveryResult{
|
||||||
|
DeliveryID: dlv.ID,
|
||||||
|
AttemptNum: attemptNumber,
|
||||||
|
Success: false,
|
||||||
|
StatusCode: attemptStatusCode,
|
||||||
|
ResponseBody: responseBody,
|
||||||
|
Error: attemptError,
|
||||||
|
Duration: attemptDurationMS,
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Omit(
|
||||||
|
clause.Associations,
|
||||||
|
).Create(result).Error)
|
||||||
|
|
||||||
|
return dlv
|
||||||
|
}
|
||||||
|
|
||||||
|
// seedFailureAndRender seeds a failed delivery against a
|
||||||
|
// target of the given type and config, and returns the
|
||||||
|
// rendered event log page.
|
||||||
|
func seedFailureAndRender(
|
||||||
|
t *testing.T,
|
||||||
|
targetType database.TargetType,
|
||||||
|
config, responseBody string,
|
||||||
|
) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
tgt := seedConfiguredTarget(
|
||||||
|
t, db, wh.ID, targetType, config,
|
||||||
|
)
|
||||||
|
|
||||||
|
seedFailedDeliveryWithResponse(
|
||||||
|
t, dbMgr, wh.ID, tgt.ID, responseBody,
|
||||||
|
)
|
||||||
|
|
||||||
|
return renderSourceLogsPage(t, h, sess, wh.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_RendersFailedAttempt is the regression
|
||||||
|
// test for the reported gap: a failed delivery used to render
|
||||||
|
// as the status word alone, so diagnosing it meant opening the
|
||||||
|
// per-webhook SQLite file by hand.
|
||||||
|
func TestHandleSourceLogs_RendersFailedAttempt(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
body := seedFailureAndRender(
|
||||||
|
t,
|
||||||
|
database.TargetTypeHTTP,
|
||||||
|
`{"url":"https://example.com/hook/abc"}`,
|
||||||
|
"upstream exploded",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert.Contains(
|
||||||
|
t, body, strconv.Itoa(attemptStatusCode),
|
||||||
|
"the attempt's status code must reach the page",
|
||||||
|
)
|
||||||
|
assert.Contains(
|
||||||
|
t, body, attemptError,
|
||||||
|
"the attempt's error must reach the page",
|
||||||
|
)
|
||||||
|
assert.Contains(
|
||||||
|
t, body, strconv.Itoa(attemptDurationMS),
|
||||||
|
"the attempt's duration must reach the page",
|
||||||
|
)
|
||||||
|
assert.Contains(
|
||||||
|
t, body, "Attempt "+strconv.Itoa(attemptNumber),
|
||||||
|
"the attempt number must reach the page",
|
||||||
|
)
|
||||||
|
assert.Contains(
|
||||||
|
t, body, "upstream exploded",
|
||||||
|
"the attempt's response body must reach the page",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_EscapesResponseBody proves the
|
||||||
|
// response body is treated as the untrusted remote content it
|
||||||
|
// is. The remote chooses these bytes and the page is rendered
|
||||||
|
// inside the operator's authenticated origin, where the
|
||||||
|
// application's own CSP allows inline script from 'self'.
|
||||||
|
func TestHandleSourceLogs_EscapesResponseBody(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const payload = `<script>alert("xss")</script>`
|
||||||
|
|
||||||
|
body := seedFailureAndRender(
|
||||||
|
t,
|
||||||
|
database.TargetTypeHTTP,
|
||||||
|
`{"url":"https://example.com/hook/abc"}`,
|
||||||
|
payload,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert.NotContains(t, body, payload)
|
||||||
|
assert.NotContains(t, body, "<script>alert")
|
||||||
|
assert.Contains(t, body, "alert")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_RedactsCredentialEchoedInResponse
|
||||||
|
// covers the case that makes rendering a response body a
|
||||||
|
// disclosure question at all: the remote echoes back the
|
||||||
|
// credential the request carried, and the page would then put
|
||||||
|
// it on the operator's screen.
|
||||||
|
func TestHandleSourceLogs_RedactsCredentialEchoedInResponse(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
body := seedFailureAndRender(
|
||||||
|
t,
|
||||||
|
database.TargetTypeSlack,
|
||||||
|
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||||
|
"no_service: "+slackWebhookURL,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert.NotContains(t, body, slackSecretPath)
|
||||||
|
assert.NotContains(t, body, "T00000000")
|
||||||
|
assert.NotContains(t, body, "B00000000")
|
||||||
|
assert.Contains(t, body, delivery.RedactionMarker)
|
||||||
|
|
||||||
|
// The rest of the response is still shown, or the
|
||||||
|
// redaction would have cost the operator the diagnosis.
|
||||||
|
assert.Contains(t, body, "no_service")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_RedactsCredentialEchoedInError covers
|
||||||
|
// the same disclosure through the error field. The delivery
|
||||||
|
// engine masks the URL out of the errors it stores, so this
|
||||||
|
// holds the read path to the rows written before it did.
|
||||||
|
func TestHandleSourceLogs_RedactsCredentialEchoedInError(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
tgt := seedConfiguredTarget(
|
||||||
|
t, db, wh.ID,
|
||||||
|
database.TargetTypeSlack,
|
||||||
|
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||||
|
)
|
||||||
|
|
||||||
|
dlv := seedFailedDeliveryWithResponse(t, dbMgr, wh.ID, tgt.ID, "")
|
||||||
|
|
||||||
|
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
// An unmasked transport error, exactly as Go's HTTP
|
||||||
|
// client renders one.
|
||||||
|
require.NoError(t, webhookDB.Model(
|
||||||
|
&database.DeliveryResult{},
|
||||||
|
).Where(
|
||||||
|
"delivery_id = ?", dlv.ID,
|
||||||
|
).Update(
|
||||||
|
"error",
|
||||||
|
`Post "`+slackWebhookURL+`": dial tcp: i/o timeout`,
|
||||||
|
).Error)
|
||||||
|
|
||||||
|
body := renderSourceLogsPage(t, h, sess, wh.ID)
|
||||||
|
|
||||||
|
assert.NotContains(t, body, slackSecretPath)
|
||||||
|
assert.Contains(t, body, delivery.RedactionMarker)
|
||||||
|
assert.Contains(t, body, "i/o timeout")
|
||||||
|
}
|
||||||
|
|
||||||
|
// severedPadding is the filler that puts the end of an echoed
|
||||||
|
// webhook URL five bytes past a cut at the response cap, so
|
||||||
|
// the cut leaves the workspace ID, the bot ID and all but the
|
||||||
|
// last few token characters behind.
|
||||||
|
func severedPadding() string {
|
||||||
|
const severedTail = 5
|
||||||
|
|
||||||
|
return strings.Repeat(
|
||||||
|
"A", responseCap-len(slackWebhookURL)+severedTail,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_RedactsCredentialSeveredByTheEngineCut
|
||||||
|
// is the regression test for a redactor gated on the SQL cut
|
||||||
|
// alone. The delivery engine stops reading a response at its
|
||||||
|
// own cap, which is the same number of bytes this page
|
||||||
|
// renders, so a row the engine cut is byte-for-byte
|
||||||
|
// indistinguishable from a complete response and that gate
|
||||||
|
// never opened on anything the engine writes.
|
||||||
|
//
|
||||||
|
// The seeded body is what the engine stores for any remote
|
||||||
|
// that sends at least that much: exactly responseCap bytes,
|
||||||
|
// ending in a severed webhook URL.
|
||||||
|
// TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog in
|
||||||
|
// internal/delivery pins that this is the size it produces.
|
||||||
|
func TestHandleSourceLogs_RedactsCredentialSeveredByTheEngineCut(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
sent := severedPadding() + slackWebhookURL +
|
||||||
|
strings.Repeat("Z", 128)
|
||||||
|
stored := sent[:responseCap]
|
||||||
|
|
||||||
|
require.Len(
|
||||||
|
t, stored, responseCap,
|
||||||
|
"the engine stores exactly the cap, never more",
|
||||||
|
)
|
||||||
|
require.Contains(
|
||||||
|
t, stored, "T00000000",
|
||||||
|
"the severed credential must be in what is seeded",
|
||||||
|
)
|
||||||
|
|
||||||
|
body := seedFailureAndRender(
|
||||||
|
t,
|
||||||
|
database.TargetTypeSlack,
|
||||||
|
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||||
|
stored,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert.NotContains(t, body, "T00000000")
|
||||||
|
assert.NotContains(t, body, "B00000000")
|
||||||
|
assert.Contains(t, body, delivery.RedactionMarker)
|
||||||
|
assert.Contains(
|
||||||
|
t, body, "reached the recording limit",
|
||||||
|
"a body the engine cut must not be shown as complete",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_RedactsCredentialSeveredBySQLCut covers
|
||||||
|
// the same severing for a row larger than the cap, which is
|
||||||
|
// SQLite's cut rather than the engine's. The current engine
|
||||||
|
// writes no such row; rows predating its cap or restored from
|
||||||
|
// an archive are not bounded by it, which is why the page cuts
|
||||||
|
// again in SQL and has to redact that cut too.
|
||||||
|
func TestHandleSourceLogs_RedactsCredentialSeveredBySQLCut(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
stored := severedPadding() + slackWebhookURL +
|
||||||
|
strings.Repeat("Z", 128)
|
||||||
|
|
||||||
|
require.Greater(
|
||||||
|
t, len(stored), responseCap,
|
||||||
|
"the stored body must exceed the cap or nothing is cut",
|
||||||
|
)
|
||||||
|
|
||||||
|
body := seedFailureAndRender(
|
||||||
|
t,
|
||||||
|
database.TargetTypeSlack,
|
||||||
|
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||||
|
stored,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert.NotContains(t, body, "T00000000")
|
||||||
|
assert.NotContains(t, body, "B00000000")
|
||||||
|
assert.NotContains(
|
||||||
|
t, body, slackWebhookURL[:len(slackWebhookURL)-10],
|
||||||
|
)
|
||||||
|
assert.Contains(t, body, delivery.RedactionMarker)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_RedactsForSoftDeletedTarget covers a
|
||||||
|
// target an operator has deleted. The row is only soft deleted
|
||||||
|
// and its deliveries survive in the per-webhook database, so
|
||||||
|
// its redactor has to survive with it or every response body
|
||||||
|
// it ever recorded renders unredacted.
|
||||||
|
func TestHandleSourceLogs_RedactsForSoftDeletedTarget(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
tgt := seedConfiguredTarget(
|
||||||
|
t, db, wh.ID,
|
||||||
|
database.TargetTypeSlack,
|
||||||
|
`{"webhookUrl":"`+slackWebhookURL+`"}`,
|
||||||
|
)
|
||||||
|
|
||||||
|
seedFailedDeliveryWithResponse(
|
||||||
|
t, dbMgr, wh.ID, tgt.ID,
|
||||||
|
"no_service: "+slackWebhookURL,
|
||||||
|
)
|
||||||
|
|
||||||
|
require.NoError(t, db.DB().Delete(tgt).Error)
|
||||||
|
|
||||||
|
body := renderSourceLogsPage(t, h, sess, wh.ID)
|
||||||
|
|
||||||
|
assert.NotContains(t, body, slackSecretPath)
|
||||||
|
assert.NotContains(t, body, "T00000000")
|
||||||
|
assert.Contains(t, body, delivery.RedactionMarker)
|
||||||
|
assert.Contains(t, body, "no_service")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_BoundsRenderedAttempts pins the ceiling
|
||||||
|
// on how many of one delivery's attempts reach the page, and
|
||||||
|
// that what it drops is counted rather than hidden.
|
||||||
|
func TestHandleSourceLogs_BoundsRenderedAttempts(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const extraAttempts = 7
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
tgt := seedConfiguredTarget(
|
||||||
|
t, db, wh.ID, database.TargetTypeLog, "",
|
||||||
|
)
|
||||||
|
|
||||||
|
dlv := seedFailedDeliveryWithResponse(t, dbMgr, wh.ID, tgt.ID, "")
|
||||||
|
|
||||||
|
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
total := handlers.MaxRenderedAttemptsForTest + extraAttempts
|
||||||
|
|
||||||
|
// seedFailedDeliveryWithResponse already recorded one attempt.
|
||||||
|
for i := range total - 1 {
|
||||||
|
require.NoError(t, webhookDB.Omit(
|
||||||
|
clause.Associations,
|
||||||
|
).Create(&database.DeliveryResult{
|
||||||
|
DeliveryID: dlv.ID,
|
||||||
|
AttemptNum: attemptNumber + 1 + i,
|
||||||
|
Error: attemptError,
|
||||||
|
}).Error)
|
||||||
|
}
|
||||||
|
|
||||||
|
views := h.LoadEventLogViewsForTest(
|
||||||
|
httptest.NewRecorder(), *wh, 1,
|
||||||
|
)
|
||||||
|
require.Len(t, views, 1)
|
||||||
|
require.Len(t, views[0].Deliveries, 1)
|
||||||
|
|
||||||
|
dv := views[0].Deliveries[0]
|
||||||
|
|
||||||
|
assert.Equal(t, total, dv.AttemptCount)
|
||||||
|
assert.Len(
|
||||||
|
t, dv.Results, handlers.MaxRenderedAttemptsForTest,
|
||||||
|
)
|
||||||
|
assert.Equal(t, extraAttempts, dv.AttemptsOmitted)
|
||||||
|
|
||||||
|
page := renderSourceLogsPage(t, h, sess, wh.ID)
|
||||||
|
|
||||||
|
assert.Contains(t, page, "attempts omitted")
|
||||||
|
assert.Contains(
|
||||||
|
t, page, strconv.Itoa(total)+" attempts",
|
||||||
|
"the header must count every recorded attempt",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceLogs_BoundsOversizeResponse proves the
|
||||||
|
// rendered page is bounded by the response cap rather than by
|
||||||
|
// the stored response size. The cut happens in SQLite, so the
|
||||||
|
// oversized value never becomes a Go string; this asserts the
|
||||||
|
// observable consequence, that neither the page nor the
|
||||||
|
// projection carries the tail.
|
||||||
|
func TestHandleSourceLogs_BoundsOversizeResponse(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const tail = "QQRESPONSETAILQQ"
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
tgt := seedConfiguredTarget(
|
||||||
|
t, db, wh.ID, database.TargetTypeLog, "",
|
||||||
|
)
|
||||||
|
|
||||||
|
stored := strings.Repeat("A", responseCap*4) + tail
|
||||||
|
seedFailedDeliveryWithResponse(t, dbMgr, wh.ID, tgt.ID, stored)
|
||||||
|
|
||||||
|
views := h.LoadEventLogViewsForTest(
|
||||||
|
httptest.NewRecorder(), *wh, 1,
|
||||||
|
)
|
||||||
|
require.Len(t, views, 1)
|
||||||
|
require.Len(t, views[0].Deliveries, 1)
|
||||||
|
require.Len(t, views[0].Deliveries[0].Results, 1)
|
||||||
|
|
||||||
|
attempt := views[0].Deliveries[0].Results[0]
|
||||||
|
|
||||||
|
assert.LessOrEqual(
|
||||||
|
t, len(attempt.ResponseBody), responseCap,
|
||||||
|
)
|
||||||
|
assert.Equal(
|
||||||
|
t, int64(len(stored)), attempt.ResponseBytes,
|
||||||
|
)
|
||||||
|
assert.True(t, attempt.ResponseTruncated)
|
||||||
|
|
||||||
|
page := renderSourceLogsPage(t, h, sess, wh.ID)
|
||||||
|
|
||||||
|
assert.NotContains(t, page, tail)
|
||||||
|
assert.Contains(
|
||||||
|
t, page, "Response truncated for display",
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -19,6 +19,14 @@ func (s *Handlers) SetLogForTest(log *slog.Logger) {
|
|||||||
// to the handlers_test package.
|
// to the handlers_test package.
|
||||||
const MaxRenderedBodyBytesForTest = maxRenderedBodyBytes
|
const MaxRenderedBodyBytesForTest = maxRenderedBodyBytes
|
||||||
|
|
||||||
|
// MaxRenderedResponseBytesForTest exposes the event log's
|
||||||
|
// delivery response cap to the handlers_test package.
|
||||||
|
const MaxRenderedResponseBytesForTest = maxRenderedResponseBytes
|
||||||
|
|
||||||
|
// MaxRenderedAttemptsForTest exposes the event log's
|
||||||
|
// per-delivery attempt ceiling to the handlers_test package.
|
||||||
|
const MaxRenderedAttemptsForTest = maxRenderedAttempts
|
||||||
|
|
||||||
// DummyVerificationsForTest reports how many equivalent-cost
|
// DummyVerificationsForTest reports how many equivalent-cost
|
||||||
// verifications were charged for usernames that do not exist. It
|
// verifications were charged for usernames that do not exist. It
|
||||||
// lets a test prove the anti-enumeration path ran without timing
|
// lets a test prove the anti-enumeration path ran without timing
|
||||||
@@ -43,7 +51,9 @@ func (s *Handlers) LoadEventLogViewsForTest(
|
|||||||
webhook database.Webhook,
|
webhook database.Webhook,
|
||||||
page int,
|
page int,
|
||||||
) []EventLogView {
|
) []EventLogView {
|
||||||
views, _ := s.loadEventsWithDeliveries(w, webhook, nil, page)
|
views, _, _ := s.loadEventsWithDeliveries(
|
||||||
|
w, webhook, nil, page,
|
||||||
|
)
|
||||||
|
|
||||||
return views
|
return views
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -60,6 +60,7 @@ type HandlersParams struct {
|
|||||||
Middleware *middleware.Middleware
|
Middleware *middleware.Middleware
|
||||||
Notifier delivery.Notifier
|
Notifier delivery.Notifier
|
||||||
Evictor delivery.WebhookEvictor
|
Evictor delivery.WebhookEvictor
|
||||||
|
SSRFGuard *delivery.Guard
|
||||||
}
|
}
|
||||||
|
|
||||||
// Handlers provides HTTP handler methods for all application
|
// Handlers provides HTTP handler methods for all application
|
||||||
@@ -77,6 +78,11 @@ type Handlers struct {
|
|||||||
mtr *metrics.Set
|
mtr *metrics.Set
|
||||||
templates map[string]*template.Template
|
templates map[string]*template.Template
|
||||||
|
|
||||||
|
// ssrf validates submitted target URLs. It is the same guard
|
||||||
|
// the delivery engine dials through, so a URL accepted here
|
||||||
|
// is one delivery will actually attempt.
|
||||||
|
ssrf *delivery.Guard
|
||||||
|
|
||||||
// dummyVerifications counts the equivalent-cost verifications
|
// dummyVerifications counts the equivalent-cost verifications
|
||||||
// charged for usernames that do not exist. It exists so a test
|
// charged for usernames that do not exist. It exists so a test
|
||||||
// can prove that path runs without measuring wall-clock time.
|
// can prove that path runs without measuring wall-clock time.
|
||||||
@@ -117,6 +123,7 @@ func New(
|
|||||||
s.notifier = params.Notifier
|
s.notifier = params.Notifier
|
||||||
s.evictor = params.Evictor
|
s.evictor = params.Evictor
|
||||||
s.mtr = metrics.Default()
|
s.mtr = metrics.Default()
|
||||||
|
s.ssrf = params.SSRFGuard
|
||||||
|
|
||||||
// Parse all page templates once at startup
|
// Parse all page templates once at startup
|
||||||
s.templates = map[string]*template.Template{
|
s.templates = map[string]*template.Template{
|
||||||
|
|||||||
@@ -110,6 +110,7 @@ func newTestApp(
|
|||||||
return r
|
return r
|
||||||
},
|
},
|
||||||
middleware.New,
|
middleware.New,
|
||||||
|
delivery.NewGuard,
|
||||||
handlers.New,
|
handlers.New,
|
||||||
),
|
),
|
||||||
fx.Populate(targets...),
|
fx.Populate(targets...),
|
||||||
|
|||||||
@@ -15,6 +15,11 @@ import (
|
|||||||
"sneak.berlin/go/webhooker/internal/session"
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// contentTypeJSON is the content type the seeded events in this
|
||||||
|
// package carry. Shared across the seed helpers so the literal
|
||||||
|
// appears once.
|
||||||
|
const contentTypeJSON = "application/json"
|
||||||
|
|
||||||
// seedDeliveredEvent records an event and a delivery for it in
|
// seedDeliveredEvent records an event and a delivery for it in
|
||||||
// the webhook's own database, so the log page has a delivery
|
// the webhook's own database, so the log page has a delivery
|
||||||
// to render against the target.
|
// to render against the target.
|
||||||
@@ -32,7 +37,7 @@ func seedDeliveredEvent(
|
|||||||
WebhookID: webhookID,
|
WebhookID: webhookID,
|
||||||
Method: http.MethodPost,
|
Method: http.MethodPost,
|
||||||
Body: `{"test":true}`,
|
Body: `{"test":true}`,
|
||||||
ContentType: "application/json",
|
ContentType: contentTypeJSON,
|
||||||
}
|
}
|
||||||
|
|
||||||
require.NoError(t, webhookDB.Omit(
|
require.NoError(t, webhookDB.Omit(
|
||||||
|
|||||||
@@ -4,11 +4,13 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/go-chi/chi"
|
"github.com/go-chi/chi"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
|
"gorm.io/gorm"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
"sneak.berlin/go/webhooker/internal/signature"
|
"sneak.berlin/go/webhooker/internal/signature"
|
||||||
@@ -101,6 +103,32 @@ type DeliveryView struct {
|
|||||||
ID string
|
ID string
|
||||||
Status database.DeliveryStatus
|
Status database.DeliveryStatus
|
||||||
Target delivery.TargetView
|
Target delivery.TargetView
|
||||||
|
|
||||||
|
// Results is this delivery's attempts in attempt order,
|
||||||
|
// bounded by maxRenderedAttempts. Without them a failure
|
||||||
|
// renders as the status word alone and says nothing about
|
||||||
|
// why.
|
||||||
|
Results []DeliveryResultView
|
||||||
|
|
||||||
|
// AttemptCount is how many attempts were recorded, which
|
||||||
|
// is more than len(Results) once the middle was dropped.
|
||||||
|
AttemptCount int
|
||||||
|
|
||||||
|
// AttemptsOmitted is how many attempts were dropped from
|
||||||
|
// the middle of Results. The page must show it, or the
|
||||||
|
// bound would hide history rather than fold it.
|
||||||
|
AttemptsOmitted int
|
||||||
|
}
|
||||||
|
|
||||||
|
// eventLogTarget is what the event log needs to know about
|
||||||
|
// one target: the display-safe view its template renders, and
|
||||||
|
// the redactor that keeps that target's own credential out of
|
||||||
|
// the text its remote peer chose. The two are kept together
|
||||||
|
// so a caller cannot pick up one without the other, and apart
|
||||||
|
// from TargetView so the secrets never reach a template.
|
||||||
|
type eventLogTarget struct {
|
||||||
|
View delivery.TargetView
|
||||||
|
Redactor delivery.Redactor
|
||||||
}
|
}
|
||||||
|
|
||||||
// HandleSourceList shows a list of user's webhooks.
|
// HandleSourceList shows a list of user's webhooks.
|
||||||
@@ -769,12 +797,24 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
targets := h.loadTargetMap(webhook.ID)
|
targets, err := h.loadTargetMap(webhook.ID)
|
||||||
|
if err != nil {
|
||||||
|
// Without the map every delivery renders through a
|
||||||
|
// zero redactor, so failing the page is the only
|
||||||
|
// safe answer.
|
||||||
|
h.serverError(w, "failed to load targets", err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
page := h.parsePage(r)
|
page := h.parsePage(r)
|
||||||
|
|
||||||
evts, total := h.loadEventsWithDeliveries(
|
evts, total, ok := h.loadEventsWithDeliveries(
|
||||||
w, webhook, targets, page,
|
w, webhook, targets, page,
|
||||||
)
|
)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
totalPages := int(total) / paginationPerPage
|
totalPages := int(total) / paginationPerPage
|
||||||
if int(total)%paginationPerPage != 0 {
|
if int(total)%paginationPerPage != 0 {
|
||||||
@@ -807,29 +847,54 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// loadTargetMap loads targets into a map of display-safe
|
// loadTargetMap loads targets into a map of display-safe
|
||||||
// views keyed by target ID. The projection happens here so
|
// views keyed by target ID, each paired with its redactor.
|
||||||
// that no caller can hand a raw target, configuration blob
|
// The projection happens here so that no caller can hand a
|
||||||
// and all, to a template.
|
// raw target, configuration blob and all, to a template: the
|
||||||
|
// raw rows do not leave this function.
|
||||||
|
//
|
||||||
|
// The load is Unscoped because deleting a target only soft
|
||||||
|
// deletes the row while its deliveries survive in the
|
||||||
|
// per-webhook database: a scoped load leaves those deliveries
|
||||||
|
// with a zero redactor, which renders their response bodies
|
||||||
|
// unredacted. Only the redactor half of the map is built from
|
||||||
|
// deleted rows. The view half, which is what the page lists,
|
||||||
|
// stays scoped.
|
||||||
func (h *Handlers) loadTargetMap(
|
func (h *Handlers) loadTargetMap(
|
||||||
webhookID string,
|
webhookID string,
|
||||||
) map[string]delivery.TargetView {
|
) (map[string]eventLogTarget, error) {
|
||||||
var targets []database.Target
|
var targets []database.Target
|
||||||
|
|
||||||
h.db.DB().Where(
|
err := h.db.DB().Unscoped().Where(
|
||||||
"webhook_id = ?", webhookID,
|
"webhook_id = ?", webhookID,
|
||||||
).Find(&targets)
|
).Find(&targets).Error
|
||||||
|
if err != nil {
|
||||||
views := delivery.NewTargetViews(targets)
|
return nil, err
|
||||||
|
|
||||||
targetMap := make(
|
|
||||||
map[string]delivery.TargetView, len(views),
|
|
||||||
)
|
|
||||||
|
|
||||||
for _, v := range views {
|
|
||||||
targetMap[v.ID] = v
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return targetMap
|
targetMap := make(
|
||||||
|
map[string]eventLogTarget, len(targets),
|
||||||
|
)
|
||||||
|
live := make([]database.Target, 0, len(targets))
|
||||||
|
|
||||||
|
for i := range targets {
|
||||||
|
targetMap[targets[i].ID] = eventLogTarget{
|
||||||
|
Redactor: delivery.NewRedactor(&targets[i]),
|
||||||
|
}
|
||||||
|
|
||||||
|
if !targets[i].DeletedAt.Valid {
|
||||||
|
live = append(live, targets[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The views come from NewTargetViews rather than being
|
||||||
|
// rebuilt here, so the masking rules stay in one place.
|
||||||
|
for _, v := range delivery.NewTargetViews(live) {
|
||||||
|
entry := targetMap[v.ID]
|
||||||
|
entry.View = v
|
||||||
|
targetMap[v.ID] = entry
|
||||||
|
}
|
||||||
|
|
||||||
|
return targetMap, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// parsePage extracts a page number from the query string.
|
// parsePage extracts a page number from the query string.
|
||||||
@@ -850,18 +915,22 @@ func (h *Handlers) parsePage(r *http.Request) int {
|
|||||||
// deliveries from the per-webhook database. Events come back
|
// deliveries from the per-webhook database. Events come back
|
||||||
// as capped projections rather than database.Event rows: see
|
// as capped projections rather than database.Event rows: see
|
||||||
// eventLogColumns for why the cut happens in SQL.
|
// eventLogColumns for why the cut happens in SQL.
|
||||||
|
//
|
||||||
|
// The bool reports whether the load succeeded. It is false
|
||||||
|
// once this has answered the request with an error, and the
|
||||||
|
// caller must then render nothing further.
|
||||||
func (h *Handlers) loadEventsWithDeliveries(
|
func (h *Handlers) loadEventsWithDeliveries(
|
||||||
w http.ResponseWriter,
|
w http.ResponseWriter,
|
||||||
webhook database.Webhook,
|
webhook database.Webhook,
|
||||||
targetMap map[string]delivery.TargetView,
|
targetMap map[string]eventLogTarget,
|
||||||
page int,
|
page int,
|
||||||
) ([]EventLogView, int64) {
|
) ([]EventLogView, int64, bool) {
|
||||||
var totalEvents int64
|
var totalEvents int64
|
||||||
|
|
||||||
var result []EventLogView
|
var result []EventLogView
|
||||||
|
|
||||||
if !h.dbMgr.DBExists(webhook.ID) {
|
if !h.dbMgr.DBExists(webhook.ID) {
|
||||||
return result, totalEvents
|
return result, totalEvents, true
|
||||||
}
|
}
|
||||||
|
|
||||||
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
|
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
|
||||||
@@ -870,7 +939,7 @@ func (h *Handlers) loadEventsWithDeliveries(
|
|||||||
w, "failed to get webhook database", err,
|
w, "failed to get webhook database", err,
|
||||||
)
|
)
|
||||||
|
|
||||||
return nil, 0
|
return nil, 0, false
|
||||||
}
|
}
|
||||||
|
|
||||||
webhookDB.Model(&database.Event{}).Where(
|
webhookDB.Model(&database.Event{}).Where(
|
||||||
@@ -890,43 +959,170 @@ func (h *Handlers) loadEventsWithDeliveries(
|
|||||||
).Find(&rows)
|
).Find(&rows)
|
||||||
|
|
||||||
result = make([]EventLogView, len(rows))
|
result = make([]EventLogView, len(rows))
|
||||||
|
eventDeliveries := make([][]database.Delivery, len(rows))
|
||||||
|
|
||||||
|
var deliveryIDs []string
|
||||||
|
|
||||||
for i := range rows {
|
for i := range rows {
|
||||||
result[i] = rows[i].view()
|
result[i] = rows[i].view()
|
||||||
|
|
||||||
var deliveries []database.Delivery
|
|
||||||
|
|
||||||
webhookDB.Where(
|
webhookDB.Where(
|
||||||
"event_id = ?", rows[i].ID,
|
"event_id = ?", rows[i].ID,
|
||||||
).Find(&deliveries)
|
).Find(&eventDeliveries[i])
|
||||||
|
|
||||||
|
for j := range eventDeliveries[i] {
|
||||||
|
deliveryIDs = append(
|
||||||
|
deliveryIDs, eventDeliveries[i][j].ID,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
attempts, err := h.loadDeliveryResults(
|
||||||
|
webhookDB, deliveryIDs,
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
h.serverError(
|
||||||
|
w, "failed to load delivery attempts", err,
|
||||||
|
)
|
||||||
|
|
||||||
|
return nil, 0, false
|
||||||
|
}
|
||||||
|
|
||||||
|
for i := range rows {
|
||||||
result[i].Deliveries = newDeliveryViews(
|
result[i].Deliveries = newDeliveryViews(
|
||||||
deliveries, targetMap,
|
eventDeliveries[i], targetMap, attempts,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
return result, totalEvents
|
return result, totalEvents, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// deliveryIDChunkSize bounds how many delivery IDs go into one
|
||||||
|
// IN clause. SQLite refuses a statement carrying more than
|
||||||
|
// SQLITE_MAX_VARIABLE_NUMBER (32766) bound parameters, and a
|
||||||
|
// page holds one delivery per target per event, so a webhook
|
||||||
|
// with enough targets would turn the whole query into an error
|
||||||
|
// and the page into zero attempts.
|
||||||
|
const deliveryIDChunkSize = 500
|
||||||
|
|
||||||
|
// loadDeliveryResults loads the recorded attempts for the
|
||||||
|
// page's deliveries, keyed by delivery ID.
|
||||||
|
//
|
||||||
|
// Each response body is cut by SQLite rather than in Go, for
|
||||||
|
// the reason deliveryResultColumns gives. How many attempts a
|
||||||
|
// delivery has is the target's MaxRetries, which the
|
||||||
|
// authenticated operator sets; how many of them reach the page
|
||||||
|
// is bounded again by maxRenderedAttempts.
|
||||||
|
func (h *Handlers) loadDeliveryResults(
|
||||||
|
webhookDB *gorm.DB,
|
||||||
|
deliveryIDs []string,
|
||||||
|
) (map[string][]deliveryResultRow, error) {
|
||||||
|
byDelivery := make(map[string][]deliveryResultRow)
|
||||||
|
|
||||||
|
for chunk := range slices.Chunk(
|
||||||
|
deliveryIDs, deliveryIDChunkSize,
|
||||||
|
) {
|
||||||
|
var rows []deliveryResultRow
|
||||||
|
|
||||||
|
err := webhookDB.Model(
|
||||||
|
&database.DeliveryResult{},
|
||||||
|
).Select(
|
||||||
|
deliveryResultColumns, maxRenderedResponseBytes,
|
||||||
|
).Where(
|
||||||
|
"delivery_id IN ?", chunk,
|
||||||
|
).Order("attempt_num ASC").Find(&rows).Error
|
||||||
|
if err != nil {
|
||||||
|
// Returning what was loaded so far renders the
|
||||||
|
// deliveries in the failed chunk as never having run,
|
||||||
|
// which is indistinguishable from ones that really
|
||||||
|
// never ran. The page fails instead.
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
for i := range rows {
|
||||||
|
byDelivery[rows[i].DeliveryID] = append(
|
||||||
|
byDelivery[rows[i].DeliveryID], rows[i],
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return byDelivery, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// newDeliveryViews projects deliveries for rendering,
|
// newDeliveryViews projects deliveries for rendering,
|
||||||
// resolving each one's target to its display-safe view.
|
// resolving each one's target to its display-safe view and
|
||||||
|
// each one's attempts through that target's redactor.
|
||||||
func newDeliveryViews(
|
func newDeliveryViews(
|
||||||
deliveries []database.Delivery,
|
deliveries []database.Delivery,
|
||||||
targetMap map[string]delivery.TargetView,
|
targetMap map[string]eventLogTarget,
|
||||||
|
attempts map[string][]deliveryResultRow,
|
||||||
) []DeliveryView {
|
) []DeliveryView {
|
||||||
views := make([]DeliveryView, len(deliveries))
|
views := make([]DeliveryView, len(deliveries))
|
||||||
|
|
||||||
for i := range deliveries {
|
for i := range deliveries {
|
||||||
|
target := targetMap[deliveries[i].TargetID]
|
||||||
|
rows := attempts[deliveries[i].ID]
|
||||||
|
|
||||||
|
results, omitted := renderedAttempts(
|
||||||
|
rows, target.Redactor,
|
||||||
|
)
|
||||||
|
|
||||||
views[i] = DeliveryView{
|
views[i] = DeliveryView{
|
||||||
ID: deliveries[i].ID,
|
ID: deliveries[i].ID,
|
||||||
Status: deliveries[i].Status,
|
Status: deliveries[i].Status,
|
||||||
Target: targetMap[deliveries[i].TargetID],
|
Target: target.View,
|
||||||
|
Results: results,
|
||||||
|
AttemptCount: len(rows),
|
||||||
|
AttemptsOmitted: omitted,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return views
|
return views
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// maxRenderedAttempts bounds how many of one delivery's
|
||||||
|
// attempts the page renders. Past it the middle is dropped and
|
||||||
|
// counted, keeping the first attempts and the last ones: how
|
||||||
|
// the delivery started failing and how it ended are what a
|
||||||
|
// reader needs, and the count says plainly that the rest was
|
||||||
|
// dropped rather than never recorded.
|
||||||
|
const (
|
||||||
|
renderedAttemptsHead = 10
|
||||||
|
renderedAttemptsTail = 10
|
||||||
|
maxRenderedAttempts = renderedAttemptsHead +
|
||||||
|
renderedAttemptsTail
|
||||||
|
)
|
||||||
|
|
||||||
|
// renderedAttempts projects a delivery's attempts through the
|
||||||
|
// target's redactor, at most maxRenderedAttempts of them, and
|
||||||
|
// reports how many it dropped.
|
||||||
|
func renderedAttempts(
|
||||||
|
rows []deliveryResultRow,
|
||||||
|
redactor delivery.Redactor,
|
||||||
|
) ([]DeliveryResultView, int) {
|
||||||
|
omitted := 0
|
||||||
|
|
||||||
|
if len(rows) > maxRenderedAttempts {
|
||||||
|
omitted = len(rows) - maxRenderedAttempts
|
||||||
|
|
||||||
|
kept := make(
|
||||||
|
[]deliveryResultRow, 0, maxRenderedAttempts,
|
||||||
|
)
|
||||||
|
kept = append(kept, rows[:renderedAttemptsHead]...)
|
||||||
|
kept = append(
|
||||||
|
kept, rows[len(rows)-renderedAttemptsTail:]...,
|
||||||
|
)
|
||||||
|
rows = kept
|
||||||
|
}
|
||||||
|
|
||||||
|
views := make([]DeliveryResultView, len(rows))
|
||||||
|
for i := range rows {
|
||||||
|
views[i] = rows[i].view(redactor)
|
||||||
|
}
|
||||||
|
|
||||||
|
return views, omitted
|
||||||
|
}
|
||||||
|
|
||||||
// HandleEntrypointCreate handles adding a new entrypoint.
|
// HandleEntrypointCreate handles adding a new entrypoint.
|
||||||
func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
|
func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -1411,7 +1607,7 @@ func (h *Handlers) validateTargetURL(
|
|||||||
return errMissingURL
|
return errMissingURL
|
||||||
}
|
}
|
||||||
|
|
||||||
err := delivery.ValidateTargetURL(
|
err := h.ssrf.ValidateTargetURL(
|
||||||
r.Context(), targetURL,
|
r.Context(), targetURL,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -164,6 +164,7 @@ func newServerApp(
|
|||||||
func() delivery.Notifier { return &noopNotifier{} },
|
func() delivery.Notifier { return &noopNotifier{} },
|
||||||
func() delivery.WebhookEvictor { return &noopEvictor{} },
|
func() delivery.WebhookEvictor { return &noopEvictor{} },
|
||||||
middleware.New,
|
middleware.New,
|
||||||
|
delivery.NewGuard,
|
||||||
handlers.New,
|
handlers.New,
|
||||||
),
|
),
|
||||||
fx.Populate(&h),
|
fx.Populate(&h),
|
||||||
|
|||||||
@@ -113,6 +113,7 @@ func newTestEnvWithConfig(
|
|||||||
func() delivery.Notifier { return &noopNotifier{} },
|
func() delivery.Notifier { return &noopNotifier{} },
|
||||||
func() delivery.WebhookEvictor { return &noopEvictor{} },
|
func() delivery.WebhookEvictor { return &noopEvictor{} },
|
||||||
middleware.New,
|
middleware.New,
|
||||||
|
delivery.NewGuard,
|
||||||
handlers.New,
|
handlers.New,
|
||||||
),
|
),
|
||||||
fx.Populate(&log, &mw, &hnd, &sess, &db, &dbMgr),
|
fx.Populate(&log, &mw, &hnd, &sess, &db, &dbMgr),
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -46,20 +46,65 @@
|
|||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if .Deliveries}}
|
{{if .Deliveries}}
|
||||||
<div class="mt-3 pt-4 border-t border-gray-200">
|
<div class="mt-4 border-t border-gray-200 pt-3">
|
||||||
|
<h3 class="text-xs font-medium uppercase tracking-wide text-gray-500">Deliveries</h3>
|
||||||
|
<div class="mt-2 divide-y divide-gray-200">
|
||||||
{{range .Deliveries}}
|
{{range .Deliveries}}
|
||||||
<div class="flex items-center justify-between py-2 text-xs text-gray-700">
|
<div class="py-2" x-data="{ attempts: false }">
|
||||||
<span><span class="font-medium">{{.Target.Name}}</span>: {{.Status}}</span>
|
<div class="flex items-center justify-between cursor-pointer" @click="attempts = !attempts">
|
||||||
|
<div class="flex items-center gap-3">
|
||||||
|
<span class="text-sm text-gray-700">{{.Target.Name}}</span>
|
||||||
|
<span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{.Status}}</span>
|
||||||
|
</div>
|
||||||
|
<div class="flex items-center gap-3">
|
||||||
{{if .Status.Terminal}}
|
{{if .Status.Terminal}}
|
||||||
<form method="POST" action="/source/{{$.Webhook.ID}}/deliveries/{{.ID}}/replay" class="inline">
|
<form method="POST" action="/source/{{$.Webhook.ID}}/deliveries/{{.ID}}/replay" class="inline" @click.stop>
|
||||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||||
<input type="hidden" name="page" value="{{$.Page}}">
|
<input type="hidden" name="page" value="{{$.Page}}">
|
||||||
<button type="submit" class="text-xs text-primary-600 hover:text-primary-700" title="Send this event to the target again">Replay</button>
|
<button type="submit" class="text-xs text-primary-600 hover:text-primary-700" title="Send this event to the target again">Replay</button>
|
||||||
</form>
|
</form>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
<span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>
|
||||||
|
<svg class="w-3 h-3 text-gray-400 transition-transform" :class="{ 'rotate-180': attempts }" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"/>
|
||||||
|
</svg>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div x-show="attempts" x-cloak class="mt-2 space-y-2">
|
||||||
|
{{if .AttemptsOmitted}}
|
||||||
|
<p class="text-xs text-gray-500">{{.AttemptsOmitted}} attempt{{if ne .AttemptsOmitted 1}}s{{end}} omitted between the first and last shown.</p>
|
||||||
|
{{end}}
|
||||||
|
{{range .Results}}
|
||||||
|
<div class="rounded-md bg-white border border-gray-200 p-2">
|
||||||
|
<div class="flex flex-wrap items-center gap-3 text-xs">
|
||||||
|
<span class="text-gray-500">Attempt {{.AttemptNum}}</span>
|
||||||
|
<span class="{{if .Success}}text-green-600{{else}}text-red-600{{end}}">{{if .Success}}success{{else}}failure{{end}}</span>
|
||||||
|
<span class="text-gray-500">Status: {{if .HasStatusCode}}{{.StatusCode}}{{else}}— (no response){{end}}</span>
|
||||||
|
<span class="text-gray-500">Duration: {{.DurationMS}} ms</span>
|
||||||
|
</div>
|
||||||
|
{{if .Error}}
|
||||||
|
<p class="mt-2 text-xs text-red-700 break-all">Error: {{.Error}}</p>
|
||||||
|
{{end}}
|
||||||
|
{{if .ResponseBody}}
|
||||||
|
<pre class="mt-2 text-xs text-gray-700 overflow-x-auto whitespace-pre-wrap break-all">{{.ResponseBody}}</pre>
|
||||||
|
{{end}}
|
||||||
|
{{if .ResponseTruncated}}
|
||||||
|
{{if .ResponseSizeKnown}}
|
||||||
|
<p class="mt-1 text-xs text-gray-500">Response truncated for display: showing {{.ResponseShownBytes}} of {{.ResponseBytes}} bytes.</p>
|
||||||
|
{{else}}
|
||||||
|
<p class="mt-1 text-xs text-gray-500">Showing {{.ResponseShownBytes}} of the {{.ResponseBytes}} recorded bytes. The response reached the recording limit, so the remote may have sent more that was never stored.</p>
|
||||||
|
{{end}}
|
||||||
|
{{end}}
|
||||||
|
</div>
|
||||||
|
{{else}}
|
||||||
|
<p class="text-xs text-gray-500">No attempts recorded yet.</p>
|
||||||
|
{{end}}
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
</div>
|
</div>
|
||||||
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
Reference in New Issue
Block a user