Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c5812c9964 | ||
|
|
eda8e80844 | ||
|
|
e584718eb5 | ||
|
|
a8d226942e | ||
|
|
6df1fc7875 |
+1
-3
@@ -72,10 +72,8 @@
|
||||
**/*.sublime-*
|
||||
|
||||
# This repository's own host-built artifacts: the binary `make build`
|
||||
# writes, and the Alpine.js file `make assets` extracts from 3p/ (the
|
||||
# build extracts its own).
|
||||
# writes.
|
||||
/bin
|
||||
/static/js/alpine.min.js
|
||||
|
||||
# SQLite databases, which hold the session key and webhook payloads, at
|
||||
# any depth.
|
||||
|
||||
@@ -77,7 +77,3 @@ data/
|
||||
# Temporary files
|
||||
tmp/
|
||||
temp/
|
||||
|
||||
# Alpine.js, extracted by `make assets` from its tarball in 3p/, which is
|
||||
# what is committed.
|
||||
/static/js/alpine.min.js
|
||||
|
||||
@@ -1,2 +1,3 @@
|
||||
node_modules/
|
||||
yarn.lock
|
||||
static/js/alpine.min.js
|
||||
|
||||
Binary file not shown.
+2
-11
@@ -92,10 +92,6 @@ COPY . .
|
||||
# fmt-check` is the markdown-check stage above.
|
||||
RUN if [ -n "$(gofmt -s -l .)" ]; then echo "gofmt needed on:"; gofmt -s -l .; exit 1; fi
|
||||
|
||||
# static/static.go embeds the Alpine.js file this extracts from 3p/; without
|
||||
# it the static package does not compile and cannot be linted.
|
||||
RUN script/assets
|
||||
|
||||
# The golangci-lint steps run with --network=none. `golangci-lint config
|
||||
# verify` is documented as fetching its JSON schema over HTTPS; this pinned
|
||||
# image resolves the schema without any network, and --network=none enforces
|
||||
@@ -130,9 +126,6 @@ RUN go mod download
|
||||
|
||||
COPY . .
|
||||
|
||||
# static/static.go embeds the Alpine.js file this extracts from 3p/.
|
||||
RUN script/assets
|
||||
|
||||
# -timeout applies to each package on its own, so 90s has only to clear the
|
||||
# slowest one. -p 4 -parallel 8 keep the run under 2 GB of memory: at most
|
||||
# four test binaries build or run at once, each with at most eight parallel
|
||||
@@ -211,10 +204,8 @@ RUN version="$(make version VERSION="$VERSION")"; \
|
||||
esac; \
|
||||
fi
|
||||
|
||||
# Builds through the Makefile's build target, which runs script/assets
|
||||
# (Alpine.js, extracted from its tarball in 3p/) first. The target builds
|
||||
# with cgo off, so the binary is statically linked and runs on the Alpine
|
||||
# runtime image below as it is.
|
||||
# The Makefile's build target builds with cgo off, so the binary is
|
||||
# statically linked and runs on the Alpine runtime image below.
|
||||
RUN make build VERSION="$VERSION"
|
||||
|
||||
# Runtime stage
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ COPY . .
|
||||
# The test binary embeds the templates and static files, so the browser
|
||||
# stage needs nothing else. It is built with cgo off and -p 4 keeps the
|
||||
# compile's memory down, both as in the test phase of Dockerfile.
|
||||
RUN make assets && CGO_ENABLED=0 go test -c -p 4 -tags browser -o /browser.test ./internal/server
|
||||
RUN CGO_ENABLED=0 go test -c -p 4 -tags browser -o /browser.test ./internal/server
|
||||
|
||||
# chromedp/headless-shell:151.0.7922.109 (Debian trixie), 2026-08-11. The
|
||||
# browser is on PATH as headless-shell, where the test's browser library
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
.PHONY: bootstrap setup assets test test-browser lint fmt fmt-check check build run dev deps docker clean hooks css css-check version
|
||||
.PHONY: bootstrap setup test test-browser lint fmt fmt-check check build run dev deps docker clean hooks css css-check version
|
||||
|
||||
# Default target
|
||||
.DEFAULT_GOAL := check
|
||||
@@ -22,9 +22,6 @@ bootstrap:
|
||||
setup:
|
||||
@script/setup
|
||||
|
||||
assets:
|
||||
@script/assets
|
||||
|
||||
test:
|
||||
@script/test
|
||||
|
||||
@@ -47,13 +44,13 @@ check:
|
||||
# REPO_POLICIES.md does: the binary carries no build paths, symbol table or
|
||||
# debug information, and on Linux, the image's included, it is statically
|
||||
# linked.
|
||||
build: assets
|
||||
build:
|
||||
CGO_ENABLED=0 go build -trimpath -ldflags '-s -w -X main.version=$(VERSION)' -o bin/webhooker ./cmd/webhooker
|
||||
|
||||
run: build
|
||||
./bin/webhooker
|
||||
|
||||
dev: assets
|
||||
dev:
|
||||
go run ./cmd/webhooker
|
||||
|
||||
deps:
|
||||
|
||||
@@ -39,15 +39,15 @@ Five words name the moving parts.
|
||||
The part of webhooker that answers requests to `/h/{uuid}` is the receiver. It
|
||||
stores the event and one pending delivery per active target, then answers `200`.
|
||||
|
||||
Delivery is at least once, not exactly once. If a delivery reaches the target
|
||||
but webhooker cannot record that outcome, it sends the delivery again later. A
|
||||
delivery in flight when the service stops is also sent again at the next start.
|
||||
webhooker adds no delivery identifier of its own to the request, so make your
|
||||
targets tolerate a duplicate.
|
||||
Delivery is at least once, not exactly once. A delivery can reach its target
|
||||
more than once. If a delivery reaches the target but webhooker cannot record
|
||||
that outcome, it sends the delivery again later. A delivery in flight when the
|
||||
service stops is also sent again at the next start. webhooker adds no delivery
|
||||
identifier of its own to the request, so make your targets tolerate a duplicate.
|
||||
|
||||
## Getting Started
|
||||
|
||||
You need the Go version named in `go.mod`, and Docker for the checks.
|
||||
You need the Go version named in `go.mod`.
|
||||
|
||||
```bash
|
||||
git clone https://git.eeqj.de/sneak/webhooker.git
|
||||
@@ -57,7 +57,8 @@ DATA_DIR=../webhooker-data make dev
|
||||
```
|
||||
|
||||
`DATA_DIR` is where webhooker keeps its databases. It defaults to
|
||||
`/var/lib/webhooker`, so point it at a directory you can write to.
|
||||
`/var/lib/webhooker`, so point it at a directory you can write to. `make build`
|
||||
writes the program to `bin/webhooker`. Running that file starts webhooker.
|
||||
|
||||
The first start creates the `admin` account and prints its password once, in a
|
||||
ruled box on standard output:
|
||||
@@ -105,9 +106,9 @@ sooner after 24 hours without use, unless `SESSION_IDLE_TIMEOUT` says otherwise.
|
||||
On the webhook list at `/hooks`, **New Webhook** asks for **Name**, an optional
|
||||
**Description** and **Retention (days)**. Events older than the retention period
|
||||
are deleted, along with their deliveries. The default is 30 days. `0` keeps
|
||||
events forever. The form also has an **HTTP target URL** field and an
|
||||
**Archive** checkbox; each creates that target with the webhook. Every new
|
||||
webhook gets one entrypoint, described as `Default entrypoint`.
|
||||
events forever. Filling in **HTTP target URL** creates an `http` target with the
|
||||
webhook. Ticking **Archive** creates an archive target. Every new webhook gets
|
||||
one entrypoint, described as `Default entrypoint`.
|
||||
|
||||
### Entrypoint URLs
|
||||
|
||||
@@ -138,6 +139,8 @@ Under **Targets**, **Add** asks for the type first, then that type's settings.
|
||||
`http` and `slack` targets also take **Delivery attempts**: how many times each
|
||||
delivery is tried in all. The default is 1. The most is 20. A target's type
|
||||
cannot be changed. Deleting an archive target leaves its files on disk.
|
||||
**Deactivate** and **Activate** on the webhook page switch a target off and on,
|
||||
and new events skip a target that is off.
|
||||
|
||||
### Events
|
||||
|
||||
@@ -147,11 +150,11 @@ body** link. **Full Event Log** shows the 50 newest events with their deliveries
|
||||
and every attempt. **Failed** and **Pending** narrow it to events with a failed
|
||||
delivery, or with one still pending or retrying.
|
||||
|
||||
**Replay** sits beside each finished delivery. It creates a new delivery of the
|
||||
same event to the same target, with the target's settings as they stand now.
|
||||
**Resubmit** sits beside each event. It stores a copy of the event as a new
|
||||
event and delivers it to every target that is active now. Both actions are
|
||||
limited to 30 per minute per client.
|
||||
A finished delivery has a **Replay** button while its target exists. **Replay**
|
||||
creates a new delivery of the same event to the same target, with the target's
|
||||
settings as they stand now. **Resubmit** sits beside each event. It stores a
|
||||
copy of the event as a new event and delivers it to every target that is active
|
||||
now. Both actions are limited to 30 per minute per client.
|
||||
|
||||
### Profile and settings
|
||||
|
||||
@@ -177,7 +180,8 @@ its published port.
|
||||
The app runs as user `webhooker`. The image's start-up script is
|
||||
`deploy/docker-entrypoint.sh`. It gives `DATA_DIR` and its contents to that user
|
||||
before the app starts. So a host directory can be mounted as it is. webhooker
|
||||
creates every SQLite file with mode `0600`.
|
||||
creates every SQLite file readable and writable by that user alone (mode
|
||||
`0600`).
|
||||
|
||||
### Running under upaas
|
||||
|
||||
@@ -192,9 +196,10 @@ Read it with `docker logs` and the name upaas gave the container.
|
||||
|
||||
webhooker speaks plain HTTP. It does not handle TLS, the encryption behind
|
||||
HTTPS: it holds no certificate and accepts no HTTPS connections. The reverse
|
||||
proxy holds the certificate and reaches webhooker over a private address. On a
|
||||
bare host that is loopback, the address only this host can reach. For the image
|
||||
it is the Docker bridge, the private network Docker puts containers on.
|
||||
proxy holds the certificate and reaches webhooker over a private address. A bare
|
||||
host is a host without Docker. On a bare host the private address is loopback
|
||||
(`127.0.0.1`), which only this host can reach. For the image it is the Docker
|
||||
bridge, the private network Docker puts containers on.
|
||||
|
||||
Four things must be right.
|
||||
|
||||
@@ -264,10 +269,11 @@ the server does and refuses while a running instance holds it.
|
||||
DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin
|
||||
```
|
||||
|
||||
Without `-generate`, the new password is read from standard input. It must be at
|
||||
least 8 characters. It is never taken as a command-line argument, because other
|
||||
accounts on the host can read those. The command creates nothing: a missing
|
||||
`DATA_DIR`, a missing `webhooker.db` or an unknown username is an error.
|
||||
`-generate` makes up a random password and prints it. Without `-generate`, the
|
||||
new password is read from standard input. It must be at least 8 bytes long. It
|
||||
is never taken as a command-line argument, because other accounts on the host
|
||||
can read those. The command creates nothing: a missing `DATA_DIR`, a missing
|
||||
`webhooker.db` or an unknown username is an error.
|
||||
|
||||
In a container, run the binary at `/app/webhooker` with the stopped container's
|
||||
volume mounted:
|
||||
@@ -281,8 +287,9 @@ docker run --rm -v /path/to/data:/var/lib/webhooker \
|
||||
|
||||
All state lives in `DATA_DIR`. Back up the whole directory. It holds:
|
||||
|
||||
- the main database, `webhooker.db`: the session key, the `admin` account's
|
||||
password hash, and the webhooks, entrypoints and targets;
|
||||
- the main database, `webhooker.db`: the session key (which signs the session
|
||||
cookies), the `admin` account's password hash, and the webhooks, entrypoints
|
||||
and targets;
|
||||
- one event database per webhook, `events-{webhook_uuid}.db`: its events,
|
||||
deliveries and attempts;
|
||||
- the files of each archive target;
|
||||
@@ -352,9 +359,8 @@ writes it. The reverse proxy's access log does hold it, in the request path.
|
||||
file's type;
|
||||
- `X-Frame-Options: DENY`, which stops other sites showing these pages in a
|
||||
frame;
|
||||
- a Content-Security-Policy (CSP), the header that tells the browser where
|
||||
scripts and styles may come from, set to
|
||||
`default-src 'self'; script-src 'self'; style-src 'self'`;
|
||||
- `Content-Security-Policy: default-src 'self'; script-src 'self'; style-src 'self'`,
|
||||
which tells the browser where scripts and styles may come from;
|
||||
- `Referrer-Policy: strict-origin-when-cross-origin`, which tells another
|
||||
site only which site a visitor came from, not which page;
|
||||
- `Permissions-Policy: camera=(), microphone=(), geolocation=()`, which
|
||||
@@ -364,13 +370,13 @@ writes it. The reverse proxy's access log does hold it, in the request path.
|
||||
|
||||
### Sign-in limits
|
||||
|
||||
The password is checked first, and only a failed sign-in is counted. A correct
|
||||
password is never answered `429`. Failures are counted per client and submitted
|
||||
username. From the fifth failure within a minute, each failure is answered
|
||||
`429`. At most two password checks run at once; a request that gets no turn
|
||||
within 5 seconds is answered `503`. So guessing speed is bounded by the limit of
|
||||
two checks at a time, not by the failure count. Choose a long random admin
|
||||
password, or rate-limit `POST /pages/login` at the reverse proxy.
|
||||
From the fifth failure within a minute, each failure is answered `429`. Only
|
||||
failures are counted. A correct password is never answered `429`. Failures are
|
||||
counted per client and submitted username. At most two password checks run at
|
||||
once; a request that gets no turn within 5 seconds is answered `503`. So
|
||||
guessing speed is bounded by the limit of two checks at a time, not by the
|
||||
failure count. Choose a long random `admin` password, or rate-limit
|
||||
`POST /pages/login` at the reverse proxy.
|
||||
|
||||
### Delivery destinations
|
||||
|
||||
@@ -426,10 +432,11 @@ sends them on in the background, and SQLite holds everything in between.
|
||||
|
||||
The receiver stores the event and one pending delivery per active target in one
|
||||
database write, then queues the deliveries. Background workers run them and
|
||||
record each attempt. At start, a recovery routine queues again every delivery
|
||||
left `pending` or `retrying`. It then runs every minute. Each run sends again a
|
||||
`retrying` delivery the engine is not working on, and a `pending` one that has
|
||||
sat untouched for 15 minutes. So a stop or a crash loses no delivery.
|
||||
record each attempt. At start, a recovery routine resumes every delivery left
|
||||
`pending` or `retrying`. It then runs every minute. Each run resumes every
|
||||
`retrying` delivery that the engine is not already handling. It also resumes a
|
||||
`pending` delivery that has sat untouched for 15 minutes. So a stop or a crash
|
||||
loses no delivery.
|
||||
|
||||
The main database holds configuration. Each webhook has its own event database,
|
||||
so writes for one webhook never wait on another webhook's database. The program
|
||||
@@ -450,18 +457,20 @@ number with a unit, such as `30s`, `15m` or `24h`.
|
||||
### `WEBHOOKER_ENVIRONMENT`
|
||||
|
||||
`dev` or `prod`. Default `prod`. This controls one thing: CORS, the headers that
|
||||
let a page on another site call this one from a browser. In `dev`, every origin
|
||||
is allowed. In `prod`, no CORS headers are sent.
|
||||
let a page on another site call this one from a browser. An origin is a site as
|
||||
a browser names it: scheme, host and port. In `dev`, every origin is allowed. In
|
||||
`prod`, no CORS headers are sent.
|
||||
|
||||
### `PORT`
|
||||
|
||||
The port the listener binds. Default `8080`. Docker marks the image's container
|
||||
healthy by a check every 30 seconds. The check requests port `8080` whatever
|
||||
`PORT` says, so a container started with another `PORT` reports unhealthy.
|
||||
The port webhooker listens on. Default `8080`. Docker marks the image's
|
||||
container healthy by a check every 30 seconds. The check requests port `8080`
|
||||
whatever `PORT` says, so a container started with another `PORT` reports
|
||||
unhealthy.
|
||||
|
||||
### `BIND_ADDRESS`
|
||||
|
||||
The IP address the listener binds. Default `127.0.0.1`. The Docker image sets
|
||||
The IP address webhooker listens on. Default `127.0.0.1`. The Docker image sets
|
||||
`0.0.0.0`. It must be an IP address literal: `localhost` and a value with a port
|
||||
are refused. `0.0.0.0` listens on IPv4 only; `::` listens on IPv6 and IPv4.
|
||||
|
||||
@@ -480,7 +489,7 @@ directory and the lock file.
|
||||
|
||||
`true` or `false`. Default `false`. It accepts the spellings Go's
|
||||
`strconv.ParseBool` accepts; `yes` and `on` are refused. `DEBUG=true` lowers the
|
||||
log level to `DEBUG`, which adds every SQL statement, with `?` in place of each
|
||||
log level to `DEBUG`, which adds SQL statements, with `?` in place of each
|
||||
value, and other diagnostic lines.
|
||||
|
||||
### `METRICS_USERNAME` and `METRICS_PASSWORD`
|
||||
@@ -530,10 +539,10 @@ private ranges. A set value replaces the default.
|
||||
The list decides whose `X-Forwarded-For` header is believed. `X-Forwarded-For`
|
||||
is the header a proxy uses to pass on the address of the client it is serving.
|
||||
It is read only when the connecting machine is inside the list. webhooker never
|
||||
reads `X-Real-IP` or `True-Client-IP`, two other headers with the same job. If
|
||||
any client can reach webhooker, or the proxy in front of it, from a private
|
||||
address, set the list to the proxy's address alone. Otherwise those clients can
|
||||
choose their own `X-Forwarded-For` and bypass every rate limit.
|
||||
reads `X-Real-IP` or `True-Client-IP`, two other headers with the same job. Some
|
||||
clients may reach webhooker, or the proxy in front of it, from a private
|
||||
address. If so, set the list to the proxy's address alone. Otherwise those
|
||||
clients can choose their own `X-Forwarded-For` and bypass every rate limit.
|
||||
|
||||
### Allowing egress to your own network
|
||||
|
||||
@@ -542,9 +551,9 @@ that delivery targets may reach although the SSRF guard refuses them by default.
|
||||
Default empty. Example: `10.0.0.0/8, 172.17.0.0/16`. When the list is set,
|
||||
startup logs it at `WARN`.
|
||||
|
||||
Anyone with the admin account can make webhooker send requests to every network
|
||||
you list, and read the responses in **Full Event Log**. List the smallest blocks
|
||||
that cover your destinations, and never `0.0.0.0/0` or `::/0`.
|
||||
Anyone with the `admin` account can make webhooker send requests to every
|
||||
network you list, and read the responses in **Full Event Log**. List the
|
||||
smallest networks that cover your destinations, and never `0.0.0.0/0` or `::/0`.
|
||||
|
||||
These addresses stay blocked whatever you list. They are link-local space, the
|
||||
unspecified addresses and cloud metadata endpoints. The unspecified addresses
|
||||
@@ -563,9 +572,9 @@ addresses are:
|
||||
- `0.0.0.0/32` and `::/128`;
|
||||
- `::a9fe:a9fe/128` and `64:ff9b::a9fe:a9fe/128`.
|
||||
|
||||
This list is not every cloud's metadata address. If yours is not here, do not
|
||||
list the block that contains it. Azure's public `168.63.129.16` is refused by
|
||||
default but is reopened by a list that covers it.
|
||||
Some clouds' metadata addresses are not on this list. If yours is not here, do
|
||||
not list the network that contains it. Azure's public metadata address,
|
||||
`168.63.129.16`, is refused by default but is reopened by a list that covers it.
|
||||
|
||||
## Delivery reference
|
||||
|
||||
@@ -574,7 +583,7 @@ event's `Content-Type` and headers. The target's configured **Headers** are
|
||||
added next and replace event headers of the same name. Every `http` and `slack`
|
||||
request carries `User-Agent: webhooker/{version}`. `User-Agent` is the header
|
||||
that names the sending program. The **Headers** field refuses `User-Agent`. A
|
||||
response with a status from 200 to 299 is a success. The first 4 KiB of the
|
||||
response with a status from `200` to `299` is a success. The first 4 KiB of the
|
||||
response body is stored with the attempt. **Timeout** is at most 300 seconds,
|
||||
and blank means the 30 second default. **Pass the query string on to this
|
||||
target** appends the event's query string to the URL.
|
||||
@@ -615,6 +624,8 @@ one JSON file compressed with gzip.
|
||||
- an unknown UUID gets `404`;
|
||||
- a deactivated entrypoint gets `410`;
|
||||
- a body over 1 MB gets `413`;
|
||||
- a body that cannot be read in full gets `400`, for example one still
|
||||
arriving after the server's 10 second read limit;
|
||||
- a failure to store the event gets `500`;
|
||||
- a stored event gets `200` with `{"status":"ok"}`.
|
||||
- `GET /pages/login` shows the sign-in form, `POST /pages/login` signs in, and
|
||||
@@ -678,15 +689,16 @@ checkout, or `VERSION` when that is set. It is shown at
|
||||
|
||||
The pages use one third-party script, Alpine.js. They load `@alpinejs/csp`, the
|
||||
version of it that works when the Content-Security-Policy forbids running text
|
||||
as code. The npm registry is where JavaScript packages are published. The
|
||||
package file in `3p/` is byte for byte the one the npm registry publishes.
|
||||
as code. The npm registry is where JavaScript packages are published.
|
||||
`static/js/alpine.min.js` is committed byte for byte as the npm registry
|
||||
publishes it.
|
||||
|
||||
A browser test in `internal/server` loads the pages in a headless browser, one
|
||||
without a window, under the real Content-Security-Policy. `make test` leaves it
|
||||
out; run `make test-browser` after changing `templates/` or `static/js/`.
|
||||
`static/css/tailwind.css` is generated by Tailwind, a CSS generator, and
|
||||
committed; `make check` fails when it is out of date, and `make css` regenerates
|
||||
it.
|
||||
A headless browser is a browser without a window. A browser test in
|
||||
`internal/server` loads the pages in one under the real Content-Security-Policy.
|
||||
`make test` leaves it out; run `make test-browser` after changing `templates/`
|
||||
or `static/js/`. `static/css/tailwind.css` is generated by Tailwind, a CSS
|
||||
generator. The file is committed. `make check` fails when it is out of date, and
|
||||
`make css` regenerates it.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
@@ -699,7 +711,6 @@ application are under [Entrypoint URLs](#entrypoint-urls).
|
||||
- `script/bootstrap` installs git, make, Go and the Go modules.
|
||||
- `script/setup` runs `script/bootstrap`, then `script/install-precommit`.
|
||||
- `script/projectname` prints `webhooker`.
|
||||
- `script/assets` extracts Alpine.js from `3p/` to `static/js/alpine.min.js`.
|
||||
- `script/test` runs the tests in Docker.
|
||||
- `script/test-browser` runs the browser test in Docker.
|
||||
- `script/lint` runs `gofmt`, `golangci-lint` and ESLint in Docker.
|
||||
|
||||
@@ -1,72 +1,77 @@
|
||||
# Workflow
|
||||
|
||||
One issue per unit of work, one branch and one PR per issue:
|
||||
One issue per unit of work, and one branch and one PR per issue:
|
||||
|
||||
- ensure a tracked issue exists with a definition of done
|
||||
- branch from `next` (never from `main`)
|
||||
- do the work; open a PR based on `next` (never on `main`)
|
||||
- pass an independent review, then the manager squash-merges into `next`
|
||||
- push; nothing stays local-only
|
||||
- make sure a tracked issue exists with a definition of done
|
||||
- branch from `next`, never from `main`
|
||||
- do the work and push the branch
|
||||
- open a PR with base `next`, never `main`
|
||||
- pass an independent review
|
||||
- the repository manager squash-merges it into `next`
|
||||
|
||||
`next` is the branch for the next milestone and must stay green and mergeable to
|
||||
`main` without notice. One `next` -> `main` PR accumulates the milestone;
|
||||
releases are cut from `main` separately.
|
||||
The repository manager is the role that merges each issue's PR into `next`.
|
||||
Nothing stays local-only: every commit is pushed.
|
||||
|
||||
The tracker (https://git.eeqj.de/sneak/webhooker/issues) is the record of what
|
||||
is done and what is under way. This file keeps no list of either.
|
||||
`next` is the branch for the next milestone. It must stay green and mergeable
|
||||
into `main` at all times. A milestone's work reaches `main` through one PR from
|
||||
`next`. The owner, @sneak, merges that PR. Cutting a release from `main` is a
|
||||
separate step after that.
|
||||
|
||||
The tracker (https://git.eeqj.de/sneak/webhooker/issues) records what is done
|
||||
and what is under way. This file keeps no list of either.
|
||||
|
||||
# Status
|
||||
|
||||
What is still open for 1.0.0 is in its milestone
|
||||
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged, and
|
||||
only @sneak tags it.
|
||||
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged. Only
|
||||
@sneak tags it.
|
||||
|
||||
Delivery is at-least-once by design, not by accident: a send whose result row
|
||||
does not land is attempted again, so a receiver can see a duplicate. That is
|
||||
deliberate — the alternative is a silent lost delivery — and the README says so
|
||||
under Rationale. It is not a defect to re-file.
|
||||
Delivery is at least once by design. When webhooker cannot record the outcome of
|
||||
a delivery, it sends that delivery again. The target can then receive the same
|
||||
event twice. The alternative is a lost delivery that nothing reports. The README
|
||||
says so. Such a duplicate is not a defect to re-file.
|
||||
|
||||
# Next Step
|
||||
|
||||
@sneak runs webhooker in limited production, then tags `v1.0.0`. Merging `next`
|
||||
into `main` is a separate act from tagging and waits on neither of those: `next`
|
||||
is kept mergeable at all times, which is the point of the branch.
|
||||
into `main` is a separate act. It waits on neither the production run nor the
|
||||
tag.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Per-webhook rate limits, set in the web UI and enforced in the receiver
|
||||
handler, on top of the receiver's existing limits, which `RECEIVER_RATE_LIMIT`
|
||||
sets for the whole server (see Rate Limiting in the README)
|
||||
- API key authentication for programmatic access (the `APIKey` model exists, but
|
||||
nothing creates a key and no middleware checks one)
|
||||
- REST API v1
|
||||
- CRUD for webhooks, entrypoints, targets
|
||||
- Per-webhook rate limits, set in the web UI and enforced at the entrypoint.
|
||||
Today `RECEIVER_RATE_LIMIT` is the only rate-limit setting. It is how many
|
||||
requests per minute one client address may send to one entrypoint. The same
|
||||
address is also capped at ten times that across all entrypoints together.
|
||||
- API key authentication for programmatic access. The `APIKey` model and its
|
||||
`api_keys` table exist. Nothing creates a key and nothing checks one.
|
||||
- REST API v1. `/api/v1` is mounted with no routes today.
|
||||
- CRUD for webhooks, entrypoints and targets
|
||||
- event viewing and filtering endpoints
|
||||
- event redelivery endpoint
|
||||
- OpenAPI specification
|
||||
- Analytics in the web UI: response times summarised per webhook or per target,
|
||||
and figures for all webhooks together. The webhook page already shows event,
|
||||
delivery and failure counts and failure percentages, the target list counts
|
||||
each target's deliveries, the event log and each event's page show every
|
||||
attempt's duration, and `/metrics` carries server-wide counts and delivery
|
||||
durations
|
||||
- A remember-me option at login. Every session already survives a browser
|
||||
restart: it lasts up to 7 days, and ends sooner after `SESSION_IDLE_TIMEOUT`
|
||||
(default 24 hours) without activity
|
||||
- Password reset flow for a forgotten password over the web. The authenticated
|
||||
password _change_ flow already landed, and a lost password is recoverable from
|
||||
the console with `webhooker resetpw`
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/208)
|
||||
and figures for all webhooks together. Today the webhook page shows event,
|
||||
delivery and failure counts and failure percentages. The target list counts
|
||||
each target's deliveries. The event log shows every attempt's duration.
|
||||
`/metrics` carries server-wide counts and delivery durations.
|
||||
- A remember-me option at login. Today every session survives a browser restart.
|
||||
It lasts up to 7 days. It ends sooner after `SESSION_IDLE_TIMEOUT` (default 24
|
||||
hours) without use.
|
||||
- Password reset over the web for a forgotten password. Today a signed-in user
|
||||
can change their password. A lost password is reset from the command line with
|
||||
`webhooker resetpw` (https://git.eeqj.de/sneak/webhooker/issues/208).
|
||||
- Later, nice to have
|
||||
- email delivery target type
|
||||
- SNS and S3 delivery targets
|
||||
- data transformations (e.g. a configurable Slack message format; the Slack
|
||||
target sends one fixed format today)
|
||||
- JSONL file delivery with periodic S3 upload (the `database` target already
|
||||
archives events to SQLite files, rotated hourly, daily or monthly if set,
|
||||
and the webhook page offers them as one gzipped JSON download)
|
||||
- data transformations, such as a configurable Slack message format. The
|
||||
Slack target sends one fixed format today.
|
||||
- JSONL file delivery with periodic S3 upload. Today an archive target
|
||||
writes events to SQLite files. An archive target can rotate those files
|
||||
hourly, daily or monthly. The webhook page offers them as one gzipped JSON
|
||||
download.
|
||||
- webhook event search, and filtering beyond the event log's failed and
|
||||
pending views
|
||||
- multi-user with role-based access control (each webhook already belongs to
|
||||
one account, but there is only the account made at first boot and no way
|
||||
to add another)
|
||||
- multi-user with role-based access control. Today each webhook belongs to
|
||||
one account. Only the account made at first boot exists. There is no way
|
||||
to add another.
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@
|
||||
// REPO_POLICIES.md states that a linter can check: const for everything,
|
||||
// let only for a variable that is reassigned, never var.
|
||||
export default [
|
||||
// Alpine.js, extracted from 3p/ by make assets; not ours to lint.
|
||||
// Alpine.js, committed as published; not ours to lint.
|
||||
{ ignores: ["static/js/alpine.min.js"] },
|
||||
{
|
||||
// The pages load static/js/app.js as a classic script, not a module.
|
||||
|
||||
@@ -96,6 +96,12 @@ func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
|
||||
return s.loadWebhookStats(webhookID, entrypoints, targets)
|
||||
}
|
||||
|
||||
// RelativeTimeForTest exposes relativeTime for use in the
|
||||
// handlers_test package.
|
||||
func RelativeTimeForTest(at, now time.Time) string {
|
||||
return relativeTime(at, now)
|
||||
}
|
||||
|
||||
// FinishedByTargetForTest exposes finishedByTarget for use in the
|
||||
// handlers_test package.
|
||||
func FinishedByTargetForTest(
|
||||
|
||||
@@ -15,6 +15,9 @@ const (
|
||||
longWindow = 24 * time.Hour
|
||||
)
|
||||
|
||||
// day is the largest unit relativeTime writes.
|
||||
const day = 24 * time.Hour
|
||||
|
||||
// percent turns a fraction into a percentage.
|
||||
const percent = 100
|
||||
|
||||
@@ -35,8 +38,12 @@ type WebhookStats struct {
|
||||
InProgress int64
|
||||
|
||||
// LastEventAt is when the newest event arrived, or nil when none
|
||||
// has. Retention does not change it.
|
||||
LastEventAt *time.Time
|
||||
// has. Retention does not change it. LastEvent is how long ago
|
||||
// that was, as relativeTime writes it, and LastEventUTC the full
|
||||
// time; both are empty when no event has arrived.
|
||||
LastEventAt *time.Time
|
||||
LastEvent string
|
||||
LastEventUTC string
|
||||
|
||||
Last10Minutes RecentWindow
|
||||
Last24Hours RecentWindow
|
||||
@@ -81,6 +88,42 @@ func (w RecentWindow) FailurePercent() string {
|
||||
)
|
||||
}
|
||||
|
||||
// relativeTime is how long before now the time at was, in whole
|
||||
// seconds, written as its two largest units out of d, h, m and s:
|
||||
// "59s", "1m1s", "23h59m", "3d4h". A second unit of zero is left off,
|
||||
// as in "5m". A time after now is "0s".
|
||||
func relativeTime(at, now time.Time) string {
|
||||
elapsed := max(now.Sub(at), 0)
|
||||
|
||||
days := int64(elapsed / day)
|
||||
hours := int64(elapsed % day / time.Hour)
|
||||
minutes := int64(elapsed % time.Hour / time.Minute)
|
||||
seconds := int64(elapsed % time.Minute / time.Second)
|
||||
|
||||
switch {
|
||||
case days > 0:
|
||||
return twoUnits(days, "d", hours, "h")
|
||||
case hours > 0:
|
||||
return twoUnits(hours, "h", minutes, "m")
|
||||
case minutes > 0:
|
||||
return twoUnits(minutes, "m", seconds, "s")
|
||||
default:
|
||||
return fmt.Sprintf("%ds", seconds)
|
||||
}
|
||||
}
|
||||
|
||||
// twoUnits writes a count of a larger unit followed by a count of the
|
||||
// next smaller one, leaving the smaller off when it is zero.
|
||||
func twoUnits(
|
||||
larger int64, largerUnit string, smaller int64, smallerUnit string,
|
||||
) string {
|
||||
if smaller == 0 {
|
||||
return fmt.Sprintf("%d%s", larger, largerUnit)
|
||||
}
|
||||
|
||||
return fmt.Sprintf("%d%s%d%s", larger, largerUnit, smaller, smallerUnit)
|
||||
}
|
||||
|
||||
// loadWebhookStats gathers the figures for the statistics pane from the
|
||||
// webhook's entrypoints and targets, as the page has already loaded
|
||||
// them, and from its event database. It returns nil, and logs why, when
|
||||
@@ -139,7 +182,7 @@ func (h *Handlers) loadWebhookStats(
|
||||
func readEventStats(
|
||||
db *gorm.DB, now time.Time, stats *WebhookStats,
|
||||
) error {
|
||||
err := readTotals(db, stats)
|
||||
err := readTotals(db, now, stats)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -169,9 +212,9 @@ func readEventStats(
|
||||
}
|
||||
|
||||
// readTotals fills in the lifetime and within-retention figures, and
|
||||
// when the last event arrived, from the running totals: the events'
|
||||
// row, and the targets' rows summed.
|
||||
func readTotals(db *gorm.DB, stats *WebhookStats) error {
|
||||
// when the last event arrived and how long before now that was, from
|
||||
// the running totals: the events' row, and the targets' rows summed.
|
||||
func readTotals(db *gorm.DB, now time.Time, stats *WebhookStats) error {
|
||||
var events database.EventTotals
|
||||
|
||||
err := db.Take(&events).Error
|
||||
@@ -190,6 +233,12 @@ func readTotals(db *gorm.DB, stats *WebhookStats) error {
|
||||
stats.WithinRetention.Events = events.Events - events.EventsRemoved
|
||||
stats.LastEventAt = events.LastEventAt
|
||||
|
||||
if events.LastEventAt != nil {
|
||||
stats.LastEvent = relativeTime(*events.LastEventAt, now)
|
||||
stats.LastEventUTC =
|
||||
events.LastEventAt.UTC().Format(time.DateTime) + " UTC"
|
||||
}
|
||||
|
||||
for _, t := range targets {
|
||||
stats.Lifetime.Deliveries += t.Deliveries
|
||||
stats.Lifetime.Failures += t.Failed
|
||||
|
||||
@@ -265,6 +265,13 @@ func statsPane(t *testing.T, page string) string {
|
||||
return strings.Join(strings.Fields(pane), " ")
|
||||
}
|
||||
|
||||
// statsLastEventShown matches the pane's last event for an event that
|
||||
// arrived at at: a figure with the full time in UTC as its tooltip.
|
||||
func statsLastEventShown(at time.Time) string {
|
||||
return `Last event</span>\s*<span [^>]*title="` +
|
||||
at.UTC().Format(time.DateTime) + ` UTC">[^<]+</span>`
|
||||
}
|
||||
|
||||
// assertStatsTargets checks, for the history seedStatsHistory builds,
|
||||
// each target's totals and its deliveries finished in the last 24
|
||||
// hours. The first target has ten deliveries and the replay, the
|
||||
@@ -300,14 +307,17 @@ func assertStatsPaneAfterPrune(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, hist.webhook.ID))
|
||||
lastEvent := hist.newest.CreatedAt.UTC().Format("2006-01-02 15:04:05 UTC")
|
||||
page := renderSourceDetailPage(t, h, sess, hist.webhook.ID)
|
||||
pane := statsPane(t, page)
|
||||
|
||||
assert.Contains(t, pane, "Entrypoints 12 (11 active) "+
|
||||
"Targets 6 (2 active) "+
|
||||
"Deliveries in progress 13 "+
|
||||
"Last event "+lastEvent+" "+
|
||||
"Retention 14 days")
|
||||
// The newest event arrived moments ago, so the last event is a
|
||||
// few seconds ago.
|
||||
assert.Regexp(t, `Entrypoints 12 \(11 active\) `+
|
||||
`Targets 6 \(2 active\) `+
|
||||
`Deliveries in progress 13 `+
|
||||
`Last event \d+s `+
|
||||
`Retention 14 days`, pane)
|
||||
assert.Regexp(t, statsLastEventShown(hist.newest.CreatedAt), page)
|
||||
assert.Contains(t, pane, "Lifetime Within retention "+
|
||||
"Events 10 9 "+
|
||||
"Deliveries 21 18 "+
|
||||
@@ -440,15 +450,14 @@ func TestWebhookStats_LastEventSurvivesPruningEveryEvent(t *testing.T) {
|
||||
require.NotNil(t, stats.LastEventAt)
|
||||
assert.True(t, arrived.Equal(*stats.LastEventAt))
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
assert.Contains(t, pane,
|
||||
"Last event "+arrived.UTC().Format("2006-01-02 15:04:05 UTC"))
|
||||
assert.Regexp(t, statsLastEventShown(arrived),
|
||||
renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
}
|
||||
|
||||
// TestWebhookStats_LastEventInUTC checks that the pane shows when the
|
||||
// last event arrived in UTC, as the event list does, when the time was
|
||||
// stored in another zone, as it is on a host whose local time is not
|
||||
// UTC.
|
||||
// TestWebhookStats_LastEventInUTC checks that the tooltip of the pane's
|
||||
// last event gives the time in UTC, as the event list does, when the
|
||||
// time was stored in another zone, as it is on a host whose local time
|
||||
// is not UTC.
|
||||
func TestWebhookStats_LastEventInUTC(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -474,8 +483,9 @@ func TestWebhookStats_LastEventInUTC(t *testing.T) {
|
||||
require.NoError(t, database.AddEventTotals(webhookDB,
|
||||
database.EventTotals{Events: 1, LastEventAt: &arrived}))
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
assert.Contains(t, pane, "Last event 2026-03-05 03:30:00 UTC")
|
||||
assert.Regexp(t,
|
||||
`Last event</span>\s*<span [^>]*title="2026-03-05 03:30:00 UTC">`,
|
||||
renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
}
|
||||
|
||||
// TestWebhookStats_PaneShowsRetentionPeriod checks that the statistics
|
||||
@@ -541,8 +551,11 @@ func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
|
||||
assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
|
||||
assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
page := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
pane := statsPane(t, page)
|
||||
assert.Contains(t, pane, "Last event none")
|
||||
assert.Regexp(t, `Last event</span>\s*<span class="[^"]*">none</span>`,
|
||||
page, "a last event of none has a tooltip")
|
||||
assert.Contains(t, pane, "Failure percentage — —")
|
||||
assert.False(t, dbMgr.DBExists(wh.ID))
|
||||
}
|
||||
@@ -567,3 +580,35 @@ func TestRecentWindow_FailurePercent(t *testing.T) {
|
||||
assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRelativeTime pins how the pane writes the time since the last
|
||||
// event, at each boundary between its units.
|
||||
func TestRelativeTime(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
now := time.Date(2026, time.October, 26, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
tests := []struct {
|
||||
ago time.Duration
|
||||
want string
|
||||
}{
|
||||
{0, "0s"},
|
||||
{59900 * time.Millisecond, "59s"},
|
||||
{60 * time.Second, "1m"},
|
||||
{61 * time.Second, "1m1s"},
|
||||
{3599 * time.Second, "59m59s"},
|
||||
{3600 * time.Second, "1h"},
|
||||
{3660 * time.Second, "1h1m"},
|
||||
{86399 * time.Second, "23h59m"},
|
||||
{86400 * time.Second, "1d"},
|
||||
{90000 * time.Second, "1d1h"},
|
||||
{400*24*time.Hour + 3*time.Hour, "400d3h"},
|
||||
// An event time ahead of the server clock.
|
||||
{-time.Hour, "0s"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
assert.Equal(t, tt.want,
|
||||
handlers.RelativeTimeForTest(now.Add(-tt.ago), now), tt.ago)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,7 +61,8 @@ const (
|
||||
// event log in a headless browser, served by the real router and so
|
||||
// under the real Content-Security-Policy, and checks that the pages'
|
||||
// Alpine.js directives, the copy control and the delete prompts work,
|
||||
// and that a target's row shows its delivery counts.
|
||||
// that a target's row shows its delivery counts, and that the
|
||||
// Statistics block shows how long ago the last event arrived.
|
||||
func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -97,6 +98,7 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
checkCopy(ctx, t, page)
|
||||
checkEntrypointEdit(ctx, t, page, page+"/events")
|
||||
checkRecentEvents(ctx, t, page)
|
||||
checkStatsLastEvent(ctx, t, env, webhook.ID, page)
|
||||
checkArchiveChoice(ctx, t, srv.URL+"/hooks/new", page)
|
||||
checkNewWebhookTargets(ctx, t, env, srv.URL+"/hooks/new")
|
||||
checkRefusedNewWebhook(ctx, t, srv.URL+"/hooks/new")
|
||||
@@ -831,6 +833,41 @@ func checkRecentEvents(ctx context.Context, t *testing.T, url string) {
|
||||
"the event's own page does not show its body")
|
||||
}
|
||||
|
||||
// checkStatsLastEvent records that the last event of the webhook
|
||||
// webhookID arrived 3 days, 4 hours and 30 minutes ago, loads its page,
|
||||
// and checks that the Statistics block shows the last event as "3d4h"
|
||||
// with the full time in UTC as its tooltip.
|
||||
func checkStatsLastEvent(
|
||||
ctx context.Context, t *testing.T, env *testEnv, webhookID, url string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
webhookDB, err := env.dbMgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
ago := 3*24*time.Hour + 4*time.Hour + 30*time.Minute
|
||||
arrived := time.Now().Add(-ago)
|
||||
require.NoError(t, database.AddEventTotals(webhookDB,
|
||||
database.EventTotals{LastEventAt: &arrived}))
|
||||
|
||||
figure := `//span[text()="Last event"]` +
|
||||
`/following-sibling::span[text()="3d4h"]`
|
||||
|
||||
var (
|
||||
title string
|
||||
found bool
|
||||
)
|
||||
|
||||
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
|
||||
require.True(t, shown(ctx, figure),
|
||||
`the Statistics block does not show the last event as "3d4h"`)
|
||||
require.NoError(t, chromedp.Run(ctx, chromedp.AttributeValue(
|
||||
figure, "title", &title, &found, chromedp.BySearch,
|
||||
)))
|
||||
assert.Equal(t, arrived.UTC().Format(time.DateTime)+" UTC", title,
|
||||
"the last event's tooltip is not the full time in UTC")
|
||||
}
|
||||
|
||||
// checkEventLog loads the event log and checks that of its events only
|
||||
// the newest, eventID, starts expanded: its row says so and its caret is
|
||||
// turned up, and the log's last event, lastEventID, starts collapsed. In
|
||||
|
||||
@@ -13,9 +13,6 @@ import (
|
||||
|
||||
// TestBaseTemplateScriptsAreServed walks every /s/ script the base
|
||||
// template loads on each page and fetches it through the real router.
|
||||
// Alpine.js is extracted from its tarball in 3p/ at build time, so the
|
||||
// file is not in the tree: this is the check that the page still gets
|
||||
// the JavaScript it asks for.
|
||||
func TestBaseTemplateScriptsAreServed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
#!/bin/sh
|
||||
# script/assets: extract Alpine.js from its npm package tarball, committed
|
||||
# in 3p/, to static/js/alpine.min.js, where go:embed reads it. The package
|
||||
# is @alpinejs/csp, Alpine's build for pages whose Content-Security-Policy
|
||||
# forbids eval. The extracted file is not committed. make build, make dev
|
||||
# and the Dockerfile's lint, test and build stages run this first.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
tar -xzOf 3p/alpinejs-csp-3.14.9.tgz package/dist/cdn.min.js \
|
||||
>static/js/alpine.min.js
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Vendored
+11
File diff suppressed because one or more lines are too long
+5
-3
@@ -6,9 +6,11 @@ import (
|
||||
)
|
||||
|
||||
// Static holds the CSS and JavaScript files the web UI's pages load. They
|
||||
// are named one by one so that a missing js/alpine.min.js, which make
|
||||
// assets extracts and git does not track, fails the build instead of
|
||||
// leaving the pages without Alpine.js.
|
||||
// are named one by one, so a missing file fails the build instead of
|
||||
// leaving the pages without it.
|
||||
//
|
||||
// js/alpine.min.js is Alpine.js, committed unchanged. It is
|
||||
// package/dist/cdn.min.js from the npm package @alpinejs/csp 3.14.9.
|
||||
//
|
||||
//go:embed css/tailwind.css css/style.css js/app.js js/alpine.min.js
|
||||
var Static embed.FS
|
||||
|
||||
@@ -22,7 +22,11 @@
|
||||
</div>
|
||||
<div>
|
||||
<span class="text-gray-500">Last event</span>
|
||||
<span class="font-medium text-gray-900">{{with .LastEventAt}}{{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
|
||||
{{if .LastEvent}}
|
||||
<span class="font-medium text-gray-900" title="{{.LastEventUTC}}">{{.LastEvent}}</span>
|
||||
{{else}}
|
||||
<span class="font-medium text-gray-900">none</span>
|
||||
{{end}}
|
||||
</div>
|
||||
<div>
|
||||
<span class="text-gray-500">Retention</span>
|
||||
|
||||
Reference in New Issue
Block a user