Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ad35798dee |
@@ -6,7 +6,5 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
# actions/checkout v4.2.2, 2026-02-22
|
# actions/checkout v4.2.2, 2026-02-22
|
||||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
||||||
# script/cibuild bootstraps, runs every check and builds the
|
# script/cibuild builds the image, whose stages run every check.
|
||||||
# image. script/bootstrap links what it installs into
|
- run: script/cibuild
|
||||||
# ~/.local/bin, so that has to be on PATH for the rest.
|
|
||||||
- run: PATH="$HOME/.local/bin:$PATH" script/cibuild
|
|
||||||
|
|||||||
+1
-3
@@ -72,14 +72,12 @@ RUN addgroup -g 1000 -S netwatch && \
|
|||||||
# conf.d; bin/entrypoint.sh says how.
|
# conf.d; bin/entrypoint.sh says how.
|
||||||
RUN rm /etc/nginx/conf.d/default.conf
|
RUN rm /etc/nginx/conf.d/default.conf
|
||||||
COPY nginx.conf /etc/nginx/templates/netwatch.conf.template
|
COPY nginx.conf /etc/nginx/templates/netwatch.conf.template
|
||||||
COPY security-headers.conf /etc/nginx/security-headers.conf
|
|
||||||
COPY --from=frontend /app/dist /usr/share/nginx/html
|
COPY --from=frontend /app/dist /usr/share/nginx/html
|
||||||
COPY --from=builder /src/netwatch-server /usr/local/bin/netwatch-server
|
COPY --from=builder /src/netwatch-server /usr/local/bin/netwatch-server
|
||||||
COPY bin/entrypoint.sh /usr/local/bin/entrypoint.sh
|
COPY bin/entrypoint.sh /usr/local/bin/entrypoint.sh
|
||||||
|
|
||||||
# bin/entrypoint.sh creates DATA_DIR at start and gives it and /data to
|
|
||||||
# the netwatch user, whatever is mounted there.
|
|
||||||
ENV DATA_DIR=/data/reports
|
ENV DATA_DIR=/data/reports
|
||||||
|
RUN mkdir -p /data/reports && chown -R netwatch:netwatch /data
|
||||||
VOLUME /data
|
VOLUME /data
|
||||||
|
|
||||||
# The default public port; PORT changes it.
|
# The default public port; PORT changes it.
|
||||||
|
|||||||
@@ -36,12 +36,12 @@ The Go backend in `backend/` has its own `script/` directory and shim Makefile
|
|||||||
(see [backend/README.md](backend/README.md)). The root scripts cover both
|
(see [backend/README.md](backend/README.md)). The root scripts cover both
|
||||||
halves, so the root `make check` fails if either one is broken. We provide:
|
halves, so the root `make check` fails if either one is broken. We provide:
|
||||||
|
|
||||||
- `script/bootstrap` — install all dependencies (the pinned node via nvm unless
|
- `script/bootstrap` — install all dependencies (pinned node via nvm if needed,
|
||||||
one new enough for the frontend's dependencies is installed, yarn via
|
yarn via corepack, `yarn install --frozen-lockfile`, the pinned Go unless one
|
||||||
corepack, `yarn install --frozen-lockfile`, the pinned Go unless one at least
|
at least as new as `backend/go.mod` asks for is installed, and the Go
|
||||||
as new as `backend/go.mod` asks for is installed, and the Go modules), linking
|
modules), linking what it installs itself into `~/.local/bin`, which has to be
|
||||||
what it installs itself into `~/.local/bin`, which has to be on `PATH`. It
|
on `PATH`. It installs no Go linter and not Docker: `make lint` runs the
|
||||||
installs no Go linter and not Docker: `make lint` runs the linter in Docker
|
linter in Docker
|
||||||
- `script/setup` — make a fresh clone ready for development: bootstrap plus the
|
- `script/setup` — make a fresh clone ready for development: bootstrap plus the
|
||||||
git pre-commit hook
|
git pre-commit hook
|
||||||
- `script/projectname` — print the project name (used for the Docker image tag)
|
- `script/projectname` — print the project name (used for the Docker image tag)
|
||||||
@@ -65,8 +65,7 @@ halves, so the root `make check` fails if either one is broken. We provide:
|
|||||||
`script/check`: it needs Docker and takes minutes.
|
`script/check`: it needs Docker and takes minutes.
|
||||||
- `script/docker` — build the image from `Dockerfile` without the build cache,
|
- `script/docker` — build the image from `Dockerfile` without the build cache,
|
||||||
tagged `netwatch` via `script/projectname`
|
tagged `netwatch` via `script/projectname`
|
||||||
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
|
- `script/cibuild` — CI entrypoint: builds the image
|
||||||
then builds the image as `script/docker` does, without the build cache
|
|
||||||
- `script/precommit` — run by the git pre-commit hook; runs `script/check`
|
- `script/precommit` — run by the git pre-commit hook; runs `script/check`
|
||||||
- `script/install-precommit` — install the git pre-commit hook
|
- `script/install-precommit` — install the git pre-commit hook
|
||||||
|
|
||||||
@@ -185,16 +184,13 @@ container: nginx serves the built frontend and passes `/api/` and
|
|||||||
only inside the container, on `127.0.0.1:8081`. The image:
|
only inside the container, on `127.0.0.1:8081`. The image:
|
||||||
|
|
||||||
- Listens on port 8080 by default (override with `PORT` env var)
|
- Listens on port 8080 by default (override with `PORT` env var)
|
||||||
- Takes the client address from `X-Forwarded-For` only on requests from the
|
- Trusts `X-Forwarded-For` from RFC1918 reverse proxies (10/8, 172.16/12,
|
||||||
reverse proxies named in `TRUSTED_PROXIES`, and by default from none
|
192.168/16)
|
||||||
- Sends access logs to stdout
|
- Sends access logs to stdout
|
||||||
- Caches static assets with immutable headers
|
- Caches static assets with immutable headers
|
||||||
- Sends the security headers `REPO_POLICIES.md` requires on every response, as
|
|
||||||
`security-headers.conf` sets them, in place of the backend's own
|
|
||||||
- Stores reports in `DATA_DIR`, `/data/reports` by default, on the `/data`
|
- Stores reports in `DATA_DIR`, `/data/reports` by default, on the `/data`
|
||||||
volume. Before the backend starts, the image creates `DATA_DIR` and gives it
|
volume. The backend runs as user `netwatch` (uid 1000), so a directory
|
||||||
and `/data` to user `netwatch` (uid 1000), which the backend runs as, so a
|
bind-mounted at `/data` must be writable by uid 1000
|
||||||
host directory bind-mounted at `/data` ends up owned by uid 1000
|
|
||||||
- Writes buffered reports to disk on `docker stop`, and exits non-zero if nginx
|
- Writes buffered reports to disk on `docker stop`, and exits non-zero if nginx
|
||||||
or the backend exits on its own, so the platform restarts it
|
or the backend exits on its own, so the platform restarts it
|
||||||
|
|
||||||
@@ -204,11 +200,20 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
|
|||||||
|
|
||||||
- **Port:** container port `8080`.
|
- **Port:** container port `8080`.
|
||||||
- **Volume:** container path `/data`; the reports are kept in `/data/reports`.
|
- **Volume:** container path `/data`; the reports are kept in `/data/reports`.
|
||||||
|
- **First run:** upaas bind-mounts the host directory it is given and does not
|
||||||
|
create it, and the backend, which runs as uid 1000, does not start unless it
|
||||||
|
can write there. Create the directory, owned by uid 1000, before the first
|
||||||
|
deploy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p /path/to/data
|
||||||
|
chown 1000:1000 /path/to/data
|
||||||
|
```
|
||||||
|
|
||||||
- **Environment variables:** none is required. An empty one counts as unset, and
|
- **Environment variables:** none is required. An empty one counts as unset, and
|
||||||
one set to a value netwatch cannot use stops the container at start, with the
|
one set to a value netwatch cannot use stops the container at start, with the
|
||||||
reason in its log.
|
reason in its log.
|
||||||
- `PORT`, default `8080`: the container port, from 1 to 65535. `8081` cannot
|
- `PORT`, default `8080`: the container port
|
||||||
be used: the backend listens on it inside the container
|
|
||||||
- `REPORTS_PER_MINUTE`, default `60`: reports each client address may send a
|
- `REPORTS_PER_MINUTE`, default `60`: reports each client address may send a
|
||||||
minute
|
minute
|
||||||
- `DATA_DIR_MAX_BYTES`, default `1073741824` (1 GiB): the most room the
|
- `DATA_DIR_MAX_BYTES`, default `1073741824` (1 GiB): the most room the
|
||||||
@@ -216,20 +221,12 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
|
|||||||
- `CORS_ALLOWED_ORIGINS`, default empty: other origins whose pages may call
|
- `CORS_ALLOWED_ORIGINS`, default empty: other origins whose pages may call
|
||||||
the API
|
the API
|
||||||
- `DEBUG`, default `false`: debug logging
|
- `DEBUG`, default `false`: debug logging
|
||||||
- `DATA_DIR`, default `/data/reports`: the directory the reports are kept
|
- `DATA_DIR`, default `/data/reports`: leave unset; reports kept outside
|
||||||
in: `/data` or a path below it, with no `.` or `..` part and no extra `/`.
|
`/data` do not survive a redeploy
|
||||||
The container also stops if a part of the path that exists, `/data`
|
- `TRUSTED_PROXIES`, default loopback and RFC1918: leave unset. The
|
||||||
included, is a symbolic link
|
backend's only client is nginx, on loopback, which passes on the client
|
||||||
- `TRUSTED_PROXIES`, default empty: set it to the address the reverse proxy
|
address; nginx takes it from `X-Forwarded-For` only from RFC1918
|
||||||
in front of the container connects from, as an IP address or CIDR; several
|
addresses.
|
||||||
are separated by commas. nginx takes the client address from
|
|
||||||
`X-Forwarded-For` only on a request from one of them, and the rate limit
|
|
||||||
counts that address. Unset, `X-Forwarded-For` is ignored and every client
|
|
||||||
behind the proxy shares the proxy's one allowance of `REPORTS_PER_MINUTE`.
|
|
||||||
Name only addresses nothing but the proxy connects from: any client that
|
|
||||||
connects from one can write its own `X-Forwarded-For`, and through a port
|
|
||||||
Docker publishes, every client may connect from the Docker network's
|
|
||||||
gateway, such as `172.17.0.1`.
|
|
||||||
- **Health check:** the image's `HEALTHCHECK` requests
|
- **Health check:** the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
|
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
|
||||||
both nginx and the backend answer. upaas reads the container's health 60
|
both nginx and the backend answer. upaas reads the container's health 60
|
||||||
|
|||||||
@@ -23,77 +23,15 @@ latest run passes.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-09-29: `bin/entrypoint.sh` checks `DATA_DIR` in full before it acts on it
|
|
||||||
as root (issue #80): `DATA_DIR` must be `/data` or a path below it with no
|
|
||||||
`.`, `..` or empty part, and no part of it that exists, `/data` included, may
|
|
||||||
be a symbolic link; anything else stops the start with a message naming
|
|
||||||
`DATA_DIR`. Only then is `DATA_DIR` created and `/data` given to `netwatch`,
|
|
||||||
so a refused start no longer creates directories outside `/data`, and
|
|
||||||
`DATA_DIR=/etc` no longer gives `/etc` to `netwatch`. The `README.md` section
|
|
||||||
"Running under upaas" says which values are accepted
|
|
||||||
- 2026-09-29: the container sets up its own data directory (issue #75):
|
|
||||||
`bin/entrypoint.sh`, still as root, creates `DATA_DIR` if missing and gives it
|
|
||||||
and `/data` to the `netwatch` user with mode 750 before starting the backend
|
|
||||||
as that user, so an empty host directory owned by root, or one holding files
|
|
||||||
from another uid, works with no step on the host. It stops the start instead
|
|
||||||
when a symbolic link is on the path to `DATA_DIR`, since root would change
|
|
||||||
whatever the link points to. The `README.md` first-run step that created and
|
|
||||||
chowned the host directory is gone, and the image no longer sets that
|
|
||||||
ownership at build time
|
|
||||||
- 2026-09-29: CI can no longer pass on checks that did not run (issue #37):
|
|
||||||
`script/cibuild` is now the org model, byte for byte. It runs
|
|
||||||
`script/bootstrap` and `script/check`, then builds the image with `--no-cache`
|
|
||||||
and the version from `git describe` as the `VERSION` build argument, where it
|
|
||||||
used to be a plain `docker build .` whose check steps could come from the
|
|
||||||
build cache. The workflow puts `~/.local/bin`, where bootstrap links what it
|
|
||||||
installs, on the step's `PATH`, and bootstrap now installs its pinned node
|
|
||||||
when the installed one is older than the frontend's dependencies need
|
|
||||||
- 2026-09-29: `backend/.golangci.yml` re-vendored from `sneak/prompts` (issue
|
|
||||||
#41): `gomodguard`, deprecated in golangci-lint v2.12.0, is disabled and its
|
|
||||||
successor `gomodguard_v2` enabled with the org block list, so lint runs print
|
|
||||||
no deprecation warning. The new file also turns `depguard` on with its
|
|
||||||
`test-support` rule, which keeps `net/http/httptest` out of non-test code;
|
|
||||||
netwatch adds no entries of its own to that rule. `backend/script/lint` checks
|
|
||||||
the new sha256
|
|
||||||
- 2026-09-29: nginx sends the security headers `REPO_POLICIES.md` requires on
|
|
||||||
every response (issue #18), including errors, `/assets/` and what it passes on
|
|
||||||
from the backend, whose own copies it drops so each header goes out once. They
|
|
||||||
live in `security-headers.conf`, which `nginx.conf` includes. The content
|
|
||||||
security policy allows no inline script or style, so the status dot's grey in
|
|
||||||
`src/main.js` is now a class; `connect-src` is `*` because probed hosts
|
|
||||||
redirect to others, and the browser checks each redirect against it
|
|
||||||
- 2026-09-29: the request log is bounded (issue #60): the method, URL, protocol,
|
|
||||||
`User-Agent`, `Referer`, request ID (which chi takes from the client's
|
|
||||||
`X-Request-Id` header) and client address it writes are each cut to 128 bytes,
|
|
||||||
the bound the report handler already used, so one request can no longer put
|
|
||||||
about 1 MiB per field into a log line. That bound and its helper now live in
|
|
||||||
the `logger` package, shared by both
|
|
||||||
- 2026-09-29: nginx takes the client address from `X-Forwarded-For` only on
|
|
||||||
requests from the reverse proxies named in the container's `TRUSTED_PROXIES`
|
|
||||||
(issue #64), and by default from none, where it trusted every RFC1918 address
|
|
||||||
before, so a client could write a new address on each request and escape the
|
|
||||||
rate limit. `bin/entrypoint.sh` writes one `set_real_ip_from` line per entry
|
|
||||||
into `/etc/nginx/trusted-proxies.conf`, which `nginx.conf` includes, refusing
|
|
||||||
an entry that is not an IP address or CIDR, as `netwatch-server check-cidr`
|
|
||||||
finds; it starts the backend with `TRUSTED_PROXIES=127.0.0.1/32`, since nginx
|
|
||||||
is its only client
|
|
||||||
- 2026-09-29: report file names can no longer collide (issue #61): each is
|
|
||||||
`reports-<timestamp>-<number>.jsonl.zst`, where the number goes up by one for
|
|
||||||
each file the server starts to write, so two flushes in the same millisecond,
|
|
||||||
such as a flush for size and the final flush at shutdown, each get a file of
|
|
||||||
their own instead of the second one failing. A failed write uses up its
|
|
||||||
number, leaving a gap if the file could not be created and otherwise a file
|
|
||||||
under that number that may be incomplete.
|
|
||||||
- 2026-09-29: ready to run under upaas (issue #59): the image has a
|
- 2026-09-29: ready to run under upaas (issue #59): the image has a
|
||||||
`HEALTHCHECK` that requests `/.well-known/healthcheck` through nginx on the
|
`HEALTHCHECK` that requests `/.well-known/healthcheck` through nginx on the
|
||||||
port from `PORT`. The backend no longer reads a bad `PORT` as 0 or a bad
|
port from `PORT`. The backend no longer reads a bad `PORT` as 0 or a bad
|
||||||
`DEBUG` as false: those, and a `BIND_ADDRESS` that is not an IP address, stop
|
`DEBUG` as false: those, and a `BIND_ADDRESS` that is not an IP address, stop
|
||||||
it from starting with an error naming the variable, as the limits,
|
it from starting with an error naming the variable, as the limits,
|
||||||
`CORS_ALLOWED_ORIGINS` and, now by name, `TRUSTED_PROXIES` already did.
|
`CORS_ALLOWED_ORIGINS` and, now by name, `TRUSTED_PROXIES` already did.
|
||||||
`bin/entrypoint.sh` also refuses a `PORT` outside 1 to 65535, and `8081`,
|
`README.md` has a "Running under upaas" section, whose first-run steps create
|
||||||
where the backend listens inside the container, naming `PORT`. `README.md` has
|
the host directory for `/data` owned by uid 1000; the image does not change
|
||||||
a "Running under upaas" section, whose first-run steps create the host
|
its owner
|
||||||
directory for `/data` owned by uid 1000; the image does not change its owner
|
|
||||||
- 2026-09-29: nginx listens on `PORT` (issue #26), 8080 when unset or empty: the
|
- 2026-09-29: nginx listens on `PORT` (issue #26), 8080 when unset or empty: the
|
||||||
nginx image renders `nginx.conf` as a template at container start, filling in
|
nginx image renders `nginx.conf` as a template at container start, filling in
|
||||||
`PORT` and no other variable. `bin/entrypoint.sh` refuses to start when `PORT`
|
`PORT` and no other variable. `bin/entrypoint.sh` refuses to start when `PORT`
|
||||||
@@ -223,3 +161,9 @@ latest run passes.
|
|||||||
(main always green policy)
|
(main always green policy)
|
||||||
- Decide what to do with untracked resume.sh: commit it, gitignore it, or delete
|
- Decide what to do with untracked resume.sh: commit it, gitignore it, or delete
|
||||||
it
|
it
|
||||||
|
- Upstream fix needed in `sneak/prompts`: the org-standard `.golangci.yml`
|
||||||
|
enables `gomodguard`, which golangci-lint v2.12.2 reports as deprecated since
|
||||||
|
v2.12.0 and replaced by `gomodguard_v2`, so every backend lint run prints a
|
||||||
|
deprecation warning. The file is standardized and must never be edited in this
|
||||||
|
repo, so nothing can be done here beyond tracking it — tracked at
|
||||||
|
<https://git.eeqj.de/sneak/netwatch/issues/41>
|
||||||
|
|||||||
+2
-66
@@ -10,20 +10,14 @@ run:
|
|||||||
|
|
||||||
linters:
|
linters:
|
||||||
default: all
|
default: all
|
||||||
enable:
|
|
||||||
# Successor to the deprecated gomodguard. Named explicitly, rather than
|
|
||||||
# left to `default: all`, because it carries the module policy below.
|
|
||||||
- gomodguard_v2
|
|
||||||
disable:
|
disable:
|
||||||
# Genuinely incompatible with project patterns
|
# Genuinely incompatible with project patterns
|
||||||
- exhaustruct # Requires all struct fields
|
- exhaustruct # Requires all struct fields
|
||||||
|
- depguard # Dependency allow/block lists
|
||||||
- godot # Requires comments to end with periods
|
- godot # Requires comments to end with periods
|
||||||
|
- wsl # Deprecated, replaced by wsl_v5
|
||||||
- wrapcheck # Too verbose for internal packages
|
- wrapcheck # Too verbose for internal packages
|
||||||
- varnamelen # Short names like db, id are idiomatic Go
|
- varnamelen # Short names like db, id are idiomatic Go
|
||||||
# Deprecated: the warning is attached to the old name, so it is
|
|
||||||
# silenced by disabling that name, not by enabling the successor.
|
|
||||||
- wsl # Deprecated, replaced by wsl_v5
|
|
||||||
- gomodguard # Deprecated, replaced by gomodguard_v2
|
|
||||||
settings:
|
settings:
|
||||||
lll:
|
lll:
|
||||||
line-length: 88
|
line-length: 88
|
||||||
@@ -34,64 +28,6 @@ linters:
|
|||||||
max-complexity: 15
|
max-complexity: 15
|
||||||
dupl:
|
dupl:
|
||||||
threshold: 100
|
threshold: 100
|
||||||
depguard:
|
|
||||||
# Test-support code must not be compiled into the shipped binary. A
|
|
||||||
# test-support package exists to hand a test privileges the program
|
|
||||||
# itself must never have, so a file that is not a test must not import
|
|
||||||
# one. Test files, and the files inside a package whose directory name
|
|
||||||
# ends in `test`, are where that code belongs, and are exempt.
|
|
||||||
#
|
|
||||||
# The deny list below is the one part of this file a repository is
|
|
||||||
# expected to extend, and the only part it may. depguard matches an
|
|
||||||
# import path against a list of prefixes, so it cannot be told "any path
|
|
||||||
# whose last segment ends in test"; a repository's own test-support
|
|
||||||
# packages have to be named here one at a time, by full import path,
|
|
||||||
# under a module path that differs from repository to repository. Add
|
|
||||||
# them; change nothing else.
|
|
||||||
rules:
|
|
||||||
test-support:
|
|
||||||
list-mode: lax
|
|
||||||
files:
|
|
||||||
- "$all"
|
|
||||||
- "!$test"
|
|
||||||
- "!**/*test/**"
|
|
||||||
deny:
|
|
||||||
- pkg: net/http/httptest
|
|
||||||
desc: >-
|
|
||||||
Test-support code belongs in test files and in packages whose
|
|
||||||
directory name ends in test, not in the shipped binary.
|
|
||||||
# Only decisions already recorded in the Go package defaults are
|
|
||||||
# listed here. Every entry matches the module path exactly.
|
|
||||||
gomodguard_v2:
|
|
||||||
blocked:
|
|
||||||
- module: github.com/rs/zerolog
|
|
||||||
recommendations:
|
|
||||||
- log/slog
|
|
||||||
reason: "Structured logging is stdlib log/slog."
|
|
||||||
# One entry per pre-fork module path, because the later releases
|
|
||||||
# are separate paths. A prefix match would be shorter but would
|
|
||||||
# also reach github.com/go-redis/redismock, the test double for
|
|
||||||
# the successor these entries recommend.
|
|
||||||
- module: github.com/go-redis/redis
|
|
||||||
recommendations:
|
|
||||||
- github.com/redis/go-redis/v9
|
|
||||||
reason: "Pre-fork module; use the maintained go-redis v9."
|
|
||||||
- module: github.com/go-redis/redis/v7
|
|
||||||
recommendations:
|
|
||||||
- github.com/redis/go-redis/v9
|
|
||||||
reason: "Pre-fork module; use the maintained go-redis v9."
|
|
||||||
- module: github.com/go-redis/redis/v8
|
|
||||||
recommendations:
|
|
||||||
- github.com/redis/go-redis/v9
|
|
||||||
reason: "Pre-fork module; use the maintained go-redis v9."
|
|
||||||
- module: github.com/sergi/go-diff
|
|
||||||
recommendations:
|
|
||||||
- github.com/aymanbagabas/go-udiff
|
|
||||||
reason: "No unified diff output; use go-udiff."
|
|
||||||
- module: github.com/hexops/gotextdiff
|
|
||||||
recommendations:
|
|
||||||
- github.com/aymanbagabas/go-udiff
|
|
||||||
reason: "Unmaintained fork; use go-udiff."
|
|
||||||
|
|
||||||
issues:
|
issues:
|
||||||
max-issues-per-linter: 0
|
max-issues-per-linter: 0
|
||||||
|
|||||||
+8
-28
@@ -87,10 +87,9 @@ Internal packages in `internal/` follow standard Go project layout:
|
|||||||
| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) |
|
| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) |
|
||||||
|
|
||||||
`TRUSTED_PROXIES` defaults to `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`.
|
`TRUSTED_PROXIES` defaults to `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`.
|
||||||
The loopback entries cover a reverse proxy on the same host. A request whose
|
The loopback entries cover the reverse proxy that shares the container; the
|
||||||
direct peer is outside this set has its forwarded headers ignored, and the
|
RFC1918 ranges match `nginx.conf`. A request whose direct peer is outside this
|
||||||
direct peer is logged and rate-limited instead. The container image does not use
|
set has its forwarded headers ignored, and the direct peer is logged instead.
|
||||||
this default; see [Container image](#container-image).
|
|
||||||
|
|
||||||
A variable set to a value the server cannot use, such as `PORT=abc`,
|
A variable set to a value the server cannot use, such as `PORT=abc`,
|
||||||
`DEBUG=maybe` or a `BIND_ADDRESS` that is not an IP address, stops it from
|
`DEBUG=maybe` or a `BIND_ADDRESS` that is not an IP address, stops it from
|
||||||
@@ -102,33 +101,14 @@ The root `Dockerfile` builds one image in which nginx listens on the public port
|
|||||||
8080, serves the frontend, and proxies `/api/` and `/.well-known/healthcheck` to
|
8080, serves the frontend, and proxies `/api/` and `/.well-known/healthcheck` to
|
||||||
this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as
|
this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as
|
||||||
user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so
|
user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so
|
||||||
only nginx reaches it, and with `TRUSTED_PROXIES=127.0.0.1/32`, so it takes the
|
only nginx reaches it. `DATA_DIR` is `/data/reports`, on the `/data` volume,
|
||||||
client address nginx passes on and no other. `DATA_DIR` is `/data/reports`, on
|
which `netwatch` owns.
|
||||||
the `/data` volume; the entrypoint creates it and gives it and `/data` to
|
|
||||||
`netwatch` before starting the server. nginx replaces the security headers
|
|
||||||
this server sets with those in the root `security-headers.conf`, so those are
|
|
||||||
what clients of the image see.
|
|
||||||
|
|
||||||
The container's own `TRUSTED_PROXIES` goes to nginx instead: IP addresses or
|
|
||||||
CIDRs, separated by commas, of the reverse proxies in front of the container.
|
|
||||||
nginx takes the client address from `X-Forwarded-For` only on a request from one
|
|
||||||
of them. Unset or empty, nginx trusts no proxy, and the client address is the
|
|
||||||
one each request comes from, so every client behind a proxy shares one rate
|
|
||||||
limit. An entry that is not an IP address or CIDR, such as a hostname or
|
|
||||||
`1.2.3`, stops the container at start with an error naming `TRUSTED_PROXIES`:
|
|
||||||
the entrypoint checks each entry with `netwatch-server check-cidr`, which parses
|
|
||||||
it as this server parses its own `TRUSTED_PROXIES`.
|
|
||||||
|
|
||||||
### Report storage
|
### Report storage
|
||||||
|
|
||||||
Reports are written as `reports-<timestamp>-<number>.jsonl.zst` files in
|
Reports are written as `reports-<timestamp>.jsonl.zst` files in `DATA_DIR`.
|
||||||
`DATA_DIR`. The timestamp is in UTC to the millisecond, so the names sort by
|
Each file contains one JSON object per line, compressed with zstd. Files are
|
||||||
time. The number starts at 1 when the server starts and goes up by one for each
|
created with `O_EXCL` to prevent overwrites.
|
||||||
file the server starts to write, so two files written in the same millisecond
|
|
||||||
still get different names. A failed write uses up its number, leaving a gap in
|
|
||||||
the numbers if the file could not be created and otherwise a file under that
|
|
||||||
number that may be incomplete. Each file contains one JSON object per line,
|
|
||||||
compressed with zstd. Files are created with `O_EXCL` to prevent overwrites.
|
|
||||||
|
|
||||||
### Report limits
|
### Report limits
|
||||||
|
|
||||||
|
|||||||
@@ -2,9 +2,6 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/config"
|
"sneak.berlin/go/netwatch/internal/config"
|
||||||
"sneak.berlin/go/netwatch/internal/globals"
|
"sneak.berlin/go/netwatch/internal/globals"
|
||||||
"sneak.berlin/go/netwatch/internal/handlers"
|
"sneak.berlin/go/netwatch/internal/handlers"
|
||||||
@@ -25,19 +22,6 @@ var (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
|
|
||||||
// this server would refuse CIDR in its TRUSTED_PROXIES.
|
|
||||||
// bin/entrypoint.sh runs it on each entry it gives nginx.
|
|
||||||
if len(os.Args) == 3 && os.Args[1] == "check-cidr" {
|
|
||||||
_, err := middleware.ParseTrustedProxies(os.Args[2:])
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintln(os.Stderr, err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
globals.Appname = Appname
|
globals.Appname = Appname
|
||||||
globals.Version = Version
|
globals.Version = Version
|
||||||
globals.Buildarch = Buildarch
|
globals.Buildarch = Buildarch
|
||||||
|
|||||||
@@ -21,11 +21,10 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// defaultTrustedProxies lists the networks whose forwarded
|
// defaultTrustedProxies lists the networks whose forwarded
|
||||||
// headers are honoured by default: IPv4 and IPv6 loopback,
|
// headers are honoured by default. It covers the RFC1918
|
||||||
// for a reverse proxy on the same host, and the RFC1918
|
// ranges (to match nginx.conf) plus IPv4 and IPv6 loopback,
|
||||||
// ranges. The container image does not use it:
|
// because the reverse proxy shares the container and reaches
|
||||||
// bin/entrypoint.sh gives the server 127.0.0.1/32, since
|
// the backend over loopback.
|
||||||
// nginx is its only client there.
|
|
||||||
const defaultTrustedProxies = "127.0.0.1/32,::1/128," +
|
const defaultTrustedProxies = "127.0.0.1/32,::1/128," +
|
||||||
"10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
|
"10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,9 @@ package handlers
|
|||||||
|
|
||||||
import "log/slog"
|
import "log/slog"
|
||||||
|
|
||||||
|
// MaxLoggedFieldBytes exposes the log bound to the external tests.
|
||||||
|
const MaxLoggedFieldBytes = maxLoggedFieldBytes
|
||||||
|
|
||||||
// NewForTest builds a Handlers around a report sink and logger,
|
// NewForTest builds a Handlers around a report sink and logger,
|
||||||
// bypassing the fx graph so handler behaviour (including the
|
// bypassing the fx graph so handler behaviour (including the
|
||||||
// storage failure path) is exercisable in unit tests.
|
// storage failure path) is exercisable in unit tests.
|
||||||
|
|||||||
@@ -5,10 +5,14 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/logger"
|
|
||||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
"sneak.berlin/go/netwatch/internal/reportbuf"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// maxLoggedFieldBytes bounds untrusted text (string fields,
|
||||||
|
// decode error text) before it is logged, so a caller cannot
|
||||||
|
// inflate log volume with an oversized value.
|
||||||
|
const maxLoggedFieldBytes = 128
|
||||||
|
|
||||||
type reportSample struct {
|
type reportSample struct {
|
||||||
T int64 `json:"t"`
|
T int64 `json:"t"`
|
||||||
Latency *int `json:"latency"`
|
Latency *int `json:"latency"`
|
||||||
@@ -79,7 +83,7 @@ func (s *Handlers) decodeErrorStatus(err error) int {
|
|||||||
// The decoder's error text can quote request bytes (a whole
|
// The decoder's error text can quote request bytes (a whole
|
||||||
// oversized number, for example), so it is bounded too.
|
// oversized number, for example), so it is bounded too.
|
||||||
s.log.Error("failed to decode report",
|
s.log.Error("failed to decode report",
|
||||||
"error", logger.BoundedForLog(err.Error()),
|
"error", boundedForLog(err.Error()),
|
||||||
)
|
)
|
||||||
|
|
||||||
return http.StatusBadRequest
|
return http.StatusBadRequest
|
||||||
@@ -111,10 +115,20 @@ func (s *Handlers) logReportReceived(rpt report) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
s.log.Info("report received",
|
s.log.Info("report received",
|
||||||
"client_id", logger.BoundedForLog(rpt.ClientID),
|
"client_id", boundedForLog(rpt.ClientID),
|
||||||
"timestamp", logger.BoundedForLog(rpt.Timestamp),
|
"timestamp", boundedForLog(rpt.Timestamp),
|
||||||
"host_count", len(rpt.Hosts),
|
"host_count", len(rpt.Hosts),
|
||||||
"total_samples", totalSamples,
|
"total_samples", totalSamples,
|
||||||
"geo_bytes", len(rpt.Geo),
|
"geo_bytes", len(rpt.Geo),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// boundedForLog truncates an untrusted string to a fixed byte
|
||||||
|
// bound so an attacker-controlled field cannot dominate the log.
|
||||||
|
func boundedForLog(s string) string {
|
||||||
|
if len(s) > maxLoggedFieldBytes {
|
||||||
|
return s[:maxLoggedFieldBytes]
|
||||||
|
}
|
||||||
|
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/handlers"
|
"sneak.berlin/go/netwatch/internal/handlers"
|
||||||
"sneak.berlin/go/netwatch/internal/logger"
|
|
||||||
"sneak.berlin/go/netwatch/internal/middleware"
|
"sneak.berlin/go/netwatch/internal/middleware"
|
||||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
"sneak.berlin/go/netwatch/internal/reportbuf"
|
||||||
)
|
)
|
||||||
@@ -175,7 +174,7 @@ func TestHandleReportDoesNotLogRawGeo(t *testing.T) {
|
|||||||
func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
|
func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
long := strings.Repeat("c", 2*logger.MaxLoggedFieldBytes)
|
long := strings.Repeat("c", 2*handlers.MaxLoggedFieldBytes)
|
||||||
|
|
||||||
var logbuf bytes.Buffer
|
var logbuf bytes.Buffer
|
||||||
|
|
||||||
@@ -198,16 +197,16 @@ func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
|
|||||||
t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String())
|
t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
want := long[:logger.MaxLoggedFieldBytes]
|
want := long[:handlers.MaxLoggedFieldBytes]
|
||||||
|
|
||||||
if logged["client_id"] != want {
|
if logged["client_id"] != want {
|
||||||
t.Fatalf("logged client_id not cut to %d bytes: %q",
|
t.Fatalf("logged client_id not cut to %d bytes: %q",
|
||||||
logger.MaxLoggedFieldBytes, logged["client_id"])
|
handlers.MaxLoggedFieldBytes, logged["client_id"])
|
||||||
}
|
}
|
||||||
|
|
||||||
if logged["timestamp"] != want {
|
if logged["timestamp"] != want {
|
||||||
t.Fatalf("logged timestamp not cut to %d bytes: %q",
|
t.Fatalf("logged timestamp not cut to %d bytes: %q",
|
||||||
logger.MaxLoggedFieldBytes, logged["timestamp"])
|
handlers.MaxLoggedFieldBytes, logged["timestamp"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -216,7 +215,7 @@ func TestHandleReportDecodeErrorLogIsBounded(t *testing.T) {
|
|||||||
|
|
||||||
// A number too large for its int64 field makes the decoder's
|
// A number too large for its int64 field makes the decoder's
|
||||||
// error text quote the whole number.
|
// error text quote the whole number.
|
||||||
huge := strings.Repeat("9", 2*logger.MaxLoggedFieldBytes)
|
huge := strings.Repeat("9", 2*handlers.MaxLoggedFieldBytes)
|
||||||
|
|
||||||
var logbuf bytes.Buffer
|
var logbuf bytes.Buffer
|
||||||
|
|
||||||
|
|||||||
@@ -11,21 +11,6 @@ import (
|
|||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
)
|
)
|
||||||
|
|
||||||
// MaxLoggedFieldBytes bounds untrusted text (request fields,
|
|
||||||
// header values, decode error text) before it is logged, so a
|
|
||||||
// caller cannot inflate log volume with an oversized value.
|
|
||||||
const MaxLoggedFieldBytes = 128
|
|
||||||
|
|
||||||
// BoundedForLog truncates an untrusted string to a fixed byte
|
|
||||||
// bound so an attacker-controlled field cannot dominate the log.
|
|
||||||
func BoundedForLog(s string) string {
|
|
||||||
if len(s) > MaxLoggedFieldBytes {
|
|
||||||
return s[:MaxLoggedFieldBytes]
|
|
||||||
}
|
|
||||||
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
// Params defines the dependencies for Logger.
|
// Params defines the dependencies for Logger.
|
||||||
type Params struct {
|
type Params struct {
|
||||||
fx.In
|
fx.In
|
||||||
|
|||||||
@@ -29,3 +29,7 @@ func ClientIP(
|
|||||||
) string {
|
) string {
|
||||||
return clientIP(remoteAddr, header, trusted)
|
return clientIP(remoteAddr, header, trusted)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func ParseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
|
||||||
|
return parseTrustedProxies(cidrs)
|
||||||
|
}
|
||||||
|
|||||||
@@ -64,7 +64,7 @@ func New(
|
|||||||
_ fx.Lifecycle,
|
_ fx.Lifecycle,
|
||||||
params Params,
|
params Params,
|
||||||
) (*Middleware, error) {
|
) (*Middleware, error) {
|
||||||
trusted, err := ParseTrustedProxies(params.Config.TrustedProxies)
|
trusted, err := parseTrustedProxies(params.Config.TrustedProxies)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -77,11 +77,9 @@ func New(
|
|||||||
return s, nil
|
return s, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseTrustedProxies converts the TRUSTED_PROXIES entries into
|
// parseTrustedProxies converts the TRUSTED_PROXIES entries into
|
||||||
// prefixes, failing fast on any malformed entry. Each entry must be
|
// prefixes, failing fast on any malformed entry.
|
||||||
// a CIDR; a lone address is refused. "netwatch-server check-cidr"
|
func parseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
|
||||||
// runs it too.
|
|
||||||
func ParseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
|
|
||||||
prefixes := make([]netip.Prefix, 0, len(cidrs))
|
prefixes := make([]netip.Prefix, 0, len(cidrs))
|
||||||
|
|
||||||
for _, cidr := range cidrs {
|
for _, cidr := range cidrs {
|
||||||
@@ -189,10 +187,7 @@ func addrInAny(s string, trusted []netip.Prefix) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Logging returns middleware that logs each request with
|
// Logging returns middleware that logs each request with
|
||||||
// timing, status code, and client information. Every string
|
// timing, status code, and client information.
|
||||||
// taken from the request is cut to logger.MaxLoggedFieldBytes,
|
|
||||||
// including the request ID, which chi takes from the client's
|
|
||||||
// X-Request-Id header when one is sent.
|
|
||||||
func (s *Middleware) Logging() func(http.Handler) http.Handler {
|
func (s *Middleware) Logging() func(http.Handler) http.Handler {
|
||||||
return func(next http.Handler) http.Handler {
|
return func(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(
|
return http.HandlerFunc(
|
||||||
@@ -205,19 +200,21 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
|
|||||||
latency := time.Since(start)
|
latency := time.Since(start)
|
||||||
s.log.InfoContext(ctx, "request",
|
s.log.InfoContext(ctx, "request",
|
||||||
"request_start", start,
|
"request_start", start,
|
||||||
"method", logger.BoundedForLog(r.Method),
|
"method", r.Method,
|
||||||
"url", logger.BoundedForLog(r.URL.String()),
|
"url", r.URL.String(),
|
||||||
"useragent", logger.BoundedForLog(r.UserAgent()),
|
"useragent", r.UserAgent(),
|
||||||
"request_id",
|
"request_id",
|
||||||
logger.BoundedForLog(middleware.GetReqID(ctx)),
|
ctx.Value(
|
||||||
"referer", logger.BoundedForLog(r.Referer()),
|
middleware.RequestIDKey,
|
||||||
"proto", logger.BoundedForLog(r.Proto),
|
),
|
||||||
|
"referer", r.Referer(),
|
||||||
|
"proto", r.Proto,
|
||||||
"remote_ip",
|
"remote_ip",
|
||||||
logger.BoundedForLog(clientIP(
|
clientIP(
|
||||||
r.RemoteAddr,
|
r.RemoteAddr,
|
||||||
r.Header,
|
r.Header,
|
||||||
s.trustedProxies,
|
s.trustedProxies,
|
||||||
)),
|
),
|
||||||
"status", lrw.statusCode,
|
"status", lrw.statusCode,
|
||||||
"latency_ms",
|
"latency_ms",
|
||||||
latency.Milliseconds(),
|
latency.Milliseconds(),
|
||||||
|
|||||||
@@ -13,10 +13,7 @@ import (
|
|||||||
"testing/synctest"
|
"testing/synctest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/logger"
|
|
||||||
"sneak.berlin/go/netwatch/internal/middleware"
|
"sneak.berlin/go/netwatch/internal/middleware"
|
||||||
|
|
||||||
chimiddleware "github.com/go-chi/chi/v5/middleware"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -39,30 +36,13 @@ func mustPrefixes(t *testing.T, cidrs ...string) []netip.Prefix {
|
|||||||
return prefixes
|
return prefixes
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestParseTrustedProxiesRejectsMalformed includes entries nginx would
|
|
||||||
// read as another address or look up as a hostname, in the CIDR form
|
|
||||||
// bin/entrypoint.sh gives "netwatch-server check-cidr".
|
|
||||||
func TestParseTrustedProxiesRejectsMalformed(t *testing.T) {
|
func TestParseTrustedProxiesRejectsMalformed(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
for _, cidr := range []string{
|
_, err := middleware.ParseTrustedProxies([]string{"not-a-cidr"})
|
||||||
"not-a-cidr", "10.0.0.1", "1.2.3/32", "172.30/32", "10/32",
|
|
||||||
"cafe/32", "999.1.1.1/32", "10.0.0.0/33", "::1/129",
|
|
||||||
"fe80::1%eth0/128",
|
|
||||||
} {
|
|
||||||
_, err := middleware.ParseTrustedProxies([]string{cidr})
|
|
||||||
if err == nil || !strings.Contains(err.Error(), "TRUSTED_PROXIES") {
|
if err == nil || !strings.Contains(err.Error(), "TRUSTED_PROXIES") {
|
||||||
t.Errorf("%q: error = %v, want one naming TRUSTED_PROXIES",
|
t.Fatalf("error = %v, want one naming TRUSTED_PROXIES", err)
|
||||||
cidr, err)
|
|
||||||
}
|
}
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestParseTrustedProxiesAcceptsCIDRs(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
mustPrefixes(t, "172.17.0.1/32", "10.0.0.0/8", "2001:db8::1/128",
|
|
||||||
"2001:db8::/32", "::ffff:192.0.2.1/128")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type clientIPCase struct {
|
type clientIPCase struct {
|
||||||
@@ -323,52 +303,6 @@ func TestRecovererRepanicsOnAbortHandler(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestLoggingCutsRequestStringsToBound sends an over-long URL and
|
|
||||||
// over-long header values, and checks the request log writes each
|
|
||||||
// one cut to logger.MaxLoggedFieldBytes.
|
|
||||||
func TestLoggingCutsRequestStringsToBound(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
long := strings.Repeat("a", 2*logger.MaxLoggedFieldBytes)
|
|
||||||
|
|
||||||
var logbuf bytes.Buffer
|
|
||||||
|
|
||||||
mw := middleware.NewWithLogger(
|
|
||||||
slog.New(slog.NewJSONHandler(&logbuf, nil)),
|
|
||||||
)
|
|
||||||
|
|
||||||
handler := chimiddleware.RequestID(mw.Logging()(okHandler()))
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(),
|
|
||||||
http.MethodGet, "/"+long, http.NoBody)
|
|
||||||
req.Header.Set("User-Agent", long)
|
|
||||||
req.Header.Set("Referer", long)
|
|
||||||
req.Header.Set("X-Request-Id", long)
|
|
||||||
|
|
||||||
handler.ServeHTTP(httptest.NewRecorder(), req)
|
|
||||||
|
|
||||||
var logged map[string]any
|
|
||||||
|
|
||||||
err := json.Unmarshal(logbuf.Bytes(), &logged)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
want := map[string]string{
|
|
||||||
"url": ("/" + long)[:logger.MaxLoggedFieldBytes],
|
|
||||||
"useragent": long[:logger.MaxLoggedFieldBytes],
|
|
||||||
"referer": long[:logger.MaxLoggedFieldBytes],
|
|
||||||
"request_id": long[:logger.MaxLoggedFieldBytes],
|
|
||||||
}
|
|
||||||
|
|
||||||
for field, value := range want {
|
|
||||||
if logged[field] != value {
|
|
||||||
t.Errorf("logged %s = %q, want it cut to %d bytes",
|
|
||||||
field, logged[field], logger.MaxLoggedFieldBytes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// okHandler stands in for the route a middleware guards.
|
// okHandler stands in for the route a middleware guards.
|
||||||
func okHandler() http.Handler {
|
func okHandler() http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
|||||||
@@ -1,15 +1,7 @@
|
|||||||
package reportbuf
|
package reportbuf
|
||||||
|
|
||||||
import "time"
|
|
||||||
|
|
||||||
// Flush writes the buffered reports to a file now, as the periodic
|
// Flush writes the buffered reports to a file now, as the periodic
|
||||||
// flush does, so tests need not wait a minute for it.
|
// flush does, so tests need not wait a minute for it.
|
||||||
func (b *Buffer) Flush() error {
|
func (b *Buffer) Flush() error {
|
||||||
return b.flushLocked()
|
return b.flushLocked()
|
||||||
}
|
}
|
||||||
|
|
||||||
// StopClock makes every report file the buffer writes from now on
|
|
||||||
// carry the timestamp at, as if all were written in one millisecond.
|
|
||||||
func (b *Buffer) StopClock(at time.Time) {
|
|
||||||
b.now = func() time.Time { return at }
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/config"
|
"sneak.berlin/go/netwatch/internal/config"
|
||||||
@@ -31,8 +30,7 @@ const (
|
|||||||
dirPerms fs.FileMode = 0o750
|
dirPerms fs.FileMode = 0o750
|
||||||
filePerms fs.FileMode = 0o640
|
filePerms fs.FileMode = 0o640
|
||||||
|
|
||||||
// Report files are named filePrefix + timestamp + "-" + number +
|
// Report files are named filePrefix + timestamp + fileSuffix.
|
||||||
// fileSuffix; see writeFile.
|
|
||||||
filePrefix = "reports-"
|
filePrefix = "reports-"
|
||||||
fileSuffix = ".jsonl.zst"
|
fileSuffix = ".jsonl.zst"
|
||||||
)
|
)
|
||||||
@@ -58,12 +56,6 @@ type Buffer struct {
|
|||||||
log *slog.Logger
|
log *slog.Logger
|
||||||
maxBytes int64
|
maxBytes int64
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
// now is the clock report files are named by: time.Now, except
|
|
||||||
// in tests that need two flushes to share a timestamp.
|
|
||||||
now func() time.Time
|
|
||||||
// seq numbers the report files, so that two named in the same
|
|
||||||
// millisecond still get different names.
|
|
||||||
seq atomic.Uint64
|
|
||||||
stopOnce sync.Once
|
stopOnce sync.Once
|
||||||
// usedBytes is what Append checks against maxBytes: the size
|
// usedBytes is what Append checks against maxBytes: the size
|
||||||
// of the report files in dataDir, plus the reports not yet
|
// of the report files in dataDir, plus the reports not yet
|
||||||
@@ -87,7 +79,6 @@ func New(
|
|||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
log: params.Logger.Get(),
|
log: params.Logger.Get(),
|
||||||
maxBytes: params.Config.DataDirMaxBytes,
|
maxBytes: params.Config.DataDirMaxBytes,
|
||||||
now: time.Now,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
lc.Append(fx.Hook{
|
lc.Append(fx.Hook{
|
||||||
@@ -220,14 +211,11 @@ func (b *Buffer) drainBuf() []byte {
|
|||||||
// writeFile creates a timestamped zstd-compressed JSONL file
|
// writeFile creates a timestamped zstd-compressed JSONL file
|
||||||
// in the data directory.
|
// in the data directory.
|
||||||
func (b *Buffer) writeFile(data []byte) error {
|
func (b *Buffer) writeFile(data []byte) error {
|
||||||
// The timestamp comes first, so the names sort by time; the number
|
ts := time.Now().UTC().Format("2006-01-02T15-04-05.000Z")
|
||||||
// after it tells apart files named in the same millisecond.
|
path := filepath.Join(b.dataDir, filePrefix+ts+fileSuffix)
|
||||||
ts := b.now().UTC().Format("2006-01-02T15-04-05.000Z")
|
|
||||||
name := fmt.Sprintf("%s%s-%d%s", filePrefix, ts, b.seq.Add(1), fileSuffix)
|
|
||||||
path := filepath.Join(b.dataDir, name)
|
|
||||||
|
|
||||||
// path is built from the operator-supplied dataDir plus a
|
// path is built from the operator-supplied dataDir plus a
|
||||||
// generated timestamp and number, so it carries no external input.
|
// generated timestamp, so it carries no external input.
|
||||||
f, err := os.OpenFile( //nolint:gosec // see comment above
|
f, err := os.OpenFile( //nolint:gosec // see comment above
|
||||||
path,
|
path,
|
||||||
os.O_WRONLY|os.O_CREATE|os.O_EXCL,
|
os.O_WRONLY|os.O_CREATE|os.O_EXCL,
|
||||||
|
|||||||
@@ -3,11 +3,9 @@ package reportbuf_test
|
|||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"slices"
|
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -20,7 +18,6 @@ import (
|
|||||||
"sneak.berlin/go/netwatch/internal/logger"
|
"sneak.berlin/go/netwatch/internal/logger"
|
||||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
"sneak.berlin/go/netwatch/internal/reportbuf"
|
||||||
|
|
||||||
"github.com/klauspost/compress/zstd"
|
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
"go.uber.org/fx/fxtest"
|
"go.uber.org/fx/fxtest"
|
||||||
)
|
)
|
||||||
@@ -213,6 +210,10 @@ func TestWrittenReportsCountAtFileSize(t *testing.T) {
|
|||||||
t.Fatalf("flush: %v", err)
|
t.Fatalf("flush: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The second report is written at shutdown, and must not land in
|
||||||
|
// the first file's millisecond (see TestWrittenReportsKeepCounting).
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
|
||||||
err = buf.Append(report)
|
err = buf.Append(report)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("second report, after the first was written: %v", err)
|
t.Fatalf("second report, after the first was written: %v", err)
|
||||||
@@ -253,6 +254,10 @@ func TestWrittenReportsKeepCounting(t *testing.T) {
|
|||||||
t.Fatalf("with %d bytes of report files: %v", used, err)
|
t.Fatalf("with %d bytes of report files: %v", used, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Report files are named to the millisecond; two in the same
|
||||||
|
// one collide (https://git.eeqj.de/sneak/netwatch/issues/61).
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
|
||||||
err = buf.Flush()
|
err = buf.Flush()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("flush: %v", err)
|
t.Fatalf("flush: %v", err)
|
||||||
@@ -310,43 +315,6 @@ func TestConcurrentAppendsStopAtCap(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestTwoFlushesInOneMillisecond flushes twice within one millisecond,
|
|
||||||
// as a flush for size and the final flush at shutdown can: each flush
|
|
||||||
// must write a file of its own, and the files must hold every report.
|
|
||||||
func TestTwoFlushesInOneMillisecond(t *testing.T) {
|
|
||||||
const flushes = 2
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
t.Setenv("DATA_DIR", dir)
|
|
||||||
|
|
||||||
buf := startBuffer(t)
|
|
||||||
buf.StopClock(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC))
|
|
||||||
|
|
||||||
for id := 1; id <= flushes; id++ {
|
|
||||||
err := buf.Append(map[string]int{"id": id})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("append report %d: %v", id, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = buf.Flush()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("flush %d: %v", id, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
files := readReportFiles(t, dir)
|
|
||||||
if len(files) != flushes {
|
|
||||||
t.Fatalf("%d report files after %d flushes", len(files), flushes)
|
|
||||||
}
|
|
||||||
|
|
||||||
for id := 1; id <= flushes; id++ {
|
|
||||||
want := fmt.Sprintf(`{"id":%d}`+"\n", id)
|
|
||||||
if !slices.Contains(files, want) {
|
|
||||||
t.Fatalf("no report file holds report %d alone", id)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// reportFilesBytes returns the total size of the report files in dir.
|
// reportFilesBytes returns the total size of the report files in dir.
|
||||||
func reportFilesBytes(t *testing.T, dir string) int64 {
|
func reportFilesBytes(t *testing.T, dir string) int64 {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -370,43 +338,6 @@ func reportFilesBytes(t *testing.T, dir string) int64 {
|
|||||||
return total
|
return total
|
||||||
}
|
}
|
||||||
|
|
||||||
// readReportFiles returns the decompressed contents of each report
|
|
||||||
// file in dir.
|
|
||||||
func readReportFiles(t *testing.T, dir string) []string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
files := os.DirFS(dir)
|
|
||||||
|
|
||||||
names, err := fs.Glob(files, "reports-*.jsonl.zst")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("list report files: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
dec, err := zstd.NewReader(nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("create zstd decoder: %v", err)
|
|
||||||
}
|
|
||||||
defer dec.Close()
|
|
||||||
|
|
||||||
contents := make([]string, 0, len(names))
|
|
||||||
|
|
||||||
for _, name := range names {
|
|
||||||
compressed, readErr := fs.ReadFile(files, name)
|
|
||||||
if readErr != nil {
|
|
||||||
t.Fatalf("read %s: %v", name, readErr)
|
|
||||||
}
|
|
||||||
|
|
||||||
data, decErr := dec.DecodeAll(compressed, nil)
|
|
||||||
if decErr != nil {
|
|
||||||
t.Fatalf("decompress %s: %v", name, decErr)
|
|
||||||
}
|
|
||||||
|
|
||||||
contents = append(contents, string(data))
|
|
||||||
}
|
|
||||||
|
|
||||||
return contents
|
|
||||||
}
|
|
||||||
|
|
||||||
func writeBytes(t *testing.T, path string, n int) {
|
func writeBytes(t *testing.T, path string, n int) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -14,7 +14,7 @@ set -eu
|
|||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
GOLANGCI_CONFIG_SHA256="a79b63a254602a5318db5d0e9a06bc71b84bf0c1d896305229d8bfed1d1b1776"
|
GOLANGCI_CONFIG_SHA256="021cc83f4e6fc7c31b95b34b846723dfcf20b66b7baeea1dc40406e643346bcb"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
|
|||||||
+6
-90
@@ -10,9 +10,8 @@ set -u
|
|||||||
|
|
||||||
# PORT is the public port nginx listens on, 8080 when unset or empty.
|
# PORT is the public port nginx listens on, 8080 when unset or empty.
|
||||||
# nginx would take a value such as localhost or unix:/tmp/x.sock as an
|
# nginx would take a value such as localhost or unix:/tmp/x.sock as an
|
||||||
# address and start anyway, and reports a bad port without naming
|
# address and start anyway, so anything but digits stops the container
|
||||||
# PORT, so a value that is not a usable port stops the container here,
|
# here, before either process starts.
|
||||||
# before either process starts.
|
|
||||||
export PORT="${PORT:-8080}"
|
export PORT="${PORT:-8080}"
|
||||||
case "$PORT" in
|
case "$PORT" in
|
||||||
*[!0-9]*)
|
*[!0-9]*)
|
||||||
@@ -20,87 +19,6 @@ case "$PORT" in
|
|||||||
exit 1
|
exit 1
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
# The length is checked first because, for a number too big for it,
|
|
||||||
# the shell's test prints an error and is false, so the range checks
|
|
||||||
# alone would let it through.
|
|
||||||
if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
|
|
||||||
echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
if [ "$PORT" -eq 8081 ]; then
|
|
||||||
echo "entrypoint: PORT cannot be 8081, netwatch-server listens there" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# TRUSTED_PROXIES names the reverse proxies in front of the container,
|
|
||||||
# as IP addresses or CIDRs separated by commas. nginx takes the client
|
|
||||||
# address from X-Forwarded-For only on a request from one of them, so
|
|
||||||
# unset or empty, it trusts no one. nginx.conf includes the file written
|
|
||||||
# here, one set_real_ip_from line per entry.
|
|
||||||
#
|
|
||||||
# nginx looks up an entry it cannot read as an address as a hostname,
|
|
||||||
# and trusts what it finds (1.2.3 is found as 1.2.0.3). So each entry
|
|
||||||
# is made a CIDR, a lone address getting /128 if it is IPv6 and /32 if
|
|
||||||
# not, and netwatch-server checks it with the parsing it gives its own
|
|
||||||
# TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
|
|
||||||
# below, naming the entry as written. set -f keeps a * in an entry from
|
|
||||||
# becoming a list of file names.
|
|
||||||
TRUSTED_PROXIES="${TRUSTED_PROXIES:-}"
|
|
||||||
set -f
|
|
||||||
for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
|
|
||||||
case "$proxy" in
|
|
||||||
*/*) cidr="$proxy" ;;
|
|
||||||
*:*) cidr="$proxy/128" ;;
|
|
||||||
*) cidr="$proxy/32" ;;
|
|
||||||
esac
|
|
||||||
if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
|
|
||||||
echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
|
|
||||||
"separated by commas; '$proxy' is neither" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "set_real_ip_from $cidr;"
|
|
||||||
done > /etc/nginx/trusted-proxies.conf
|
|
||||||
|
|
||||||
# netwatch-server keeps its report files in DATA_DIR, on the /data
|
|
||||||
# volume, which may be a host directory owned by root or by another
|
|
||||||
# uid. /data and everything in it are given to the netwatch user here,
|
|
||||||
# and /data and DATA_DIR get the mode the server gives a directory it
|
|
||||||
# creates, so the host directory needs no preparing.
|
|
||||||
#
|
|
||||||
# This runs as root, so nothing is created or changed until DATA_DIR is
|
|
||||||
# known to be /data or a path below it, with no '.', '..' or empty
|
|
||||||
# part, and no part of it that exists, /data included, is a symbolic
|
|
||||||
# link: the netwatch user can put one in /data, and root would follow
|
|
||||||
# it anywhere in the container. Nothing else runs in the container yet,
|
|
||||||
# so no link can appear after the check.
|
|
||||||
export DATA_DIR="${DATA_DIR:-/data/reports}"
|
|
||||||
data_dir_ok() {
|
|
||||||
# With a / added at the end, a last part of '.' or '..', and a / at
|
|
||||||
# the end, match these patterns too.
|
|
||||||
case "$DATA_DIR/" in
|
|
||||||
*/./* | */../* | *//*) return 1 ;;
|
|
||||||
/data/*) ;;
|
|
||||||
*) return 1 ;;
|
|
||||||
esac
|
|
||||||
# Each part from DATA_DIR up to /data. [ -L ] is false for a part
|
|
||||||
# that does not exist.
|
|
||||||
dir="$DATA_DIR"
|
|
||||||
while [ "$dir" != /data ]; do
|
|
||||||
[ -L "$dir" ] && return 1
|
|
||||||
dir="${dir%/*}"
|
|
||||||
done
|
|
||||||
[ ! -L /data ]
|
|
||||||
}
|
|
||||||
if ! data_dir_ok; then
|
|
||||||
echo "entrypoint: DATA_DIR must be /data or a path below it, with no" \
|
|
||||||
"'.', '..', extra '/' or symbolic link on it, not '$DATA_DIR'" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
mkdir -p "$DATA_DIR" || exit 1
|
|
||||||
# -h: a symbolic link in /data is itself given to netwatch, not what it
|
|
||||||
# points to.
|
|
||||||
chown -R -h netwatch:netwatch /data || exit 1
|
|
||||||
chmod 750 /data "$DATA_DIR" || exit 1
|
|
||||||
|
|
||||||
# A stop signal is only noted here; the loop below acts on it.
|
# A stop signal is only noted here; the loop below acts on it.
|
||||||
stop_requested=""
|
stop_requested=""
|
||||||
@@ -108,12 +26,10 @@ trap 'stop_requested=yes' TERM INT
|
|||||||
|
|
||||||
# netwatch-server runs as the netwatch user and listens on loopback
|
# netwatch-server runs as the netwatch user and listens on loopback
|
||||||
# only, on a port other than the public one; nginx.conf proxies to this
|
# only, on a port other than the public one; nginx.conf proxies to this
|
||||||
# address. Its only client is nginx, so it takes the client address
|
# address. The netwatch user has no login shell, hence -s /bin/sh.
|
||||||
# nginx passes on from 127.0.0.1 alone, whatever TRUSTED_PROXIES the
|
# busybox su replaces itself with the command instead of staying on as
|
||||||
# container has. The netwatch user has no login shell, hence -s
|
# its parent, so $! is the server's own PID.
|
||||||
# /bin/sh. busybox su replaces itself with the command instead of
|
BIND_ADDRESS=127.0.0.1 PORT=8081 \
|
||||||
# staying on as its parent, so $! is the server's own PID.
|
|
||||||
BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
|
|
||||||
su -s /bin/sh netwatch -c 'exec netwatch-server' &
|
su -s /bin/sh netwatch -c 'exec netwatch-server' &
|
||||||
backend=$!
|
backend=$!
|
||||||
|
|
||||||
|
|||||||
+4
-22
@@ -8,20 +8,13 @@ server {
|
|||||||
# Keep the nginx version out of the Server header and error pages.
|
# Keep the nginx version out of the Server header and error pages.
|
||||||
server_tokens off;
|
server_tokens off;
|
||||||
|
|
||||||
# The security headers, on every response. An add_header in a
|
|
||||||
# location drops every add_header from here, so a location with one
|
|
||||||
# of its own includes this file again.
|
|
||||||
include /etc/nginx/security-headers.conf;
|
|
||||||
|
|
||||||
root /usr/share/nginx/html;
|
root /usr/share/nginx/html;
|
||||||
index index.html;
|
index index.html;
|
||||||
|
|
||||||
# The client address comes from X-Forwarded-For only on a request
|
# Trust RFC1918 reverse proxies for X-Forwarded-For
|
||||||
# from the reverse proxies in TRUSTED_PROXIES: bin/entrypoint.sh
|
set_real_ip_from 10.0.0.0/8;
|
||||||
# writes one set_real_ip_from line for each into this file, and
|
set_real_ip_from 172.16.0.0/12;
|
||||||
# leaves it empty when TRUSTED_PROXIES is unset, so that by default
|
set_real_ip_from 192.168.0.0/16;
|
||||||
# the client address is the one each request comes from.
|
|
||||||
include /etc/nginx/trusted-proxies.conf;
|
|
||||||
real_ip_header X-Forwarded-For;
|
real_ip_header X-Forwarded-For;
|
||||||
real_ip_recursive on;
|
real_ip_recursive on;
|
||||||
|
|
||||||
@@ -37,7 +30,6 @@ server {
|
|||||||
location /assets/ {
|
location /assets/ {
|
||||||
expires 1y;
|
expires 1y;
|
||||||
add_header Cache-Control "public, immutable";
|
add_header Cache-Control "public, immutable";
|
||||||
include /etc/nginx/security-headers.conf;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# netwatch-server, the Go backend, runs in the same container and
|
# netwatch-server, the Go backend, runs in the same container and
|
||||||
@@ -51,16 +43,6 @@ server {
|
|||||||
proxy_set_header X-Forwarded-For $remote_addr;
|
proxy_set_header X-Forwarded-For $remote_addr;
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
|
||||||
# netwatch-server sets the same security headers on its own
|
|
||||||
# responses. Its copies are dropped so that each header goes out
|
|
||||||
# once, as security-headers.conf sets it.
|
|
||||||
proxy_hide_header Strict-Transport-Security;
|
|
||||||
proxy_hide_header Content-Security-Policy;
|
|
||||||
proxy_hide_header X-Frame-Options;
|
|
||||||
proxy_hide_header X-Content-Type-Options;
|
|
||||||
proxy_hide_header Referrer-Policy;
|
|
||||||
proxy_hide_header Permissions-Policy;
|
|
||||||
|
|
||||||
location /api/ {
|
location /api/ {
|
||||||
proxy_pass http://127.0.0.1:8081;
|
proxy_pass http://127.0.0.1:8081;
|
||||||
}
|
}
|
||||||
|
|||||||
+7
-25
@@ -3,12 +3,12 @@
|
|||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes nothing is present. Node is
|
# or apk (detected in that order); assumes nothing is present. Node is
|
||||||
# used directly if it is at least NODE_MIN_VERSION; otherwise it is
|
# used directly if installed; otherwise it is installed at a pinned
|
||||||
# installed at a pinned version via nvm (installing nvm itself first,
|
# version via nvm (installing nvm itself first, from a hash-verified
|
||||||
# from a hash-verified release archive, never curl | sh). Go, with its
|
# release archive, never curl | sh). Go, with its gofmt, is used
|
||||||
# gofmt, is used directly if it is at least the version backend/go.mod
|
# directly if it is at least the version backend/go.mod asks for;
|
||||||
# asks for; otherwise the pinned Go release is installed from its
|
# otherwise the pinned Go release is installed from its hash-verified
|
||||||
# hash-verified archive.
|
# archive.
|
||||||
#
|
#
|
||||||
# What this script installs outside the system package manager lives
|
# What this script installs outside the system package manager lives
|
||||||
# under $HOME and is linked into ~/.local/bin, where make and the git
|
# under $HOME and is linked into ~/.local/bin, where make and the git
|
||||||
@@ -23,10 +23,6 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
|
|
||||||
# Pinned versions, 2026-07-07
|
# Pinned versions, 2026-07-07
|
||||||
NODE_VERSION="22.17.0"
|
NODE_VERSION="22.17.0"
|
||||||
# The oldest node the frontend's dependencies accept: the "engines"
|
|
||||||
# field of puppeteer-core 25.5.0, the most demanding of them, asks for
|
|
||||||
# 22.12.0 or newer, 2026-09-29. An older installed node is not used.
|
|
||||||
NODE_MIN_VERSION="22.12.0"
|
|
||||||
NVM_VERSION="0.40.3"
|
NVM_VERSION="0.40.3"
|
||||||
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
|
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
|
||||||
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
|
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
|
||||||
@@ -140,22 +136,8 @@ ensure_nvm() {
|
|||||||
rm -rf "$tmp"
|
rm -rf "$tmp"
|
||||||
}
|
}
|
||||||
|
|
||||||
# node_ok: the node on PATH is at least NODE_MIN_VERSION. node itself
|
|
||||||
# compares the two: major, then minor, then patch.
|
|
||||||
node_ok() {
|
|
||||||
if missing node; then return 1; fi
|
|
||||||
node -e '
|
|
||||||
const have = process.versions.node.split(".").map(Number);
|
|
||||||
const want = process.argv[1].split(".").map(Number);
|
|
||||||
for (let i = 0; i < 3; i++) {
|
|
||||||
if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1);
|
|
||||||
}
|
|
||||||
' "$NODE_MIN_VERSION"
|
|
||||||
}
|
|
||||||
|
|
||||||
# ensure_node: unless node_ok, install NODE_VERSION and link its node.
|
|
||||||
ensure_node() {
|
ensure_node() {
|
||||||
if node_ok; then return 0; fi
|
if ! missing node; then return 0; fi
|
||||||
ensure_nvm
|
ensure_nvm
|
||||||
nvm_sh "nvm install $NODE_VERSION"
|
nvm_sh "nvm install $NODE_VERSION"
|
||||||
link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node
|
link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node
|
||||||
|
|||||||
+6
-20
@@ -1,29 +1,15 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
# script/cibuild: run the CI build: build the one image from Dockerfile,
|
||||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
# whose stages run the checks as build steps (the backend's fmt-check,
|
||||||
# the formatter on the host, which a pristine checkout cannot do.
|
# lint and tests, and the frontend's test, lint and fmt-check). This is
|
||||||
# --no-cache for the same reason as script/docker: the gate phases the
|
# the only build step the Gitea workflow runs.
|
||||||
# final stage depends on are RUN steps, and a cached one is a check that
|
|
||||||
# did not run.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$SCRIPT_DIR/bootstrap"
|
timeout 300 docker build .
|
||||||
"$SCRIPT_DIR/check"
|
|
||||||
# Own line: a failing command substitution inside an argument does
|
|
||||||
# not trip `set -e`, so the inline form degrades silently to an
|
|
||||||
# empty constant. VERSION is computed here because .dockerignore
|
|
||||||
# excludes .git, so `git describe` in a build stage yields an empty
|
|
||||||
# version without failing.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
|
||||||
[ -n "$version" ] || version="unknown"
|
|
||||||
docker build --no-cache \
|
|
||||||
--build-arg VERSION="$version" \
|
|
||||||
-t "$("$SCRIPT_DIR/projectname")" .
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -63,16 +63,11 @@ main() {
|
|||||||
|
|
||||||
# nginx.conf is a template: the image renders it over its own
|
# nginx.conf is a template: the image renders it over its own
|
||||||
# default.conf, with the same port and limit bin/entrypoint.sh uses.
|
# default.conf, with the same port and limit bin/entrypoint.sh uses.
|
||||||
# The empty file it includes trusts no proxy, as bin/entrypoint.sh
|
|
||||||
# writes it when TRUSTED_PROXIES is unset. nginx.conf also includes
|
|
||||||
# the security headers, so the page runs under the shipped policy.
|
|
||||||
docker run -d --rm --name "$SERVER" \
|
docker run -d --rm --name "$SERVER" \
|
||||||
--network "$NETWORK" --network-alias netwatch \
|
--network "$NETWORK" --network-alias netwatch \
|
||||||
-e PORT=8080 -e NGINX_ENVSUBST_FILTER='^PORT$' \
|
-e PORT=8080 -e NGINX_ENVSUBST_FILTER='^PORT$' \
|
||||||
-v "$ROOT/dist:/usr/share/nginx/html:ro" \
|
-v "$ROOT/dist:/usr/share/nginx/html:ro" \
|
||||||
-v "$ROOT/nginx.conf:/etc/nginx/templates/default.conf.template:ro" \
|
-v "$ROOT/nginx.conf:/etc/nginx/templates/default.conf.template:ro" \
|
||||||
-v /dev/null:/etc/nginx/trusted-proxies.conf:ro \
|
|
||||||
-v "$ROOT/security-headers.conf:/etc/nginx/security-headers.conf:ro" \
|
|
||||||
"$SERVER_IMAGE" > /dev/null
|
"$SERVER_IMAGE" > /dev/null
|
||||||
|
|
||||||
# The image's own entrypoint already exposes CDP on 9222 and passes
|
# The image's own entrypoint already exposes CDP on 9222 and passes
|
||||||
|
|||||||
@@ -1,24 +0,0 @@
|
|||||||
# The security headers REPO_POLICIES.md requires on every response.
|
|
||||||
# nginx.conf includes this file, which Dockerfile copies to
|
|
||||||
# /etc/nginx/security-headers.conf. always sends each header on error
|
|
||||||
# responses too.
|
|
||||||
|
|
||||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
|
||||||
|
|
||||||
# Scripts and styles load only from the page's own origin. Inline ones
|
|
||||||
# are blocked, style attributes in markup included, so style elements
|
|
||||||
# through classes or element.style. data: images are for the favicon
|
|
||||||
# in index.html. connect-src is * because the browser checks each probe in
|
|
||||||
# src/main.js against it, and also every redirect the probe follows,
|
|
||||||
# and several of those hosts redirect to others; a list of hosts here
|
|
||||||
# would block those probes. It also covers the reports the page sends
|
|
||||||
# to its own origin.
|
|
||||||
add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
|
|
||||||
|
|
||||||
add_header X-Frame-Options DENY always;
|
|
||||||
add_header X-Content-Type-Options nosniff always;
|
|
||||||
|
|
||||||
# The probed hosts are not told where the page is served from.
|
|
||||||
add_header Referrer-Policy no-referrer always;
|
|
||||||
|
|
||||||
add_header Permissions-Policy "accelerometer=(), camera=(), display-capture=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), usb=()" always;
|
|
||||||
+1
-1
@@ -716,7 +716,7 @@ function hostRowHTML(host, index, showPin = true) {
|
|||||||
${pinBtn}
|
${pinBtn}
|
||||||
<div class="w-[420px] flex-shrink-0 grid grid-cols-[minmax(0,1fr)_auto] items-center">
|
<div class="w-[420px] flex-shrink-0 grid grid-cols-[minmax(0,1fr)_auto] items-center">
|
||||||
<div class="flex items-center gap-2 min-w-[200px]">
|
<div class="flex items-center gap-2 min-w-[200px]">
|
||||||
<div class="w-3 h-3 rounded-full flex-shrink-0 bg-[#6b7280]"></div>
|
<div class="w-3 h-3 rounded-full flex-shrink-0" style="background-color: ${latencyHex(null)}"></div>
|
||||||
<span class="font-medium text-white truncate">${host.name}</span>
|
<span class="font-medium text-white truncate">${host.name}</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="latency-value text-4xl font-bold tabular-nums text-right mt-3" data-host="${index}">
|
<div class="latency-value text-4xl font-bold tabular-nums text-right mt-3" data-host="${index}">
|
||||||
|
|||||||
Reference in New Issue
Block a user