Compare commits
1
Commits
next
..
f665ab17bc
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f665ab17bc |
@@ -72,7 +72,6 @@ RUN addgroup -g 1000 -S netwatch && \
|
||||
# conf.d; bin/entrypoint.sh says how.
|
||||
RUN rm /etc/nginx/conf.d/default.conf
|
||||
COPY nginx.conf /etc/nginx/templates/netwatch.conf.template
|
||||
COPY security-headers.conf /etc/nginx/security-headers.conf
|
||||
COPY --from=frontend /app/dist /usr/share/nginx/html
|
||||
COPY --from=builder /src/netwatch-server /usr/local/bin/netwatch-server
|
||||
COPY bin/entrypoint.sh /usr/local/bin/entrypoint.sh
|
||||
@@ -84,12 +83,6 @@ VOLUME /data
|
||||
# The default public port; PORT changes it.
|
||||
EXPOSE 8080
|
||||
|
||||
# Requests the backend's health check through nginx, on the port from
|
||||
# PORT, so it fails unless both answer. upaas reads the result 60
|
||||
# seconds after a deploy and fails the deploy unless it is healthy.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||
CMD wget -q -O /dev/null "http://127.0.0.1:${PORT:-8080}/.well-known/healthcheck"
|
||||
|
||||
# The nginx image stops its container with SIGQUIT; the entrypoint
|
||||
# acts on TERM and INT.
|
||||
STOPSIGNAL SIGTERM
|
||||
|
||||
@@ -184,64 +184,16 @@ container: nginx serves the built frontend and passes `/api/` and
|
||||
only inside the container, on `127.0.0.1:8081`. The image:
|
||||
|
||||
- Listens on port 8080 by default (override with `PORT` env var)
|
||||
- Takes the client address from `X-Forwarded-For` only on requests from the
|
||||
reverse proxies named in `TRUSTED_PROXIES`, and by default from none
|
||||
- Trusts `X-Forwarded-For` from RFC1918 reverse proxies (10/8, 172.16/12,
|
||||
192.168/16)
|
||||
- Sends access logs to stdout
|
||||
- Caches static assets with immutable headers
|
||||
- Sends the security headers `REPO_POLICIES.md` requires on every response, as
|
||||
`security-headers.conf` sets them, in place of the backend's own
|
||||
- Stores reports in `DATA_DIR`, `/data/reports` by default, on the `/data`
|
||||
volume. The backend runs as user `netwatch` (uid 1000), so a directory
|
||||
bind-mounted at `/data` must be writable by uid 1000
|
||||
- Writes buffered reports to disk on `docker stop`, and exits non-zero if nginx
|
||||
or the backend exits on its own, so the platform restarts it
|
||||
|
||||
## Running under upaas
|
||||
|
||||
What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
|
||||
|
||||
- **Port:** container port `8080`.
|
||||
- **Volume:** container path `/data`; the reports are kept in `/data/reports`.
|
||||
- **First run:** upaas bind-mounts the host directory it is given and does not
|
||||
create it, and the backend, which runs as uid 1000, does not start unless it
|
||||
can write there. Create the directory, owned by uid 1000, before the first
|
||||
deploy:
|
||||
|
||||
```bash
|
||||
mkdir -p /path/to/data
|
||||
chown 1000:1000 /path/to/data
|
||||
```
|
||||
|
||||
- **Environment variables:** none is required. An empty one counts as unset, and
|
||||
one set to a value netwatch cannot use stops the container at start, with the
|
||||
reason in its log.
|
||||
- `PORT`, default `8080`: the container port, from 1 to 65535. `8081` cannot
|
||||
be used: the backend listens on it inside the container
|
||||
- `REPORTS_PER_MINUTE`, default `60`: reports each client address may send a
|
||||
minute
|
||||
- `DATA_DIR_MAX_BYTES`, default `1073741824` (1 GiB): the most room the
|
||||
report files may take
|
||||
- `CORS_ALLOWED_ORIGINS`, default empty: other origins whose pages may call
|
||||
the API
|
||||
- `DEBUG`, default `false`: debug logging
|
||||
- `DATA_DIR`, default `/data/reports`: leave unset; reports kept outside
|
||||
`/data` do not survive a redeploy
|
||||
- `TRUSTED_PROXIES`, default empty: set it to the address the reverse proxy
|
||||
in front of the container connects from, as an IP address or CIDR; several
|
||||
are separated by commas. nginx takes the client address from
|
||||
`X-Forwarded-For` only on a request from one of them, and the rate limit
|
||||
counts that address. Unset, `X-Forwarded-For` is ignored and every client
|
||||
behind the proxy shares the proxy's one allowance of `REPORTS_PER_MINUTE`.
|
||||
Name only addresses nothing but the proxy connects from: any client that
|
||||
connects from one can write its own `X-Forwarded-For`, and through a port
|
||||
Docker publishes, every client may connect from the Docker network's
|
||||
gateway, such as `172.17.0.1`.
|
||||
- **Health check:** the image's `HEALTHCHECK` requests
|
||||
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
|
||||
both nginx and the backend answer. upaas reads the container's health 60
|
||||
seconds after a deploy and fails the deploy unless it is `healthy`. The
|
||||
container also stops when either process exits.
|
||||
|
||||
## Browser Compatibility
|
||||
|
||||
Requires a modern browser with ES modules, Fetch API, Canvas API, and CSS custom
|
||||
|
||||
@@ -23,45 +23,12 @@ latest run passes.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-29: nginx sends the security headers `REPO_POLICIES.md` requires on
|
||||
every response (issue #18), including errors, `/assets/` and what it passes on
|
||||
from the backend, whose own copies it drops so each header goes out once. They
|
||||
live in `security-headers.conf`, which `nginx.conf` includes. The content
|
||||
security policy allows no inline script or style, so the status dot's grey in
|
||||
`src/main.js` is now a class; `connect-src` is `*` because probed hosts
|
||||
redirect to others, and the browser checks each redirect against it
|
||||
- 2026-09-29: the request log is bounded (issue #60): the method, URL, protocol,
|
||||
`User-Agent`, `Referer`, request ID (which chi takes from the client's
|
||||
`X-Request-Id` header) and client address it writes are each cut to 128 bytes,
|
||||
the bound the report handler already used, so one request can no longer put
|
||||
about 1 MiB per field into a log line. That bound and its helper now live in
|
||||
the `logger` package, shared by both
|
||||
- 2026-09-29: nginx takes the client address from `X-Forwarded-For` only on
|
||||
requests from the reverse proxies named in the container's `TRUSTED_PROXIES`
|
||||
(issue #64), and by default from none, where it trusted every RFC1918 address
|
||||
before, so a client could write a new address on each request and escape the
|
||||
rate limit. `bin/entrypoint.sh` writes one `set_real_ip_from` line per entry
|
||||
into `/etc/nginx/trusted-proxies.conf`, which `nginx.conf` includes, refusing
|
||||
an entry that is not an IP address or CIDR, as `netwatch-server check-cidr`
|
||||
finds; it starts the backend with `TRUSTED_PROXIES=127.0.0.1/32`, since nginx
|
||||
is its only client
|
||||
- 2026-09-29: report file names can no longer collide (issue #61): each is
|
||||
`reports-<timestamp>-<number>.jsonl.zst`, where the number goes up by one for
|
||||
each file the server starts to write, so two flushes in the same millisecond,
|
||||
such as a flush for size and the final flush at shutdown, each get a file of
|
||||
their own instead of the second one failing. A failed write uses up its
|
||||
number, leaving a gap if the file could not be created and otherwise a file
|
||||
under that number that may be incomplete.
|
||||
- 2026-09-29: ready to run under upaas (issue #59): the image has a
|
||||
`HEALTHCHECK` that requests `/.well-known/healthcheck` through nginx on the
|
||||
port from `PORT`. The backend no longer reads a bad `PORT` as 0 or a bad
|
||||
`DEBUG` as false: those, and a `BIND_ADDRESS` that is not an IP address, stop
|
||||
it from starting with an error naming the variable, as the limits,
|
||||
`CORS_ALLOWED_ORIGINS` and, now by name, `TRUSTED_PROXIES` already did.
|
||||
`bin/entrypoint.sh` also refuses a `PORT` outside 1 to 65535, and `8081`,
|
||||
where the backend listens inside the container, naming `PORT`. `README.md` has
|
||||
a "Running under upaas" section, whose first-run steps create the host
|
||||
directory for `/data` owned by uid 1000; the image does not change its owner
|
||||
their own instead of the second one failing. A failed write leaves a gap in
|
||||
the numbers.
|
||||
- 2026-09-29: nginx listens on `PORT` (issue #26), 8080 when unset or empty: the
|
||||
nginx image renders `nginx.conf` as a template at container start, filling in
|
||||
`PORT` and no other variable. `bin/entrypoint.sh` refuses to start when `PORT`
|
||||
|
||||
+8
-27
@@ -87,14 +87,9 @@ Internal packages in `internal/` follow standard Go project layout:
|
||||
| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) |
|
||||
|
||||
`TRUSTED_PROXIES` defaults to `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`.
|
||||
The loopback entries cover a reverse proxy on the same host. A request whose
|
||||
direct peer is outside this set has its forwarded headers ignored, and the
|
||||
direct peer is logged and rate-limited instead. The container image does not use
|
||||
this default; see [Container image](#container-image).
|
||||
|
||||
A variable set to a value the server cannot use, such as `PORT=abc`,
|
||||
`DEBUG=maybe` or a `BIND_ADDRESS` that is not an IP address, stops it from
|
||||
starting, with an error naming the variable. An empty variable counts as unset.
|
||||
The loopback entries cover the reverse proxy that shares the container; the
|
||||
RFC1918 ranges match `nginx.conf`. A request whose direct peer is outside this
|
||||
set has its forwarded headers ignored, and the direct peer is logged instead.
|
||||
|
||||
### Container image
|
||||
|
||||
@@ -102,21 +97,8 @@ The root `Dockerfile` builds one image in which nginx listens on the public port
|
||||
8080, serves the frontend, and proxies `/api/` and `/.well-known/healthcheck` to
|
||||
this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as
|
||||
user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so
|
||||
only nginx reaches it, and with `TRUSTED_PROXIES=127.0.0.1/32`, so it takes the
|
||||
client address nginx passes on and no other. `DATA_DIR` is `/data/reports`, on
|
||||
the `/data` volume, which `netwatch` owns. nginx replaces the security headers
|
||||
this server sets with those in the root `security-headers.conf`, so those are
|
||||
what clients of the image see.
|
||||
|
||||
The container's own `TRUSTED_PROXIES` goes to nginx instead: IP addresses or
|
||||
CIDRs, separated by commas, of the reverse proxies in front of the container.
|
||||
nginx takes the client address from `X-Forwarded-For` only on a request from one
|
||||
of them. Unset or empty, nginx trusts no proxy, and the client address is the
|
||||
one each request comes from, so every client behind a proxy shares one rate
|
||||
limit. An entry that is not an IP address or CIDR, such as a hostname or
|
||||
`1.2.3`, stops the container at start with an error naming `TRUSTED_PROXIES`:
|
||||
the entrypoint checks each entry with `netwatch-server check-cidr`, which parses
|
||||
it as this server parses its own `TRUSTED_PROXIES`.
|
||||
only nginx reaches it. `DATA_DIR` is `/data/reports`, on the `/data` volume,
|
||||
which `netwatch` owns.
|
||||
|
||||
### Report storage
|
||||
|
||||
@@ -124,10 +106,9 @@ Reports are written as `reports-<timestamp>-<number>.jsonl.zst` files in
|
||||
`DATA_DIR`. The timestamp is in UTC to the millisecond, so the names sort by
|
||||
time. The number starts at 1 when the server starts and goes up by one for each
|
||||
file the server starts to write, so two files written in the same millisecond
|
||||
still get different names. A failed write uses up its number, leaving a gap in
|
||||
the numbers if the file could not be created and otherwise a file under that
|
||||
number that may be incomplete. Each file contains one JSON object per line,
|
||||
compressed with zstd. Files are created with `O_EXCL` to prevent overwrites.
|
||||
still get different names, and a failed write leaves a gap in the numbers. Each
|
||||
file contains one JSON object per line, compressed with zstd. Files are created
|
||||
with `O_EXCL` to prevent overwrites.
|
||||
|
||||
### Report limits
|
||||
|
||||
|
||||
@@ -2,9 +2,6 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"sneak.berlin/go/netwatch/internal/config"
|
||||
"sneak.berlin/go/netwatch/internal/globals"
|
||||
"sneak.berlin/go/netwatch/internal/handlers"
|
||||
@@ -25,19 +22,6 @@ var (
|
||||
)
|
||||
|
||||
func main() {
|
||||
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
|
||||
// this server would refuse CIDR in its TRUSTED_PROXIES.
|
||||
// bin/entrypoint.sh runs it on each entry it gives nginx.
|
||||
if len(os.Args) == 3 && os.Args[1] == "check-cidr" {
|
||||
_, err := middleware.ParseTrustedProxies(os.Args[2:])
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
globals.Appname = Appname
|
||||
globals.Version = Version
|
||||
globals.Buildarch = Buildarch
|
||||
|
||||
@@ -6,10 +6,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"math"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"sneak.berlin/go/netwatch/internal/globals"
|
||||
@@ -21,11 +18,10 @@ import (
|
||||
)
|
||||
|
||||
// defaultTrustedProxies lists the networks whose forwarded
|
||||
// headers are honoured by default: IPv4 and IPv6 loopback,
|
||||
// for a reverse proxy on the same host, and the RFC1918
|
||||
// ranges. The container image does not use it:
|
||||
// bin/entrypoint.sh gives the server 127.0.0.1/32, since
|
||||
// nginx is its only client there.
|
||||
// headers are honoured by default. It covers the RFC1918
|
||||
// ranges (to match nginx.conf) plus IPv4 and IPv6 loopback,
|
||||
// because the reverse proxy shares the container and reaches
|
||||
// the backend over loopback.
|
||||
const defaultTrustedProxies = "127.0.0.1/32,::1/128," +
|
||||
"10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
|
||||
|
||||
@@ -41,9 +37,6 @@ var (
|
||||
errNotOrigin = errors.New(
|
||||
"must be an origin, scheme://host with an optional port",
|
||||
)
|
||||
errNotPort = errors.New("must be a port number, 1 to 65535")
|
||||
errNotBool = errors.New("must be true or false")
|
||||
errNotIP = errors.New("must be an IP address, or empty")
|
||||
)
|
||||
|
||||
// Params defines the dependencies for Config.
|
||||
@@ -72,8 +65,7 @@ type Config struct {
|
||||
}
|
||||
|
||||
// New loads configuration from env, .env files, and config
|
||||
// files, returning a fully resolved Config. It fails, with an error
|
||||
// naming the setting, on a value the server cannot use.
|
||||
// files, returning a fully resolved Config.
|
||||
func New(
|
||||
_ fx.Lifecycle,
|
||||
params Params,
|
||||
@@ -111,29 +103,15 @@ func New(
|
||||
}
|
||||
}
|
||||
|
||||
// Read with strconv: viper's GetInt and GetBool would read a value
|
||||
// they cannot parse as 0 or false instead of failing.
|
||||
port, err := strconv.Atoi(viper.GetString("PORT"))
|
||||
if err != nil || port < 1 || port > math.MaxUint16 {
|
||||
return nil, fmt.Errorf("PORT %q: %w",
|
||||
viper.GetString("PORT"), errNotPort)
|
||||
}
|
||||
|
||||
debug, err := strconv.ParseBool(viper.GetString("DEBUG"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("DEBUG %q: %w",
|
||||
viper.GetString("DEBUG"), errNotBool)
|
||||
}
|
||||
|
||||
s := &Config{
|
||||
BindAddress: viper.GetString("BIND_ADDRESS"),
|
||||
CORSAllowedOrigins: splitList(viper.GetString("CORS_ALLOWED_ORIGINS")),
|
||||
DataDir: viper.GetString("DATA_DIR"),
|
||||
DataDirMaxBytes: viper.GetInt64("DATA_DIR_MAX_BYTES"),
|
||||
Debug: debug,
|
||||
Debug: viper.GetBool("DEBUG"),
|
||||
MetricsPassword: viper.GetString("METRICS_PASSWORD"),
|
||||
MetricsUsername: viper.GetString("METRICS_USERNAME"),
|
||||
Port: port,
|
||||
Port: viper.GetInt("PORT"),
|
||||
ReportsPerMinute: viper.GetInt("REPORTS_PER_MINUTE"),
|
||||
SentryDSN: viper.GetString("SENTRY_DSN"),
|
||||
TrustedProxies: splitList(viper.GetString("TRUSTED_PROXIES")),
|
||||
@@ -141,7 +119,19 @@ func New(
|
||||
params: ¶ms,
|
||||
}
|
||||
|
||||
err = s.check()
|
||||
// viper reads a value that is not a number as 0, so this also
|
||||
// catches a mistyped setting.
|
||||
if s.ReportsPerMinute <= 0 {
|
||||
return nil, fmt.Errorf("REPORTS_PER_MINUTE %q: %w",
|
||||
viper.GetString("REPORTS_PER_MINUTE"), errNotPositive)
|
||||
}
|
||||
|
||||
if s.DataDirMaxBytes <= 0 {
|
||||
return nil, fmt.Errorf("DATA_DIR_MAX_BYTES %q: %w",
|
||||
viper.GetString("DATA_DIR_MAX_BYTES"), errNotPositive)
|
||||
}
|
||||
|
||||
err = checkOrigins(s.CORSAllowedOrigins)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -154,32 +144,6 @@ func New(
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// check fails with an error naming the first setting here whose value
|
||||
// the server cannot use. New checks PORT and DEBUG as it reads them,
|
||||
// and the middleware checks TRUSTED_PROXIES as it parses it.
|
||||
func (s *Config) check() error {
|
||||
// viper reads a value that is not a number as 0, so this also
|
||||
// catches a mistyped setting.
|
||||
if s.ReportsPerMinute <= 0 {
|
||||
return fmt.Errorf("REPORTS_PER_MINUTE %q: %w",
|
||||
viper.GetString("REPORTS_PER_MINUTE"), errNotPositive)
|
||||
}
|
||||
|
||||
if s.DataDirMaxBytes <= 0 {
|
||||
return fmt.Errorf("DATA_DIR_MAX_BYTES %q: %w",
|
||||
viper.GetString("DATA_DIR_MAX_BYTES"), errNotPositive)
|
||||
}
|
||||
|
||||
if s.BindAddress != "" {
|
||||
_, err := netip.ParseAddr(s.BindAddress)
|
||||
if err != nil {
|
||||
return fmt.Errorf("BIND_ADDRESS %q: %w", s.BindAddress, errNotIP)
|
||||
}
|
||||
}
|
||||
|
||||
return checkOrigins(s.CORSAllowedOrigins)
|
||||
}
|
||||
|
||||
// checkOrigins fails on the first CORS_ALLOWED_ORIGINS entry that is
|
||||
// not a plain origin, scheme://host with an optional port, as browsers
|
||||
// send it; anything more, such as a trailing "/", would match no page.
|
||||
|
||||
@@ -29,63 +29,6 @@ func requireConfigError(t *testing.T, setting string) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSettingsLoadAsGiven: valid values pass the checks and are used
|
||||
// as given. bin/entrypoint.sh starts the server with these
|
||||
// BIND_ADDRESS and PORT values.
|
||||
func TestSettingsLoadAsGiven(t *testing.T) {
|
||||
t.Setenv("BIND_ADDRESS", "127.0.0.1")
|
||||
t.Setenv("PORT", "8081")
|
||||
t.Setenv("DEBUG", "true")
|
||||
|
||||
var cfg *config.Config
|
||||
|
||||
app := fx.New(
|
||||
fx.NopLogger,
|
||||
fx.Provide(globals.New, logger.New, config.New),
|
||||
fx.Populate(&cfg),
|
||||
)
|
||||
|
||||
err := app.Err()
|
||||
if err != nil {
|
||||
t.Fatalf("config error = %v", err)
|
||||
}
|
||||
|
||||
if cfg.BindAddress != "127.0.0.1" || cfg.Port != 8081 || !cfg.Debug {
|
||||
t.Fatalf("BindAddress, Port, Debug = %q, %d, %t; "+
|
||||
"want \"127.0.0.1\", 8081, true",
|
||||
cfg.BindAddress, cfg.Port, cfg.Debug)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPortMustBeAPortNumber: viper reads a value that is not a number
|
||||
// as 0, on which the server would listen on a random port.
|
||||
func TestPortMustBeAPortNumber(t *testing.T) {
|
||||
for _, value := range []string{"abc", "0", "65536", "8080.5"} {
|
||||
t.Run(value, func(t *testing.T) {
|
||||
t.Setenv("PORT", value)
|
||||
|
||||
requireConfigError(t, "PORT")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDebugMustBeTrueOrFalse: viper reads any other value, such as
|
||||
// "yes", as false.
|
||||
func TestDebugMustBeTrueOrFalse(t *testing.T) {
|
||||
t.Setenv("DEBUG", "yes")
|
||||
|
||||
requireConfigError(t, "DEBUG")
|
||||
}
|
||||
|
||||
// TestBindAddressMustBeAnIPAddress: a host name would be looked up
|
||||
// only once the server starts listening, and a mistyped one would stop
|
||||
// it then with an error that does not name the setting.
|
||||
func TestBindAddressMustBeAnIPAddress(t *testing.T) {
|
||||
t.Setenv("BIND_ADDRESS", "localhost")
|
||||
|
||||
requireConfigError(t, "BIND_ADDRESS")
|
||||
}
|
||||
|
||||
// TestReportsPerMinuteMustBePositive: unchecked, zero would panic
|
||||
// when the routes are built, and a negative rate would lift the
|
||||
// limit.
|
||||
|
||||
@@ -2,6 +2,9 @@ package handlers
|
||||
|
||||
import "log/slog"
|
||||
|
||||
// MaxLoggedFieldBytes exposes the log bound to the external tests.
|
||||
const MaxLoggedFieldBytes = maxLoggedFieldBytes
|
||||
|
||||
// NewForTest builds a Handlers around a report sink and logger,
|
||||
// bypassing the fx graph so handler behaviour (including the
|
||||
// storage failure path) is exercisable in unit tests.
|
||||
|
||||
@@ -5,10 +5,14 @@ import (
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"sneak.berlin/go/netwatch/internal/logger"
|
||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
||||
)
|
||||
|
||||
// maxLoggedFieldBytes bounds untrusted text (string fields,
|
||||
// decode error text) before it is logged, so a caller cannot
|
||||
// inflate log volume with an oversized value.
|
||||
const maxLoggedFieldBytes = 128
|
||||
|
||||
type reportSample struct {
|
||||
T int64 `json:"t"`
|
||||
Latency *int `json:"latency"`
|
||||
@@ -79,7 +83,7 @@ func (s *Handlers) decodeErrorStatus(err error) int {
|
||||
// The decoder's error text can quote request bytes (a whole
|
||||
// oversized number, for example), so it is bounded too.
|
||||
s.log.Error("failed to decode report",
|
||||
"error", logger.BoundedForLog(err.Error()),
|
||||
"error", boundedForLog(err.Error()),
|
||||
)
|
||||
|
||||
return http.StatusBadRequest
|
||||
@@ -111,10 +115,20 @@ func (s *Handlers) logReportReceived(rpt report) {
|
||||
}
|
||||
|
||||
s.log.Info("report received",
|
||||
"client_id", logger.BoundedForLog(rpt.ClientID),
|
||||
"timestamp", logger.BoundedForLog(rpt.Timestamp),
|
||||
"client_id", boundedForLog(rpt.ClientID),
|
||||
"timestamp", boundedForLog(rpt.Timestamp),
|
||||
"host_count", len(rpt.Hosts),
|
||||
"total_samples", totalSamples,
|
||||
"geo_bytes", len(rpt.Geo),
|
||||
)
|
||||
}
|
||||
|
||||
// boundedForLog truncates an untrusted string to a fixed byte
|
||||
// bound so an attacker-controlled field cannot dominate the log.
|
||||
func boundedForLog(s string) string {
|
||||
if len(s) > maxLoggedFieldBytes {
|
||||
return s[:maxLoggedFieldBytes]
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
@@ -12,7 +12,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/netwatch/internal/handlers"
|
||||
"sneak.berlin/go/netwatch/internal/logger"
|
||||
"sneak.berlin/go/netwatch/internal/middleware"
|
||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
||||
)
|
||||
@@ -175,7 +174,7 @@ func TestHandleReportDoesNotLogRawGeo(t *testing.T) {
|
||||
func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
long := strings.Repeat("c", 2*logger.MaxLoggedFieldBytes)
|
||||
long := strings.Repeat("c", 2*handlers.MaxLoggedFieldBytes)
|
||||
|
||||
var logbuf bytes.Buffer
|
||||
|
||||
@@ -198,16 +197,16 @@ func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
|
||||
t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String())
|
||||
}
|
||||
|
||||
want := long[:logger.MaxLoggedFieldBytes]
|
||||
want := long[:handlers.MaxLoggedFieldBytes]
|
||||
|
||||
if logged["client_id"] != want {
|
||||
t.Fatalf("logged client_id not cut to %d bytes: %q",
|
||||
logger.MaxLoggedFieldBytes, logged["client_id"])
|
||||
handlers.MaxLoggedFieldBytes, logged["client_id"])
|
||||
}
|
||||
|
||||
if logged["timestamp"] != want {
|
||||
t.Fatalf("logged timestamp not cut to %d bytes: %q",
|
||||
logger.MaxLoggedFieldBytes, logged["timestamp"])
|
||||
handlers.MaxLoggedFieldBytes, logged["timestamp"])
|
||||
}
|
||||
}
|
||||
|
||||
@@ -216,7 +215,7 @@ func TestHandleReportDecodeErrorLogIsBounded(t *testing.T) {
|
||||
|
||||
// A number too large for its int64 field makes the decoder's
|
||||
// error text quote the whole number.
|
||||
huge := strings.Repeat("9", 2*logger.MaxLoggedFieldBytes)
|
||||
huge := strings.Repeat("9", 2*handlers.MaxLoggedFieldBytes)
|
||||
|
||||
var logbuf bytes.Buffer
|
||||
|
||||
|
||||
@@ -11,21 +11,6 @@ import (
|
||||
"go.uber.org/fx"
|
||||
)
|
||||
|
||||
// MaxLoggedFieldBytes bounds untrusted text (request fields,
|
||||
// header values, decode error text) before it is logged, so a
|
||||
// caller cannot inflate log volume with an oversized value.
|
||||
const MaxLoggedFieldBytes = 128
|
||||
|
||||
// BoundedForLog truncates an untrusted string to a fixed byte
|
||||
// bound so an attacker-controlled field cannot dominate the log.
|
||||
func BoundedForLog(s string) string {
|
||||
if len(s) > MaxLoggedFieldBytes {
|
||||
return s[:MaxLoggedFieldBytes]
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// Params defines the dependencies for Logger.
|
||||
type Params struct {
|
||||
fx.In
|
||||
|
||||
@@ -29,3 +29,7 @@ func ClientIP(
|
||||
) string {
|
||||
return clientIP(remoteAddr, header, trusted)
|
||||
}
|
||||
|
||||
func ParseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
|
||||
return parseTrustedProxies(cidrs)
|
||||
}
|
||||
|
||||
@@ -64,7 +64,7 @@ func New(
|
||||
_ fx.Lifecycle,
|
||||
params Params,
|
||||
) (*Middleware, error) {
|
||||
trusted, err := ParseTrustedProxies(params.Config.TrustedProxies)
|
||||
trusted, err := parseTrustedProxies(params.Config.TrustedProxies)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -77,18 +77,16 @@ func New(
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// ParseTrustedProxies converts the TRUSTED_PROXIES entries into
|
||||
// prefixes, failing fast on any malformed entry. Each entry must be
|
||||
// a CIDR; a lone address is refused. "netwatch-server check-cidr"
|
||||
// runs it too.
|
||||
func ParseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
|
||||
// parseTrustedProxies converts CIDR strings into prefixes,
|
||||
// failing fast on any malformed entry.
|
||||
func parseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
|
||||
prefixes := make([]netip.Prefix, 0, len(cidrs))
|
||||
|
||||
for _, cidr := range cidrs {
|
||||
prefix, err := netip.ParsePrefix(cidr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"TRUSTED_PROXIES %q: %w", cidr, err,
|
||||
"trusted proxy %q: %w", cidr, err,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -189,10 +187,7 @@ func addrInAny(s string, trusted []netip.Prefix) bool {
|
||||
}
|
||||
|
||||
// Logging returns middleware that logs each request with
|
||||
// timing, status code, and client information. Every string
|
||||
// taken from the request is cut to logger.MaxLoggedFieldBytes,
|
||||
// including the request ID, which chi takes from the client's
|
||||
// X-Request-Id header when one is sent.
|
||||
// timing, status code, and client information.
|
||||
func (s *Middleware) Logging() func(http.Handler) http.Handler {
|
||||
return func(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(
|
||||
@@ -205,19 +200,21 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
|
||||
latency := time.Since(start)
|
||||
s.log.InfoContext(ctx, "request",
|
||||
"request_start", start,
|
||||
"method", logger.BoundedForLog(r.Method),
|
||||
"url", logger.BoundedForLog(r.URL.String()),
|
||||
"useragent", logger.BoundedForLog(r.UserAgent()),
|
||||
"method", r.Method,
|
||||
"url", r.URL.String(),
|
||||
"useragent", r.UserAgent(),
|
||||
"request_id",
|
||||
logger.BoundedForLog(middleware.GetReqID(ctx)),
|
||||
"referer", logger.BoundedForLog(r.Referer()),
|
||||
"proto", logger.BoundedForLog(r.Proto),
|
||||
ctx.Value(
|
||||
middleware.RequestIDKey,
|
||||
),
|
||||
"referer", r.Referer(),
|
||||
"proto", r.Proto,
|
||||
"remote_ip",
|
||||
logger.BoundedForLog(clientIP(
|
||||
clientIP(
|
||||
r.RemoteAddr,
|
||||
r.Header,
|
||||
s.trustedProxies,
|
||||
)),
|
||||
),
|
||||
"status", lrw.statusCode,
|
||||
"latency_ms",
|
||||
latency.Milliseconds(),
|
||||
|
||||
@@ -13,10 +13,7 @@ import (
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/netwatch/internal/logger"
|
||||
"sneak.berlin/go/netwatch/internal/middleware"
|
||||
|
||||
chimiddleware "github.com/go-chi/chi/v5/middleware"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -39,32 +36,15 @@ func mustPrefixes(t *testing.T, cidrs ...string) []netip.Prefix {
|
||||
return prefixes
|
||||
}
|
||||
|
||||
// TestParseTrustedProxiesRejectsMalformed includes entries nginx would
|
||||
// read as another address or look up as a hostname, in the CIDR form
|
||||
// bin/entrypoint.sh gives "netwatch-server check-cidr".
|
||||
func TestParseTrustedProxiesRejectsMalformed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, cidr := range []string{
|
||||
"not-a-cidr", "10.0.0.1", "1.2.3/32", "172.30/32", "10/32",
|
||||
"cafe/32", "999.1.1.1/32", "10.0.0.0/33", "::1/129",
|
||||
"fe80::1%eth0/128",
|
||||
} {
|
||||
_, err := middleware.ParseTrustedProxies([]string{cidr})
|
||||
if err == nil || !strings.Contains(err.Error(), "TRUSTED_PROXIES") {
|
||||
t.Errorf("%q: error = %v, want one naming TRUSTED_PROXIES",
|
||||
cidr, err)
|
||||
}
|
||||
_, err := middleware.ParseTrustedProxies([]string{"not-a-cidr"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for malformed CIDR, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseTrustedProxiesAcceptsCIDRs(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mustPrefixes(t, "172.17.0.1/32", "10.0.0.0/8", "2001:db8::1/128",
|
||||
"2001:db8::/32", "::ffff:192.0.2.1/128")
|
||||
}
|
||||
|
||||
type clientIPCase struct {
|
||||
name string
|
||||
remoteAddr string
|
||||
@@ -323,52 +303,6 @@ func TestRecovererRepanicsOnAbortHandler(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoggingCutsRequestStringsToBound sends an over-long URL and
|
||||
// over-long header values, and checks the request log writes each
|
||||
// one cut to logger.MaxLoggedFieldBytes.
|
||||
func TestLoggingCutsRequestStringsToBound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
long := strings.Repeat("a", 2*logger.MaxLoggedFieldBytes)
|
||||
|
||||
var logbuf bytes.Buffer
|
||||
|
||||
mw := middleware.NewWithLogger(
|
||||
slog.New(slog.NewJSONHandler(&logbuf, nil)),
|
||||
)
|
||||
|
||||
handler := chimiddleware.RequestID(mw.Logging()(okHandler()))
|
||||
|
||||
req := httptest.NewRequestWithContext(t.Context(),
|
||||
http.MethodGet, "/"+long, http.NoBody)
|
||||
req.Header.Set("User-Agent", long)
|
||||
req.Header.Set("Referer", long)
|
||||
req.Header.Set("X-Request-Id", long)
|
||||
|
||||
handler.ServeHTTP(httptest.NewRecorder(), req)
|
||||
|
||||
var logged map[string]any
|
||||
|
||||
err := json.Unmarshal(logbuf.Bytes(), &logged)
|
||||
if err != nil {
|
||||
t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String())
|
||||
}
|
||||
|
||||
want := map[string]string{
|
||||
"url": ("/" + long)[:logger.MaxLoggedFieldBytes],
|
||||
"useragent": long[:logger.MaxLoggedFieldBytes],
|
||||
"referer": long[:logger.MaxLoggedFieldBytes],
|
||||
"request_id": long[:logger.MaxLoggedFieldBytes],
|
||||
}
|
||||
|
||||
for field, value := range want {
|
||||
if logged[field] != value {
|
||||
t.Errorf("logged %s = %q, want it cut to %d bytes",
|
||||
field, logged[field], logger.MaxLoggedFieldBytes)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// okHandler stands in for the route a middleware guards.
|
||||
func okHandler() http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
|
||||
+6
-49
@@ -10,9 +10,8 @@ set -u
|
||||
|
||||
# PORT is the public port nginx listens on, 8080 when unset or empty.
|
||||
# nginx would take a value such as localhost or unix:/tmp/x.sock as an
|
||||
# address and start anyway, and reports a bad port without naming
|
||||
# PORT, so a value that is not a usable port stops the container here,
|
||||
# before either process starts.
|
||||
# address and start anyway, so anything but digits stops the container
|
||||
# here, before either process starts.
|
||||
export PORT="${PORT:-8080}"
|
||||
case "$PORT" in
|
||||
*[!0-9]*)
|
||||
@@ -20,46 +19,6 @@ case "$PORT" in
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
# The length is checked first because, for a number too big for it,
|
||||
# the shell's test prints an error and is false, so the range checks
|
||||
# alone would let it through.
|
||||
if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
|
||||
echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$PORT" -eq 8081 ]; then
|
||||
echo "entrypoint: PORT cannot be 8081, netwatch-server listens there" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# TRUSTED_PROXIES names the reverse proxies in front of the container,
|
||||
# as IP addresses or CIDRs separated by commas. nginx takes the client
|
||||
# address from X-Forwarded-For only on a request from one of them, so
|
||||
# unset or empty, it trusts no one. nginx.conf includes the file written
|
||||
# here, one set_real_ip_from line per entry.
|
||||
#
|
||||
# nginx looks up an entry it cannot read as an address as a hostname,
|
||||
# and trusts what it finds (1.2.3 is found as 1.2.0.3). So each entry
|
||||
# is made a CIDR, a lone address getting /128 if it is IPv6 and /32 if
|
||||
# not, and netwatch-server checks it with the parsing it gives its own
|
||||
# TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
|
||||
# below, naming the entry as written. set -f keeps a * in an entry from
|
||||
# becoming a list of file names.
|
||||
TRUSTED_PROXIES="${TRUSTED_PROXIES:-}"
|
||||
set -f
|
||||
for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
|
||||
case "$proxy" in
|
||||
*/*) cidr="$proxy" ;;
|
||||
*:*) cidr="$proxy/128" ;;
|
||||
*) cidr="$proxy/32" ;;
|
||||
esac
|
||||
if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
|
||||
echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
|
||||
"separated by commas; '$proxy' is neither" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "set_real_ip_from $cidr;"
|
||||
done > /etc/nginx/trusted-proxies.conf
|
||||
|
||||
# A stop signal is only noted here; the loop below acts on it.
|
||||
stop_requested=""
|
||||
@@ -67,12 +26,10 @@ trap 'stop_requested=yes' TERM INT
|
||||
|
||||
# netwatch-server runs as the netwatch user and listens on loopback
|
||||
# only, on a port other than the public one; nginx.conf proxies to this
|
||||
# address. Its only client is nginx, so it takes the client address
|
||||
# nginx passes on from 127.0.0.1 alone, whatever TRUSTED_PROXIES the
|
||||
# container has. The netwatch user has no login shell, hence -s
|
||||
# /bin/sh. busybox su replaces itself with the command instead of
|
||||
# staying on as its parent, so $! is the server's own PID.
|
||||
BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
|
||||
# address. The netwatch user has no login shell, hence -s /bin/sh.
|
||||
# busybox su replaces itself with the command instead of staying on as
|
||||
# its parent, so $! is the server's own PID.
|
||||
BIND_ADDRESS=127.0.0.1 PORT=8081 \
|
||||
su -s /bin/sh netwatch -c 'exec netwatch-server' &
|
||||
backend=$!
|
||||
|
||||
|
||||
+4
-22
@@ -8,20 +8,13 @@ server {
|
||||
# Keep the nginx version out of the Server header and error pages.
|
||||
server_tokens off;
|
||||
|
||||
# The security headers, on every response. An add_header in a
|
||||
# location drops every add_header from here, so a location with one
|
||||
# of its own includes this file again.
|
||||
include /etc/nginx/security-headers.conf;
|
||||
|
||||
root /usr/share/nginx/html;
|
||||
index index.html;
|
||||
|
||||
# The client address comes from X-Forwarded-For only on a request
|
||||
# from the reverse proxies in TRUSTED_PROXIES: bin/entrypoint.sh
|
||||
# writes one set_real_ip_from line for each into this file, and
|
||||
# leaves it empty when TRUSTED_PROXIES is unset, so that by default
|
||||
# the client address is the one each request comes from.
|
||||
include /etc/nginx/trusted-proxies.conf;
|
||||
# Trust RFC1918 reverse proxies for X-Forwarded-For
|
||||
set_real_ip_from 10.0.0.0/8;
|
||||
set_real_ip_from 172.16.0.0/12;
|
||||
set_real_ip_from 192.168.0.0/16;
|
||||
real_ip_header X-Forwarded-For;
|
||||
real_ip_recursive on;
|
||||
|
||||
@@ -37,7 +30,6 @@ server {
|
||||
location /assets/ {
|
||||
expires 1y;
|
||||
add_header Cache-Control "public, immutable";
|
||||
include /etc/nginx/security-headers.conf;
|
||||
}
|
||||
|
||||
# netwatch-server, the Go backend, runs in the same container and
|
||||
@@ -51,16 +43,6 @@ server {
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# netwatch-server sets the same security headers on its own
|
||||
# responses. Its copies are dropped so that each header goes out
|
||||
# once, as security-headers.conf sets it.
|
||||
proxy_hide_header Strict-Transport-Security;
|
||||
proxy_hide_header Content-Security-Policy;
|
||||
proxy_hide_header X-Frame-Options;
|
||||
proxy_hide_header X-Content-Type-Options;
|
||||
proxy_hide_header Referrer-Policy;
|
||||
proxy_hide_header Permissions-Policy;
|
||||
|
||||
location /api/ {
|
||||
proxy_pass http://127.0.0.1:8081;
|
||||
}
|
||||
|
||||
@@ -63,16 +63,11 @@ main() {
|
||||
|
||||
# nginx.conf is a template: the image renders it over its own
|
||||
# default.conf, with the same port and limit bin/entrypoint.sh uses.
|
||||
# The empty file it includes trusts no proxy, as bin/entrypoint.sh
|
||||
# writes it when TRUSTED_PROXIES is unset. nginx.conf also includes
|
||||
# the security headers, so the page runs under the shipped policy.
|
||||
docker run -d --rm --name "$SERVER" \
|
||||
--network "$NETWORK" --network-alias netwatch \
|
||||
-e PORT=8080 -e NGINX_ENVSUBST_FILTER='^PORT$' \
|
||||
-v "$ROOT/dist:/usr/share/nginx/html:ro" \
|
||||
-v "$ROOT/nginx.conf:/etc/nginx/templates/default.conf.template:ro" \
|
||||
-v /dev/null:/etc/nginx/trusted-proxies.conf:ro \
|
||||
-v "$ROOT/security-headers.conf:/etc/nginx/security-headers.conf:ro" \
|
||||
"$SERVER_IMAGE" > /dev/null
|
||||
|
||||
# The image's own entrypoint already exposes CDP on 9222 and passes
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
# The security headers REPO_POLICIES.md requires on every response.
|
||||
# nginx.conf includes this file, which Dockerfile copies to
|
||||
# /etc/nginx/security-headers.conf. always sends each header on error
|
||||
# responses too.
|
||||
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
|
||||
# Scripts and styles load only from the page's own origin. Inline ones
|
||||
# are blocked, style attributes in markup included, so style elements
|
||||
# through classes or element.style. data: images are for the favicon
|
||||
# in index.html. connect-src is * because the browser checks each probe in
|
||||
# src/main.js against it, and also every redirect the probe follows,
|
||||
# and several of those hosts redirect to others; a list of hosts here
|
||||
# would block those probes. It also covers the reports the page sends
|
||||
# to its own origin.
|
||||
add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
|
||||
|
||||
add_header X-Frame-Options DENY always;
|
||||
add_header X-Content-Type-Options nosniff always;
|
||||
|
||||
# The probed hosts are not told where the page is served from.
|
||||
add_header Referrer-Policy no-referrer always;
|
||||
|
||||
add_header Permissions-Policy "accelerometer=(), camera=(), display-capture=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), usb=()" always;
|
||||
+1
-1
@@ -716,7 +716,7 @@ function hostRowHTML(host, index, showPin = true) {
|
||||
${pinBtn}
|
||||
<div class="w-[420px] flex-shrink-0 grid grid-cols-[minmax(0,1fr)_auto] items-center">
|
||||
<div class="flex items-center gap-2 min-w-[200px]">
|
||||
<div class="w-3 h-3 rounded-full flex-shrink-0 bg-[#6b7280]"></div>
|
||||
<div class="w-3 h-3 rounded-full flex-shrink-0" style="background-color: ${latencyHex(null)}"></div>
|
||||
<span class="font-medium text-white truncate">${host.name}</span>
|
||||
</div>
|
||||
<div class="latency-value text-4xl font-bold tabular-nums text-right mt-3" data-host="${index}">
|
||||
|
||||
Reference in New Issue
Block a user