diff --git a/README.md b/README.md index 4fa31fb..9628494 100644 --- a/README.md +++ b/README.md @@ -153,8 +153,26 @@ report a delta, so only new samples are sent; the mark advances only on a delivered report, and while paused nothing is sent. Changing the interval clears each host's history on the page; the samples in it not yet reported still go in the next report. Delivery failure is quiet — one debug-log line per outage, -retried at the next interval, never blocking probing. The report-building step -is a pure function of host state. +never blocking probing. The report-building step is a pure function of host +state. + +A sample in the history that no report has delivered is sent only as long as the +history holds it, so after a refused or failed report the next report carries +the samples the history still has. Samples an interval change cleared from the +history are kept apart, the last 100 per host, until they are delivered. The +history keeps each host's last 100 rounds: 100 seconds at the 1-second interval, +200 seconds at 2 seconds, 5 minutes at the default of 3 seconds and 100 minutes +at 60 seconds. At 1 second, one refused or failed report loses the oldest 20 +seconds of each host's samples, as the next delivered report comes 2 minutes +after the last. At 2 seconds and longer one loses nothing; samples are lost from +the third refused report in a row at 2 seconds, from the fifth at 3 seconds, and +later still at the longer intervals. + +No report is sent while the page is paused, but each paused round still adds an +entry to the history, so each report due during a pause counts as a refused one. +A pause longer than the history spans loses the samples in the history recorded +between the last delivered report and the pause: up to a minute's worth for each +host if the report before the pause was delivered. ### Backend @@ -240,17 +258,19 @@ Files are created with `O_EXCL` to prevent overwrites. The reports of a failed write stay in memory, and the next flush, within a minute, tries again to write them. From the failed write until a write succeeds, -`POST /api/v1/reports` refuses every report with 500 and keeps nothing of it, so -the page keeps its samples and sends them again at its next report. Reports -already answered 200 are lost only if the server stops before a write succeeds; -the final flush at stop tries them too, and if it fails, the process exits -non-zero. +`POST /api/v1/reports` refuses every report with 500 and keeps nothing of it. +The page sends the refused samples again in its next report, as long as it still +holds them: at the 1-second interval one refused report already loses some of +those in the history (see [Reporting](#reporting)). Reports already answered 200 +are lost only if the server stops before a write succeeds; the final flush at +stop tries them too, and if it fails, the process exits non-zero. #### Report limits `POST /api/v1/reports` takes reports from anyone who can reach it, without credentials, so it is bounded instead. Each refusal below answers with the same -`{"status":"error"}` body as any other error. +`{"status":"error"}` body as any other error, except nginx's refusal of a body +over the limit in the image. - **JSON only.** A report must be sent with `Content-Type: application/json`, parameters such as `charset` allowed, as the page sends it; any other is @@ -258,6 +278,13 @@ credentials, so it is bounded instead. Each refusal below answers with the same send a `text/plain` POST, or a form, without a CORS preflight; refusing them keeps any site from making its visitors' browsers store reports (see [CORS](#cors)). +- **Body limit.** A report body may be at most 1 MiB, the backend's limit for + any request. Run on its own, the backend refuses a larger body with 413: + before the route when the request declares a length over the limit, otherwise + at the route once reading the body passes it. In the image, nginx refuses it + first, with or without a declared length, as its `client_max_body_size`, which + `configs/nginx.conf` does not set, is 1 MiB by default. nginx answers 413 with + its own HTML error page, and the backend never sees the request. - **Rate limit.** Each client address, resolved through `TRUSTED_PROXIES`, may send `REPORTS_PER_MINUTE` reports a minute; past that it gets 429 with `Retry-After: 60`. The minute slides: reports from the minute before still @@ -281,6 +308,12 @@ credentials, so it is bounded instead. Each refusal below answers with the same how much of the newest reports is kept: the default of 1 GiB is small enough for any host; set it to the space you can give `DATA_DIR`. +At the defaults, one address may send 60 reports of up to 1 MiB a minute, and a +report made up to compress badly takes about as much room in the report files as +its request body, so in about 17 minutes one address can fill the 1 GiB cap with +its own reports, every report stored before them deleted to make room. Lowering +`REPORTS_PER_MINUTE` or raising `DATA_DIR_MAX_BYTES` makes that take longer. + #### CORS The page calls the API from the origin it is served from, so by default the @@ -302,12 +335,15 @@ the metrics record its duration and response size, labelled with its path, method and status; they also count those requests in progress, and include Go's runtime and process metrics. No other request is recorded: not those to `/metrics` itself, and not those answered before they reach either route, such -as a CORS preflight, or a request refused with 404 for a path no route has, 405 -for a method its route does not take, or 413 for declaring a body length over -the 1 MiB limit. A report whose body goes over the limit without declaring its -length reaches the route, is answered 413 there, and is recorded with that -status. Clients can make up any number of paths and methods, and each would add -labels to the metrics for as long as the server runs. +as a CORS preflight, or a request refused with 404 for a path no route has or +405 for a method its route does not take. Run on its own, the backend also +answers 413 before the route to a request declaring a body length over the 1 MiB +limit, which is not recorded; a report whose body goes over the limit without +declaring its length reaches the route, is answered 413 there, and is recorded +with that status. In the image neither reaches the backend: nginx answers both +with 413 itself (see [Report limits](#report-limits)), and nothing of them is +recorded. Clients can make up any number of paths and methods, and each would +add labels to the metrics for as long as the server runs. `/metrics` is rate limited, so that its password cannot be guessed quickly: each client address, resolved through `TRUSTED_PROXIES`, may make 60 requests to it a diff --git a/TODO.md b/TODO.md index ec56ac3..28dbb9f 100644 --- a/TODO.md +++ b/TODO.md @@ -21,6 +21,23 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of # Completed Steps +- 2026-10-18: `README.md` says what the page loses while reports are refused or + the page is paused ([#145](https://git.eeqj.de/sneak/netwatch/issues/145)), + how fast one address can replace the stored reports + ([#146](https://git.eeqj.de/sneak/netwatch/issues/146)), and that in the image + nginx answers a body over 1 MiB itself + ([#147](https://git.eeqj.de/sneak/netwatch/issues/147)). A sample in the + history that no report has delivered is sent only as long as the history holds + it: at the 1-second interval one refused or failed report loses the oldest 20 + seconds of each host's samples, and a pause longer than the history loses + every sample between the last delivered report and the pause. Samples an + interval change cleared are kept apart, the last 100 per host, until they are + delivered. At the defaults, one address sending 1 MiB reports made up to + compress badly can fill the 1 GiB cap in about 17 minutes. nginx's + `client_max_body_size` is also 1 MiB by default, so in the image it answers a + larger body with 413 and its own HTML page, and the backend never sees or + records it; the backend's own 413 sentences now say they apply to it run on + its own. The reporter, the body limit and nginx are unchanged - 2026-10-18: a browser that cannot format the header's clocks still probes ([#149](https://git.eeqj.de/sneak/netwatch/issues/149)). The clocks name the local time zone through `Intl.DateTimeFormat`'s `shortGeneric` option, which