Delete the oldest report files to stay under the size cap (closes #54)
check / check (push) Successful in 1m53s
check / check (push) Successful in 1m53s
When a report would take the report files past DATA_DIR_MAX_BYTES, reportbuf now deletes the oldest report files until it fits, and does the same at start when files left by an earlier run are already past it. The buffer keeps the files it may delete in a list, oldest first; a file joins it only once it is completely written, so a file still being written is never deleted. A report is refused with 507 only when the reports not yet written leave no room for it on their own, and then no file is deleted. A file whose deletion fails keeps counting; one already deleted by hand counts as freed. Model: opus-5-5
This commit is contained in:
+10
-7
@@ -80,7 +80,7 @@ Internal packages in `internal/` follow standard Go project layout:
|
||||
| `BIND_ADDRESS` | empty | IP address to listen on; empty listens on every interface |
|
||||
| `PORT` | `8080` | HTTP listen port |
|
||||
| `DATA_DIR` | `./data/reports` | Directory for compressed reports |
|
||||
| `DATA_DIR_MAX_BYTES` | `1073741824` (1 GiB) | Largest total size of the report files in `DATA_DIR`; see [Report limits](#report-limits) |
|
||||
| `DATA_DIR_MAX_BYTES` | `1073741824` (1 GiB) | Most bytes of report files kept in `DATA_DIR`, oldest deleted first; see [Report limits](#report-limits) |
|
||||
| `DEBUG` | `false` | Enable debug logging |
|
||||
| `TRUSTED_PROXIES` | loopback + RFC1918 | Comma-separated CIDRs whose `X-Forwarded-For` / `X-Real-IP` headers are trusted for client IP resolution |
|
||||
| `REPORTS_PER_MINUTE` | `60` | Reports each client address may send a minute; see [Report limits](#report-limits) |
|
||||
@@ -148,11 +148,15 @@ credentials, so it is bounded instead. Both refusals below answer with the same
|
||||
`X-RateLimit-Reset` headers.
|
||||
- **Size cap.** The report files in `DATA_DIR` may total at most
|
||||
`DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports
|
||||
waiting in memory count at their uncompressed size until they are written, so
|
||||
a report that would take the total past the cap is refused with 507, and
|
||||
nothing of it is stored. Deleting report files frees room only at the next
|
||||
start, when the files are counted again. The default of 1 GiB is small enough
|
||||
for any host; set it to the space you can give `DATA_DIR`.
|
||||
waiting in memory count at their uncompressed size until they are written.
|
||||
When a report would take the total past the cap, the oldest report files are
|
||||
deleted to make room, and each deletion is logged with the file's name and
|
||||
size; a file still being written is never deleted. A report is refused with
|
||||
507, and nothing of it is stored, only when the reports not yet written leave
|
||||
no room for it on their own, and then no file is deleted. At start, report
|
||||
files past the cap, as after lowering it, are deleted the same way. So the cap
|
||||
is how much of the newest reports is kept: the default of 1 GiB is small
|
||||
enough for any host; set it to the space you can give `DATA_DIR`.
|
||||
|
||||
### CORS
|
||||
|
||||
@@ -170,7 +174,6 @@ starting, with an error naming `CORS_ALLOWED_ORIGINS`.
|
||||
- Add integration test that POSTs a report and verifies the compressed output
|
||||
- Add report decompression/query endpoint
|
||||
- Add metrics (Prometheus) for buffer size, flush count, report count
|
||||
- Add retention policy to prune old report files
|
||||
|
||||
## License
|
||||
|
||||
|
||||
Reference in New Issue
Block a user