diff --git a/README.md b/README.md index cc1a877..8e36dc0 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,9 @@ Configured via YAML file (`--config`). Key settings: - `upstream_max_response_size` — max origin response size - `downstream_timeout` — client response timeout - `signing_key` — HMAC secret for URL signatures +- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the + disk cache entirely; omitted defaults to 75% of the free space on + the filesystem containing `/cache/` (minimum 500 MiB) See `config.example.yml` for all options with defaults. diff --git a/TODO.md b/TODO.md index 5c291cc..250904e 100644 --- a/TODO.md +++ b/TODO.md @@ -12,18 +12,35 @@ pre-1.0. No git tags exist. Recent work extracted the internal/magic, internal/allowlist, internal/httpfetcher, and internal/signature -packages. The gosec findings from the 2026-07-06 survey are resolved: -the last two open findings (G124, session cookie attributes in -internal/session) are fixed as of this change, so `make check` is green -on main. +packages. The gosec findings from the 2026-07-06 survey are resolved +and `make check` is green on main. The disk cache is now size-bounded +with LRU eviction (`cache_max_bytes`), closing the unbounded disk +growth DoS vector. # Next Step -P0: implement cache size management and eviction so the disk cannot -fill up +P1: implement blocked networks configuration to extend SSRF protection # Completed Steps +- 2026-08-07 implement cache size management and eviction (closes + #51): new `cache_max_bytes` config key validated by the startup + framework (explicit values used exactly with no floor, `0` disables + the disk cache entirely, omitted defaults to max(75% of free space + on the filesystem containing `/cache/`, 500 MiB), logged + at startup); processed variants are now tracked in the database + (migration 002 adds `variant_content` and an LRU timestamp on + `source_content`) so total usage is two SUMs, never a directory scan + on the hot path; a background goroutine evicts globally + least-recently-used entries (variants and source blobs merged) to + the limit, woken by a periodic ticker and by write-pressure + notifications from stores; a source blob and ALL of its + `source_metadata` references are deleted in one transaction before + the file is unlinked, so multi-referenced blobs are never removed + while referenced and rows never point at deleted files; a startup + reconciliation pass adopts untracked variant files, drops rows for + missing files, removes unreachable source blobs, and sweeps stale + temp files - 2026-08-07 validate configuration on startup, fail fast on bad config (closes #52): a config value that is set but unparseable or invalid aborts startup naming the key and value (defaults apply only @@ -79,8 +96,6 @@ fill up # Future Steps -- P1: implement blocked networks configuration to extend SSRF - protection - P1: rate limit global concurrent upstream fetches to prevent resource exhaustion - P1: strip EXIF and other metadata from processed images (privacy) diff --git a/config.example.yml b/config.example.yml index 900a0b6..122189e 100644 --- a/config.example.yml +++ b/config.example.yml @@ -28,6 +28,13 @@ allow_http: false # Maximum concurrent connections per upstream host (default: 20) upstream_connections_per_host: 20 +# Maximum disk cache size in bytes. Explicit values are used exactly as +# given; 0 disables the disk cache entirely (every request fetches and +# processes uncached). When omitted, the default is 75% of the free +# space on the filesystem containing /cache/ at startup, +# with a minimum of 500 MiB. +# cache_max_bytes: 10737418240 + # Sentry error reporting (optional) sentry_dsn: ""