clawbot
63fbc98e63
Some checks failed
check / check (push) Has been cancelled
Implements #51 per the issue DoD and the owner direction comment (issuecomment-44068). ## Behavior **Config: `cache_max_bytes`** (integrates with the #52/#53 validation framework) - Strict int64 parsing via a new `getInt64`/`int64Val` getter in the existing strict-loader pattern; the key is registered in the known-keys list. A SET but invalid value — negative, float, null, non-numeric string, boolean, list — aborts startup with exit 1 naming the key and the offending value. - Explicit values are used exactly as given, any non-negative amount, no floor. `cache_max_bytes: 0` is a valid value that disables the disk cache entirely. - Omitted: after `state_dir` validation, the default resolves to `max(75% of free bytes on the filesystem containing <state_dir>/cache/, 500 MiB)`. The cache directory is created first and statfs runs on that actual path, so the measurement hits the right filesystem. The probe is injectable (`FreeSpaceProbeFunc`) so tests do not depend on the host disk. The effective limit (and disabled state) is logged at startup. **Size accounting** (no directory scans on the hot path) - Migration `002` adds a `variant_content` table — processed variants were previously untracked anywhere — and a `last_accessed_at` column on `source_content`, both indexed. Total usage is two SUM queries. - Stores record accounting rows; cache hits touch the LRU timestamps (same cost class as the existing per-request stats UPDATEs). The variant accounting insert is best-effort with a warning: the reconciliation pass (below) adopts any file that missed its row, and this keeps the pre-migration inline test schema working. **Eviction policy: global LRU across both content classes** - Candidates are the least-recently-used entries from `variant_content` and `source_content` (batched, 100 per class per pass, merged oldest-first by `COALESCE(last_accessed_at, fetched_at)`), evicted until usage is at or below the limit. - Why global LRU: recency of actual use is the best cheap predictor of future use for a CDN-style cache, and treating both classes in one ordering avoids pathologies of class-priority schemes (e.g. evicting every variant before any cold source blob, which would tank hit rate, or the reverse, which would hoard stale sources). Byte-for-byte, the coldest data goes first regardless of what kind it is. LFU-style schemes need more bookkeeping for marginal gain at this scale. - Reference safety (the multi-reference DoD case): evicting a source blob deletes ALL `source_metadata` rows referencing it plus its `source_content` row in a single transaction BEFORE the file is unlinked. A blob referenced by multiple source paths is only ever removed together with all of its references, and DB rows never point at deleted files (the crash window leaves at worst an orphaned file, which reconciliation sweeps). The JSON metadata sidecars for removed rows are deleted as well. **Triggers, off the request path** - A background goroutine (started in the handlers OnStart hook, stopped in OnStop) runs an eviction pass on a periodic ticker (5 min) and on write pressure: every store sends a non-blocking notification on a capacity-1 channel. Requests never wait on eviction. - On startup the goroutine first reconciles accounting with the disk (off the hot path): adopts untracked variant files (size/mtime from disk, content type from the `.meta` sidecar), drops accounting rows whose files are missing, removes source blob files the DB does not know (unreachable, since lookups go through `source_metadata`), removes rows whose files are gone, and sweeps `.tmp-*` files older than an hour. **`cache_max_bytes: 0` disables the disk cache** - No cache directories are created, lookups always miss, `StoreSource`/`StoreVariant` are no-ops, no evictor runs; every request fetches and processes uncached. Verified end-to-end (below). ## Notes for review - At the `imgcache.CacheConfig` layer, disabling is an explicit `DisableDiskCache` flag rather than `MaxBytes == 0`, because existing test fixtures construct `CacheConfig` without `MaxBytes` and rely on the legacy "no limit" behavior; per repo rules those tests were not touched. The config layer maps `cache_max_bytes: 0` to the flag in `handlers`. `MaxBytes == 0` at that layer means "no limit enforced" and is unreachable from production config (the computed default is always at least 500 MiB). - The negative cache stays active in disabled mode: it is DB-backed (TTL-expired rows in SQLite), not part of the disk cache this issue bounds, and it protects against hammering failing upstreams. Flagging explicitly since the direction said "no cache reads, no cache writes" — I read that as the disk cache; happy to disable it too if intended. - One deviation from pure red/green: after the red commit I extended the new-test fixture helper (`newEvictionTestCache`) to pass `DisableDiskCache: maxBytes == 0`, mirroring the production mapping, when the flag design emerged. Assertions were not touched; no pre-existing tests were modified. - Sidecar files (`.meta`, metadata JSON) are not counted in usage; they are bounded by entry counts and small (tens of bytes to ~1 KiB per entry) while content bytes dominate. Documented here for transparency. - Discovered while working: `Cache.Stats` reads the never-populated `output_content`/`request_cache` tables, so `TotalItems`/`TotalSizeBytes` are always 0. Out of scope here; filing as a separate issue. ## Verification - TDD: commit `3963ec3` adds the failing tests first (18 new tests covering strict parsing, default computation with injected probe including floor and 75% branches, explicit-no-floor, zero-disables, size accounting, dedup accounting, LRU order, multi-reference blob eviction with the no-dangling-references invariant, under-limit no-op, write-pressure trigger, periodic trigger, reconciliation); implementation follows in `8cb09b6`/`bdd86a4` until green. - `make check` green (all tests, lint 0 issues, fmt-check) at HEAD. - Pinned CI lint gate: `docker build --target lint .` green (golangci-lint v2.10.1). - End-to-end with the built binary: - omitted key: startup logs `computed default cache size limit from free space` and `effective cache size limit` (75% of the test host's free space); - `cache_max_bytes: banana`: exit 1 with `config key "cache_max_bytes": value "banana" is not an integer`; - `cache_max_bytes: 0`: `cache_disabled=true` logged, two identical requests both fetch upstream (2 upstream fetches logged), 200 `image/jpeg` responses, no `cache/` directory created, only `state.sqlite3` in the state dir; - enabled: second request served from cache (1 upstream fetch), `variant_content` and `source_content` rows match the on-disk file sizes. Co-authored-by: sneak <sneak@sneak.berlin> Reviewed-on: #55 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org>
pixa
pixa is a GPL-3.0-licensed Go web server by @sneak that proxies images from upstream sources, optionally resizing or transforming them, and serves the results. Both source and transformed images are cached to disk so that subsequent requests are served without origin fetches or additional processing.
Getting Started
# clone and build
git clone https://git.eeqj.de/sneak/pixa.git
cd pixa
make build
# run with a config file
./bin/pixad --config config.example.yml
# or build and run via Docker
make docker
docker run -p 8080:8080 pixad:latest
Rationale
Image-heavy web applications need a fast, caching reverse proxy that can resize and transcode images on the fly. pixa fills that role as a single, self-contained binary with no external runtime dependencies beyond libvips. It supports HMAC-SHA256 signed URLs with expiration to prevent abuse, and allowlisted source hosts for open access.
Design
Storage
- Source content:
<statedir>/cache/src-content/<ab>/<cd>/<sha256 of source content> - Source metadata:
<statedir>/cache/src-metadata/<hostname>/<sha256 of path>.json(fetch time, original headers, request, content hash) - Database:
<statedir>/state.sqlite3(SQLite) - Output documents:
<statedir>/cache/dst-content/<ab>/<cd>/<sha256 of output content>
Multiple source paths may reference the same content blob; the database tracks references rather than using filesystem refcounting. In-process caching of request-to-output mappings targets 1-5k r/s.
Routes
/v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>
Images are only fetched from origins using TLS with valid certificates.
<format>: one oforig,png,jpeg,webp<size>:origor<width>x<height>(e.g.800x600)
Source Hosts
Source hosts may be allowlisted in the configuration. Non-allowlisted hosts require an HMAC-SHA256 signature.
Signature Specification
Signatures use HMAC-SHA256 and include an expiration timestamp to prevent replay attacks. Signatures are exact match only: every component (host, path, query, dimensions, format, expiration) must match exactly what was signed. No suffix matching, wildcard matching, or partial matching is supported.
Signed data format (colon-separated):
HMAC-SHA256(secret, "host:path:query:width:height:format:expiration")
Where:
host— source origin hostname (e.g.cdn.example.com)path— source path (e.g./photos/cat.jpg)query— source query string, empty string if nonewidth— requested width in pixels,0for originalheight— requested height in pixels,0for originalformat— output format (jpeg, png, webp, avif, gif, orig)expiration— Unix timestamp when signature expires
Example: resize
https://cdn.example.com/photos/cat.jpg to 800x600 WebP with
expiration 1704067200:
- Build input:
cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200 - Compute HMAC-SHA256 with your secret key
- Base64URL-encode the result
- URL:
/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=<base64url>&exp=1704067200
Allowlist patterns:
- Exact match:
cdn.example.com— matches only that host - Suffix match:
.example.com— matchescdn.example.com,images.example.com, andexample.com
Configuration
Configured via YAML file (--config). Key settings:
access_control_allow_origin— CORS originallowlist_hosts— list of allowed upstream hostsupstream_fetch_timeout— timeout for origin requestsupstream_max_response_size— max origin response sizedownstream_timeout— client response timeoutsigning_key— HMAC secret for URL signaturescache_max_bytes— disk cache size limit in bytes;0disables the disk cache entirely; omitted defaults to 75% of the free space on the filesystem containing<state_dir>/cache/(minimum 500 MiB)
See config.example.yml for all options with defaults.
Architecture
- Dependency injection: Uber fx
- HTTP router: go-chi
- Image processing: govips (CGO wrapper for libvips)
- Database: SQLite via modernc.org/sqlite
- Static assets: embedded via
//go:embed - Metrics: Prometheus
- Logging: stdlib slog
Entrypoints
This repository adheres to the
Scripts to Rule Them All
standard: normalized scripts in script/ are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
script/bootstrap— install all dependencies (idempotent)script/setup— make a fresh clone ready for development (bootstrap, then install-precommit)script/projectname— output the project name ("pixa")script/test— run the test suitescript/lint— run golangci-lintscript/fmt— format all code (writes)script/fmt-check— check formatting (read-only)script/check— run test, lint, and fmt-checkscript/docker— build the Docker image tagged viascript/projectnamescript/cibuild— CI entrypoint:docker build .(the Dockerfile runs the checks, so a green build implies a green repo)script/precommit— pre-commit checks (go mod tidyguard, thenscript/check)script/install-precommit— install the git pre-commit hook that runsscript/precommit
TODO
See TODO.md for the full prioritized task list.
License
GPL-3.0. See LICENSE.