sneak/pixa
1
0
Fork 0
Code Issues 51 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
63fbc98e63874b947cd2b591afe74fad88ed0e33
pixa/TODO.md
clawbot 63fbc98e63
Some checks failed
check / check (push) Has been cancelled
Details
feat: cache size management and LRU eviction (closes #51) (#55) 
Implements #51 per the issue DoD and the owner direction comment (issuecomment-44068).

## Behavior

**Config: `cache_max_bytes`** (integrates with the #52/#53 validation framework)

- Strict int64 parsing via a new `getInt64`/`int64Val` getter in the existing strict-loader pattern; the key is registered in the known-keys list. A SET but invalid value — negative, float, null, non-numeric string, boolean, list — aborts startup with exit 1 naming the key and the offending value.
- Explicit values are used exactly as given, any non-negative amount, no floor. `cache_max_bytes: 0` is a valid value that disables the disk cache entirely.
- Omitted: after `state_dir` validation, the default resolves to `max(75% of free bytes on the filesystem containing <state_dir>/cache/, 500 MiB)`. The cache directory is created first and statfs runs on that actual path, so the measurement hits the right filesystem. The probe is injectable (`FreeSpaceProbeFunc`) so tests do not depend on the host disk. The effective limit (and disabled state) is logged at startup.

**Size accounting** (no directory scans on the hot path)

- Migration `002` adds a `variant_content` table — processed variants were previously untracked anywhere — and a `last_accessed_at` column on `source_content`, both indexed. Total usage is two SUM queries.
- Stores record accounting rows; cache hits touch the LRU timestamps (same cost class as the existing per-request stats UPDATEs). The variant accounting insert is best-effort with a warning: the reconciliation pass (below) adopts any file that missed its row, and this keeps the pre-migration inline test schema working.

**Eviction policy: global LRU across both content classes**

- Candidates are the least-recently-used entries from `variant_content` and `source_content` (batched, 100 per class per pass, merged oldest-first by `COALESCE(last_accessed_at, fetched_at)`), evicted until usage is at or below the limit.
- Why global LRU: recency of actual use is the best cheap predictor of future use for a CDN-style cache, and treating both classes in one ordering avoids pathologies of class-priority schemes (e.g. evicting every variant before any cold source blob, which would tank hit rate, or the reverse, which would hoard stale sources). Byte-for-byte, the coldest data goes first regardless of what kind it is. LFU-style schemes need more bookkeeping for marginal gain at this scale.
- Reference safety (the multi-reference DoD case): evicting a source blob deletes ALL `source_metadata` rows referencing it plus its `source_content` row in a single transaction BEFORE the file is unlinked. A blob referenced by multiple source paths is only ever removed together with all of its references, and DB rows never point at deleted files (the crash window leaves at worst an orphaned file, which reconciliation sweeps). The JSON metadata sidecars for removed rows are deleted as well.

**Triggers, off the request path**

- A background goroutine (started in the handlers OnStart hook, stopped in OnStop) runs an eviction pass on a periodic ticker (5 min) and on write pressure: every store sends a non-blocking notification on a capacity-1 channel. Requests never wait on eviction.
- On startup the goroutine first reconciles accounting with the disk (off the hot path): adopts untracked variant files (size/mtime from disk, content type from the `.meta` sidecar), drops accounting rows whose files are missing, removes source blob files the DB does not know (unreachable, since lookups go through `source_metadata`), removes rows whose files are gone, and sweeps `.tmp-*` files older than an hour.

**`cache_max_bytes: 0` disables the disk cache**

- No cache directories are created, lookups always miss, `StoreSource`/`StoreVariant` are no-ops, no evictor runs; every request fetches and processes uncached. Verified end-to-end (below).

## Notes for review

- At the `imgcache.CacheConfig` layer, disabling is an explicit `DisableDiskCache` flag rather than `MaxBytes == 0`, because existing test fixtures construct `CacheConfig` without `MaxBytes` and rely on the legacy "no limit" behavior; per repo rules those tests were not touched. The config layer maps `cache_max_bytes: 0` to the flag in `handlers`. `MaxBytes == 0` at that layer means "no limit enforced" and is unreachable from production config (the computed default is always at least 500 MiB).
- The negative cache stays active in disabled mode: it is DB-backed (TTL-expired rows in SQLite), not part of the disk cache this issue bounds, and it protects against hammering failing upstreams. Flagging explicitly since the direction said "no cache reads, no cache writes" — I read that as the disk cache; happy to disable it too if intended.
- One deviation from pure red/green: after the red commit I extended the new-test fixture helper (`newEvictionTestCache`) to pass `DisableDiskCache: maxBytes == 0`, mirroring the production mapping, when the flag design emerged. Assertions were not touched; no pre-existing tests were modified.
- Sidecar files (`.meta`, metadata JSON) are not counted in usage; they are bounded by entry counts and small (tens of bytes to ~1 KiB per entry) while content bytes dominate. Documented here for transparency.
- Discovered while working: `Cache.Stats` reads the never-populated `output_content`/`request_cache` tables, so `TotalItems`/`TotalSizeBytes` are always 0. Out of scope here; filing as a separate issue.

## Verification

- TDD: commit `3963ec3` adds the failing tests first (18 new tests covering strict parsing, default computation with injected probe including floor and 75% branches, explicit-no-floor, zero-disables, size accounting, dedup accounting, LRU order, multi-reference blob eviction with the no-dangling-references invariant, under-limit no-op, write-pressure trigger, periodic trigger, reconciliation); implementation follows in `8cb09b6`/`bdd86a4` until green.
- `make check` green (all tests, lint 0 issues, fmt-check) at HEAD.
- Pinned CI lint gate: `docker build --target lint .` green (golangci-lint v2.10.1).
- End-to-end with the built binary:
  - omitted key: startup logs `computed default cache size limit from free space` and `effective cache size limit` (75% of the test host's free space);
  - `cache_max_bytes: banana`: exit 1 with `config key "cache_max_bytes": value "banana" is not an integer`;
  - `cache_max_bytes: 0`: `cache_disabled=true` logged, two identical requests both fetch upstream (2 upstream fetches logged), 200 `image/jpeg` responses, no `cache/` directory created, only `state.sqlite3` in the state dir;
  - enabled: second request served from cache (1 upstream fetch), `variant_content` and `source_content` rows match the on-disk file sizes.

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #55
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-09 13:22:51 +02:00

5.8 KiB
Raw Blame History

Workflow

  • branch (from main)
  • do the work in Next Step
  • move Next Step to the top of Completed Steps
  • move the top item of Future Steps into Next Step
  • commit (TODO.md changes in the same commit as the work)
  • merge to main if the branch is not protected, otherwise open a PR
  • push

Status

pre-1.0. No git tags exist. Recent work extracted the internal/magic, internal/allowlist, internal/httpfetcher, and internal/signature packages. The gosec findings from the 2026-07-06 survey are resolved and make check is green on main. The disk cache is now size-bounded with LRU eviction (cache_max_bytes), closing the unbounded disk growth DoS vector.

Next Step

P1: implement blocked networks configuration to extend SSRF protection

Completed Steps

  • 2026-08-07 implement cache size management and eviction (closes #51): new cache_max_bytes config key validated by the startup framework (explicit values used exactly with no floor, 0 disables the disk cache entirely, omitted defaults to max(75% of free space on the filesystem containing <state_dir>/cache/, 500 MiB), logged at startup); processed variants are now tracked in the database (a new variant_content table and an LRU timestamp on source_content) so total usage is two SUMs, never a directory scan on the hot path; a background goroutine evicts globally least-recently-used entries (variants and source blobs merged) to the limit, woken by a periodic ticker and by write-pressure notifications from stores; a source blob and ALL of its source_metadata references are deleted in one transaction before the file is unlinked, so multi-referenced blobs are never removed while referenced and rows never point at deleted files; a startup and periodic reconciliation pass adopts untracked variant files, drops rows for missing files, removes unreachable source blobs, and sweeps stale temp files
  • 2026-08-07 validate configuration on startup, fail fast on bad config (closes #52): a config value that is set but unparseable or invalid aborts startup naming the key and value (defaults apply only to omitted keys), unknown config keys abort startup, a malformed config file aborts instead of being skipped, and state_dir is verified creatable and writable before the listener binds
  • 2026-08-07 manual test pass of the auth and encrypted URL flows against a locally built and running pixad (built from main at 6573b9d, port 18099, local throwaway config); all six checks passed, plus all nine tests in scripts/manual-test.sh (closes #49):
    • visit / and see the login form: HTTP 200, Pixa - Login page with name="key" password form
    • wrong key shows an error: POST / with key=wrong-key returned HTTP 200 login page containing "Invalid signing key"
    • correct signing key shows the generator form: POST / returned HTTP 303 to / with Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict; GET / with that cookie rendered Pixa - URL Generator with the /generate form and logout link
    • a generated encrypted URL serves the image: POST /generate (ttl=3600) produced a /v1/e/<token>/img.jpeg URL that returned HTTP 200, Content-Type: image/jpeg, an 800x600 baseline JPEG of 61706 bytes
    • an expired URL (short TTL) returns 410: a ttl=1 URL fetched after 3 s returned HTTP 410 Gone with {"error":"URL has expired","status":410,...}
    • logout redirects back to login: GET /logout returned HTTP 303 to / with Set-Cookie: pixa_session=; Max-Age=0; subsequent GET / rendered the login form again
  • 2026-08-07 fix the two remaining gosec findings (G124 in internal/session): session cookies now always carry Secure/HttpOnly/SameSite=Strict on both the set and clear paths; make check green (closes #47)
  • 2026-07-07 Adopted scripts-to-rule-them-all: script/ entrypoints, Makefile shims, README Entrypoints section
  • 2026-04-07 extract magic byte detection into internal/magic (#42)
  • 2026-03-25 extract allowlist package from internal/imgcache (#41)
  • 2026-03-25 move schema_migrations table creation into 000.sql (#36)
  • 2026-03-20 enforce and document exact-match-only signature verification (#40)
  • 2026-03-20 bound imageprocessor.Process input read to prevent unbounded memory use (#37); consolidate appname into an internal/globals constant (#34)
  • 2026-03-18 parse version prefix from migration filenames (#33)
  • 2026-03-15 QA audit fixes for 1.0/MVP readiness (#25)
  • 2026-03-02 split Dockerfile with pre-built golangci-lint stage for faster CI (#23)
  • 2026-02-25 repo policy compliance: CI workflow, hash-pinned images, golangci-lint and gosec fixes of that date (#14); arm64 Docker build fix (#16)
  • 2026-01-08 WebP and AVIF encoding support via govips (both former P0 image processing items, now done)

Future Steps

  • P1: rate limit global concurrent upstream fetches to prevent resource exhaustion
  • P1: strip EXIF and other metadata from processed images (privacy)
  • P2: security
    • referer blacklist
    • per-IP rate limiting
    • per-origin rate limiting
  • P2: HTTP response handling
    • Last-Modified headers
    • Vary header for content negotiation
    • X-Request-ID propagation
  • P2: auto format selection (format=auto based on Accept header)
  • P2: configuration
    • add all configuration options from README
    • environment variable overrides
    • YAML config file support
  • P2: operational
    • optional Sentry error reporting
    • comprehensive request logging
    • Prometheus performance metrics
    • integration tests for the image proxy flow
    • load tests to verify the 1k to 5k req/s target
  • P2: documentation
    • configuration options
    • API endpoints
    • deployment guide
    • example nginx or caddy reverse proxy config
Reference in New Issue View Git Blame Copy Permalink