3 Commits
Author SHA1 Message Date
clawbot d98d4dc119 Test a fetch whose context ends waiting for a shared connection
check / check (push) Successful in 11m43s
A fetch from a host with nothing open waits for the connection shared by
all hosts, and its context ends long before the wait timeout; once every
response is closed, no semaphore may be left in hostSems.

Model: opus-5-5
2026-10-04 03:47:37 +00:00
clawbot c9867d801a Remove idle host semaphores and delete .meta with its variant (closes #87)
Each upstream host's semaphore now counts the fetches holding or waiting
for one of its slots, and is removed from hostSems when the last of them
gives its slot back or stops waiting, so a long-running pixad no longer
keeps one semaphore per host it ever fetched from. The semLen test helper
reads hostSems directly, as getHostSemaphore now counts its caller.

VariantStorage.Delete removes the variant's .meta file too, a missing one
not being an error; DeleteWithMeta, which eviction called for that, is
gone.

Model: opus-5-5
2026-10-04 03:43:09 +00:00
clawbot 96be48f127 Test that idle host semaphores and .meta files are removed
New tests, failing before the fix: fetches from many hosts, and fetches
that end without a connection, must leave no semaphore in hostSems once
they finish; VariantStorage.Delete must remove the variant's .meta file,
and must succeed when that file is already missing.

Model: opus-5-5
2026-10-04 03:42:53 +00:00
3 changed files with 13 additions and 33 deletions
+10 -21
View File
@@ -71,17 +71,13 @@ prevent abuse, and allowlisted source hosts for open access.
### Storage
- **Source content**:
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
`<statedir>/cache/src-content/<ab>/<cd>/<sha256 of source content>`
- **Source metadata**:
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json`
(host, path and query, content hash, upstream status and headers, fetch time)
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
- **Transformed images**:
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
each with a `.meta` file beside it holding its content type
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
name.
`<statedir>/cache/src-metadata/<hostname>/<sha256 of path>.json`
(fetch time, original headers, request, content hash)
- **Database**: `<statedir>/state.sqlite3` (SQLite)
- **Output documents**:
`<statedir>/cache/dst-content/<ab>/<cd>/<sha256 of output content>`
Multiple source paths may reference the same content blob; the
database tracks references rather than using filesystem refcounting.
@@ -96,15 +92,12 @@ the metadata file stored beside it.
/v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>
```
Images are only fetched from origins using TLS with valid certificates, unless
`allow_http` is set: then pixa fetches every image over plain HTTP, which is for
testing only.
Images are only fetched from origins using TLS with valid certificates.
A request whose query string cannot be decoded, or gives any parameter more
than once, is refused with 400.
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
`avif`, `gif`
- `<format>`: one of `orig`, `png`, `jpeg`, `webp`
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
@@ -180,8 +173,7 @@ Where:
- `query` — source query string, empty string if none
- `width` — requested width in pixels, `0` for original
- `height` — requested height in pixels, `0` for original
- `format` — output format, one of those listed under Routes, with `original`
signed as `orig` and `jpg` as `jpeg`
- `format` — output format (jpeg, png, webp, avif, gif, orig)
- `expiration` — the URL's `exp` query parameter, the Unix timestamp when
the signature expires; a request whose `exp` is not a whole number, an
empty `exp=` included, is refused with 400
@@ -338,10 +330,7 @@ See `config.example.yml` for all options with defaults.
- **Image processing**: govips (CGO wrapper for libvips)
- **Database**: SQLite via modernc.org/sqlite
- **Static assets**: embedded via `//go:embed`
- **Metrics**: Prometheus, at `/metrics`: generic HTTP request metrics
(duration, response size, requests in flight) and the Go runtime and process
metrics; requests are measured and `/metrics` is served only when
`metrics.username` and `metrics.password` are set
- **Metrics**: Prometheus
- **Logging**: stdlib slog
## Entrypoints
-8
View File
@@ -36,14 +36,6 @@ P2: security: referer blacklist
semaphore each until restart; `VariantStorage.Delete` removes the variant's
`.meta` file along with it, a missing `.meta` file not being an error, and
`DeleteWithMeta`, which eviction called for that, is gone.
- 2026-10-04 `README.md` matches the code (closes #74): "Storage" names the
cache directories pixa uses (`cache/sources`, `cache/metadata`,
`cache/variants`) and how files are named in each, and the comments in
`001_schema.sql` name the same paths; the routes and the signature section
list the same output formats, `jpg` and `original` included; the TLS
sentence names `allow_http` as its exception; "Metrics" says only generic
HTTP and Go runtime metrics exist, measured and served only when the metrics
username and password are set.
- 2026-10-03 shutdown sets the exit code and waits for image processing
(closes #86): fx alone handles SIGINT and SIGTERM, and the server's own
signal handler is gone; fx's `Run` in `cmd/pixad` exits with the shutdown's
+3 -4
View File
@@ -2,7 +2,7 @@
-- Creates all tables for the pixa caching image proxy
-- Source content blobs
-- Files stored at: cache/sources/<ab>/<cd>/<sha256>
-- Files stored at: cache/src-content/<ab>/<cd>/<sha256>
-- last_accessed_at is NULL until the first LRU touch; eviction falls
-- back to fetched_at for rows that have never been touched.
CREATE TABLE IF NOT EXISTS source_content (
@@ -16,7 +16,7 @@ CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
ON source_content(last_accessed_at);
-- Source URL metadata - maps URLs to content hashes
-- JSON stored at: cache/metadata/<hostname>/<path_hash>.json
-- JSON stored at: cache/src-metadata/<hostname>/<path_hash>.json
CREATE TABLE IF NOT EXISTS source_metadata (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_host TEXT NOT NULL,
@@ -56,8 +56,7 @@ CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed
ON variant_content(last_accessed_at);
-- Output/transformed content blobs
-- Not written: transformed images are stored in cache/variants and
-- tracked in variant_content above.
-- Files stored at: cache/dst-content/<ab>/<cd>/<sha256>
CREATE TABLE IF NOT EXISTS output_content (
content_hash TEXT PRIMARY KEY,
content_type TEXT NOT NULL,