Name pixa's concepts at the start of the README's Design section (closes #321)
check / check (push) Waiting to run

The Design section starts with a Concepts list. Each concept gets one
sentence, under the name the code gives it. Storage calls the
transformed images variants, and the sentences it changed for that are
shorter. Its paths end in the source key and the variant key, and it
lists the image cache's tables in the database. Architecture names
internal/providers and lists the packages that work with the concepts.
No code or test changes.

Model: opus-5-5
This commit was merged in pull request #338.
This commit is contained in:
2026-10-26 23:19:19 +01:00
parent 93b13d6638
commit 6de69f7349
2 changed files with 101 additions and 34 deletions
+94 -34
View File
@@ -178,47 +178,91 @@ allowlisted source hosts for open access.
## Design
### Concepts
These terms are the names the code uses:
- **Source**: the image at one upstream URL, named by its host, path and query.
- **Source key**: the SHA-256 of a source's host, path and query, which names
the source's file and its row in the database.
- **Transform**: the output size, format, quality and fit an image request asks
for; with the format `auto`, pixa chooses the output format from the `Accept`
header.
- **Image request**: a source and a transform, which each image route reads from
its URL.
- **Variant**: the image pixa serves for an image request, made from its source
with its transform.
- **Variant key**: the SHA-256 of a variant's source, size, format, quality and
fit, which names the variant's file and its row in the database, and gives its
`ETag`.
- **Signed URL**: a `/v1/image/` URL whose `sig` signs its image request and its
expiry `exp` with `signing_key`.
- **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any
expiry, encrypted with a key derived from `signing_key`.
- **Login session**: an encrypted cookie that keeps a browser logged in to the
generator page for 30 days, or until a logout replaces the random value pixa
keeps in the `logout` table (see Encrypted URLs).
- **Negative cache**: the source URLs pixa does not fetch again for 5 minutes,
after one of these failed fetches:
- the upstream answered with an error status
- the body's first bytes were not those of the format its `Content-Type`
names
- **Image cache**: the part of pixa that keeps the sources, the variants and the
negative cache (see Storage).
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
sources and variants may reach together.
- **Image proxy**: the part of pixa that serves a variant from the image cache
when it can, and otherwise makes it from the source.
### Storage
- **Source images**:
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one
file for each source URL
- **Sources**: `<state_dir>/cache/sources/<ab>/<cd>/<source key>`, one file for
each source URL
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
- **Transformed images**:
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`
- **Variants**: `<state_dir>/cache/variants/<ab>/<cd>/<variant key>`
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
name.
Each source image and each transformed image has one row in the database,
holding its size and when it was last used. pixa reads a cached source image's
format from its file's first bytes. When they match none of the source formats,
pixa fetches the source again. Two source URLs that serve the same bytes are
stored twice, once under each URL. A source image is stored by writing its file
and then its row; when the row cannot be written, the file is removed again.
The image cache keeps these tables in the database:
Source and transformed image files are written under a temporary name, synced to
disk, and only then renamed into place, so a crash leaves no file under its own
name with part of its data. When pixa starts, and every five minutes after, it
removes each cached source or transformed image whose file is empty, or not the
size the database records for it, with a warning naming the file; the next
request for it fetches the source or converts the image again. Looking for those
files reads the database once for every cached file, so pixa skips each
five-minute look that would start less than ten times as long after the last
look ended as that look took. After a look that took one minute, the next one
starts 10 to 15 minutes after that look ended.
- `source_content`: a row for each source, by source key, with its size and when
it was last used
- `variant_content`: a row for each variant, by variant key (column
`cache_key`), with its size and when it was last used
- `cache_usage`: the total size of the cached sources and variants, which
eviction reads
- `negative_cache`: a row for each source URL in the negative cache, with when
it expires
pixa reads a cached source image's format from its file's first bytes. When they
match none of the source formats, pixa fetches the source again. Two source URLs
that serve the same bytes are stored twice, once under each URL. A source image
is stored by writing its file and then its row; when the row cannot be written,
the file is removed again.
Each source and variant file is written under a temporary name and synced to
disk before it is renamed into place. So a crash never leaves part of a file
under its own name. When pixa starts, and every five minutes after, it removes
each source or variant whose file is empty, or not the size its row records. It
logs a warning naming the file. A later request for that image fetches the
source or makes the variant again. Looking for those files reads the database
once for every cached file, so pixa skips each five-minute look that would start
less than ten times as long after the last look ended as that look took. After a
look that took one minute, the next one starts 10 to 15 minutes after that look
ended.
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
Test). Toward it, a cache hit opens the transformed image's file and nothing
else, and makes no database call. The content type comes from the file's first
bytes, which are those of the format pixa wrote it in; a file whose first bytes
are not those of a format pixa writes is not served, and the image is converted
again. A hit hands the time it was served to a background writer and does not
wait for the writer. Eviction orders by that time. The hit is counted in memory.
A miss that uses a cached source image also hands that writer the time it used
the image. Once a second that writer writes the waiting times in one
transaction. While 10,000 times of transformed images, or 10,000 of source
images, wait to be written, a further one is dropped, and that image only looks
Test). Toward it, a cache hit opens the variant's file and nothing else, and
makes no database call. The content type comes from the file's first bytes,
which are those of the format pixa wrote it in; a file whose first bytes are not
those of a format pixa writes is not served, and the image is converted again. A
hit hands the time it was served to a background writer and does not wait for
the writer. Eviction orders by that time. The hit is counted in memory. A miss
that uses a cached source image also hands that writer the time it used the
image. Once a second that writer writes the waiting times in one transaction. Up
to 10,000 access times of variants, and as many of sources, can wait to be
written. A time handed over beyond that is dropped. Its image then only looks
less recently used to eviction. `script/bench` measures how many hits pixa
serves per second (see Load Test).
@@ -880,9 +924,25 @@ See `configs/config.example.yml` for all options with defaults.
### Architecture
- **Dependency injection**: Uber fx. `internal/app` builds every part of pixad
from the config in one fx graph. Each setting's default is declared once, in
`internal/config`.
- **Dependency injection**: Uber fx. `internal/app` puts every part of pixad in
one fx graph. `internal/providers` builds the image cache, the image proxy and
the other parts that serve images and logins. Each setting's default is
declared once, in `internal/config`.
- **Packages**: these packages under `internal/` work with the concepts above:
- `handlers`: the routes, which read the image request from the URL, check
the signed or encrypted URL, and make the login session
- `imagerequest`: the image request, with the source key and the variant key
- `imageproxy`: the image proxy
- `imgcache`: the image cache, which keeps its sources in one `FileStore`
and its variants in another
- `httpfetcher`: fetching sources from upstream hosts
- `magic`: reading an image's format from its first bytes
- `imageprocessor`: making a variant from a source with libvips
- `signature`: checking the `sig` of a signed URL
- `encurl`: making and reading the tokens of encrypted URLs
- `seal`: the keys derived from `signing_key`, and the encryption of those
tokens
- `session`: the login session
- **HTTP router**: go-chi
- **Image processing**: govips (CGO wrapper for libvips)
- **Database**: SQLite via modernc.org/sqlite
+7
View File
@@ -30,6 +30,13 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-26 `README.md`'s Design section starts with a Concepts list (closes
#321). Each concept gets one sentence, under the name the code gives it.
- Storage calls the transformed images variants, in shorter sentences.
- Storage's paths end in the source key and the variant key.
- Storage lists the image cache's tables in the database.
- Architecture names `internal/providers`, and lists the packages that work
with the concepts.
- 2026-10-26 one table in `internal/config` lists every setting once (closes
#320). `Config.settings` gives each setting's key, its environment variable
and the function that loads it.