diff --git a/README.md b/README.md index b748270..8628e67 100644 --- a/README.md +++ b/README.md @@ -178,47 +178,86 @@ allowlisted source hosts for open access. ## Design +### Concepts + +These terms are the names the code uses: + +- **Source**: the image at one upstream URL, named by its host, path and query. +- **Source key**: the SHA-256 of a source's host, path and query, which names + the source's file and its row in the database. +- **Transform**: the output size and format an image request asks for, with its + quality and fit. +- **Image request**: a source and a transform, which each image route reads from + its URL. +- **Variant**: the image pixa serves for an image request, made from its source + with its transform. +- **Variant key**: the SHA-256 of an image request, which names the variant's + file and its row in the database, and gives its `ETag`. +- **Signed URL**: a `/v1/image/` URL whose `sig` signs its image request and its + expiry `exp` with `signing_key`. +- **Encrypted URL**: a `/v1/e/` URL whose token holds an image request and its + expiry, encrypted with a key derived from `signing_key`. +- **Negative cache**: the source URLs pixa does not fetch again for 5 minutes, + after one of these failed fetches: + - the upstream answered with an error status + - the body's first bytes were not those of the format its `Content-Type` + names +- **Image cache**: the part of pixa that keeps the sources, the variants and the + negative cache (see Storage). +- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached + sources and variants may reach together. +- **Image proxy**: the part of pixa that serves a variant from the image cache + when it can, and otherwise makes it from the source. + ### Storage -- **Source images**: - `/cache/sources///`, one - file for each source URL +- **Sources**: `/cache/sources///`, one file for + each source URL - **Database**: `/state.sqlite3` (SQLite) -- **Transformed images**: - `/cache/variants///` +- **Variants**: `/cache/variants///` `` and `` are the first and second pairs of characters of the file's name. -Each source image and each transformed image has one row in the database, -holding its size and when it was last used. pixa reads a cached source image's -format from its file's first bytes. When they match none of the source formats, -pixa fetches the source again. Two source URLs that serve the same bytes are -stored twice, once under each URL. A source image is stored by writing its file -and then its row; when the row cannot be written, the file is removed again. +The image cache keeps these tables in the database: -Source and transformed image files are written under a temporary name, synced to -disk, and only then renamed into place, so a crash leaves no file under its own -name with part of its data. When pixa starts, and every five minutes after, it -removes each cached source or transformed image whose file is empty, or not the -size the database records for it, with a warning naming the file; the next -request for it fetches the source or converts the image again. Looking for those -files reads the database once for every cached file, so pixa skips each -five-minute look that would start less than ten times as long after the last -look ended as that look took. After a look that took one minute, the next one -starts 10 to 15 minutes after that look ended. +- `source_content`: a row for each source, by source key, with its size and when + it was last used +- `variant_content`: a row for each variant, by variant key (column + `cache_key`), with its size and when it was last used +- `cache_usage`: the total size of the cached sources and variants, which + eviction reads +- `negative_cache`: a row for each source URL in the negative cache, with when + it expires + +pixa reads a cached source image's format from its file's first bytes. When they +match none of the source formats, pixa fetches the source again. Two source URLs +that serve the same bytes are stored twice, once under each URL. A source image +is stored by writing its file and then its row; when the row cannot be written, +the file is removed again. + +Each source and variant file is written under a temporary name and synced to +disk before it is renamed into place. So a crash never leaves part of a file +under its own name. When pixa starts, and every five minutes after, it removes +each source or variant whose file is empty, or not the size its row records. It +logs a warning naming the file. A later request for that image fetches the +source or makes the variant again. Looking for those files reads the database +once for every cached file, so pixa skips each five-minute look that would start +less than ten times as long after the last look ended as that look took. After a +look that took one minute, the next one starts 10 to 15 minutes after that look +ended. pixa's target is 1-5k r/s, which has not been measured at that rate (see Load -Test). Toward it, a cache hit opens the transformed image's file and nothing -else, and makes no database call. The content type comes from the file's first -bytes, which are those of the format pixa wrote it in; a file whose first bytes -are not those of a format pixa writes is not served, and the image is converted -again. A hit hands the time it was served to a background writer and does not -wait for the writer. Eviction orders by that time. The hit is counted in memory. -A miss that uses a cached source image also hands that writer the time it used -the image. Once a second that writer writes the waiting times in one -transaction. While 10,000 times of transformed images, or 10,000 of source -images, wait to be written, a further one is dropped, and that image only looks +Test). Toward it, a cache hit opens the variant's file and nothing else, and +makes no database call. The content type comes from the file's first bytes, +which are those of the format pixa wrote it in; a file whose first bytes are not +those of a format pixa writes is not served, and the image is converted again. A +hit hands the time it was served to a background writer and does not wait for +the writer. Eviction orders by that time. The hit is counted in memory. A miss +that uses a cached source image also hands that writer the time it used the +image. Once a second that writer writes the waiting times in one transaction. Up +to 10,000 times of variants, and as many of sources, can be waiting to be +written. A time handed over beyond that is dropped. Its image then only looks less recently used to eviction. `script/bench` measures how many hits pixa serves per second (see Load Test). @@ -880,9 +919,22 @@ See `configs/config.example.yml` for all options with defaults. ### Architecture -- **Dependency injection**: Uber fx. `internal/app` builds every part of pixad - from the config in one fx graph. Each setting's default is declared once, in - `internal/config`. +- **Dependency injection**: Uber fx. `internal/app` puts every part of pixad in + one fx graph. `internal/providers` builds the image cache, the image proxy and + the other parts that serve images and logins. Each setting's default is + declared once, in `internal/config`. +- **Packages**: these packages under `internal/` work with the concepts above: + - `imagerequest`: the image request, with the source key and the variant key + - `imageproxy`: the image proxy + - `imgcache`: the image cache, which keeps its sources in one `FileStore` + and its variants in another + - `httpfetcher`: fetching sources from upstream hosts + - `magic`: reading an image's format from its first bytes + - `imageprocessor`: making a variant from a source with libvips + - `signature`: checking the `sig` of a signed URL + - `encurl`: making and reading the tokens of encrypted URLs + - `seal`: the keys derived from `signing_key`, and the encryption of those + tokens - **HTTP router**: go-chi - **Image processing**: govips (CGO wrapper for libvips) - **Database**: SQLite via modernc.org/sqlite diff --git a/TODO.md b/TODO.md index f0cba91..92228e4 100644 --- a/TODO.md +++ b/TODO.md @@ -30,6 +30,13 @@ P2: security: per-origin rate limiting # Completed Steps +- 2026-10-26 `README.md`'s Design section starts with a Concepts list (closes + #321). Each concept gets one sentence, under the name the code gives it. + - Storage calls the transformed images variants, in shorter sentences. + - Storage's paths end in the source key and the variant key. + - Storage lists the image cache's tables in the database. + - Architecture names `internal/providers`, and lists the packages that work + with the concepts. - 2026-10-26 one table in `internal/config` lists every setting once (closes #320). `Config.settings` gives each setting's key, its environment variable and the function that loads it.