Check cached sources with upstream after the cache TTL (closes #69)
check / check (push) Canceled after 0s

A cached source and its variants were served until eviction, so a
changed image at the same URL was never picked up.

After 24 hours a request checks the source with upstream, sending its
stored ETag and Last-Modified. A 304 renews the source with no download
or conversion. New bytes replace the source, and older variants are
converted again. A failed check serves the cached image and puts the
URL in the negative cache. This includes the full fetch made for a
variant whose source was evicted. Images not checked for 30 days are
evicted.

A variant file's modification time is its source's check time, so a
hit still makes no database call.

Model: opus-5-5
This commit is contained in:
clawbot committed 2026-11-11 09:53:04 +00:00
1 parent 9f1e9f5253
commit 049a051596
28 files changed
+1445 -231

No files matched your search

+49 -6
View File
@@ -2,9 +2,8 @@
pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that
proxies images from upstream sources, optionally resizing or transforming them,
and serves the results. Both source and transformed images are cached to disk so
that subsequent requests are served without origin fetches or additional
processing.
and serves the results. Both source and transformed images are cached on disk.
pixa checks a cached image with upstream again after 24 hours (see Cache TTL).
## Getting Started
@@ -207,8 +206,11 @@ These terms are the names the code uses:
- the upstream answered with an error status
- the body's first bytes were not those of the format its `Content-Type`
names
- a check of a cached image failed in any way (see Cache TTL)
- **Image cache**: the part of pixa that keeps the sources, the variants and the
negative cache (see Storage).
- **Cache TTL**: how long pixa serves a cached source and its variants before it
checks the source with upstream again: 24 hours (see Cache TTL).
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
sources and variants may reach together.
- **Image proxy**: the part of pixa that serves a variant from the image cache
@@ -226,10 +228,15 @@ name.
The image cache keeps these tables in the database:
- `source_content`: a row for each source, by source key, with its size and when
it was last used
- `source_content`: a row for each source, by source key, with:
- its size and when it was last used
- when it was fetched, and when upstream last sent it or confirmed it
unchanged
- the `ETag` and `Last-Modified` headers upstream sent with it
- `variant_content`: a row for each variant, by variant key (column
`cache_key`), with its size and when it was last used
`cache_key`), with:
- its size and when it was last used
- when its source was last checked with upstream
- `cache_usage`: the total size of the cached sources and variants, which
eviction reads
- `negative_cache`: a row for each source URL in the negative cache, with when
@@ -275,6 +282,42 @@ written. A time handed over beyond that is dropped. Its image then only looks
less recently used to eviction. `script/bench` measures how many hits pixa
serves per second (see Load Test).
### Cache TTL
pixa serves a cached source and its variants for 24 hours after upstream last
sent the source or confirmed it unchanged. After that, the next request for an
image of that source checks the source with upstream first:
- pixa asks upstream to send the source only if it changed. It sends the
source's `ETag` as `If-None-Match` and its `Last-Modified` as
`If-Modified-Since`, when upstream gave them. With neither, upstream sends the
whole image again.
- When upstream answers 304 Not Modified, pixa serves the cached source and its
variants for another 24 hours. Nothing is downloaded or converted again.
- When upstream sends the image, it replaces the cached source. Each variant
made from the old image is converted again from the new one when it is next
requested.
- When the check fails, pixa serves the cached image. A failure can be an error
status, a timeout or no connection. The failure puts the source URL in the
negative cache, so pixa does not ask upstream again for 5 minutes, however
often it serves the image.
pixa evicts a source and its variants 30 days after upstream last sent the
source or confirmed it unchanged. So a source whose checks keep failing is
served for at most 30 days. Eviction to stay under the disk cache limit can
remove images sooner.
That eviction can remove a source and keep its variants. A request for such a
variant after 24 hours fetches the whole image again, since pixa no longer has
the source's `ETag` or `Last-Modified`. pixa then converts the variant again.
When that fetch fails, pixa serves the cached variant and puts the source URL in
the negative cache, as for any failed check.
A variant file's modification time is when its source was last checked with
upstream. So a cache hit still reads no database row to know whether the variant
is older than 24 hours. A request that checks its source with upstream counts as
a miss.
### Routes
pixa answers these routes; any other path answers 404. A path in this list asked