Check cached sources with upstream after the cache TTL (closes #69, closes #83)
check / check (push) Canceled after 0s

A cached source and its variants were served until eviction, so a
changed image at the same URL was never picked up.

Once a source was last checked 24 hours ago, the next request for one
of its images asks upstream with If-None-Match and If-Modified-Since,
from the ETag and Last-Modified it stored. A 304 renews the source and
its variant without a download or a conversion. New bytes replace the
source, and each variant made before them is converted again. A failed
check serves the cached image and puts the URL in the negative cache.
Sources and variants not checked successfully for 30 days are evicted.

A variant file's modification time is its source's check time, so a
hit still makes no database call.

Existing tests change only in call signatures. The test that asserted
two-day-old entries are served without asking upstream is replaced by
the new ones.

Model: opus-5-5
This commit is contained in:
clawbot committed 2026-11-11 00:24:06 +00:00
1 parent 9f1e9f5253
commit 8feed69143
28 files changed
+1262 -226

No files matched your search

+40 -3
View File
@@ -207,8 +207,11 @@ These terms are the names the code uses:
- the upstream answered with an error status
- the body's first bytes were not those of the format its `Content-Type`
names
- a check of a cached source failed in any way (see Cache TTL)
- **Image cache**: the part of pixa that keeps the sources, the variants and the
negative cache (see Storage).
- **Cache TTL**: how long pixa serves a cached source and its variants before it
checks the source with upstream again: 24 hours (see Cache TTL).
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
sources and variants may reach together.
- **Image proxy**: the part of pixa that serves a variant from the image cache
@@ -226,10 +229,14 @@ name.
The image cache keeps these tables in the database:
- `source_content`: a row for each source, by source key, with its size and when
it was last used
- `source_content`: a row for each source, by source key, with:
- its size and when it was last used
- when it was fetched, and when upstream last sent it or confirmed it
unchanged
- the `ETag` and `Last-Modified` headers upstream sent with it
- `variant_content`: a row for each variant, by variant key (column
`cache_key`), with its size and when it was last used
`cache_key`), with its size, when it was last used, and when its source was
last checked with upstream
- `cache_usage`: the total size of the cached sources and variants, which
eviction reads
- `negative_cache`: a row for each source URL in the negative cache, with when
@@ -275,6 +282,36 @@ written. A time handed over beyond that is dropped. Its image then only looks
less recently used to eviction. `script/bench` measures how many hits pixa
serves per second (see Load Test).
### Cache TTL
pixa serves a cached source and its variants for 24 hours after upstream last
sent the source or confirmed it unchanged. After that, the next request for an
image of that source checks the source with upstream first:
- pixa asks upstream to send the source only if it changed. It sends the
source's `ETag` as `If-None-Match` and its `Last-Modified` as
`If-Modified-Since`, when upstream gave them. With neither, upstream sends the
whole image again.
- When upstream answers 304 Not Modified, pixa serves the cached source and its
variants for another 24 hours. Nothing is downloaded or converted again.
- When upstream sends the image, it replaces the cached source. Each variant
made from the old image is converted again from the new one when it is next
requested.
- When the check fails, pixa serves the cached image. A failure can be an error
status, a timeout or no connection. The failure puts the source URL in the
negative cache, so pixa does not ask upstream again for 5 minutes, however
often it serves the image.
pixa evicts a source and its variants 30 days after upstream last sent the
source or confirmed it unchanged. So a source whose checks keep failing is
served for at most 30 days. Eviction to stay under the disk cache limit can
remove images sooner.
A variant file's modification time is when its source was last checked with
upstream. So a cache hit still reads no database row to know whether the variant
is older than 24 hours. A request that checks its source with upstream counts as
a miss.
### Routes
pixa answers these routes; any other path answers 404. A path in this list asked