Check cached sources with upstream after the cache TTL (closes #69, closes #83)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
A cached source and its variants were served until eviction, so a changed image at the same URL was never picked up. Once a source was last checked 24 hours ago, the next request for one of its images asks upstream with If-None-Match and If-Modified-Since, from the ETag and Last-Modified it stored. A 304 renews the source and its variant without a download or a conversion. New bytes replace the source, and each variant made before them is converted again. A failed check serves the cached image and puts the URL in the negative cache. Sources and variants not checked successfully for 30 days are evicted. A variant file's modification time is its source's check time, so a hit still makes no database call. Existing tests change only in call signatures. The test that asserted two-day-old entries are served without asking upstream is replaced by the new ones. Model: opus-5-5
This commit is contained in:
1 parent
9f1e9f5253
commit
8feed69143
28 files changed
+1262
-226
No files matched your search
@@ -207,8 +207,11 @@ These terms are the names the code uses:
|
||||
- the upstream answered with an error status
|
||||
- the body's first bytes were not those of the format its `Content-Type`
|
||||
names
|
||||
- a check of a cached source failed in any way (see Cache TTL)
|
||||
- **Image cache**: the part of pixa that keeps the sources, the variants and the
|
||||
negative cache (see Storage).
|
||||
- **Cache TTL**: how long pixa serves a cached source and its variants before it
|
||||
checks the source with upstream again: 24 hours (see Cache TTL).
|
||||
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
|
||||
sources and variants may reach together.
|
||||
- **Image proxy**: the part of pixa that serves a variant from the image cache
|
||||
@@ -226,10 +229,14 @@ name.
|
||||
|
||||
The image cache keeps these tables in the database:
|
||||
|
||||
- `source_content`: a row for each source, by source key, with its size and when
|
||||
it was last used
|
||||
- `source_content`: a row for each source, by source key, with:
|
||||
- its size and when it was last used
|
||||
- when it was fetched, and when upstream last sent it or confirmed it
|
||||
unchanged
|
||||
- the `ETag` and `Last-Modified` headers upstream sent with it
|
||||
- `variant_content`: a row for each variant, by variant key (column
|
||||
`cache_key`), with its size and when it was last used
|
||||
`cache_key`), with its size, when it was last used, and when its source was
|
||||
last checked with upstream
|
||||
- `cache_usage`: the total size of the cached sources and variants, which
|
||||
eviction reads
|
||||
- `negative_cache`: a row for each source URL in the negative cache, with when
|
||||
@@ -275,6 +282,36 @@ written. A time handed over beyond that is dropped. Its image then only looks
|
||||
less recently used to eviction. `script/bench` measures how many hits pixa
|
||||
serves per second (see Load Test).
|
||||
|
||||
### Cache TTL
|
||||
|
||||
pixa serves a cached source and its variants for 24 hours after upstream last
|
||||
sent the source or confirmed it unchanged. After that, the next request for an
|
||||
image of that source checks the source with upstream first:
|
||||
|
||||
- pixa asks upstream to send the source only if it changed. It sends the
|
||||
source's `ETag` as `If-None-Match` and its `Last-Modified` as
|
||||
`If-Modified-Since`, when upstream gave them. With neither, upstream sends the
|
||||
whole image again.
|
||||
- When upstream answers 304 Not Modified, pixa serves the cached source and its
|
||||
variants for another 24 hours. Nothing is downloaded or converted again.
|
||||
- When upstream sends the image, it replaces the cached source. Each variant
|
||||
made from the old image is converted again from the new one when it is next
|
||||
requested.
|
||||
- When the check fails, pixa serves the cached image. A failure can be an error
|
||||
status, a timeout or no connection. The failure puts the source URL in the
|
||||
negative cache, so pixa does not ask upstream again for 5 minutes, however
|
||||
often it serves the image.
|
||||
|
||||
pixa evicts a source and its variants 30 days after upstream last sent the
|
||||
source or confirmed it unchanged. So a source whose checks keep failing is
|
||||
served for at most 30 days. Eviction to stay under the disk cache limit can
|
||||
remove images sooner.
|
||||
|
||||
A variant file's modification time is when its source was last checked with
|
||||
upstream. So a cache hit still reads no database row to know whether the variant
|
||||
is older than 24 hours. A request that checks its source with upstream counts as
|
||||
a miss.
|
||||
|
||||
### Routes
|
||||
|
||||
pixa answers these routes; any other path answers 404. A path in this list asked
|
||||
|
||||
Reference in new issue
Block a user