Check cached sources with upstream after the cache TTL (closes #69)
check / check (push) Canceled after 0s

Cached images were served until eviction, so upstream changes were
missed.

After 24 hours a request checks the source with upstream, sending its
stored ETag and Last-Modified. A 304 renews it with no download or
conversion. New bytes replace it; older variants are converted again.
A failed check serves the cached image and puts the URL in the
negative cache. Images not checked for 30 days are evicted.

Variant rows keep the source's ETag and Last-Modified for checking a
variant of an evicted source.

A variant file's modification time is its source's check time when it
was made or renewed. Hits read it without a database call. The ETag
includes it, so new bytes get a new ETag.

Model: opus-5-5
This commit is contained in:
clawbot committed 2026-11-11 11:40:36 +00:00
1 parent b3b8408d18
commit 445ff628fd
29 files changed
+1709 -269

No files matched your search

+20
View File
@@ -30,6 +30,26 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-11-11 a cached source is checked with upstream once it is 24 hours old
(closes #69). Before, it was used and its variants served until eviction, so a
changed image at the same URL was never picked up.
- `source_content` keeps the source's `fetched_at`, `checked_at`, and the
`ETag` and `Last-Modified` upstream sent.
- The check sends them as `If-None-Match` and `If-Modified-Since`. A 304
renews the source; new bytes replace it.
- A variant file's modification time is its source's `checked_at` when the
variant was made or last renewed, so a hit makes no database call. A
variant made before the source's `fetched_at` is converted again.
- A variant's `ETag` is made from its variant key and its file's
modification time, so a variant made again from new bytes gets a new
`ETag`.
- `variant_content` keeps the `ETag` and `Last-Modified` of the source a
variant was made from. A variant whose source was evicted is checked with
them, and a 304 renews the variant.
- A failed check serves the cached image and puts the source URL in the
negative cache, so upstream is asked at most once per 5 minutes.
- The eviction goroutine evicts sources and variants whose `checked_at` is
30 days old (`EvictionAfterLastCheck`).
- 2026-10-27 the reconciliation pass skips a directory it cannot list and a file
whose info it cannot read, and goes on (closes #343). It stopped before its
later checks at a directory it could not list, or at a source or variant file