Compare commits

..
1 Commits
Author SHA1 Message Date
clawbot 049a051596 Check cached sources with upstream after the cache TTL (closes #69)
check / check (push) Canceled after 0s
A cached source and its variants were served until eviction, so a
changed image at the same URL was never picked up.

After 24 hours a request checks the source with upstream, sending its
stored ETag and Last-Modified. A 304 renews the source with no download
or conversion. New bytes replace the source, and older variants are
converted again. A failed check serves the cached image and puts the
URL in the negative cache. This includes the full fetch made for a
variant whose source was evicted. Images not checked for 30 days are
evicted.

A variant file's modification time is its source's check time, so a
hit still makes no database call.

Model: opus-5-5
2026-11-11 09:53:04 +00:00
7 changed files with 254 additions and 76 deletions

No files matched your search

+12 -6
View File
@@ -2,9 +2,8 @@
pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that
proxies images from upstream sources, optionally resizing or transforming them, proxies images from upstream sources, optionally resizing or transforming them,
and serves the results. Both source and transformed images are cached to disk so and serves the results. Both source and transformed images are cached on disk.
that subsequent requests are served without origin fetches or additional pixa checks a cached image with upstream again after 24 hours (see Cache TTL).
processing.
## Getting Started ## Getting Started
@@ -207,7 +206,7 @@ These terms are the names the code uses:
- the upstream answered with an error status - the upstream answered with an error status
- the body's first bytes were not those of the format its `Content-Type` - the body's first bytes were not those of the format its `Content-Type`
names names
- a check of a cached source failed in any way (see Cache TTL) - a check of a cached image failed in any way (see Cache TTL)
- **Image cache**: the part of pixa that keeps the sources, the variants and the - **Image cache**: the part of pixa that keeps the sources, the variants and the
negative cache (see Storage). negative cache (see Storage).
- **Cache TTL**: how long pixa serves a cached source and its variants before it - **Cache TTL**: how long pixa serves a cached source and its variants before it
@@ -235,8 +234,9 @@ The image cache keeps these tables in the database:
unchanged unchanged
- the `ETag` and `Last-Modified` headers upstream sent with it - the `ETag` and `Last-Modified` headers upstream sent with it
- `variant_content`: a row for each variant, by variant key (column - `variant_content`: a row for each variant, by variant key (column
`cache_key`), with its size, when it was last used, and when its source was `cache_key`), with:
last checked with upstream - its size and when it was last used
- when its source was last checked with upstream
- `cache_usage`: the total size of the cached sources and variants, which - `cache_usage`: the total size of the cached sources and variants, which
eviction reads eviction reads
- `negative_cache`: a row for each source URL in the negative cache, with when - `negative_cache`: a row for each source URL in the negative cache, with when
@@ -307,6 +307,12 @@ source or confirmed it unchanged. So a source whose checks keep failing is
served for at most 30 days. Eviction to stay under the disk cache limit can served for at most 30 days. Eviction to stay under the disk cache limit can
remove images sooner. remove images sooner.
That eviction can remove a source and keep its variants. A request for such a
variant after 24 hours fetches the whole image again, since pixa no longer has
the source's `ETag` or `Last-Modified`. pixa then converts the variant again.
When that fetch fails, pixa serves the cached variant and puts the source URL in
the negative cache, as for any failed check.
A variant file's modification time is when its source was last checked with A variant file's modification time is when its source was last checked with
upstream. So a cache hit still reads no database row to know whether the variant upstream. So a cache hit still reads no database row to know whether the variant
is older than 24 hours. A request that checks its source with upstream counts as is older than 24 hours. A request that checks its source with upstream counts as
+6 -4
View File
@@ -31,8 +31,8 @@ P2: security: per-origin rate limiting
# Completed Steps # Completed Steps
- 2026-11-11 a cached source is checked with upstream once it is 24 hours old - 2026-11-11 a cached source is checked with upstream once it is 24 hours old
(closes #69, #83). Before, it and its variants were served until eviction, so (closes #69). Before, it and its variants were served until eviction, so a
a changed image at the same URL was never picked up. changed image at the same URL was never picked up.
- `source_content` keeps the source's `fetched_at`, `checked_at`, and the - `source_content` keeps the source's `fetched_at`, `checked_at`, and the
`ETag` and `Last-Modified` upstream sent. `ETag` and `Last-Modified` upstream sent.
- The check sends them as `If-None-Match` and `If-Modified-Since`. A 304 - The check sends them as `If-None-Match` and `If-Modified-Since`. A 304
@@ -41,9 +41,11 @@ P2: security: per-origin rate limiting
still makes no database call. A variant made before the source's still makes no database call. A variant made before the source's
`fetched_at` is converted again. `fetched_at` is converted again.
- A failed check serves the cached image and puts the source URL in the - A failed check serves the cached image and puts the source URL in the
negative cache, so upstream is asked at most once per 5 minutes. negative cache, so upstream is asked at most once per 5 minutes. A variant
whose source was evicted is checked with a full fetch. When that fetch
fails, the cached variant is served the same way.
- The eviction goroutine evicts sources and variants whose `checked_at` is - The eviction goroutine evicts sources and variants whose `checked_at` is
30 days old (`CacheMaxAge`). 30 days old (`EvictionAfterLastCheck`).
- 2026-10-27 the reconciliation pass skips a directory it cannot list and a file - 2026-10-27 the reconciliation pass skips a directory it cannot list and a file
whose info it cannot read, and goes on (closes #343). It stopped before its whose info it cannot read, and goes on (closes #343). It stopped before its
later checks at a directory it could not list, or at a source or variant file later checks at a directory it could not list, or at a source or variant file
+99 -12
View File
@@ -26,6 +26,22 @@ const (
// errConnectionRefused stands for an upstream that cannot be reached. // errConnectionRefused stands for an upstream that cannot be reached.
var errConnectionRefused = errors.New("connection refused") var errConnectionRefused = errors.New("connection refused")
// upstreamFailure is one way a fetch from upstream fails.
type upstreamFailure struct {
name string
err error
}
// upstreamFailures returns the ways a check of a cached image with upstream
// can fail: an error status, a timeout and no connection.
func upstreamFailures() []upstreamFailure {
return []upstreamFailure{
{"error status", fmt.Errorf("%w: status 503", httpfetcher.ErrUpstreamError)},
{"timeout", httpfetcher.ErrUpstreamTimeout},
{"no connection", errConnectionRefused},
}
}
// conditionalUpstream serves one JPEG image with an ETag or a Last-Modified // conditionalUpstream serves one JPEG image with an ETag or a Last-Modified
// header, or both. It answers 304 Not Modified to a fetch whose If-None-Match // header, or both. It answers 304 Not Modified to a fetch whose If-None-Match
// is its ETag, or, when it has no ETag, whose If-Modified-Since is its // is its ETag, or, when it has no ETag, whose If-Modified-Since is its
@@ -292,17 +308,8 @@ func TestProxy_Get_ChangedSourceAfterTTLReplacesSourceAndVariants(t *testing.T)
func TestProxy_Get_UpstreamFailureAfterTTLServesCachedImage(t *testing.T) { func TestProxy_Get_UpstreamFailureAfterTTLServesCachedImage(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { for _, failure := range upstreamFailures() {
name string t.Run(failure.name, func(t *testing.T) {
err error
}{
{"error status", fmt.Errorf("%w: status 503", httpfetcher.ErrUpstreamError)},
{"timeout", httpfetcher.ErrUpstreamTimeout},
{"no connection", errConnectionRefused},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t) svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
@@ -310,7 +317,7 @@ func TestProxy_Get_UpstreamFailureAfterTTLServesCachedImage(t *testing.T) {
first, _ := getImage(t, svc, variant) first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2)) setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
upstream.fail(tt.err) upstream.fail(failure.err)
second, _ := getImage(t, svc, variant) second, _ := getImage(t, svc, variant)
t.Logf("upstream fetches = %d", upstream.fetches()) t.Logf("upstream fetches = %d", upstream.fetches())
@@ -356,6 +363,86 @@ func TestProxy_Get_FailingUpstreamIsAskedOnceWhileImageIsServedOften(t *testing.
} }
} }
// TestProxy_Get_UpstreamFailureAfterTTLServesVariantOfEvictedSource caches a
// variant, sets its file time two days back, removes its source's row and
// file, as eviction does, and makes upstream fail. The cached variant must
// still be served.
func TestProxy_Get_UpstreamFailureAfterTTLServesVariantOfEvictedSource(t *testing.T) {
t.Parallel()
for _, failure := range upstreamFailures() {
t.Run(failure.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
evictSource(t, svc, variant.Source)
upstream.fail(failure.err)
second, _ := getImage(t, svc, variant)
t.Logf("upstream fetches = %d", upstream.fetches())
if !bytes.Equal(first, second) {
t.Error("the variant served while upstream fails differs from the cached one")
}
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2", n)
}
})
}
}
// TestProxy_Get_FailingUpstreamIsAskedOnceForVariantOfEvictedSource caches a
// variant, sets its file time two days back, removes its source's row and
// file, as eviction does, and makes every fetch time out. Serving the variant
// ten times must ask upstream once: the failed check puts the source URL in
// the negative cache.
func TestProxy_Get_FailingUpstreamIsAskedOnceForVariantOfEvictedSource(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
evictSource(t, svc, variant.Source)
upstream.fail(httpfetcher.ErrUpstreamTimeout)
for range 10 {
image, _ := getImage(t, svc, variant)
if !bytes.Equal(image, first) {
t.Fatal("the variant served while upstream fails differs from the cached one")
}
}
t.Logf("upstream fetches = %d", upstream.fetches())
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2: the first fetch and one check", n)
}
}
// evictSource removes the row and then the file of src's cached source, as
// the cache's eviction does, and leaves its variants cached.
func evictSource(t *testing.T, svc *testProxy, src imagerequest.Source) {
t.Helper()
_, err := svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(src.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(sourceFilePath(svc, src.Key()))
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
}
// setBackSourceAndVariant sets back to when the times the cache keeps for // setBackSourceAndVariant sets back to when the times the cache keeps for
// req's source and for the variant req asks for: the source's fetch and check // req's source and for the variant req asks for: the source's fetch and check
// times, both last access times, the variant's check time, and the variant // times, both last access times, the variant's check time, and the variant
+63 -14
View File
@@ -348,7 +348,9 @@ func (p *Proxy) loadCachedSource(
// changed, the new bytes replace the cached source and the variant is made // changed, the new bytes replace the cached source and the variant is made
// from them. When it did not change, or the check fails, the cached source // from them. When it did not change, or the check fails, the cached source
// is used: the variant's cached file is served again if it was made from the // is used: the variant's cached file is served again if it was made from the
// cached source's bytes, and otherwise the variant is made from them. // cached source's bytes, and otherwise the variant is made from them. When
// the source is not cached, a failed fetch still serves the variant's cached
// file (see fetchAndProcess).
func (p *Proxy) processFromSourceOrFetch( func (p *Proxy) processFromSourceOrFetch(
ctx context.Context, ctx context.Context,
req *imagerequest.Request, req *imagerequest.Request,
@@ -371,15 +373,7 @@ func (p *Proxy) processFromSourceOrFetch(
cached.CheckedAt = time.Now().UTC() cached.CheckedAt = time.Now().UTC()
p.renewSource(ctx, req.Source, cached.CheckedAt) p.renewSource(ctx, req.Source, cached.CheckedAt)
case err != nil: case err != nil:
// Any failure, a timeout included, puts the source URL in the p.recordFailedCheck(ctx, req, err)
// negative cache, so upstream is asked at most once per
// NegativeTTL while it fails, however often the image is served
if !errors.Is(err, ErrNegativeCached) {
p.log.Warn("checking source with upstream failed, using cached source",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path, "error", err)
p.storeNegative(ctx, req)
}
default: default:
return p.processFetched(ctx, req, cacheKey, fetched) return p.processFetched(ctx, req, cacheKey, fetched)
} }
@@ -406,6 +400,24 @@ func (p *Proxy) processFromSourceOrFetch(
source, sourceFormat, sourceSize) source, sourceFormat, sourceSize)
} }
// recordFailedCheck logs a failed check of req's source with upstream, after
// which the image is still served from the cache. It puts the source URL in
// the negative cache, whatever the failure, a timeout included, so upstream
// is asked at most once per NegativeTTL while it fails, however often the
// image is served. A failure that was a negative cache hit records nothing.
func (p *Proxy) recordFailedCheck(
ctx context.Context, req *imagerequest.Request, err error,
) {
if errors.Is(err, ErrNegativeCached) {
return
}
p.log.Warn("checking source with upstream failed, serving from the cache",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path, "error", err)
p.storeNegative(ctx, req)
}
// renewSource records that upstream answered at checkedAt that src's cached // renewSource records that upstream answered at checkedAt that src's cached
// source had not changed. A failure to record it is only logged: the source // source had not changed. A failure to record it is only logged: the source
// is then checked again on its next use. // is then checked again on its next use.
@@ -437,12 +449,39 @@ func (p *Proxy) renewVariant(
return nil return nil
} }
return p.readCachedVariant(cacheKey, reader, contentType)
}
// cachedVariantIgnoringTTL returns the cached variant of cacheKey whatever
// its age. It returns nil when the variant is not cached or cannot be read.
func (p *Proxy) cachedVariantIgnoringTTL(
cacheKey imagerequest.VariantKey,
) *processedVariant {
reader, _, contentType, err := p.cache.VariantIgnoringTTL(cacheKey)
if err != nil {
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("cached variant not served", "cache_key", cacheKey,
"error", err)
}
return nil
}
return p.readCachedVariant(cacheKey, reader, contentType)
}
// readCachedVariant reads the cached variant of cacheKey that reader opened,
// whose content type is contentType, and closes reader. It returns nil when
// the variant cannot be read.
func (p *Proxy) readCachedVariant(
cacheKey imagerequest.VariantKey, reader io.ReadCloser, contentType string,
) *processedVariant {
defer func() { _ = reader.Close() }() defer func() { _ = reader.Close() }()
data, err := io.ReadAll(reader) data, err := io.ReadAll(reader)
if err != nil { if err != nil {
p.log.Warn("failed to read cached variant, making it again", p.log.Warn("failed to read cached variant", "cache_key", cacheKey,
"cache_key", cacheKey, "error", err) "error", err)
return nil return nil
} }
@@ -451,7 +490,10 @@ func (p *Proxy) renewVariant(
} }
// fetchAndProcess fetches the source from upstream, stores it, and makes // fetchAndProcess fetches the source from upstream, stores it, and makes
// and stores the variant from it. // and stores the variant from it. When the fetch fails while the variant is
// still cached, as it can be after eviction removed its source, the cached
// variant is served whatever its age, and the failure is recorded as a failed
// check (see recordFailedCheck).
func (p *Proxy) fetchAndProcess( func (p *Proxy) fetchAndProcess(
ctx context.Context, ctx context.Context,
req *imagerequest.Request, req *imagerequest.Request,
@@ -459,7 +501,14 @@ func (p *Proxy) fetchAndProcess(
) (*processedVariant, error) { ) (*processedVariant, error) {
fetched, err := p.fetchSource(ctx, req, "", "") fetched, err := p.fetchSource(ctx, req, "", "")
if err != nil { if err != nil {
return nil, err variant := p.cachedVariantIgnoringTTL(cacheKey)
if variant == nil {
return nil, err
}
p.recordFailedCheck(ctx, req, err)
return variant, nil
} }
return p.processFetched(ctx, req, cacheKey, fetched) return p.processFetched(ctx, req, cacheKey, fetched)
+48 -15
View File
@@ -3,8 +3,12 @@
// - the variants made from them // - the variants made from them
// - the negative cache: the source URLs whose fetch failed recently // - the negative cache: the source URLs whose fetch failed recently
// //
// Each image is a file with a row in the database. Eviction removes the // Each image is a file with a row in the database. A source and its
// least recently used images to keep the cache under its size limit. // variants are served until the cache TTL has passed since upstream last sent
// the source or confirmed it unchanged. Then the source is due to be checked
// with upstream again (see Stale). Eviction removes:
// - the least recently used images, to keep the cache under its size limit
// - the images not checked with upstream for EvictionAfterLastCheck
package imgcache package imgcache
import ( import (
@@ -45,10 +49,11 @@ const DefaultNegativeTTL = 5 * time.Minute
// variants without asking upstream again. // variants without asking upstream again.
const DefaultCacheTTL = 24 * time.Hour const DefaultCacheTTL = 24 * time.Hour
// CacheMaxAge is how long after upstream last sent a cached source, or // EvictionAfterLastCheck is how long after upstream last sent a cached
// answered that it had not changed, the source and its variants are evicted // source, or answered that it had not changed, the source and its variants
// (see EvictPastMaxAge). Until then a source whose checks fail is served. // are evicted (see EvictNotCheckedRecently). Until then a source whose checks
const CacheMaxAge = 30 * 24 * time.Hour // fail is served.
const EvictionAfterLastCheck = 30 * 24 * time.Hour
// ErrStale is returned by Variant for a variant whose source was last // ErrStale is returned by Variant for a variant whose source was last
// checked with upstream CacheTTL or more ago. // checked with upstream CacheTTL or more ago.
@@ -208,25 +213,34 @@ func (c *Cache) Variant(
return nil, 0, "", ErrStale return nil, 0, "", ErrStale
} }
format, err := readFormat(file) return c.finishOpeningVariant(cacheKey, file, info)
if err != nil { }
_ = file.Close()
// VariantIgnoringTTL opens a cached variant whatever its age, and returns it
// as Variant does. The image proxy serves it when a fetch of the variant's
// source fails while the source is not cached.
func (c *Cache) VariantIgnoringTTL(
cacheKey imagerequest.VariantKey,
) (io.ReadCloser, int64, string, error) {
if c.config.MaxBytes == 0 {
return nil, 0, "", ErrNotFound
}
file, info, err := c.openVariant(cacheKey)
if err != nil {
return nil, 0, "", err return nil, 0, "", err
} }
c.touchVariant(cacheKey) return c.finishOpeningVariant(cacheKey, file, info)
return file, info.Size(), format.MIMEType(), nil
} }
// RenewVariant opens a cached variant made from the source bytes that source // RenewVariant opens a cached variant made from the source bytes that source
// describes, whatever its age, and returns it as Variant does. It moves the // describes, whatever its age, and returns it as Variant does. It moves the
// variant file's modification time, and the checked_at of its row, forward // variant file's modification time, and the checked_at of its row, forward
// to source.CheckedAt, so Variant serves the file until CacheTTL after that // to source.CheckedAt, so Variant serves the file until CacheTTL after that
// check and eviction keeps it until CacheMaxAge after it. A variant made // check and eviction keeps it until EvictionAfterLastCheck after it. A
// before source.FetchedAt was made from older bytes, and is ErrNotFound, as // variant made before source.FetchedAt was made from older bytes, and is
// is a variant not in the cache. // ErrNotFound, as is a variant not in the cache.
func (c *Cache) RenewVariant( func (c *Cache) RenewVariant(
ctx context.Context, cacheKey imagerequest.VariantKey, source SourceInfo, ctx context.Context, cacheKey imagerequest.VariantKey, source SourceInfo,
) (io.ReadCloser, int64, string, error) { ) (io.ReadCloser, int64, string, error) {
@@ -646,6 +660,25 @@ func (c *Cache) openVariant(
return file, info, nil return file, info, nil
} }
// finishOpeningVariant reads the content type of a variant file openVariant
// opened from its first bytes, hands the variant's access time to the
// background writer (see touchVariant), and returns the file as Variant does.
// It closes the file when its first bytes are not those of an output format.
func (c *Cache) finishOpeningVariant(
cacheKey imagerequest.VariantKey, file *os.File, info fs.FileInfo,
) (io.ReadCloser, int64, string, error) {
format, err := readFormat(file)
if err != nil {
_ = file.Close()
return nil, 0, "", err
}
c.touchVariant(cacheKey)
return file, info.Size(), format.MIMEType(), nil
}
// readFormat reads the first bytes of a source or variant file, without // readFormat reads the first bytes of a source or variant file, without
// moving its offset, and returns the format they show. Sources and variants // moving its offset, and returns the format they show. Sources and variants
// are in the same six formats. // are in the same six formats.
@@ -10,11 +10,11 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestEvictPastMaxAge stores a source and a variant last checked with // TestEvictNotCheckedRecently stores a source and a variant last checked with
// upstream 31 days ago and a source and a variant checked 29 days ago. // upstream 31 days ago and a source and a variant checked 29 days ago.
// EvictPastMaxAge must remove the rows and files of the first two and keep // EvictNotCheckedRecently must remove the rows and files of the first two and
// the other two. // keep the other two.
func TestEvictPastMaxAge(t *testing.T) { func TestEvictNotCheckedRecently(t *testing.T) {
t.Parallel() t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
@@ -47,9 +47,9 @@ func TestEvictPastMaxAge(t *testing.T) {
} }
} }
err := cache.EvictPastMaxAge(t.Context()) err := cache.EvictNotCheckedRecently(t.Context())
if err != nil { if err != nil {
t.Fatalf("EvictPastMaxAge() error = %v", err) t.Fatalf("EvictNotCheckedRecently() error = %v", err)
} }
sourceRows, err := queryStringColumn[imagerequest.SourceKey](t.Context(), sourceRows, err := queryStringColumn[imagerequest.SourceKey](t.Context(),
+20 -19
View File
@@ -131,20 +131,20 @@ func (c *Cache) EvictToLimit(ctx context.Context) error {
} }
} }
// EvictPastMaxAge evicts each source and variant whose checked_at is // EvictNotCheckedRecently evicts each source and variant whose checked_at is
// CacheMaxAge or more ago, evictionBatchSize of each at a time, until none // EvictionAfterLastCheck or more ago, evictionBatchSize of each at a time,
// is left. It is a no-op when the cache is disabled. // until none is left. It is a no-op when the cache is disabled.
func (c *Cache) EvictPastMaxAge(ctx context.Context) error { func (c *Cache) EvictNotCheckedRecently(ctx context.Context) error {
if c.config.MaxBytes == 0 { if c.config.MaxBytes == 0 {
return nil return nil
} }
cutoff := time.Now().UTC().Add(-CacheMaxAge) cutoff := time.Now().UTC().Add(-EvictionAfterLastCheck)
for { for {
sources, err := queryStringColumn[imagerequest.SourceKey](ctx, c.db, ` sources, err := queryStringColumn[imagerequest.SourceKey](ctx, c.db, `
SELECT source_key FROM source_content WHERE checked_at < ? LIMIT ? SELECT source_key FROM source_content WHERE checked_at < ? LIMIT ?
`, "sources past the maximum age", "source key", cutoff, evictionBatchSize) `, "sources not checked recently", "source key", cutoff, evictionBatchSize)
if err != nil { if err != nil {
return err return err
} }
@@ -158,7 +158,7 @@ func (c *Cache) EvictPastMaxAge(ctx context.Context) error {
variants, err := queryStringColumn[imagerequest.VariantKey](ctx, c.db, ` variants, err := queryStringColumn[imagerequest.VariantKey](ctx, c.db, `
SELECT cache_key FROM variant_content WHERE checked_at < ? LIMIT ? SELECT cache_key FROM variant_content WHERE checked_at < ? LIMIT ?
`, "variants past the maximum age", "variant key", cutoff, evictionBatchSize) `, "variants not checked recently", "variant key", cutoff, evictionBatchSize)
if err != nil { if err != nil {
return err return err
} }
@@ -171,7 +171,7 @@ func (c *Cache) EvictPastMaxAge(ctx context.Context) error {
} }
if len(sources) > 0 || len(variants) > 0 { if len(sources) > 0 || len(variants) > 0 {
c.log.Info("evicted cache content past the maximum age", c.log.Info("evicted cache content not checked with upstream recently",
"sources", len(sources), "variants", len(variants)) "sources", len(sources), "variants", len(variants))
} }
@@ -394,9 +394,9 @@ func (c *Cache) notifyWritePressure() {
} }
// StartEviction launches the background eviction goroutine, which // StartEviction launches the background eviction goroutine, which
// removes the expired negative cache rows, and the sources and variants // removes the expired negative cache rows, and the sources and variants not
// past CacheMaxAge, at startup and again on every periodic tick // checked for EvictionAfterLastCheck, at startup and again on every periodic
// thereafter, reconciles the database accounting with the // tick thereafter, reconciles the database accounting with the
// cache directories at startup and again on the first periodic tick // cache directories at startup and again on the first periodic tick
// that comes at least ten times as long after a reconciliation pass // that comes at least ten times as long after a reconciliation pass
// ended as that pass took, and evicts to the configured limit on the given // ended as that pass took, and evicts to the configured limit on the given
@@ -443,7 +443,7 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
nextReconciliation := c.runReconciliationPass(ctx) nextReconciliation := c.runReconciliationPass(ctx)
c.runCleanExpired(ctx) c.runCleanExpired(ctx)
c.runEvictPastMaxAge(ctx) c.runEvictNotCheckedRecently(ctx)
c.runEvictionPass(ctx) c.runEvictionPass(ctx)
ticker := time.NewTicker(interval) ticker := time.NewTicker(interval)
@@ -469,7 +469,7 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
} }
c.runCleanExpired(ctx) c.runCleanExpired(ctx)
c.runEvictPastMaxAge(ctx) c.runEvictNotCheckedRecently(ctx)
case <-c.evictionPressure: case <-c.evictionPressure:
} }
@@ -527,17 +527,18 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
} }
} }
// runEvictPastMaxAge runs EvictPastMaxAge, logging a failure instead of // runEvictNotCheckedRecently runs EvictNotCheckedRecently, logging a failure
// propagating it (the loop must keep running). It does nothing once ctx is // instead of propagating it (the loop must keep running). It does nothing
// cancelled. // once ctx is cancelled.
func (c *Cache) runEvictPastMaxAge(ctx context.Context) { func (c *Cache) runEvictNotCheckedRecently(ctx context.Context) {
if ctx.Err() != nil { if ctx.Err() != nil {
return return
} }
err := c.EvictPastMaxAge(ctx) err := c.EvictNotCheckedRecently(ctx)
if err != nil { if err != nil {
c.log.Warn("eviction past the maximum age failed", "error", err) c.log.Warn("evicting cache content not checked recently failed",
"error", err)
} }
} }