feat: cache size management and LRU eviction (closes #51) (#55)
Some checks failed
check / check (push) Has been cancelled
Some checks failed
check / check (push) Has been cancelled
Implements #51 per the issue DoD and the owner direction comment (issuecomment-44068). ## Behavior **Config: `cache_max_bytes`** (integrates with the #52/#53 validation framework) - Strict int64 parsing via a new `getInt64`/`int64Val` getter in the existing strict-loader pattern; the key is registered in the known-keys list. A SET but invalid value — negative, float, null, non-numeric string, boolean, list — aborts startup with exit 1 naming the key and the offending value. - Explicit values are used exactly as given, any non-negative amount, no floor. `cache_max_bytes: 0` is a valid value that disables the disk cache entirely. - Omitted: after `state_dir` validation, the default resolves to `max(75% of free bytes on the filesystem containing <state_dir>/cache/, 500 MiB)`. The cache directory is created first and statfs runs on that actual path, so the measurement hits the right filesystem. The probe is injectable (`FreeSpaceProbeFunc`) so tests do not depend on the host disk. The effective limit (and disabled state) is logged at startup. **Size accounting** (no directory scans on the hot path) - Migration `002` adds a `variant_content` table — processed variants were previously untracked anywhere — and a `last_accessed_at` column on `source_content`, both indexed. Total usage is two SUM queries. - Stores record accounting rows; cache hits touch the LRU timestamps (same cost class as the existing per-request stats UPDATEs). The variant accounting insert is best-effort with a warning: the reconciliation pass (below) adopts any file that missed its row, and this keeps the pre-migration inline test schema working. **Eviction policy: global LRU across both content classes** - Candidates are the least-recently-used entries from `variant_content` and `source_content` (batched, 100 per class per pass, merged oldest-first by `COALESCE(last_accessed_at, fetched_at)`), evicted until usage is at or below the limit. - Why global LRU: recency of actual use is the best cheap predictor of future use for a CDN-style cache, and treating both classes in one ordering avoids pathologies of class-priority schemes (e.g. evicting every variant before any cold source blob, which would tank hit rate, or the reverse, which would hoard stale sources). Byte-for-byte, the coldest data goes first regardless of what kind it is. LFU-style schemes need more bookkeeping for marginal gain at this scale. - Reference safety (the multi-reference DoD case): evicting a source blob deletes ALL `source_metadata` rows referencing it plus its `source_content` row in a single transaction BEFORE the file is unlinked. A blob referenced by multiple source paths is only ever removed together with all of its references, and DB rows never point at deleted files (the crash window leaves at worst an orphaned file, which reconciliation sweeps). The JSON metadata sidecars for removed rows are deleted as well. **Triggers, off the request path** - A background goroutine (started in the handlers OnStart hook, stopped in OnStop) runs an eviction pass on a periodic ticker (5 min) and on write pressure: every store sends a non-blocking notification on a capacity-1 channel. Requests never wait on eviction. - On startup the goroutine first reconciles accounting with the disk (off the hot path): adopts untracked variant files (size/mtime from disk, content type from the `.meta` sidecar), drops accounting rows whose files are missing, removes source blob files the DB does not know (unreachable, since lookups go through `source_metadata`), removes rows whose files are gone, and sweeps `.tmp-*` files older than an hour. **`cache_max_bytes: 0` disables the disk cache** - No cache directories are created, lookups always miss, `StoreSource`/`StoreVariant` are no-ops, no evictor runs; every request fetches and processes uncached. Verified end-to-end (below). ## Notes for review - At the `imgcache.CacheConfig` layer, disabling is an explicit `DisableDiskCache` flag rather than `MaxBytes == 0`, because existing test fixtures construct `CacheConfig` without `MaxBytes` and rely on the legacy "no limit" behavior; per repo rules those tests were not touched. The config layer maps `cache_max_bytes: 0` to the flag in `handlers`. `MaxBytes == 0` at that layer means "no limit enforced" and is unreachable from production config (the computed default is always at least 500 MiB). - The negative cache stays active in disabled mode: it is DB-backed (TTL-expired rows in SQLite), not part of the disk cache this issue bounds, and it protects against hammering failing upstreams. Flagging explicitly since the direction said "no cache reads, no cache writes" — I read that as the disk cache; happy to disable it too if intended. - One deviation from pure red/green: after the red commit I extended the new-test fixture helper (`newEvictionTestCache`) to pass `DisableDiskCache: maxBytes == 0`, mirroring the production mapping, when the flag design emerged. Assertions were not touched; no pre-existing tests were modified. - Sidecar files (`.meta`, metadata JSON) are not counted in usage; they are bounded by entry counts and small (tens of bytes to ~1 KiB per entry) while content bytes dominate. Documented here for transparency. - Discovered while working: `Cache.Stats` reads the never-populated `output_content`/`request_cache` tables, so `TotalItems`/`TotalSizeBytes` are always 0. Out of scope here; filing as a separate issue. ## Verification - TDD: commit `3963ec3` adds the failing tests first (18 new tests covering strict parsing, default computation with injected probe including floor and 75% branches, explicit-no-floor, zero-disables, size accounting, dedup accounting, LRU order, multi-reference blob eviction with the no-dangling-references invariant, under-limit no-op, write-pressure trigger, periodic trigger, reconciliation); implementation follows in `8cb09b6`/`bdd86a4` until green. - `make check` green (all tests, lint 0 issues, fmt-check) at HEAD. - Pinned CI lint gate: `docker build --target lint .` green (golangci-lint v2.10.1). - End-to-end with the built binary: - omitted key: startup logs `computed default cache size limit from free space` and `effective cache size limit` (75% of the test host's free space); - `cache_max_bytes: banana`: exit 1 with `config key "cache_max_bytes": value "banana" is not an integer`; - `cache_max_bytes: 0`: `cache_disabled=true` logged, two identical requests both fetch upstream (2 upstream fetches logged), 200 `image/jpeg` responses, no `cache/` directory created, only `state.sqlite3` in the state dir; - enabled: second request served from cache (1 upstream fetch), `variant_content` and `source_content` rows match the on-disk file sizes. Co-authored-by: sneak <sneak@sneak.berlin> Reviewed-on: #55 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org>
This commit was merged in pull request #55.
This commit is contained in:
@@ -2,12 +2,16 @@ package imgcache
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
@@ -27,6 +31,22 @@ type CacheConfig struct {
|
||||
StateDir string
|
||||
CacheTTL time.Duration
|
||||
NegativeTTL time.Duration
|
||||
|
||||
// MaxBytes is the disk cache size limit in bytes that eviction
|
||||
// enforces. Zero means no limit is enforced (no eviction). The
|
||||
// config layer supplies the computed default when the operator
|
||||
// omits cache_max_bytes.
|
||||
MaxBytes int64
|
||||
|
||||
// DisableDiskCache turns the disk cache off entirely: no cache
|
||||
// directories are created, lookups always miss, stores are
|
||||
// no-ops, and no eviction machinery runs. The config layer sets
|
||||
// this when the operator configures cache_max_bytes: 0.
|
||||
DisableDiskCache bool
|
||||
|
||||
// Logger receives accounting and eviction log output. A nil
|
||||
// Logger means slog.Default().
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// variantMeta stores content type for fast cache hits without reading .meta file.
|
||||
@@ -42,13 +62,60 @@ type Cache struct {
|
||||
variants *VariantStorage // processed variants by cache key
|
||||
srcMetadata *MetadataStorage // source metadata by host/path
|
||||
config CacheConfig
|
||||
log *slog.Logger
|
||||
|
||||
// disabled means the disk cache is turned off entirely: lookups
|
||||
// always miss, stores are no-ops, and no eviction runs.
|
||||
disabled bool
|
||||
|
||||
// Eviction machinery. The channels are created in NewCache so
|
||||
// stores can signal write pressure without racing StartEviction.
|
||||
evictionPressure chan struct{}
|
||||
evictionStop chan struct{}
|
||||
evictionDone chan struct{}
|
||||
evictionStarted bool
|
||||
evictionStopOnce sync.Once
|
||||
|
||||
// In-memory cache of variant metadata (content type, size) to avoid reading .meta files
|
||||
metaCache map[VariantKey]variantMeta
|
||||
|
||||
// contentLocks serializes StoreSource and evictSourceBlob per
|
||||
// content hash, closing the race window between an eviction's row
|
||||
// deletion and its file unlink against a concurrent store of
|
||||
// identical content.
|
||||
contentLocks *contentLock
|
||||
|
||||
// evictSourceBlobTestHook, when set, is invoked by evictSourceBlob
|
||||
// after its row-deletion transaction commits and before the
|
||||
// content file is unlinked. It exists solely so tests can
|
||||
// deterministically pause inside that window to exercise
|
||||
// concurrent stores against it; production code leaves it nil.
|
||||
evictSourceBlobTestHook func(ContentHash)
|
||||
}
|
||||
|
||||
// NewCache creates a new cache instance.
|
||||
func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
|
||||
log := config.Logger
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
|
||||
c := &Cache{
|
||||
db: db,
|
||||
config: config,
|
||||
log: log,
|
||||
disabled: config.DisableDiskCache,
|
||||
evictionPressure: make(chan struct{}, 1),
|
||||
evictionStop: make(chan struct{}),
|
||||
evictionDone: make(chan struct{}),
|
||||
metaCache: make(map[VariantKey]variantMeta),
|
||||
contentLocks: newContentLock(),
|
||||
}
|
||||
|
||||
if c.disabled {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
srcContent, err := NewContentStorage(filepath.Join(config.StateDir, "cache", "sources"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create source content storage: %w", err)
|
||||
@@ -64,14 +131,11 @@ func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
|
||||
return nil, fmt.Errorf("failed to create source metadata storage: %w", err)
|
||||
}
|
||||
|
||||
return &Cache{
|
||||
db: db,
|
||||
srcContent: srcContent,
|
||||
variants: variants,
|
||||
srcMetadata: srcMetadata,
|
||||
config: config,
|
||||
metaCache: make(map[VariantKey]variantMeta),
|
||||
}, nil
|
||||
c.srcContent = srcContent
|
||||
c.variants = variants
|
||||
c.srcMetadata = srcMetadata
|
||||
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// LookupResult contains the result of a cache lookup.
|
||||
@@ -83,12 +147,15 @@ type LookupResult struct {
|
||||
CacheStatus CacheStatus
|
||||
}
|
||||
|
||||
// Lookup checks if a processed variant exists on disk (no DB access for hits).
|
||||
func (c *Cache) Lookup(_ context.Context, req *ImageRequest) (*LookupResult, error) {
|
||||
// Lookup checks if a processed variant exists on disk. Hits touch the
|
||||
// variant's LRU timestamp; a disabled cache always misses.
|
||||
func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, error) {
|
||||
cacheKey := CacheKey(req)
|
||||
|
||||
// Check variant storage directly - no DB needed for cache hits
|
||||
if c.variants.Exists(cacheKey) {
|
||||
if !c.disabled && c.variants.Exists(cacheKey) {
|
||||
c.touchVariant(ctx, cacheKey)
|
||||
|
||||
return &LookupResult{
|
||||
Hit: true,
|
||||
CacheKey: cacheKey,
|
||||
@@ -103,20 +170,80 @@ func (c *Cache) Lookup(_ context.Context, req *ImageRequest) (*LookupResult, err
|
||||
}, nil
|
||||
}
|
||||
|
||||
// touchVariant updates the LRU timestamp of a variant, best-effort:
|
||||
// a failed touch only makes the entry look colder to eviction.
|
||||
func (c *Cache) touchVariant(ctx context.Context, cacheKey VariantKey) {
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
UPDATE variant_content SET last_accessed_at = CURRENT_TIMESTAMP
|
||||
WHERE cache_key = ?
|
||||
`, string(cacheKey))
|
||||
if err != nil {
|
||||
c.log.Debug("failed to touch variant LRU timestamp",
|
||||
"cache_key", cacheKey, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// touchSourceContent updates the LRU timestamp of a source content
|
||||
// blob, best-effort: a failed touch only makes the blob look colder.
|
||||
func (c *Cache) touchSourceContent(ctx context.Context, contentHash ContentHash) {
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
UPDATE source_content SET last_accessed_at = CURRENT_TIMESTAMP
|
||||
WHERE content_hash = ?
|
||||
`, string(contentHash))
|
||||
if err != nil {
|
||||
c.log.Debug("failed to touch source content LRU timestamp",
|
||||
"content_hash", contentHash, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// GetVariant returns a reader, size, and content type for a cached variant.
|
||||
func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, error) {
|
||||
if c.disabled {
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
return c.variants.LoadWithMeta(cacheKey)
|
||||
}
|
||||
|
||||
// StoreSource stores fetched source content and metadata.
|
||||
// StoreSource stores fetched source content and metadata. On a
|
||||
// disabled cache it is a no-op returning an empty hash.
|
||||
func (c *Cache) StoreSource(
|
||||
ctx context.Context,
|
||||
req *ImageRequest,
|
||||
content io.Reader,
|
||||
result *httpfetcher.FetchResult,
|
||||
) (ContentHash, error) {
|
||||
// Store content
|
||||
contentHash, size, err := c.srcContent.Store(content)
|
||||
if c.disabled {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// Hash the content ourselves (rather than via srcContent.Store,
|
||||
// which would hash internally) so the content hash is known before
|
||||
// any file or database work happens: that lets the entire store be
|
||||
// serialized, per hash, against a concurrent eviction of the same
|
||||
// content below.
|
||||
data, err := io.ReadAll(content)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read source content: %w", err)
|
||||
}
|
||||
|
||||
sum := sha256.Sum256(data)
|
||||
contentHash := ContentHash(hex.EncodeToString(sum[:]))
|
||||
|
||||
// Hold the content hash's lock for the whole store operation. A
|
||||
// concurrent eviction of this exact hash (the real SHA-256 dedup
|
||||
// case: a different source path whose bytes hash identically)
|
||||
// deletes the accounting rows and unlinks the file inside the same
|
||||
// lock, so the two can never interleave: either this store
|
||||
// completes first (and a subsequent eviction removes it together
|
||||
// with its rows and file, correctly), or eviction completes first
|
||||
// (and this store finds the file already gone and recreates it
|
||||
// fresh) — never a fresh row left pointing at a file eviction is
|
||||
// mid-unlink on.
|
||||
unlock := c.contentLocks.Lock(string(contentHash))
|
||||
defer unlock()
|
||||
|
||||
size, err := c.srcContent.StoreHashed(contentHash, data)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to store source content: %w", err)
|
||||
}
|
||||
@@ -171,19 +298,52 @@ func (c *Cache) StoreSource(
|
||||
_ = err
|
||||
}
|
||||
|
||||
c.notifyWritePressure()
|
||||
|
||||
return contentHash, nil
|
||||
}
|
||||
|
||||
// StoreVariant stores a processed variant by its cache key.
|
||||
// StoreVariant stores a processed variant by its cache key and records
|
||||
// it in the size accounting. On a disabled cache it is a no-op. The
|
||||
// accounting insert is best-effort (the startup reconciliation pass
|
||||
// adopts any variant file that misses its accounting row).
|
||||
func (c *Cache) StoreVariant(cacheKey VariantKey, content io.Reader, contentType string) error {
|
||||
_, err := c.variants.Store(cacheKey, content, contentType)
|
||||
if c.disabled {
|
||||
return nil
|
||||
}
|
||||
|
||||
return err
|
||||
size, err := c.variants.Store(cacheKey, content, contentType)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = c.db.Exec(`
|
||||
INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(cache_key) DO UPDATE SET
|
||||
size_bytes = excluded.size_bytes,
|
||||
content_type = excluded.content_type,
|
||||
last_accessed_at = CURRENT_TIMESTAMP
|
||||
`, string(cacheKey), size, contentType)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to record variant in size accounting",
|
||||
"cache_key", cacheKey, "error", err)
|
||||
}
|
||||
|
||||
c.notifyWritePressure()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// LookupSource checks if we have cached source content for a request.
|
||||
// Returns the content hash and content type if found, or empty values if not.
|
||||
// Returns the content hash and content type if found, or empty values
|
||||
// if not. Hits touch the blob's LRU timestamp; a disabled cache always
|
||||
// reports no cached source.
|
||||
func (c *Cache) LookupSource(ctx context.Context, req *ImageRequest) (ContentHash, string, error) {
|
||||
if c.disabled {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
var hashStr, contentType string
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
@@ -206,6 +366,8 @@ func (c *Cache) LookupSource(ctx context.Context, req *ImageRequest) (ContentHas
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
c.touchSourceContent(ctx, contentHash)
|
||||
|
||||
return contentHash, contentType, nil
|
||||
}
|
||||
|
||||
@@ -278,6 +440,10 @@ func (c *Cache) GetSourceMetadataID(ctx context.Context, req *ImageRequest) (int
|
||||
|
||||
// GetSourceContent returns a reader for cached source content by its hash.
|
||||
func (c *Cache) GetSourceContent(contentHash ContentHash) (io.ReadCloser, error) {
|
||||
if c.disabled {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
|
||||
return c.srcContent.Load(contentHash)
|
||||
}
|
||||
|
||||
|
||||
64
internal/imgcache/contentlock.go
Normal file
64
internal/imgcache/contentlock.go
Normal file
@@ -0,0 +1,64 @@
|
||||
package imgcache
|
||||
|
||||
import "sync"
|
||||
|
||||
// contentLock provides per-key mutual exclusion for content-hash keyed
|
||||
// operations. StoreSource and evictSourceBlob each hold a content
|
||||
// hash's lock for the full duration of their file-plus-accounting-row
|
||||
// work, so a store and an eviction racing on identical content bytes
|
||||
// (the real SHA-256 content-addressed dedup case, not a contrived one)
|
||||
// can never interleave: the unlink of an evicted blob's file can never
|
||||
// race the creation of a fresh database row for a concurrently
|
||||
// re-stored copy of the same content. Entries are removed once no
|
||||
// goroutine holds or is waiting for them, so a long-running process
|
||||
// does not accumulate memory proportional to the number of distinct
|
||||
// content hashes it has ever seen.
|
||||
type contentLock struct {
|
||||
mu sync.Mutex
|
||||
entries map[string]*contentLockEntry
|
||||
}
|
||||
|
||||
// contentLockEntry is one key's exclusion lock plus a count of
|
||||
// goroutines currently holding or waiting to acquire it, used to know
|
||||
// when it is safe to remove the entry from the map.
|
||||
type contentLockEntry struct {
|
||||
mu sync.Mutex
|
||||
count int
|
||||
}
|
||||
|
||||
// newContentLock creates an empty contentLock.
|
||||
func newContentLock() *contentLock {
|
||||
return &contentLock{entries: make(map[string]*contentLockEntry)}
|
||||
}
|
||||
|
||||
// Lock acquires exclusive access for key, blocking until it is
|
||||
// available, and returns a function that releases it. The caller must
|
||||
// invoke the returned function exactly once to release the lock.
|
||||
func (c *contentLock) Lock(key string) func() {
|
||||
c.mu.Lock()
|
||||
|
||||
entry, ok := c.entries[key]
|
||||
if !ok {
|
||||
entry = &contentLockEntry{}
|
||||
c.entries[key] = entry
|
||||
}
|
||||
|
||||
entry.count++
|
||||
|
||||
c.mu.Unlock()
|
||||
|
||||
entry.mu.Lock()
|
||||
|
||||
return func() {
|
||||
entry.mu.Unlock()
|
||||
|
||||
c.mu.Lock()
|
||||
|
||||
entry.count--
|
||||
if entry.count == 0 {
|
||||
delete(c.entries, key)
|
||||
}
|
||||
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
133
internal/imgcache/contentlock_test.go
Normal file
133
internal/imgcache/contentlock_test.go
Normal file
@@ -0,0 +1,133 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestContentLockExcludesSameKey verifies that two goroutines locking
|
||||
// the same key never run their critical sections concurrently.
|
||||
func TestContentLockExcludesSameKey(t *testing.T) {
|
||||
lock := newContentLock()
|
||||
|
||||
var (
|
||||
active int32
|
||||
maxSeen int32
|
||||
wg sync.WaitGroup
|
||||
)
|
||||
|
||||
const goroutines = 20
|
||||
|
||||
wg.Add(goroutines)
|
||||
|
||||
for i := 0; i < goroutines; i++ {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
|
||||
unlock := lock.Lock("same-key")
|
||||
defer unlock()
|
||||
|
||||
n := atomic.AddInt32(&active, 1)
|
||||
|
||||
for {
|
||||
seen := atomic.LoadInt32(&maxSeen)
|
||||
if n <= seen || atomic.CompareAndSwapInt32(&maxSeen, seen, n) {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
|
||||
atomic.AddInt32(&active, -1)
|
||||
}()
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
|
||||
if maxSeen != 1 {
|
||||
t.Errorf("max concurrent holders of the same key = %d, want 1", maxSeen)
|
||||
}
|
||||
}
|
||||
|
||||
// TestContentLockAllowsDifferentKeys verifies that locking distinct
|
||||
// keys does not serialize unrelated work: all goroutines must be able
|
||||
// to enter their critical sections at once, proven by every one of
|
||||
// them reaching the rendezvous point before any is allowed to
|
||||
// proceed.
|
||||
func TestContentLockAllowsDifferentKeys(t *testing.T) {
|
||||
lock := newContentLock()
|
||||
|
||||
const goroutines = 20
|
||||
|
||||
var (
|
||||
wg sync.WaitGroup
|
||||
inside int32
|
||||
reached = make(chan struct{}, goroutines)
|
||||
)
|
||||
|
||||
wg.Add(goroutines)
|
||||
|
||||
release := make(chan struct{})
|
||||
|
||||
for i := 0; i < goroutines; i++ {
|
||||
key := string(rune('a' + i))
|
||||
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
|
||||
unlock := lock.Lock(key)
|
||||
defer unlock()
|
||||
|
||||
atomic.AddInt32(&inside, 1)
|
||||
reached <- struct{}{}
|
||||
<-release
|
||||
}()
|
||||
}
|
||||
|
||||
// Every goroutine must reach the rendezvous point (i.e. acquire its
|
||||
// own key's lock) without needing any other to release first. If
|
||||
// keys were incorrectly serialized onto one underlying lock, only
|
||||
// one would get here and this would time out.
|
||||
for i := 0; i < goroutines; i++ {
|
||||
select {
|
||||
case <-reached:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatalf("only %d/%d goroutines locking distinct keys made progress; "+
|
||||
"keys may be incorrectly serialized", i, goroutines)
|
||||
}
|
||||
}
|
||||
|
||||
if n := atomic.LoadInt32(&inside); n != goroutines {
|
||||
t.Errorf("goroutines inside their critical section = %d, want %d", n, goroutines)
|
||||
}
|
||||
|
||||
close(release)
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// TestContentLockRemovesEntryAfterUnlock verifies that the internal
|
||||
// entries map does not grow without bound: once no goroutine holds or
|
||||
// awaits a key, its entry is removed.
|
||||
func TestContentLockRemovesEntryAfterUnlock(t *testing.T) {
|
||||
lock := newContentLock()
|
||||
|
||||
unlock := lock.Lock("k")
|
||||
|
||||
lock.mu.Lock()
|
||||
if _, ok := lock.entries["k"]; !ok {
|
||||
lock.mu.Unlock()
|
||||
t.Fatal("entry missing while lock is held")
|
||||
}
|
||||
lock.mu.Unlock()
|
||||
|
||||
unlock()
|
||||
|
||||
lock.mu.Lock()
|
||||
defer lock.mu.Unlock()
|
||||
|
||||
if _, ok := lock.entries["k"]; ok {
|
||||
t.Error("entry for key still present after the last holder unlocked")
|
||||
}
|
||||
}
|
||||
776
internal/imgcache/eviction.go
Normal file
776
internal/imgcache/eviction.go
Normal file
@@ -0,0 +1,776 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DefaultEvictionInterval is how often the background evictor checks
|
||||
// cache usage against the configured limit, in addition to the
|
||||
// write-pressure wakeups triggered by stores.
|
||||
const DefaultEvictionInterval = 5 * time.Minute
|
||||
|
||||
// evictionBatchSize is how many LRU candidates of each class (variants
|
||||
// and source blobs) one eviction pass fetches from the database.
|
||||
const evictionBatchSize = 100
|
||||
|
||||
// staleTempFileAge is how old an orphaned temp file (left behind by a
|
||||
// crashed write) must be before reconciliation removes it. Fresh temp
|
||||
// files may still belong to an in-flight store.
|
||||
const staleTempFileAge = time.Hour
|
||||
|
||||
// sqliteTimestampLayout matches SQLite's CURRENT_TIMESTAMP format, so
|
||||
// timestamps written by reconciliation order correctly against ones
|
||||
// written by the hot path.
|
||||
const sqliteTimestampLayout = "2006-01-02 15:04:05"
|
||||
|
||||
// tempFilePrefix is the prefix os.CreateTemp uses for in-flight cache
|
||||
// writes (".tmp-*" patterns in the storage layer).
|
||||
const tempFilePrefix = ".tmp-"
|
||||
|
||||
// variantMetaSuffix is the sidecar suffix VariantStorage writes next
|
||||
// to each variant file.
|
||||
const variantMetaSuffix = ".meta"
|
||||
|
||||
// fallbackContentType is recorded when a reconciled variant file has
|
||||
// no readable .meta sidecar.
|
||||
const fallbackContentType = "application/octet-stream"
|
||||
|
||||
// UsageBytes returns the total number of bytes of cache content
|
||||
// tracked in the database (source content blobs plus processed
|
||||
// variants). It never scans the cache directories.
|
||||
func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
|
||||
if c.disabled {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
var total int64
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT (SELECT COALESCE(SUM(size_bytes), 0) FROM source_content)
|
||||
+ (SELECT COALESCE(SUM(size_bytes), 0) FROM variant_content)
|
||||
`).Scan(&total)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to compute cache usage: %w", err)
|
||||
}
|
||||
|
||||
return total, nil
|
||||
}
|
||||
|
||||
// evictionCandidate is one LRU eviction victim candidate: either a
|
||||
// processed variant (isVariant true, identified by cacheKey) or a
|
||||
// source content blob (identified by contentHash).
|
||||
type evictionCandidate struct {
|
||||
isVariant bool
|
||||
cacheKey VariantKey
|
||||
contentHash ContentHash
|
||||
sizeBytes int64
|
||||
lastAccessedAt string
|
||||
}
|
||||
|
||||
// EvictToLimit evicts least-recently-used cache entries until total
|
||||
// tracked usage is at or below the configured MaxBytes limit. It is a
|
||||
// no-op when the cache is disabled or no limit is configured.
|
||||
func (c *Cache) EvictToLimit(ctx context.Context) error {
|
||||
if c.disabled || c.config.MaxBytes <= 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
for {
|
||||
usage, err := c.UsageBytes(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if usage <= c.config.MaxBytes {
|
||||
return nil
|
||||
}
|
||||
|
||||
freed, err := c.evictBatch(ctx, usage-c.config.MaxBytes)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if freed == 0 {
|
||||
c.log.Warn("cache eviction made no progress",
|
||||
"usage_bytes", usage,
|
||||
"cache_max_bytes", c.config.MaxBytes,
|
||||
)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
c.log.Info("evicted cache content",
|
||||
"freed_bytes", freed,
|
||||
"usage_bytes", usage-freed,
|
||||
"cache_max_bytes", c.config.MaxBytes,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// evictBatch fetches one batch of LRU candidates across variants and
|
||||
// source blobs and evicts them oldest-first until excessBytes are
|
||||
// freed or the batch is exhausted. It returns the bytes freed.
|
||||
func (c *Cache) evictBatch(ctx context.Context, excessBytes int64) (int64, error) {
|
||||
candidates, err := c.evictionCandidates(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
var freed int64
|
||||
|
||||
for _, candidate := range candidates {
|
||||
if freed >= excessBytes {
|
||||
break
|
||||
}
|
||||
|
||||
if err := c.evictCandidate(ctx, candidate); err != nil {
|
||||
c.log.Warn("failed to evict cache entry",
|
||||
"cache_key", candidate.cacheKey,
|
||||
"content_hash", candidate.contentHash,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
freed += candidate.sizeBytes
|
||||
}
|
||||
|
||||
return freed, nil
|
||||
}
|
||||
|
||||
// evictCandidate removes a single eviction victim.
|
||||
func (c *Cache) evictCandidate(ctx context.Context, candidate evictionCandidate) error {
|
||||
if candidate.isVariant {
|
||||
return c.evictVariant(ctx, candidate.cacheKey)
|
||||
}
|
||||
|
||||
return c.evictSourceBlob(ctx, candidate.contentHash)
|
||||
}
|
||||
|
||||
// evictionCandidates returns up to evictionBatchSize variants and
|
||||
// evictionBatchSize source blobs, merged into a single list ordered by
|
||||
// last access time (oldest first).
|
||||
func (c *Cache) evictionCandidates(ctx context.Context) ([]evictionCandidate, error) {
|
||||
variants, err := c.variantCandidates(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
sources, err := c.sourceCandidates(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Merge the two lists, each already sorted oldest-first. SQLite
|
||||
// CURRENT_TIMESTAMP strings compare correctly lexicographically.
|
||||
merged := make([]evictionCandidate, 0, len(variants)+len(sources))
|
||||
|
||||
for len(variants) > 0 && len(sources) > 0 {
|
||||
if variants[0].lastAccessedAt <= sources[0].lastAccessedAt {
|
||||
merged = append(merged, variants[0])
|
||||
variants = variants[1:]
|
||||
} else {
|
||||
merged = append(merged, sources[0])
|
||||
sources = sources[1:]
|
||||
}
|
||||
}
|
||||
|
||||
merged = append(merged, variants...)
|
||||
merged = append(merged, sources...)
|
||||
|
||||
return merged, nil
|
||||
}
|
||||
|
||||
// variantCandidates returns the least recently used variants.
|
||||
func (c *Cache) variantCandidates(ctx context.Context) ([]evictionCandidate, error) {
|
||||
rows, err := c.db.QueryContext(ctx, `
|
||||
SELECT cache_key, size_bytes, last_accessed_at
|
||||
FROM variant_content
|
||||
ORDER BY last_accessed_at ASC, cache_key ASC
|
||||
LIMIT ?
|
||||
`, evictionBatchSize)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query variant eviction candidates: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var candidates []evictionCandidate
|
||||
|
||||
for rows.Next() {
|
||||
candidate := evictionCandidate{isVariant: true}
|
||||
|
||||
var key string
|
||||
if err := rows.Scan(&key, &candidate.sizeBytes, &candidate.lastAccessedAt); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan variant candidate: %w", err)
|
||||
}
|
||||
|
||||
candidate.cacheKey = VariantKey(key)
|
||||
candidates = append(candidates, candidate)
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("variant candidate iteration failed: %w", err)
|
||||
}
|
||||
|
||||
return candidates, nil
|
||||
}
|
||||
|
||||
// sourceCandidates returns the least recently used source blobs. Rows
|
||||
// written before the LRU column existed fall back to fetched_at.
|
||||
func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, error) {
|
||||
rows, err := c.db.QueryContext(ctx, `
|
||||
SELECT content_hash, size_bytes,
|
||||
COALESCE(last_accessed_at, fetched_at, '1970-01-01 00:00:00') AS lru
|
||||
FROM source_content
|
||||
ORDER BY lru ASC, content_hash ASC
|
||||
LIMIT ?
|
||||
`, evictionBatchSize)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query source eviction candidates: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var candidates []evictionCandidate
|
||||
|
||||
for rows.Next() {
|
||||
var candidate evictionCandidate
|
||||
|
||||
var hash string
|
||||
if err := rows.Scan(&hash, &candidate.sizeBytes, &candidate.lastAccessedAt); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan source candidate: %w", err)
|
||||
}
|
||||
|
||||
candidate.contentHash = ContentHash(hash)
|
||||
candidates = append(candidates, candidate)
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("source candidate iteration failed: %w", err)
|
||||
}
|
||||
|
||||
return candidates, nil
|
||||
}
|
||||
|
||||
// evictVariant removes one variant: accounting row first, then the
|
||||
// content and .meta files, so the database never references a deleted
|
||||
// file.
|
||||
func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM variant_content WHERE cache_key = ?`, string(cacheKey))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to delete variant accounting row: %w", err)
|
||||
}
|
||||
|
||||
if err := c.variants.DeleteWithMeta(cacheKey); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// sourceReference identifies one source_metadata row's JSON sidecar.
|
||||
type sourceReference struct {
|
||||
host string
|
||||
pathHash PathHash
|
||||
}
|
||||
|
||||
// evictSourceBlob removes one source content blob. All source_metadata
|
||||
// rows referencing the blob are deleted together with its
|
||||
// source_content row in a single transaction BEFORE the file is
|
||||
// unlinked: a blob referenced by multiple source paths is only ever
|
||||
// removed together with all of its references, and database rows never
|
||||
// point at deleted files. The JSON metadata sidecars for the removed
|
||||
// rows are deleted afterwards.
|
||||
//
|
||||
// The whole operation holds the content hash's lock (the same one
|
||||
// StoreSource holds for its full store), so a concurrent store of
|
||||
// identical content bytes can never observe the file gone but a row
|
||||
// still present, or insert a fresh row between this transaction's
|
||||
// commit and the file unlink below: it either runs entirely before
|
||||
// this eviction starts, or is blocked until this eviction (row
|
||||
// deletion and unlink together) has fully completed.
|
||||
func (c *Cache) evictSourceBlob(ctx context.Context, contentHash ContentHash) error {
|
||||
unlock := c.contentLocks.Lock(string(contentHash))
|
||||
defer unlock()
|
||||
|
||||
references, err := c.sourceReferences(ctx, contentHash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tx, err := c.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to begin eviction transaction: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
if _, err := tx.ExecContext(ctx,
|
||||
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash)); err != nil {
|
||||
return fmt.Errorf("failed to delete source metadata rows: %w", err)
|
||||
}
|
||||
|
||||
if _, err := tx.ExecContext(ctx,
|
||||
`DELETE FROM source_content WHERE content_hash = ?`, string(contentHash)); err != nil {
|
||||
return fmt.Errorf("failed to delete source content row: %w", err)
|
||||
}
|
||||
|
||||
if err := tx.Commit(); err != nil {
|
||||
return fmt.Errorf("failed to commit eviction transaction: %w", err)
|
||||
}
|
||||
|
||||
if c.evictSourceBlobTestHook != nil {
|
||||
c.evictSourceBlobTestHook(contentHash)
|
||||
}
|
||||
|
||||
// Only after the rows are gone may the files be removed.
|
||||
for _, reference := range references {
|
||||
if err := c.srcMetadata.Delete(reference.host, reference.pathHash); err != nil {
|
||||
c.log.Warn("failed to delete metadata sidecar",
|
||||
"host", reference.host, "path_hash", reference.pathHash, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := c.srcContent.Delete(contentHash); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// sourceReferences lists the metadata sidecar locations of every
|
||||
// source_metadata row referencing the given blob.
|
||||
func (c *Cache) sourceReferences(
|
||||
ctx context.Context, contentHash ContentHash,
|
||||
) ([]sourceReference, error) {
|
||||
rows, err := c.db.QueryContext(ctx, `
|
||||
SELECT source_host, path_hash FROM source_metadata WHERE content_hash = ?
|
||||
`, string(contentHash))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query source references: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var references []sourceReference
|
||||
|
||||
for rows.Next() {
|
||||
var reference sourceReference
|
||||
|
||||
var pathHash string
|
||||
if err := rows.Scan(&reference.host, &pathHash); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan source reference: %w", err)
|
||||
}
|
||||
|
||||
reference.pathHash = PathHash(pathHash)
|
||||
references = append(references, reference)
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("source reference iteration failed: %w", err)
|
||||
}
|
||||
|
||||
return references, nil
|
||||
}
|
||||
|
||||
// notifyWritePressure wakes the background evictor after a store, so
|
||||
// eviction under write pressure happens promptly without blocking the
|
||||
// storing request. The notification channel has capacity one and drops
|
||||
// when a wakeup is already pending.
|
||||
func (c *Cache) notifyWritePressure() {
|
||||
if c.disabled || c.config.MaxBytes <= 0 {
|
||||
return
|
||||
}
|
||||
|
||||
select {
|
||||
case c.evictionPressure <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// StartEviction launches the background eviction goroutine, which
|
||||
// reconciles the database accounting with the cache directories at
|
||||
// startup and again on every periodic tick thereafter, and evicts to
|
||||
// the configured limit on the given periodic interval and on
|
||||
// write-pressure notifications. It is a no-op on a disabled cache or
|
||||
// when already started.
|
||||
func (c *Cache) StartEviction(interval time.Duration) {
|
||||
if c.disabled || c.evictionStarted {
|
||||
return
|
||||
}
|
||||
|
||||
c.evictionStarted = true
|
||||
|
||||
go c.evictionLoop(interval)
|
||||
}
|
||||
|
||||
// StopEviction stops the background eviction goroutine and waits for
|
||||
// it to exit. It is safe to call when eviction was never started, and
|
||||
// safe to call more than once.
|
||||
func (c *Cache) StopEviction() {
|
||||
if !c.evictionStarted {
|
||||
return
|
||||
}
|
||||
|
||||
c.evictionStopOnce.Do(func() {
|
||||
close(c.evictionStop)
|
||||
<-c.evictionDone
|
||||
})
|
||||
}
|
||||
|
||||
// evictionLoop is the body of the background eviction goroutine.
|
||||
func (c *Cache) evictionLoop(interval time.Duration) {
|
||||
defer close(c.evictionDone)
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
c.runReconciliationPass(ctx)
|
||||
c.runEvictionPass(ctx)
|
||||
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-c.evictionStop:
|
||||
return
|
||||
case <-ticker.C:
|
||||
// Reconciliation walks the cache directories, so it only
|
||||
// runs on the periodic ticker rather than on every
|
||||
// write-pressure wakeup, keeping it off the per-store hot
|
||||
// path. Reusing the eviction interval itself (rather than a
|
||||
// separate, longer one) is a deliberate choice: it is the
|
||||
// simplest option that still bounds how long a store's
|
||||
// best-effort accounting insert can stay silently
|
||||
// unaccounted for to one interval, on a process that is
|
||||
// already running this loop regardless.
|
||||
c.runReconciliationPass(ctx)
|
||||
case <-c.evictionPressure:
|
||||
}
|
||||
|
||||
c.runEvictionPass(ctx)
|
||||
}
|
||||
}
|
||||
|
||||
// runEvictionPass runs one eviction pass, logging failures instead of
|
||||
// propagating them (the loop must keep running).
|
||||
func (c *Cache) runEvictionPass(ctx context.Context) {
|
||||
if err := c.EvictToLimit(ctx); err != nil {
|
||||
c.log.Warn("cache eviction pass failed", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// runReconciliationPass runs one reconciliation pass, logging failures
|
||||
// instead of propagating them (the loop must keep running).
|
||||
func (c *Cache) runReconciliationPass(ctx context.Context) {
|
||||
if err := c.reconcileAccounting(ctx); err != nil {
|
||||
c.log.Warn("cache accounting reconciliation failed", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// reconcileAccounting synchronizes the database size accounting with
|
||||
// the actual contents of the cache directories. It runs at startup and
|
||||
// again on every periodic eviction tick thereafter, off the request
|
||||
// hot path: it adopts variant files that predate the accounting table
|
||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||
// insert under transient DB contention), drops accounting rows whose
|
||||
// files are missing, removes source blob files the database does not
|
||||
// know (and rows whose files are gone), and sweeps stale temp files
|
||||
// left behind by crashed writes. Running it periodically, not just
|
||||
// once, bounds how long such drift can accumulate unaccounted for on a
|
||||
// long-running process to one eviction interval.
|
||||
func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||
if c.disabled {
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := c.reconcileVariantFiles(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := c.reconcileVariantRows(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := c.reconcileSourceFiles(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := c.reconcileSourceRows(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// reconcileVariantFiles walks the variant storage directory, adopting
|
||||
// files without accounting rows and sweeping stale temp files.
|
||||
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||
return filepath.WalkDir(c.variants.baseDir, func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
|
||||
name := entry.Name()
|
||||
|
||||
if strings.HasPrefix(name, tempFilePrefix) {
|
||||
c.sweepStaleTempFile(path, entry)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if strings.HasSuffix(name, variantMetaSuffix) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return c.adoptVariantFile(ctx, path, entry, VariantKey(name))
|
||||
})
|
||||
}
|
||||
|
||||
// adoptVariantFile inserts an accounting row for a variant file that
|
||||
// has none, using the file's size and modification time.
|
||||
func (c *Cache) adoptVariantFile(
|
||||
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
|
||||
) error {
|
||||
var rowExists int
|
||||
|
||||
err := c.db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, string(cacheKey),
|
||||
).Scan(&rowExists)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check variant accounting row: %w", err)
|
||||
}
|
||||
|
||||
if rowExists > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to stat variant file: %w", err)
|
||||
}
|
||||
|
||||
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
|
||||
contentType := c.variantContentTypeFromSidecar(path)
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO variant_content
|
||||
(cache_key, size_bytes, content_type, created_at, last_accessed_at)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
`, string(cacheKey), info.Size(), contentType, modTime, modTime)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to adopt variant file into accounting: %w", err)
|
||||
}
|
||||
|
||||
c.log.Info("adopted untracked variant file into size accounting",
|
||||
"cache_key", cacheKey, "size_bytes", info.Size())
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// variantContentTypeFromSidecar reads the content type from a variant
|
||||
// .meta sidecar, falling back to application/octet-stream.
|
||||
func (c *Cache) variantContentTypeFromSidecar(variantPath string) string {
|
||||
metaData, err := os.ReadFile(variantPath + variantMetaSuffix) //nolint:gosec // path from cache walk
|
||||
if err != nil {
|
||||
return fallbackContentType
|
||||
}
|
||||
|
||||
var meta VariantMeta
|
||||
if json.Unmarshal(metaData, &meta) != nil || meta.ContentType == "" {
|
||||
return fallbackContentType
|
||||
}
|
||||
|
||||
return meta.ContentType
|
||||
}
|
||||
|
||||
// reconcileVariantRows drops accounting rows whose variant files are
|
||||
// missing, so the database never references deleted content.
|
||||
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||
keys, err := c.allVariantKeys(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, key := range keys {
|
||||
if c.variants.Exists(key) {
|
||||
continue
|
||||
}
|
||||
|
||||
if _, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM variant_content WHERE cache_key = ?`, string(key)); err != nil {
|
||||
return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
|
||||
}
|
||||
|
||||
c.log.Info("dropped accounting row for missing variant file", "cache_key", key)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// allVariantKeys returns every tracked variant cache key.
|
||||
func (c *Cache) allVariantKeys(ctx context.Context) ([]VariantKey, error) {
|
||||
rows, err := c.db.QueryContext(ctx, `SELECT cache_key FROM variant_content`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query variant keys: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var keys []VariantKey
|
||||
|
||||
for rows.Next() {
|
||||
var key string
|
||||
if err := rows.Scan(&key); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan variant key: %w", err)
|
||||
}
|
||||
|
||||
keys = append(keys, VariantKey(key))
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("variant key iteration failed: %w", err)
|
||||
}
|
||||
|
||||
return keys, nil
|
||||
}
|
||||
|
||||
// reconcileSourceFiles walks the source content directory, removing
|
||||
// blob files the database does not track (they are unreachable: source
|
||||
// lookups always go through source_metadata) and sweeping stale temp
|
||||
// files.
|
||||
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
||||
return filepath.WalkDir(c.srcContent.baseDir, func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
|
||||
name := entry.Name()
|
||||
|
||||
if strings.HasPrefix(name, tempFilePrefix) {
|
||||
c.sweepStaleTempFile(path, entry)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return c.removeUntrackedSourceFile(ctx, path, ContentHash(name))
|
||||
})
|
||||
}
|
||||
|
||||
// removeUntrackedSourceFile deletes a source blob file that has no
|
||||
// source_content row. Any source_metadata rows referencing the hash
|
||||
// are removed first so no row ever points at a deleted file.
|
||||
func (c *Cache) removeUntrackedSourceFile(
|
||||
ctx context.Context, path string, contentHash ContentHash,
|
||||
) error {
|
||||
var rowExists int
|
||||
|
||||
err := c.db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(contentHash),
|
||||
).Scan(&rowExists)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check source content row: %w", err)
|
||||
}
|
||||
|
||||
if rowExists > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
if _, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash)); err != nil {
|
||||
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
|
||||
}
|
||||
|
||||
//nolint:gosec // G703: path comes from walking our own cache directory
|
||||
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to remove untracked source file: %w", err)
|
||||
}
|
||||
|
||||
c.log.Info("removed untracked source content file", "content_hash", contentHash)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// reconcileSourceRows removes source_content rows (and their metadata
|
||||
// references and sidecars) whose blob files are missing on disk.
|
||||
func (c *Cache) reconcileSourceRows(ctx context.Context) error {
|
||||
hashes, err := c.allSourceContentHashes(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, hash := range hashes {
|
||||
if c.srcContent.Exists(hash) {
|
||||
continue
|
||||
}
|
||||
|
||||
// The blob file is already gone; evictSourceBlob removes the
|
||||
// rows and sidecars and tolerates the missing file.
|
||||
if err := c.evictSourceBlob(ctx, hash); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
c.log.Info("dropped rows for missing source content file", "content_hash", hash)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// allSourceContentHashes returns every tracked source content hash.
|
||||
func (c *Cache) allSourceContentHashes(ctx context.Context) ([]ContentHash, error) {
|
||||
rows, err := c.db.QueryContext(ctx, `SELECT content_hash FROM source_content`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query source content hashes: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var hashes []ContentHash
|
||||
|
||||
for rows.Next() {
|
||||
var hash string
|
||||
if err := rows.Scan(&hash); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan content hash: %w", err)
|
||||
}
|
||||
|
||||
hashes = append(hashes, ContentHash(hash))
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("content hash iteration failed: %w", err)
|
||||
}
|
||||
|
||||
return hashes, nil
|
||||
}
|
||||
|
||||
// sweepStaleTempFile removes a temp file left behind by a crashed
|
||||
// write once it is old enough that no in-flight store can own it.
|
||||
func (c *Cache) sweepStaleTempFile(path string, entry fs.DirEntry) {
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Since(info.ModTime()) < staleTempFileAge {
|
||||
return
|
||||
}
|
||||
|
||||
//nolint:gosec // G703: path comes from walking our own cache directory
|
||||
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
|
||||
c.log.Warn("failed to remove stale temp file", "path", path, "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
c.log.Info("removed stale temp file", "path", path)
|
||||
}
|
||||
837
internal/imgcache/eviction_test.go
Normal file
837
internal/imgcache/eviction_test.go
Normal file
@@ -0,0 +1,837 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
)
|
||||
|
||||
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
|
||||
// produces, so injected timestamps compare correctly against ones the
|
||||
// implementation writes.
|
||||
const sqliteTimestampFormat = "2006-01-02 15:04:05"
|
||||
|
||||
// evictionTestDB creates an in-memory SQLite database with the real
|
||||
// production schema, limited to a single connection so the background
|
||||
// eviction goroutine shares the same in-memory database as the test.
|
||||
func evictionTestDB(t *testing.T) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
db, err := sql.Open("sqlite", ":memory:")
|
||||
if err != nil {
|
||||
t.Fatalf("failed to open test db: %v", err)
|
||||
}
|
||||
|
||||
db.SetMaxOpenConns(1)
|
||||
|
||||
if err := database.ApplyMigrations(context.Background(), db, nil); err != nil {
|
||||
t.Fatalf("failed to apply migrations: %v", err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
// newEvictionTestCache creates a Cache backed by a temp directory and
|
||||
// an in-memory database, with the given size limit.
|
||||
func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
db := evictionTestDB(t)
|
||||
|
||||
// maxBytes zero mirrors the production mapping of
|
||||
// cache_max_bytes: 0 (handlers sets DisableDiskCache); at the
|
||||
// CacheConfig layer itself a zero MaxBytes means "no limit" for
|
||||
// backwards compatibility with existing fixtures.
|
||||
cache, err := NewCache(db, CacheConfig{
|
||||
StateDir: tmpDir,
|
||||
CacheTTL: time.Hour,
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: maxBytes,
|
||||
DisableDiskCache: maxBytes == 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
|
||||
return cache, tmpDir
|
||||
}
|
||||
|
||||
// storeEvictionTestSource stores content as a fetched source for
|
||||
// host/path and returns the resulting content hash.
|
||||
func storeEvictionTestSource(
|
||||
t *testing.T, cache *Cache, host, path string, content []byte,
|
||||
) ContentHash {
|
||||
t.Helper()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: host,
|
||||
SourcePath: path,
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
|
||||
result := &httpfetcher.FetchResult{
|
||||
StatusCode: 200,
|
||||
ContentType: "image/jpeg",
|
||||
ContentLength: int64(len(content)),
|
||||
Headers: map[string][]string{"Content-Type": {"image/jpeg"}},
|
||||
}
|
||||
|
||||
hash, err := cache.StoreSource(context.Background(), req, bytes.NewReader(content), result)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err)
|
||||
}
|
||||
|
||||
return hash
|
||||
}
|
||||
|
||||
// storeEvictionTestVariant stores content as a processed variant under
|
||||
// the given cache key.
|
||||
func storeEvictionTestVariant(t *testing.T, cache *Cache, key VariantKey, content []byte) {
|
||||
t.Helper()
|
||||
|
||||
if err := cache.StoreVariant(key, bytes.NewReader(content), "image/webp"); err != nil {
|
||||
t.Fatalf("StoreVariant(%s) failed: %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
// setVariantLastAccessed backdates the last access time of a tracked
|
||||
// variant, to make LRU ordering deterministic in tests.
|
||||
func setVariantLastAccessed(t *testing.T, cache *Cache, key VariantKey, when time.Time) {
|
||||
t.Helper()
|
||||
|
||||
res, err := cache.db.Exec(
|
||||
`UPDATE variant_content SET last_accessed_at = ? WHERE cache_key = ?`,
|
||||
when.UTC().Format(sqliteTimestampFormat), string(key),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set variant last_accessed_at: %v", err)
|
||||
}
|
||||
|
||||
affected, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read affected rows: %v", err)
|
||||
}
|
||||
|
||||
if affected != 1 {
|
||||
t.Fatalf("variant %s has no accounting row (affected=%d); "+
|
||||
"stores must track variants in the database", key, affected)
|
||||
}
|
||||
}
|
||||
|
||||
// setSourceLastAccessed backdates the last access time of a tracked
|
||||
// source content blob.
|
||||
func setSourceLastAccessed(t *testing.T, cache *Cache, hash ContentHash, when time.Time) {
|
||||
t.Helper()
|
||||
|
||||
res, err := cache.db.Exec(
|
||||
`UPDATE source_content SET last_accessed_at = ? WHERE content_hash = ?`,
|
||||
when.UTC().Format(sqliteTimestampFormat), string(hash),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set source last_accessed_at: %v", err)
|
||||
}
|
||||
|
||||
affected, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read affected rows: %v", err)
|
||||
}
|
||||
|
||||
if affected != 1 {
|
||||
t.Fatalf("source %s has no accounting row (affected=%d)", hash, affected)
|
||||
}
|
||||
}
|
||||
|
||||
// countRows returns the number of rows the given query yields.
|
||||
func countRows(t *testing.T, cache *Cache, query string, args ...interface{}) int {
|
||||
t.Helper()
|
||||
|
||||
var n int
|
||||
if err := cache.db.QueryRow(query, args...).Scan(&n); err != nil {
|
||||
t.Fatalf("count query %q failed: %v", query, err)
|
||||
}
|
||||
|
||||
return n
|
||||
}
|
||||
|
||||
// assertNoDanglingReferences verifies the core eviction invariant:
|
||||
// every database row that references cache content on disk points at a
|
||||
// file that actually exists.
|
||||
func assertNoDanglingReferences(t *testing.T, cache *Cache) {
|
||||
t.Helper()
|
||||
|
||||
rows, err := cache.db.Query(
|
||||
`SELECT content_hash FROM source_metadata
|
||||
WHERE content_hash IS NOT NULL AND content_hash != ''`,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to query source_metadata: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
var hash string
|
||||
if err := rows.Scan(&hash); err != nil {
|
||||
t.Fatalf("failed to scan content_hash: %v", err)
|
||||
}
|
||||
|
||||
if !cache.srcContent.Exists(ContentHash(hash)) {
|
||||
t.Errorf("source_metadata references content %s but the file is missing", hash)
|
||||
}
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
t.Fatalf("source_metadata iteration failed: %v", err)
|
||||
}
|
||||
|
||||
variantRows, err := cache.db.Query(`SELECT cache_key FROM variant_content`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to query variant_content: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = variantRows.Close() }()
|
||||
|
||||
for variantRows.Next() {
|
||||
var key string
|
||||
if err := variantRows.Scan(&key); err != nil {
|
||||
t.Fatalf("failed to scan cache_key: %v", err)
|
||||
}
|
||||
|
||||
if !cache.variants.Exists(VariantKey(key)) {
|
||||
t.Errorf("variant_content references key %s but the file is missing", key)
|
||||
}
|
||||
}
|
||||
|
||||
if err := variantRows.Err(); err != nil {
|
||||
t.Fatalf("variant_content iteration failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// waitForUsageAtOrBelow polls UsageBytes until it reaches limit or the
|
||||
// timeout expires, returning the last observed usage.
|
||||
func waitForUsageAtOrBelow(t *testing.T, cache *Cache, limit int64, timeout time.Duration) int64 {
|
||||
t.Helper()
|
||||
|
||||
deadline := time.Now().Add(timeout)
|
||||
|
||||
var usage int64
|
||||
|
||||
for time.Now().Before(deadline) {
|
||||
var err error
|
||||
|
||||
usage, err = cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage <= limit {
|
||||
return usage
|
||||
}
|
||||
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
|
||||
return usage
|
||||
}
|
||||
|
||||
func TestUsageBytesAccountsSourceAndVariantBytes(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg",
|
||||
bytes.Repeat([]byte{0xAA}, 1000))
|
||||
storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg",
|
||||
bytes.Repeat([]byte{0xAB}, 2000))
|
||||
storeEvictionTestVariant(t, cache, "aabbccdd0001", bytes.Repeat([]byte{0xAC}, 500))
|
||||
storeEvictionTestVariant(t, cache, "aabbccdd0002", bytes.Repeat([]byte{0xAD}, 250))
|
||||
|
||||
usage, err := cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage != 3750 {
|
||||
t.Errorf("UsageBytes = %d, want 3750 (1000+2000+500+250)", usage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUsageBytesCountsMultiReferencedBlobOnce(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
content := bytes.Repeat([]byte{0xCC}, 1200)
|
||||
|
||||
hashOne := storeEvictionTestSource(t, cache, "src.example.com", "/one.jpg", content)
|
||||
hashTwo := storeEvictionTestSource(t, cache, "src.example.com", "/two.jpg", content)
|
||||
|
||||
if hashOne != hashTwo {
|
||||
t.Fatalf("identical content produced different hashes: %s vs %s", hashOne, hashTwo)
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage != 1200 {
|
||||
t.Errorf("UsageBytes = %d, want 1200 (deduplicated blob counted once)", usage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
const limit = 3000
|
||||
|
||||
cache, _ := newEvictionTestCache(t, limit)
|
||||
|
||||
now := time.Now()
|
||||
keys := []VariantKey{"aabbccdd0001", "aabbccdd0002", "aabbccdd0003", "aabbccdd0004"}
|
||||
fills := []byte{0x01, 0x02, 0x03, 0x04}
|
||||
ages := []time.Duration{4 * time.Hour, 3 * time.Hour, 2 * time.Hour, 1 * time.Hour}
|
||||
|
||||
for i, key := range keys {
|
||||
storeEvictionTestVariant(t, cache, key, bytes.Repeat([]byte{fills[i]}, 1000))
|
||||
setVariantLastAccessed(t, cache, key, now.Add(-ages[i]))
|
||||
}
|
||||
|
||||
if err := cache.EvictToLimit(context.Background()); err != nil {
|
||||
t.Fatalf("EvictToLimit failed: %v", err)
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage > limit {
|
||||
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
|
||||
}
|
||||
|
||||
if cache.variants.Exists(keys[0]) {
|
||||
t.Errorf("least recently used variant %s must be evicted", keys[0])
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, string(keys[0]),
|
||||
); n != 0 {
|
||||
t.Errorf("evicted variant %s still has %d accounting rows", keys[0], n)
|
||||
}
|
||||
|
||||
for _, key := range keys[1:] {
|
||||
if !cache.variants.Exists(key) {
|
||||
t.Errorf("more recently used variant %s must survive eviction", key)
|
||||
}
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
func TestEvictionRemovesMultiReferencedBlobTogetherWithAllReferences(t *testing.T) {
|
||||
const limit = 1000
|
||||
|
||||
cache, _ := newEvictionTestCache(t, limit)
|
||||
|
||||
now := time.Now()
|
||||
|
||||
// One 800-byte blob referenced by two source paths.
|
||||
sharedContent := bytes.Repeat([]byte{0xDD}, 800)
|
||||
sharedHash := storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg", sharedContent)
|
||||
|
||||
if h := storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg", sharedContent); h != sharedHash {
|
||||
t.Fatalf("identical content produced different hashes: %s vs %s", h, sharedHash)
|
||||
}
|
||||
|
||||
// A newer 600-byte blob referenced by one source path.
|
||||
recentHash := storeEvictionTestSource(t, cache, "src.example.com", "/c.jpg",
|
||||
bytes.Repeat([]byte{0xEE}, 600))
|
||||
|
||||
setSourceLastAccessed(t, cache, sharedHash, now.Add(-2*time.Hour))
|
||||
setSourceLastAccessed(t, cache, recentHash, now.Add(-time.Minute))
|
||||
|
||||
if err := cache.EvictToLimit(context.Background()); err != nil {
|
||||
t.Fatalf("EvictToLimit failed: %v", err)
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage > limit {
|
||||
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
|
||||
}
|
||||
|
||||
// The multi-referenced blob must be gone from disk, from
|
||||
// source_content, and from BOTH source_metadata rows: references
|
||||
// are removed together with the blob, never left dangling.
|
||||
if cache.srcContent.Exists(sharedHash) {
|
||||
t.Errorf("evicted blob %s still exists on disk", sharedHash)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(sharedHash),
|
||||
); n != 0 {
|
||||
t.Errorf("evicted blob %s still has %d source_content rows", sharedHash, n)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(sharedHash),
|
||||
); n != 0 {
|
||||
t.Errorf("evicted blob %s still has %d source_metadata references", sharedHash, n)
|
||||
}
|
||||
|
||||
// The JSON metadata sidecars for both referencing paths must be
|
||||
// removed along with the rows.
|
||||
for _, path := range []string{"/a.jpg", "/b.jpg"} {
|
||||
pathHash := HashPath(path + "?")
|
||||
if cache.srcMetadata.Exists("src.example.com", pathHash) {
|
||||
t.Errorf("metadata sidecar for %s must be removed with its row", path)
|
||||
}
|
||||
}
|
||||
|
||||
// The more recently used blob survives fully intact.
|
||||
if !cache.srcContent.Exists(recentHash) {
|
||||
t.Errorf("recently used blob %s must survive eviction", recentHash)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(recentHash),
|
||||
); n != 1 {
|
||||
t.Errorf("recently used blob %s has %d source_metadata rows, want 1", recentHash, n)
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
content := bytes.Repeat([]byte{0xDF}, 800)
|
||||
hash := storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg", content)
|
||||
|
||||
if h := storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg", content); h != hash {
|
||||
t.Fatalf("identical content produced different hashes: %s vs %s", h, hash)
|
||||
}
|
||||
|
||||
storeEvictionTestVariant(t, cache, "aabbccdd0001", bytes.Repeat([]byte{0xE0}, 500))
|
||||
|
||||
if err := cache.EvictToLimit(context.Background()); err != nil {
|
||||
t.Fatalf("EvictToLimit failed: %v", err)
|
||||
}
|
||||
|
||||
if !cache.srcContent.Exists(hash) {
|
||||
t.Errorf("blob %s must not be evicted while usage is under the limit", hash)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(hash),
|
||||
); n != 2 {
|
||||
t.Errorf("blob %s has %d source_metadata rows, want 2", hash, n)
|
||||
}
|
||||
|
||||
if !cache.variants.Exists("aabbccdd0001") {
|
||||
t.Error("variant must not be evicted while usage is under the limit")
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
func TestZeroMaxBytesDisablesDiskCache(t *testing.T) {
|
||||
cache, tmpDir := newEvictionTestCache(t, 0)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: "src.example.com",
|
||||
SourcePath: "/a.jpg",
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
|
||||
// Writes are no-ops that report success.
|
||||
if err := cache.StoreVariant(CacheKey(req), bytes.NewReader([]byte("data")), "image/webp"); err != nil {
|
||||
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err)
|
||||
}
|
||||
|
||||
result := &httpfetcher.FetchResult{
|
||||
StatusCode: 200,
|
||||
ContentType: "image/jpeg",
|
||||
ContentLength: 4,
|
||||
Headers: map[string][]string{},
|
||||
}
|
||||
|
||||
hash, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")), result)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err)
|
||||
}
|
||||
|
||||
if hash != "" {
|
||||
t.Errorf("StoreSource on disabled cache returned hash %q, want empty", hash)
|
||||
}
|
||||
|
||||
// Reads always miss.
|
||||
lookup, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup on disabled cache failed: %v", err)
|
||||
}
|
||||
|
||||
if lookup.Hit {
|
||||
t.Error("Lookup on disabled cache must always miss")
|
||||
}
|
||||
|
||||
srcHash, srcType, err := cache.LookupSource(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("LookupSource on disabled cache failed: %v", err)
|
||||
}
|
||||
|
||||
if srcHash != "" || srcType != "" {
|
||||
t.Errorf("LookupSource on disabled cache = (%q, %q), want empty", srcHash, srcType)
|
||||
}
|
||||
|
||||
// Nothing is tracked and nothing is written to disk.
|
||||
usage, err := cache.UsageBytes(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage != 0 {
|
||||
t.Errorf("UsageBytes on disabled cache = %d, want 0", usage)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_content`); n != 0 {
|
||||
t.Errorf("disabled cache wrote %d source_content rows, want 0", n)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_metadata`); n != 0 {
|
||||
t.Errorf("disabled cache wrote %d source_metadata rows, want 0", n)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(filepath.Join(tmpDir, "cache")); !os.IsNotExist(err) {
|
||||
t.Errorf("disabled cache must not create the cache directory tree (stat err=%v)", err)
|
||||
}
|
||||
|
||||
var foundFiles []string
|
||||
|
||||
walkErr := filepath.WalkDir(tmpDir, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if !d.IsDir() {
|
||||
foundFiles = append(foundFiles, path)
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
if walkErr != nil {
|
||||
t.Fatalf("failed to walk state dir: %v", walkErr)
|
||||
}
|
||||
|
||||
if len(foundFiles) != 0 {
|
||||
t.Errorf("disabled cache wrote files to disk: %v", foundFiles)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvictionRunsUnderWritePressure(t *testing.T) {
|
||||
const limit = 1500
|
||||
|
||||
cache, _ := newEvictionTestCache(t, limit)
|
||||
|
||||
// An interval far longer than the test ensures only write
|
||||
// pressure can trigger eviction here.
|
||||
cache.StartEviction(time.Hour)
|
||||
defer cache.StopEviction()
|
||||
|
||||
keys := []VariantKey{"aabbccdd0001", "aabbccdd0002", "aabbccdd0003"}
|
||||
fills := []byte{0x11, 0x12, 0x13}
|
||||
|
||||
for i, key := range keys {
|
||||
storeEvictionTestVariant(t, cache, key, bytes.Repeat([]byte{fills[i]}, 1000))
|
||||
}
|
||||
|
||||
usage := waitForUsageAtOrBelow(t, cache, limit, 5*time.Second)
|
||||
if usage > limit {
|
||||
t.Errorf("write pressure did not trigger eviction: usage = %d, want <= %d",
|
||||
usage, limit)
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
||||
const limit = 1500
|
||||
|
||||
cache, _ := newEvictionTestCache(t, limit)
|
||||
|
||||
// Start the evictor while the cache is empty, then create tracked
|
||||
// over-limit state WITHOUT going through the store methods, so no
|
||||
// write-pressure notification fires and only the periodic ticker
|
||||
// can trigger eviction.
|
||||
cache.StartEviction(100 * time.Millisecond)
|
||||
defer cache.StopEviction()
|
||||
|
||||
keys := []VariantKey{"aabbccdd0001", "aabbccdd0002", "aabbccdd0003"}
|
||||
fills := []byte{0x21, 0x22, 0x23}
|
||||
|
||||
for i, key := range keys {
|
||||
content := bytes.Repeat([]byte{fills[i]}, 1000)
|
||||
|
||||
if _, err := cache.variants.Store(key, bytes.NewReader(content), "image/webp"); err != nil {
|
||||
t.Fatalf("failed to store variant file: %v", err)
|
||||
}
|
||||
|
||||
if _, err := cache.db.Exec(
|
||||
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
||||
VALUES (?, ?, ?)`,
|
||||
string(key), len(content), "image/webp",
|
||||
); err != nil {
|
||||
t.Fatalf("failed to insert variant accounting row: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
usage := waitForUsageAtOrBelow(t, cache, limit, 5*time.Second)
|
||||
if usage > limit {
|
||||
t.Errorf("periodic schedule did not trigger eviction: usage = %d, want <= %d",
|
||||
usage, limit)
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
// An untracked variant file on disk (e.g. written before this
|
||||
// feature existed) must be adopted into the accounting.
|
||||
untracked := bytes.Repeat([]byte{0x31}, 1000)
|
||||
if _, err := cache.variants.Store("aabbccdd0001", bytes.NewReader(untracked), "image/webp"); err != nil {
|
||||
t.Fatalf("failed to store untracked variant file: %v", err)
|
||||
}
|
||||
|
||||
// An accounting row whose file is missing must be dropped.
|
||||
if _, err := cache.db.Exec(
|
||||
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
||||
VALUES (?, ?, ?)`,
|
||||
"deadbeef0001", 700, "image/webp",
|
||||
); err != nil {
|
||||
t.Fatalf("failed to insert stale variant accounting row: %v", err)
|
||||
}
|
||||
|
||||
cache.StartEviction(time.Hour)
|
||||
defer cache.StopEviction()
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
var usage int64
|
||||
|
||||
for time.Now().Before(deadline) {
|
||||
var err error
|
||||
|
||||
usage, err = cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage == 1000 {
|
||||
break
|
||||
}
|
||||
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
|
||||
if usage != 1000 {
|
||||
t.Errorf("usage after reconciliation = %d, want 1000 "+
|
||||
"(untracked file adopted, stale row dropped)", usage)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, "aabbccdd0001",
|
||||
); n != 1 {
|
||||
t.Errorf("untracked variant file was not adopted into accounting (rows=%d)", n)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, "deadbeef0001",
|
||||
); n != 0 {
|
||||
t.Errorf("stale accounting row without a file was not dropped (rows=%d)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves
|
||||
// reconciliation is not a one-shot startup-only pass: it must also run
|
||||
// on the periodic ticker, so a variant file that lands on disk with no
|
||||
// accounting row well after startup (e.g. because StoreVariant's
|
||||
// best-effort accounting insert failed under transient contention, or
|
||||
// any other cause of an untracked file appearing during steady-state
|
||||
// operation) is still adopted into accounting eventually, rather than
|
||||
// staying invisible to UsageBytes/EvictToLimit until the next process
|
||||
// restart.
|
||||
func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
const interval = 100 * time.Millisecond
|
||||
|
||||
cache.StartEviction(interval)
|
||||
defer cache.StopEviction()
|
||||
|
||||
// Let startup reconciliation run and settle on an empty cache
|
||||
// before introducing the untracked file, so the adoption we assert
|
||||
// below can only be the work of a later, periodic pass.
|
||||
time.Sleep(3 * interval)
|
||||
|
||||
// Simulate a variant whose accounting insert failed after the
|
||||
// process was already running and serving requests: the content
|
||||
// file is written directly, bypassing StoreVariant's (and thus its
|
||||
// accounting insert) entirely, exactly as would happen if that
|
||||
// insert had failed and only the file write had succeeded.
|
||||
untracked := bytes.Repeat([]byte{0x41}, 900)
|
||||
if _, err := cache.variants.Store("aabbccdd0099", bytes.NewReader(untracked), "image/webp"); err != nil {
|
||||
t.Fatalf("failed to store untracked variant file: %v", err)
|
||||
}
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
var usage int64
|
||||
|
||||
for time.Now().Before(deadline) {
|
||||
var err error
|
||||
|
||||
usage, err = cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage == 900 {
|
||||
break
|
||||
}
|
||||
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
|
||||
if usage != 900 {
|
||||
t.Errorf("usage after periodic reconciliation = %d, want 900 "+
|
||||
"(a file that appeared after startup reconciliation already ran must still "+
|
||||
"be adopted by a later periodic pass)", usage)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, "aabbccdd0099",
|
||||
); n != 1 {
|
||||
t.Errorf("file that appeared after startup was not adopted by periodic "+
|
||||
"reconciliation (rows=%d)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent exercises
|
||||
// the exact TOCTOU window between evictSourceBlob's row-deletion
|
||||
// transaction commit and its content file unlink: a concurrent
|
||||
// StoreSource for a different source path whose content hashes to the
|
||||
// same value (real SHA-256 dedup, not a contrived case) must not be
|
||||
// able to insert a fresh row referencing the file while eviction is
|
||||
// mid-unlink, and must not lose its own store once eviction has fully
|
||||
// released the content hash.
|
||||
func TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
ctx := context.Background()
|
||||
|
||||
content := bytes.Repeat([]byte{0x55}, 400)
|
||||
|
||||
hash := storeEvictionTestSource(t, cache, "race.example.com", "/first.jpg", content)
|
||||
|
||||
proceed := make(chan struct{})
|
||||
storeAttempted := make(chan struct{})
|
||||
|
||||
cache.evictSourceBlobTestHook = func(gotHash ContentHash) {
|
||||
if gotHash != hash {
|
||||
t.Errorf("test hook invoked for hash %s, want %s", gotHash, hash)
|
||||
}
|
||||
|
||||
close(storeAttempted)
|
||||
<-proceed
|
||||
}
|
||||
|
||||
evictDone := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
evictDone <- cache.evictSourceBlob(ctx, hash)
|
||||
}()
|
||||
|
||||
// Wait until eviction has committed its delete transaction and is
|
||||
// paused (inside the test hook) immediately before unlinking the
|
||||
// content file: exactly the window the review flagged.
|
||||
<-storeAttempted
|
||||
|
||||
storeDone := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
req := &ImageRequest{
|
||||
SourceHost: "race.example.com",
|
||||
SourcePath: "/dup.jpg",
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
result := &httpfetcher.FetchResult{
|
||||
StatusCode: 200,
|
||||
ContentType: "image/jpeg",
|
||||
ContentLength: int64(len(content)),
|
||||
Headers: map[string][]string{"Content-Type": {"image/jpeg"}},
|
||||
}
|
||||
|
||||
_, err := cache.StoreSource(ctx, req, bytes.NewReader(content), result)
|
||||
storeDone <- err
|
||||
}()
|
||||
|
||||
// The concurrent store must not be able to complete while eviction
|
||||
// still holds the content hash (i.e. before the file is unlinked):
|
||||
// if it could, it would insert a row referencing a file about to be
|
||||
// removed out from under it.
|
||||
select {
|
||||
case err := <-storeDone:
|
||||
t.Fatalf("StoreSource for identical content completed (err=%v) while eviction "+
|
||||
"still held the content hash open between commit and unlink; the store and "+
|
||||
"the evict of identical content are not mutually exclusive", err)
|
||||
case <-time.After(200 * time.Millisecond):
|
||||
// Expected: the store is blocked behind eviction's exclusion.
|
||||
}
|
||||
|
||||
close(proceed)
|
||||
|
||||
if err := <-evictDone; err != nil {
|
||||
t.Fatalf("evictSourceBlob failed: %v", err)
|
||||
}
|
||||
|
||||
if err := <-storeDone; err != nil {
|
||||
t.Fatalf("StoreSource failed: %v", err)
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
|
||||
dupHash, _, err := cache.LookupSource(ctx, &ImageRequest{
|
||||
SourceHost: "race.example.com",
|
||||
SourcePath: "/dup.jpg",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("LookupSource failed: %v", err)
|
||||
}
|
||||
|
||||
if dupHash == "" {
|
||||
t.Fatal("re-stored blob was lost: the store legitimately ran after eviction " +
|
||||
"released the content hash and must have recreated the file and row")
|
||||
}
|
||||
|
||||
if !cache.srcContent.Exists(dupHash) {
|
||||
t.Errorf("source_content/source_metadata references %s but its file is missing", dupHash)
|
||||
}
|
||||
}
|
||||
@@ -65,24 +65,49 @@ func (s *ContentStorage) Store(r io.Reader) (hash ContentHash, size int64, err e
|
||||
hash = ContentHash(hex.EncodeToString(h[:]))
|
||||
size = int64(len(data))
|
||||
|
||||
if err := s.writeIfAbsent(hash, data); err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
|
||||
return hash, size, nil
|
||||
}
|
||||
|
||||
// StoreHashed writes pre-hashed content to storage at the path derived
|
||||
// from hash, without recomputing it. Callers that already know the
|
||||
// hash before writing (e.g. because they must hold a hash-keyed lock
|
||||
// across the whole store operation) use this instead of Store. Like
|
||||
// Store, it is idempotent: content already on disk at that path is
|
||||
// left untouched.
|
||||
func (s *ContentStorage) StoreHashed(hash ContentHash, data []byte) (size int64, err error) {
|
||||
if err := s.writeIfAbsent(hash, data); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return int64(len(data)), nil
|
||||
}
|
||||
|
||||
// writeIfAbsent writes data to the path derived from hash, unless
|
||||
// content already exists there, via a temp-file-plus-rename so
|
||||
// concurrent readers never observe a partial file.
|
||||
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) (err error) {
|
||||
// Build path: <basedir>/<ab>/<cd>/<hash>
|
||||
path := s.hashToPath(hash)
|
||||
|
||||
// Check if already exists
|
||||
if _, err := os.Stat(path); err == nil {
|
||||
return hash, size, nil
|
||||
if _, statErr := os.Stat(path); statErr == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Create directory structure
|
||||
dir := filepath.Dir(path)
|
||||
if err := os.MkdirAll(dir, StorageDirPerm); err != nil {
|
||||
return "", 0, fmt.Errorf("failed to create directory: %w", err)
|
||||
return fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
// Write to temp file first, then rename for atomicity
|
||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("failed to create temp file: %w", err)
|
||||
return fmt.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
tmpPath := tmpFile.Name()
|
||||
|
||||
@@ -95,20 +120,20 @@ func (s *ContentStorage) Store(r io.Reader) (hash ContentHash, size int64, err e
|
||||
if _, err := tmpFile.Write(data); err != nil {
|
||||
_ = tmpFile.Close()
|
||||
|
||||
return "", 0, fmt.Errorf("failed to write content: %w", err)
|
||||
return fmt.Errorf("failed to write content: %w", err)
|
||||
}
|
||||
|
||||
if err := tmpFile.Close(); err != nil {
|
||||
return "", 0, fmt.Errorf("failed to close temp file: %w", err)
|
||||
return fmt.Errorf("failed to close temp file: %w", err)
|
||||
}
|
||||
|
||||
// Atomic rename
|
||||
//nolint:gosec // G703: paths from internal SHA256 hashes
|
||||
if err := os.Rename(filepath.Clean(tmpPath), filepath.Clean(path)); err != nil {
|
||||
return "", 0, fmt.Errorf("failed to rename temp file: %w", err)
|
||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||
}
|
||||
|
||||
return hash, size, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// Load returns a reader for the content with the given hash.
|
||||
@@ -493,6 +518,24 @@ func (s *VariantStorage) Delete(key VariantKey) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteWithMeta removes the content at the given key together with
|
||||
// its .meta sidecar file. A missing file is not an error.
|
||||
func (s *VariantStorage) DeleteWithMeta(key VariantKey) error {
|
||||
if err := s.Delete(key); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
metaPath := s.keyToPath(key) + ".meta"
|
||||
|
||||
//nolint:gosec // G703: path derived from cache key
|
||||
err := os.Remove(metaPath)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to delete variant metadata: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
|
||||
func (s *VariantStorage) keyToPath(key VariantKey) string {
|
||||
k := string(key)
|
||||
|
||||
Reference in New Issue
Block a user