Export cache, eviction, upstream and transform metrics (closes #85)
check / check (push) Canceled after 0s

New pixa_ metrics, none with labels:
- source cache hits and misses, counted once per variant made or checked;
  a source whose cached file cannot be used is a miss
- cache usage and limit gauges; usage is set each time eviction reads it
- eviction count and bytes, from the size_bytes the deleted row returns
- upstream fetch duration histogram and fetch error counter
- transform duration histogram and an in-progress gauge, both including
  the wait for a processing slot

Removing a damaged file during reconciliation is not counted as an
eviction. README lists every metric; the metrics config comment now says
the pixa_ metrics are measured whether or not /metrics is served.

Model: opus-5-5
This commit was merged in pull request #353.
This commit is contained in:
sneak committed 2026-11-11 14:32:38 +00:00
1 parent a041e73550
commit fa2ec3da6a
9 files changed
+319 -113

No files matched your search

+12 -17
View File
@@ -111,10 +111,16 @@ type Cache struct {
pendingAccessTimesDone chan struct{}
pendingAccessTimesCancel context.CancelFunc
// The counts of cache hits and misses, made and registered by
// registerCounters (see counters.go)
cacheHits prometheus.Counter
cacheMisses prometheus.Counter
// The cache's metrics, made and registered by registerMetrics (see
// metrics.go)
cacheHits prometheus.Counter
cacheMisses prometheus.Counter
sourceCacheHits prometheus.Counter
sourceCacheMisses prometheus.Counter
usageBytes prometheus.Gauge
maxBytes prometheus.Gauge
evictions prometheus.Counter
evictedBytes prometheus.Counter
// reconciliationPageSize is the most rows one read of a content table
// returns in the reconciliation pass. NewCache sets it to
@@ -128,7 +134,7 @@ type Cache struct {
reconciliationReadTestHook func(rows int)
}
// NewCache creates a new cache instance and registers its counters on
// NewCache creates a new cache instance and registers its metrics on
// metricsRegistry. pixad passes Prometheus' default registry, the one
// /metrics serves. It makes no database call.
func NewCache(
@@ -157,7 +163,7 @@ func NewCache(
reconciliationPageSize: defaultReconciliationPageSize,
}
err := c.registerCounters(metricsRegistry)
err := c.registerMetrics(metricsRegistry)
if err != nil {
return nil, err
}
@@ -656,17 +662,6 @@ func (c *Cache) CleanExpired(ctx context.Context) error {
}
}
// IncrementStats counts a cache hit or miss. It only adds to the cache's
// Prometheus counters in memory (see counters.go), so it never waits for the
// database or fails.
func (c *Cache) IncrementStats(hit bool) {
if hit {
c.cacheHits.Inc()
} else {
c.cacheMisses.Inc()
}
}
// touchVariant hands the access time of a variant served from the cache to
// the goroutine StartPendingAccessTimeWrites starts, which writes it in its
// next batch. It never waits: while maxPendingAccessTimes access times of