2 Commits
Author SHA1 Message Date
clawbot 536e85d09d Correct trusted_proxies advice and state signature padding (closes #150)
check / check (push) Successful in 4m4s
The README told operators to set trusted_proxies to the proxy's own
address. A proxy on the Docker host that connects over 127.0.0.1 reaches
pixa from the Docker network's gateway, so that advice made pixa count
every user as one client for the login limit. The login-limit paragraph,
the trusted_proxies entry and config.example.yml now say to use the
address pixa sees for requests through the proxy, that a proxy connecting
through another host address is seen with that address, and how to read
it from the request log.

The signature section now says sig is base64url with the = padding
kept, since pixa compares it exactly, and shows the example's sig for a
stated key, computed with pixa's signer.

Model: opus-5-5
2026-09-29 03:27:35 +00:00
clawbot e98b998cb6 Make the cache stats count what is cached, fetched and transcoded (closes #56)
check / check (push) Successful in 18s
Cache.Stats read request_cache and output_content, which nothing
writes, so TotalItems and TotalSizeBytes were always 0. They now count
source_content plus variant_content and use UsageBytes; a disabled disk
cache reports 0 for both. The upstream fetch count and bytes and the
transform count never moved: Get now passes the bytes it fetched
(including those read before a failed body read) and counts each
successful transcode. Hits, misses and these counters are written with
context.WithoutCancel, so a client disconnect or the request timeout no
longer loses them. The unused tables stay; metaCache is #70.

Model: opus-5-5
2026-09-29 05:25:10 +02:00
7 changed files with 417 additions and 65 deletions
+16 -14
View File
@@ -115,15 +115,16 @@ refused with 429 and a `Retry-After` header. Behind a reverse proxy the client
address comes from `X-Forwarded-For` only when the address pixa sees for address comes from `X-Forwarded-For` only when the address pixa sees for
requests that come through the proxy is in `trusted_proxies`; otherwise all requests that come through the proxy is in `trusted_proxies`; otherwise all
users behind the proxy are counted as one client. That address is not always users behind the proxy are counted as one client. That address is not always
the proxy's own: for a proxy on the Docker host it is the gateway of the the proxy's own: a proxy on the Docker host that connects to pixa over
container's Docker network, such as `172.17.0.1` on the default bridge. pixa's `127.0.0.1` is seen as the gateway of the container's Docker network, such as
request log shows it as `remoteIP` while it is not in `trusted_proxies` (see `172.17.0.1` on the default bridge, and one that connects through another of the
`trusted_proxies` under Configuration). With the default `trusted_proxies` host's addresses is seen with that address. To be sure, read it as `remoteIP` in
(the RFC 1918 ranges), a client with a private address can choose the address pixa's request log while it is not in `trusted_proxies` (see `trusted_proxies`
it is counted by through its own `X-Forwarded-For`, whether it connects under Configuration). With the default `trusted_proxies` (the RFC 1918 ranges),
directly or through the proxy, because its own address is trusted too. Setting a client with a private address can choose the address it is counted by through
`trusted_proxies` to only the address pixa sees for requests that come through its own `X-Forwarded-For`, whether it connects directly or through the proxy,
the proxy closes this. because its own address is trusted too. Setting `trusted_proxies` to only the
address pixa sees for requests that come through the proxy closes this.
### Image Metadata ### Image Metadata
@@ -262,11 +263,12 @@ Key settings in more detail:
startup. Set this to the address pixa sees for requests that come through startup. Set this to the address pixa sees for requests that come through
your proxy, such as `172.17.0.1/32`, when the defaults do not cover it, or your proxy, such as `172.17.0.1/32`, when the defaults do not cover it, or
to trust nothing else (see the login limit under Routes). For a proxy on to trust nothing else (see the login limit under Routes). For a proxy on
the Docker host that address is the gateway of the container's Docker the Docker host that connects to pixa over `127.0.0.1`, that address is the
network (`172.17.0.1` on the default bridge), not the proxy's own address. gateway of the container's Docker network (`172.17.0.1` on the default
To find it, set this to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a bridge), not the proxy's own address; a proxy that connects through another of
request through the proxy, and read `remoteIP` in pixa's request log line the host's addresses is seen with that address. To be sure which address it
for it is, set this to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request
through the proxy, and read `remoteIP` in pixa's request log line for it
- `upstream_fetch_timeout` — timeout for origin requests - `upstream_fetch_timeout` — timeout for origin requests
- `upstream_max_response_size` — max origin response size - `upstream_max_response_size` — max origin response size
- `downstream_timeout` — client response timeout - `downstream_timeout` — client response timeout
+17 -5
View File
@@ -33,11 +33,11 @@ exhaustion
- 2026-09-29 `trusted_proxies` advice and signature padding in `README.md` - 2026-09-29 `trusted_proxies` advice and signature padding in `README.md`
(closes #150): the login-limit paragraph, the `trusted_proxies` entry and (closes #150): the login-limit paragraph, the `trusted_proxies` entry and
`config.example.yml` say to set `trusted_proxies` to the address pixa sees for `config.example.yml` say to set `trusted_proxies` to the address pixa sees for
requests that come through the proxy, not the proxy's own address; for a proxy requests that come through the proxy, which the request log shows as
on the Docker host that is the Docker network's gateway, which the request log `remoteIP` while it is not trusted; for a proxy on the Docker host that
shows as `remoteIP` while it is not trusted; the signature section says `sig` connects over `127.0.0.1` that is the Docker network's gateway, not the
is base64url with the `=` padding kept, and gives the example's `sig` for a proxy's own address; the signature section says `sig` is base64url with the
stated signing key. `=` padding kept, and gives the example's `sig` for a stated signing key.
- 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the - 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the
`pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of `pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of
the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad` the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad`
@@ -50,6 +50,18 @@ exhaustion
that is sooner, never negative; an allowlisted host's URL that has an `exp` that is sooner, never negative; an allowlisted host's URL that has an `exp`
follows it too; `immutable` stays, as freshness now ends at the expiry; follows it too; `immutable` stays, as freshness now ends at the expiry;
documented in `README.md`. documented in `README.md`.
- 2026-09-28 cache stats report real numbers (closes #56): `Cache.Stats`
counts the cached source images and processed variants (`source_content`
plus `variant_content`) and takes their size from `Cache.UsageBytes`,
instead of reading `request_cache` and `output_content`, which nothing
writes; those two tables are left in the schema; a disabled disk cache
reports no items and no size. A hit is counted even when the request
context has ended. A miss is counted after it is served or fails, also
when the request context has ended by then, with the bytes it read from
upstream, so `upstream_fetch_count` and `upstream_fetch_bytes` move,
including for an upstream body that fails partway or a fetched source
that then fails the magic byte check; `transform_count` counts each image
the image processor transcodes.
- 2026-09-28 strip metadata from processed images (closes #82): every output is - 2026-09-28 strip metadata from processed images (closes #82): every output is
exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC
profile; the image is first turned upright with `AutoRotate` (before sizes are profile; the image is first turned upright with `AutoRotate` (before sizes are
+6 -5
View File
@@ -51,11 +51,12 @@ allowlist_hosts:
# a private network. An explicitly empty list ([]) trusts no one; an # a private network. An explicitly empty list ([]) trusts no one; an
# explicit list replaces the default. An invalid CIDR aborts startup. # explicit list replaces the default. An invalid CIDR aborts startup.
# Uncomment to override the defaults with the address pixa sees for # Uncomment to override the defaults with the address pixa sees for
# requests that come through your proxy. That is not always the proxy's # requests that come through your proxy. That is not always the proxy's own
# own address: for a proxy on the Docker host it is the gateway of the # address: a proxy on the Docker host that connects over 127.0.0.1 is seen
# container's Docker network (172.17.0.1 on the default bridge). The # as the gateway of the container's Docker network (172.17.0.1 on the
# trusted_proxies entry in README.md says how to find it in the request # default bridge), and one that connects through another host address is
# log. # seen with that address. To be sure, look it up in the request log as the
# trusted_proxies entry in README.md describes.
# trusted_proxies: # trusted_proxies:
# - 10.0.0.0/8 # - 10.0.0.0/8
# - 2001:db8::/32 # - 2001:db8::/32
+27 -12
View File
@@ -419,19 +419,21 @@ func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
return nil, fmt.Errorf("failed to get cache stats: %w", err) return nil, fmt.Errorf("failed to get cache stats: %w", err)
} }
// Get actual item count and total size from content tables // Count and size the cached source images and processed variants. A
err = c.db.QueryRowContext(ctx, // disabled cache holds none, whatever rows an earlier run left.
`SELECT COUNT(*) FROM request_cache`, if !c.disabled {
).Scan(&stats.TotalItems) err = c.db.QueryRowContext(ctx, `
if err != nil { SELECT (SELECT COUNT(*) FROM source_content)
c.log.Warn("failed to count cache items for stats", "error", err) + (SELECT COUNT(*) FROM variant_content)
} `).Scan(&stats.TotalItems)
if err != nil {
c.log.Warn("failed to count cache items for stats", "error", err)
}
err = c.db.QueryRowContext(ctx, stats.TotalSizeBytes, err = c.UsageBytes(ctx)
`SELECT COALESCE(SUM(size_bytes), 0) FROM output_content`, if err != nil {
).Scan(&stats.TotalSizeBytes) c.log.Warn("failed to sum cache size for stats", "error", err)
if err != nil { }
c.log.Warn("failed to sum cache size for stats", "error", err)
} }
// Compute hit rate as a ratio // Compute hit rate as a ratio
@@ -481,6 +483,19 @@ func (c *Cache) IncrementStats(ctx context.Context, hit bool, fetchBytes int64)
} }
} }
// IncrementTransformCount counts one image transcoded by the image processor.
func (c *Cache) IncrementTransformCount(ctx context.Context) {
_, err := c.db.ExecContext(ctx, `
UPDATE cache_stats
SET transform_count = transform_count + 1,
last_updated_at = CURRENT_TIMESTAMP
WHERE id = 1
`)
if err != nil {
c.log.Warn("failed to count transform", "error", err)
}
}
// writeMetadataSidecar writes the JSON metadata sidecar of a stored source. // writeMetadataSidecar writes the JSON metadata sidecar of a stored source.
// A failure is logged and is otherwise non-fatal; the metadata is in the // A failure is logged and is otherwise non-fatal; the metadata is in the
// database. // database.
+4 -2
View File
@@ -163,9 +163,11 @@ type ImageCache interface {
// CacheStats contains cache statistics // CacheStats contains cache statistics
type CacheStats struct { type CacheStats struct {
// TotalItems is the number of cached items // TotalItems is the number of cached source images plus processed
// variants
TotalItems int64 TotalItems int64
// TotalSizeBytes is the total size of cached content // TotalSizeBytes is the total size of cached source images and
// processed variants
TotalSizeBytes int64 TotalSizeBytes int64
// HitCount is the number of cache hits // HitCount is the number of cache hits
HitCount int64 HitCount int64
+32 -26
View File
@@ -143,7 +143,8 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
s.log.Error("failed to get cached variant", "key", result.CacheKey, "error", err) s.log.Error("failed to get cached variant", "key", result.CacheKey, "error", err)
// Fall through to re-process // Fall through to re-process
} else { } else {
s.cache.IncrementStats(ctx, true, 0) // Counted also when the request context has ended meanwhile
s.cache.IncrementStats(context.WithoutCancel(ctx), true, 0)
return &ImageResponse{ return &ImageResponse{
Content: reader, Content: reader,
@@ -155,12 +156,15 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
} }
} }
// Cache miss - check if we have source content cached // Cache miss - process the cached source or fetch it, then count the
// miss with the bytes it fetched from upstream, also when it failed or
// the request context has ended meanwhile
cacheKey := CacheKey(req) cacheKey := CacheKey(req)
s.cache.IncrementStats(ctx, false, 0) response, fetchedBytes, err := s.processFromSourceOrFetch(ctx, req, cacheKey)
s.cache.IncrementStats(context.WithoutCancel(ctx), false, fetchedBytes)
response, err := s.processFromSourceOrFetch(ctx, req, cacheKey)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -268,22 +272,20 @@ func (s *Service) loadCachedSource(contentHash ContentHash) []byte {
} }
// processFromSourceOrFetch processes an image, using cached source content // processFromSourceOrFetch processes an image, using cached source content
// if available. // if available. It also returns the number of bytes fetched from upstream,
// as fetchAndProcess does, or 0 when the cached source was used.
func (s *Service) processFromSourceOrFetch( func (s *Service) processFromSourceOrFetch(
ctx context.Context, ctx context.Context,
req *ImageRequest, req *ImageRequest,
cacheKey VariantKey, cacheKey VariantKey,
) (*ImageResponse, error) { ) (*ImageResponse, int64, error) {
// Check if we have cached source content // Check if we have cached source content
contentHash, _, err := s.cache.LookupSource(ctx, req) contentHash, _, err := s.cache.LookupSource(ctx, req)
if err != nil { if err != nil {
s.log.Warn("source lookup failed", "error", err) s.log.Warn("source lookup failed", "error", err)
} }
var ( var sourceData []byte
sourceData []byte
fetchBytes int64
)
if contentHash != "" { if contentHash != "" {
s.log.Debug("using cached source", "hash", contentHash) s.log.Debug("using cached source", "hash", contentHash)
@@ -292,26 +294,25 @@ func (s *Service) processFromSourceOrFetch(
// Fetch from upstream if we don't have source data or it's empty // Fetch from upstream if we don't have source data or it's empty
if len(sourceData) == 0 { if len(sourceData) == 0 {
resp, err := s.fetchAndProcess(ctx, req, cacheKey) return s.fetchAndProcess(ctx, req, cacheKey)
if err != nil {
return nil, err
}
return resp, nil
} }
// Process using cached source // Process using cached source; nothing was fetched from upstream
fetchBytes = int64(len(sourceData)) resp, err := s.processAndStore(
ctx, req, cacheKey, sourceData, int64(len(sourceData)),
)
return s.processAndStore(ctx, req, cacheKey, sourceData, fetchBytes) return resp, 0, err
} }
// fetchAndProcess fetches from upstream, processes, and caches the result. // fetchAndProcess fetches from upstream, processes, and caches the result.
// It also returns the number of bytes read from upstream, including when
// reading the response or a later step fails.
func (s *Service) fetchAndProcess( func (s *Service) fetchAndProcess(
ctx context.Context, ctx context.Context,
req *ImageRequest, req *ImageRequest,
cacheKey VariantKey, cacheKey VariantKey,
) (*ImageResponse, error) { ) (*ImageResponse, int64, error) {
// Fetch from upstream // Fetch from upstream
sourceURL := req.SourceURL() sourceURL := req.SourceURL()
@@ -330,20 +331,20 @@ func (s *Service) fetchAndProcess(
} }
} }
return nil, fmt.Errorf("upstream fetch failed: %w", err) return nil, 0, fmt.Errorf("upstream fetch failed: %w", err)
} }
defer func() { _ = fetchResult.Content.Close() }() defer func() { _ = fetchResult.Content.Close() }()
// Read and validate the source content // Read and validate the source content
sourceData, err := io.ReadAll(fetchResult.Content) sourceData, err := io.ReadAll(fetchResult.Content)
fetchBytes := int64(len(sourceData))
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read upstream response: %w", err) return nil, fetchBytes, fmt.Errorf("failed to read upstream response: %w", err)
} }
// Calculate download bitrate // Calculate download bitrate
fetchBytes := int64(len(sourceData))
var downloadRate string var downloadRate string
if fetchResult.FetchDurationMs > 0 { if fetchResult.FetchDurationMs > 0 {
@@ -368,7 +369,7 @@ func (s *Service) fetchAndProcess(
// Validate magic bytes match content type // Validate magic bytes match content type
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType) err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType)
if err != nil { if err != nil {
return nil, fmt.Errorf("content validation failed: %w", err) return nil, fetchBytes, fmt.Errorf("content validation failed: %w", err)
} }
// Store source content // Store source content
@@ -378,7 +379,9 @@ func (s *Service) fetchAndProcess(
// Continue even if caching fails // Continue even if caching fails
} }
return s.processAndStore(ctx, req, cacheKey, sourceData, fetchBytes) resp, err := s.processAndStore(ctx, req, cacheKey, sourceData, fetchBytes)
return resp, fetchBytes, err
} }
// processAndStore processes an image and stores the result. // processAndStore processes an image and stores the result.
@@ -406,6 +409,9 @@ func (s *Service) processAndStore(
processDuration := time.Since(processStart) processDuration := time.Since(processStart)
// Counted also when the request context has ended meanwhile
s.cache.IncrementTransformCount(context.WithoutCancel(ctx))
// Read processed content // Read processed content
processedData, err := io.ReadAll(processResult.Content) processedData, err := io.ReadAll(processResult.Content)
_ = processResult.Content.Close() _ = processResult.Content.Close()
+315 -1
View File
@@ -4,6 +4,9 @@ import (
"bytes" "bytes"
"context" "context"
"database/sql" "database/sql"
"image/color"
"io"
"io/fs"
"log/slog" "log/slog"
"math" "math"
"strings" "strings"
@@ -11,6 +14,7 @@ import (
"time" "time"
"sneak.berlin/go/pixa/internal/database" "sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
) )
func setupStatsTestDB(t *testing.T) *sql.DB { func setupStatsTestDB(t *testing.T) *sql.DB {
@@ -125,7 +129,7 @@ func TestStats_LogsFailedCountQueries(t *testing.T) {
} }
_, err = db.ExecContext(t.Context(), _, err = db.ExecContext(t.Context(),
`DROP TABLE request_cache; DROP TABLE output_content`) `DROP TABLE source_content; DROP TABLE variant_content`)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -183,3 +187,313 @@ func TestIncrementStats_LogsFailedUpdates(t *testing.T) {
} }
} }
} }
// TestStats_TotalsCountSourcesAndVariants verifies that TotalItems and
// TotalSizeBytes cover the stored source images and processed variants.
func TestStats_TotalsCountSourcesAndVariants(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
storeEvictionTestSource(t, cache, testHostCDN, testPathCat,
bytes.Repeat([]byte{0xAA}, 1000))
storeEvictionTestVariant(t, cache, testVariantKeyOne,
bytes.Repeat([]byte{0xAB}, 500))
storeEvictionTestVariant(t, cache, testVariantKeyTwo,
bytes.Repeat([]byte{0xAC}, 250))
stats, err := cache.Stats(t.Context())
if err != nil {
t.Fatalf("Stats() error = %v", err)
}
if stats.TotalItems != 3 {
t.Errorf("TotalItems = %d, want 3 (1 source, 2 variants)", stats.TotalItems)
}
if stats.TotalSizeBytes != 1750 {
t.Errorf("TotalSizeBytes = %d, want 1750 (1000+500+250)",
stats.TotalSizeBytes)
}
}
// TestStats_DisabledCacheReportsNoItems verifies that a disabled disk cache
// reports no items and no size, even when its database still holds the
// rows of an earlier run with the disk cache enabled.
func TestStats_DisabledCacheReportsNoItems(t *testing.T) {
t.Parallel()
enabled, _ := newEvictionTestCache(t, 1<<30)
storeEvictionTestSource(t, enabled, testHostCDN, testPathCat,
bytes.Repeat([]byte{0xAA}, 1000))
storeEvictionTestVariant(t, enabled, testVariantKeyOne,
bytes.Repeat([]byte{0xAB}, 500))
disabled, err := NewCache(enabled.db, CacheConfig{
StateDir: t.TempDir(),
CacheTTL: time.Hour,
NegativeTTL: 5 * time.Minute,
DisableDiskCache: true,
})
if err != nil {
t.Fatal(err)
}
stats, err := disabled.Stats(t.Context())
if err != nil {
t.Fatalf("Stats() error = %v", err)
}
if stats.TotalItems != 0 || stats.TotalSizeBytes != 0 {
t.Errorf("TotalItems = %d, TotalSizeBytes = %d, want 0 and 0",
stats.TotalItems, stats.TotalSizeBytes)
}
}
// cacheStatsCounters holds the counters of the cache_stats row, in column
// order.
type cacheStatsCounters struct {
hitCount int64
missCount int64
upstreamFetchCount int64
upstreamFetchBytes int64
transformCount int64
}
// readCacheStatsCounters reads the counters of the cache_stats row.
func readCacheStatsCounters(t *testing.T, cache *Cache) cacheStatsCounters {
t.Helper()
var got cacheStatsCounters
err := cache.db.QueryRowContext(t.Context(), `
SELECT hit_count, miss_count, upstream_fetch_count,
upstream_fetch_bytes, transform_count
FROM cache_stats WHERE id = 1
`).Scan(&got.hitCount, &got.missCount, &got.upstreamFetchCount,
&got.upstreamFetchBytes, &got.transformCount)
if err != nil {
t.Fatalf("failed to read cache_stats: %v", err)
}
return got
}
// TestService_Get_CountsStats walks Get through a miss that fetches the
// source, a hit, a miss that reuses the cached source, and two misses whose
// source cannot be used, checking every cache_stats counter after each.
func TestService_Get_CountsStats(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
// NewTestFS builds the same files the test service's fetcher serves.
testFS, _ := NewTestFS(t)
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
if err != nil {
t.Fatal(err)
}
fake, err := fs.ReadFile(testFS, fixtures.InvalidFile)
if err != nil {
t.Fatal(err)
}
photoBytes, fakeBytes := int64(len(photo)), int64(len(fake))
// want is hits, misses, upstream fetches, upstream bytes, transforms.
steps := []struct {
name string
path string
size int
wantErr bool
want cacheStatsCounters
}{
{"miss that fetches the source", testPathPhoto, 50, false,
cacheStatsCounters{0, 1, 1, photoBytes, 1}},
{"hit", testPathPhoto, 50, false,
cacheStatsCounters{1, 1, 1, photoBytes, 1}},
{"miss that reuses the cached source", testPathPhoto, 25, false,
cacheStatsCounters{1, 2, 1, photoBytes, 2}},
{"miss whose source fails the magic byte check", "/images/fake.jpg", 50, true,
cacheStatsCounters{1, 3, 2, photoBytes + fakeBytes, 2}},
{"miss whose source is not found", "/images/nonexistent.jpg", 50, true,
cacheStatsCounters{1, 4, 2, photoBytes + fakeBytes, 2}},
}
for _, step := range steps {
resp, err := svc.Get(t.Context(), &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: step.path,
Size: Size{Width: step.size, Height: step.size},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
})
if (err != nil) != step.wantErr {
t.Fatalf("%s: Get() error = %v, want error %t", step.name, err, step.wantErr)
}
if err == nil {
_ = resp.Content.Close()
}
got := readCacheStatsCounters(t, svc.cache)
if got != step.want {
t.Fatalf("after the %s: counters = %+v, want %+v", step.name, got, step.want)
}
}
}
// fakeUpstream answers every fetch with itself as a JPEG body. The body
// serves data, then calls cancel, when set, and returns err; io.EOF ends
// the body normally.
type fakeUpstream struct {
data *bytes.Reader
cancel context.CancelFunc
err error
}
func (u *fakeUpstream) Fetch(
context.Context, string,
) (*httpfetcher.FetchResult, error) {
return &httpfetcher.FetchResult{
Content: io.NopCloser(u),
ContentLength: -1,
ContentType: testContentTypeJPEG,
}, nil
}
func (u *fakeUpstream) Read(p []byte) (int, error) {
if u.data.Len() > 0 {
return u.data.Read(p)
}
if u.cancel != nil {
u.cancel()
}
return 0, u.err
}
// TestService_Get_CountsInterruptedMisses checks every cache_stats counter
// after a miss whose request context ends during or after the upstream
// fetch, and after a miss whose upstream body is over the size limit.
func TestService_Get_CountsInterruptedMisses(t *testing.T) {
t.Parallel()
photo := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
half := len(photo) / 2
// want is hits, misses, upstream fetches, upstream bytes, transforms.
tests := []struct {
name string
served int // bytes of the photo the upstream body serves
cancel bool // whether the body then ends the request context
readErr error // what the body then returns
wantErr bool
want cacheStatsCounters
}{
{"request context ends during the fetch", half, true, context.Canceled, true,
cacheStatsCounters{0, 1, 1, int64(half), 0}},
{"request context ends after the fetch", len(photo), true, io.EOF, false,
cacheStatsCounters{0, 1, 1, int64(len(photo)), 1}},
{"upstream body over the size limit", half, false,
httpfetcher.ErrResponseTooLarge, true,
cacheStatsCounters{0, 1, 1, int64(half), 0}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
upstream := &fakeUpstream{
data: bytes.NewReader(photo[:tc.served]),
err: tc.readErr,
}
if tc.cancel {
upstream.cancel = cancel
}
svc.fetcher = upstream
resp, err := svc.Get(ctx, &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
})
t.Logf("Get() error = %v", err)
if (err != nil) != tc.wantErr {
t.Fatalf("Get() error = %v, want error %t", err, tc.wantErr)
}
if err == nil {
_ = resp.Content.Close()
}
got := readCacheStatsCounters(t, svc.cache)
if got != tc.want {
t.Errorf("counters = %+v, want %+v", got, tc.want)
}
})
}
}
// TestService_Get_CountsHitAfterRequestEnds checks every cache_stats counter
// after a hit served with a request context that has already ended: only
// the hit count moves.
func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
// A first request caches the variant.
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("first Get() error = %v", err)
}
_ = resp.Content.Close()
want := readCacheStatsCounters(t, svc.cache)
want.hitCount++
ctx, cancel := context.WithCancel(t.Context())
cancel()
resp, err = svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() with an ended request context: error = %v", err)
}
_ = resp.Content.Close()
if resp.CacheStatus != CacheHit {
t.Fatalf("CacheStatus = %v, want %v", resp.CacheStatus, CacheHit)
}
got := readCacheStatsCounters(t, svc.cache)
if got != want {
t.Errorf("counters = %+v, want %+v", got, want)
}
}