Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3f487ee0ce |
No files matched your search
@@ -233,8 +233,8 @@ These terms are the names the code uses:
|
||||
- **Variant**: the image pixa serves for an image request, made from its source
|
||||
with its transform.
|
||||
- **Variant key**: the SHA-256 hash of a variant's source, size, format, quality
|
||||
and fit. It names the variant's file and its row in the database. The
|
||||
variant's `ETag` is made from it and the file's modification time.
|
||||
and fit. It names the variant's file and its row in the database, and gives
|
||||
its `ETag`.
|
||||
- **Signed URL**: a `/v1/image/` URL whose `sig` parameter signs its image
|
||||
request and its expiry `exp` with `signing_key`.
|
||||
- **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any
|
||||
@@ -249,8 +249,8 @@ These terms are the names the code uses:
|
||||
- a check of a cached image failed in any way (see Cache TTL).
|
||||
- **Image cache**: the part of pixa that keeps the sources, the variants and the
|
||||
negative cache (see Storage).
|
||||
- **Cache TTL**: how long pixa uses a cached source and serves its variants
|
||||
before it checks the source with upstream again: 24 hours (see Cache TTL).
|
||||
- **Cache TTL**: how long pixa serves a cached source and its variants before it
|
||||
checks the source with upstream again: 24 hours (see Cache TTL).
|
||||
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
|
||||
sources and variants may reach together.
|
||||
- **Image proxy**: the part of pixa that serves a variant from the image cache
|
||||
@@ -522,9 +522,7 @@ the image after pixa would refuse the URL. A URL with no expiry gets one year.
|
||||
`immutable` only stops a client revalidating while its copy is fresh. The
|
||||
response also carries `X-Pixa-Cache`: `HIT` for an image served from the cache,
|
||||
and `MISS` for one made for this request or older than 24 hours (see Cache TTL).
|
||||
The `ETag` is made from the first 16 characters of the variant key and the
|
||||
variant file's modification time (see Cache TTL). So a variant made again from a
|
||||
changed source gets a new `ETag`.
|
||||
The `ETag` is the first 16 characters of the variant key, in quotes.
|
||||
|
||||
### Signed URLs
|
||||
|
||||
@@ -896,17 +894,16 @@ also removes the expired rows of the negative cache every five minutes, at most
|
||||
|
||||
### Cache TTL
|
||||
|
||||
pixa uses a cached source for 24 hours after upstream last sent it or confirmed
|
||||
it unchanged. During that time it serves the source's variants from the cache.
|
||||
After that, the next request for an image of that source checks the source with
|
||||
upstream first:
|
||||
pixa serves a cached source and its variants for 24 hours after upstream last
|
||||
sent the source or confirmed it unchanged. After that, the next request for an
|
||||
image of that source checks the source with upstream first:
|
||||
|
||||
- pixa asks upstream to send the source only if it changed. It sends the
|
||||
source's `ETag` as `If-None-Match` and its `Last-Modified` as
|
||||
`If-Modified-Since`, when upstream gave them. With neither, upstream sends the
|
||||
whole image again.
|
||||
- When upstream answers 304 Not Modified, pixa uses the cached source and serves
|
||||
its variants for another 24 hours. Nothing is downloaded or converted again.
|
||||
- When upstream answers 304 Not Modified, pixa serves the cached source and its
|
||||
variants for another 24 hours. Nothing is downloaded or converted again.
|
||||
- When upstream sends the image, it replaces the cached source. Each variant
|
||||
made from the old image is converted again from the new one when it is next
|
||||
requested.
|
||||
@@ -916,8 +913,8 @@ upstream first:
|
||||
often it serves the image.
|
||||
|
||||
pixa evicts a source and its variants 30 days after upstream last sent the
|
||||
source or confirmed it unchanged. So pixa uses a source whose checks keep
|
||||
failing for at most 30 days. Eviction to stay under the disk cache limit can
|
||||
source or confirmed it unchanged. So a source whose checks keep failing is
|
||||
served for at most 30 days. Eviction to stay under the disk cache limit can
|
||||
remove images sooner.
|
||||
|
||||
That eviction can remove a source and keep its variants. Each variant's row
|
||||
|
||||
@@ -31,7 +31,7 @@ P2: security: per-origin rate limiting
|
||||
# Completed Steps
|
||||
|
||||
- 2026-11-11 a cached source is checked with upstream once it is 24 hours old
|
||||
(closes #69). Before, it was used and its variants served until eviction, so a
|
||||
(closes #69). Before, it and its variants were served until eviction, so a
|
||||
changed image at the same URL was never picked up.
|
||||
- `source_content` keeps the source's `fetched_at`, `checked_at`, and the
|
||||
`ETag` and `Last-Modified` upstream sent.
|
||||
@@ -40,9 +40,6 @@ P2: security: per-origin rate limiting
|
||||
- A variant file's modification time is its source's `checked_at` when the
|
||||
variant was made or last renewed, so a hit makes no database call. A
|
||||
variant made before the source's `fetched_at` is converted again.
|
||||
- A variant's `ETag` is made from its variant key and its file's
|
||||
modification time, so a variant made again from new bytes gets a new
|
||||
`ETag`.
|
||||
- `variant_content` keeps the `ETag` and `Last-Modified` of the source a
|
||||
variant was made from. A variant whose source was evicted is checked with
|
||||
them, and a 304 renews the variant.
|
||||
|
||||
@@ -257,90 +257,6 @@ func TestHandleImage_ConditionalRequest_IfNoneMatch_DifferentETag(t *testing.T)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_ConditionalRequest_EarlierETagGetsChangedImage serves an
|
||||
// image, changes it upstream, and requests it again with the ETag first
|
||||
// served, once the cache TTL has passed. The changed image must be served,
|
||||
// not 304 Not Modified.
|
||||
func TestHandleImage_ConditionalRequest_EarlierETagGetsChangedImage(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const host = "goodhost.example.com"
|
||||
|
||||
upstream := fstest.MapFS{
|
||||
host + photoPath: &fstest.MapFile{
|
||||
Data: generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}),
|
||||
},
|
||||
}
|
||||
|
||||
// A cache TTL of one nanosecond has passed by the second request
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
|
||||
StateDir: t.TempDir(),
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
CacheTTL: time.Nanosecond,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
if err != nil {
|
||||
t.Fatalf("imgcache.NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
svc, err := imageproxy.New(&imageproxy.Config{
|
||||
Cache: cache,
|
||||
Fetcher: newMockFetcher(upstream),
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
Logger: slog.New(slog.DiscardHandler),
|
||||
}, prometheus.NewRegistry())
|
||||
if err != nil {
|
||||
t.Fatalf("imageproxy.New() error = %v", err)
|
||||
}
|
||||
|
||||
h := &Handlers{
|
||||
imgSvc: svc,
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
signer: signature.New(testSigningKey),
|
||||
allowlist: allowlist.New([]string{host}),
|
||||
}
|
||||
|
||||
r := chi.NewRouter()
|
||||
r.Get("/v1/image/*", h.HandleImage())
|
||||
|
||||
target := "/v1/image/" + host + photoPath + "/50x50.jpeg"
|
||||
|
||||
first := httptest.NewRecorder()
|
||||
r.ServeHTTP(first,
|
||||
httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil))
|
||||
|
||||
etag := first.Header().Get("ETag")
|
||||
if first.Code != http.StatusOK || etag == "" {
|
||||
t.Fatalf("first request: status = %d, ETag = %q, want %d and an ETag",
|
||||
first.Code, etag, http.StatusOK)
|
||||
}
|
||||
|
||||
upstream[host+photoPath] = &fstest.MapFile{
|
||||
Data: generateTestJPEG(t, 100, 100, color.RGBA{0, 0, 255, 255}),
|
||||
}
|
||||
|
||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
|
||||
req.Header.Set("If-None-Match", etag)
|
||||
|
||||
second := httptest.NewRecorder()
|
||||
r.ServeHTTP(second, req)
|
||||
t.Logf("first ETag %s; with it after the change: status %d, ETag %s, %d bytes",
|
||||
etag, second.Code, second.Header().Get("ETag"), second.Body.Len())
|
||||
|
||||
if second.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d with the changed image",
|
||||
second.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
if bytes.Equal(second.Body.Bytes(), first.Body.Bytes()) {
|
||||
t.Error("the image served is the one made before the change")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleImage_ETagHeader(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -168,16 +168,16 @@ func (p *Proxy) Get(
|
||||
) (*ImageResponse, error) {
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
|
||||
cached, err := p.cache.Variant(cacheKey)
|
||||
reader, size, contentType, err := p.cache.Variant(cacheKey)
|
||||
if err == nil {
|
||||
p.cache.IncrementStats(true)
|
||||
|
||||
return &ImageResponse{
|
||||
Content: cached.Content,
|
||||
ContentLength: cached.Size,
|
||||
ContentType: cached.ContentType,
|
||||
Content: reader,
|
||||
ContentLength: size,
|
||||
ContentType: contentType,
|
||||
CacheStatus: CacheHit,
|
||||
ETag: formatETag(cacheKey, cached.CheckedAt),
|
||||
ETag: formatETag(cacheKey),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -288,7 +288,7 @@ func (p *Proxy) processOrWait(
|
||||
ContentLength: int64(len(variant.data)),
|
||||
ContentType: variant.contentType,
|
||||
FetchedBytes: variant.fetchedBytes,
|
||||
ETag: formatETag(cacheKey, variant.checkedAt),
|
||||
ETag: formatETag(cacheKey),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -439,7 +439,7 @@ func (p *Proxy) renewVariant(
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey,
|
||||
source imgcache.SourceInfo,
|
||||
) *processedVariant {
|
||||
cached, err := p.cache.RenewVariant(ctx, cacheKey, source)
|
||||
reader, _, contentType, err := p.cache.RenewVariant(ctx, cacheKey, source)
|
||||
if err != nil {
|
||||
if !errors.Is(err, imgcache.ErrNotFound) {
|
||||
p.log.Warn("cached variant not renewed, making it again",
|
||||
@@ -449,7 +449,7 @@ func (p *Proxy) renewVariant(
|
||||
return nil
|
||||
}
|
||||
|
||||
return p.readCachedVariant(cacheKey, cached)
|
||||
return p.readCachedVariant(cacheKey, reader, contentType)
|
||||
}
|
||||
|
||||
// cachedVariantIgnoringTTL returns the cached variant of cacheKey whatever
|
||||
@@ -460,7 +460,7 @@ func (p *Proxy) renewVariant(
|
||||
func (p *Proxy) cachedVariantIgnoringTTL(
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey,
|
||||
) (*processedVariant, imgcache.SourceInfo) {
|
||||
cached, err := p.cache.VariantIgnoringTTL(cacheKey)
|
||||
reader, _, contentType, err := p.cache.VariantIgnoringTTL(cacheKey)
|
||||
if err != nil {
|
||||
if !errors.Is(err, imgcache.ErrNotFound) {
|
||||
p.log.Warn("cached variant not served", "cache_key", cacheKey,
|
||||
@@ -470,7 +470,7 @@ func (p *Proxy) cachedVariantIgnoringTTL(
|
||||
return nil, imgcache.SourceInfo{}
|
||||
}
|
||||
|
||||
variant := p.readCachedVariant(cacheKey, cached)
|
||||
variant := p.readCachedVariant(cacheKey, reader, contentType)
|
||||
if variant == nil {
|
||||
return nil, imgcache.SourceInfo{}
|
||||
}
|
||||
@@ -484,14 +484,15 @@ func (p *Proxy) cachedVariantIgnoringTTL(
|
||||
return variant, madeFrom
|
||||
}
|
||||
|
||||
// readCachedVariant reads the cached variant of cacheKey from its opened file,
|
||||
// and closes the file. It returns nil when the variant cannot be read.
|
||||
// readCachedVariant reads the cached variant of cacheKey that reader opened,
|
||||
// whose content type is contentType, and closes reader. It returns nil when
|
||||
// the variant cannot be read.
|
||||
func (p *Proxy) readCachedVariant(
|
||||
cacheKey imagerequest.VariantKey, file imgcache.VariantFile,
|
||||
cacheKey imagerequest.VariantKey, reader io.ReadCloser, contentType string,
|
||||
) *processedVariant {
|
||||
defer func() { _ = file.Content.Close() }()
|
||||
defer func() { _ = reader.Close() }()
|
||||
|
||||
data, err := io.ReadAll(file.Content)
|
||||
data, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
p.log.Warn("failed to read cached variant", "cache_key", cacheKey,
|
||||
"error", err)
|
||||
@@ -499,11 +500,7 @@ func (p *Proxy) readCachedVariant(
|
||||
return nil
|
||||
}
|
||||
|
||||
return &processedVariant{
|
||||
data: data,
|
||||
contentType: file.ContentType,
|
||||
checkedAt: file.CheckedAt,
|
||||
}
|
||||
return &processedVariant{data: data, contentType: contentType}
|
||||
}
|
||||
|
||||
// fetchAndProcess fetches the source from upstream, stores it, and makes
|
||||
@@ -531,9 +528,7 @@ func (p *Proxy) fetchAndProcess(
|
||||
case variant == nil:
|
||||
return nil, err
|
||||
case errors.Is(err, errSourceNotModified):
|
||||
variant.checkedAt = time.Now().UTC()
|
||||
|
||||
err = p.cache.SetVariantCheckedAt(ctx, cacheKey, variant.checkedAt)
|
||||
err = p.cache.SetVariantCheckedAt(ctx, cacheKey, time.Now().UTC())
|
||||
if err != nil {
|
||||
p.log.Warn("failed to renew cached variant", "cache_key", cacheKey,
|
||||
"error", err)
|
||||
@@ -750,9 +745,6 @@ type processedVariant struct {
|
||||
data []byte
|
||||
contentType string
|
||||
fetchedBytes int64
|
||||
// checkedAt is its source's check time when the variant was made or last
|
||||
// renewed, which its cached file has as its modification time
|
||||
checkedAt time.Time
|
||||
}
|
||||
|
||||
// processAndStore processes the image read from source, which is in
|
||||
@@ -834,7 +826,6 @@ func (p *Proxy) processAndStore(
|
||||
data: processedData,
|
||||
contentType: processResult.ContentType,
|
||||
fetchedBytes: fetchBytes,
|
||||
checkedAt: sourceInfo.CheckedAt,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -846,16 +837,13 @@ func isNegativeCacheable(err error) bool {
|
||||
// etagHashLength is the number of hash characters to use for ETags.
|
||||
const etagHashLength = 16
|
||||
|
||||
// formatETag returns the quoted ETag of the variant of key: the first
|
||||
// etagHashLength characters of key, then checkedAt in Unix nanoseconds.
|
||||
// checkedAt is its source's check time when the variant was made or last
|
||||
// renewed, so a variant made again from new source bytes gets another ETag.
|
||||
func formatETag(key imagerequest.VariantKey, checkedAt time.Time) string {
|
||||
// formatETag formats a VariantKey as a quoted ETag value.
|
||||
func formatETag(key imagerequest.VariantKey) string {
|
||||
hash := string(key)
|
||||
// Use first 16 characters of hash for a shorter but still unique ETag
|
||||
if len(hash) > etagHashLength {
|
||||
hash = hash[:etagHashLength]
|
||||
}
|
||||
|
||||
return fmt.Sprintf(`"%s-%d"`, hash, checkedAt.UnixNano())
|
||||
return `"` + hash + `"`
|
||||
}
|
||||
+62
-69
@@ -3,8 +3,8 @@
|
||||
// - the variants made from them
|
||||
// - the negative cache: the source URLs whose fetch failed recently
|
||||
//
|
||||
// Each image is a file with a row in the database. A source is used, and its
|
||||
// variants are served, until the cache TTL has passed since upstream last sent
|
||||
// Each image is a file with a row in the database. A source and its
|
||||
// variants are served until the cache TTL has passed since upstream last sent
|
||||
// the source or confirmed it unchanged. Then the source is due to be checked
|
||||
// with upstream again (see Stale). Eviction removes:
|
||||
// - the least recently used images, to keep the cache under its size limit
|
||||
@@ -45,14 +45,14 @@ const cleanExpiredBatchSize = 1000
|
||||
const DefaultNegativeTTL = 5 * time.Minute
|
||||
|
||||
// DefaultCacheTTL is how long after upstream last sent a cached source, or
|
||||
// answered that it had not changed, pixad uses the source and serves its
|
||||
// answered that it had not changed, pixad serves the source and its
|
||||
// variants without asking upstream again.
|
||||
const DefaultCacheTTL = 24 * time.Hour
|
||||
|
||||
// EvictionAfterLastCheck is how long after upstream last sent a cached
|
||||
// source, or answered that it had not changed, the source and its variants
|
||||
// are evicted (see EvictNotCheckedRecently). Until then a source whose checks
|
||||
// fail is used.
|
||||
// fail is served.
|
||||
const EvictionAfterLastCheck = 30 * 24 * time.Hour
|
||||
|
||||
// ErrStale is returned by Variant for a variant whose file's modification
|
||||
@@ -64,7 +64,7 @@ type Config struct {
|
||||
StateDir string
|
||||
NegativeTTL time.Duration
|
||||
|
||||
// CacheTTL is how long a cached source is used, and its variants served,
|
||||
// CacheTTL is how long a cached source and its variants are served
|
||||
// before upstream is asked again whether the source changed. Zero means
|
||||
// DefaultCacheTTL.
|
||||
CacheTTL time.Duration
|
||||
@@ -186,43 +186,31 @@ func NewCache(
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// VariantFile is the open file of a cached variant, as Variant,
|
||||
// VariantIgnoringTTL and RenewVariant return it.
|
||||
type VariantFile struct {
|
||||
// Content reads the file. The caller closes it.
|
||||
Content io.ReadCloser
|
||||
// Size is the file's size in bytes.
|
||||
Size int64
|
||||
// ContentType is the MIME type of the variant's format, read from the
|
||||
// file's first bytes.
|
||||
ContentType string
|
||||
// CheckedAt is the file's modification time: its source's check time
|
||||
// when the variant was made or last renewed.
|
||||
CheckedAt time.Time
|
||||
}
|
||||
|
||||
// Variant opens a cached variant and returns its file (see VariantFile). It
|
||||
// opens that one file and makes no database call: the file's modification
|
||||
// time is its source's check time when the variant was made or last renewed
|
||||
// (see PutVariant), and a variant whose time is CacheTTL or more ago is
|
||||
// ErrStale. A variant it opens has its access time handed to the background
|
||||
// writer without waiting (see touchVariant). A variant not in the cache, or
|
||||
// any variant of a disabled cache, is ErrNotFound; a variant file whose first
|
||||
// Variant opens a cached variant and returns a reader, its size and its
|
||||
// content type, read from the variant file's first bytes. It opens that one
|
||||
// file and makes no database call: the file's modification time is its
|
||||
// source's check time when the variant was made or last renewed (see
|
||||
// PutVariant), and a variant whose time is CacheTTL or more ago is ErrStale.
|
||||
// A variant it opens has its access time handed to the background writer
|
||||
// without waiting (see touchVariant). A variant not in the cache, or any
|
||||
// variant of a disabled cache, is ErrNotFound; a variant file whose first
|
||||
// bytes are not those of an output format is another error.
|
||||
func (c *Cache) Variant(cacheKey imagerequest.VariantKey) (VariantFile, error) {
|
||||
func (c *Cache) Variant(
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
return VariantFile{}, ErrNotFound
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
file, info, err := c.variants.Open(string(cacheKey))
|
||||
file, info, err := c.openVariant(cacheKey)
|
||||
if err != nil {
|
||||
return VariantFile{}, err
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
if c.Stale(info.ModTime()) {
|
||||
_ = file.Close()
|
||||
|
||||
return VariantFile{}, ErrStale
|
||||
return nil, 0, "", ErrStale
|
||||
}
|
||||
|
||||
return c.finishOpeningVariant(cacheKey, file, info)
|
||||
@@ -233,14 +221,14 @@ func (c *Cache) Variant(cacheKey imagerequest.VariantKey) (VariantFile, error) {
|
||||
// cached, before it checks the source with upstream (see LookupVariant).
|
||||
func (c *Cache) VariantIgnoringTTL(
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (VariantFile, error) {
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
return VariantFile{}, ErrNotFound
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
file, info, err := c.variants.Open(string(cacheKey))
|
||||
file, info, err := c.openVariant(cacheKey)
|
||||
if err != nil {
|
||||
return VariantFile{}, err
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
return c.finishOpeningVariant(cacheKey, file, info)
|
||||
@@ -253,50 +241,41 @@ func (c *Cache) VariantIgnoringTTL(
|
||||
// ErrNotFound, as is a variant not in the cache.
|
||||
func (c *Cache) RenewVariant(
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey, source SourceInfo,
|
||||
) (VariantFile, error) {
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
return VariantFile{}, ErrNotFound
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
file, info, err := c.variants.Open(string(cacheKey))
|
||||
file, info, err := c.openVariant(cacheKey)
|
||||
if err != nil {
|
||||
return VariantFile{}, err
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
if info.ModTime().Before(source.FetchedAt) {
|
||||
_ = file.Close()
|
||||
|
||||
return VariantFile{}, ErrNotFound
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
format, err := readFormat(file)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return VariantFile{}, err
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
checkedAt := info.ModTime()
|
||||
|
||||
if checkedAt.Before(source.CheckedAt) {
|
||||
if info.ModTime().Before(source.CheckedAt) {
|
||||
err = c.SetVariantCheckedAt(ctx, cacheKey, source.CheckedAt)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return VariantFile{}, err
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
checkedAt = source.CheckedAt
|
||||
}
|
||||
|
||||
c.touchVariant(cacheKey)
|
||||
|
||||
return VariantFile{
|
||||
Content: file,
|
||||
Size: info.Size(),
|
||||
ContentType: format.MIMEType(),
|
||||
CheckedAt: checkedAt,
|
||||
}, nil
|
||||
return file, info.Size(), format.MIMEType(), nil
|
||||
}
|
||||
|
||||
// SetVariantCheckedAt records checkedAt as when upstream last sent the source
|
||||
@@ -427,7 +406,7 @@ func (c *Cache) Source(
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
file, info, err := c.srcContent.Open(string(key))
|
||||
file, size, err := c.srcContent.Open(string(key))
|
||||
if err != nil {
|
||||
return nil, 0, "", err
|
||||
}
|
||||
@@ -441,7 +420,7 @@ func (c *Cache) Source(
|
||||
|
||||
c.touchSourceContent(key)
|
||||
|
||||
return file, info.Size(), format, nil
|
||||
return file, size, format, nil
|
||||
}
|
||||
|
||||
// SourceInfo is what the cache keeps in a cached source's row besides its
|
||||
@@ -715,29 +694,43 @@ func (c *Cache) cleanExpiredBatch(ctx context.Context) (int64, error) {
|
||||
return deleted, nil
|
||||
}
|
||||
|
||||
// finishOpeningVariant returns a variant file that FileStore.Open opened, with
|
||||
// its information info, as Variant does. It reads the content type from the
|
||||
// file's first bytes and hands the variant's access time to the background
|
||||
// writer (see touchVariant). It closes the file when its first bytes are not
|
||||
// those of an output format.
|
||||
// openVariant opens the file of a cached variant and returns it with its
|
||||
// information, which gives its size and its modification time.
|
||||
func (c *Cache) openVariant(
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (*os.File, fs.FileInfo, error) {
|
||||
file, _, err := c.variants.Open(string(cacheKey))
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
info, err := file.Stat()
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return nil, nil, fmt.Errorf("failed to stat variant: %w", err)
|
||||
}
|
||||
|
||||
return file, info, nil
|
||||
}
|
||||
|
||||
// finishOpeningVariant reads the content type of a variant file openVariant
|
||||
// opened from its first bytes, hands the variant's access time to the
|
||||
// background writer (see touchVariant), and returns the file as Variant does.
|
||||
// It closes the file when its first bytes are not those of an output format.
|
||||
func (c *Cache) finishOpeningVariant(
|
||||
cacheKey imagerequest.VariantKey, file *os.File, info fs.FileInfo,
|
||||
) (VariantFile, error) {
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
format, err := readFormat(file)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return VariantFile{}, err
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
c.touchVariant(cacheKey)
|
||||
|
||||
return VariantFile{
|
||||
Content: file,
|
||||
Size: info.Size(),
|
||||
ContentType: format.MIMEType(),
|
||||
CheckedAt: info.ModTime(),
|
||||
}, nil
|
||||
return file, info.Size(), format.MIMEType(), nil
|
||||
}
|
||||
|
||||
// readFormat reads the first bytes of a source or variant file, without
|
||||
|
||||
@@ -55,13 +55,11 @@ func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
|
||||
t.Fatalf("PutVariant(%s) error = %v", format, err)
|
||||
}
|
||||
|
||||
variant, err := cache.Variant(key)
|
||||
reader, size, contentType, err := cache.Variant(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Variant(%s) error = %v", format, err)
|
||||
}
|
||||
|
||||
reader, size, contentType := variant.Content, variant.Size, variant.ContentType
|
||||
|
||||
served, err := io.ReadAll(reader)
|
||||
_ = reader.Close()
|
||||
|
||||
@@ -112,19 +110,16 @@ func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
|
||||
t.Fatalf("%s: storing the variant file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
variant, err := cache.Variant(key)
|
||||
reader, _, contentType, err := cache.Variant(key)
|
||||
t.Logf("%s: Variant() error = %v", tt.name, err)
|
||||
|
||||
reader, contentType := variant.Content, variant.ContentType
|
||||
if reader != nil || contentType != "" || !errors.Is(err, tt.wantErr) {
|
||||
t.Errorf("%s: Variant() = reader %v, content type %q, error %v; "+
|
||||
"want no reader and %v", tt.name, reader, contentType, err, tt.wantErr)
|
||||
}
|
||||
}
|
||||
|
||||
variant, err := cache.Variant(testVariantKeyFour)
|
||||
|
||||
reader := variant.Content
|
||||
reader, _, _, err := cache.Variant(testVariantKeyFour)
|
||||
if reader != nil || !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Variant() of a variant with no file = reader %v, error %v; "+
|
||||
"want no reader and %v", reader, err, ErrNotFound)
|
||||
@@ -132,9 +127,7 @@ func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
|
||||
|
||||
disabled, _ := newEvictionTestCache(t, 0)
|
||||
|
||||
variant, err = disabled.Variant(testVariantKeyOne)
|
||||
|
||||
reader = variant.Content
|
||||
reader, _, _, err = disabled.Variant(testVariantKeyOne)
|
||||
if reader != nil || !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Variant() on a disabled cache = reader %v, error %v; "+
|
||||
"want no reader and %v", reader, err, ErrNotFound)
|
||||
|
||||
@@ -3,7 +3,6 @@ package imgcache
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
@@ -56,26 +55,26 @@ func (s *FileStore) Write(key string, data []byte) error {
|
||||
return writeFileSynced(path, data)
|
||||
}
|
||||
|
||||
// Open opens the file for key and returns it with its information, which
|
||||
// gives its size and its modification time. A missing file is ErrNotFound.
|
||||
func (s *FileStore) Open(key string) (*os.File, fs.FileInfo, error) {
|
||||
// Open opens the file for key and returns it with its size in bytes. A
|
||||
// missing file is ErrNotFound.
|
||||
func (s *FileStore) Open(key string) (*os.File, int64, error) {
|
||||
f, err := os.Open(s.keyToPath(key))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil, ErrNotFound
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
return nil, nil, fmt.Errorf("failed to open content: %w", err)
|
||||
return nil, 0, fmt.Errorf("failed to open content: %w", err)
|
||||
}
|
||||
|
||||
info, err := f.Stat()
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
|
||||
return nil, nil, fmt.Errorf("failed to stat content: %w", err)
|
||||
return nil, 0, fmt.Errorf("failed to stat content: %w", err)
|
||||
}
|
||||
|
||||
return f, info, nil
|
||||
return f, stat.Size(), nil
|
||||
}
|
||||
|
||||
// Size returns the size in bytes of the file for key. A missing file is
|
||||
|
||||
@@ -38,15 +38,15 @@ func TestFileStore_WriteAndOpen(t *testing.T) {
|
||||
}
|
||||
|
||||
// Open and verify size and content
|
||||
r, info, err := storage.Open(key)
|
||||
r, size, err := storage.Open(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Open() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = r.Close() }()
|
||||
|
||||
if info.Size() != int64(len(content)) {
|
||||
t.Errorf("Open() size = %d, want %d", info.Size(), len(content))
|
||||
if size != int64(len(content)) {
|
||||
t.Errorf("Open() size = %d, want %d", size, len(content))
|
||||
}
|
||||
|
||||
loaded, err := io.ReadAll(r)
|
||||
|
||||
Reference in new issue
Block a user