Compare commits
2
Commits
6bc354f249
...
2c5af66094
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2c5af66094 | ||
|
|
5b17d1f555 |
@@ -92,8 +92,61 @@ the metadata file stored beside it.
|
||||
|
||||
### Routes
|
||||
|
||||
pixa answers these routes; any other path answers 404. A path in this list asked
|
||||
with a method the list does not give answers 405, except `/static/<file>`, which
|
||||
answers any method as it answers `GET`. A browser's CORS preflight request
|
||||
(`OPTIONS` with `Origin` and `Access-Control-Request-Method` headers) to any
|
||||
path under `/v1/` answers 200, in maintenance mode too.
|
||||
|
||||
- `GET /` — the login page, or the URL generator page with a login session
|
||||
(see Encrypted URLs). Needs: nothing. Answers: 200.
|
||||
- `POST /` — log in with the signing key typed into the login page. Needs: the
|
||||
login page's form (below). Answers: 303 to `/` with a login session cookie
|
||||
that lasts 30 days for the right key; 200 with the login page and an error for
|
||||
a wrong key; 429 over the login limit (below).
|
||||
- `POST /generate` — make an encrypted URL from the generator page's form.
|
||||
Needs: a login session and the generator page's form (below); without a login
|
||||
session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400
|
||||
with the page naming a field that is not valid; 500 when the URL cannot be
|
||||
made.
|
||||
- `GET /logout` — end the login session. Needs: nothing. Answers: 303 to `/`.
|
||||
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>` — an image, fetched,
|
||||
resized and converted (below). Needs: a signature, unless the host is
|
||||
allowlisted (see Source Hosts). Answers: 200; 304 when `If-None-Match` matches
|
||||
the image's `ETag`; 400 for a URL or parameter that is not valid; 401 for a
|
||||
missing or wrong signature, a missing `exp` or an `exp` in the past; 403 when
|
||||
the upstream host, or a host it redirects to, is `localhost`, ends in
|
||||
`.localhost` or `.local`, or has an address in a blocked network (see
|
||||
`blocked_networks`); 502 when the upstream answered with an error status, and
|
||||
for 5 minutes after that for the same source URL; 503 when pixa is busy or in
|
||||
maintenance mode; 500 for any other failure.
|
||||
- `GET /v1/e/<token>/<name>` — an image through an encrypted URL (see Encrypted
|
||||
URLs). Needs: nothing but the URL. Answers: 200; 400 for a token that does not
|
||||
decrypt, or that asks for a size or fit that is not valid; 410 once it has
|
||||
expired; 504 when the upstream has not sent its response headers within
|
||||
`upstream_fetch_timeout`, but 500 when that time runs out while the image
|
||||
itself is still arriving; 403, 502, 503 and 500 as for `/v1/image/`.
|
||||
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
|
||||
nothing. Answers: 200.
|
||||
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
|
||||
`uptime_seconds`, `uptime_human`, `version`, `appname` and
|
||||
`maintenance_mode`. Needs: nothing. Answers: 200, always.
|
||||
- `GET /static/<file>` — the script the login and generator pages load. Needs:
|
||||
nothing. Answers: 200, or 404 for a file that does not exist.
|
||||
- `GET /metrics` — Prometheus metrics (see Architecture). Needs: HTTP basic
|
||||
authentication with `metrics.username` and `metrics.password`. Answers: 200;
|
||||
401 without them; 404 when they are not set, as the route then does not exist.
|
||||
|
||||
Both `POST` routes accept only a form that pixa's own page served: the page puts
|
||||
a token in the form and sets a cookie to match, and a request without both is
|
||||
refused with 403, so another site cannot submit the form from a visitor's
|
||||
browser. A form body over 1 MiB is refused with 413. The image routes answer the
|
||||
errors listed for them with JSON holding `error`, `status` and `timestamp`.
|
||||
|
||||
An image URL has this form:
|
||||
|
||||
```
|
||||
/v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>
|
||||
/v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit>
|
||||
```
|
||||
|
||||
Images are only fetched from origins using TLS with valid certificates, unless
|
||||
@@ -106,6 +159,11 @@ than once, is refused with 400.
|
||||
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
|
||||
`avif`, `gif`
|
||||
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
|
||||
- `sig` and `exp`: the signature and its expiry, needed unless the host is
|
||||
allowlisted (see Signature Specification)
|
||||
- `q` and `fit`: the output quality and how the image is fitted to `<size>`,
|
||||
both optional (values under Signature Specification). Both are part of what
|
||||
is cached, so each value of either is a separate cached image.
|
||||
|
||||
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
|
||||
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL),
|
||||
@@ -139,6 +197,38 @@ its own `X-Forwarded-For`, whether it connects directly or through the proxy,
|
||||
because its own address is trusted too. Setting `trusted_proxies` to only the
|
||||
address pixa sees for requests that come through the proxy closes this.
|
||||
|
||||
### Encrypted URLs
|
||||
|
||||
An encrypted URL is an image URL made on pixa's own web page by someone who
|
||||
knows the signing key. It works for any upstream host, allowlisted or not,
|
||||
without a signature, and whoever gets it can neither read the source URL from it
|
||||
nor change what it asks for.
|
||||
|
||||
1. Open `/` in a browser and log in with the signing key (`signing_key`). The
|
||||
login session lasts 30 days, or until `/logout`.
|
||||
2. On the generator page, give the source image's URL, the width and height, the
|
||||
format, quality and fit, and how long the URL lasts, then submit the form
|
||||
(`POST /generate`). Width and height both empty or `0` keep the original
|
||||
size; if only one of them is empty or `0`, that side is scaled to keep the
|
||||
image's proportions.
|
||||
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
||||
it expires. `<host>` is the host the page was opened on, and the URL starts
|
||||
with `http` instead while `debug` is on. The name after the token is ignored
|
||||
and only gives the URL a file extension, `jpg` for `orig`.
|
||||
|
||||
The token holds the source's host, path and query and the size, format,
|
||||
quality, fit and expiry, encrypted with a key derived from `signing_key`. The
|
||||
source URL's scheme is not kept: the image is fetched like any other (see
|
||||
Routes), and the blocked networks still apply.
|
||||
|
||||
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or
|
||||
never. The expiry is fixed in the token when the URL is made and cannot be
|
||||
changed or revoked afterwards. Until then the image is served with a `max-age`
|
||||
that ends at the expiry (see Routes); after it the URL answers 410
|
||||
`URL has expired`. A URL made to last forever stops working only when
|
||||
`signing_key` changes: changing it makes every encrypted URL already handed out
|
||||
answer 400, and ends every login session.
|
||||
|
||||
### Image Metadata
|
||||
|
||||
pixa decodes and re-encodes every image it serves, and removes all metadata from
|
||||
@@ -237,6 +327,14 @@ startup naming it, as an unknown config key does. The one other accepted
|
||||
name is `PIXA_CONFIG_PATH`, the config file's path (like `--config`). The
|
||||
variables set by the file's `env:` section are checked the same way.
|
||||
|
||||
pixa reads at most one config file: the one given with `--config` (or `-c`),
|
||||
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that
|
||||
exists: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`,
|
||||
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml`
|
||||
and `config.yaml` in the working directory. A file that cannot be read or does
|
||||
not parse aborts startup, and so does a named file that does not exist; with no
|
||||
file, pixa uses the environment and the defaults.
|
||||
|
||||
| Variable | Config key | Meaning |
|
||||
| ------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| `PIXA_SIGNING_KEY` | `signing_key` | Required: secret for signed and encrypted URLs and login, 32+ characters |
|
||||
|
||||
@@ -29,6 +29,21 @@ P2: security: referer blacklist
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-04 routes, encrypted URLs and config file documented (closes #75):
|
||||
"Routes" in `README.md` lists every route with its method, purpose, what it
|
||||
needs and the status codes it answers with, and says `q` and `fit` are part
|
||||
of what is cached; "Encrypted URLs" covers logging in, making one on the
|
||||
generator page, how long it lasts and the 410 once it has expired;
|
||||
"Configuration" gives the order in which pixa looks for its config file;
|
||||
`config.example.yml` lists `db_url` and `env` and gives every key's default;
|
||||
`scripts/manual-test.sh` is left to #97.
|
||||
- 2026-10-04 upstream host semaphores and variant `.meta` files no longer
|
||||
outlive their use (closes #87): the fetcher counts the fetches holding or
|
||||
waiting for a slot of each upstream host's semaphore and removes the host's
|
||||
semaphore once none is left, so fetches from many hosts no longer leave one
|
||||
semaphore each until restart; `VariantStorage.Delete` removes the variant's
|
||||
`.meta` file along with it, a missing `.meta` file not being an error, and
|
||||
`DeleteWithMeta`, which eviction called for that, is gone.
|
||||
- 2026-10-04 `README.md` matches the code (closes #74): "Storage" names the
|
||||
cache directories pixa uses (`cache/sources`, `cache/metadata`,
|
||||
`cache/variants`) and how files are named in each, and the comments in
|
||||
@@ -420,7 +435,5 @@ P2: security: referer blacklist
|
||||
- integration tests for the image proxy flow
|
||||
- load tests to verify the 1k to 5k req/s target
|
||||
- P2: documentation
|
||||
- configuration options
|
||||
- API endpoints
|
||||
- deployment guide
|
||||
- example nginx or caddy reverse proxy config
|
||||
|
||||
+23
-5
@@ -12,9 +12,13 @@
|
||||
# Durations are Go duration strings such as 30s or 2m and must be
|
||||
# positive; a bare number has no unit and aborts startup. Sizes are a
|
||||
# whole number of bytes.
|
||||
#
|
||||
# A key left out takes the default its comment gives.
|
||||
|
||||
# Server settings
|
||||
# Port to listen on (default: 8080)
|
||||
port: 8080
|
||||
|
||||
# Debug logging and plain-HTTP local development (default: false)
|
||||
debug: false
|
||||
|
||||
# While true, the image routes (/v1/image/ and /v1/e/) answer every request
|
||||
@@ -22,17 +26,24 @@ debug: false
|
||||
# reports maintenance_mode as true. It stays 200 because the image's Docker
|
||||
# HEALTHCHECK requests it: a 503 there would make the container unhealthy, and
|
||||
# upaas marks a deploy failed when its container is unhealthy.
|
||||
# (default: false)
|
||||
maintenance_mode: false
|
||||
|
||||
# Data directory for SQLite database and cache files
|
||||
# (default: /var/lib/pixa)
|
||||
state_dir: ./data
|
||||
|
||||
# SQLite database URL (default:
|
||||
# file:<state_dir>/state.sqlite3?_journal_mode=WAL). An empty value aborts
|
||||
# startup; leave the key out to use the default.
|
||||
# db_url: "file:./data/state.sqlite3?_journal_mode=WAL"
|
||||
|
||||
# Image proxy settings
|
||||
# HMAC signing key for URL signatures (required, at least 32 characters)
|
||||
# Generate with: openssl rand -base64 32
|
||||
signing_key: "CHANGE_ME_generate_with_openssl_rand_base64_32"
|
||||
|
||||
# Hosts that don't require signatures
|
||||
# Hosts that don't require signatures (default: none)
|
||||
# Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com")
|
||||
allowlist_hosts:
|
||||
- s3.sneak.cloud
|
||||
@@ -45,7 +56,7 @@ allowlist_hosts:
|
||||
# SSRF protection. These are added to the always-enforced built-in ranges
|
||||
# (loopback, RFC 1918 private, link-local, CGNAT, benchmark, NAT64, and
|
||||
# similar), never replacing them. Each entry must be a valid CIDR in IPv4
|
||||
# or IPv6 form; an invalid entry aborts startup.
|
||||
# or IPv6 form; an invalid entry aborts startup. (default: none)
|
||||
# blocked_networks:
|
||||
# - 100.64.0.0/10
|
||||
# - 2001:db8::/32
|
||||
@@ -72,6 +83,7 @@ allowlist_hosts:
|
||||
# - 2001:db8::/32
|
||||
|
||||
# Allow HTTP upstream (only for testing, always use HTTPS in production)
|
||||
# (default: false)
|
||||
allow_http: false
|
||||
|
||||
# Maximum concurrent connections per upstream host (default: 20)
|
||||
@@ -121,10 +133,16 @@ access_control_allow_origin: "*"
|
||||
# with a minimum of 500 MiB.
|
||||
# cache_max_bytes: 10737418240
|
||||
|
||||
# Sentry error reporting (optional)
|
||||
# Sentry DSN for error reporting (default: empty, which turns it off)
|
||||
sentry_dsn: ""
|
||||
|
||||
# Metrics endpoint authentication (optional)
|
||||
# Username and password for /metrics, set together (default: unset). Metrics
|
||||
# are measured and /metrics is served only when both are set.
|
||||
# metrics:
|
||||
# username: "admin"
|
||||
# password: "secret"
|
||||
|
||||
# Environment variables set while this file loads, as described at the top
|
||||
# (default: none)
|
||||
# env:
|
||||
# PIXA_DEBUG: "true"
|
||||
|
||||
@@ -191,7 +191,23 @@ func fetchBody(t *testing.T, f *HTTPFetcher, path string) string {
|
||||
|
||||
// semLen reports how many per-host semaphore slots are currently held.
|
||||
func semLen(f *HTTPFetcher, host string) int {
|
||||
return len(f.getHostSemaphore(host))
|
||||
f.hostSemMu.Lock()
|
||||
defer f.hostSemMu.Unlock()
|
||||
|
||||
sem, ok := f.hostSems[host]
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
|
||||
return len(sem.slots)
|
||||
}
|
||||
|
||||
// hostSemCount reports how many hosts have a semaphore in hostSems.
|
||||
func hostSemCount(f *HTTPFetcher) int {
|
||||
f.hostSemMu.Lock()
|
||||
defer f.hostSemMu.Unlock()
|
||||
|
||||
return len(f.hostSems)
|
||||
}
|
||||
|
||||
func TestFetchRedirectToPrivateIPBlocked(t *testing.T) {
|
||||
|
||||
@@ -160,10 +160,13 @@ func DefaultConfig() *Config {
|
||||
// HTTPFetcher implements Fetcher with SSRF protection and connection limits
|
||||
// per host and for all hosts together.
|
||||
type HTTPFetcher struct {
|
||||
client *http.Client
|
||||
config *Config
|
||||
hostSems map[string]chan struct{} // per-host semaphores
|
||||
hostSemMu sync.Mutex // protects hostSems map
|
||||
client *http.Client
|
||||
config *Config
|
||||
// hostSems holds the semaphore of each host with a fetch holding or
|
||||
// waiting for one of its slots; the entry is removed when the host's
|
||||
// last such fetch gives its slot back or stops waiting.
|
||||
hostSems map[string]*hostSemaphore
|
||||
hostSemMu sync.Mutex // protects hostSems and each entry's count
|
||||
// allHostsSemaphore has one slot per connection allowed to all hosts
|
||||
// together (config.MaxConnections).
|
||||
allHostsSemaphore chan struct{}
|
||||
@@ -171,6 +174,14 @@ type HTTPFetcher struct {
|
||||
connectionWaitTimeout time.Duration
|
||||
}
|
||||
|
||||
// hostSemaphore is one host's connection slots
|
||||
// (config.MaxConnectionsPerHost) and the number of fetches holding or
|
||||
// waiting for one of them.
|
||||
type hostSemaphore struct {
|
||||
slots chan struct{}
|
||||
count int
|
||||
}
|
||||
|
||||
// New creates a new HTTPFetcher with SSRF protection.
|
||||
func New(config *Config) *HTTPFetcher {
|
||||
if config == nil {
|
||||
@@ -211,7 +222,7 @@ func New(config *Config) *HTTPFetcher {
|
||||
return &HTTPFetcher{
|
||||
client: client,
|
||||
config: config,
|
||||
hostSems: make(map[string]chan struct{}),
|
||||
hostSems: make(map[string]*hostSemaphore),
|
||||
allHostsSemaphore: make(chan struct{}, config.MaxConnections),
|
||||
connectionWaitTimeout: ConnectionWaitTimeout,
|
||||
}
|
||||
@@ -307,6 +318,8 @@ func (f *HTTPFetcher) acquireConnection(
|
||||
select {
|
||||
case hostSem <- struct{}{}:
|
||||
case <-ctx.Done():
|
||||
f.putHostSemaphore(host)
|
||||
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
|
||||
@@ -314,32 +327,55 @@ func (f *HTTPFetcher) acquireConnection(
|
||||
case f.allHostsSemaphore <- struct{}{}:
|
||||
case <-time.After(f.connectionWaitTimeout):
|
||||
<-hostSem
|
||||
f.putHostSemaphore(host)
|
||||
|
||||
return nil, ErrTooManyConnections
|
||||
case <-ctx.Done():
|
||||
<-hostSem
|
||||
f.putHostSemaphore(host)
|
||||
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
|
||||
return func() {
|
||||
<-hostSem
|
||||
f.putHostSemaphore(host)
|
||||
<-f.allHostsSemaphore
|
||||
}, nil
|
||||
}
|
||||
|
||||
// getHostSemaphore returns the semaphore for a host, creating it if necessary.
|
||||
// getHostSemaphore returns the semaphore for a host, creating it if
|
||||
// necessary, and counts the caller among the fetches using it. The caller
|
||||
// calls putHostSemaphore once it holds no slot and waits for none.
|
||||
func (f *HTTPFetcher) getHostSemaphore(host string) chan struct{} {
|
||||
f.hostSemMu.Lock()
|
||||
defer f.hostSemMu.Unlock()
|
||||
|
||||
sem, ok := f.hostSems[host]
|
||||
if !ok {
|
||||
sem = make(chan struct{}, f.config.MaxConnectionsPerHost)
|
||||
sem = &hostSemaphore{
|
||||
slots: make(chan struct{}, f.config.MaxConnectionsPerHost),
|
||||
}
|
||||
f.hostSems[host] = sem
|
||||
}
|
||||
|
||||
return sem
|
||||
sem.count++
|
||||
|
||||
return sem.slots
|
||||
}
|
||||
|
||||
// putHostSemaphore stops counting the caller among the fetches using the
|
||||
// host's semaphore, and removes the semaphore when no fetch uses it.
|
||||
func (f *HTTPFetcher) putHostSemaphore(host string) {
|
||||
f.hostSemMu.Lock()
|
||||
defer f.hostSemMu.Unlock()
|
||||
|
||||
sem := f.hostSems[host]
|
||||
|
||||
sem.count--
|
||||
if sem.count == 0 {
|
||||
delete(f.hostSems, host)
|
||||
}
|
||||
}
|
||||
|
||||
// buildResult validates the upstream response and assembles a FetchResult
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"errors"
|
||||
"net"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
@@ -119,6 +120,98 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestFetchRemovesIdleHostSemaphores checks that a host's semaphore is
|
||||
// removed once no fetch holds or waits for one of its slots: after 100
|
||||
// concurrent fetches from 50 hosts have all finished, no semaphore is left.
|
||||
func TestFetchRemovesIdleHostSemaphores(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
srv := startUpstream(t)
|
||||
f, _ := newServerFetcher(t, srv, nil)
|
||||
ctx := testContext(t)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
for i := range 100 {
|
||||
wg.Go(func() {
|
||||
res, err := f.Fetch(ctx, imageURLOnPort(1+i%50))
|
||||
if err != nil {
|
||||
t.Errorf("Fetch() error = %v", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
_ = res.Content.Close()
|
||||
})
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
|
||||
if n := hostSemCount(f); n != 0 {
|
||||
t.Errorf("%d host semaphores left after every fetch finished, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFetchRemovesHostSemaphoreWhenNoConnection checks that a fetch that
|
||||
// ends without a connection leaves no semaphore behind: when it is refused
|
||||
// after waiting for a connection shared by all hosts, when its context ends
|
||||
// while it waits for its host's slot, and when its context ends while it
|
||||
// waits for a connection shared by all hosts, long before the 10 second
|
||||
// wait timeout.
|
||||
func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxConnections = 1
|
||||
cfg.MaxConnectionsPerHost = 1
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
f.connectionWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
open, err := f.Fetch(testContext(t), imageURLOnPort(81))
|
||||
if err != nil {
|
||||
t.Fatalf("first Fetch() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = f.Fetch(testContext(t), imageURLOnPort(82))
|
||||
if !errors.Is(err, ErrTooManyConnections) {
|
||||
t.Fatalf("Fetch() from another host: error = %v, "+
|
||||
"want ErrTooManyConnections", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err = f.Fetch(ctx, imageURLOnPort(81))
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("Fetch() from the busy host: error = %v, "+
|
||||
"want context.DeadlineExceeded", err)
|
||||
}
|
||||
|
||||
// Back to the 10 second wait, so the next fetch's context ends first.
|
||||
f.connectionWaitTimeout = ConnectionWaitTimeout
|
||||
|
||||
ctx, cancel = context.WithTimeout(t.Context(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err = f.Fetch(ctx, imageURLOnPort(83))
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("Fetch() from a host with nothing open: error = %v, "+
|
||||
"want context.DeadlineExceeded", err)
|
||||
}
|
||||
|
||||
err = open.Content.Close()
|
||||
if err != nil {
|
||||
t.Fatalf("close first body: %v", err)
|
||||
}
|
||||
|
||||
if n := hostSemCount(f); n != 0 {
|
||||
t.Errorf("%d host semaphores left after every fetch finished, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFetchReleasesConnectionOnError checks that a fetch that fails after
|
||||
// taking its connection gives it back: with MaxConnections at 1, the slot
|
||||
// must be free after the failure and the next fetch must succeed.
|
||||
|
||||
@@ -283,7 +283,7 @@ func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
|
||||
|
||||
c.metaCache.Remove(cacheKey)
|
||||
|
||||
err = c.variants.DeleteWithMeta(cacheKey)
|
||||
err = c.variants.Delete(cacheKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -564,7 +564,8 @@ func (s *VariantStorage) Exists(key VariantKey) bool {
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// Delete removes content at the given key.
|
||||
// Delete removes the content at the given key together with its .meta
|
||||
// sidecar file. A missing file is not an error.
|
||||
func (s *VariantStorage) Delete(key VariantKey) error {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
@@ -573,18 +574,7 @@ func (s *VariantStorage) Delete(key VariantKey) error {
|
||||
return fmt.Errorf("failed to delete content: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteWithMeta removes the content at the given key together with
|
||||
// its .meta sidecar file. A missing file is not an error.
|
||||
func (s *VariantStorage) DeleteWithMeta(key VariantKey) error {
|
||||
err := s.Delete(key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
metaPath := s.keyToPath(key) + ".meta"
|
||||
metaPath := path + ".meta"
|
||||
|
||||
err = os.Remove(metaPath)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
|
||||
@@ -438,3 +438,73 @@ func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
|
||||
t.Errorf("log missing %s; got %q", want, logBuf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// storeTestVariant stores one variant, with its .meta file, in a new
|
||||
// VariantStorage and returns the storage and the variant's key.
|
||||
func storeTestVariant(t *testing.T) (*VariantStorage, VariantKey) {
|
||||
t.Helper()
|
||||
|
||||
storage, err := NewVariantStorage(t.TempDir(), slog.New(slog.DiscardHandler))
|
||||
if err != nil {
|
||||
t.Fatalf("NewVariantStorage() error = %v", err)
|
||||
}
|
||||
|
||||
key := CacheKey(&ImageRequest{SourceHost: testHostCDN, SourcePath: testPathCat})
|
||||
|
||||
_, err = storage.Store(key, bytes.NewReader([]byte("variant data")), "image/webp")
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
return storage, key
|
||||
}
|
||||
|
||||
// TestVariantStorage_DeleteRemovesMeta verifies that Delete removes the
|
||||
// variant's .meta file along with the variant file.
|
||||
func TestVariantStorage_DeleteRemovesMeta(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
storage, key := storeTestVariant(t)
|
||||
metaPath := storage.keyToPath(key) + ".meta"
|
||||
|
||||
_, err := os.Stat(metaPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() wrote no .meta file: %v", err)
|
||||
}
|
||||
|
||||
err = storage.Delete(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete() error = %v", err)
|
||||
}
|
||||
|
||||
if storage.Exists(key) {
|
||||
t.Error("Exists() = true after delete, want false")
|
||||
}
|
||||
|
||||
_, err = os.Stat(metaPath)
|
||||
if !os.IsNotExist(err) {
|
||||
t.Errorf(".meta file left after Delete() (stat err=%v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestVariantStorage_DeleteWithoutMeta verifies that Delete succeeds for a
|
||||
// variant whose .meta file is missing.
|
||||
func TestVariantStorage_DeleteWithoutMeta(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
storage, key := storeTestVariant(t)
|
||||
|
||||
err := os.Remove(storage.keyToPath(key) + ".meta")
|
||||
if err != nil {
|
||||
t.Fatalf("removing .meta file: %v", err)
|
||||
}
|
||||
|
||||
err = storage.Delete(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
if storage.Exists(key) {
|
||||
t.Error("Exists() = true after delete, want false")
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user