Compare commits
1
Commits
next
..
2dd2134f7f
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
2dd2134f7f |
@@ -1,4 +1,4 @@
|
||||
.PHONY: bootstrap setup check lint test fmt fmt-check build clean docker docker-smoke docker-versioned docker-test devserver devserver-stop hooks loadtest bench
|
||||
.PHONY: bootstrap setup check lint test fmt fmt-check build clean docker docker-smoke docker-versioned docker-test devserver devserver-stop hooks loadtest
|
||||
|
||||
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
|
||||
LDFLAGS := -X main.Version=$(VERSION)
|
||||
@@ -64,12 +64,6 @@ docker-smoke:
|
||||
loadtest:
|
||||
@script/loadtest
|
||||
|
||||
# Serve cache hits in parallel through the image handler and report hits per
|
||||
# second (needs Go on the host after `script/bootstrap --cgo`, not Docker; a
|
||||
# benchmark, not part of check)
|
||||
bench:
|
||||
@script/bench
|
||||
|
||||
# Build Docker image as `make docker` does, and also tag it pixa:$(VERSION)
|
||||
docker-versioned:
|
||||
@script/docker
|
||||
|
||||
@@ -76,9 +76,9 @@ database and the disk cache:
|
||||
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
||||
must be able to write the directory.
|
||||
- `cache_max_bytes` limits the source and transformed images together. The
|
||||
database and files still being written come on top, and eviction runs in the
|
||||
background, so the cache can pass the limit for a while: leave room on the
|
||||
volume beyond it.
|
||||
database, the `.meta` file beside each transformed image and files still being
|
||||
written come on top, and eviction runs in the background, so the cache can
|
||||
pass the limit for a while: leave room on the volume beyond it.
|
||||
- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each
|
||||
time pixa starts, is 75% of the sum of the space free on the volume and the
|
||||
space the cached images already take, so a restart keeps the limit the cache
|
||||
@@ -88,17 +88,13 @@ A load balancer's health check can request `/.well-known/healthcheck.json`,
|
||||
which answers 200 whenever pixa is running, in maintenance mode too (see
|
||||
`maintenance_mode`).
|
||||
|
||||
On SIGTERM or SIGINT pixa stops accepting connections. The requests in progress
|
||||
and the images being processed get 5 seconds to finish. A request not finished
|
||||
by then is cut off. pixa then writes the waiting last-used times of cached
|
||||
images to the database (see Storage) and exits. It exits with 1 when:
|
||||
|
||||
- images were still being processed after those 5 seconds,
|
||||
- some of those times could not be written, or
|
||||
- another part of pixa failed to stop.
|
||||
|
||||
Otherwise it exits with 0. `docker stop` waits 10 seconds before it kills the
|
||||
container.
|
||||
On SIGTERM or SIGINT pixa stops accepting connections, gives the requests in
|
||||
progress and the images being processed 5 seconds to finish, then writes to the
|
||||
database the counts of cache hits, misses, fetches and conversions that requests
|
||||
could not write by their deadline, and exits: with 0, or with 1 when images were
|
||||
still being processed after those 5 seconds, some of those counts could not be
|
||||
written, or another part of pixa failed to stop. A request not finished by then
|
||||
is cut off. `docker stop` waits 10 seconds before it kills the container.
|
||||
|
||||
Outside Docker, pixa needs libvips (the image has 8.16) and libheif to run, as
|
||||
it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL
|
||||
@@ -133,7 +129,7 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
||||
filesystem's free space and what the cache holds, and at least 500 MiB, so
|
||||
the cache can grow to three quarters or more of the space they all have
|
||||
left, and to all of it when that is 500 MiB or less.
|
||||
- the rest are in the list under Configuration below
|
||||
- the rest are in the table under Configuration below
|
||||
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
|
||||
the images it processes at once, plus `upstream_connections` times
|
||||
`upstream_max_response_size` for the source images it has fetched that wait
|
||||
@@ -178,93 +174,41 @@ allowlisted source hosts for open access.
|
||||
|
||||
## Design
|
||||
|
||||
### Concepts
|
||||
|
||||
These terms are the names the code uses:
|
||||
|
||||
- **Source**: the image at one upstream URL, named by its host, path and query.
|
||||
- **Source key**: the SHA-256 of a source's host, path and query, which names
|
||||
the source's file and its row in the database.
|
||||
- **Transform**: the output size, format, quality and fit an image request asks
|
||||
for; with the format `auto`, pixa chooses the output format from the `Accept`
|
||||
header.
|
||||
- **Image request**: a source and a transform, which each image route reads from
|
||||
its URL.
|
||||
- **Variant**: the image pixa serves for an image request, made from its source
|
||||
with its transform.
|
||||
- **Variant key**: the SHA-256 of a variant's source, size, format, quality and
|
||||
fit, which names the variant's file and its row in the database, and gives its
|
||||
`ETag`.
|
||||
- **Signed URL**: a `/v1/image/` URL whose `sig` signs its image request and its
|
||||
expiry `exp` with `signing_key`.
|
||||
- **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any
|
||||
expiry, encrypted with a key derived from `signing_key`.
|
||||
- **Login session**: an encrypted cookie that keeps a browser logged in to the
|
||||
generator page for 30 days, or until a logout replaces the random value pixa
|
||||
keeps in the `logout` table (see Encrypted URLs).
|
||||
- **Negative cache**: the source URLs pixa does not fetch again for 5 minutes,
|
||||
after one of these failed fetches:
|
||||
- the upstream answered with an error status
|
||||
- the body's first bytes were not those of the format its `Content-Type`
|
||||
names
|
||||
- **Image cache**: the part of pixa that keeps the sources, the variants and the
|
||||
negative cache (see Storage).
|
||||
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
|
||||
sources and variants may reach together.
|
||||
- **Image proxy**: the part of pixa that serves a variant from the image cache
|
||||
when it can, and otherwise makes it from the source.
|
||||
|
||||
### Storage
|
||||
|
||||
- **Sources**: `<state_dir>/cache/sources/<ab>/<cd>/<source key>`, one file for
|
||||
each source URL
|
||||
- **Source images**:
|
||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one
|
||||
file for each source URL
|
||||
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
||||
- **Variants**: `<state_dir>/cache/variants/<ab>/<cd>/<variant key>`
|
||||
- **Transformed images**:
|
||||
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
|
||||
each with a `.meta` file beside it holding its content type
|
||||
|
||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||
name.
|
||||
|
||||
The image cache keeps these tables in the database:
|
||||
Each source image and each transformed image has one row in the database,
|
||||
holding its size and when it was last used. Two source URLs that serve the same
|
||||
bytes are stored twice, once under each URL. A source image is stored by writing
|
||||
its file and then its row; when the row cannot be written, the file is removed
|
||||
again.
|
||||
|
||||
- `source_content`: a row for each source, by source key, with its size and when
|
||||
it was last used
|
||||
- `variant_content`: a row for each variant, by variant key (column
|
||||
`cache_key`), with its size and when it was last used
|
||||
- `cache_usage`: the total size of the cached sources and variants, which
|
||||
eviction reads
|
||||
- `negative_cache`: a row for each source URL in the negative cache, with when
|
||||
it expires
|
||||
|
||||
pixa reads a cached source image's format from its file's first bytes. When they
|
||||
match none of the source formats, pixa fetches the source again. Two source URLs
|
||||
that serve the same bytes are stored twice, once under each URL. A source image
|
||||
is stored by writing its file and then its row; when the row cannot be written,
|
||||
the file is removed again.
|
||||
|
||||
Each source and variant file is written under a temporary name and synced to
|
||||
disk before it is renamed into place. So a crash never leaves part of a file
|
||||
under its own name. When pixa starts, and every five minutes after, it removes
|
||||
each source or variant whose file is empty, or not the size its row records. It
|
||||
logs a warning naming the file. A later request for that image fetches the
|
||||
source or makes the variant again. Looking for those files reads the database
|
||||
once for every cached file, so pixa skips each five-minute look that would start
|
||||
less than ten times as long after the last look ended as that look took. After a
|
||||
look that took one minute, the next one starts 10 to 15 minutes after that look
|
||||
Source and transformed image files, and the `.meta` file beside each transformed
|
||||
image, are written under a temporary name, synced to disk, and only then renamed
|
||||
into place, so a crash leaves no file under its own name with part of its data.
|
||||
When pixa starts, and every five minutes after, it removes each cached source or
|
||||
transformed image whose file is empty, or not the size the database records for
|
||||
it, with a warning naming the file; the next request for it fetches the source
|
||||
or converts the image again. Looking for those files reads the database once for
|
||||
every cached file, so pixa skips each five-minute look that would start less
|
||||
than ten times as long after the last look ended as that look took. After a look
|
||||
that took one minute, the next one starts 10 to 15 minutes after that look
|
||||
ended.
|
||||
|
||||
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||
Test). Toward it, a cache hit opens the variant's file and nothing else, and
|
||||
makes no database call. The content type comes from the file's first bytes,
|
||||
which are those of the format pixa wrote it in; a file whose first bytes are not
|
||||
those of a format pixa writes is not served, and the image is converted again. A
|
||||
hit hands the time it was served to a background writer and does not wait for
|
||||
the writer. Eviction orders by that time. The hit is counted in memory. A miss
|
||||
that uses a cached source image also hands that writer the time it used the
|
||||
image. Once a second that writer writes the waiting times in one transaction. Up
|
||||
to 10,000 access times of variants, and as many of sources, can wait to be
|
||||
written. A time handed over beyond that is dropped. Its image then only looks
|
||||
less recently used to eviction. `script/bench` measures how many hits pixa
|
||||
serves per second (see Load Test).
|
||||
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
||||
transformed images most recently cached or served, so a cache hit on one of them
|
||||
reads only the image file from disk and not the metadata file stored beside it.
|
||||
|
||||
### Routes
|
||||
|
||||
@@ -293,58 +237,32 @@ path under `/v1/` answers 200, in maintenance mode too.
|
||||
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
|
||||
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
|
||||
and converted (below). Needs: a signature, unless the host is allowlisted (see
|
||||
Source Hosts). Answers:
|
||||
- 200 with the image.
|
||||
- 304 when `If-None-Match` matches the image's `ETag`.
|
||||
- 400 for a URL or parameter that is not valid.
|
||||
- 400 when the format is `auto` and the `Accept` header is not valid.
|
||||
- 406 when the format is `auto` and `Accept` allows none of the formats it
|
||||
chooses from.
|
||||
- 401 for a missing or wrong signature.
|
||||
- 401 for a missing `exp`.
|
||||
- 401 for an `exp` in the past.
|
||||
- 403 when the request's `Referer` names a host in `referer_blocklist`,
|
||||
checked before the signature, the cache and the upstream fetch.
|
||||
- 403 when the upstream host, or a host it redirects to, is `localhost` or
|
||||
ends in `.localhost` or `.local`.
|
||||
- 403 when the upstream host, or a host it redirects to, has an address in a
|
||||
blocked network (see `blocked_networks`).
|
||||
- 429 over `image_requests_per_minute` (below).
|
||||
- 415 when the upstream answers with a `Content-Type` that is not one of the
|
||||
source formats (below).
|
||||
- 415 when the upstream sends a body whose first bytes are not those of the
|
||||
format its `Content-Type` names.
|
||||
- 415 when the upstream sends an image whose first bytes libvips reads as
|
||||
another format.
|
||||
- 502 when the upstream answers with an error status.
|
||||
- 502 for 5 minutes after that for the same source URL, when pixa has
|
||||
neither the image nor its source image in its cache.
|
||||
- 502 likewise for 5 minutes after a body whose first bytes are not those of
|
||||
its `Content-Type`.
|
||||
- 503 when pixa is busy.
|
||||
- 503 when pixa is in maintenance mode.
|
||||
- 504 when the upstream has not sent its response headers within
|
||||
`upstream_fetch_timeout`.
|
||||
- 500 when that time runs out while the image itself is still arriving.
|
||||
- 500 for any other failure.
|
||||
Source Hosts). Answers: 200; 304 when `If-None-Match` matches the image's
|
||||
`ETag`; 400 for a URL or parameter that is not valid, or for the format `auto`
|
||||
an `Accept` header that is not valid; 406 for the format `auto` when `Accept`
|
||||
allows none of the formats it chooses from; 401 for a missing or wrong
|
||||
signature, a missing `exp` or an `exp` in the past; 403 when the request's
|
||||
`Referer` names a host in `referer_blocklist`, checked before the signature,
|
||||
the cache and the upstream fetch; 403 when the upstream host, or a host it
|
||||
redirects to, is `localhost`, ends in `.localhost` or `.local`, or has an
|
||||
address in a blocked network (see `blocked_networks`); 429 over
|
||||
`image_requests_per_minute` (below); 415 when the upstream answers with a
|
||||
`Content-Type` that is not one of the source formats (below), with a body
|
||||
whose first bytes are not those of the format its `Content-Type` names, or
|
||||
with an image whose first bytes libvips reads as another format; 502 when the
|
||||
upstream answered with an error status, and for 5 minutes after that, or after
|
||||
a body whose first bytes are not those of its `Content-Type`, for an image of
|
||||
the same source URL that pixa can neither serve from its cache nor make from
|
||||
the source image still in its cache; 503 when pixa is busy or in maintenance
|
||||
mode; 500 for any other failure.
|
||||
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
|
||||
(see Encrypted URLs). Needs: nothing but the URL. Answers:
|
||||
- 200 with the image.
|
||||
- 304 when `If-None-Match` matches the image's `ETag`.
|
||||
- 400 for a token that does not decrypt.
|
||||
- 400 for a token that asks for a size or fit that is not valid.
|
||||
- 410 once the token has expired.
|
||||
- 400 when the token's format is `auto` and the `Accept` header is not
|
||||
valid.
|
||||
- These answers, for the same causes as on `/v1/image/`:
|
||||
- 406
|
||||
- 403
|
||||
- 415
|
||||
- 429
|
||||
- 502
|
||||
- 503
|
||||
- 504
|
||||
- 500
|
||||
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
|
||||
`If-None-Match` matches the image's `ETag`; 400 for a token that does not
|
||||
decrypt, or that asks for a size or fit that is not valid; 410 once it has
|
||||
expired; 504 when the upstream has not sent its response headers within
|
||||
`upstream_fetch_timeout`, but 500 when that time runs out while the image
|
||||
itself is still arriving; 400 for an `Accept` header that is not valid, and
|
||||
406, 403, 415, 429, 502, 503 and 500, as for `/v1/image/`.
|
||||
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
|
||||
nothing. Answers: 200.
|
||||
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
|
||||
@@ -722,85 +640,54 @@ refused with 400.
|
||||
Every setting can be given as an environment variable, in a YAML config file
|
||||
(`--config`), or both. A variable present in the environment wins over the file,
|
||||
even when it is empty, and the file wins over the built-in default. The one
|
||||
exception is a variable set in the file's `env:` section. pixa sets that
|
||||
variable while loading the file, so the variable wins over the environment pixa
|
||||
started with. It also wins over the file's key for the same setting. A
|
||||
variable's value is parsed as the same text in the file would be. A setting that
|
||||
holds a list takes comma-separated entries, with the spaces around each trimmed.
|
||||
An empty variable is an empty list. A value that does not parse or is invalid
|
||||
aborts startup, naming the variable. pixa refuses to start when a variable
|
||||
starts with `PIXA_` but is not in the list below, such as a misspelled one or
|
||||
`PIXA_PORT`. The error names the variable, as it names an unknown config key.
|
||||
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
|
||||
`--config`. The variables set by the file's `env:` section are checked the same
|
||||
exception is a variable named in the file's `env:` section: it is set while the
|
||||
file loads, so it overrides both the environment the process was started with
|
||||
and the file's own key. A variable's value is parsed as the same text in the
|
||||
file would be. The three lists take comma-separated entries, with the spaces
|
||||
around each trimmed; an empty variable is an empty list. A value that does not
|
||||
parse or is invalid aborts startup, naming the variable. A variable whose name
|
||||
starts with `PIXA_` but is not in the table below, such as a misspelled one or
|
||||
`PIXA_PORT`, aborts startup naming it, as an unknown config key does. The one
|
||||
other accepted name is `PIXA_CONFIG_PATH`, the config file's path (like
|
||||
`--config`). The variables set by the file's `env:` section are checked the same
|
||||
way.
|
||||
|
||||
pixa reads at most one config file. It reads the file given with `--config` (or
|
||||
`-c`), or else the one `PIXA_CONFIG_PATH` names. pixa refuses to start when that
|
||||
file does not exist, cannot be read or does not parse. Without `--config` or
|
||||
`PIXA_CONFIG_PATH`, pixa looks in `/etc/pixa/`, then in `~/.config/pixa/`, then
|
||||
in the working directory. In each directory it looks for `config.yml`, then
|
||||
`config.yaml`. It reads the first file it finds. It skips a missing file without
|
||||
a message. pixa refuses to start when a file it finds cannot be read or does not
|
||||
parse, and the error names the file. It also refuses to start when it may not
|
||||
enter one of these directories, whether or not the file is there, since it
|
||||
cannot tell. With no file, pixa uses the environment and the defaults.
|
||||
pixa reads at most one config file: the one given with `--config` (or `-c`),
|
||||
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that
|
||||
pixa finds: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`,
|
||||
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml` and
|
||||
`config.yaml` in the working directory. A named file that does not exist, cannot
|
||||
be read or does not parse aborts startup. Of the files pixa looks for on its
|
||||
own, only one that does not exist is passed over, without a message. One that
|
||||
pixa cannot read or parse aborts startup, naming the file. So does one in a
|
||||
directory pixa may not enter, whether or not it is there, since pixa cannot
|
||||
tell. With no file, pixa uses the environment and the defaults.
|
||||
|
||||
Each item below is one setting, named by its config key:
|
||||
|
||||
- `signing_key`: variable `PIXA_SIGNING_KEY`, required. The secret for signed
|
||||
URLs, encrypted URLs and login. It must be 32 characters or longer.
|
||||
- `port`: variable `PORT`, default `8080`. The port to listen on.
|
||||
- `state_dir`: variable `PIXA_STATE_DIR`, default `/var/lib/pixa`. The directory
|
||||
for the database and the disk cache.
|
||||
- `db_url`: variable `PIXA_DB_URL`, default `state.sqlite3` in the state
|
||||
directory. The URL of the SQLite database.
|
||||
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default 75% of the sum of
|
||||
the free space and what the cache holds. The disk cache limit in bytes. `0`
|
||||
turns the disk cache off.
|
||||
- `allowlist_hosts`: variable `PIXA_ALLOWLIST_HOSTS`, default none. The upstream
|
||||
hosts whose images pixa serves without a signature.
|
||||
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. The
|
||||
image routes refuse a request with 403 when its `Referer` header names one of
|
||||
these hosts.
|
||||
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. The CIDR
|
||||
ranges pixa never fetches from, on top of the built-in ones.
|
||||
- `trusted_proxies`: variable `PIXA_TRUSTED_PROXIES`, default the RFC 1918
|
||||
ranges. The CIDR ranges of the proxies whose `X-Forwarded-For` header pixa
|
||||
believes.
|
||||
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. Allows plain-HTTP
|
||||
upstream hosts. It is for testing only.
|
||||
- `upstream_connections_per_host`: variable
|
||||
`PIXA_UPSTREAM_CONNECTIONS_PER_HOST`, default `20`. The most connections to
|
||||
one upstream host at once.
|
||||
- `upstream_connections`: variable `PIXA_UPSTREAM_CONNECTIONS`, default `64`.
|
||||
The most connections to all upstream hosts together at once.
|
||||
- `max_concurrent_processing`: variable `PIXA_MAX_CONCURRENT_PROCESSING`,
|
||||
default half the CPUs, at least 1. The most images processed at once.
|
||||
- `image_requests_per_minute`: variable `PIXA_IMAGE_REQUESTS_PER_MINUTE`,
|
||||
default `600`. How many image requests one client may send per minute. `0`
|
||||
turns the limit off.
|
||||
- `upstream_fetch_timeout`: variable `PIXA_UPSTREAM_FETCH_TIMEOUT`, default
|
||||
`30s`. The time allowed for one fetch from an upstream host.
|
||||
- `upstream_max_response_size`: variable `PIXA_UPSTREAM_MAX_RESPONSE_SIZE`,
|
||||
default 50 MiB. The largest upstream response accepted, in bytes.
|
||||
- `downstream_timeout`: variable `PIXA_DOWNSTREAM_TIMEOUT`, default `60s`. The
|
||||
time allowed for answering one client request.
|
||||
- `access_control_allow_origin`: variable `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN`,
|
||||
default `*`. The CORS origin allowed to read image responses. It is `*` or one
|
||||
origin.
|
||||
- `metrics.username`: variable `PIXA_METRICS_USERNAME`, default empty. The
|
||||
username for `/metrics`. pixa serves `/metrics` only when the username and the
|
||||
password are both set.
|
||||
- `metrics.password`: variable `PIXA_METRICS_PASSWORD`, default empty. The
|
||||
password for `/metrics`. Set it together with the username.
|
||||
- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. The Sentry DSN for
|
||||
error reporting. An empty DSN turns error reporting off.
|
||||
- `debug`: variable `PIXA_DEBUG`, default `false`. Turns on debug logging and
|
||||
plain-HTTP local development.
|
||||
- `maintenance_mode`: variable `PIXA_MAINTENANCE_MODE`, default `false`. While
|
||||
`true`, pixa answers image requests with 503. The health check still
|
||||
answers 200.
|
||||
| Variable | Config key | Meaning |
|
||||
| ------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| `PIXA_SIGNING_KEY` | `signing_key` | Required: secret for signed and encrypted URLs and login, 32+ characters |
|
||||
| `PORT` | `port` | Port to listen on; default `8080` |
|
||||
| `PIXA_STATE_DIR` | `state_dir` | Directory for the database and the disk cache; default `/var/lib/pixa` |
|
||||
| `PIXA_DB_URL` | `db_url` | SQLite database URL; default `state.sqlite3` in the state directory |
|
||||
| `PIXA_CACHE_MAX_BYTES` | `cache_max_bytes` | Disk cache limit in bytes; `0` disables it; default 75% of (free + cached) |
|
||||
| `PIXA_ALLOWLIST_HOSTS` | `allowlist_hosts` | Upstream hosts served without a signature |
|
||||
| `PIXA_REFERER_BLOCKLIST` | `referer_blocklist` | Hosts whose pages the image routes refuse with 403, by `Referer` |
|
||||
| `PIXA_BLOCKED_NETWORKS` | `blocked_networks` | CIDR ranges never fetched from, on top of the built-in ones |
|
||||
| `PIXA_TRUSTED_PROXIES` | `trusted_proxies` | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 |
|
||||
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
||||
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
|
||||
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
|
||||
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 |
|
||||
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
|
||||
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
|
||||
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
|
||||
| `PIXA_DOWNSTREAM_TIMEOUT` | `downstream_timeout` | Time allowed for answering one client request; default `60s` |
|
||||
| `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN` | `access_control_allow_origin` | CORS origin allowed to read image responses: `*` or one origin; default `*` |
|
||||
| `PIXA_METRICS_USERNAME` | `metrics.username` | Username for `/metrics`, which is served only when both are set |
|
||||
| `PIXA_METRICS_PASSWORD` | `metrics.password` | Password for `/metrics`; set together with the username |
|
||||
| `PIXA_SENTRY_DSN` | `sentry_dsn` | Sentry DSN for error reporting; empty disables it |
|
||||
| `PIXA_DEBUG` | `debug` | Debug logging and plain-HTTP local development; default `false` |
|
||||
| `PIXA_MAINTENANCE_MODE` | `maintenance_mode` | Answer image requests with 503; the health check stays 200; default `false` |
|
||||
|
||||
Key settings in more detail:
|
||||
|
||||
@@ -867,17 +754,7 @@ Key settings in more detail:
|
||||
duration; default `60s`. The upstream fetch counts toward it, and so do the
|
||||
waits for an upstream connection and for a processing slot (up to 10 seconds
|
||||
each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds
|
||||
- `signing_key` — the secret that protects signed URLs, encrypted URLs and
|
||||
logins. pixa uses it as it is for two things:
|
||||
- the HMAC key of signed URLs (see Signature Specification)
|
||||
- the password of the login page
|
||||
|
||||
pixa derives every other key it uses from it at startup, with HKDF-SHA256:
|
||||
- a key that signs the session cookie
|
||||
- a key that encrypts the session cookie
|
||||
- a key that encrypts the token of an encrypted URL
|
||||
- a key that signs the CSRF token cookie of the login and generator pages
|
||||
|
||||
- `signing_key` — HMAC secret for URL signatures
|
||||
- `db_url` — the SQLite database to open; omitted, it is
|
||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
||||
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
|
||||
@@ -924,50 +801,20 @@ See `configs/config.example.yml` for all options with defaults.
|
||||
|
||||
### Architecture
|
||||
|
||||
- **Dependency injection**: Uber fx. `internal/app` puts every part of pixad in
|
||||
one fx graph. `internal/providers` builds the image cache, the image proxy and
|
||||
the other parts that serve images and logins. Each setting's default is
|
||||
declared once, in `internal/config`.
|
||||
- **Packages**: these packages under `internal/` work with the concepts above:
|
||||
- `handlers`: the routes, which read the image request from the URL, check
|
||||
the signed or encrypted URL, and make the login session
|
||||
- `imagerequest`: the image request, with the source key and the variant key
|
||||
- `imageproxy`: the image proxy
|
||||
- `imgcache`: the image cache, which keeps its sources in one `FileStore`
|
||||
and its variants in another
|
||||
- `httpfetcher`: fetching sources from upstream hosts
|
||||
- `magic`: reading an image's format from its first bytes
|
||||
- `imageprocessor`: making a variant from a source with libvips
|
||||
- `signature`: checking the `sig` of a signed URL
|
||||
- `encurl`: making and reading the tokens of encrypted URLs
|
||||
- `seal`: the keys derived from `signing_key`, and the encryption of those
|
||||
tokens
|
||||
- `session`: the login session
|
||||
- **Dependency injection**: Uber fx
|
||||
- **HTTP router**: go-chi
|
||||
- **Image processing**: govips (CGO wrapper for libvips)
|
||||
- **Database**: SQLite via modernc.org/sqlite
|
||||
- **Static assets**: embedded via `//go:embed`
|
||||
- **Metrics**: Prometheus, at `/metrics`. pixa serves `/metrics` and measures
|
||||
HTTP requests only when `metrics.username` and `metrics.password` are set.
|
||||
`/metrics` holds:
|
||||
- the duration of HTTP requests
|
||||
- the size of HTTP responses
|
||||
- the HTTP requests in flight
|
||||
- the Go runtime and process metrics
|
||||
- `pixa_cache_hits_total`: image requests served from the cache
|
||||
- `pixa_cache_misses_total`: image requests not served from the cache
|
||||
- `pixa_upstream_fetches_total`: fetches of a source image that read any
|
||||
bytes
|
||||
- `pixa_upstream_fetch_bytes_total`: the bytes those fetches read
|
||||
- `pixa_transforms_total`: images converted
|
||||
|
||||
A request's `handler` label is the pattern of the route it matches, such as
|
||||
`/v1/image/*`, never its path. Requests that match no route share the label
|
||||
`unmatched`: those the router answers 404 or 405, and CORS preflights. A
|
||||
request's `method` label is its method when that is `GET`, `HEAD`, `POST`,
|
||||
`PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` or `TRACE`, and `other` for
|
||||
any other method.
|
||||
|
||||
- **Metrics**: Prometheus, at `/metrics`: generic HTTP request metrics
|
||||
(duration, response size, requests in flight) and the Go runtime and process
|
||||
metrics; requests are measured and `/metrics` is served only when
|
||||
`metrics.username` and `metrics.password` are set. A request's `handler` label
|
||||
is the pattern of the route it matches, such as `/v1/image/*`, never its path.
|
||||
Requests that match no route share the label `unmatched`: those the router
|
||||
answers 404 or 405, and CORS preflights. A request's `method` label is its
|
||||
method when that is `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`,
|
||||
`OPTIONS`, `CONNECT` or `TRACE`, and `other` for any other method
|
||||
- **Logging**: stdlib slog
|
||||
|
||||
## Entrypoints
|
||||
@@ -999,9 +846,6 @@ provide:
|
||||
- `script/docker-smoke` — build the image, start it, wait for it to be healthy
|
||||
- `script/loadtest` — measure pixad's throughput, latency and peak memory; a
|
||||
benchmark, not part of `script/check` (see Load Test)
|
||||
- `script/bench` — run the Go benchmark that serves cache hits in parallel
|
||||
through the image handler, with Go on the host after `script/bootstrap --cgo`;
|
||||
a benchmark, not part of `script/check` (see Load Test)
|
||||
- `script/cibuild` — CI entrypoint: run `script/bootstrap` (without `--cgo`),
|
||||
then `script/check`, then build the image as `script/docker` does
|
||||
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
||||
@@ -1082,17 +926,6 @@ The numbers depend on the machine and on whatever else runs on it. The first
|
||||
measurement, made on a shared machine with few clients, is in `TODO.md`; it says
|
||||
nothing about the target.
|
||||
|
||||
`script/bench` (or `make bench`) measures cache hits alone, without the network
|
||||
and with no pixad process: a Go benchmark in `internal/handlers` serves one
|
||||
cached image through the `/v1/image/` handler from as many goroutines at once as
|
||||
the machine has CPUs, and reports the hits served per second as `hits/s`. Each
|
||||
run starts with an empty cache and an empty database, a SQLite file of its own,
|
||||
and puts the image in the cache before it measures. The benchmark fails if
|
||||
serving the hits ran any statement on the database. Like the load test it is not
|
||||
part of `script/check`. It runs Go on the host, as the load test does, and
|
||||
compiles pixa there, so it needs the C compiler and libraries that
|
||||
`script/bootstrap --cgo` installs, but not Docker.
|
||||
|
||||
## TODO
|
||||
|
||||
See [TODO.md](TODO.md) for the full prioritized task list.
|
||||
|
||||
@@ -30,249 +30,6 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-26 `README.md`'s Design section starts with a Concepts list (closes
|
||||
#321). Each concept gets one sentence, under the name the code gives it.
|
||||
- Storage calls the transformed images variants, in shorter sentences.
|
||||
- Storage's paths end in the source key and the variant key.
|
||||
- Storage lists the image cache's tables in the database.
|
||||
- Architecture names `internal/providers`, and lists the packages that work
|
||||
with the concepts.
|
||||
- 2026-10-26 one table in `internal/config` lists every setting once (closes
|
||||
#320). `Config.settings` gives each setting's key, its environment variable
|
||||
and the function that loads it.
|
||||
- Most rows load through `field`, whose arguments give the setting's type,
|
||||
its default and its check.
|
||||
- `db_url` and `cache_max_bytes` have their own load functions. The default
|
||||
of `db_url` comes from `state_dir`, and `cache_max_bytes` records whether
|
||||
it is set.
|
||||
- Loading, the unknown-key check, the unknown-variable check and the names
|
||||
in error messages all read the table.
|
||||
- `allowlist_hosts` loads as `referer_blocklist` does, through
|
||||
`parseHostList`.
|
||||
- The metrics username and password are checked together once the table has
|
||||
loaded.
|
||||
- When a config holds more than one invalid value, the error names the one
|
||||
that comes first in the table.
|
||||
- `README.md` lists the settings one item each, where it had a table. A test
|
||||
checks that this list and `configs/config.example.yml` name exactly the
|
||||
keys of the table.
|
||||
- The key constants moved to the tests, their only users.
|
||||
- Removed:
|
||||
- `isKnownConfigKey`, `envVarNames`, `strictLoader` and `validate`
|
||||
- `validateAllowlistHostsValue` and `getStringSlice`, with the three
|
||||
tests of `getStringSlice`
|
||||
- 2026-10-26 a fetched image's format is detected from its bytes once, as an
|
||||
`imagerequest.Format` (closes #319). The upstream `Content-Type` check in the
|
||||
fetcher stays as the filter before downloading.
|
||||
- `magic.DetectFormat` returns an `imagerequest.Format`, and
|
||||
`magic.ValidateMagicBytes` returns the format it checked against the
|
||||
`Content-Type`.
|
||||
- `Format.MIMEType` gives each format's MIME type. The fetcher, the mock
|
||||
fetcher, the cache's read of a variant's type and the handler's `Accept`
|
||||
matching use it.
|
||||
- `Cache.Source` returns a source's format, read from its file's first bytes
|
||||
as `Cache.Variant` reads a variant's. A file in no source format is an
|
||||
error, and the proxy fetches the source again.
|
||||
- The proxy hands the format to `ImageProcessor.Process`. The processor
|
||||
refuses input that libvips would load as another format, as before, with
|
||||
`ErrUnsupportedInputFormat`. It refuses a format that is not one of the
|
||||
six source formats the same way.
|
||||
- `internal/magic` no longer detects SVG, which pixa does not read. SVG data
|
||||
is `ErrUnknownFormat`.
|
||||
- Removed:
|
||||
- `magic.MIMEType`, `magic.ImageFormat` and `magic.MIMEToImageFormat`
|
||||
- `detectFormat`, `formatFromString`, `FormatToMIME` and
|
||||
`Result.InputFormat` of the processor
|
||||
- the fetcher's MIME constants of the six formats
|
||||
- `errNotAnOutputFormat` of the cache
|
||||
- 2026-10-26 the cache keeps its source images in one `imgcache.FileStore` and
|
||||
its variants in another (closes #318). A `FileStore` is a directory of files,
|
||||
each named by a hex key, at `<ab>/<cd>/<key>`.
|
||||
- `ContentStorage` and `VariantStorage` are gone.
|
||||
- `FileStore` has these methods:
|
||||
- `Write`, which writes a file under a temp name, syncs it and renames
|
||||
it into place
|
||||
- `Open`, which returns the file and its size
|
||||
- `Size`
|
||||
- `Delete`
|
||||
- `FileStore` has no `Exists`. The reconciliation pass checks that a
|
||||
variant's file is there with `Size`. The tests look for the file on disk.
|
||||
- `Cache.Variant` reads the content type from the variant file's first
|
||||
bytes, as `VariantStorage.LoadWithContentType` did. A cache hit still
|
||||
makes no database call.
|
||||
- `storage.go` is now `filestore.go`. `internal/imgcache/testdata/red.avif`,
|
||||
which no test used, is gone.
|
||||
- 2026-10-26 `internal/imgcache` holds the cache only (closes #317). The new
|
||||
package `internal/imageproxy` fetches, transforms, caches and serves images.
|
||||
It imports `imgcache`, and `imgcache` does not import it.
|
||||
- `imgcache.Service` is `imageproxy.Proxy`, built by `imageproxy.New` from
|
||||
an `imageproxy.Config`.
|
||||
- `ImageResponse`, `CacheStatus`, `ErrNegativeCached` and
|
||||
`ErrProcessingTimedOut` moved with it, under their names.
|
||||
- `imgcache.CacheConfig` is `imgcache.Config`.
|
||||
- The cache has one read and one write for each kind it keeps. Each takes
|
||||
the place of an earlier method:
|
||||
- `Variant`, of `GetVariant`
|
||||
- `PutVariant`, of `StoreVariant`
|
||||
- `Source`, of `LookupSource` followed by `GetSourceContent`
|
||||
- `PutSource`, of `StoreSource`
|
||||
- `Failure`, of `checkNegativeCache`
|
||||
- `PutFailure`, of `StoreNegative`
|
||||
- `Variant` and `Source` each open the file and hand its access time to the
|
||||
background writer. A cache hit still makes no database call.
|
||||
- `imgcache.go` and `module.go` are gone.
|
||||
- `imageproxy.New` makes the counters of upstream fetches and transforms,
|
||||
and registers them on the registry it is given. The cache keeps only the
|
||||
hit and miss counters.
|
||||
- A cached source whose row cannot be read is logged as "failed to load
|
||||
cached source, fetching". It was logged as "source lookup failed".
|
||||
- The proxy's tests moved to `internal/imageproxy`. They reach the cache
|
||||
through its database, its state directory and its metrics registry.
|
||||
- The tests of an empty or cut cached file are split in two:
|
||||
- `internal/imgcache` checks that the reconciliation pass removes the
|
||||
file and its row, with a warning naming the file.
|
||||
- `internal/imageproxy` removes the file and its row as the pass does,
|
||||
and checks that `Get` fetches or converts again.
|
||||
- 2026-10-26 both image routes serve an image with one function, `serveImage`,
|
||||
and answer its errors with one, `respondImageError` (closes #316).
|
||||
`HandleImage` parses the path and verifies the signature, and `HandleImageEnc`
|
||||
decrypts the token. Each then calls `serveImage`.
|
||||
- `handleImageError`, the encrypted route's own mapping, is gone.
|
||||
- `/v1/image/` answers 504 when the upstream has not sent its response
|
||||
headers within `upstream_fetch_timeout`, as `/v1/e/` did. It answered 500.
|
||||
- `respondImageError` logs the image service's errors as "failed to get
|
||||
image". Each answer gets the same level on `/v1/image/` and on `/v1/e/`:
|
||||
- 500: error, as pixa's own failure
|
||||
- 400: warn
|
||||
- 403: warn
|
||||
- 415: warn
|
||||
- 502: warn
|
||||
- 503: warn
|
||||
- 504: warn
|
||||
- Before, `/v1/image/` logged all of these answers at error level. `/v1/e/`
|
||||
logged only its 500, as "image request failed", without the host and the
|
||||
path.
|
||||
- `respondImageError` does not log the 415 for a body refused for its first
|
||||
bytes, on either route. The image service logs it as "source refused", at
|
||||
warn level.
|
||||
- The "image served" line carries the host, the path and `fetched_bytes` on
|
||||
both routes. The debug line "encrypted image request" is gone.
|
||||
- The debug line "failed to decrypt URL" is now "invalid encrypted URL". A
|
||||
token that decrypts to a payload that does not decode also gets it.
|
||||
- A failed write is logged once, as "failed to write response" at warn
|
||||
level, since most often the client has gone away. It carries the
|
||||
`request_id`, the host, the path, `served_bytes` and the error, and is not
|
||||
also logged as "image served".
|
||||
- 2026-10-26 every key derived from the signing key is derived in one place
|
||||
(closes #315): `seal.DeriveKeys` returns them as one `seal.Keys`, with their
|
||||
salts declared beside it. `internal/providers` calls it once at startup. Each
|
||||
key is derived as before, with `seal.DeriveKey` and the same salt.
|
||||
- The session manager, the encrypted URL generator and the CSRF protection
|
||||
each take their own key instead of the signing key.
|
||||
- `session.NewManager`, `encurl.NewGenerator`, `newCSRFProtect` and
|
||||
`handlers.New` return no error, as nothing in them can fail now.
|
||||
- The signing key itself stays the HMAC key of signed URLs and the login
|
||||
password.
|
||||
- `README.md` says under `signing_key` what it protects.
|
||||
- 2026-10-26 the image request holds only what names the cached object (closes
|
||||
#314): its source and its transform, which the variant key covers. The signed
|
||||
URL format, the signing algorithm and every route's answers are unchanged.
|
||||
- The `/v1/image/` handler checks `allowlist_hosts` and verifies the
|
||||
signature itself, with `Signer.Verify(req, sig, expires)`.
|
||||
- Each image handler keeps its URL's expiry as a value of its own, for
|
||||
`Cache-Control`. `encurl`'s `Generate` takes the expiry and its `Parse`
|
||||
returns it.
|
||||
- The fetcher takes the source and builds its URL with `Source.URL`, over
|
||||
http only when the fetcher's own `AllowHTTP` is set. `validateURL` takes
|
||||
that URL, so no URL string is built and parsed again.
|
||||
- `chooseAutoFormat` returns the format it chooses, and the handler puts it
|
||||
into the request it parsed.
|
||||
- The image service serves images through `Get`, and has `WaitForProcessing`
|
||||
for shutdown.
|
||||
- Removed:
|
||||
- `Signature`, `Expires`, `AllowHTTP` and `SourceURL` of the image
|
||||
request
|
||||
- `Service.ValidateRequest`, and the service's signer, allowlist and
|
||||
`AllowHTTP`
|
||||
- `extractHost` of the fetcher and `urlToFSPath` of the mock fetcher
|
||||
- 2026-10-26 pixad's fx graph builds the image pipeline (closes #313).
|
||||
`database.New` now opens the database, so the parts below can read it while
|
||||
they are built; its stop hook still runs after the cache's. The cache's
|
||||
`MaxBytes` is the one value for the disk cache limit, and 0 turns the disk
|
||||
cache off. For an omitted `cache_max_bytes`, `internal/providers` works out
|
||||
`config.DefaultCacheMaxBytes` before it builds the cache. `internal/app` and
|
||||
the server tests build the parts below with `providers.Constructors()`; the
|
||||
image proxy test replaces only the fetcher's dial function.
|
||||
- Built by the providers in `internal/providers`, and given to
|
||||
`handlers.New`:
|
||||
- the cache
|
||||
- the fetcher
|
||||
- the processor
|
||||
- the image service
|
||||
- the session manager
|
||||
- the encrypted URL generator
|
||||
- Removed:
|
||||
- `initImageService` and `handlers.Params.Fetcher`
|
||||
- `CacheConfig.UseDefaultMaxBytes`, `CacheConfig.DisableDiskCache` and
|
||||
`Cache.disabled`
|
||||
- `httpfetcher.DefaultConfig()` and the fetcher's copies of setting
|
||||
defaults
|
||||
- `imageprocessor.DefaultMaxInputBytes`, and the processor's defaults
|
||||
for a zero `MaxInputBytes` or `MaxConcurrentProcessing`
|
||||
- 2026-10-26 the code nothing in production called is gone (closes #312). The
|
||||
handlers tests sign their image URLs with `generateSignedURL`, a test helper
|
||||
that writes them as the removed generator did. Removed:
|
||||
- `Service.Warm` and the `ImageCache` interface
|
||||
- `Service.GenerateSignedURL` and `Signer.GenerateSignedURL`
|
||||
- `signature.ParseParams`
|
||||
- `imagerequest.ParseImagePath`
|
||||
- `PeekAndValidate`, `IsSupportedMIMEType`, `ImageFormatToMIME` and
|
||||
`FormatOriginal` of `internal/magic`
|
||||
- `SupportedInputFormats` and `SupportedOutputFormats` of the image
|
||||
processor
|
||||
- `IsEmpty` and `Count` of the allowlist
|
||||
- `ssrfSafeDialer`
|
||||
- `CacheStale`, `ErrCacheMiss`, `ErrNegativeCache` and
|
||||
`ImageResponse.LastModified`
|
||||
- `ContentStorage.Load` and `VariantStorage.Load`
|
||||
- 2026-10-26 cache hits, misses, upstream fetches, the bytes those fetches read
|
||||
and transforms are counted as Prometheus counters in memory (closes #311).
|
||||
They are `pixa_cache_hits_total`, `pixa_cache_misses_total`,
|
||||
`pixa_upstream_fetches_total`, `pixa_upstream_fetch_bytes_total` and
|
||||
`pixa_transforms_total`, fields of the cache. `NewCache` makes them and
|
||||
registers them on the registry it is given (`internal/imgcache/counters.go`).
|
||||
pixad gives it Prometheus' default registry, which `/metrics` serves; each
|
||||
test gives it a new one. The `Increment` methods add to their cache's
|
||||
counters. `cache_stats`, `Cache.Stats`, `Service.Stats`, `CacheStats` and the
|
||||
pending counts are gone. The background goroutine writes only access times
|
||||
now: `StartPendingAccessTimeWrites` starts it and
|
||||
`StopPendingAccessTimeWrites` stops it, in `pendingaccesstimes.go`.
|
||||
- 2026-10-21 the schema file declares only the tables and columns pixa writes
|
||||
and reads (closes #310): `output_content` and `request_cache` are gone, and so
|
||||
are the columns nothing read: the host, path, query, `content_type` and
|
||||
`fetched_at` of `source_content`, the `content_type` and `created_at` of
|
||||
`variant_content`, and the `status_code`, `error_message` and `fetched_at` of
|
||||
`negative_cache`. `StoreSource` no longer takes the fetch result,
|
||||
`LookupSource` returns only the source key, `StoreVariant` no longer takes a
|
||||
content type, `StoreNegative` no longer takes a status code or message, and
|
||||
reconciliation adopts a variant file without reading its first bytes.
|
||||
- 2026-10-21 a cache hit makes no database call (closes #309): `Service.Get`
|
||||
opens the variant file with `Cache.GetVariant` and nothing else, with no
|
||||
existence check first. The content type comes from the file's first bytes; a
|
||||
file whose first bytes are not those of an output format, SVG included, is a
|
||||
miss, and the variant is made again. A hit hands its access time to a channel
|
||||
that holds up to 10,000 and drops the rest, and its hit count to the pending
|
||||
counts, without waiting; every count is now a pending count. A miss that finds
|
||||
a cached source hands that source's access time to a second such channel. The
|
||||
goroutine `StartPendingCountWrites` starts writes, once a second, the access
|
||||
times in one transaction and the counts in one `UPDATE`;
|
||||
`StopPendingCountWrites` writes what is left once at shutdown and logs
|
||||
separately the access times and the counts it could not write.
|
||||
`checkNegativeCache` no longer deletes an expired row; `CleanExpired` in the
|
||||
eviction goroutine does. The `.meta` files, `VariantMeta`, `LoadWithMeta`,
|
||||
`variantContentTypeFromSidecar`, `metaCache`, `Lookup` and `golang-lru` are
|
||||
gone. `make bench` (`script/bench`) runs `BenchmarkHandleImage_CacheHits` with
|
||||
Go on the host after `script/bootstrap --cgo`, outside `make check`.
|
||||
- 2026-10-21 a source is stored the way a variant is (closes #308): one
|
||||
`source_content` row and one file per source URL, the file named by the
|
||||
source's `SourceKey`, the SHA-256 of its host, path and query made as the
|
||||
|
||||
@@ -14,6 +14,7 @@ require (
|
||||
github.com/go-chi/httprate v0.16.0
|
||||
github.com/gorilla/csrf v1.7.3
|
||||
github.com/gorilla/securecookie v1.1.2
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||
github.com/prometheus/client_golang v1.23.2
|
||||
github.com/slok/go-http-metrics v0.13.0
|
||||
github.com/spf13/cobra v1.10.2
|
||||
|
||||
@@ -228,6 +228,8 @@ github.com/hashicorp/go-version v1.2.1/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09
|
||||
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
|
||||
github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
|
||||
github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/hashicorp/hcl v1.0.1-vault-7 h1:ag5OxFVy3QYTFTJODRzTKVZ6xvdfLLCA1cy/Y6xGI0I=
|
||||
github.com/hashicorp/hcl v1.0.1-vault-7/go.mod h1:XYhtn6ijBSAj6n4YqAaf7RBPS4I06AItNorpy+MoQNM=
|
||||
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
|
||||
|
||||
@@ -72,3 +72,13 @@ func (w *HostAllowList) IsAllowed(u *url.URL) bool {
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// IsEmpty returns true if the allow list has no entries.
|
||||
func (w *HostAllowList) IsEmpty() bool {
|
||||
return len(w.exactHosts) == 0 && len(w.suffixHosts) == 0
|
||||
}
|
||||
|
||||
// Count returns the total number of allow list entries.
|
||||
func (w *HostAllowList) Count() int {
|
||||
return len(w.exactHosts) + len(w.suffixHosts)
|
||||
}
|
||||
|
||||
@@ -139,3 +139,87 @@ func TestHostAllowList_IsAllowed_SuffixMatch(t *testing.T) {
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestHostAllowList_IsEmpty(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
patterns []string
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "empty",
|
||||
patterns: []string{},
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "nil",
|
||||
patterns: nil,
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "whitespace only",
|
||||
patterns: []string{" ", ""},
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "has entries",
|
||||
patterns: []string{"example.com"},
|
||||
want: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w := allowlist.New(tt.patterns)
|
||||
if got := w.IsEmpty(); got != tt.want {
|
||||
t.Errorf("IsEmpty() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHostAllowList_Count(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
patterns []string
|
||||
want int
|
||||
}{
|
||||
{
|
||||
name: "empty",
|
||||
patterns: []string{},
|
||||
want: 0,
|
||||
},
|
||||
{
|
||||
name: "exact hosts only",
|
||||
patterns: []string{"a.com", "b.com", "c.com"},
|
||||
want: 3,
|
||||
},
|
||||
{
|
||||
name: "suffix hosts only",
|
||||
patterns: []string{".a.com", ".b.com"},
|
||||
want: 2,
|
||||
},
|
||||
{
|
||||
name: "mixed",
|
||||
patterns: []string{"exact.com", ".suffix.com"},
|
||||
want: 2,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w := allowlist.New(tt.patterns)
|
||||
if got := w.Count(); got != tt.want {
|
||||
t.Errorf("Count() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/spf13/cobra"
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
@@ -17,7 +16,6 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||
"sneak.berlin/go/pixa/internal/logger"
|
||||
"sneak.berlin/go/pixa/internal/middleware"
|
||||
"sneak.berlin/go/pixa/internal/providers"
|
||||
"sneak.berlin/go/pixa/internal/server"
|
||||
)
|
||||
|
||||
@@ -63,10 +61,7 @@ func run(_ *cobra.Command, _ []string) {
|
||||
server.New,
|
||||
middleware.New,
|
||||
healthcheck.New,
|
||||
// The registry /metrics serves
|
||||
func() prometheus.Registerer { return prometheus.DefaultRegisterer },
|
||||
),
|
||||
fx.Provide(providers.Constructors()...),
|
||||
fx.Invoke(
|
||||
func(log *logger.Logger) { log.Identify() },
|
||||
func(*server.Server) {},
|
||||
|
||||
@@ -135,38 +135,10 @@ func TestCacheMaxBytesInvalidValuesAbortStartup(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDefaultCacheMaxBytesAppliesFloor verifies that when 75% of the
|
||||
// free space plus what the cache holds is below 500 MiB, the default is
|
||||
// floored at DefaultCacheMaxBytesFloor.
|
||||
func TestDefaultCacheMaxBytesAppliesFloor(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
freeBytes uint64
|
||||
}{
|
||||
{name: "100 MiB free", freeBytes: 104857600},
|
||||
{name: "zero free", freeBytes: 0},
|
||||
{name: "just below floor threshold", freeBytes: 699050665},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := DefaultCacheMaxBytes(tc.freeBytes, 0)
|
||||
if got != DefaultCacheMaxBytesFloor {
|
||||
t.Errorf("DefaultCacheMaxBytes = %d, want floor %d",
|
||||
got, DefaultCacheMaxBytesFloor)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheMaxBytesExplicitIsRecorded verifies that an omitted
|
||||
// cache_max_bytes is recorded as not explicit, so pixad works out the
|
||||
// default when it builds the cache, and that an explicit zero is recorded
|
||||
// as explicit, so it disables the disk cache instead.
|
||||
// cache_max_bytes is recorded as not explicit, so the cache works out
|
||||
// the default when it opens, and that an explicit zero is recorded as
|
||||
// explicit, so it disables the disk cache instead.
|
||||
func TestCacheMaxBytesExplicitIsRecorded(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
+593
-379
File diff suppressed because it is too large
Load Diff
@@ -1,12 +1,97 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.eeqj.de/sneak/smartconfig"
|
||||
)
|
||||
|
||||
// writeTestConfig writes yamlContent to a temp config file and returns
|
||||
// the file path.
|
||||
func writeTestConfig(t *testing.T, yamlContent string) string {
|
||||
t.Helper()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
configPath := filepath.Join(tmpDir, "config.yml")
|
||||
|
||||
err := os.WriteFile(configPath, []byte(yamlContent), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to write test config: %v", err)
|
||||
}
|
||||
|
||||
return configPath
|
||||
}
|
||||
|
||||
// checkAllowlistHosts loads the config at configPath and asserts that
|
||||
// getStringSlice returns the three expected hosts.
|
||||
func checkAllowlistHosts(t *testing.T, configPath string) {
|
||||
t.Helper()
|
||||
|
||||
sc, err := loadTestConfig(configPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to load config: %v", err)
|
||||
}
|
||||
|
||||
hosts := getStringSlice(sc)
|
||||
|
||||
if len(hosts) != 3 {
|
||||
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts)
|
||||
}
|
||||
|
||||
expected := []string{"static.sneak.cloud", "sneak.berlin", testHostS3}
|
||||
for i, want := range expected {
|
||||
if i >= len(hosts) {
|
||||
t.Errorf("missing host at index %d: want %q", i, want)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if hosts[i] != want {
|
||||
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetStringSlice_YAMLList(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
yamlContent := `
|
||||
allowlist_hosts:
|
||||
- static.sneak.cloud
|
||||
- sneak.berlin
|
||||
- s3.sneak.cloud
|
||||
`
|
||||
|
||||
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
|
||||
}
|
||||
|
||||
func TestGetStringSlice_CommaSeparated(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Backwards compatibility with comma-separated string values.
|
||||
yamlContent := `allowlist_hosts: "static.sneak.cloud, sneak.berlin, s3.sneak.cloud"`
|
||||
|
||||
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
|
||||
}
|
||||
|
||||
func TestGetStringSlice_Empty(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
configPath := writeTestConfig(t, `port: 8080`)
|
||||
|
||||
sc, err := loadTestConfig(configPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to load config: %v", err)
|
||||
}
|
||||
|
||||
hosts := getStringSlice(sc)
|
||||
if len(hosts) != 0 {
|
||||
t.Errorf("expected nil or empty slice, got %v", hosts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestExampleConfigAllowlistsNoHost loads configs/config.example.yml
|
||||
// with a real signing key, as Getting Started in README.md has it copied
|
||||
// and edited, and checks that it allowlists no host: every image on an
|
||||
|
||||
@@ -15,36 +15,8 @@ import (
|
||||
_ "modernc.org/sqlite" // SQLite driver registration
|
||||
)
|
||||
|
||||
// The config keys the tests name in the errors they expect.
|
||||
const (
|
||||
keyDebug = "debug"
|
||||
keyMaintenanceMode = "maintenance_mode"
|
||||
keyPort = "port"
|
||||
keyStateDir = "state_dir"
|
||||
keySentryDSN = "sentry_dsn"
|
||||
keyDBURL = "db_url"
|
||||
keyMetrics = "metrics"
|
||||
keyMetricsUsername = "metrics.username"
|
||||
keyMetricsPassword = "metrics.password"
|
||||
keySigningKey = "signing_key"
|
||||
keyAllowlistHosts = "allowlist_hosts"
|
||||
keyRefererBlocklist = "referer_blocklist"
|
||||
keyAllowHTTP = "allow_http"
|
||||
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
|
||||
keyUpstreamConnections = "upstream_connections"
|
||||
keyMaxConcurrentProcessing = "max_concurrent_processing"
|
||||
keyCacheMaxBytes = "cache_max_bytes"
|
||||
keyBlockedNetworks = "blocked_networks"
|
||||
keyTrustedProxies = "trusted_proxies"
|
||||
keyAccessControlAllowOrigin = "access_control_allow_origin"
|
||||
keyUpstreamFetchTimeout = "upstream_fetch_timeout"
|
||||
keyUpstreamMaxResponseSize = "upstream_max_response_size"
|
||||
keyDownstreamTimeout = "downstream_timeout"
|
||||
keyImageRequestsPerMinute = "image_requests_per_minute"
|
||||
)
|
||||
|
||||
// validTestSigningKey is a 32-character signing key that satisfies the
|
||||
// minimum length requirement in validateSigningKey.
|
||||
// minimum length requirement in validate().
|
||||
const validTestSigningKey = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
// signingKeyLine is a valid signing_key config line used as the base of
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestReadmeNamesEverySetting checks that the list of settings in
|
||||
// README.md names exactly the keys of the settings table, one item each.
|
||||
// The list is the first one in the Configuration section, and it ends at
|
||||
// the first blank line. Each item starts with its key in backticks.
|
||||
func TestReadmeNamesEverySetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
readme, err := os.ReadFile(filepath.Join("..", "..", "README.md"))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read README.md: %v", err)
|
||||
}
|
||||
|
||||
_, section, found := strings.Cut(string(readme), "\n### Configuration\n")
|
||||
if !found {
|
||||
t.Fatal("README.md has no Configuration section")
|
||||
}
|
||||
|
||||
// The section ends at the next heading of its level or above.
|
||||
section, _, _ = strings.Cut(section, "\n## ")
|
||||
section, _, _ = strings.Cut(section, "\n### ")
|
||||
|
||||
_, list, found := strings.Cut(section, "\n- ")
|
||||
if !found {
|
||||
t.Fatal("README.md's Configuration section has no list")
|
||||
}
|
||||
|
||||
list, _, _ = strings.Cut("- "+list, "\n\n")
|
||||
|
||||
itemKey := regexp.MustCompile("^- `([^`]+)`")
|
||||
named := map[string]bool{}
|
||||
|
||||
for line := range strings.SplitSeq(list, "\n") {
|
||||
// A line that does not start with "- " goes on with the item above.
|
||||
if !strings.HasPrefix(line, "- ") {
|
||||
continue
|
||||
}
|
||||
|
||||
match := itemKey.FindStringSubmatch(line)
|
||||
if match == nil {
|
||||
t.Errorf("README.md's list of settings has an item "+
|
||||
"that does not start with a key in backticks: %s", line)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if named[match[1]] {
|
||||
t.Errorf("README.md's list of settings names %s twice", match[1])
|
||||
}
|
||||
|
||||
named[match[1]] = true
|
||||
}
|
||||
|
||||
for _, s := range new(Config).settings() {
|
||||
if !named[s.key] {
|
||||
t.Errorf("README.md's list of settings does not name %s", s.key)
|
||||
}
|
||||
|
||||
delete(named, s.key)
|
||||
}
|
||||
|
||||
for key := range named {
|
||||
t.Errorf("README.md's list of settings names %s, "+
|
||||
"which is not a setting", key)
|
||||
}
|
||||
}
|
||||
|
||||
// TestExampleConfigNamesEverySetting checks that
|
||||
// configs/config.example.yml names exactly the keys of the settings
|
||||
// table, each one set or commented out with "# ". A key line holds the
|
||||
// key, a colon and at most a one-word value, which tells it from a
|
||||
// comment line that starts with a word and a colon. An indented key is
|
||||
// inside the map of the key above it, as username is inside metrics.
|
||||
func TestExampleConfigNamesEverySetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const examplePath = "configs/config.example.yml"
|
||||
|
||||
example, err := os.ReadFile(filepath.Join("..", "..", examplePath))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read %s: %v", examplePath, err)
|
||||
}
|
||||
|
||||
keyLine := regexp.MustCompile(`(?m)^(?:# )?( *)([a-z_]+):(?: \S+)?$`)
|
||||
named := map[string]bool{}
|
||||
|
||||
var mapKey string
|
||||
|
||||
for _, match := range keyLine.FindAllStringSubmatch(string(example), -1) {
|
||||
indent, key := match[1], match[2]
|
||||
if indent == "" {
|
||||
mapKey = key
|
||||
named[key] = true
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
// The key holding the map is not itself a setting.
|
||||
delete(named, mapKey)
|
||||
|
||||
named[mapKey+"."+key] = true
|
||||
}
|
||||
|
||||
// The env section sets environment variables; it is not a setting.
|
||||
delete(named, "env")
|
||||
|
||||
for _, s := range new(Config).settings() {
|
||||
if !named[s.key] {
|
||||
t.Errorf("%s does not name %s", examplePath, s.key)
|
||||
}
|
||||
|
||||
delete(named, s.key)
|
||||
}
|
||||
|
||||
for key := range named {
|
||||
t.Errorf("%s names %s, which is not a setting", examplePath, key)
|
||||
}
|
||||
}
|
||||
@@ -138,8 +138,7 @@ func TestPixaPortAbortsStartupPointingToPort(t *testing.T) {
|
||||
|
||||
// TestSettingVariablesAndConfigPathAreAccepted checks that every
|
||||
// setting's variable and PIXA_CONFIG_PATH pass the check for unknown
|
||||
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the
|
||||
// settings table.
|
||||
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the list.
|
||||
func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
|
||||
// A config file's env section loaded by another test can leave a
|
||||
// PIXA_ variable set for the whole process, so every one is unset
|
||||
@@ -160,8 +159,8 @@ func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
|
||||
|
||||
t.Setenv("PIXA_CONFIG_PATH", "/etc/pixa/config.yml")
|
||||
|
||||
for _, s := range new(Config).settings() {
|
||||
t.Setenv(s.envVar, "")
|
||||
for _, name := range envVarNames() {
|
||||
t.Setenv(name, "")
|
||||
}
|
||||
|
||||
err := validateKnownEnvVars()
|
||||
|
||||
@@ -104,31 +104,41 @@ func writeConcurrently(t *testing.T, db *sql.DB) {
|
||||
|
||||
wg.Wait()
|
||||
|
||||
var sources int
|
||||
var hits, sources int
|
||||
|
||||
err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM source_content`).
|
||||
Scan(&sources)
|
||||
err := db.QueryRowContext(ctx, `
|
||||
SELECT hit_count, (SELECT COUNT(*) FROM source_content)
|
||||
FROM cache_stats WHERE id = 1
|
||||
`).Scan(&hits, &sources)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to count the writes: %v", err)
|
||||
}
|
||||
|
||||
if sources != totalRequests {
|
||||
t.Errorf("%d source_content rows, want %d", sources, totalRequests)
|
||||
if hits != totalRequests || sources != totalRequests {
|
||||
t.Errorf("hit_count = %d and %d source_content rows, want %d of each",
|
||||
hits, sources, totalRequests)
|
||||
}
|
||||
}
|
||||
|
||||
// writeLikeOneRequest makes the writes one request and the eviction pass
|
||||
// make: it stores a source, records a transformed image and deletes that
|
||||
// record again.
|
||||
// make: it counts a cache hit, stores a source, records a transformed image
|
||||
// and deletes that record again.
|
||||
func writeLikeOneRequest(ctx context.Context, db *sql.DB, key string) error {
|
||||
_, err := db.ExecContext(ctx, `INSERT INTO source_content
|
||||
(source_key, size_bytes) VALUES (?, 1)`, key)
|
||||
_, err := db.ExecContext(ctx,
|
||||
`UPDATE cache_stats SET hit_count = hit_count + 1 WHERE id = 1`)
|
||||
if err != nil {
|
||||
return fmt.Errorf("counting a cache hit: %w", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(ctx, `INSERT INTO source_content
|
||||
(source_key, source_host, source_path, content_type, size_bytes)
|
||||
VALUES (?, 'example.com', '/image.png', 'image/png', 1)`, key)
|
||||
if err != nil {
|
||||
return fmt.Errorf("storing a source: %w", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(ctx, `INSERT INTO variant_content
|
||||
(cache_key, size_bytes) VALUES (?, 1)`, key)
|
||||
(cache_key, size_bytes, content_type) VALUES (?, 1, 'image/png')`, key)
|
||||
if err != nil {
|
||||
return fmt.Errorf("recording a transformed image: %w", err)
|
||||
}
|
||||
|
||||
@@ -82,11 +82,7 @@ func ParseMigrationVersion(filename string) (int, error) {
|
||||
return version, nil
|
||||
}
|
||||
|
||||
// New opens the database and applies the migrations, so that the parts
|
||||
// fx builds after it can read the database while they are built. Its
|
||||
// stop hook closes the database. fx runs stop hooks in the reverse of the
|
||||
// order it built their parts in, so the stop hooks of those parts run
|
||||
// before this one.
|
||||
// New creates a new Database instance.
|
||||
func New(lc fx.Lifecycle, params Params) (*Database, error) {
|
||||
s := &Database{
|
||||
log: params.Logger.Get(),
|
||||
@@ -95,16 +91,20 @@ func New(lc fx.Lifecycle, params Params) (*Database, error) {
|
||||
|
||||
s.log.Info("Database instantiated")
|
||||
|
||||
err := s.connect(context.Background())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
lc.Append(fx.Hook{
|
||||
OnStart: func(ctx context.Context) error {
|
||||
s.log.Info("Database OnStart Hook")
|
||||
|
||||
return s.connect(ctx)
|
||||
},
|
||||
OnStop: func(_ context.Context) error {
|
||||
s.log.Info("Database OnStop Hook")
|
||||
|
||||
return s.db.Close()
|
||||
if s.db != nil {
|
||||
return s.db.Close()
|
||||
}
|
||||
|
||||
return nil
|
||||
},
|
||||
})
|
||||
|
||||
|
||||
@@ -159,7 +159,10 @@ func TestApplyMigrations_CreatesSchemaAndTables(t *testing.T) {
|
||||
}
|
||||
|
||||
// Verify that the application tables created by 001.sql exist.
|
||||
tables := []string{"source_content", "negative_cache"}
|
||||
tables := []string{
|
||||
"source_content", "output_content",
|
||||
"request_cache", "negative_cache", "cache_stats",
|
||||
}
|
||||
for _, table := range tables {
|
||||
var count int
|
||||
|
||||
|
||||
@@ -6,22 +6,27 @@
|
||||
-- is the SHA-256 of the source's host, path and query
|
||||
CREATE TABLE IF NOT EXISTS source_content (
|
||||
source_key TEXT PRIMARY KEY,
|
||||
source_host TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_query TEXT NOT NULL DEFAULT '',
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
|
||||
ON source_content(last_accessed_at);
|
||||
|
||||
-- Processed variant blobs
|
||||
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key>. Tracked here
|
||||
-- (like source content blobs above) so total cache usage is known
|
||||
-- without a directory scan, and so LRU eviction has a timestamp to order
|
||||
-- on. A cache hit does not read this table: it takes the content type
|
||||
-- from the file's first bytes, and its last_accessed_at is written
|
||||
-- later, in a batch.
|
||||
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key> (plus a .meta
|
||||
-- sidecar with the content type). Tracked here (like source content
|
||||
-- blobs above) so total cache usage is known without a directory scan,
|
||||
-- and so LRU eviction has a timestamp to order on.
|
||||
CREATE TABLE IF NOT EXISTS variant_content (
|
||||
cache_key TEXT PRIMARY KEY,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed
|
||||
@@ -95,19 +100,63 @@ BEGIN
|
||||
WHERE id = 1;
|
||||
END;
|
||||
|
||||
-- Negative cache: source URLs whose fetch failed with an upstream error
|
||||
-- status, or whose first bytes did not match their Content-Type. pixa does
|
||||
-- not fetch such a URL again until its row expires.
|
||||
-- Output/transformed content blobs
|
||||
-- Not written: transformed images are stored in cache/variants and
|
||||
-- tracked in variant_content above.
|
||||
CREATE TABLE IF NOT EXISTS output_content (
|
||||
content_hash TEXT PRIMARY KEY,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Request cache - maps full request params to output content
|
||||
CREATE TABLE IF NOT EXISTS request_cache (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
cache_key TEXT NOT NULL UNIQUE,
|
||||
source_metadata_id INTEGER NOT NULL,
|
||||
output_hash TEXT NOT NULL,
|
||||
width INTEGER NOT NULL,
|
||||
height INTEGER NOT NULL,
|
||||
format TEXT NOT NULL,
|
||||
quality INTEGER NOT NULL DEFAULT 85,
|
||||
fit_mode TEXT NOT NULL DEFAULT 'cover',
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
access_count INTEGER NOT NULL DEFAULT 1,
|
||||
FOREIGN KEY (source_metadata_id) REFERENCES source_metadata(id),
|
||||
FOREIGN KEY (output_hash) REFERENCES output_content(content_hash)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_request_cache_key ON request_cache(cache_key);
|
||||
CREATE INDEX IF NOT EXISTS idx_request_cache_source ON request_cache(source_metadata_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_request_cache_output ON request_cache(output_hash);
|
||||
CREATE INDEX IF NOT EXISTS idx_request_cache_fetched ON request_cache(fetched_at);
|
||||
|
||||
-- Negative cache for failed fetches (404s, timeouts, etc.)
|
||||
CREATE TABLE IF NOT EXISTS negative_cache (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_host TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_query TEXT NOT NULL DEFAULT '',
|
||||
status_code INTEGER NOT NULL,
|
||||
error_message TEXT,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at DATETIME NOT NULL,
|
||||
UNIQUE(source_host, source_path, source_query)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_negative_cache_expires ON negative_cache(expires_at);
|
||||
|
||||
-- Cache statistics for monitoring
|
||||
CREATE TABLE IF NOT EXISTS cache_stats (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
hit_count INTEGER NOT NULL DEFAULT 0,
|
||||
miss_count INTEGER NOT NULL DEFAULT 0,
|
||||
upstream_fetch_count INTEGER NOT NULL DEFAULT 0,
|
||||
upstream_fetch_bytes INTEGER NOT NULL DEFAULT 0,
|
||||
transform_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
INSERT OR IGNORE INTO cache_stats (id) VALUES (1);
|
||||
|
||||
-- A random value that a login session carries from its login. pixa makes it
|
||||
-- when the table has no row, and each logout replaces it; a login session
|
||||
-- carrying another value is refused, so logging out ends every login session.
|
||||
|
||||
+28
-21
@@ -16,6 +16,9 @@ const (
|
||||
DefaultQuality = 85
|
||||
DefaultFormat = imagerequest.FormatJXL
|
||||
DefaultFitMode = imagerequest.FitCover
|
||||
|
||||
// HKDF salt for URL encryption key derivation
|
||||
urlKeySalt = "pixa-urlenc-v1"
|
||||
)
|
||||
|
||||
// Errors returned by encurl operations.
|
||||
@@ -45,20 +48,23 @@ type Generator struct {
|
||||
key [seal.KeySize]byte
|
||||
}
|
||||
|
||||
// NewGenerator creates an encrypted URL generator that encrypts its tokens
|
||||
// with key.
|
||||
func NewGenerator(key [seal.KeySize]byte) *Generator {
|
||||
return &Generator{key: key}
|
||||
// NewGenerator creates an encrypted URL generator with a key derived
|
||||
// from the signing key.
|
||||
func NewGenerator(signingKey string) (*Generator, error) {
|
||||
key, err := seal.DeriveKey([]byte(signingKey), urlKeySalt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &Generator{key: key}, nil
|
||||
}
|
||||
|
||||
// Generate creates an encrypted URL token from the request's source and
|
||||
// transform, each as the request holds it, and from expires; an empty format,
|
||||
// a quality of 0 or an empty fit mode is left out of the token, and a zero
|
||||
// expires makes a token that never expires. The token is CBOR-encoded,
|
||||
// Generate creates an encrypted URL token from the request's source,
|
||||
// transform and expiry, each as the request holds it; an empty format, a
|
||||
// quality of 0 or an empty fit mode is left out of the token, and a zero
|
||||
// Expires makes a token that never expires. The token is CBOR-encoded,
|
||||
// encrypted with NaCl secretbox, and base64url-encoded.
|
||||
func (g *Generator) Generate(
|
||||
req *imagerequest.Request, expires time.Time,
|
||||
) (string, error) {
|
||||
func (g *Generator) Generate(req *imagerequest.Request) (string, error) {
|
||||
p := &payload{
|
||||
SourceHost: req.Source.Host,
|
||||
SourcePath: req.Source.Path,
|
||||
@@ -70,8 +76,8 @@ func (g *Generator) Generate(
|
||||
FitMode: req.Transform.FitMode,
|
||||
}
|
||||
|
||||
if !expires.IsZero() {
|
||||
p.ExpiresAt = expires.Unix()
|
||||
if !req.Expires.IsZero() {
|
||||
p.ExpiresAt = req.Expires.Unix()
|
||||
}
|
||||
|
||||
// CBOR encode the payload
|
||||
@@ -85,19 +91,19 @@ func (g *Generator) Generate(
|
||||
}
|
||||
|
||||
// Parse decrypts and validates an encrypted URL token and returns the
|
||||
// request it holds, with default values for omitted optional fields, and its
|
||||
// expiry. A token that never expires gives the zero time.
|
||||
// request it holds, with default values for omitted optional fields. A token
|
||||
// that never expires gives the zero Expires.
|
||||
// Returns ErrExpired if the token has expired, or other errors for invalid tokens.
|
||||
func (g *Generator) Parse(token string) (*imagerequest.Request, time.Time, error) {
|
||||
func (g *Generator) Parse(token string) (*imagerequest.Request, error) {
|
||||
// Decrypt
|
||||
data, err := seal.Decrypt(g.key, token)
|
||||
if err != nil {
|
||||
if errors.Is(err, seal.ErrDecryptionFailed) ||
|
||||
errors.Is(err, seal.ErrInvalidPayload) {
|
||||
return nil, time.Time{}, ErrDecryptFailed
|
||||
return nil, ErrDecryptFailed
|
||||
}
|
||||
|
||||
return nil, time.Time{}, err
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// CBOR decode
|
||||
@@ -105,12 +111,12 @@ func (g *Generator) Parse(token string) (*imagerequest.Request, time.Time, error
|
||||
|
||||
err = cbor.Unmarshal(data, &p)
|
||||
if err != nil {
|
||||
return nil, time.Time{}, ErrInvalidFormat
|
||||
return nil, ErrInvalidFormat
|
||||
}
|
||||
|
||||
// Check expiration (0 = never expires)
|
||||
if p.ExpiresAt != 0 && time.Now().Unix() > p.ExpiresAt {
|
||||
return nil, time.Time{}, ErrExpired
|
||||
return nil, ErrExpired
|
||||
}
|
||||
|
||||
format := p.Format
|
||||
@@ -145,5 +151,6 @@ func (g *Generator) Parse(token string) (*imagerequest.Request, time.Time, error
|
||||
Quality: quality,
|
||||
FitMode: fitMode,
|
||||
},
|
||||
}, expires, nil
|
||||
Expires: expires,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
)
|
||||
|
||||
// Shared test fixture strings.
|
||||
@@ -20,13 +19,11 @@ const (
|
||||
func TestGenerator_GenerateAndParse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
keys, err := seal.DeriveKeys("test-signing-key-12345")
|
||||
gen, err := encurl.NewGenerator("test-signing-key-12345")
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
gen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
@@ -39,10 +36,10 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
|
||||
Quality: 90,
|
||||
FitMode: imagerequest.FitContain,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
expires := time.Now().Add(time.Hour)
|
||||
|
||||
token, err := gen.Generate(req, expires)
|
||||
token, err := gen.Generate(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
@@ -51,7 +48,7 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
|
||||
t.Fatal("Generate() returned empty token")
|
||||
}
|
||||
|
||||
parsed, parsedExpires, err := gen.Parse(token)
|
||||
parsed, err := gen.Parse(token)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse() error = %v", err)
|
||||
}
|
||||
@@ -94,30 +91,30 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
|
||||
parsed.Transform.FitMode, req.Transform.FitMode)
|
||||
}
|
||||
|
||||
if parsedExpires.Unix() != expires.Unix() {
|
||||
t.Errorf("ExpiresAt = %d, want %d", parsedExpires.Unix(), expires.Unix())
|
||||
if parsed.Expires.Unix() != req.Expires.Unix() {
|
||||
t.Errorf("ExpiresAt = %d, want %d", parsed.Expires.Unix(), req.Expires.Unix())
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerator_Parse_Expired(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
keys, _ := seal.DeriveKeys("test-signing-key-12345")
|
||||
gen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(-time.Hour), // Already expired
|
||||
}
|
||||
|
||||
token, err := gen.Generate(req, time.Now().Add(-time.Hour)) // Already expired
|
||||
token, err := gen.Generate(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
_, _, err = gen.Parse(token)
|
||||
_, err = gen.Parse(token)
|
||||
if err == nil {
|
||||
t.Error("Parse() should fail for expired token")
|
||||
}
|
||||
@@ -130,10 +127,9 @@ func TestGenerator_Parse_Expired(t *testing.T) {
|
||||
func TestGenerator_Parse_InvalidToken(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
keys, _ := seal.DeriveKeys("test-signing-key-12345")
|
||||
gen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
_, _, err := gen.Parse("not-a-valid-token")
|
||||
_, err := gen.Parse("not-a-valid-token")
|
||||
if err == nil {
|
||||
t.Error("Parse() should fail for invalid token")
|
||||
}
|
||||
@@ -142,17 +138,17 @@ func TestGenerator_Parse_InvalidToken(t *testing.T) {
|
||||
func TestGenerator_Parse_TamperedToken(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
keys, _ := seal.DeriveKeys("test-signing-key-12345")
|
||||
gen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
|
||||
token, _ := gen.Generate(req, time.Now().Add(time.Hour))
|
||||
token, _ := gen.Generate(req)
|
||||
|
||||
// Tamper with the token
|
||||
tampered := []byte(token)
|
||||
@@ -160,7 +156,7 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
|
||||
tampered[10] ^= 0x01
|
||||
}
|
||||
|
||||
_, _, err := gen.Parse(string(tampered))
|
||||
_, err := gen.Parse(string(tampered))
|
||||
if err == nil {
|
||||
t.Error("Parse() should fail for tampered token")
|
||||
}
|
||||
@@ -169,21 +165,20 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
|
||||
func TestGenerator_Parse_WrongKey(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
keys1, _ := seal.DeriveKeys("signing-key-1")
|
||||
keys2, _ := seal.DeriveKeys("signing-key-2")
|
||||
gen1 := encurl.NewGenerator(keys1.EncryptedURLKey)
|
||||
gen2 := encurl.NewGenerator(keys2.EncryptedURLKey)
|
||||
gen1, _ := encurl.NewGenerator("signing-key-1")
|
||||
gen2, _ := encurl.NewGenerator("signing-key-2")
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
|
||||
token, _ := gen1.Generate(req, time.Now().Add(time.Hour))
|
||||
token, _ := gen1.Generate(req)
|
||||
|
||||
_, _, err := gen2.Parse(token)
|
||||
_, err := gen2.Parse(token)
|
||||
if err == nil {
|
||||
t.Error("Parse() should fail with different signing key")
|
||||
}
|
||||
@@ -192,8 +187,7 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
|
||||
func TestGenerator_Parse_Defaults(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
keys, _ := seal.DeriveKeys("test-signing-key-12345")
|
||||
gen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
// Request with only required fields - should get defaults
|
||||
token, err := gen.Generate(&imagerequest.Request{
|
||||
@@ -201,12 +195,13 @@ func TestGenerator_Parse_Defaults(t *testing.T) {
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
}, time.Now().Add(time.Hour))
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
req, _, err := gen.Parse(token)
|
||||
req, err := gen.Parse(token)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse() error = %v", err)
|
||||
}
|
||||
@@ -230,17 +225,17 @@ func TestGenerator_Parse_Defaults(t *testing.T) {
|
||||
func TestGenerator_TokenIsURLSafe(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
keys, _ := seal.DeriveKeys("test-signing-key-12345")
|
||||
gen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
|
||||
token, _ := gen.Generate(req, time.Now().Add(time.Hour))
|
||||
token, _ := gen.Generate(req)
|
||||
|
||||
// Check that token only contains URL-safe characters
|
||||
for _, c := range token {
|
||||
|
||||
+17
-14
@@ -153,20 +153,20 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
// The image URL must fetch the source URL as it was written. The
|
||||
// fetcher fetches over http only when allow_http is set.
|
||||
fetchURL := req.Source.URL(s.config.AllowHTTP)
|
||||
// The image URL must fetch the source URL as it was written
|
||||
req.AllowHTTP = s.config.AllowHTTP
|
||||
fetchURL := req.SourceURL()
|
||||
|
||||
if !fetchedAsWritten(parsed, fetchURL) {
|
||||
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
|
||||
"written: pixa would fetch "+fetchURL.String(),
|
||||
"written: pixa would fetch "+fetchURL,
|
||||
r.Form, http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Generate encrypted token
|
||||
token, err := s.encGen.Generate(req, expiresAt)
|
||||
token, err := s.encGen.Generate(req)
|
||||
if err != nil {
|
||||
s.log.Error("failed to generate encrypted URL", "error", err)
|
||||
s.renderGeneratorWithForm(w, r, "Failed to generate URL", r.Form,
|
||||
@@ -226,23 +226,25 @@ func parseSourceURL(sourceURL string) (*url.URL, error) {
|
||||
// in RawPath when it is not the decoded path percent-encoded again.
|
||||
// EscapedPath ignores RawPath when it also holds a character net/url encodes,
|
||||
// such as a space, so RawPath is compared itself.
|
||||
func fetchedAsWritten(source, fetchURL *url.URL) bool {
|
||||
func fetchedAsWritten(source *url.URL, fetchURL string) bool {
|
||||
writtenPath := source.RawPath
|
||||
if writtenPath == "" {
|
||||
writtenPath = source.EscapedPath()
|
||||
}
|
||||
|
||||
return source.User == nil && !source.ForceQuery &&
|
||||
fetchURL.EscapedPath() == writtenPath &&
|
||||
fetchURL.RawQuery == source.RawQuery
|
||||
fetched, err := url.Parse(fetchURL)
|
||||
|
||||
return err == nil && source.User == nil && !source.ForceQuery &&
|
||||
fetched.EscapedPath() == writtenPath &&
|
||||
fetched.RawQuery == source.RawQuery
|
||||
}
|
||||
|
||||
// buildGenerateRequest parses the numeric form fields and assembles the
|
||||
// image request the encrypted URL's token is made from, and its expiry.
|
||||
// ttl=0 means never expires (the expiry is the zero time). A non-numeric or
|
||||
// out-of-range width, height, quality or ttl, or an unrecognized fit mode, is
|
||||
// a client error naming the offending field. The format field is passed
|
||||
// through unchecked.
|
||||
// image request the encrypted URL's token is made from. ttl=0 means never
|
||||
// expires (Expires stays the zero time). A non-numeric or out-of-range
|
||||
// width, height, quality or ttl, or an unrecognized fit mode, is a client
|
||||
// error naming the offending field. The format field is passed through
|
||||
// unchecked.
|
||||
func buildGenerateRequest(
|
||||
parsed *url.URL, form url.Values,
|
||||
) (*imagerequest.Request, time.Time, int, error) {
|
||||
@@ -292,6 +294,7 @@ func buildGenerateRequest(
|
||||
Quality: quality,
|
||||
FitMode: fitMode,
|
||||
},
|
||||
Expires: expiresAt,
|
||||
}
|
||||
|
||||
return req, expiresAt, ttl, nil
|
||||
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
)
|
||||
|
||||
@@ -45,17 +44,27 @@ func newCSRFTestRouter(t *testing.T) (*Handlers, http.Handler) {
|
||||
|
||||
cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
sessMgr, err := session.NewManager(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("session.NewManager() error = %v", err)
|
||||
}
|
||||
|
||||
encGen, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
|
||||
if err != nil {
|
||||
t.Fatalf("newCSRFProtect() error = %v", err)
|
||||
}
|
||||
|
||||
h := &Handlers{
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
config: cfg,
|
||||
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
|
||||
encGen: encurl.NewGenerator(keys.EncryptedURLKey),
|
||||
csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug),
|
||||
sessMgr: sessMgr,
|
||||
encGen: encGen,
|
||||
csrfProtect: protect,
|
||||
}
|
||||
|
||||
r := chi.NewRouter()
|
||||
|
||||
@@ -10,7 +10,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
)
|
||||
|
||||
@@ -23,15 +22,15 @@ func TestLoginLogLeavesOutSubmittedKey(t *testing.T) {
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
sessMgr, err := session.NewManager(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("session.NewManager() error = %v", err)
|
||||
}
|
||||
|
||||
h := &Handlers{
|
||||
log: slog.New(slog.NewJSONHandler(&buf, nil)),
|
||||
config: &config.Config{SigningKey: testSigningKey},
|
||||
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
|
||||
sessMgr: sessMgr,
|
||||
}
|
||||
|
||||
submittedKeys := []string{wrongKey, testSigningKey}
|
||||
|
||||
@@ -11,7 +11,6 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
)
|
||||
|
||||
@@ -233,13 +232,8 @@ func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
|
||||
|
||||
db := setupTestDB(t)
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
}
|
||||
|
||||
sessMgr, err := session.NewManagerWithDatabase(
|
||||
t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
|
||||
t.Context(), testSigningKey, db)
|
||||
if err != nil {
|
||||
t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -12,11 +12,11 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError, which
|
||||
// answers the errors of both image routes, answers 400 naming the AVIF pixel
|
||||
// limit when the image processor refuses an AVIF output above it, wrapped as
|
||||
// the service wraps it. The request's source only goes into a log line, so it
|
||||
// is left empty.
|
||||
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
|
||||
// handleImageError, which answer the errors of the two image routes, answer
|
||||
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
|
||||
// output above it, wrapped as the service wraps it. The request's source only
|
||||
// goes into a log line, so it is left empty.
|
||||
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -25,15 +25,25 @@ func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
|
||||
imageprocessor.ErrAVIFOutputTooLarge)
|
||||
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.respondImageError(rec, &imagerequest.Request{}, err)
|
||||
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, &imagerequest.Request{}, err)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
if !strings.Contains(rec.Body.String(), wantBody) {
|
||||
t.Errorf("body = %q, want it to say %q", rec.Body.String(), wantBody)
|
||||
for name, rec := range map[string]*httptest.ResponseRecorder{
|
||||
"respondImageError": plain, "handleImageError": encrypted,
|
||||
} {
|
||||
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status = %d, want %d",
|
||||
name, rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), wantBody) {
|
||||
t.Errorf("%s: body = %q, want it to say %q",
|
||||
name, rec.Body.String(), wantBody)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,7 +11,6 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
)
|
||||
|
||||
@@ -33,17 +32,27 @@ func newBodyLimitTestRouter(
|
||||
|
||||
cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
sessMgr, err := session.NewManager(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("session.NewManager() error = %v", err)
|
||||
}
|
||||
|
||||
encGen, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
|
||||
if err != nil {
|
||||
t.Fatalf("newCSRFProtect() error = %v", err)
|
||||
}
|
||||
|
||||
h := &Handlers{
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
config: cfg,
|
||||
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
|
||||
encGen: encurl.NewGenerator(keys.EncryptedURLKey),
|
||||
csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug),
|
||||
sessMgr: sessMgr,
|
||||
encGen: encGen,
|
||||
csrfProtect: protect,
|
||||
}
|
||||
|
||||
r := chi.NewRouter()
|
||||
|
||||
@@ -1,278 +0,0 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"image"
|
||||
"image/jpeg"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// countingHost is the allowlisted host whose photoPath the server of
|
||||
// newCountingDatabaseServer serves.
|
||||
const countingHost = "counting.example.com"
|
||||
|
||||
// countingConnector opens connections to the SQLite database file name with
|
||||
// sqliteDriver, each one counting in statements the statements run on it.
|
||||
type countingConnector struct {
|
||||
sqliteDriver driver.Driver
|
||||
name string
|
||||
statements *atomic.Int64
|
||||
}
|
||||
|
||||
func (c countingConnector) Connect(context.Context) (driver.Conn, error) {
|
||||
conn, err := c.sqliteDriver.Open(c.name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return countingConn{Conn: conn, statements: c.statements}, nil
|
||||
}
|
||||
|
||||
func (c countingConnector) Driver() driver.Driver {
|
||||
return c.sqliteDriver
|
||||
}
|
||||
|
||||
// countingConn counts the statements run on its connection. It has only the
|
||||
// methods of driver.Conn, so database/sql prepares every query and every
|
||||
// other statement with Prepare, which counts it.
|
||||
type countingConn struct {
|
||||
driver.Conn
|
||||
|
||||
statements *atomic.Int64
|
||||
}
|
||||
|
||||
func (c countingConn) Prepare(query string) (driver.Stmt, error) {
|
||||
c.statements.Add(1)
|
||||
|
||||
return c.Conn.Prepare(query)
|
||||
}
|
||||
|
||||
// newCountingDatabase returns a database with pixa's schema, in a file of
|
||||
// its own and with one connection, and the count of the statements run on
|
||||
// it once the schema is in place.
|
||||
func newCountingDatabase(tb testing.TB) (*sql.DB, *atomic.Int64) {
|
||||
tb.Helper()
|
||||
|
||||
path := filepath.Join(tb.TempDir(), "state.sqlite3")
|
||||
|
||||
schemaDB, err := sql.Open("sqlite", path)
|
||||
if err != nil {
|
||||
tb.Fatalf("opening the database: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = schemaDB.Close() }()
|
||||
|
||||
err = database.ApplyMigrations(tb.Context(), schemaDB, nil)
|
||||
if err != nil {
|
||||
tb.Fatalf("applying the migrations: %v", err)
|
||||
}
|
||||
|
||||
statements := new(atomic.Int64)
|
||||
|
||||
db := sql.OpenDB(countingConnector{
|
||||
sqliteDriver: schemaDB.Driver(),
|
||||
name: path,
|
||||
statements: statements,
|
||||
})
|
||||
db.SetMaxOpenConns(1)
|
||||
|
||||
tb.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return db, statements
|
||||
}
|
||||
|
||||
// newCountingDatabaseServer returns a router for both image routes whose
|
||||
// image cache uses newCountingDatabase, the count of the statements run on
|
||||
// that database, and a URL on each route for a different image of
|
||||
// photoPath on countingHost. Nothing writes the cache's access times in the
|
||||
// background, so every statement counted comes from a request.
|
||||
func newCountingDatabaseServer(
|
||||
tb testing.TB,
|
||||
) (http.Handler, *atomic.Int64, []string) {
|
||||
tb.Helper()
|
||||
|
||||
db, statements := newCountingDatabase(tb)
|
||||
|
||||
var photo bytes.Buffer
|
||||
|
||||
err := jpeg.Encode(&photo, image.NewRGBA(image.Rect(0, 0, 100, 100)), nil)
|
||||
if err != nil {
|
||||
tb.Fatalf("encoding the photo: %v", err)
|
||||
}
|
||||
|
||||
cache, err := imgcache.NewCache(db, imgcache.Config{
|
||||
StateDir: tb.TempDir(),
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
if err != nil {
|
||||
tb.Fatalf("imgcache.NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
svc, err := imageproxy.New(&imageproxy.Config{
|
||||
Cache: cache,
|
||||
Fetcher: newMockFetcher(fstest.MapFS{
|
||||
countingHost + photoPath: &fstest.MapFile{Data: photo.Bytes()},
|
||||
}),
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
Logger: slog.New(slog.DiscardHandler),
|
||||
}, prometheus.NewRegistry())
|
||||
if err != nil {
|
||||
tb.Fatalf("imageproxy.New() error = %v", err)
|
||||
}
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
if err != nil {
|
||||
tb.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
}
|
||||
|
||||
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
|
||||
token, err := encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: countingHost, Path: photoPath},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 40, Height: 40},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}, time.Time{})
|
||||
if err != nil {
|
||||
tb.Fatalf("encGen.Generate() error = %v", err)
|
||||
}
|
||||
|
||||
h := &Handlers{
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
imgSvc: svc,
|
||||
encGen: encGen,
|
||||
signer: signature.New(testSigningKey),
|
||||
allowlist: allowlist.New([]string{countingHost}),
|
||||
}
|
||||
|
||||
r := chi.NewRouter()
|
||||
r.Get("/v1/image/*", h.HandleImage())
|
||||
r.Get("/v1/e/{token}/*", h.HandleImageEnc())
|
||||
|
||||
return r, statements, []string{
|
||||
"/v1/image/" + countingHost + photoPath + "/50x50.jpeg",
|
||||
"/v1/e/" + token + "/img.jpg",
|
||||
}
|
||||
}
|
||||
|
||||
// serveImage sends srv a GET for target and returns the response, failing
|
||||
// the test unless it is a 200 with an image.
|
||||
func serveImage(
|
||||
tb testing.TB, srv http.Handler, target string,
|
||||
) *httptest.ResponseRecorder {
|
||||
tb.Helper()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
srv.ServeHTTP(rec,
|
||||
httptest.NewRequestWithContext(tb.Context(), http.MethodGet, target, nil))
|
||||
|
||||
if rec.Code != http.StatusOK || rec.Body.Len() == 0 {
|
||||
tb.Errorf("GET %s: %d with %d bytes, want 200 with an image",
|
||||
target, rec.Code, rec.Body.Len())
|
||||
}
|
||||
|
||||
return rec
|
||||
}
|
||||
|
||||
// TestImageRoutes_CacheHitsMakeNoDatabaseCall serves an image on each image
|
||||
// route once, which makes and caches it, then serves it again from the cache.
|
||||
// Serving those cache hits must run no statement on the database.
|
||||
func TestImageRoutes_CacheHitsMakeNoDatabaseCall(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
srv, statements, targets := newCountingDatabaseServer(t)
|
||||
|
||||
for _, target := range targets {
|
||||
before := statements.Load()
|
||||
|
||||
serveImage(t, srv, target)
|
||||
|
||||
afterMiss := statements.Load()
|
||||
t.Logf("GET %s, a miss: %d statements", target, afterMiss-before)
|
||||
|
||||
if afterMiss == before {
|
||||
t.Fatalf("GET %s, a miss: no statement counted, want some", target)
|
||||
}
|
||||
|
||||
for range 3 {
|
||||
rec := serveImage(t, srv, target)
|
||||
|
||||
cacheStatus := rec.Header().Get("X-Pixa-Cache")
|
||||
if cacheStatus != string(imageproxy.CacheHit) {
|
||||
t.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
|
||||
target, cacheStatus, imageproxy.CacheHit)
|
||||
}
|
||||
}
|
||||
|
||||
if hitStatements := statements.Load() - afterMiss; hitStatements != 0 {
|
||||
t.Errorf("GET %s: 3 cache hits ran %d statements, want 0",
|
||||
target, hitStatements)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkHandleImage_CacheHits serves one cached image through the
|
||||
// /v1/image/ route from b.RunParallel's goroutines, and reports the cache
|
||||
// hits served per second. It fails if serving them ran any statement on the
|
||||
// database. script/bench runs it.
|
||||
func BenchmarkHandleImage_CacheHits(b *testing.B) {
|
||||
srv, statements, targets := newCountingDatabaseServer(b)
|
||||
target := targets[0]
|
||||
|
||||
serveImage(b, srv, target)
|
||||
|
||||
before := statements.Load()
|
||||
|
||||
b.ResetTimer()
|
||||
|
||||
b.RunParallel(func(pb *testing.PB) {
|
||||
for pb.Next() {
|
||||
rec := serveImage(b, srv, target)
|
||||
|
||||
cacheStatus := rec.Header().Get("X-Pixa-Cache")
|
||||
if cacheStatus != string(imageproxy.CacheHit) {
|
||||
b.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
|
||||
target, cacheStatus, imageproxy.CacheHit)
|
||||
|
||||
return
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
b.ReportMetric(float64(b.N)/b.Elapsed().Seconds(), "hits/s")
|
||||
|
||||
if hitStatements := statements.Load() - before; hitStatements != 0 {
|
||||
b.Fatalf("%d cache hits ran %d statements, want 0", b.N, hitStatements)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"go.uber.org/fx/fxtest"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/globals"
|
||||
"sneak.berlin/go/pixa/internal/logger"
|
||||
)
|
||||
|
||||
// TestNewCacheConfigFromCacheMaxBytes checks the cache configuration
|
||||
// built from cache_max_bytes: omitted, the cache works out the default
|
||||
// limit; 0 turns the disk cache off; a positive value is the limit,
|
||||
// unchanged.
|
||||
func TestNewCacheConfigFromCacheMaxBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const oneGiB = 1 << 30
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
cacheMaxBytes int64
|
||||
cacheMaxBytesExplicit bool
|
||||
wantMaxBytes int64
|
||||
wantUseDefaultMaxBytes bool
|
||||
wantDisableDiskCache bool
|
||||
}{
|
||||
{
|
||||
name: "cache_max_bytes omitted",
|
||||
cacheMaxBytes: 0,
|
||||
cacheMaxBytesExplicit: false,
|
||||
wantMaxBytes: 0,
|
||||
wantUseDefaultMaxBytes: true,
|
||||
wantDisableDiskCache: false,
|
||||
},
|
||||
{
|
||||
name: "cache_max_bytes: 0",
|
||||
cacheMaxBytes: 0,
|
||||
cacheMaxBytesExplicit: true,
|
||||
wantMaxBytes: 0,
|
||||
wantUseDefaultMaxBytes: false,
|
||||
wantDisableDiskCache: true,
|
||||
},
|
||||
{
|
||||
name: "cache_max_bytes: 1 GiB",
|
||||
cacheMaxBytes: oneGiB,
|
||||
cacheMaxBytesExplicit: true,
|
||||
wantMaxBytes: oneGiB,
|
||||
wantUseDefaultMaxBytes: false,
|
||||
wantDisableDiskCache: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := &config.Config{
|
||||
CacheMaxBytes: tc.cacheMaxBytes,
|
||||
CacheMaxBytesExplicit: tc.cacheMaxBytesExplicit,
|
||||
}
|
||||
|
||||
got := newCacheConfig(cfg, nil)
|
||||
t.Logf("MaxBytes = %d, UseDefaultMaxBytes = %v, DisableDiskCache = %v",
|
||||
got.MaxBytes, got.UseDefaultMaxBytes, got.DisableDiskCache)
|
||||
|
||||
if got.MaxBytes != tc.wantMaxBytes {
|
||||
t.Errorf("MaxBytes = %d, want %d", got.MaxBytes, tc.wantMaxBytes)
|
||||
}
|
||||
|
||||
if got.UseDefaultMaxBytes != tc.wantUseDefaultMaxBytes {
|
||||
t.Errorf("UseDefaultMaxBytes = %v, want %v",
|
||||
got.UseDefaultMaxBytes, tc.wantUseDefaultMaxBytes)
|
||||
}
|
||||
|
||||
if got.DisableDiskCache != tc.wantDisableDiskCache {
|
||||
t.Errorf("DisableDiskCache = %v, want %v",
|
||||
got.DisableDiskCache, tc.wantDisableDiskCache)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDiskCacheOffOnlyForExplicitZeroCacheMaxBytes starts the handlers
|
||||
// once with cache_max_bytes omitted and once with cache_max_bytes: 0,
|
||||
// and checks by whether the cache directories were created that the
|
||||
// disk cache is on in the first case and off in the second.
|
||||
func TestDiskCacheOffOnlyForExplicitZeroCacheMaxBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
cacheMaxBytesExplicit bool
|
||||
wantDiskCache bool
|
||||
}{
|
||||
{name: "cache_max_bytes omitted", cacheMaxBytesExplicit: false, wantDiskCache: true},
|
||||
{name: "cache_max_bytes: 0", cacheMaxBytesExplicit: true, wantDiskCache: false},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
stateDir := t.TempDir()
|
||||
cfg := &config.Config{
|
||||
SigningKey: testSigningKey,
|
||||
StateDir: stateDir,
|
||||
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||
CacheMaxBytes: 0,
|
||||
CacheMaxBytesExplicit: tc.cacheMaxBytesExplicit,
|
||||
}
|
||||
|
||||
lc := fxtest.NewLifecycle(t)
|
||||
|
||||
log, err := logger.New(lc, logger.Params{Globals: &globals.Globals{}})
|
||||
if err != nil {
|
||||
t.Fatalf("logger.New() error = %v", err)
|
||||
}
|
||||
|
||||
db, err := database.New(lc, database.Params{Logger: log, Config: cfg})
|
||||
if err != nil {
|
||||
t.Fatalf("database.New() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = New(lc, Params{Logger: log, Database: db, Config: cfg})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
|
||||
lc.RequireStart()
|
||||
t.Cleanup(lc.RequireStop)
|
||||
|
||||
_, err = os.Stat(filepath.Join(stateDir, "cache", "variants"))
|
||||
gotDiskCache := err == nil
|
||||
|
||||
if gotDiskCache != tc.wantDiskCache {
|
||||
t.Errorf("cache directories created = %v, want %v",
|
||||
gotDiskCache, tc.wantDiskCache)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -9,10 +9,15 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
)
|
||||
|
||||
// csrfKeySalt provides domain separation for the CSRF authentication key,
|
||||
// derived from the signing key so tokens survive restarts without extra
|
||||
// configuration and never reuse the session or encrypted-URL key material.
|
||||
const csrfKeySalt = "pixa-csrf-v1"
|
||||
|
||||
// newCSRFProtect builds the CSRF-protection middleware for the
|
||||
// state-mutating HTML form routes. The token lives in its own cookie,
|
||||
// independent of the session cookie, so it also protects the login POST
|
||||
// where no session exists yet (login CSRF). key signs that cookie.
|
||||
// where no session exists yet (login CSRF).
|
||||
//
|
||||
// When plaintext is true (local HTTP development), requests are marked
|
||||
// plaintext so the library neither demands an https Referer nor sets a
|
||||
@@ -20,8 +25,13 @@ import (
|
||||
// service runs behind a TLS-terminating proxy, so plaintext is false and
|
||||
// the library enforces its https Referer origin check.
|
||||
func newCSRFProtect(
|
||||
key [seal.KeySize]byte, plaintext bool,
|
||||
) func(http.Handler) http.Handler {
|
||||
signingKey string, plaintext bool,
|
||||
) (func(http.Handler) http.Handler, error) {
|
||||
key, err := seal.DeriveKey([]byte(signingKey), csrfKeySalt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
protect := csrf.Protect(
|
||||
key[:],
|
||||
csrf.Path("/"),
|
||||
@@ -30,7 +40,7 @@ func newCSRFProtect(
|
||||
)
|
||||
|
||||
if !plaintext {
|
||||
return protect
|
||||
return protect, nil
|
||||
}
|
||||
|
||||
return func(next http.Handler) http.Handler {
|
||||
@@ -39,7 +49,7 @@ func newCSRFProtect(
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
protected.ServeHTTP(w, csrf.PlaintextHTTPRequest(r))
|
||||
})
|
||||
}
|
||||
}, nil
|
||||
}
|
||||
|
||||
// CSRF returns the CSRF-protection middleware for the login and generator
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxtest"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/globals"
|
||||
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||
"sneak.berlin/go/pixa/internal/logger"
|
||||
)
|
||||
|
||||
// TestHandlersBuildTheirOwnFetcherWhenNoneIsProvided builds the handlers as
|
||||
// pixad does, in an fx app that provides no fetcher, and requests an image
|
||||
// from 192.0.2.10, which is on the allowlist and in blocked_networks. The URL
|
||||
// check accepts that address; only the dialer that refuses internal
|
||||
// addresses checks blocked_networks, so the answer is 403 only if the
|
||||
// fetcher the handlers build from the config connects with that dialer. Any
|
||||
// other dialer would try to connect until the upstream fetch timeout, which
|
||||
// is short so that the test then fails quickly.
|
||||
func TestHandlersBuildTheirOwnFetcherWhenNoneIsProvided(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const host = "192.0.2.10"
|
||||
|
||||
stateDir := t.TempDir()
|
||||
cfg := &config.Config{
|
||||
SigningKey: testSigningKey,
|
||||
StateDir: stateDir,
|
||||
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||
AllowlistHosts: []string{host},
|
||||
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("192.0.2.0/24")},
|
||||
UpstreamFetchTimeout: 2 * time.Second,
|
||||
// With no connection slots, the fetch would fail before dialing.
|
||||
UpstreamConnections: config.DefaultUpstreamConnections,
|
||||
}
|
||||
|
||||
var h *Handlers
|
||||
|
||||
app := fxtest.New(t,
|
||||
fx.Supply(cfg),
|
||||
fx.Provide(globals.New, logger.New, database.New, healthcheck.New, New),
|
||||
fx.Populate(&h),
|
||||
)
|
||||
app.RequireStart()
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
r := chi.NewRouter()
|
||||
r.Get("/v1/image/*", h.HandleImage())
|
||||
|
||||
rec := sendGet(t, r, photoURL(host))
|
||||
checkErrorBody(t, rec, http.StatusForbidden, "forbidden")
|
||||
}
|
||||
@@ -12,16 +12,10 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// mismatchedSourcePath is the path of the source that
|
||||
@@ -31,51 +25,38 @@ import (
|
||||
const mismatchedSourcePath = "/images/source.png"
|
||||
|
||||
// newRoutesFetchingThrough returns both image routes, and the Handlers behind
|
||||
// them, whose image service fetches through fetcher, and which have
|
||||
// allowlistedHost on their allowlist. The handlers and the image service log
|
||||
// to log.
|
||||
// them, whose image service fetches through fetcher and has allowlistedHost on
|
||||
// its allowlist. The handlers and the image service log to log.
|
||||
func newRoutesFetchingThrough(
|
||||
t *testing.T, fetcher httpfetcher.Fetcher, log *slog.Logger,
|
||||
) (*Handlers, http.Handler) {
|
||||
t.Helper()
|
||||
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
|
||||
StateDir: t.TempDir(),
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("imgcache.NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
svc, err := imageproxy.New(&imageproxy.Config{
|
||||
Cache: cache,
|
||||
Fetcher: fetcher,
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
Logger: log,
|
||||
}, prometheus.NewRegistry())
|
||||
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||
Cache: cache,
|
||||
Fetcher: fetcher,
|
||||
SigningKey: testSigningKey,
|
||||
Allowlist: []string{allowlistedHost},
|
||||
Logger: log,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("imageproxy.New() error = %v", err)
|
||||
t.Fatalf("imgcache.NewService() error = %v", err)
|
||||
}
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
encGen, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
|
||||
h := &Handlers{
|
||||
log: log,
|
||||
imgSvc: svc,
|
||||
encGen: encGen,
|
||||
signer: signature.New(testSigningKey),
|
||||
allowlist: allowlist.New([]string{allowlistedHost}),
|
||||
}
|
||||
h := &Handlers{log: log, imgSvc: svc, encGen: encGen}
|
||||
|
||||
r := chi.NewRouter()
|
||||
r.Get("/v1/image/*", h.HandleImage())
|
||||
@@ -104,7 +85,7 @@ func mismatchedSourceEncURL(t *testing.T, h *Handlers) string {
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Time{})
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -18,17 +18,16 @@ var (
|
||||
"image/jxl, image/avif, image/webp or image/jpeg")
|
||||
)
|
||||
|
||||
// chooseAutoFormat returns the format to serve for the requested format: for
|
||||
// the format auto, the format formatForAccept chooses from r's Accept header,
|
||||
// and any other format as it is. For auto it adds Vary: Accept to the
|
||||
// chooseAutoFormat replaces the format auto in req with the format
|
||||
// formatForAccept chooses from r's Accept header, and adds Vary: Accept to the
|
||||
// response, which then depends on that header. It answers 400 for an Accept
|
||||
// header that is not valid and 406 for one that allows none of the formats,
|
||||
// and reports whether the image can be served.
|
||||
// and reports whether req can be served. Any other format is left as it is.
|
||||
func (s *Handlers) chooseAutoFormat(
|
||||
w http.ResponseWriter, r *http.Request, requested imagerequest.Format,
|
||||
) (imagerequest.Format, bool) {
|
||||
if requested != imagerequest.FormatAuto {
|
||||
return requested, true
|
||||
w http.ResponseWriter, r *http.Request, req *imagerequest.Request,
|
||||
) bool {
|
||||
if req.Transform.Format != imagerequest.FormatAuto {
|
||||
return true
|
||||
}
|
||||
|
||||
w.Header().Add("Vary", "Accept")
|
||||
@@ -37,16 +36,18 @@ func (s *Handlers) chooseAutoFormat(
|
||||
if errors.Is(err, errNotAcceptable) {
|
||||
s.respondError(w, err.Error(), http.StatusNotAcceptable)
|
||||
|
||||
return "", false
|
||||
return false
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
s.respondError(w, err.Error(), http.StatusBadRequest)
|
||||
|
||||
return "", false
|
||||
return false
|
||||
}
|
||||
|
||||
return format, true
|
||||
req.Transform.Format = format
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// formatForAccept returns the format an auto URL is served in for the Accept
|
||||
@@ -65,20 +66,20 @@ func formatForAccept(accept string) (imagerequest.Format, error) {
|
||||
return imagerequest.FormatJPEG, nil
|
||||
}
|
||||
|
||||
if qualities[imagerequest.FormatJXL.MIMEType()] > 0 {
|
||||
if qualities["image/jxl"] > 0 {
|
||||
return imagerequest.FormatJXL, nil
|
||||
}
|
||||
|
||||
if qualities[imagerequest.FormatAVIF.MIMEType()] > 0 {
|
||||
if qualities["image/avif"] > 0 {
|
||||
return imagerequest.FormatAVIF, nil
|
||||
}
|
||||
|
||||
if qualities[imagerequest.FormatWebP.MIMEType()] > 0 {
|
||||
if qualities["image/webp"] > 0 {
|
||||
return imagerequest.FormatWebP, nil
|
||||
}
|
||||
|
||||
// For JPEG, the most specific entry the header has decides
|
||||
quality, named := qualities[imagerequest.FormatJPEG.MIMEType()]
|
||||
quality, named := qualities["image/jpeg"]
|
||||
if !named {
|
||||
quality, named = qualities["image/*"]
|
||||
}
|
||||
|
||||
@@ -75,13 +75,11 @@ func TestFormatForAccept(t *testing.T) {
|
||||
|
||||
// autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL,
|
||||
// both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in
|
||||
// the format auto. The first is signed with testSigningKey, the key the
|
||||
// handlers that newSignedHostServer builds check signatures with. The second
|
||||
// is made with h's generator.
|
||||
// the format auto, made with h's image service and generator.
|
||||
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
|
||||
t.Helper()
|
||||
|
||||
signedURL := generateSignedURL(testSigningKey, &imagerequest.Request{
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
@@ -91,6 +89,9 @@ func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
|
||||
Format: imagerequest.FormatAuto,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
token, err := h.encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
@@ -101,7 +102,8 @@ func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatAuto,
|
||||
},
|
||||
}, time.Now().Add(time.Minute))
|
||||
Expires: time.Now().Add(time.Minute),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
@@ -186,9 +188,9 @@ func TestFormatAuto_ChosenFromAccept(t *testing.T) {
|
||||
func TestFormatAuto_SignatureCoversAuto(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
|
||||
signedForAVIF := generateSignedURL(testSigningKey, &imagerequest.Request{
|
||||
signedForAVIF, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
@@ -198,6 +200,9 @@ func TestFormatAuto_SignatureCoversAuto(t *testing.T) {
|
||||
Format: imagerequest.FormatAVIF,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
target := strings.Replace(signedForAVIF, "/50x50.avif?", "/50x50.auto?", 1)
|
||||
if target == signedForAVIF {
|
||||
|
||||
+132
-21
@@ -4,6 +4,7 @@ package handlers
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"time"
|
||||
@@ -11,13 +12,13 @@ import (
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/logger"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// Params defines dependencies for Handlers.
|
||||
@@ -26,48 +27,70 @@ type Params struct {
|
||||
|
||||
Logger *logger.Logger
|
||||
Healthcheck *healthcheck.Healthcheck
|
||||
Database *database.Database
|
||||
Config *config.Config
|
||||
Keys seal.Keys
|
||||
|
||||
ImageService *imageproxy.Proxy
|
||||
SessionManager *session.Manager
|
||||
EncryptedURLGenerator *encurl.Generator
|
||||
// Fetcher, when provided, fetches upstream images in place of the
|
||||
// fetcher the handlers build from the config. Only tests provide one;
|
||||
// pixad does not.
|
||||
Fetcher httpfetcher.Fetcher `optional:"true"`
|
||||
}
|
||||
|
||||
// Handlers provides HTTP request handlers.
|
||||
type Handlers struct {
|
||||
log *slog.Logger
|
||||
hc *healthcheck.Healthcheck
|
||||
db *database.Database
|
||||
config *config.Config
|
||||
imgSvc *imageproxy.Proxy
|
||||
fetcher httpfetcher.Fetcher
|
||||
imgSvc *imgcache.Service
|
||||
imgCache *imgcache.Cache
|
||||
sessMgr *session.Manager
|
||||
encGen *encurl.Generator
|
||||
csrfProtect func(http.Handler) http.Handler
|
||||
|
||||
// signer checks the signature of a /v1/image/ URL with signing_key.
|
||||
signer *signature.Signer
|
||||
// allowlist matches the hosts of allowlist_hosts, which a /v1/image/
|
||||
// URL may name without a signature.
|
||||
allowlist *allowlist.HostAllowList
|
||||
// refererBlocklist matches the hosts of referer_blocklist; its IsAllowed
|
||||
// reports whether a URL's host is on that list.
|
||||
refererBlocklist *allowlist.HostAllowList
|
||||
}
|
||||
|
||||
// New creates a new Handlers instance.
|
||||
func New(_ fx.Lifecycle, params Params) *Handlers {
|
||||
return &Handlers{
|
||||
func New(lc fx.Lifecycle, params Params) (*Handlers, error) {
|
||||
csrfProtect, err := newCSRFProtect(params.Config.SigningKey, params.Config.Debug)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
s := &Handlers{
|
||||
log: params.Logger.Get(),
|
||||
hc: params.Healthcheck,
|
||||
db: params.Database,
|
||||
config: params.Config,
|
||||
imgSvc: params.ImageService,
|
||||
sessMgr: params.SessionManager,
|
||||
encGen: params.EncryptedURLGenerator,
|
||||
csrfProtect: newCSRFProtect(params.Keys.CSRFKey, params.Config.Debug),
|
||||
signer: signature.New(params.Config.SigningKey),
|
||||
allowlist: allowlist.New(params.Config.AllowlistHosts),
|
||||
fetcher: params.Fetcher,
|
||||
csrfProtect: csrfProtect,
|
||||
refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
|
||||
}
|
||||
|
||||
lc.Append(fx.Hook{
|
||||
//nolint:contextcheck // the cache's goroutines outlive OnStart; OnStop stops them
|
||||
OnStart: func(_ context.Context) error {
|
||||
return s.initImageService()
|
||||
},
|
||||
// The pending counts are written here, before the database's stop
|
||||
// hook closes it.
|
||||
OnStop: func(ctx context.Context) error {
|
||||
if s.imgCache == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return errors.Join(
|
||||
s.imgCache.StopEviction(ctx),
|
||||
s.imgCache.StopPendingCountWrites(ctx),
|
||||
)
|
||||
},
|
||||
})
|
||||
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// WaitForProcessing waits until no image is being processed, or until ctx
|
||||
@@ -76,6 +99,94 @@ func (s *Handlers) WaitForProcessing(ctx context.Context) int {
|
||||
return s.imgSvc.WaitForProcessing(ctx)
|
||||
}
|
||||
|
||||
// newCacheConfig builds the image cache's configuration from cfg.
|
||||
// cache_max_bytes: 0 disables the disk cache entirely; any other value
|
||||
// is the eviction limit in bytes; when it is omitted, the cache works
|
||||
// out the default limit itself.
|
||||
func newCacheConfig(cfg *config.Config, log *slog.Logger) imgcache.CacheConfig {
|
||||
return imgcache.CacheConfig{
|
||||
StateDir: cfg.StateDir,
|
||||
NegativeTTL: imgcache.DefaultNegativeTTL,
|
||||
MaxBytes: cfg.CacheMaxBytes,
|
||||
UseDefaultMaxBytes: !cfg.CacheMaxBytesExplicit,
|
||||
DisableDiskCache: cfg.CacheMaxBytesExplicit && cfg.CacheMaxBytes == 0,
|
||||
Logger: log,
|
||||
}
|
||||
}
|
||||
|
||||
// initImageService initializes the image cache and service.
|
||||
func (s *Handlers) initImageService() error {
|
||||
cache, err := imgcache.NewCache(s.db.DB(), newCacheConfig(s.config, s.log))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.imgCache = cache
|
||||
|
||||
// Background eviction: startup reconciliation, then periodic and
|
||||
// write-pressure passes, and the removal of expired negative cache
|
||||
// rows, which is all it does when the disk cache is disabled.
|
||||
cache.StartEviction(imgcache.DefaultEvictionInterval)
|
||||
|
||||
// Writes the counts requests could not write by their deadline
|
||||
cache.StartPendingCountWrites()
|
||||
|
||||
// Create the fetcher config
|
||||
fetcherCfg := httpfetcher.DefaultConfig()
|
||||
fetcherCfg.AllowHTTP = s.config.AllowHTTP
|
||||
fetcherCfg.Timeout = s.config.UpstreamFetchTimeout
|
||||
fetcherCfg.MaxResponseSize = s.config.UpstreamMaxResponseSize
|
||||
|
||||
if s.config.UpstreamConnectionsPerHost > 0 {
|
||||
fetcherCfg.MaxConnectionsPerHost = s.config.UpstreamConnectionsPerHost
|
||||
}
|
||||
|
||||
fetcherCfg.MaxConnections = s.config.UpstreamConnections
|
||||
fetcherCfg.BlockedNetworks = s.config.BlockedNetworks
|
||||
|
||||
// Create the service. With no fetcher provided, it builds its own from
|
||||
// fetcherCfg.
|
||||
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||
Cache: cache,
|
||||
FetcherConfig: fetcherCfg,
|
||||
Fetcher: s.fetcher,
|
||||
SigningKey: s.config.SigningKey,
|
||||
Allowlist: s.config.AllowlistHosts,
|
||||
MaxConcurrentProcessing: s.config.MaxConcurrentProcessing,
|
||||
Logger: s.log,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.imgSvc = svc
|
||||
s.log.Info("image service initialized")
|
||||
|
||||
// Initialize session manager (signing key is validated at config load
|
||||
// time). Session cookies are always Secure/HttpOnly/SameSite=Strict.
|
||||
// The random value that each logout replaces is saved in the database,
|
||||
// so it outlasts a restart.
|
||||
sessMgr, err := session.NewManagerWithDatabase(
|
||||
context.Background(), s.config.SigningKey, s.db.DB())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.sessMgr = sessMgr
|
||||
|
||||
// Initialize encrypted URL generator
|
||||
encGen, err := encurl.NewGenerator(s.config.SigningKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.encGen = encGen
|
||||
|
||||
s.log.Info("session manager and URL generator initialized")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Handlers) respondJSON(w http.ResponseWriter, data any, status int) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
|
||||
@@ -18,20 +18,15 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// testFixtures contains test data.
|
||||
type testFixtures struct {
|
||||
handler *Handlers
|
||||
service *imgcache.Service
|
||||
goodHost string
|
||||
}
|
||||
|
||||
@@ -50,37 +45,32 @@ func setupTestHandler(t *testing.T) *testFixtures {
|
||||
tmpDir := t.TempDir()
|
||||
db := setupTestDB(t)
|
||||
|
||||
cache, err := imgcache.NewCache(db, imgcache.Config{
|
||||
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{
|
||||
StateDir: tmpDir,
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
|
||||
svc, err := imageproxy.New(&imageproxy.Config{
|
||||
Cache: cache,
|
||||
Fetcher: newMockFetcher(mockFS),
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
}, prometheus.NewRegistry())
|
||||
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||
Cache: cache,
|
||||
Fetcher: newMockFetcher(mockFS),
|
||||
SigningKey: "test-signing-key-must-be-32-chars",
|
||||
Allowlist: []string{goodHost},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create proxy: %v", err)
|
||||
t.Fatalf("failed to create service: %v", err)
|
||||
}
|
||||
|
||||
h := &Handlers{
|
||||
imgSvc: svc,
|
||||
log: slog.Default(),
|
||||
signer: signature.New("test-signing-key-must-be-32-chars"),
|
||||
allowlist: allowlist.New([]string{goodHost}),
|
||||
imgSvc: svc,
|
||||
log: slog.Default(),
|
||||
}
|
||||
|
||||
return &testFixtures{
|
||||
handler: h,
|
||||
service: svc,
|
||||
goodHost: goodHost,
|
||||
}
|
||||
}
|
||||
@@ -131,9 +121,10 @@ func newMockFetcher(fs fs.FS) *mockFetcher {
|
||||
}
|
||||
|
||||
func (f *mockFetcher) Fetch(
|
||||
_ context.Context, source imagerequest.Source,
|
||||
_ context.Context, url string,
|
||||
) (*httpfetcher.FetchResult, error) {
|
||||
path := source.Host + source.Path
|
||||
// Remove https:// prefix
|
||||
path := url[8:] // Remove "https://"
|
||||
|
||||
data, err := fs.ReadFile(f.fs, path)
|
||||
if err != nil {
|
||||
|
||||
+99
-132
@@ -7,15 +7,14 @@ import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
@@ -28,13 +27,13 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
req, sig, expires, ok := s.parseImageRequest(w, r)
|
||||
req, ok := s.parseImageRequest(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
// Validate signature if required
|
||||
err := s.validateSignature(req, sig, expires)
|
||||
err := s.imgSvc.ValidateRequest(req)
|
||||
if err != nil {
|
||||
s.log.Warn("signature validation failed",
|
||||
"host", req.Source.Host,
|
||||
@@ -46,7 +45,27 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
s.serveImage(w, r, req, expires)
|
||||
// The signature covers the format auto, not the format chosen
|
||||
if !s.chooseAutoFormat(w, r, req) {
|
||||
return
|
||||
}
|
||||
|
||||
// Get cache key for logging
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
|
||||
// Get the image (from cache or fetch/process)
|
||||
startTime := time.Now()
|
||||
|
||||
resp, err := s.imgSvc.Get(r.Context(), req)
|
||||
if err != nil {
|
||||
s.respondImageError(w, req, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
s.writeImageResponse(w, r, req, resp, cacheKey, startTime)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,11 +82,11 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
|
||||
}
|
||||
|
||||
// parseImageRequest parses the image URL's path and query parameters into
|
||||
// an image request, and returns it with the URL's signature and expiry. On
|
||||
// invalid input it writes an error response and returns false.
|
||||
// an image request. On invalid input it writes an error response and
|
||||
// returns false.
|
||||
func (s *Handlers) parseImageRequest(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
) (*imagerequest.Request, string, time.Time, bool) {
|
||||
) (*imagerequest.Request, bool) {
|
||||
// The path is parsed decoded once, as r.URL.Path holds it, and not from
|
||||
// chi's wildcard: chi routes on the path as the client wrote it whenever
|
||||
// that is not Go's own encoding of it, so its wildcard would still hold
|
||||
@@ -81,7 +100,7 @@ func (s *Handlers) parseImageRequest(
|
||||
)
|
||||
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Parse signature params from query string. r.URL.Query() would silently
|
||||
@@ -94,7 +113,7 @@ func (s *Handlers) parseImageRequest(
|
||||
s.respondError(w, fmt.Sprintf("invalid query string %q: %v",
|
||||
r.URL.RawQuery, err), http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
for name, values := range query {
|
||||
@@ -102,15 +121,17 @@ func (s *Handlers) parseImageRequest(
|
||||
s.respondError(w, fmt.Sprintf("invalid %s: given more than once",
|
||||
name), http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
|
||||
expires, err := parseExpires(query)
|
||||
req.Signature = query.Get("sig")
|
||||
|
||||
req.Expires, err = parseExpires(query)
|
||||
if err != nil {
|
||||
s.respondError(w, err.Error(), http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Parse optional quality and fit params. Only a q missing from the URL is
|
||||
@@ -122,7 +143,7 @@ func (s *Handlers) parseImageRequest(
|
||||
s.respondError(w, `invalid q: not a number, got ""`,
|
||||
http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
req.Transform.Quality, err = parseFormInt(query, "q",
|
||||
@@ -131,7 +152,7 @@ func (s *Handlers) parseImageRequest(
|
||||
s.respondError(w, fmt.Sprintf("%v, got %q", err, qStr),
|
||||
http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Only a fit missing from the URL is cover. A fit in the URL that is not a
|
||||
@@ -141,7 +162,7 @@ func (s *Handlers) parseImageRequest(
|
||||
if query.Has("fit") && req.Transform.FitMode == "" {
|
||||
s.respondError(w, `invalid fit: not a fit mode, got ""`, http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Default fit mode if not set
|
||||
@@ -158,40 +179,15 @@ func (s *Handlers) parseImageRequest(
|
||||
s.respondError(w, "invalid image request: "+err.Error(),
|
||||
http.StatusBadRequest)
|
||||
|
||||
return nil, "", time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
return req, query.Get("sig"), expires, true
|
||||
}
|
||||
|
||||
// validateSignature returns nil when the source host of req is on
|
||||
// allowlist_hosts and names no port. Otherwise it verifies sig, the URL's
|
||||
// signature of req, and expires, the URL's expiry.
|
||||
func (s *Handlers) validateSignature(
|
||||
req *imagerequest.Request, sig string, expires time.Time,
|
||||
) error {
|
||||
// An allowlist entry names no port, so a host followed by ":", with or
|
||||
// without a port, needs a signature. Go's URL type reads the port as the
|
||||
// fetcher does: in an IPv6 address written without brackets, such as
|
||||
// 2001:db8::1:8443, it is what follows the last ":". This is checked here
|
||||
// and not in allowlist.IsAllowed, which also matches referer_blocklist,
|
||||
// where a Referer must be blocked whatever its port. Port and IsAllowed
|
||||
// read only the host of a URL.
|
||||
sourceURL := &url.URL{Host: req.Source.Host}
|
||||
hostHasPort := sourceURL.Port() != "" ||
|
||||
strings.HasSuffix(sourceURL.Host, ":")
|
||||
|
||||
if !hostHasPort && s.allowlist.IsAllowed(sourceURL) {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Signature required for non-allowed hosts
|
||||
return s.signer.Verify(req, sig, expires)
|
||||
return req, true
|
||||
}
|
||||
|
||||
// parseExpires reads the exp query parameter, a Unix time in seconds. An exp
|
||||
// missing from the URL gives the zero time, which the signature check refuses
|
||||
// as a missing exp. An exp in the URL that is not a whole number, an empty
|
||||
// missing from the URL gives the zero time, which the signature check takes
|
||||
// as no expiration. An exp in the URL that is not a whole number, an empty
|
||||
// one included, is an error naming exp and the value.
|
||||
func parseExpires(query url.Values) (time.Time, error) {
|
||||
if !query.Has("exp") {
|
||||
@@ -219,11 +215,7 @@ func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// respondImageError answers err, the image service's error for req, on both
|
||||
// image routes. It logs a 500 at error level, as pixa's own failure. It logs
|
||||
// the other answers at warn level, since the client, the upstream or a full
|
||||
// limit caused them. The 415 for a source refused for its first bytes is not
|
||||
// logged here, because the image service logs it.
|
||||
// respondImageError maps image retrieval errors to HTTP responses.
|
||||
func (s *Handlers) respondImageError(
|
||||
w http.ResponseWriter, req *imagerequest.Request, err error,
|
||||
) {
|
||||
@@ -239,53 +231,59 @@ func (s *Handlers) respondImageError(
|
||||
return
|
||||
}
|
||||
|
||||
var (
|
||||
status int
|
||||
message string
|
||||
)
|
||||
|
||||
switch {
|
||||
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
|
||||
status, message = http.StatusForbidden, "forbidden"
|
||||
case errors.Is(err, httpfetcher.ErrUpstreamError):
|
||||
status, message = http.StatusBadGateway, "upstream error"
|
||||
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
|
||||
status, message = http.StatusGatewayTimeout, "upstream timeout"
|
||||
case errors.Is(err, httpfetcher.ErrTooManyConnections),
|
||||
errors.Is(err, imageprocessor.ErrTooManyImages),
|
||||
errors.Is(err, imageproxy.ErrProcessingTimedOut):
|
||||
status = http.StatusServiceUnavailable
|
||||
message = "server busy, try again later"
|
||||
case errors.Is(err, httpfetcher.ErrInvalidContentType),
|
||||
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
|
||||
status = http.StatusUnsupportedMediaType
|
||||
message = "unsupported source format"
|
||||
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
|
||||
status = http.StatusBadRequest
|
||||
message = fmt.Sprintf("size too large for the source: the side "+
|
||||
"computed from its proportions is above %d",
|
||||
imageprocessor.MaxOutputDimension)
|
||||
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
|
||||
status = http.StatusBadRequest
|
||||
message = fmt.Sprintf("size too large for AVIF: the output "+
|
||||
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels)
|
||||
default:
|
||||
s.log.Error("failed to get image",
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"error", err,
|
||||
)
|
||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
s.log.Warn("failed to get image",
|
||||
s.log.Error("failed to get image",
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"error", err,
|
||||
)
|
||||
s.respondError(w, message, status)
|
||||
|
||||
// Check for specific error types
|
||||
if errors.Is(err, httpfetcher.ErrSSRFBlocked) {
|
||||
s.respondError(w, "forbidden", http.StatusForbidden)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, httpfetcher.ErrUpstreamError) {
|
||||
s.respondError(w, "upstream error", http.StatusBadGateway)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
|
||||
errors.Is(err, imageprocessor.ErrTooManyImages) ||
|
||||
errors.Is(err, imgcache.ErrProcessingTimedOut) {
|
||||
s.respondError(w, "server busy, try again later",
|
||||
http.StatusServiceUnavailable)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, httpfetcher.ErrInvalidContentType) ||
|
||||
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat) {
|
||||
s.respondError(w, "unsupported source format",
|
||||
http.StatusUnsupportedMediaType)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, imageprocessor.ErrOutputTooLarge) {
|
||||
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
|
||||
"computed from its proportions is above %d",
|
||||
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
|
||||
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||
http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||
}
|
||||
|
||||
// cacheControl returns the Cache-Control header for an image served through a
|
||||
@@ -342,34 +340,13 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// serveImage serves the image that req asks for. Both image routes call it
|
||||
// once they have the request and its URL's expiry, which ends the
|
||||
// Cache-Control max-age. It chooses the format for the format auto, answers
|
||||
// If-None-Match with 304 and HEAD with the headers only, and answers an error
|
||||
// of the image service with respondImageError.
|
||||
func (s *Handlers) serveImage(
|
||||
// writeImageResponse writes headers and streams the image content,
|
||||
// handling conditional and HEAD requests.
|
||||
func (s *Handlers) writeImageResponse(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
req *imagerequest.Request, expires time.Time,
|
||||
req *imagerequest.Request, resp *imgcache.ImageResponse,
|
||||
cacheKey imagerequest.VariantKey, startTime time.Time,
|
||||
) {
|
||||
// The signature or the token covers the format auto, not the format chosen
|
||||
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
req.Transform.Format = format
|
||||
|
||||
startTime := time.Now()
|
||||
|
||||
resp, err := s.imgSvc.Get(r.Context(), req)
|
||||
if err != nil {
|
||||
s.respondImageError(w, req, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
// Set response headers
|
||||
w.Header().Set("Content-Type", resp.ContentType)
|
||||
|
||||
@@ -378,7 +355,7 @@ func (s *Handlers) serveImage(
|
||||
}
|
||||
|
||||
// Cache control headers
|
||||
w.Header().Set("Cache-Control", cacheControl(expires))
|
||||
w.Header().Set("Cache-Control", cacheControl(req.Expires))
|
||||
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
|
||||
|
||||
if notModified(w, r, resp.ETag) {
|
||||
@@ -395,28 +372,18 @@ func (s *Handlers) serveImage(
|
||||
// Stream the response
|
||||
w.WriteHeader(http.StatusOK)
|
||||
|
||||
// A failed write is most often a client that has gone away, not pixa's
|
||||
// failure, so it is logged at warn level
|
||||
servedBytes, err := io.Copy(w, resp.Content)
|
||||
if err != nil {
|
||||
s.log.Warn("failed to write response",
|
||||
"request_id", middleware.GetReqID(r.Context()),
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"served_bytes", servedBytes,
|
||||
s.log.Error("failed to write response",
|
||||
"error", err,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Log cache status and timing after serving
|
||||
duration := time.Since(startTime)
|
||||
s.log.Info("image served",
|
||||
"request_id", middleware.GetReqID(r.Context()),
|
||||
"cache_key", imagerequest.CacheKey(req),
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"cache_key", cacheKey,
|
||||
"cache_status", resp.CacheStatus,
|
||||
"duration_ms", duration.Milliseconds(),
|
||||
"format", req.Transform.Format,
|
||||
|
||||
@@ -12,16 +12,10 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// photoPath is the path of the JPEG that newSignedHostServer serves.
|
||||
@@ -36,46 +30,37 @@ func newSignedHostServer(
|
||||
) (*Handlers, http.Handler) {
|
||||
t.Helper()
|
||||
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
|
||||
StateDir: t.TempDir(),
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("imgcache.NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
|
||||
|
||||
svc, err := imageproxy.New(&imageproxy.Config{
|
||||
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||
Cache: cache,
|
||||
Fetcher: newMockFetcher(fstest.MapFS{
|
||||
signedHost + photoPath: &fstest.MapFile{Data: jpegData},
|
||||
}),
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
Logger: log,
|
||||
}, prometheus.NewRegistry())
|
||||
SigningKey: testSigningKey,
|
||||
Logger: log,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("imageproxy.New() error = %v", err)
|
||||
t.Fatalf("imgcache.NewService() error = %v", err)
|
||||
}
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
encGen, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
|
||||
h := &Handlers{
|
||||
log: log,
|
||||
imgSvc: svc,
|
||||
encGen: encGen,
|
||||
signer: signature.New(testSigningKey),
|
||||
allowlist: allowlist.New(nil),
|
||||
log: log,
|
||||
imgSvc: svc,
|
||||
encGen: encGen,
|
||||
}
|
||||
|
||||
r := chi.NewRouter()
|
||||
@@ -121,9 +106,9 @@ func getMaxAge(t *testing.T, srv http.Handler, target string) int {
|
||||
func TestHandleImage_SignedURL_MaxAgeEndsAtExp(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
|
||||
signedURL := generateSignedURL(testSigningKey, &imagerequest.Request{
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
@@ -133,6 +118,9 @@ func TestHandleImage_SignedURL_MaxAgeEndsAtExp(t *testing.T) {
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
maxAge := getMaxAge(t, srv, signedURL)
|
||||
if maxAge < 50 || maxAge > 60 {
|
||||
@@ -209,7 +197,8 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, tt.expires)
|
||||
Expires: tt.expires,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -1,28 +1,19 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"image/color"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
@@ -33,38 +24,29 @@ import (
|
||||
const allowlistedHost = "allowed.example.com"
|
||||
|
||||
// newImageRoute returns the image route of a Handlers whose service fetches
|
||||
// with fetcher, and which checks signatures with testSigningKey.
|
||||
// with fetcher and checks signatures with testSigningKey.
|
||||
func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
|
||||
t.Helper()
|
||||
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
|
||||
StateDir: t.TempDir(),
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
|
||||
svc, err := imageproxy.New(&imageproxy.Config{
|
||||
Cache: cache,
|
||||
Fetcher: fetcher,
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
}, prometheus.NewRegistry())
|
||||
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||
Cache: cache,
|
||||
Fetcher: fetcher,
|
||||
SigningKey: testSigningKey,
|
||||
Allowlist: []string{allowlistedHost},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create proxy: %v", err)
|
||||
t.Fatalf("failed to create service: %v", err)
|
||||
}
|
||||
|
||||
h := &Handlers{
|
||||
imgSvc: svc,
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
signer: signature.New(testSigningKey),
|
||||
allowlist: allowlist.New([]string{allowlistedHost}),
|
||||
}
|
||||
h := &Handlers{imgSvc: svc, log: slog.New(slog.DiscardHandler)}
|
||||
|
||||
r := chi.NewRouter()
|
||||
r.Get("/v1/image/*", h.HandleImage())
|
||||
@@ -73,7 +55,7 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
|
||||
}
|
||||
|
||||
// newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on
|
||||
// each of hosts, and answers any other source with an upstream error.
|
||||
// each of hosts, and answers any other URL with an upstream error.
|
||||
func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher {
|
||||
t.Helper()
|
||||
|
||||
@@ -100,10 +82,10 @@ func photoURLWithSig(host, sig string, expires time.Time) string {
|
||||
return fmt.Sprintf("%s?sig=%s&exp=%d", photoURL(host), sig, expires.Unix())
|
||||
}
|
||||
|
||||
// photoRequest returns the image request of photoURL(host) at the default
|
||||
// quality and fit.
|
||||
func photoRequest(host string) *imagerequest.Request {
|
||||
return &imagerequest.Request{
|
||||
// photoSignature returns the signature of photoURL(host) at the default
|
||||
// quality and fit, made with key and expiring at expires.
|
||||
func photoSignature(key, host string, expires time.Time) string {
|
||||
return signature.New(key).Sign(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: host,
|
||||
Path: photoPath,
|
||||
@@ -114,13 +96,8 @@ func photoRequest(host string) *imagerequest.Request {
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// photoSignature returns the signature of photoURL(host) at the default
|
||||
// quality and fit, made with key and expiring at expires.
|
||||
func photoSignature(key, host string, expires time.Time) string {
|
||||
return signature.New(key).Sign(photoRequest(host), expires)
|
||||
Expires: expires,
|
||||
})
|
||||
}
|
||||
|
||||
// sendGet sends a GET for target to route and returns the response.
|
||||
@@ -191,12 +168,7 @@ func TestHandleImage_ErrorAnswers(t *testing.T) {
|
||||
photos := newPhotoFetcher(t,
|
||||
signedHost, parentHost, siblingHost, subdomainHost, appendedHost)
|
||||
// The real fetcher refuses localhost before any lookup or connection.
|
||||
realFetcher := httpfetcher.New(&httpfetcher.Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
})
|
||||
realFetcher := httpfetcher.New(httpfetcher.DefaultConfig())
|
||||
|
||||
exp := time.Now().Add(time.Hour)
|
||||
expired := time.Now().Add(-time.Hour)
|
||||
@@ -301,9 +273,10 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
|
||||
// host can be, and checks that each is refused with 400 naming the host as
|
||||
// decoded: allowlistedHost with a character after it that a URL host cannot
|
||||
// hold, percent-encoded, an IPv6 address without brackets, and brackets
|
||||
// around something that is not an IPv6 address. Later steps put the host
|
||||
// unparsed into the allowlist check, the cache key and the upstream URL, so
|
||||
// such a host is refused first.
|
||||
// around something that is not an IPv6 address. A "?" or "#" after
|
||||
// allowlistedHost would otherwise end the host when the source URL is read,
|
||||
// so the request would pass the allowlist and be fetched and cached under a
|
||||
// host that is not the one fetched.
|
||||
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -374,9 +347,10 @@ func TestHandleImage_HostForms_Served(t *testing.T) {
|
||||
|
||||
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
|
||||
// source path or source query holds a "#", written as %23, and checks that
|
||||
// each is refused with 400, the one signed for that source path included.
|
||||
// A "#" starts a URL's fragment, so neither the source path nor the source
|
||||
// query may hold one.
|
||||
// each is refused with 400, the one signed for that source path included. The
|
||||
// source URL would otherwise end at the "#" when it is fetched, so each text
|
||||
// after it would fetch the image at photoPath and cache it under keys of its
|
||||
// own.
|
||||
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -394,7 +368,8 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}, exp)
|
||||
Expires: exp,
|
||||
})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -420,168 +395,3 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// failingFetcher answers every fetch with its error.
|
||||
type failingFetcher struct {
|
||||
err error
|
||||
}
|
||||
|
||||
// Fetch returns f's error.
|
||||
func (f failingFetcher) Fetch(
|
||||
context.Context, imagerequest.Source,
|
||||
) (*httpfetcher.FetchResult, error) {
|
||||
return nil, f.err
|
||||
}
|
||||
|
||||
// photoURLsByRoute returns, for each image route, the function that makes the
|
||||
// URL of the JPEG at photoPath on signedHost on that route.
|
||||
func photoURLsByRoute() map[string]func(*testing.T, *Handlers) string {
|
||||
return map[string]func(*testing.T, *Handlers) string{
|
||||
"/v1/image/": signedPhotoURL,
|
||||
"/v1/e/": encPhotoURL,
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageRoutes_SameErrorAnswers checks that both image routes answer each
|
||||
// upstream fetch error below with the same status. They log it as "failed to
|
||||
// get image" at the same level: error for the 500 and warn for the others.
|
||||
// Each request gets routes of its own, so no request finds another's failed
|
||||
// fetch in the negative cache.
|
||||
func TestImageRoutes_SameErrorAnswers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
wantStatus int
|
||||
wantLevel slog.Level
|
||||
}{
|
||||
{"blocked upstream address", httpfetcher.ErrSSRFBlocked,
|
||||
http.StatusForbidden, slog.LevelWarn},
|
||||
{"upstream error status",
|
||||
fmt.Errorf("%w: status 404", httpfetcher.ErrUpstreamError),
|
||||
http.StatusBadGateway, slog.LevelWarn},
|
||||
{"no response headers within the fetch timeout",
|
||||
httpfetcher.ErrUpstreamTimeout, http.StatusGatewayTimeout,
|
||||
slog.LevelWarn},
|
||||
{"no free upstream connection", httpfetcher.ErrTooManyConnections,
|
||||
http.StatusServiceUnavailable, slog.LevelWarn},
|
||||
{"Content-Type not a source format",
|
||||
fmt.Errorf("%w: image/svg+xml", httpfetcher.ErrInvalidContentType),
|
||||
http.StatusUnsupportedMediaType, slog.LevelWarn},
|
||||
{"any other failure", io.ErrUnexpectedEOF,
|
||||
http.StatusInternalServerError, slog.LevelError},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
for route, imageURL := range photoURLsByRoute() {
|
||||
t.Run(route+" "+tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var logs bytes.Buffer
|
||||
|
||||
h, routes := newRoutesFetchingThrough(t,
|
||||
failingFetcher{err: tt.err},
|
||||
slog.New(slog.NewJSONHandler(&logs, nil)))
|
||||
|
||||
rec := sendGet(t, routes, imageURL(t, h))
|
||||
if rec.Code != tt.wantStatus {
|
||||
t.Errorf("status = %d, want %d: %s",
|
||||
rec.Code, tt.wantStatus, rec.Body)
|
||||
}
|
||||
|
||||
levels := loggedLevels(t, &logs)
|
||||
if !slices.Contains(levels[tt.wantLevel.String()],
|
||||
"failed to get image") {
|
||||
t.Errorf(`no "failed to get image" line logged at level %s`,
|
||||
tt.wantLevel)
|
||||
}
|
||||
|
||||
if tt.wantLevel != slog.LevelError &&
|
||||
len(levels[slog.LevelError.String()]) != 0 {
|
||||
t.Errorf("logged at error level: %q",
|
||||
levels[slog.LevelError.String()])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// failingResponseWriter is a response writer whose Write always fails, as when
|
||||
// the client has gone away.
|
||||
type failingResponseWriter struct {
|
||||
*httptest.ResponseRecorder
|
||||
}
|
||||
|
||||
// Write writes nothing and returns io.ErrClosedPipe.
|
||||
func (failingResponseWriter) Write([]byte) (int, error) {
|
||||
return 0, io.ErrClosedPipe
|
||||
}
|
||||
|
||||
// TestImageRoutes_FailedWriteLoggedOnce checks that both image routes log a
|
||||
// failed write of the image once, at warn level, with the request's ID, the
|
||||
// source host and path, the bytes served and the error. Neither may also log
|
||||
// the image as served.
|
||||
func TestImageRoutes_FailedWriteLoggedOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const requestID = "test-request-id"
|
||||
|
||||
for route, imageURL := range photoURLsByRoute() {
|
||||
t.Run(route, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var logs bytes.Buffer
|
||||
|
||||
h, srv := newSignedHostServer(t,
|
||||
slog.New(slog.NewJSONHandler(&logs, nil)))
|
||||
|
||||
ctx := context.WithValue(t.Context(),
|
||||
middleware.RequestIDKey, requestID)
|
||||
srv.ServeHTTP(failingResponseWriter{httptest.NewRecorder()},
|
||||
httptest.NewRequestWithContext(
|
||||
ctx, http.MethodGet, imageURL(t, h), nil))
|
||||
|
||||
t.Logf("logged:\n%s", logs.String())
|
||||
|
||||
var failedWrites []map[string]any
|
||||
|
||||
dec := json.NewDecoder(&logs)
|
||||
for dec.More() {
|
||||
var line map[string]any
|
||||
|
||||
err := dec.Decode(&line)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding log line: %v", err)
|
||||
}
|
||||
|
||||
switch line["msg"] {
|
||||
case "image served":
|
||||
t.Error(`"image served" logged after a failed write`)
|
||||
case "failed to write response":
|
||||
failedWrites = append(failedWrites, line)
|
||||
}
|
||||
}
|
||||
|
||||
if len(failedWrites) != 1 {
|
||||
t.Fatalf(`"failed to write response" logged %d times, want 1`,
|
||||
len(failedWrites))
|
||||
}
|
||||
|
||||
want := map[string]any{
|
||||
"level": slog.LevelWarn.String(),
|
||||
"request_id": requestID,
|
||||
"host": signedHost,
|
||||
"path": photoPath,
|
||||
"served_bytes": float64(0),
|
||||
"error": io.ErrClosedPipe.Error(),
|
||||
}
|
||||
|
||||
for key, wantValue := range want {
|
||||
if got := failedWrites[0][key]; got != wantValue {
|
||||
t.Errorf("%s = %v, want %v", key, got, wantValue)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -37,47 +36,6 @@ func getImage(t *testing.T, fix *testFixtures, target string) int {
|
||||
return rec.Code
|
||||
}
|
||||
|
||||
// generateSignedURL returns the /v1/image/ URL of req, signed with key and
|
||||
// expiring after ttl. It signs a copy of req, so req itself is not changed.
|
||||
// An unset quality or fit in the copy is set to 85 or cover, the values the
|
||||
// route uses when a URL has no q or fit. The URL carries q and fit next to sig
|
||||
// and exp, so the route verifies it for the quality and fit it was signed
|
||||
// with.
|
||||
func generateSignedURL(
|
||||
key string, req *imagerequest.Request, ttl time.Duration,
|
||||
) string {
|
||||
signed := *req
|
||||
|
||||
if signed.Transform.Quality == 0 {
|
||||
signed.Transform.Quality = 85
|
||||
}
|
||||
|
||||
if signed.Transform.FitMode == "" {
|
||||
signed.Transform.FitMode = imagerequest.FitCover
|
||||
}
|
||||
|
||||
expires := time.Now().Add(ttl)
|
||||
sig := signature.New(key).Sign(&signed, expires)
|
||||
|
||||
// A source query follows the source path after a "?". The image route
|
||||
// reads the path decoded once, so the whole path is written in Go's own
|
||||
// percent-encoding of a URL path, which writes "?" as %3F and "%" as %25,
|
||||
// and leaves "/", ";", ",", "=" and "&" as they are.
|
||||
sourcePathAndQuery := signed.Source.Path
|
||||
if signed.Source.Query != "" {
|
||||
sourcePathAndQuery += "?" + signed.Source.Query
|
||||
}
|
||||
|
||||
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%dx%d.%s",
|
||||
signed.Source.Host, sourcePathAndQuery,
|
||||
signed.Transform.Size.Width, signed.Transform.Size.Height,
|
||||
signed.Transform.Format)
|
||||
|
||||
return fmt.Sprintf("%s?sig=%s&exp=%d&q=%d&fit=%s",
|
||||
(&url.URL{Path: unescapedPath}).EscapedPath(), sig, expires.Unix(),
|
||||
signed.Transform.Quality, signed.Transform.FitMode)
|
||||
}
|
||||
|
||||
// TestHandleImage_SignatureCoversQualityAndFit signs a URL for quality 85
|
||||
// and fit cover, the values the route uses when a URL has no q or fit, and
|
||||
// sends that signature with each q and fit below.
|
||||
@@ -97,7 +55,8 @@ func TestHandleImage_SignatureCoversQualityAndFit(t *testing.T) {
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}, expires)
|
||||
Expires: expires,
|
||||
})
|
||||
signedURL := fmt.Sprintf("/v1/image/%s/images/photo.jpg/50x50.jpeg?sig=%s&exp=%d",
|
||||
signedHost, sig, expires.Unix())
|
||||
|
||||
@@ -124,8 +83,8 @@ func TestHandleImage_SignatureCoversQualityAndFit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_GeneratedSignedURLVerifies sends URLs built by
|
||||
// generateSignedURL to the route.
|
||||
// TestHandleImage_GeneratedSignedURLVerifies sends URLs built by the
|
||||
// service's signed-URL generator to the route.
|
||||
func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -142,21 +101,25 @@ func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
signedURL := generateSignedURL("test-signing-key-must-be-32-chars",
|
||||
&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: "/images/photo.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: tt.quality,
|
||||
FitMode: tt.fitMode,
|
||||
},
|
||||
}, time.Hour)
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
status := getImage(t, setupTestHandler(t), signedURL)
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: "/images/photo.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: tt.quality,
|
||||
FitMode: tt.fitMode,
|
||||
},
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
status := getImage(t, fix, signedURL)
|
||||
if status != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||
}
|
||||
@@ -215,10 +178,10 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
|
||||
// by generateSignedURL for source URLs with a query to the route, through a
|
||||
// chi router: chi routes on the path as the client wrote it whenever that is
|
||||
// not Go's own encoding of it. A 502 means the signature passed, so the route
|
||||
// read the source path and query that were signed.
|
||||
// by the service's signed-URL generator for source URLs with a query to the
|
||||
// route, through a chi router: chi routes on the path as the client wrote it
|
||||
// whenever that is not Go's own encoding of it. A 502 means the signature
|
||||
// passed, so the route read the source path and query that were signed.
|
||||
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -238,20 +201,24 @@ func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
signedURL := generateSignedURL("test-signing-key-must-be-32-chars",
|
||||
&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
Query: tt.query,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Hour)
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
status := getImage(t, setupTestHandler(t), signedURL)
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
Query: tt.query,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
status := getImage(t, fix, signedURL)
|
||||
if status != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||
}
|
||||
@@ -280,7 +247,8 @@ func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}, expires)
|
||||
Expires: expires,
|
||||
})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -312,7 +280,7 @@ func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
|
||||
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
|
||||
// source query that each hold %25, the percent-escape of a literal "%", and
|
||||
// sends each to the route written as the README says, with that "%" written as
|
||||
// %25, and as generateSignedURL writes it. A 502 means the signature passed.
|
||||
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
|
||||
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -346,7 +314,8 @@ func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}, expires)
|
||||
Expires: expires,
|
||||
})
|
||||
|
||||
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
|
||||
@@ -357,23 +326,27 @@ func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(tt.name+" written by generateSignedURL", func(t *testing.T) {
|
||||
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
signedURL := generateSignedURL("test-signing-key-must-be-32-chars",
|
||||
&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: tt.sourcePath,
|
||||
Query: tt.sourceQuery,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Hour)
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
status := getImage(t, setupTestHandler(t), signedURL)
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: tt.sourcePath,
|
||||
Query: tt.sourceQuery,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
status := getImage(t, fix, signedURL)
|
||||
if status != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||
}
|
||||
|
||||
+123
-17
@@ -2,13 +2,21 @@ package handlers
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// HandleImageEnc handles requests to /v1/e/{token}/* for encrypted
|
||||
@@ -20,59 +28,157 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
req, expires, ok := s.parseImageEncRequest(w, r)
|
||||
if !ok {
|
||||
ctx := r.Context()
|
||||
start := time.Now()
|
||||
|
||||
req, ok := s.parseImageEncRequest(w, r)
|
||||
if !ok || !s.chooseAutoFormat(w, r, req) {
|
||||
return
|
||||
}
|
||||
|
||||
s.serveImage(w, r, req, expires)
|
||||
// Log the request
|
||||
s.log.Debug("encrypted image request",
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"dimensions", fmt.Sprintf("%dx%d",
|
||||
req.Transform.Size.Width, req.Transform.Size.Height),
|
||||
"format", req.Transform.Format,
|
||||
)
|
||||
|
||||
// Fetch and process the image (no signature validation
|
||||
// needed - encrypted URL is trusted)
|
||||
resp, err := s.imgSvc.Get(ctx, req)
|
||||
if err != nil {
|
||||
s.handleImageError(w, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
// Set response headers
|
||||
w.Header().Set("Content-Type", resp.ContentType)
|
||||
|
||||
if resp.ContentLength > 0 {
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
|
||||
}
|
||||
|
||||
// Cache headers: max-age ends at the URL's expiry
|
||||
w.Header().Set("Cache-Control", cacheControl(req.Expires))
|
||||
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
|
||||
|
||||
if notModified(w, r, resp.ETag) {
|
||||
return
|
||||
}
|
||||
|
||||
// A HEAD request gets the headers only
|
||||
if r.Method == http.MethodHead {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Stream the response
|
||||
written, err := io.Copy(w, resp.Content)
|
||||
if err != nil {
|
||||
s.log.Error("failed to write response", "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Log completion
|
||||
duration := time.Since(start)
|
||||
s.log.Info("image served",
|
||||
"request_id", middleware.GetReqID(ctx),
|
||||
"cache_key", imagerequest.CacheKey(req),
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"format", req.Transform.Format,
|
||||
"cache_status", resp.CacheStatus,
|
||||
"served_bytes", written,
|
||||
"duration_ms", duration.Milliseconds(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// parseImageEncRequest decrypts the token of an encrypted image URL into an
|
||||
// image request and checks it, and returns it with the token's expiry. On a
|
||||
// token that is missing, does not decrypt, has expired or asks for something
|
||||
// not valid, it writes an error response and returns false.
|
||||
// image request and checks it. On a token that is missing, does not decrypt,
|
||||
// has expired or asks for something not valid, it writes an error response
|
||||
// and returns false.
|
||||
func (s *Handlers) parseImageEncRequest(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
) (*imagerequest.Request, time.Time, bool) {
|
||||
) (*imagerequest.Request, bool) {
|
||||
// Extract token from URL
|
||||
token := chi.URLParam(r, "token")
|
||||
if token == "" {
|
||||
s.respondError(w, "missing token", http.StatusBadRequest)
|
||||
|
||||
return nil, time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Decrypt and validate the payload
|
||||
req, expires, err := s.encGen.Parse(token)
|
||||
req, err := s.encGen.Parse(token)
|
||||
if err != nil {
|
||||
if errors.Is(err, encurl.ErrExpired) {
|
||||
s.log.Debug("encrypted URL expired", "error", err)
|
||||
s.respondError(w, "URL has expired", http.StatusGone)
|
||||
|
||||
return nil, time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
s.log.Debug("invalid encrypted URL", "error", err)
|
||||
s.log.Debug("failed to decrypt URL", "error", err)
|
||||
s.respondError(w, "invalid encrypted URL", http.StatusBadRequest)
|
||||
|
||||
return nil, time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Apply the same dimension and fit-mode bounds as the plain image
|
||||
// route: a sealed payload is trusted for its origin, not for staying
|
||||
// within limits. This refuses an over-limit size before the source is
|
||||
// fetched. An unknown fit mode is a 400 here rather than a 500 from the
|
||||
// processor.
|
||||
// within limits, so an over-limit size or unknown fit mode is a 400
|
||||
// here rather than an out-of-memory or a 500 from the processor.
|
||||
err = imagerequest.ValidateImageRequest(req)
|
||||
if err != nil {
|
||||
s.log.Debug("encrypted URL failed validation", "error", err)
|
||||
s.respondError(w, "invalid encrypted URL: "+err.Error(),
|
||||
http.StatusBadRequest)
|
||||
|
||||
return nil, time.Time{}, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
return req, expires, true
|
||||
return req, true
|
||||
}
|
||||
|
||||
// handleImageError converts image service errors to HTTP responses.
|
||||
func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
|
||||
s.respondError(w, "forbidden", http.StatusForbidden)
|
||||
case errors.Is(err, httpfetcher.ErrUpstreamError):
|
||||
s.respondError(w, "upstream error", http.StatusBadGateway)
|
||||
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
|
||||
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
|
||||
case errors.Is(err, httpfetcher.ErrTooManyConnections),
|
||||
errors.Is(err, imageprocessor.ErrTooManyImages),
|
||||
errors.Is(err, imgcache.ErrProcessingTimedOut):
|
||||
s.respondError(w, "server busy, try again later",
|
||||
http.StatusServiceUnavailable)
|
||||
case errors.Is(err, httpfetcher.ErrInvalidContentType),
|
||||
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat),
|
||||
errors.Is(err, magic.ErrMagicByteMismatch),
|
||||
errors.Is(err, magic.ErrUnknownFormat),
|
||||
errors.Is(err, magic.ErrNotEnoughData):
|
||||
s.respondError(w, "unsupported source format",
|
||||
http.StatusUnsupportedMediaType)
|
||||
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
|
||||
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
|
||||
"computed from its proportions is above %d",
|
||||
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
|
||||
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
|
||||
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||
http.StatusBadRequest)
|
||||
default:
|
||||
s.log.Error("image request failed", "error", err)
|
||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,13 +6,11 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
)
|
||||
|
||||
// newEncTestServer builds a router serving the encrypted-URL route with a
|
||||
@@ -22,13 +20,11 @@ import (
|
||||
func newEncTestServer(t *testing.T) (*encurl.Generator, http.Handler) {
|
||||
t.Helper()
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
encGen, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
|
||||
h := &Handlers{
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
encGen: encGen,
|
||||
@@ -66,7 +62,7 @@ func TestHandleImageEnc_OverLimitDimension_Returns400(t *testing.T) {
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100000, Height: 100000},
|
||||
},
|
||||
}, time.Time{})
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
@@ -95,7 +91,7 @@ func TestHandleImageEnc_InvalidFitMode_Returns400(t *testing.T) {
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
FitMode: imagerequest.Fit("bogus"),
|
||||
},
|
||||
}, time.Time{})
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -10,7 +10,6 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
)
|
||||
|
||||
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
|
||||
@@ -71,24 +70,25 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
valid, err := h.encGen.Generate(&photo, time.Time{})
|
||||
valid, err := h.encGen.Generate(&photo)
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
expired, err := h.encGen.Generate(&photo, time.Now().Add(-time.Minute))
|
||||
expiredPhoto := photo
|
||||
expiredPhoto.Expires = time.Now().Add(-time.Minute)
|
||||
|
||||
expired, err := h.encGen.Generate(&expiredPhoto)
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
keys, err := seal.DeriveKeys("another-signing-key-fedcba9876543210")
|
||||
otherGen, err := encurl.NewGenerator("another-signing-key-fedcba9876543210")
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
otherGen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
|
||||
otherKey, err := otherGen.Generate(&photo, time.Time{})
|
||||
otherKey, err := otherGen.Generate(&photo)
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -69,7 +69,8 @@ func TestImageWithoutFormat_SignedAsJXL(t *testing.T) {
|
||||
Quality: encurl.DefaultQuality,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}, expires)
|
||||
Expires: expires,
|
||||
})
|
||||
query := fmt.Sprintf("?sig=%s&exp=%d", sig, expires.Unix())
|
||||
|
||||
for _, size := range []string{"50x50.jxl", "50x50"} {
|
||||
@@ -93,7 +94,7 @@ func TestImageEncWithoutFormat_ServesJPEGXL(t *testing.T) {
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
},
|
||||
}, time.Time{})
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -10,16 +10,10 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// blockedReferer is a page on leech.example, which newRefererRoutes puts on
|
||||
@@ -35,11 +29,11 @@ type countingFetcher struct {
|
||||
|
||||
// Fetch counts the fetch and passes it on.
|
||||
func (f *countingFetcher) Fetch(
|
||||
ctx context.Context, source imagerequest.Source,
|
||||
ctx context.Context, url string,
|
||||
) (*httpfetcher.FetchResult, error) {
|
||||
f.fetches.Add(1)
|
||||
|
||||
return f.Fetcher.Fetch(ctx, source)
|
||||
return f.Fetcher.Fetch(ctx, url)
|
||||
}
|
||||
|
||||
// newRefererRoutes returns both image routes of a Handlers whose
|
||||
@@ -53,41 +47,33 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
|
||||
Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
|
||||
}
|
||||
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
|
||||
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
|
||||
StateDir: t.TempDir(),
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("imgcache.NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
svc, err := imageproxy.New(&imageproxy.Config{
|
||||
Cache: cache,
|
||||
Fetcher: fetcher,
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
}, prometheus.NewRegistry())
|
||||
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||
Cache: cache,
|
||||
Fetcher: fetcher,
|
||||
SigningKey: testSigningKey,
|
||||
Allowlist: []string{allowlistedHost},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("imageproxy.New() error = %v", err)
|
||||
t.Fatalf("imgcache.NewService() error = %v", err)
|
||||
}
|
||||
|
||||
keys, err := seal.DeriveKeys(testSigningKey)
|
||||
encGen, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("seal.DeriveKeys() error = %v", err)
|
||||
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
|
||||
|
||||
h := &Handlers{
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
imgSvc: svc,
|
||||
encGen: encGen,
|
||||
signer: signature.New(testSigningKey),
|
||||
allowlist: allowlist.New([]string{allowlistedHost}),
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
imgSvc: svc,
|
||||
encGen: encGen,
|
||||
refererBlocklist: allowlist.New(
|
||||
[]string{"leech.example", ".hotlinker.example"}),
|
||||
}
|
||||
|
||||
@@ -17,13 +17,11 @@ import (
|
||||
)
|
||||
|
||||
// signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the
|
||||
// JPEG at photoPath on signedHost at 50x50. It is signed with testSigningKey,
|
||||
// the key the handlers that newSignedHostServer builds check signatures with.
|
||||
// It takes a Handlers it does not use, so that it has the type of encPhotoURL.
|
||||
func signedPhotoURL(t *testing.T, _ *Handlers) string {
|
||||
// JPEG at photoPath on signedHost at 50x50, made with h's image service.
|
||||
func signedPhotoURL(t *testing.T, h *Handlers) string {
|
||||
t.Helper()
|
||||
|
||||
return generateSignedURL(testSigningKey, &imagerequest.Request{
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
@@ -33,6 +31,11 @@ func signedPhotoURL(t *testing.T, _ *Handlers) string {
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
return signedURL
|
||||
}
|
||||
|
||||
// encPhotoURL returns an encrypted /v1/e/ URL, which never expires, for the
|
||||
@@ -49,7 +52,7 @@ func encPhotoURL(t *testing.T, h *Handlers) string {
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Time{})
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -11,14 +11,13 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imageproxy"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// TestServerBusyAnswers503 checks that respondImageError, which answers the
|
||||
// errors of both image routes, answers 503 with a clear error when the image
|
||||
// service gives up waiting for a free upstream connection or processing slot,
|
||||
// wrapped as the service wraps them.
|
||||
// TestServerBusyAnswers503 checks that both image routes answer 503 with
|
||||
// a clear error when the image service gives up waiting for a free upstream
|
||||
// connection or processing slot, wrapped as the service wraps them.
|
||||
func TestServerBusyAnswers503(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -31,44 +30,61 @@ func TestServerBusyAnswers503(t *testing.T) {
|
||||
fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections),
|
||||
fmt.Errorf("image processing failed: %w", imageprocessor.ErrTooManyImages),
|
||||
} {
|
||||
rec := httptest.NewRecorder()
|
||||
h.respondImageError(rec, req, err)
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, req, err)
|
||||
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Errorf("for %v: status = %d, want %d",
|
||||
err, rec.Code, http.StatusServiceUnavailable)
|
||||
}
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
|
||||
t.Errorf("for %v: body = %q, want the server busy error",
|
||||
err, rec.Body.String())
|
||||
for route, rec := range map[string]*httptest.ResponseRecorder{
|
||||
"/v1/image/": plain, "/v1/e/": encrypted,
|
||||
} {
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Errorf("%s for %v: status = %d, want %d",
|
||||
route, err, rec.Code, http.StatusServiceUnavailable)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
|
||||
t.Errorf("%s for %v: body = %q, want the server busy error",
|
||||
route, err, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestProcessingTimedOutAnswers503 checks that respondImageError, which
|
||||
// answers the errors of both image routes, answers 503 with the server busy
|
||||
// error when the image service reports that processing stopped at the
|
||||
// deadline of the request doing it, wrapped as the service wraps it. A request
|
||||
// that waited for that processing gets this error while its own deadline may
|
||||
// have time left. The request's source only goes into a log line, so it is
|
||||
// left empty.
|
||||
// TestProcessingTimedOutAnswers503 checks that respondImageError and
|
||||
// handleImageError, which answer the errors of the two image routes, answer
|
||||
// 503 with the server busy error when the image service reports that
|
||||
// processing stopped at the deadline of the request doing it, wrapped as the
|
||||
// service wraps it. A request that waited for that processing gets this error
|
||||
// while its own deadline may have time left. The request's source only goes
|
||||
// into a log line, so it is left empty.
|
||||
func TestProcessingTimedOutAnswers503(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||
err := fmt.Errorf("%w: failed to encode: %w",
|
||||
imageproxy.ErrProcessingTimedOut, context.DeadlineExceeded)
|
||||
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.respondImageError(rec, &imagerequest.Request{}, err)
|
||||
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, &imagerequest.Request{}, err)
|
||||
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Errorf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
|
||||
}
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
|
||||
t.Errorf("body = %q, want the server busy error", rec.Body.String())
|
||||
for name, rec := range map[string]*httptest.ResponseRecorder{
|
||||
"respondImageError": plain, "handleImageError": encrypted,
|
||||
} {
|
||||
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
|
||||
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Errorf("%s: status = %d, want %d",
|
||||
name, rec.Code, http.StatusServiceUnavailable)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
|
||||
t.Errorf("%s: body = %q, want the server busy error",
|
||||
name, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -13,11 +13,10 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestUnsupportedSourceFormatAnswers415 checks that respondImageError, which
|
||||
// answers the errors of both image routes, answers 415 with a clear error when
|
||||
// the upstream answers with a content type pixa does not accept, such as SVG,
|
||||
// and when libvips does not read a source as the format its first bytes
|
||||
// show, wrapped as the service wraps them.
|
||||
// TestUnsupportedSourceFormatAnswers415 checks that both image routes answer
|
||||
// 415 with a clear error when the upstream answers with a content type pixa
|
||||
// does not accept, such as SVG, and when orig is asked of a source in a
|
||||
// format the processor cannot name, wrapped as the service wraps them.
|
||||
func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -29,21 +28,29 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
|
||||
for _, err := range []error{
|
||||
fmt.Errorf("upstream fetch failed: %w: image/svg+xml",
|
||||
httpfetcher.ErrInvalidContentType),
|
||||
fmt.Errorf("image processing failed: %w: libvips does not read the input as avif",
|
||||
fmt.Errorf("image processing failed: %w: unknown",
|
||||
imageprocessor.ErrUnsupportedInputFormat),
|
||||
} {
|
||||
rec := httptest.NewRecorder()
|
||||
h.respondImageError(rec, req, err)
|
||||
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, req, err)
|
||||
|
||||
if rec.Code != http.StatusUnsupportedMediaType {
|
||||
t.Errorf("for %v: status = %d, want %d",
|
||||
err, rec.Code, http.StatusUnsupportedMediaType)
|
||||
}
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
if !strings.Contains(rec.Body.String(), "unsupported source format") {
|
||||
t.Errorf("for %v: body = %q, want the unsupported source format error",
|
||||
err, rec.Body.String())
|
||||
for route, rec := range map[string]*httptest.ResponseRecorder{
|
||||
"HandleImage": plain, "HandleImageEnc": encrypted,
|
||||
} {
|
||||
t.Logf("%s for %v: %d %s", route, err, rec.Code, rec.Body.String())
|
||||
|
||||
if rec.Code != http.StatusUnsupportedMediaType {
|
||||
t.Errorf("%s for %v: status = %d, want %d",
|
||||
route, err, rec.Code, http.StatusUnsupportedMediaType)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), "unsupported source format") {
|
||||
t.Errorf("%s for %v: body = %q, want the unsupported source format error",
|
||||
route, err, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,211 +0,0 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// Addresses the port tests below put on the allowlist.
|
||||
const (
|
||||
allowlistedIPv4 = "203.0.113.5"
|
||||
allowlistedIPv6 = "2001:db8::1"
|
||||
)
|
||||
|
||||
// newSignatureChecker returns Handlers that check signatures with
|
||||
// testSigningKey and have allowlistHosts on their allowlist, as New builds
|
||||
// them from signing_key and allowlist_hosts.
|
||||
func newSignatureChecker(allowlistHosts ...string) *Handlers {
|
||||
return &Handlers{
|
||||
signer: signature.New(testSigningKey),
|
||||
allowlist: allowlist.New(allowlistHosts),
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateSignature_NonAllowlistedHost_NoSignature(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker(allowlistedHost)
|
||||
|
||||
// Should fail validation - not allowlisted and no signature
|
||||
err := h.validateSignature(photoRequest(signedHost), "", time.Time{})
|
||||
if err == nil {
|
||||
t.Error("validateSignature() expected error for non-allowlisted " +
|
||||
"host without signature")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateSignature_NonAllowlistedHost_ValidSignature(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker(allowlistedHost)
|
||||
req := photoRequest(signedHost)
|
||||
|
||||
// Generate a valid signature
|
||||
expires := time.Now().Add(time.Hour)
|
||||
sig := signature.New(testSigningKey).Sign(req, expires)
|
||||
|
||||
// Should pass validation
|
||||
err := h.validateSignature(req, sig, expires)
|
||||
if err != nil {
|
||||
t.Errorf("validateSignature() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateSignature_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker(allowlistedHost)
|
||||
req := photoRequest(signedHost)
|
||||
|
||||
// Generate an expired signature
|
||||
expires := time.Now().Add(-time.Hour)
|
||||
sig := signature.New(testSigningKey).Sign(req, expires)
|
||||
|
||||
// Should fail validation
|
||||
err := h.validateSignature(req, sig, expires)
|
||||
if err == nil {
|
||||
t.Error("validateSignature() expected error for expired signature")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateSignature_NonAllowlistedHost_InvalidSignature(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker(allowlistedHost)
|
||||
|
||||
// Should fail validation
|
||||
err := h.validateSignature(photoRequest(signedHost), "invalid-signature",
|
||||
time.Now().Add(time.Hour))
|
||||
if err == nil {
|
||||
t.Error("validateSignature() expected error for invalid signature")
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateSignature_NoAllowlist checks that with no host on the
|
||||
// allowlist, a request without a signature fails validation.
|
||||
func TestValidateSignature_NoAllowlist(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker()
|
||||
|
||||
err := h.validateSignature(photoRequest(signedHost), "", time.Time{})
|
||||
if err == nil {
|
||||
t.Error("validateSignature() expected error when no host is allowlisted")
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateSignature_SignatureExactHostMatch verifies that
|
||||
// validateSignature enforces exact host matching for signatures. A
|
||||
// signature for one host must not verify for a different host, even
|
||||
// if they share a domain suffix.
|
||||
func TestValidateSignature_SignatureExactHostMatch(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker()
|
||||
|
||||
// Sign a request for signedHost
|
||||
signedReq := photoRequest(signedHost)
|
||||
expires := time.Now().Add(time.Hour)
|
||||
sig := signature.New(testSigningKey).Sign(signedReq, expires)
|
||||
|
||||
// The original request should pass validation
|
||||
t.Run("exact host passes", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := h.validateSignature(signedReq, sig, expires)
|
||||
if err != nil {
|
||||
t.Errorf("validateSignature() exact host failed: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
// Try to reuse the signature with different hosts
|
||||
tests := []struct {
|
||||
name string
|
||||
host string
|
||||
}{
|
||||
{"parent domain", "example.com"},
|
||||
{"sibling subdomain", "images.example.com"},
|
||||
{"deeper subdomain", "a." + signedHost},
|
||||
{"evil suffix domain", signedHost + ".evil.com"},
|
||||
{"prefixed host", "evil" + signedHost},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name+" rejected", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := h.validateSignature(photoRequest(tt.host), sig, expires)
|
||||
if err == nil {
|
||||
t.Errorf("validateSignature() should reject signature for "+
|
||||
"host %q (signed for %q)", tt.host, signedHost)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateSignature_HostWithPortNeedsSignature verifies that a host
|
||||
// followed by a port, or by a colon with no port, matches no allowlist entry,
|
||||
// so that a request for it without a signature is refused, while the same
|
||||
// host with no port is allowlisted.
|
||||
func TestValidateSignature_HostWithPortNeedsSignature(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker(allowlistedHost, allowlistedIPv4, allowlistedIPv6)
|
||||
|
||||
tests := []struct {
|
||||
host string
|
||||
allowlisted bool
|
||||
}{
|
||||
{allowlistedHost, true},
|
||||
{allowlistedHost + ":8443", false},
|
||||
{allowlistedHost + ":443", false},
|
||||
{allowlistedHost + ":", false},
|
||||
{allowlistedIPv4, true},
|
||||
{allowlistedIPv4 + ":8080", false},
|
||||
{"[" + allowlistedIPv6 + "]", true},
|
||||
{"[" + allowlistedIPv6 + "]:8443", false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.host, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := h.validateSignature(photoRequest(tt.host), "", time.Time{})
|
||||
t.Logf("validateSignature() for host %q with no signature: %v",
|
||||
tt.host, err)
|
||||
|
||||
if allowlisted := err == nil; allowlisted != tt.allowlisted {
|
||||
t.Errorf("host %q served without a signature = %v, want %v",
|
||||
tt.host, allowlisted, tt.allowlisted)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateSignature_UnbracketedIPv6PortNeedsSignature verifies that an
|
||||
// allowlisted IPv6 address written without brackets and followed by a port,
|
||||
// or by a colon with no port, needs a signature. Go's URL type reads
|
||||
// 2001:db8::1:8443 as the host 2001:db8::1 with the port 8443.
|
||||
func TestValidateSignature_UnbracketedIPv6PortNeedsSignature(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newSignatureChecker(allowlistedIPv6)
|
||||
|
||||
for _, host := range []string{allowlistedIPv6 + ":8443", allowlistedIPv6 + ":"} {
|
||||
t.Run(host, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := h.validateSignature(photoRequest(host), "", time.Time{})
|
||||
t.Logf("validateSignature() for host %q with no signature: %v",
|
||||
host, err)
|
||||
|
||||
if err == nil {
|
||||
t.Errorf("host %q served without a signature, want refused",
|
||||
host)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestIsPrivateIPBlocksSpecialRanges covers the internal and special-use
|
||||
@@ -71,15 +70,12 @@ func transportOf(t *testing.T, f *HTTPFetcher) *http.Transport {
|
||||
func TestDialerEnforcesBlockedNetworks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
transport := transportOf(t, New(&Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
// TEST-NET-2 (198.51.100.0/24) is public to the built-in check, so
|
||||
// blocking it can only come from the operator-supplied list.
|
||||
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
|
||||
}))
|
||||
cfg := DefaultConfig()
|
||||
// TEST-NET-2 (198.51.100.0/24) is public to the built-in check, so
|
||||
// blocking it can only come from the operator-supplied list.
|
||||
cfg.BlockedNetworks = []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")}
|
||||
|
||||
transport := transportOf(t, New(cfg))
|
||||
|
||||
blocked := []string{
|
||||
"198.51.100.5:80", // operator-supplied range
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"errors"
|
||||
"net"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestNewUsesCheckedDialerWithoutDialContext checks that a fetcher built
|
||||
@@ -14,12 +13,7 @@ func TestNewUsesCheckedDialerWithoutDialContext(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
srv := startUpstream(t)
|
||||
transport := transportOf(t, New(&Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
}))
|
||||
transport := transportOf(t, New(DefaultConfig()))
|
||||
|
||||
addr := srv.Listener.Addr().String()
|
||||
|
||||
@@ -38,25 +32,21 @@ func TestDialContextReplacesOnlyTheDialer(t *testing.T) {
|
||||
srv := startUpstream(t)
|
||||
dialer := &recordingDialer{target: srv.Listener.Addr().String()}
|
||||
|
||||
f := New(&Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
AllowHTTP: true,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
DialContext: dialer.dialContext,
|
||||
})
|
||||
cfg := DefaultConfig()
|
||||
cfg.AllowHTTP = true
|
||||
cfg.DialContext = dialer.dialContext
|
||||
f := New(cfg)
|
||||
|
||||
if body := fetchBody(t, f, "/image"); body != imagePayload {
|
||||
t.Errorf("body = %q, want %q", body, imagePayload)
|
||||
}
|
||||
|
||||
_, err := f.Fetch(testContext(t), imageSourceOn("127.0.0.1"))
|
||||
_, err := f.Fetch(testContext(t), "http://127.0.0.1/image")
|
||||
if !errors.Is(err, ErrSSRFBlocked) {
|
||||
t.Errorf("Fetch(loopback URL) error = %v, want ErrSSRFBlocked", err)
|
||||
}
|
||||
|
||||
_, err = f.Fetch(testContext(t), upstreamSource("/redirect/private"))
|
||||
_, err = f.Fetch(testContext(t), upstreamURL("/redirect/private"))
|
||||
if !errors.Is(err, ErrSSRFBlocked) {
|
||||
t.Errorf("Fetch(/redirect/private) error = %v, want ErrSSRFBlocked", err)
|
||||
}
|
||||
|
||||
@@ -14,8 +14,6 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// testPublicHost is a TEST-NET-1 (RFC 5737) literal. isPrivateIP treats it as
|
||||
@@ -31,9 +29,9 @@ const imagePayload = "fake-jpeg-bytes"
|
||||
// would mean SSRF protection let a forbidden target reach the transport.
|
||||
var errUnexpectedDial = errors.New("unexpected dial target")
|
||||
|
||||
// upstreamSource returns the source at path on the fake public host.
|
||||
func upstreamSource(path string) imagerequest.Source {
|
||||
return imagerequest.Source{Host: testPublicHost, Path: path}
|
||||
// upstreamURL builds a fetch URL on the fake public host for the given path.
|
||||
func upstreamURL(path string) string {
|
||||
return "http://" + testPublicHost + path
|
||||
}
|
||||
|
||||
// recordingDialer records every address the transport asks it to dial and
|
||||
@@ -83,7 +81,7 @@ func startUpstream(t *testing.T) *httptest.Server {
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/image", func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
_, _ = io.WriteString(w, imagePayload)
|
||||
})
|
||||
mux.HandleFunc("/status/500", func(w http.ResponseWriter, _ *http.Request) {
|
||||
@@ -122,12 +120,7 @@ func newServerFetcher(
|
||||
t.Helper()
|
||||
|
||||
if cfg == nil {
|
||||
cfg = &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
}
|
||||
cfg = DefaultConfig()
|
||||
}
|
||||
|
||||
cfg.AllowHTTP = true
|
||||
@@ -160,7 +153,7 @@ func testContext(t *testing.T) context.Context {
|
||||
func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
|
||||
t.Helper()
|
||||
|
||||
res, err := f.Fetch(testContext(t), upstreamSource(path))
|
||||
res, err := f.Fetch(testContext(t), upstreamURL(path))
|
||||
if err != nil {
|
||||
t.Fatalf("Fetch(%s) error = %v", path, err)
|
||||
}
|
||||
@@ -172,7 +165,7 @@ func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
|
||||
func fetchExpectError(t *testing.T, f *HTTPFetcher, path string) error {
|
||||
t.Helper()
|
||||
|
||||
res, err := f.Fetch(testContext(t), upstreamSource(path))
|
||||
res, err := f.Fetch(testContext(t), upstreamURL(path))
|
||||
if err == nil {
|
||||
_ = res.Content.Close()
|
||||
|
||||
@@ -224,7 +217,7 @@ func TestFetchRedirectToPrivateIPBlocked(t *testing.T) {
|
||||
srv := startUpstream(t)
|
||||
f, dialer := newServerFetcher(t, srv, nil)
|
||||
|
||||
_, err := f.Fetch(testContext(t), upstreamSource("/redirect/private"))
|
||||
_, err := f.Fetch(testContext(t), upstreamURL("/redirect/private"))
|
||||
if !errors.Is(err, ErrSSRFBlocked) {
|
||||
t.Fatalf("Fetch() error = %v, want ErrSSRFBlocked", err)
|
||||
}
|
||||
@@ -287,12 +280,8 @@ func TestFetchMaxResponseSizeEnforced(t *testing.T) {
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 8,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxResponseSize = 8
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
|
||||
@@ -321,12 +310,8 @@ func fetchWithMaxResponseSize(
|
||||
srv := httptest.NewServer(handler)
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: maxResponseSize,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxResponseSize = maxResponseSize
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
|
||||
@@ -358,7 +343,7 @@ func TestFetchContentLengthBodyAtMaxResponseSize(t *testing.T) {
|
||||
|
||||
res := fetchWithMaxResponseSize(t, maxResponseSize,
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
w.Header().Set("Content-Length", strconv.Itoa(tc.size))
|
||||
_, _ = io.WriteString(w, body)
|
||||
})
|
||||
@@ -406,7 +391,7 @@ func TestFetchChunkedBodyAtMaxResponseSize(t *testing.T) {
|
||||
|
||||
res := fetchWithMaxResponseSize(t, maxResponseSize,
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
_, _ = io.WriteString(w, body)
|
||||
// With no Content-Length, flushing sends the body
|
||||
// chunked; its end is sent when the handler returns.
|
||||
@@ -451,12 +436,8 @@ func TestFetchSemaphoreReleasedOnError(t *testing.T) {
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 1,
|
||||
MaxConnections: 64,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxConnectionsPerHost = 1
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
|
||||
@@ -485,12 +466,8 @@ func assertSlotReleasedByClose(
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 1,
|
||||
MaxConnections: 64,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxConnectionsPerHost = 1
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
|
||||
@@ -547,7 +524,7 @@ func TestFetchSemaphoreReleasedOnPartialReadClose(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
// The dial-time re-resolution in dialSSRFSafe is what closes the DNS
|
||||
// The dial-time re-resolution in ssrfSafeDialer is what closes the DNS
|
||||
// rebinding window: even if validateURL saw a public answer earlier, the
|
||||
// dialer independently re-checks the address it is about to connect to. A full
|
||||
// rebinding simulation (a resolver returning public, then private) would mean
|
||||
@@ -567,9 +544,9 @@ func TestSSRFSafeDialerBlocksPrivateTarget(t *testing.T) {
|
||||
t.Run(addr, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := dialSSRFSafe(context.Background(), "tcp", addr, nil)
|
||||
_, err := ssrfSafeDialer(context.Background(), "tcp", addr)
|
||||
if !errors.Is(err, ErrSSRFBlocked) {
|
||||
t.Errorf("dialSSRFSafe(%q) = %v, want ErrSSRFBlocked", addr, err)
|
||||
t.Errorf("ssrfSafeDialer(%q) = %v, want ErrSSRFBlocked", addr, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -583,7 +560,7 @@ func TestSSRFSafeDialerAllowsPublicTarget(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
_, err := dialSSRFSafe(ctx, "tcp", testPublicHost+":80", nil)
|
||||
_, err := ssrfSafeDialer(ctx, "tcp", testPublicHost+":80")
|
||||
if err == nil {
|
||||
t.Fatal("expected a dial error for an unreachable public target")
|
||||
}
|
||||
@@ -592,53 +569,3 @@ func TestSSRFSafeDialerAllowsPublicTarget(t *testing.T) {
|
||||
t.Errorf("public target was SSRF-blocked: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFetchSchemeFollowsAllowHTTP checks that the fetcher fetches a source
|
||||
// over https, on port 443, unless AllowHTTP is set, and then over http, on
|
||||
// port 80. The test server speaks plain HTTP, so only the fetch over http
|
||||
// succeeds.
|
||||
func TestFetchSchemeFollowsAllowHTTP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
allowHTTP bool
|
||||
wantAddr string
|
||||
}{
|
||||
{false, net.JoinHostPort(testPublicHost, "443")},
|
||||
{true, net.JoinHostPort(testPublicHost, "80")},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(fmt.Sprintf("AllowHTTP %v", tt.allowHTTP), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
srv := startUpstream(t)
|
||||
dialer := &recordingDialer{target: srv.Listener.Addr().String()}
|
||||
|
||||
f := New(&Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
AllowHTTP: tt.allowHTTP,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
DialContext: dialer.dialContext,
|
||||
})
|
||||
|
||||
res, err := f.Fetch(testContext(t), upstreamSource("/image"))
|
||||
if err == nil {
|
||||
_ = res.Content.Close()
|
||||
}
|
||||
|
||||
dialed := dialer.dialedAddrs()
|
||||
t.Logf("Fetch(): error %v, dialed %v", err, dialed)
|
||||
|
||||
if served := err == nil; served != tt.allowHTTP {
|
||||
t.Errorf("served = %v, want %v", served, tt.allowHTTP)
|
||||
}
|
||||
|
||||
if len(dialed) != 1 || dialed[0] != tt.wantAddr {
|
||||
t.Errorf("dialed %v, want only %s", dialed, tt.wantAddr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,24 +20,36 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Fetcher configuration constants.
|
||||
const (
|
||||
DefaultTLSTimeout = 10 * time.Second
|
||||
DefaultMaxIdleConns = 100
|
||||
DefaultIdleConnTimeout = 90 * time.Second
|
||||
DefaultMaxRedirects = 10
|
||||
DefaultFetchTimeout = 30 * time.Second
|
||||
DefaultMaxResponseSize = 50 << 20 // 50MB
|
||||
DefaultTLSTimeout = 10 * time.Second
|
||||
DefaultMaxIdleConns = 100
|
||||
DefaultIdleConnTimeout = 90 * time.Second
|
||||
DefaultMaxRedirects = 10
|
||||
DefaultMaxConnectionsPerHost = 20
|
||||
DefaultMaxConnections = 64
|
||||
)
|
||||
|
||||
// userAgent is the User-Agent header sent to upstream servers.
|
||||
const userAgent = "pixa/1.0"
|
||||
|
||||
// ConnectionWaitTimeout is how long Fetch waits for a free connection when
|
||||
// MaxConnections fetches are already in progress.
|
||||
const ConnectionWaitTimeout = 10 * time.Second
|
||||
|
||||
// MIME content types.
|
||||
const (
|
||||
contentTypeJPEG = "image/jpeg"
|
||||
contentTypePNG = "image/png"
|
||||
contentTypeGIF = "image/gif"
|
||||
contentTypeWebP = "image/webp"
|
||||
contentTypeAVIF = "image/avif"
|
||||
contentTypeJXL = "image/jxl"
|
||||
contentTypeSVG = "image/svg+xml"
|
||||
contentTypeOctetStream = "application/octet-stream"
|
||||
)
|
||||
|
||||
// Loopback addresses blocked by SSRF protection.
|
||||
const (
|
||||
localhostIPv4 = "127.0.0.1"
|
||||
@@ -80,8 +92,8 @@ var (
|
||||
|
||||
// Fetcher retrieves content from upstream origins.
|
||||
type Fetcher interface {
|
||||
// Fetch retrieves the content of the source image.
|
||||
Fetch(ctx context.Context, source imagerequest.Source) (*FetchResult, error)
|
||||
// Fetch retrieves content from the given URL.
|
||||
Fetch(ctx context.Context, url string) (*FetchResult, error)
|
||||
}
|
||||
|
||||
// FetchResult contains the result of fetching from upstream.
|
||||
@@ -108,29 +120,17 @@ type FetchResult struct {
|
||||
TLSCipherSuite string
|
||||
}
|
||||
|
||||
// allowedContentTypes returns the content types an upstream response may
|
||||
// have, which Fetch also sends as its Accept header: the MIME types of the
|
||||
// formats pixa reads.
|
||||
func allowedContentTypes() []string {
|
||||
return []string{
|
||||
imagerequest.FormatJPEG.MIMEType(),
|
||||
imagerequest.FormatPNG.MIMEType(),
|
||||
imagerequest.FormatGIF.MIMEType(),
|
||||
imagerequest.FormatWebP.MIMEType(),
|
||||
imagerequest.FormatAVIF.MIMEType(),
|
||||
imagerequest.FormatJXL.MIMEType(),
|
||||
}
|
||||
}
|
||||
|
||||
// Config holds configuration for the upstream fetcher. Every field but
|
||||
// AllowHTTP, BlockedNetworks and DialContext must be set.
|
||||
// Config holds configuration for the upstream fetcher.
|
||||
type Config struct {
|
||||
// Timeout for upstream requests.
|
||||
Timeout time.Duration
|
||||
// MaxResponseSize is the maximum allowed response body size.
|
||||
MaxResponseSize int64
|
||||
// AllowHTTP makes the fetcher fetch sources over http instead of https,
|
||||
// and follow redirects to http (for testing only).
|
||||
// UserAgent to send to upstream servers.
|
||||
UserAgent string
|
||||
// AllowedContentTypes is an allow list of MIME types to accept.
|
||||
AllowedContentTypes []string
|
||||
// AllowHTTP allows non-TLS connections (for testing only).
|
||||
AllowHTTP bool
|
||||
// MaxConnectionsPerHost limits concurrent connections to each upstream host.
|
||||
MaxConnectionsPerHost int
|
||||
@@ -147,6 +147,26 @@ type Config struct {
|
||||
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
|
||||
}
|
||||
|
||||
// DefaultConfig returns a Config with sensible defaults.
|
||||
func DefaultConfig() *Config {
|
||||
return &Config{
|
||||
Timeout: DefaultFetchTimeout,
|
||||
MaxResponseSize: DefaultMaxResponseSize,
|
||||
UserAgent: "pixa/1.0",
|
||||
AllowedContentTypes: []string{
|
||||
contentTypeJPEG,
|
||||
contentTypePNG,
|
||||
contentTypeGIF,
|
||||
contentTypeWebP,
|
||||
contentTypeAVIF,
|
||||
contentTypeJXL,
|
||||
},
|
||||
AllowHTTP: false,
|
||||
MaxConnectionsPerHost: DefaultMaxConnectionsPerHost,
|
||||
MaxConnections: DefaultMaxConnections,
|
||||
}
|
||||
}
|
||||
|
||||
// HTTPFetcher implements Fetcher with SSRF protection and connection limits
|
||||
// per host and for all hosts together.
|
||||
type HTTPFetcher struct {
|
||||
@@ -174,6 +194,10 @@ type hostSemaphore struct {
|
||||
|
||||
// New creates a new HTTPFetcher with SSRF protection.
|
||||
func New(config *Config) *HTTPFetcher {
|
||||
if config == nil {
|
||||
config = DefaultConfig()
|
||||
}
|
||||
|
||||
// Unless config.DialContext replaces it, the transport connects with
|
||||
// the SSRF-safe dialer, which re-resolves and re-checks at connect time
|
||||
// (closing the DNS-rebinding window) against both the built-in ranges
|
||||
@@ -202,7 +226,7 @@ func New(config *Config) *HTTPFetcher {
|
||||
}
|
||||
|
||||
// Validate the redirect target
|
||||
err := validateURL(req.Context(), req.URL, config.AllowHTTP)
|
||||
err := validateURL(req.Context(), req.URL.String(), config.AllowHTTP)
|
||||
if err != nil {
|
||||
return fmt.Errorf("redirect blocked: %w", err)
|
||||
}
|
||||
@@ -220,22 +244,18 @@ func New(config *Config) *HTTPFetcher {
|
||||
}
|
||||
}
|
||||
|
||||
// Fetch retrieves the source image with SSRF protection, over https, or over
|
||||
// http when AllowHTTP is set. When MaxConnections fetches are already in
|
||||
// progress, it waits up to ConnectionWaitTimeout for one to finish, then
|
||||
// fails with ErrTooManyConnections.
|
||||
func (f *HTTPFetcher) Fetch(
|
||||
ctx context.Context, source imagerequest.Source,
|
||||
) (*FetchResult, error) {
|
||||
sourceURL := source.URL(f.config.AllowHTTP)
|
||||
|
||||
// Fetch retrieves content from the given URL with SSRF protection. When
|
||||
// MaxConnections fetches are already in progress, it waits up to
|
||||
// ConnectionWaitTimeout for one to finish, then fails with
|
||||
// ErrTooManyConnections.
|
||||
func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, error) {
|
||||
// Validate URL before making request
|
||||
err := validateURL(ctx, sourceURL, f.config.AllowHTTP)
|
||||
err := validateURL(ctx, url, f.config.AllowHTTP)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
release, err := f.acquireConnection(ctx, source.Host)
|
||||
release, err := f.acquireConnection(ctx, extractHost(url))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -249,14 +269,19 @@ func (f *HTTPFetcher) Fetch(
|
||||
}
|
||||
}()
|
||||
|
||||
parsedURL, err := neturl.Parse(url)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to parse URL: %w", err)
|
||||
}
|
||||
|
||||
req := &http.Request{
|
||||
Method: http.MethodGet,
|
||||
URL: sourceURL,
|
||||
URL: parsedURL,
|
||||
Header: make(http.Header),
|
||||
}
|
||||
|
||||
req.Header.Set("User-Agent", userAgent)
|
||||
req.Header.Set("Accept", strings.Join(allowedContentTypes(), ", "))
|
||||
req.Header.Set("User-Agent", f.config.UserAgent)
|
||||
req.Header.Set("Accept", strings.Join(f.config.AllowedContentTypes, ", "))
|
||||
|
||||
// The ID of the request this fetch serves, so the fetch can be found in
|
||||
// the upstream host's logs
|
||||
@@ -435,7 +460,7 @@ func (f *HTTPFetcher) isAllowedContentType(contentType string) bool {
|
||||
// Extract the MIME type without parameters
|
||||
mediaType := strings.TrimSpace(strings.Split(contentType, ";")[0])
|
||||
|
||||
for _, allowed := range allowedContentTypes() {
|
||||
for _, allowed := range f.config.AllowedContentTypes {
|
||||
if strings.EqualFold(mediaType, allowed) {
|
||||
return true
|
||||
}
|
||||
@@ -445,14 +470,19 @@ func (f *HTTPFetcher) isAllowedContentType(contentType string) bool {
|
||||
}
|
||||
|
||||
// validateURL checks if a URL is safe to fetch (not internal/private).
|
||||
func validateURL(ctx context.Context, fetchURL *neturl.URL, allowHTTP bool) error {
|
||||
if !allowHTTP && fetchURL.Scheme != "https" {
|
||||
func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
|
||||
if !allowHTTP && !strings.HasPrefix(rawURL, "https://") {
|
||||
return ErrUnsupportedScheme
|
||||
}
|
||||
|
||||
parsedURL, err := neturl.Parse(rawURL)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
|
||||
}
|
||||
|
||||
// Hostname drops any port, and the brackets of an IPv6 address whether
|
||||
// or not a port follows them
|
||||
host := fetchURL.Hostname()
|
||||
host := parsedURL.Hostname()
|
||||
if host == "" {
|
||||
return ErrInvalidHost
|
||||
}
|
||||
@@ -478,6 +508,25 @@ func validateURL(ctx context.Context, fetchURL *neturl.URL, allowHTTP bool) erro
|
||||
return nil
|
||||
}
|
||||
|
||||
// extractHost extracts the host from a URL string.
|
||||
func extractHost(rawURL string) string {
|
||||
// Simple extraction without full URL parsing
|
||||
url := rawURL
|
||||
if idx := strings.Index(url, "://"); idx != -1 {
|
||||
url = url[idx+3:]
|
||||
}
|
||||
|
||||
if idx := strings.Index(url, "/"); idx != -1 {
|
||||
url = url[:idx]
|
||||
}
|
||||
|
||||
if idx := strings.Index(url, "?"); idx != -1 {
|
||||
url = url[:idx]
|
||||
}
|
||||
|
||||
return url
|
||||
}
|
||||
|
||||
// isLocalhost checks if the host is localhost.
|
||||
func isLocalhost(host string) bool {
|
||||
host = strings.ToLower(host)
|
||||
@@ -590,6 +639,13 @@ func ipv4CompatibleToIPv4(ip net.IP) net.IP {
|
||||
return ip
|
||||
}
|
||||
|
||||
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges
|
||||
// before connecting. New wraps dialSSRFSafe with the operator-supplied
|
||||
// blocklist; this entry point enforces the built-in ranges alone.
|
||||
func ssrfSafeDialer(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||
return dialSSRFSafe(ctx, network, addr, nil)
|
||||
}
|
||||
|
||||
// dialSSRFSafe re-resolves addr and refuses to connect to any built-in
|
||||
// internal range or operator-supplied blocked prefix, closing the
|
||||
// DNS-rebinding window at connect time.
|
||||
|
||||
@@ -5,34 +5,70 @@ import (
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/url"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// testHost is the hostname used by mock fetch tests.
|
||||
const testHost = "example.com"
|
||||
|
||||
func TestDefaultConfig(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := DefaultConfig()
|
||||
|
||||
if cfg.Timeout != DefaultFetchTimeout {
|
||||
t.Errorf("Timeout = %v, want %v", cfg.Timeout, DefaultFetchTimeout)
|
||||
}
|
||||
|
||||
if cfg.MaxResponseSize != DefaultMaxResponseSize {
|
||||
t.Errorf("MaxResponseSize = %d, want %d", cfg.MaxResponseSize, DefaultMaxResponseSize)
|
||||
}
|
||||
|
||||
if cfg.MaxConnectionsPerHost != DefaultMaxConnectionsPerHost {
|
||||
t.Errorf("MaxConnectionsPerHost = %d, want %d",
|
||||
cfg.MaxConnectionsPerHost, DefaultMaxConnectionsPerHost)
|
||||
}
|
||||
|
||||
if cfg.AllowHTTP {
|
||||
t.Error("AllowHTTP should default to false")
|
||||
}
|
||||
|
||||
if len(cfg.AllowedContentTypes) == 0 {
|
||||
t.Error("AllowedContentTypes should not be empty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewWithNilConfigUsesDefaults(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := New(nil)
|
||||
|
||||
if f == nil {
|
||||
t.Fatal("New(nil) returned nil")
|
||||
}
|
||||
|
||||
if f.config == nil {
|
||||
t.Fatal("config should be populated from DefaultConfig")
|
||||
}
|
||||
|
||||
if f.config.Timeout != DefaultFetchTimeout {
|
||||
t.Errorf("Timeout = %v, want %v", f.config.Timeout, DefaultFetchTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsAllowedContentType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := New(&Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
})
|
||||
f := New(DefaultConfig())
|
||||
|
||||
tests := []struct {
|
||||
contentType string
|
||||
want bool
|
||||
}{
|
||||
{imagerequest.FormatJPEG.MIMEType(), true},
|
||||
{imagerequest.FormatPNG.MIMEType(), true},
|
||||
{imagerequest.FormatWebP.MIMEType(), true},
|
||||
{contentTypeJPEG, true},
|
||||
{contentTypePNG, true},
|
||||
{contentTypeWebP, true},
|
||||
{"image/jpeg; charset=utf-8", true},
|
||||
{"IMAGE/JPEG", true},
|
||||
{"text/html", false},
|
||||
@@ -52,6 +88,33 @@ func TestIsAllowedContentType(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractHost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
url string
|
||||
want string
|
||||
}{
|
||||
{"https://example.com/path", testHost},
|
||||
{"http://example.com:8080/path", "example.com:8080"},
|
||||
{"https://example.com", testHost},
|
||||
{"https://example.com?q=1", testHost},
|
||||
{"example.com/path", testHost},
|
||||
{"", ""},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.url, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := extractHost(tc.url)
|
||||
if got != tc.want {
|
||||
t.Errorf("extractHost(%q) = %q, want %q", tc.url, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsLocalhost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -164,7 +227,7 @@ func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
|
||||
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := validateURL(t.Context(), &url.URL{Scheme: "http", Host: testHost}, false)
|
||||
err := validateURL(t.Context(), "http://example.com/path", false)
|
||||
if !errors.Is(err, ErrUnsupportedScheme) {
|
||||
t.Errorf("validateURL http = %v, want ErrUnsupportedScheme", err)
|
||||
}
|
||||
@@ -174,8 +237,7 @@ func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Use a host that won't resolve (explicit .invalid TLD) so we don't hit DNS.
|
||||
err := validateURL(t.Context(),
|
||||
&url.URL{Scheme: "http", Host: "nonexistent.invalid"}, true)
|
||||
err := validateURL(t.Context(), "http://nonexistent.invalid/path", true)
|
||||
// We expect a host resolution error, not ErrUnsupportedScheme.
|
||||
if errors.Is(err, ErrUnsupportedScheme) {
|
||||
t.Error("validateURL with AllowHTTP should not return ErrUnsupportedScheme")
|
||||
@@ -185,7 +247,7 @@ func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
|
||||
func TestValidateURL_RejectsLocalhost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := validateURL(t.Context(), &url.URL{Scheme: "https", Host: "localhost"}, false)
|
||||
err := validateURL(t.Context(), "https://localhost/path", false)
|
||||
if !errors.Is(err, ErrSSRFBlocked) {
|
||||
t.Errorf("validateURL localhost = %v, want ErrSSRFBlocked", err)
|
||||
}
|
||||
@@ -194,7 +256,7 @@ func TestValidateURL_RejectsLocalhost(t *testing.T) {
|
||||
func TestValidateURL_EmptyHost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := validateURL(t.Context(), &url.URL{Scheme: "https"}, false)
|
||||
err := validateURL(t.Context(), "https:///path", false)
|
||||
if !errors.Is(err, ErrInvalidHost) {
|
||||
t.Errorf("validateURL empty host = %v, want ErrInvalidHost", err)
|
||||
}
|
||||
@@ -209,15 +271,14 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
|
||||
|
||||
m := NewMock(mockFS)
|
||||
|
||||
result, err := m.Fetch(context.Background(),
|
||||
imagerequest.Source{Host: testHost, Path: "/images/photo.jpg"})
|
||||
result, err := m.Fetch(context.Background(), "https://example.com/images/photo.jpg")
|
||||
if err != nil {
|
||||
t.Fatalf("Fetch() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
if result.ContentType != imagerequest.FormatJPEG.MIMEType() {
|
||||
if result.ContentType != contentTypeJPEG {
|
||||
t.Errorf("ContentType = %q, want image/jpeg", result.ContentType)
|
||||
}
|
||||
|
||||
@@ -241,8 +302,7 @@ func TestMockFetcher_MissingFileReturnsUpstreamError(t *testing.T) {
|
||||
mockFS := fstest.MapFS{}
|
||||
m := NewMock(mockFS)
|
||||
|
||||
_, err := m.Fetch(context.Background(),
|
||||
imagerequest.Source{Host: testHost, Path: "/missing.jpg"})
|
||||
_, err := m.Fetch(context.Background(), "https://example.com/missing.jpg")
|
||||
if !errors.Is(err, ErrUpstreamError) {
|
||||
t.Errorf("Fetch() error = %v, want ErrUpstreamError", err)
|
||||
}
|
||||
@@ -259,7 +319,7 @@ func TestMockFetcher_RespectsContextCancellation(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
_, err := m.Fetch(ctx, imagerequest.Source{Host: testHost, Path: "/photo.jpg"})
|
||||
_, err := m.Fetch(ctx, "https://example.com/photo.jpg")
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("Fetch() error = %v, want context.Canceled", err)
|
||||
}
|
||||
@@ -272,13 +332,13 @@ func TestDetectContentTypeFromPath(t *testing.T) {
|
||||
path string
|
||||
want string
|
||||
}{
|
||||
{"foo/bar.jpg", imagerequest.FormatJPEG.MIMEType()},
|
||||
{"foo/bar.JPG", imagerequest.FormatJPEG.MIMEType()},
|
||||
{"foo/bar.jpeg", imagerequest.FormatJPEG.MIMEType()},
|
||||
{"foo/bar.png", imagerequest.FormatPNG.MIMEType()},
|
||||
{"foo/bar.gif", imagerequest.FormatGIF.MIMEType()},
|
||||
{"foo/bar.webp", imagerequest.FormatWebP.MIMEType()},
|
||||
{"foo/bar.avif", imagerequest.FormatAVIF.MIMEType()},
|
||||
{"foo/bar.jpg", contentTypeJPEG},
|
||||
{"foo/bar.JPG", contentTypeJPEG},
|
||||
{"foo/bar.jpeg", contentTypeJPEG},
|
||||
{"foo/bar.png", contentTypePNG},
|
||||
{"foo/bar.gif", contentTypeGIF},
|
||||
{"foo/bar.webp", contentTypeWebP},
|
||||
{"foo/bar.avif", contentTypeAVIF},
|
||||
{"foo/bar.svg", contentTypeSVG},
|
||||
{"foo/bar.bin", contentTypeOctetStream},
|
||||
{"foo/bar", contentTypeOctetStream},
|
||||
|
||||
@@ -1,26 +1,16 @@
|
||||
package httpfetcher
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
import "testing"
|
||||
|
||||
// TestJPEGXLContentType verifies that the fetcher accepts an upstream answer
|
||||
// of type image/jxl, and that the mock fetcher serves a .jxl file as
|
||||
// image/jxl.
|
||||
// of type image/jxl by default, and that the mock fetcher serves a .jxl file
|
||||
// as image/jxl.
|
||||
func TestJPEGXLContentType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const jxlType = "image/jxl"
|
||||
|
||||
f := New(&Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 64,
|
||||
})
|
||||
|
||||
if !f.isAllowedContentType(jxlType) {
|
||||
if !New(DefaultConfig()).isAllowedContentType(jxlType) {
|
||||
t.Errorf("isAllowedContentType(%q) = false, want true", jxlType)
|
||||
}
|
||||
|
||||
|
||||
@@ -8,20 +8,22 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// imageSourceOn is the fake upstream's image route on host.
|
||||
func imageSourceOn(host string) imagerequest.Source {
|
||||
return imagerequest.Source{Host: host, Path: "/image"}
|
||||
}
|
||||
|
||||
// imageSourceOnPort is the fake upstream's image route on testPublicHost at
|
||||
// imageURLOnPort is the fake upstream's image route on testPublicHost at
|
||||
// port. Each port is a different host to the per-host limit, while the test
|
||||
// dialer sends every port to the one test server.
|
||||
func imageSourceOnPort(port int) imagerequest.Source {
|
||||
return imageSourceOn(net.JoinHostPort(testPublicHost, strconv.Itoa(port)))
|
||||
func imageURLOnPort(port int) string {
|
||||
return "http://" + net.JoinHostPort(testPublicHost, strconv.Itoa(port)) +
|
||||
"/image"
|
||||
}
|
||||
|
||||
func TestDefaultConfigMaxConnections(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
if got := DefaultConfig().MaxConnections; got != DefaultMaxConnections {
|
||||
t.Errorf("MaxConnections = %d, want %d", got, DefaultMaxConnections)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFetchLimitsConnectionsToAllHostsTogether checks that MaxConnections
|
||||
@@ -35,22 +37,18 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 2,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxConnections = 2
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
f.connectionWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
first, err := f.Fetch(testContext(t), imageSourceOnPort(81))
|
||||
first, err := f.Fetch(testContext(t), imageURLOnPort(81))
|
||||
if err != nil {
|
||||
t.Fatalf("first Fetch() error = %v", err)
|
||||
}
|
||||
|
||||
second, err := f.Fetch(testContext(t), imageSourceOnPort(82))
|
||||
second, err := f.Fetch(testContext(t), imageURLOnPort(82))
|
||||
if err != nil {
|
||||
t.Fatalf("second Fetch() error = %v", err)
|
||||
}
|
||||
@@ -59,7 +57,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
|
||||
|
||||
start := time.Now()
|
||||
|
||||
_, err = f.Fetch(testContext(t), imageSourceOnPort(83))
|
||||
_, err = f.Fetch(testContext(t), imageURLOnPort(83))
|
||||
if !errors.Is(err, ErrTooManyConnections) {
|
||||
t.Fatalf("third Fetch() error = %v, want ErrTooManyConnections", err)
|
||||
}
|
||||
@@ -78,7 +76,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
|
||||
t.Fatalf("close first body: %v", err)
|
||||
}
|
||||
|
||||
third, err := f.Fetch(testContext(t), imageSourceOnPort(83))
|
||||
third, err := f.Fetch(testContext(t), imageURLOnPort(83))
|
||||
if err != nil {
|
||||
t.Fatalf("Fetch() after a response was closed: error = %v", err)
|
||||
}
|
||||
@@ -96,16 +94,12 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 1,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxConnections = 1
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
|
||||
first, err := f.Fetch(testContext(t), imageSourceOnPort(81))
|
||||
first, err := f.Fetch(testContext(t), imageURLOnPort(81))
|
||||
if err != nil {
|
||||
t.Fatalf("first Fetch() error = %v", err)
|
||||
}
|
||||
@@ -115,7 +109,7 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err = f.Fetch(ctx, imageSourceOnPort(82))
|
||||
_, err = f.Fetch(ctx, imageURLOnPort(82))
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("second Fetch() error = %v, want context.DeadlineExceeded", err)
|
||||
}
|
||||
@@ -140,7 +134,7 @@ func TestFetchRemovesIdleHostSemaphores(t *testing.T) {
|
||||
|
||||
for i := range 100 {
|
||||
wg.Go(func() {
|
||||
res, err := f.Fetch(ctx, imageSourceOnPort(1+i%50))
|
||||
res, err := f.Fetch(ctx, imageURLOnPort(1+i%50))
|
||||
if err != nil {
|
||||
t.Errorf("Fetch() error = %v", err)
|
||||
|
||||
@@ -169,22 +163,19 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 1,
|
||||
MaxConnections: 1,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxConnections = 1
|
||||
cfg.MaxConnectionsPerHost = 1
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
f.connectionWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
open, err := f.Fetch(testContext(t), imageSourceOnPort(81))
|
||||
open, err := f.Fetch(testContext(t), imageURLOnPort(81))
|
||||
if err != nil {
|
||||
t.Fatalf("first Fetch() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = f.Fetch(testContext(t), imageSourceOnPort(82))
|
||||
_, err = f.Fetch(testContext(t), imageURLOnPort(82))
|
||||
if !errors.Is(err, ErrTooManyConnections) {
|
||||
t.Fatalf("Fetch() from another host: error = %v, "+
|
||||
"want ErrTooManyConnections", err)
|
||||
@@ -193,7 +184,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err = f.Fetch(ctx, imageSourceOnPort(81))
|
||||
_, err = f.Fetch(ctx, imageURLOnPort(81))
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("Fetch() from the busy host: error = %v, "+
|
||||
"want context.DeadlineExceeded", err)
|
||||
@@ -205,7 +196,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
|
||||
ctx, cancel = context.WithTimeout(t.Context(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err = f.Fetch(ctx, imageSourceOnPort(83))
|
||||
_, err = f.Fetch(ctx, imageURLOnPort(83))
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("Fetch() from a host with nothing open: error = %v, "+
|
||||
"want context.DeadlineExceeded", err)
|
||||
@@ -228,15 +219,15 @@ func TestFetchReleasesConnectionOnError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
source imagerequest.Source
|
||||
want error
|
||||
name string
|
||||
url string
|
||||
want error
|
||||
}{
|
||||
{"upstream answers 500", upstreamSource("/status/500"), ErrUpstreamError},
|
||||
{"upstream sends HTML", upstreamSource("/html"), ErrInvalidContentType},
|
||||
{"upstream answers 500", upstreamURL("/status/500"), ErrUpstreamError},
|
||||
{"upstream sends HTML", upstreamURL("/html"), ErrInvalidContentType},
|
||||
// 198.51.100.7 (TEST-NET-2) passes the SSRF checks, and the test
|
||||
// dialer refuses every host but testPublicHost.
|
||||
{"connecting fails", imageSourceOn("198.51.100.7"), errUnexpectedDial},
|
||||
{"connecting fails", "http://198.51.100.7/image", errUnexpectedDial},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
@@ -245,17 +236,13 @@ func TestFetchReleasesConnectionOnError(t *testing.T) {
|
||||
|
||||
srv := startUpstream(t)
|
||||
|
||||
cfg := &Config{
|
||||
Timeout: 30 * time.Second,
|
||||
MaxResponseSize: 50 << 20,
|
||||
MaxConnectionsPerHost: 20,
|
||||
MaxConnections: 1,
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxConnections = 1
|
||||
|
||||
f, _ := newServerFetcher(t, srv, cfg)
|
||||
f.connectionWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
_, err := f.Fetch(testContext(t), tc.source)
|
||||
_, err := f.Fetch(testContext(t), tc.url)
|
||||
if !errors.Is(err, tc.want) {
|
||||
t.Fatalf("Fetch() error = %v, want %v", err, tc.want)
|
||||
}
|
||||
|
||||
@@ -7,14 +7,14 @@ import (
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// errEmptyURLPath is returned when a mock URL has no usable path.
|
||||
var errEmptyURLPath = errors.New("empty URL path")
|
||||
|
||||
// MockFetcher implements Fetcher using an embedded filesystem.
|
||||
// Files are organized as: hostname/path/to/file.ext
|
||||
// The source with host example.com and path /images/photo.jpg maps to
|
||||
// example.com/images/photo.jpg; the source query is ignored.
|
||||
// URLs like https://example.com/images/photo.jpg map to example.com/images/photo.jpg.
|
||||
type MockFetcher struct {
|
||||
fs fs.FS
|
||||
}
|
||||
@@ -24,10 +24,8 @@ func NewMock(fsys fs.FS) *MockFetcher {
|
||||
return &MockFetcher{fs: fsys}
|
||||
}
|
||||
|
||||
// Fetch retrieves the source's content from the mock filesystem.
|
||||
func (m *MockFetcher) Fetch(
|
||||
ctx context.Context, source imagerequest.Source,
|
||||
) (*FetchResult, error) {
|
||||
// Fetch retrieves content from the mock filesystem.
|
||||
func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, error) {
|
||||
// Check context cancellation
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
@@ -35,7 +33,11 @@ func (m *MockFetcher) Fetch(
|
||||
default:
|
||||
}
|
||||
|
||||
path := source.Host + source.Path
|
||||
// Parse URL to get filesystem path
|
||||
path, err := urlToFSPath(url)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Open the file
|
||||
f, err := m.fs.Open(path)
|
||||
@@ -66,11 +68,31 @@ func (m *MockFetcher) Fetch(
|
||||
}, nil
|
||||
}
|
||||
|
||||
// The content types the mock fetcher gives files in no format pixa reads.
|
||||
const (
|
||||
contentTypeSVG = "image/svg+xml"
|
||||
contentTypeOctetStream = "application/octet-stream"
|
||||
)
|
||||
// urlToFSPath converts a URL to a filesystem path.
|
||||
// https://example.com/images/photo.jpg -> example.com/images/photo.jpg
|
||||
func urlToFSPath(rawURL string) (string, error) {
|
||||
// Strip scheme
|
||||
url := rawURL
|
||||
if idx := strings.Index(url, "://"); idx != -1 {
|
||||
url = url[idx+3:]
|
||||
}
|
||||
|
||||
// Remove query string
|
||||
if idx := strings.Index(url, "?"); idx != -1 {
|
||||
url = url[:idx]
|
||||
}
|
||||
|
||||
// Remove fragment
|
||||
if idx := strings.Index(url, "#"); idx != -1 {
|
||||
url = url[:idx]
|
||||
}
|
||||
|
||||
if url == "" {
|
||||
return "", errEmptyURLPath
|
||||
}
|
||||
|
||||
return url, nil
|
||||
}
|
||||
|
||||
// detectContentTypeFromPath returns the MIME type based on file extension.
|
||||
func detectContentTypeFromPath(path string) string {
|
||||
@@ -78,17 +100,17 @@ func detectContentTypeFromPath(path string) string {
|
||||
|
||||
switch {
|
||||
case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"):
|
||||
return imagerequest.FormatJPEG.MIMEType()
|
||||
return contentTypeJPEG
|
||||
case strings.HasSuffix(path, ".png"):
|
||||
return imagerequest.FormatPNG.MIMEType()
|
||||
return contentTypePNG
|
||||
case strings.HasSuffix(path, ".gif"):
|
||||
return imagerequest.FormatGIF.MIMEType()
|
||||
return contentTypeGIF
|
||||
case strings.HasSuffix(path, ".webp"):
|
||||
return imagerequest.FormatWebP.MIMEType()
|
||||
return contentTypeWebP
|
||||
case strings.HasSuffix(path, ".avif"):
|
||||
return imagerequest.FormatAVIF.MIMEType()
|
||||
return contentTypeAVIF
|
||||
case strings.HasSuffix(path, ".jxl"):
|
||||
return imagerequest.FormatJXL.MIMEType()
|
||||
return contentTypeJXL
|
||||
case strings.HasSuffix(path, ".svg"):
|
||||
return contentTypeSVG
|
||||
default:
|
||||
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestFetchSendsRequestID verifies that a fetch sends the ID of the request
|
||||
@@ -26,7 +25,7 @@ func TestFetchSendsRequestID(t *testing.T) {
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
received <- r.Header.Get("X-Request-Id")
|
||||
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
_, _ = io.WriteString(w, imagePayload)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
@@ -35,7 +34,7 @@ func TestFetchSendsRequestID(t *testing.T) {
|
||||
|
||||
ctx := context.WithValue(testContext(t), middleware.RequestIDKey, requestID)
|
||||
|
||||
res, err := f.Fetch(ctx, upstreamSource("/image"))
|
||||
res, err := f.Fetch(ctx, upstreamURL("/image"))
|
||||
if err != nil {
|
||||
t.Fatalf("Fetch() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -39,41 +39,37 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input []byte
|
||||
sourceFormat imagerequest.Format
|
||||
format imagerequest.Format
|
||||
width, height int
|
||||
fit imagerequest.Fit
|
||||
refused bool
|
||||
}{
|
||||
{"AVIF at the limit", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 4096, 4096, imagerequest.FitCover, false},
|
||||
{"AVIF above the limit", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 4096, 4097, imagerequest.FitCover, true},
|
||||
{"WebP above the AVIF limit", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatWebP, 4096, 4097, imagerequest.FitCover, false},
|
||||
{"AVIF at the limit", cutOff, imagerequest.FormatAVIF,
|
||||
4096, 4096, imagerequest.FitCover, false},
|
||||
{"AVIF above the limit", cutOff, imagerequest.FormatAVIF,
|
||||
4096, 4097, imagerequest.FitCover, true},
|
||||
{"WebP above the AVIF limit", cutOff, imagerequest.FormatWebP,
|
||||
4096, 4097, imagerequest.FitCover, false},
|
||||
// FitInside does not enlarge, so the output stays 64x32
|
||||
{"AVIF inside a larger size", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 8192, 8192, imagerequest.FitInside, false},
|
||||
{"AVIF inside a larger size", cutOff, imagerequest.FormatAVIF,
|
||||
8192, 8192, imagerequest.FitInside, false},
|
||||
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
|
||||
{"AVIF outside a smaller size", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 8192, 1, imagerequest.FitOutside, true},
|
||||
{"AVIF outside a smaller size", cutOff, imagerequest.FormatAVIF,
|
||||
8192, 1, imagerequest.FitOutside, true},
|
||||
// With no size, the output keeps the source's 4097x4096
|
||||
{"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF,
|
||||
imagerequest.FormatAVIF, 0, 0, imagerequest.FitCover, true},
|
||||
{"orig of an AVIF source", largeAVIF, imagerequest.FormatAVIF,
|
||||
imagerequest.FormatOriginal, 0, 0, imagerequest.FitCover, true},
|
||||
{"JPEG of the source's size", largeAVIF, imagerequest.FormatAVIF,
|
||||
imagerequest.FormatJPEG, 0, 0, imagerequest.FitCover, false},
|
||||
0, 0, imagerequest.FitCover, true},
|
||||
{"orig of an AVIF source", largeAVIF, imagerequest.FormatOriginal,
|
||||
0, 0, imagerequest.FitCover, true},
|
||||
{"JPEG of the source's size", largeAVIF, imagerequest.FormatJPEG,
|
||||
0, 0, imagerequest.FitCover, false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(), bytes.NewReader(tt.input), tt.sourceFormat,
|
||||
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(tt.input),
|
||||
&imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
|
||||
Format: tt.format,
|
||||
|
||||
@@ -38,11 +38,8 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
@@ -107,11 +104,8 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
|
||||
@@ -7,8 +7,6 @@ import (
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// readerAfterDeadline is a Process input whose first Read waits until done
|
||||
@@ -86,10 +84,9 @@ func checkProcessStopsAtDeadline(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{MaxConcurrentProcessing: 1})
|
||||
|
||||
result, err := proc.Process(ctx, input, imagerequest.FormatJPEG,
|
||||
smallJPEGTransform())
|
||||
result, err := proc.Process(ctx, input, smallJPEGTransform())
|
||||
if result != nil {
|
||||
t.Logf("Process() made a %dx%d image after its deadline passed",
|
||||
result.Width, result.Height)
|
||||
|
||||
@@ -14,12 +14,9 @@ import (
|
||||
func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
_, err := New(Params{}).Process(
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)),
|
||||
imagerequest.FormatJPEG, &imagerequest.Transform{},
|
||||
&imagerequest.Transform{},
|
||||
)
|
||||
if !errors.Is(err, ErrUnsupportedOutputFormat) {
|
||||
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
|
||||
|
||||
@@ -11,18 +11,14 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// processedSize runs input, an image in sourceFormat, through Process and
|
||||
// returns the output's size in bytes.
|
||||
// processedSize runs input through Process and returns the output's size in
|
||||
// bytes.
|
||||
func processedSize(
|
||||
t *testing.T, input []byte, sourceFormat imagerequest.Format,
|
||||
transform *imagerequest.Transform,
|
||||
t *testing.T, input []byte, transform *imagerequest.Transform,
|
||||
) int64 {
|
||||
t.Helper()
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(), bytes.NewReader(input), sourceFormat, transform)
|
||||
result, err := New(Params{}).Process(t.Context(), bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -75,7 +71,7 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
size := processedSize(t, encodePNG(t, flat), imagerequest.FormatPNG,
|
||||
size := processedSize(t, encodePNG(t, flat),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatPNG})
|
||||
|
||||
const rawBytes = width * height * 3
|
||||
@@ -93,7 +89,7 @@ func TestImageProcessor_WebPAtEffort2(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, imagerequest.FormatJPEG,
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85})
|
||||
|
||||
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
|
||||
@@ -135,7 +131,7 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
|
||||
|
||||
input := encodePNG(t, source)
|
||||
|
||||
size := processedSize(t, input, imagerequest.FormatPNG,
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85})
|
||||
|
||||
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
|
||||
@@ -162,7 +158,7 @@ func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, imagerequest.FormatJPEG,
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85})
|
||||
|
||||
params := vips.NewJxlExportParams()
|
||||
|
||||
@@ -36,11 +36,8 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
@@ -109,11 +106,8 @@ func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
|
||||
|
||||
input := createBandsPNG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatPNG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatPNG,
|
||||
@@ -236,11 +230,8 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
|
||||
|
||||
input := createSquarePNG(t, tt.side, tt.squareSide)
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatPNG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatPNG,
|
||||
|
||||
@@ -41,11 +41,8 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
@@ -98,11 +95,8 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
@@ -70,12 +71,18 @@ type Result struct {
|
||||
InputWidth int
|
||||
// InputHeight is the original image height before processing.
|
||||
InputHeight int
|
||||
// InputFormat is the detected input format (e.g., "jpeg", "png").
|
||||
InputFormat string
|
||||
}
|
||||
|
||||
// MaxInputDimension is the maximum allowed width or height for input images.
|
||||
// Images larger than this are rejected to prevent DoS via decompression bombs.
|
||||
const MaxInputDimension = 8192
|
||||
|
||||
// DefaultMaxInputBytes is the default maximum input size in bytes (50 MiB).
|
||||
// This matches the default upstream fetcher limit.
|
||||
const DefaultMaxInputBytes = 50 << 20
|
||||
|
||||
// ErrInputTooLarge is returned when input image dimensions exceed MaxInputDimension.
|
||||
var ErrInputTooLarge = errors.New("input image dimensions exceed maximum")
|
||||
|
||||
@@ -111,10 +118,9 @@ var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
|
||||
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
|
||||
|
||||
// ErrUnsupportedInputFormat is returned, before the input is decoded, when
|
||||
// libvips reads the input's first bytes as another format than its source
|
||||
// format, or as none. An SVG whose first bytes pass the first-bytes check for
|
||||
// AVIF is one example. A source format other than the six pixa reads is also
|
||||
// refused with it.
|
||||
// libvips reads the input's first bytes as another format than the source
|
||||
// formats, or as none: for example an SVG whose first bytes pass the
|
||||
// first-bytes check for AVIF.
|
||||
var ErrUnsupportedInputFormat = errors.New("unsupported input format")
|
||||
|
||||
// ErrTooManyImages is returned when MaxConcurrentProcessing images are being
|
||||
@@ -136,30 +142,42 @@ type ImageProcessor struct {
|
||||
processingWaitTimeout time.Duration
|
||||
}
|
||||
|
||||
// Params holds configuration for creating an ImageProcessor. Both fields
|
||||
// must be at least 1.
|
||||
// Params holds configuration for creating an ImageProcessor.
|
||||
// Zero values use sensible defaults (MaxInputBytes defaults to DefaultMaxInputBytes).
|
||||
type Params struct {
|
||||
// MaxInputBytes is the maximum allowed input size in bytes.
|
||||
// If <= 0, DefaultMaxInputBytes is used.
|
||||
MaxInputBytes int64
|
||||
// MaxConcurrentProcessing is the most images processed at once.
|
||||
// If <= 0, config.DefaultMaxConcurrentProcessing() is used.
|
||||
MaxConcurrentProcessing int
|
||||
}
|
||||
|
||||
// New creates a new image processor with the given parameters.
|
||||
// A zero-value Params{} uses sensible defaults.
|
||||
func New(params Params) *ImageProcessor {
|
||||
initVips()
|
||||
|
||||
maxInputBytes := params.MaxInputBytes
|
||||
if maxInputBytes <= 0 {
|
||||
maxInputBytes = DefaultMaxInputBytes
|
||||
}
|
||||
|
||||
maxConcurrentProcessing := params.MaxConcurrentProcessing
|
||||
if maxConcurrentProcessing <= 0 {
|
||||
maxConcurrentProcessing = config.DefaultMaxConcurrentProcessing()
|
||||
}
|
||||
|
||||
return &ImageProcessor{
|
||||
maxInputBytes: params.MaxInputBytes,
|
||||
processingSemaphore: make(chan struct{}, params.MaxConcurrentProcessing),
|
||||
maxInputBytes: maxInputBytes,
|
||||
processingSemaphore: make(chan struct{}, maxConcurrentProcessing),
|
||||
processingWaitTimeout: ProcessingWaitTimeout,
|
||||
}
|
||||
}
|
||||
|
||||
// Process transforms the image read from input according to transform.
|
||||
// sourceFormat is the input's format, which magic detected from its first
|
||||
// bytes. When MaxConcurrentProcessing images are already being processed,
|
||||
// it waits up to ProcessingWaitTimeout for one to finish, then fails with
|
||||
// Process transforms an image according to transform. When
|
||||
// MaxConcurrentProcessing images are already being processed, it waits up
|
||||
// to ProcessingWaitTimeout for one to finish, then fails with
|
||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
|
||||
// no way to stop libvips once an encode has begun, so an encode keeps its
|
||||
@@ -168,7 +186,6 @@ func New(params Params) *ImageProcessor {
|
||||
func (p *ImageProcessor) Process(
|
||||
ctx context.Context,
|
||||
input io.Reader,
|
||||
sourceFormat imagerequest.Format,
|
||||
transform *imagerequest.Transform,
|
||||
) (*Result, error) {
|
||||
release, err := p.acquireSlot(ctx)
|
||||
@@ -183,9 +200,13 @@ func (p *ImageProcessor) Process(
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Input libvips would not load as sourceFormat is refused before it is
|
||||
// decoded
|
||||
err = checkInputFormat(data, sourceFormat)
|
||||
// govips picks the libvips loader from the input's first bytes with
|
||||
// DetermineImageType. Input whose first bytes it reads as no source
|
||||
// format, such as SVG, which librsvg would render, is refused before
|
||||
// libvips reads it.
|
||||
inputFormat := p.detectFormat(vips.DetermineImageType(data))
|
||||
|
||||
sourceFormat, err := p.formatFromString(inputFormat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -251,11 +272,12 @@ func (p *ImageProcessor) Process(
|
||||
return &Result{
|
||||
Content: io.NopCloser(bytes.NewReader(output)),
|
||||
ContentLength: int64(len(output)),
|
||||
ContentType: outputFormat.MIMEType(),
|
||||
ContentType: FormatToMIME(outputFormat),
|
||||
Width: img.Width(),
|
||||
Height: img.Height(),
|
||||
InputWidth: origWidth,
|
||||
InputHeight: origHeight,
|
||||
InputFormat: inputFormat,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -290,6 +312,62 @@ func targetDimensions(
|
||||
return width, height, nil
|
||||
}
|
||||
|
||||
// MIME types for the supported image formats.
|
||||
const (
|
||||
mimeJPEG = "image/jpeg"
|
||||
mimePNG = "image/png"
|
||||
mimeGIF = "image/gif"
|
||||
mimeWebP = "image/webp"
|
||||
mimeAVIF = "image/avif"
|
||||
mimeJXL = "image/jxl"
|
||||
)
|
||||
|
||||
// SupportedInputFormats returns MIME types this processor can read.
|
||||
func (p *ImageProcessor) SupportedInputFormats() []string {
|
||||
return []string{
|
||||
mimeJPEG,
|
||||
mimePNG,
|
||||
mimeGIF,
|
||||
mimeWebP,
|
||||
mimeAVIF,
|
||||
mimeJXL,
|
||||
}
|
||||
}
|
||||
|
||||
// SupportedOutputFormats returns formats this processor can write.
|
||||
func (p *ImageProcessor) SupportedOutputFormats() []imagerequest.Format {
|
||||
return []imagerequest.Format{
|
||||
imagerequest.FormatJPEG,
|
||||
imagerequest.FormatPNG,
|
||||
imagerequest.FormatGIF,
|
||||
imagerequest.FormatWebP,
|
||||
imagerequest.FormatAVIF,
|
||||
imagerequest.FormatJXL,
|
||||
}
|
||||
}
|
||||
|
||||
// FormatToMIME converts a Format to its MIME type string.
|
||||
func FormatToMIME(format imagerequest.Format) string {
|
||||
switch format {
|
||||
case imagerequest.FormatJPEG:
|
||||
return mimeJPEG
|
||||
case imagerequest.FormatPNG:
|
||||
return mimePNG
|
||||
case imagerequest.FormatWebP:
|
||||
return mimeWebP
|
||||
case imagerequest.FormatGIF:
|
||||
return mimeGIF
|
||||
case imagerequest.FormatAVIF:
|
||||
return mimeAVIF
|
||||
case imagerequest.FormatJXL:
|
||||
return mimeJXL
|
||||
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
|
||||
return "application/octet-stream"
|
||||
default:
|
||||
return "application/octet-stream"
|
||||
}
|
||||
}
|
||||
|
||||
// WaitForProcessing waits until no image is being processed, or until ctx
|
||||
// ends, and returns how many images were still being processed then. It
|
||||
// waits by taking each slot in processingSemaphore as it frees up until it
|
||||
@@ -358,49 +436,27 @@ func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// checkInputFormat returns ErrUnsupportedInputFormat unless libvips would
|
||||
// load data with the loader of sourceFormat. govips picks the libvips loader
|
||||
// from the input's first bytes with DetermineImageType. It refuses an SVG
|
||||
// whose first bytes pass the first-bytes check for AVIF, which librsvg would
|
||||
// otherwise render.
|
||||
func checkInputFormat(data []byte, sourceFormat imagerequest.Format) error {
|
||||
// A format with no loader, such as orig or auto, has the type govips also
|
||||
// gives input it cannot read, which the check below would let through.
|
||||
sourceType := vipsImageType(sourceFormat)
|
||||
if sourceType == vips.ImageTypeUnknown {
|
||||
return fmt.Errorf("%w: %q is not a source format",
|
||||
ErrUnsupportedInputFormat, sourceFormat)
|
||||
}
|
||||
|
||||
if vips.DetermineImageType(data) != sourceType {
|
||||
return fmt.Errorf("%w: libvips does not read the input as %s",
|
||||
ErrUnsupportedInputFormat, sourceFormat)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// vipsImageType returns the govips image type whose libvips loader reads
|
||||
// format. orig and auto stand for another format, so they have
|
||||
// vips.ImageTypeUnknown, which govips has no loader for.
|
||||
func vipsImageType(format imagerequest.Format) vips.ImageType {
|
||||
switch format {
|
||||
case imagerequest.FormatJPEG:
|
||||
return vips.ImageTypeJPEG
|
||||
case imagerequest.FormatPNG:
|
||||
return vips.ImageTypePNG
|
||||
case imagerequest.FormatGIF:
|
||||
return vips.ImageTypeGIF
|
||||
case imagerequest.FormatWebP:
|
||||
return vips.ImageTypeWEBP
|
||||
case imagerequest.FormatAVIF:
|
||||
return vips.ImageTypeAVIF
|
||||
case imagerequest.FormatJXL:
|
||||
return vips.ImageTypeJXL
|
||||
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
|
||||
return vips.ImageTypeUnknown
|
||||
// detectFormat returns the format string for a govips image type.
|
||||
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
switch imageType {
|
||||
case vips.ImageTypeJPEG:
|
||||
return "jpeg"
|
||||
case vips.ImageTypePNG:
|
||||
return "png"
|
||||
case vips.ImageTypeGIF:
|
||||
return "gif"
|
||||
case vips.ImageTypeWEBP:
|
||||
return "webp"
|
||||
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
|
||||
return string(imagerequest.FormatAVIF)
|
||||
case vips.ImageTypeJXL:
|
||||
return string(imagerequest.FormatJXL)
|
||||
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
|
||||
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
|
||||
vips.ImageTypeJP2K:
|
||||
return "unknown"
|
||||
default:
|
||||
return vips.ImageTypeUnknown
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -754,3 +810,26 @@ func jxlDistance(quality int) float64 {
|
||||
|
||||
return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0
|
||||
}
|
||||
|
||||
// formatFromString converts a format string from detectFormat to Format. A
|
||||
// format detectFormat could not name is ErrUnsupportedInputFormat.
|
||||
func (p *ImageProcessor) formatFromString(
|
||||
format string,
|
||||
) (imagerequest.Format, error) {
|
||||
switch format {
|
||||
case "jpeg":
|
||||
return imagerequest.FormatJPEG, nil
|
||||
case "png":
|
||||
return imagerequest.FormatPNG, nil
|
||||
case "gif":
|
||||
return imagerequest.FormatGIF, nil
|
||||
case "webp":
|
||||
return imagerequest.FormatWebP, nil
|
||||
case string(imagerequest.FormatAVIF):
|
||||
return imagerequest.FormatAVIF, nil
|
||||
case string(imagerequest.FormatJXL):
|
||||
return imagerequest.FormatJXL, nil
|
||||
default:
|
||||
return "", fmt.Errorf("%w: %s", ErrUnsupportedInputFormat, format)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -94,23 +94,23 @@ func isAVIF(data []byte) bool {
|
||||
// detectMIME is a minimal magic-byte detector for test assertions.
|
||||
func detectMIME(data []byte) string {
|
||||
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
|
||||
return imagerequest.FormatJPEG.MIMEType()
|
||||
return mimeJPEG
|
||||
}
|
||||
|
||||
if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" {
|
||||
return imagerequest.FormatPNG.MIMEType()
|
||||
return mimePNG
|
||||
}
|
||||
|
||||
if len(data) >= 4 && string(data[:4]) == "GIF8" {
|
||||
return imagerequest.FormatGIF.MIMEType()
|
||||
return mimeGIF
|
||||
}
|
||||
|
||||
if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" {
|
||||
return imagerequest.FormatWebP.MIMEType()
|
||||
return mimeWebP
|
||||
}
|
||||
|
||||
if isAVIF(data) {
|
||||
return imagerequest.FormatAVIF.MIMEType()
|
||||
return mimeAVIF
|
||||
}
|
||||
|
||||
return ""
|
||||
@@ -119,7 +119,7 @@ func detectMIME(data []byte) string {
|
||||
func TestImageProcessor_ResizeJPEG(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
input := createTestJPEG(t, 800, 600)
|
||||
@@ -131,8 +131,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -158,7 +157,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
|
||||
}
|
||||
|
||||
mime := detectMIME(data)
|
||||
if mime != imagerequest.FormatJPEG.MIMEType() {
|
||||
if mime != mimeJPEG {
|
||||
t.Errorf("Output format = %v, want image/jpeg", mime)
|
||||
}
|
||||
}
|
||||
@@ -166,7 +165,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
|
||||
func TestImageProcessor_ConvertToPNG(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
@@ -177,8 +176,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -191,7 +189,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
|
||||
}
|
||||
|
||||
mime := detectMIME(data)
|
||||
if mime != imagerequest.FormatPNG.MIMEType() {
|
||||
if mime != mimePNG {
|
||||
t.Errorf("Output format = %v, want image/png", mime)
|
||||
}
|
||||
}
|
||||
@@ -203,7 +201,7 @@ func processAndCheckSize(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
input := createTestJPEG(t, inputW, inputH)
|
||||
@@ -215,8 +213,7 @@ func processAndCheckSize(
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -242,7 +239,7 @@ func TestImageProcessor_OriginalSize(t *testing.T) {
|
||||
func TestImageProcessor_FitContain(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
// 800x400 image (2:1 aspect) into 400x400 box with contain
|
||||
@@ -256,8 +253,7 @@ func TestImageProcessor_FitContain(t *testing.T) {
|
||||
FitMode: imagerequest.FitContain,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -289,7 +285,7 @@ func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
|
||||
func TestImageProcessor_ProcessPNG(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
input := createTestPNG(t, 400, 300)
|
||||
@@ -300,8 +296,7 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatPNG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -317,6 +312,22 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestImageProcessor_SupportedFormats(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{})
|
||||
|
||||
inputFormats := proc.SupportedInputFormats()
|
||||
if len(inputFormats) == 0 {
|
||||
t.Error("SupportedInputFormats() returned empty slice")
|
||||
}
|
||||
|
||||
outputFormats := proc.SupportedOutputFormats()
|
||||
if len(outputFormats) == 0 {
|
||||
t.Error("SupportedOutputFormats() returned empty slice")
|
||||
}
|
||||
}
|
||||
|
||||
func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -335,7 +346,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
input := createTestJPEG(t, tt.width, tt.height)
|
||||
|
||||
@@ -346,8 +357,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err == nil {
|
||||
t.Error("Process() should reject oversized input images")
|
||||
}
|
||||
@@ -362,7 +372,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
|
||||
func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
// Create an image at exactly MaxInputDimension - should be accepted
|
||||
@@ -375,8 +385,7 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf(
|
||||
"Process() should accept images at MaxInputDimension, got error: %v",
|
||||
@@ -394,7 +403,7 @@ func encodeAndCheck(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
@@ -406,8 +415,7 @@ func encodeAndCheck(
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil", err)
|
||||
}
|
||||
@@ -438,13 +446,13 @@ func encodeAndCheck(
|
||||
func TestImageProcessor_EncodeWebP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
encodeAndCheck(t, imagerequest.FormatWebP, 80, imagerequest.FormatWebP.MIMEType())
|
||||
encodeAndCheck(t, imagerequest.FormatWebP, 80, mimeWebP)
|
||||
}
|
||||
|
||||
func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
ctx := context.Background()
|
||||
|
||||
// Load test AVIF file
|
||||
@@ -461,8 +469,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatAVIF, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
|
||||
}
|
||||
@@ -476,7 +483,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
}
|
||||
|
||||
mime := detectMIME(data)
|
||||
if mime != imagerequest.FormatJPEG.MIMEType() {
|
||||
if mime != mimeJPEG {
|
||||
t.Errorf("Output format = %v, want image/jpeg", mime)
|
||||
}
|
||||
}
|
||||
@@ -487,7 +494,7 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
|
||||
// Create a processor with a very small byte limit
|
||||
const limit = 1024
|
||||
|
||||
proc := New(Params{MaxInputBytes: limit, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{MaxInputBytes: limit})
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a valid JPEG that exceeds the byte limit
|
||||
@@ -503,8 +510,7 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err == nil {
|
||||
t.Fatal("Process() should reject input exceeding maxInputBytes")
|
||||
}
|
||||
@@ -521,7 +527,7 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
|
||||
input := createTestJPEG(t, 10, 10)
|
||||
limit := int64(len(input)) * 10 // 10× headroom
|
||||
|
||||
proc := New(Params{MaxInputBytes: limit, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{MaxInputBytes: limit})
|
||||
ctx := context.Background()
|
||||
|
||||
transform := &imagerequest.Transform{
|
||||
@@ -531,8 +537,7 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil", err)
|
||||
}
|
||||
@@ -540,25 +545,37 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
}
|
||||
|
||||
func TestImageProcessor_DefaultMaxInputBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Passing 0 should use the default
|
||||
proc := New(Params{})
|
||||
if proc.maxInputBytes != DefaultMaxInputBytes {
|
||||
t.Errorf("maxInputBytes = %d, want %d", proc.maxInputBytes, DefaultMaxInputBytes)
|
||||
}
|
||||
|
||||
// Passing negative should also use the default
|
||||
proc = New(Params{MaxInputBytes: -1})
|
||||
if proc.maxInputBytes != DefaultMaxInputBytes {
|
||||
t.Errorf("maxInputBytes = %d, want %d", proc.maxInputBytes, DefaultMaxInputBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestImageProcessor_EncodeAVIF(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
encodeAndCheck(t, imagerequest.FormatAVIF, 85, imagerequest.FormatAVIF.MIMEType())
|
||||
encodeAndCheck(t, imagerequest.FormatAVIF, 85, mimeAVIF)
|
||||
}
|
||||
|
||||
// processAndDecode runs input, a JPEG, through Process and decodes the output
|
||||
// with vips, so a test can inspect the image a client would receive.
|
||||
// processAndDecode runs input through Process and decodes the output with
|
||||
// vips, so a test can inspect the image a client would receive.
|
||||
func processAndDecode(
|
||||
t *testing.T, input []byte, transform *imagerequest.Transform,
|
||||
) *vips.ImageRef {
|
||||
t.Helper()
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
context.Background(), bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
transform,
|
||||
result, err := New(Params{}).Process(
|
||||
context.Background(), bytes.NewReader(input), transform,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
|
||||
@@ -21,12 +21,8 @@ func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
|
||||
Format: imagerequest.FormatJXL,
|
||||
}
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)),
|
||||
imagerequest.FormatJPEG, transform,
|
||||
result, err := New(Params{}).Process(
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), transform,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
@@ -68,11 +64,8 @@ func TestImageProcessor_JPEGXLQuality(t *testing.T) {
|
||||
outputBytes := make(map[int]int64)
|
||||
|
||||
for _, quality := range []int{30, 90} {
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() at quality %d error = %v", quality, err)
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
@@ -93,8 +94,7 @@ func processInBackground(
|
||||
proc *ImageProcessor, reader *gatedReader, results chan<- error,
|
||||
) {
|
||||
go func() {
|
||||
result, err := proc.Process(context.Background(), reader,
|
||||
imagerequest.FormatJPEG, smallJPEGTransform())
|
||||
result, err := proc.Process(context.Background(), reader, smallJPEGTransform())
|
||||
if err == nil {
|
||||
_ = result.Content.Close()
|
||||
}
|
||||
@@ -117,15 +117,47 @@ func waitForEntries(t *testing.T, entered <-chan struct{}, count int) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewMakesMaxConcurrentProcessingSlots(t *testing.T) {
|
||||
func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 3})
|
||||
for _, limit := range []int{0, -1} {
|
||||
proc := New(Params{MaxConcurrentProcessing: limit})
|
||||
if got := cap(proc.processingSemaphore); got != max(runtime.GOMAXPROCS(0)/2, 1) {
|
||||
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, half the CPUs",
|
||||
limit, got, max(runtime.GOMAXPROCS(0)/2, 1))
|
||||
}
|
||||
}
|
||||
|
||||
proc := New(Params{MaxConcurrentProcessing: 3})
|
||||
if got := cap(proc.processingSemaphore); got != 3 {
|
||||
t.Errorf("MaxConcurrentProcessing 3: %d slots, want 3", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewDefaultIsHalfTheCPUsAndAtLeastOne checks that New, given no
|
||||
// MaxConcurrentProcessing, makes one slot for every two CPUs Go uses,
|
||||
// rounded down, and one slot when Go uses one CPU.
|
||||
//
|
||||
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
|
||||
func TestNewDefaultIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
|
||||
cpusBefore := runtime.GOMAXPROCS(0)
|
||||
|
||||
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
|
||||
|
||||
for _, tc := range []struct{ cpus, want int }{
|
||||
{1, 1}, {2, 1}, {3, 1}, {8, 4},
|
||||
} {
|
||||
runtime.GOMAXPROCS(tc.cpus)
|
||||
|
||||
got := cap(New(Params{}).processingSemaphore)
|
||||
t.Logf("GOMAXPROCS %d: %d slots", tc.cpus, got)
|
||||
|
||||
if got != tc.want {
|
||||
t.Errorf("GOMAXPROCS %d: %d slots, want %d", tc.cpus, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
|
||||
// than MaxConcurrentProcessing allows and holds the first ones inside
|
||||
// Process until the test lets them go. No more than the limit may be
|
||||
@@ -139,7 +171,7 @@ func TestProcessNeverExceedsMaxConcurrentProcessing(t *testing.T) {
|
||||
calls = 6
|
||||
)
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: limit})
|
||||
proc := New(Params{MaxConcurrentProcessing: limit})
|
||||
input := createTestJPEG(t, 50, 50)
|
||||
|
||||
counter := &readingCounter{}
|
||||
@@ -186,7 +218,7 @@ func TestProcessNeverExceedsMaxConcurrentProcessing(t *testing.T) {
|
||||
func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{MaxConcurrentProcessing: 1})
|
||||
proc.processingWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
input := createTestJPEG(t, 10, 10)
|
||||
@@ -207,7 +239,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
|
||||
start := time.Now()
|
||||
|
||||
_, err := proc.Process(context.Background(), bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, smallJPEGTransform())
|
||||
smallJPEGTransform())
|
||||
if !errors.Is(err, ErrTooManyImages) {
|
||||
t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
|
||||
}
|
||||
@@ -275,8 +307,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
|
||||
proc.processingWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
_, err := proc.Process(context.Background(), tc.input,
|
||||
imagerequest.FormatJPEG, tc.transform)
|
||||
_, err := proc.Process(context.Background(), tc.input, tc.transform)
|
||||
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
|
||||
t.Fatalf("Process() error = %v, want %v", err, tc.want)
|
||||
}
|
||||
@@ -286,7 +317,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
}
|
||||
|
||||
result, err := proc.Process(context.Background(), bytes.NewReader(valid),
|
||||
imagerequest.FormatJPEG, smallJPEGTransform())
|
||||
smallJPEGTransform())
|
||||
if err != nil {
|
||||
t.Fatalf("Process() after the error = %v, want nil", err)
|
||||
}
|
||||
@@ -303,7 +334,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
func TestWaitForProcessing(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 2})
|
||||
proc := New(Params{MaxConcurrentProcessing: 2})
|
||||
|
||||
gate := make(chan struct{})
|
||||
entered := make(chan struct{}, 1)
|
||||
|
||||
@@ -10,9 +10,8 @@ import (
|
||||
)
|
||||
|
||||
// TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF
|
||||
// input is refused with ErrUnsupportedInputFormat before it is decoded, and
|
||||
// not saved as JPEG. TIFF is not a source format, and the test passes the
|
||||
// input to Process as a JPEG.
|
||||
// input, which is not a source format, is refused with
|
||||
// ErrUnsupportedInputFormat before it is decoded, and not saved as JPEG.
|
||||
func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -28,11 +27,8 @@ func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
|
||||
t.Fatalf("failed to save test TIFF: %v", err)
|
||||
}
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
t.Context(), bytes.NewReader(tiff), imagerequest.FormatJPEG,
|
||||
result, err := New(Params{}).Process(
|
||||
t.Context(), bytes.NewReader(tiff),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatOriginal},
|
||||
)
|
||||
if result != nil {
|
||||
|
||||
@@ -2,7 +2,6 @@ package imageprocessor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"io"
|
||||
"testing"
|
||||
|
||||
@@ -15,14 +14,14 @@ import (
|
||||
func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
proc := New(Params{})
|
||||
|
||||
for _, format := range []imagerequest.Format{
|
||||
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
|
||||
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
|
||||
} {
|
||||
saved, err := proc.Process(t.Context(),
|
||||
bytes.NewReader(createTestPNG(t, 20, 20)), imagerequest.FormatPNG,
|
||||
bytes.NewReader(createTestPNG(t, 20, 20)),
|
||||
&imagerequest.Transform{Format: format})
|
||||
if err != nil {
|
||||
t.Fatalf("saving the test image as %s: %v", format, err)
|
||||
@@ -33,7 +32,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
t.Fatalf("reading the %s image: %v", format, err)
|
||||
}
|
||||
|
||||
result, err := proc.Process(t.Context(), bytes.NewReader(source), format,
|
||||
result, err := proc.Process(t.Context(), bytes.NewReader(source),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatOriginal})
|
||||
if err != nil {
|
||||
t.Errorf("orig of the %s image: %v", format, err)
|
||||
@@ -43,33 +42,9 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
|
||||
t.Logf("orig of the %s image: %s", format, result.ContentType)
|
||||
|
||||
if result.ContentType != format.MIMEType() {
|
||||
if result.ContentType != FormatToMIME(format) {
|
||||
t.Errorf("orig of the %s image is %s, want %s",
|
||||
format, result.ContentType, format.MIMEType())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_NotASourceFormatRefused verifies that a source format
|
||||
// other than the six, such as orig, auto or none, is refused with
|
||||
// ErrUnsupportedInputFormat. The input is text, which libvips reads as no
|
||||
// format.
|
||||
func TestImageProcessor_NotASourceFormatRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
|
||||
for _, sourceFormat := range []imagerequest.Format{
|
||||
imagerequest.FormatOriginal, imagerequest.FormatAuto, "",
|
||||
} {
|
||||
result, err := proc.Process(t.Context(),
|
||||
bytes.NewReader([]byte("this is text, not an image")), sourceFormat,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJPEG})
|
||||
t.Logf("source format %q: error = %v", sourceFormat, err)
|
||||
|
||||
if result != nil || !errors.Is(err, ErrUnsupportedInputFormat) {
|
||||
t.Errorf("Process() with source format %q = %v, error %v; want %v",
|
||||
sourceFormat, result, err, ErrUnsupportedInputFormat)
|
||||
format, result.ContentType, FormatToMIME(format))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,100 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// TestProxy_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
|
||||
// variant's file with text, and checks that the next request is a miss
|
||||
// answered with the variant made again, and the one after a hit.
|
||||
func TestProxy_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
|
||||
want, _ := getImage(t, svc, req)
|
||||
|
||||
path := variantFilePath(svc, imagerequest.CacheKey(req))
|
||||
|
||||
err := os.WriteFile(path, []byte("this is text, not an image"),
|
||||
imgcache.StorageFilePerm)
|
||||
if err != nil {
|
||||
t.Fatalf("replacing the variant file: %v", err)
|
||||
}
|
||||
|
||||
got, status := getImage(t, svc, req)
|
||||
t.Logf("after the replacement: %d bytes, %s", len(got), status)
|
||||
|
||||
if status != CacheMiss || !bytes.Equal(got, want) {
|
||||
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
|
||||
"made again, %s", len(got), status, len(want), CacheMiss)
|
||||
}
|
||||
|
||||
got, status = getImage(t, svc, req)
|
||||
|
||||
if status != CacheHit || !bytes.Equal(got, want) {
|
||||
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
|
||||
len(got), status, len(want), CacheHit)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_HitWaitsForNoDatabaseWrite holds the database's one
|
||||
// connection while a cached variant is requested. The hit must be answered
|
||||
// while the connection is still held, and its access time must reach the
|
||||
// database once it is free.
|
||||
func TestProxy_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
key := imagerequest.CacheKey(req)
|
||||
|
||||
getImage(t, svc, req)
|
||||
|
||||
// Each connection to an in-memory SQLite database opens a new, empty one,
|
||||
// so the goroutine that writes the access times must share the
|
||||
// connection that holds the schema.
|
||||
svc.db.SetMaxOpenConns(1)
|
||||
svc.cache.StartPendingAccessTimeWrites()
|
||||
|
||||
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
|
||||
|
||||
anHourAgo := time.Now().Add(-time.Hour)
|
||||
setVariantLastAccessed(t, svc.db, key, anHourAgo)
|
||||
|
||||
releaseDatabase := holdDatabase(t, svc.db)
|
||||
defer releaseDatabase()
|
||||
|
||||
results := startGet(t.Context(), svc, req)
|
||||
|
||||
select {
|
||||
case got := <-results:
|
||||
if got.err != nil {
|
||||
t.Fatalf("Get() of the cached variant error = %v", got.err)
|
||||
}
|
||||
case <-time.After(pendingAccessTimesWait):
|
||||
t.Fatal("the cache hit waited for the database")
|
||||
}
|
||||
|
||||
releaseDatabase()
|
||||
|
||||
deadline := time.Now().Add(pendingAccessTimesWait)
|
||||
|
||||
for countRows(t, svc.db, `
|
||||
SELECT COUNT(*) FROM variant_content
|
||||
WHERE cache_key = ? AND last_accessed_at > ?
|
||||
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the hit's access time was not written")
|
||||
}
|
||||
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -1,77 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// TestNew_RegistersItsCountersOnItsRegistry checks that New registers the
|
||||
// proxy's counters on the registry it is given, under the names README.md
|
||||
// gives them. The cache registers its own counters on another registry.
|
||||
func TestNew_RegistersItsCountersOnItsRegistry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, err := imgcache.NewCache(setupTestDB(t),
|
||||
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
|
||||
prometheus.NewRegistry())
|
||||
if err != nil {
|
||||
t.Fatalf("imgcache.NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
registry := prometheus.NewRegistry()
|
||||
|
||||
_, err = New(&Config{Cache: cache}, registry)
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
|
||||
names := []string{
|
||||
"pixa_upstream_fetches_total",
|
||||
"pixa_upstream_fetch_bytes_total",
|
||||
"pixa_transforms_total",
|
||||
}
|
||||
|
||||
count, err := testutil.GatherAndCount(registry, names...)
|
||||
if err != nil {
|
||||
t.Fatalf("testutil.GatherAndCount() error = %v", err)
|
||||
}
|
||||
|
||||
if count != len(names) {
|
||||
t.Errorf("%d of the counters %v are on the registry, want %d",
|
||||
count, names, len(names))
|
||||
}
|
||||
}
|
||||
|
||||
// TestNew_FailsWhenItsCountersCannotBeRegistered makes a second proxy on
|
||||
// the registry of a first. Its counters have the same names, so the
|
||||
// registry refuses them, and New must return that error.
|
||||
func TestNew_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, err := imgcache.NewCache(setupTestDB(t),
|
||||
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
|
||||
prometheus.NewRegistry())
|
||||
if err != nil {
|
||||
t.Fatalf("imgcache.NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
registry := prometheus.NewRegistry()
|
||||
|
||||
_, err = New(&Config{Cache: cache}, registry)
|
||||
if err != nil {
|
||||
t.Fatalf("New() of the first proxy error = %v", err)
|
||||
}
|
||||
|
||||
_, err = New(&Config{Cache: cache}, registry)
|
||||
t.Logf("New() of the second proxy error = %v", err)
|
||||
|
||||
var alreadyRegistered prometheus.AlreadyRegisteredError
|
||||
if !errors.As(err, &alreadyRegistered) {
|
||||
t.Errorf("New() of the second proxy error = %v, want %T",
|
||||
err, alreadyRegistered)
|
||||
}
|
||||
}
|
||||
@@ -1,575 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// Test data literals used repeatedly in this file (goconst).
|
||||
const (
|
||||
testPathPhoto = "/images/photo.jpg"
|
||||
testPathUpload = "/uploads/image.jpg"
|
||||
)
|
||||
|
||||
func TestProxy_Get_GoodHost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
// Verify we got content
|
||||
data, err := io.ReadAll(resp.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read response: %v", err)
|
||||
}
|
||||
|
||||
if len(data) == 0 {
|
||||
t.Error("expected non-empty response")
|
||||
}
|
||||
|
||||
if resp.ContentType != testContentTypeJPEG {
|
||||
t.Errorf("ContentType = %q, want %q", resp.ContentType, testContentTypeJPEG)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_OtherHost checks that the proxy serves a second host as it
|
||||
// serves GoodHost.
|
||||
func TestProxy_Get_OtherHost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.OtherHost,
|
||||
Path: testPathUpload,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Should fetch successfully
|
||||
resp, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
data, err := io.ReadAll(resp.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read response: %v", err)
|
||||
}
|
||||
|
||||
if len(data) == 0 {
|
||||
t.Error("expected non-empty response")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_InvalidFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
req.Source.Path = "/images/fake.jpg"
|
||||
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
|
||||
|
||||
// Should fail because magic bytes don't match
|
||||
_, err := svc.Get(ctx, req)
|
||||
if err == nil {
|
||||
t.Error("Get() expected error for invalid file")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_NotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
req.Source.Path = "/images/nonexistent.jpg"
|
||||
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
|
||||
|
||||
_, err := svc.Get(ctx, req)
|
||||
if err == nil {
|
||||
t.Error("Get() expected error for nonexistent file")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_FormatConversion(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
sourcePath string
|
||||
outFormat imagerequest.Format
|
||||
wantMIME string
|
||||
}{
|
||||
{
|
||||
name: "JPEG to PNG",
|
||||
sourcePath: testPathPhoto,
|
||||
outFormat: imagerequest.FormatPNG,
|
||||
wantMIME: "image/png",
|
||||
},
|
||||
{
|
||||
name: "PNG to JPEG",
|
||||
sourcePath: "/images/logo.png",
|
||||
outFormat: imagerequest.FormatJPEG,
|
||||
wantMIME: testContentTypeJPEG,
|
||||
},
|
||||
{
|
||||
name: "GIF to PNG",
|
||||
sourcePath: "/images/animation.gif",
|
||||
outFormat: imagerequest.FormatPNG,
|
||||
wantMIME: "image/png",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: fixtures.GoodHost, Path: tt.sourcePath},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: tt.outFormat, Quality: 85, FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
if resp.ContentType != tt.wantMIME {
|
||||
t.Errorf("ContentType = %q, want %q", resp.ContentType, tt.wantMIME)
|
||||
}
|
||||
|
||||
// Verify magic bytes match the expected format
|
||||
data, err := io.ReadAll(resp.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read response: %v", err)
|
||||
}
|
||||
|
||||
detectedFormat, err := magic.DetectFormat(data)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to detect format: %v", err)
|
||||
}
|
||||
|
||||
if detectedFormat.MIMEType() != tt.wantMIME {
|
||||
t.Errorf("detected format = %q, want the format of %s",
|
||||
detectedFormat, tt.wantMIME)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_Caching(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// First request - should be a cache miss
|
||||
resp1, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() first request error = %v", err)
|
||||
}
|
||||
|
||||
if resp1.CacheStatus != CacheMiss {
|
||||
t.Errorf("first request CacheStatus = %q, want %q", resp1.CacheStatus, CacheMiss)
|
||||
}
|
||||
|
||||
data1, err := io.ReadAll(resp1.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read first response: %v", err)
|
||||
}
|
||||
|
||||
_ = resp1.Content.Close()
|
||||
|
||||
// Second request - should be a cache hit
|
||||
resp2, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() second request error = %v", err)
|
||||
}
|
||||
|
||||
if resp2.CacheStatus != CacheHit {
|
||||
t.Errorf("second request CacheStatus = %q, want %q", resp2.CacheStatus, CacheHit)
|
||||
}
|
||||
|
||||
data2, err := io.ReadAll(resp2.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read second response: %v", err)
|
||||
}
|
||||
|
||||
_ = resp2.Content.Close()
|
||||
|
||||
// Content should be identical
|
||||
if len(data1) != len(data2) {
|
||||
t.Errorf("response sizes differ: %d vs %d", len(data1), len(data2))
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo caches a source and
|
||||
// a variant of it and sets back two days every time pixa stored for them.
|
||||
// For the source: last_accessed_at in source_content, and the access and
|
||||
// modification times of its file. For the variant: last_accessed_at in
|
||||
// variant_content, and the access and modification times of its file. It
|
||||
// then checks that the variant is still served from the cache and that a new
|
||||
// variant is made from the cached source without fetching it again.
|
||||
//
|
||||
// Left at the present: the files' status change and birth times, which a
|
||||
// program cannot set, and the times of the directories holding the files,
|
||||
// which other entries share.
|
||||
func TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
|
||||
resp, err := svc.Get(t.Context(), variant)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() first request error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
twoDaysAgo := time.Now().AddDate(0, 0, -2)
|
||||
backdateSource(t, svc, variant, twoDaysAgo)
|
||||
backdateVariant(t, svc, variant, twoDaysAgo)
|
||||
|
||||
resp, err = svc.Get(t.Context(), variant)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of the cached variant error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
if resp.CacheStatus != CacheHit {
|
||||
t.Errorf("cached variant CacheStatus = %q, want %q",
|
||||
resp.CacheStatus, CacheHit)
|
||||
}
|
||||
|
||||
// Another quality is a new variant, made from the cached source
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of a new variant error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
fetches := readCounters(t, svc).upstreamFetchCount
|
||||
if fetches != 1 {
|
||||
t.Errorf("upstream fetches = %d, want 1", fetches)
|
||||
}
|
||||
}
|
||||
|
||||
// backdateSource sets every time pixa stored for req's cached source to when,
|
||||
// as listed on TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo.
|
||||
func backdateSource(
|
||||
t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
key := req.Source.Key()
|
||||
setSourceLastAccessed(t, svc.db, key, when)
|
||||
|
||||
path := sourceFilePath(svc, key)
|
||||
|
||||
err := os.Chtimes(path, when, when)
|
||||
if err != nil {
|
||||
t.Fatalf("setting back the times of %s: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
// backdateVariant sets every time pixa stored for the cached variant req asks
|
||||
// for to when, as listed on
|
||||
// TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo.
|
||||
func backdateVariant(
|
||||
t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
key := imagerequest.CacheKey(req)
|
||||
path := variantFilePath(svc, key)
|
||||
|
||||
setVariantLastAccessed(t, svc.db, key, when)
|
||||
|
||||
err := os.Chtimes(path, when, when)
|
||||
if err != nil {
|
||||
t.Fatalf("setting back the times of %s: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_DifferentSizes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
// Request same image at different sizes
|
||||
sizes := []imagerequest.Size{
|
||||
{Width: 25, Height: 25},
|
||||
{Width: 50, Height: 50},
|
||||
{Width: 75, Height: 75},
|
||||
}
|
||||
|
||||
responses := make([][]byte, 0, len(sizes))
|
||||
|
||||
for _, size := range sizes {
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() error for size %v = %v", size, err)
|
||||
}
|
||||
|
||||
data, err := io.ReadAll(resp.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read response: %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
responses = append(responses, data)
|
||||
}
|
||||
|
||||
// All responses should be different sizes (different cache entries)
|
||||
for i := range len(responses) - 1 {
|
||||
if len(responses[i]) == len(responses[i+1]) {
|
||||
// Not necessarily an error, but worth noting
|
||||
t.Logf("responses %d and %d have same size: %d bytes",
|
||||
i, i+1, len(responses[i]))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_ContextCancellation(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // Cancel immediately
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
_, err := svc.Get(ctx, req)
|
||||
if err == nil {
|
||||
t.Error("Get() expected error for cancelled context")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_ReturnsETag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
// ETag should be set
|
||||
if resp.ETag == "" {
|
||||
t.Error("ETag should be set in response")
|
||||
}
|
||||
|
||||
// ETag should be properly quoted
|
||||
if len(resp.ETag) < 2 || resp.ETag[0] != '"' || resp.ETag[len(resp.ETag)-1] != '"' {
|
||||
t.Errorf("ETag should be quoted, got %q", resp.ETag)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_ETagConsistency(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// First request
|
||||
resp1, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() first request error = %v", err)
|
||||
}
|
||||
|
||||
etag1 := resp1.ETag
|
||||
|
||||
_ = resp1.Content.Close()
|
||||
|
||||
// Second request (from cache)
|
||||
resp2, err := svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() second request error = %v", err)
|
||||
}
|
||||
|
||||
etag2 := resp2.ETag
|
||||
|
||||
_ = resp2.Content.Close()
|
||||
|
||||
// ETags should be identical for the same content
|
||||
if etag1 != etag2 {
|
||||
t.Errorf("ETags should match: %q != %q", etag1, etag2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_DifferentETagsForDifferentContent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
ctx := context.Background()
|
||||
|
||||
// Request same image at different sizes - should get different ETags
|
||||
req1 := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 25, Height: 25},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
req2 := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp1, err := svc.Get(ctx, req1)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() first request error = %v", err)
|
||||
}
|
||||
|
||||
etag1 := resp1.ETag
|
||||
|
||||
_ = resp1.Content.Close()
|
||||
|
||||
resp2, err := svc.Get(ctx, req2)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() second request error = %v", err)
|
||||
}
|
||||
|
||||
etag2 := resp2.ETag
|
||||
|
||||
_ = resp2.Content.Close()
|
||||
|
||||
// ETags should be different for different content
|
||||
if etag1 == etag2 {
|
||||
t.Error("ETags should differ for different sizes")
|
||||
}
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestProxy_Get_ServesCachedVariantOfNegativeCachedURL caches a variant of
|
||||
// the test photo, then keeps a failed fetch of the photo's URL in the negative
|
||||
// cache, as a failed fetch for another variant does once the source has been
|
||||
// evicted. The cached variant is still served, and a variant not in the cache
|
||||
// is made from the cached source. Once the source is evicted, a variant not in
|
||||
// the cache needs a fetch and is refused.
|
||||
func TestProxy_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
if err != nil {
|
||||
t.Fatalf("Get() first request error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
err = svc.cache.PutFailure(t.Context(),
|
||||
photoVariant(fixtures, 85, imagerequest.FitCover).Source)
|
||||
if err != nil {
|
||||
t.Fatalf("PutFailure() error = %v", err)
|
||||
}
|
||||
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of the cached variant error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
if resp.CacheStatus != CacheHit {
|
||||
t.Errorf("cached variant CacheStatus = %q, want %q",
|
||||
resp.CacheStatus, CacheHit)
|
||||
}
|
||||
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
t.Logf("Get() of a variant made from the cached source: error = %v", err)
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of a variant made from the cached source error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
|
||||
|
||||
reader, _, _, err := svc.cache.Source(t.Context(), source)
|
||||
if err != nil {
|
||||
t.Fatalf("Source() error = %v, want the cached source", err)
|
||||
}
|
||||
|
||||
_ = reader.Close()
|
||||
|
||||
// Evicts the source as the cache's eviction does: its row, then its file
|
||||
_, err = svc.db.ExecContext(t.Context(),
|
||||
`DELETE FROM source_content WHERE source_key = ?`, string(source.Key()))
|
||||
if err != nil {
|
||||
t.Fatalf("deleting the source's row: %v", err)
|
||||
}
|
||||
|
||||
err = os.Remove(sourceFilePath(svc, source.Key()))
|
||||
if err != nil {
|
||||
t.Fatalf("removing the source's file: %v", err)
|
||||
}
|
||||
|
||||
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
|
||||
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
|
||||
|
||||
if !errors.Is(err, ErrNegativeCached) {
|
||||
t.Errorf("Get() of a variant whose source was evicted error = %v, want %v",
|
||||
err, ErrNegativeCached)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxy_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// This test verifies that Proxy.Get() checks the negative cache
|
||||
// We can't easily test the full pipeline without vips, but we can
|
||||
// verify the error type
|
||||
err := ErrNegativeCached
|
||||
if !errors.Is(err, ErrNegativeCached) {
|
||||
t.Error("ErrNegativeCached should be identifiable with errors.Is")
|
||||
}
|
||||
}
|
||||
@@ -1,184 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// getImage calls Get for req and returns the image it answered with and
|
||||
// its cache status.
|
||||
func getImage(
|
||||
t *testing.T, svc *testProxy, req *imagerequest.Request,
|
||||
) ([]byte, CacheStatus) {
|
||||
t.Helper()
|
||||
|
||||
resp, err := svc.Get(t.Context(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
body, err := io.ReadAll(resp.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
|
||||
}
|
||||
|
||||
return body, resp.CacheStatus
|
||||
}
|
||||
|
||||
// fileSize returns the size of the file at path.
|
||||
func fileSize(t *testing.T, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
t.Fatalf("stat %s: %v", path, err)
|
||||
}
|
||||
|
||||
return info.Size()
|
||||
}
|
||||
|
||||
// cachedSourcePath returns the path of the cached source file req uses.
|
||||
func cachedSourcePath(
|
||||
t *testing.T, svc *testProxy, req *imagerequest.Request,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
reader, _, _, err := svc.cache.Source(t.Context(), req.Source)
|
||||
if err != nil {
|
||||
t.Fatalf("Source() error = %v; want the cached source", err)
|
||||
}
|
||||
|
||||
_ = reader.Close()
|
||||
|
||||
return sourceFilePath(svc, req.Source.Key())
|
||||
}
|
||||
|
||||
// TestProxy_Get_FetchesAgainForRemovedCachedSource removes the cached
|
||||
// source's row and file, as the reconciliation pass does when the file is
|
||||
// empty or cut short. internal/imgcache tests that the pass does so. The
|
||||
// test then asks for two new sizes of the source. The first must fetch it
|
||||
// again and cache it whole, so the second is made from the cache without a
|
||||
// third fetch.
|
||||
func TestProxy_Get_FetchesAgainForRemovedCachedSource(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
|
||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
wholeSize := fileSize(t, path)
|
||||
|
||||
// Removes the source as the reconciliation pass does: its row, then its file
|
||||
_, err := svc.db.ExecContext(t.Context(),
|
||||
`DELETE FROM source_content WHERE source_key = ?`, string(source.Key()))
|
||||
if err != nil {
|
||||
t.Fatalf("deleting the source's row: %v", err)
|
||||
}
|
||||
|
||||
err = os.Remove(path)
|
||||
if err != nil {
|
||||
t.Fatalf("removing the source's file: %v", err)
|
||||
}
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
|
||||
|
||||
fetches := readCounters(t, svc).upstreamFetchCount
|
||||
t.Logf("upstream fetches = %d", fetches)
|
||||
|
||||
if fetches != 2 {
|
||||
t.Errorf("upstream fetches = %d, want 2", fetches)
|
||||
}
|
||||
|
||||
size := fileSize(t, path)
|
||||
if size != wholeSize {
|
||||
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_FetchesAgainSourceOfNoSourceFormat replaces the cached
|
||||
// source's file with text. The next request for a new size of the source
|
||||
// must fetch it again and cache it whole.
|
||||
func TestProxy_Get_FetchesAgainSourceOfNoSourceFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
wholeSize := fileSize(t, path)
|
||||
|
||||
err := os.WriteFile(path, []byte("this is text, not an image"),
|
||||
imgcache.StorageFilePerm)
|
||||
if err != nil {
|
||||
t.Fatalf("replacing the source's file: %v", err)
|
||||
}
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
|
||||
fetches := readCounters(t, svc).upstreamFetchCount
|
||||
t.Logf("upstream fetches = %d", fetches)
|
||||
|
||||
if fetches != 2 {
|
||||
t.Errorf("upstream fetches = %d, want 2", fetches)
|
||||
}
|
||||
|
||||
size := fileSize(t, path)
|
||||
if size != wholeSize {
|
||||
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_ConvertsAgainForRemovedCachedVariant caches a variant of
|
||||
// the photo and removes its row and file, as the reconciliation pass does
|
||||
// when the file is empty or cut short. internal/imgcache tests that the
|
||||
// pass does so. The test then checks that Get converts the image again, and
|
||||
// that the next Get serves the whole variant from the cache.
|
||||
func TestProxy_Get_ConvertsAgainForRemovedCachedVariant(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
|
||||
want, _ := getImage(t, svc, req)
|
||||
|
||||
key := imagerequest.CacheKey(req)
|
||||
|
||||
// Removes the variant as the reconciliation pass does: its row, then its file
|
||||
_, err := svc.db.ExecContext(t.Context(),
|
||||
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
|
||||
if err != nil {
|
||||
t.Fatalf("deleting the variant's row: %v", err)
|
||||
}
|
||||
|
||||
err = os.Remove(variantFilePath(svc, key))
|
||||
if err != nil {
|
||||
t.Fatalf("removing the variant's file: %v", err)
|
||||
}
|
||||
|
||||
got, status := getImage(t, svc, req)
|
||||
t.Logf("after the removal: %d bytes, %s", len(got), status)
|
||||
|
||||
if status != CacheMiss || !bytes.Equal(got, want) {
|
||||
t.Errorf("after the removal: Get() = %d bytes, %s; want the %d bytes "+
|
||||
"converted again, %s", len(got), status, len(want), CacheMiss)
|
||||
}
|
||||
|
||||
got, status = getImage(t, svc, req)
|
||||
t.Logf("next request: %d bytes, %s", len(got), status)
|
||||
|
||||
if status != CacheHit || !bytes.Equal(got, want) {
|
||||
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
|
||||
len(got), status, len(want), CacheHit)
|
||||
}
|
||||
}
|
||||
@@ -1,78 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// pendingAccessTimesWait is how long a test waits for the database's
|
||||
// connection, for a cache hit to be answered, for an access time to reach
|
||||
// the database, or for a request to return past its deadline.
|
||||
const pendingAccessTimesWait = 5 * time.Second
|
||||
|
||||
// holdDatabase takes the one connection of db, so that every other query
|
||||
// waits for it, and returns the func that frees it.
|
||||
func holdDatabase(t *testing.T, db *sql.DB) func() {
|
||||
t.Helper()
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), pendingAccessTimesWait)
|
||||
defer cancel()
|
||||
|
||||
conn, err := db.Conn(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to take the database connection: %v", err)
|
||||
}
|
||||
|
||||
return func() { _ = conn.Close() }
|
||||
}
|
||||
|
||||
// TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
|
||||
// database's one connection while a request whose fetch is held reaches its
|
||||
// deadline. The request must return with the deadline's error while the
|
||||
// connection is still held, as it waits for no database write past its
|
||||
// deadline.
|
||||
func TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
|
||||
svc.cache.StartPendingAccessTimeWrites()
|
||||
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
|
||||
|
||||
// Room for the request to reach its fetch on a busy host
|
||||
const timeout = 2 * time.Second
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), timeout)
|
||||
defer cancel()
|
||||
|
||||
deadline, _ := ctx.Deadline()
|
||||
|
||||
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
// The request has made its database reads by the time it fetches.
|
||||
select {
|
||||
case <-fetcher.started:
|
||||
case <-time.After(timeout):
|
||||
t.Fatal("request did not reach its fetch by its deadline")
|
||||
}
|
||||
|
||||
releaseDatabase := holdDatabase(t, svc.db)
|
||||
defer releaseDatabase()
|
||||
|
||||
select {
|
||||
case got := <-results:
|
||||
t.Logf("Get() returned %v after its deadline, error = %v",
|
||||
time.Since(deadline), got.err)
|
||||
|
||||
if !errors.Is(got.err, context.DeadlineExceeded) {
|
||||
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
|
||||
}
|
||||
case <-time.After(time.Until(deadline) + pendingAccessTimesWait):
|
||||
t.Fatal("request did not return while the database was busy")
|
||||
}
|
||||
}
|
||||
@@ -1,245 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"image/color"
|
||||
"io"
|
||||
"io/fs"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestProxy_Get_CountsStats walks Get through a miss that fetches the
|
||||
// source, a hit, a miss that reuses the cached source, and two misses whose
|
||||
// source cannot be used, checking every counter of the proxy and its cache
|
||||
// after each.
|
||||
func TestProxy_Get_CountsStats(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
// NewTestFS builds the same files the test proxy's fetcher serves.
|
||||
testFS, _ := NewTestFS(t)
|
||||
|
||||
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
fake, err := fs.ReadFile(testFS, fixtures.InvalidFile)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
photoBytes, fakeBytes := int64(len(photo)), int64(len(fake))
|
||||
|
||||
// want is hits, misses, upstream fetches, upstream bytes, transforms.
|
||||
steps := []struct {
|
||||
name string
|
||||
path string
|
||||
size int
|
||||
wantErr bool
|
||||
want counterValues
|
||||
}{
|
||||
{"miss that fetches the source", testPathPhoto, 50, false,
|
||||
counterValues{0, 1, 1, photoBytes, 1}},
|
||||
{"hit", testPathPhoto, 50, false,
|
||||
counterValues{1, 1, 1, photoBytes, 1}},
|
||||
{"miss that reuses the cached source", testPathPhoto, 25, false,
|
||||
counterValues{1, 2, 1, photoBytes, 2}},
|
||||
{"miss whose source fails the magic byte check", "/images/fake.jpg", 50, true,
|
||||
counterValues{1, 3, 2, photoBytes + fakeBytes, 2}},
|
||||
{"miss whose source is not found", "/images/nonexistent.jpg", 50, true,
|
||||
counterValues{1, 4, 2, photoBytes + fakeBytes, 2}},
|
||||
}
|
||||
|
||||
for _, step := range steps {
|
||||
resp, err := svc.Get(t.Context(), &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: step.path,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: step.size, Height: step.size},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
})
|
||||
if (err != nil) != step.wantErr {
|
||||
t.Fatalf("%s: Get() error = %v, want error %t", step.name, err, step.wantErr)
|
||||
}
|
||||
|
||||
if err == nil {
|
||||
_ = resp.Content.Close()
|
||||
}
|
||||
|
||||
got := readCounters(t, svc)
|
||||
if got != step.want {
|
||||
t.Fatalf("after the %s: counters = %+v, want %+v", step.name, got, step.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// fakeUpstream answers every fetch with itself as a JPEG body. The body
|
||||
// serves data, then calls cancel, when set, and returns err; io.EOF ends
|
||||
// the body normally.
|
||||
type fakeUpstream struct {
|
||||
data *bytes.Reader
|
||||
cancel context.CancelFunc
|
||||
err error
|
||||
}
|
||||
|
||||
func (u *fakeUpstream) Fetch(
|
||||
context.Context, imagerequest.Source,
|
||||
) (*httpfetcher.FetchResult, error) {
|
||||
return &httpfetcher.FetchResult{
|
||||
Content: io.NopCloser(u),
|
||||
ContentLength: -1,
|
||||
ContentType: testContentTypeJPEG,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (u *fakeUpstream) Read(p []byte) (int, error) {
|
||||
if u.data.Len() > 0 {
|
||||
return u.data.Read(p)
|
||||
}
|
||||
|
||||
if u.cancel != nil {
|
||||
u.cancel()
|
||||
}
|
||||
|
||||
return 0, u.err
|
||||
}
|
||||
|
||||
// TestProxy_Get_CountsInterruptedMisses checks every counter of the proxy
|
||||
// and its cache after a miss whose request context ends during or after the
|
||||
// upstream fetch, and after a miss whose upstream body is over the size
|
||||
// limit.
|
||||
func TestProxy_Get_CountsInterruptedMisses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
photo := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
|
||||
half := len(photo) / 2
|
||||
|
||||
// want is hits, misses, upstream fetches, upstream bytes, transforms.
|
||||
tests := []struct {
|
||||
name string
|
||||
served int // bytes of the photo the upstream body serves
|
||||
cancel bool // whether the body then ends the request context
|
||||
readErr error // what the body then returns
|
||||
wantErr bool
|
||||
want counterValues
|
||||
}{
|
||||
{"request context ends during the fetch", half, true, context.Canceled, true,
|
||||
counterValues{0, 1, 1, int64(half), 0}},
|
||||
{"request context ends after the fetch", len(photo), true, io.EOF, false,
|
||||
counterValues{0, 1, 1, int64(len(photo)), 1}},
|
||||
{"upstream body over the size limit", half, false,
|
||||
httpfetcher.ErrResponseTooLarge, true,
|
||||
counterValues{0, 1, 1, int64(half), 0}},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
|
||||
upstream := &fakeUpstream{
|
||||
data: bytes.NewReader(photo[:tc.served]),
|
||||
err: tc.readErr,
|
||||
}
|
||||
if tc.cancel {
|
||||
upstream.cancel = cancel
|
||||
}
|
||||
|
||||
svc.fetcher = upstream
|
||||
|
||||
resp, err := svc.Get(ctx, &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
})
|
||||
t.Logf("Get() error = %v", err)
|
||||
|
||||
if (err != nil) != tc.wantErr {
|
||||
t.Fatalf("Get() error = %v, want error %t", err, tc.wantErr)
|
||||
}
|
||||
|
||||
if err == nil {
|
||||
_ = resp.Content.Close()
|
||||
}
|
||||
|
||||
got := readCounters(t, svc)
|
||||
if got != tc.want {
|
||||
t.Errorf("counters = %+v, want %+v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_CountsHitAfterRequestEnds checks every counter of the
|
||||
// proxy and its cache after a hit served with a request context that has
|
||||
// already ended: only the hit count moves.
|
||||
func TestProxy_Get_CountsHitAfterRequestEnds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// A first request caches the variant.
|
||||
resp, err := svc.Get(t.Context(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("first Get() error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
want := readCounters(t, svc)
|
||||
want.hitCount++
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
cancel()
|
||||
|
||||
resp, err = svc.Get(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() with an ended request context: error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
if resp.CacheStatus != CacheHit {
|
||||
t.Fatalf("CacheStatus = %v, want %v", resp.CacheStatus, CacheHit)
|
||||
}
|
||||
|
||||
got := readCounters(t, svc)
|
||||
if got != want {
|
||||
t.Errorf("counters = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -1,355 +0,0 @@
|
||||
package imageproxy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/gif"
|
||||
"image/jpeg"
|
||||
"image/png"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// Shared test data literals, extracted as constants for goconst.
|
||||
const (
|
||||
testContentTypeJPEG = "image/jpeg"
|
||||
)
|
||||
|
||||
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
|
||||
// produces, so injected timestamps compare correctly against ones the
|
||||
// cache writes.
|
||||
const sqliteTimestampFormat = "2006-01-02 15:04:05"
|
||||
|
||||
// TestFixtures contains paths to test files in the mock filesystem.
|
||||
type TestFixtures struct {
|
||||
// Valid image files
|
||||
GoodHostJPEG string // valid JPEG on GoodHost
|
||||
GoodHostPNG string // valid PNG on GoodHost
|
||||
GoodHostGIF string // valid GIF on GoodHost
|
||||
OtherHostJPEG string // valid JPEG on OtherHost
|
||||
OtherHostPNG string // valid PNG on OtherHost
|
||||
|
||||
// Invalid/edge case files
|
||||
InvalidFile string // file with wrong magic bytes
|
||||
EmptyFile string // zero-byte file
|
||||
TextFile string // text file masquerading as image
|
||||
|
||||
// Hostnames
|
||||
GoodHost string // a host the mock fetcher serves
|
||||
OtherHost string // a second host the mock fetcher serves
|
||||
}
|
||||
|
||||
// DefaultFixtures returns the standard test fixture paths.
|
||||
func DefaultFixtures() *TestFixtures {
|
||||
return &TestFixtures{
|
||||
GoodHostJPEG: "goodhost.example.com/images/photo.jpg",
|
||||
GoodHostPNG: "goodhost.example.com/images/logo.png",
|
||||
GoodHostGIF: "goodhost.example.com/images/animation.gif",
|
||||
OtherHostJPEG: "otherhost.example.com/uploads/image.jpg",
|
||||
OtherHostPNG: "otherhost.example.com/uploads/icon.png",
|
||||
InvalidFile: "goodhost.example.com/images/fake.jpg",
|
||||
EmptyFile: "goodhost.example.com/images/empty.jpg",
|
||||
TextFile: "goodhost.example.com/images/text.png",
|
||||
GoodHost: "goodhost.example.com",
|
||||
OtherHost: "otherhost.example.com",
|
||||
}
|
||||
}
|
||||
|
||||
// NewTestFS creates a mock filesystem with test images.
|
||||
func NewTestFS(t *testing.T) (fs.FS, *TestFixtures) {
|
||||
t.Helper()
|
||||
|
||||
fixtures := DefaultFixtures()
|
||||
|
||||
// Generate test images
|
||||
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
|
||||
pngData := generateTestPNG(t, 100, 100, color.RGBA{0, 255, 0, 255})
|
||||
gifData := generateTestGIF(t, 100, 100, color.RGBA{0, 0, 255, 255})
|
||||
|
||||
// Create the mock filesystem
|
||||
mockFS := fstest.MapFS{
|
||||
// Good host files
|
||||
fixtures.GoodHostJPEG: &fstest.MapFile{Data: jpegData},
|
||||
fixtures.GoodHostPNG: &fstest.MapFile{Data: pngData},
|
||||
fixtures.GoodHostGIF: &fstest.MapFile{Data: gifData},
|
||||
|
||||
// Other host files
|
||||
fixtures.OtherHostJPEG: &fstest.MapFile{Data: jpegData},
|
||||
fixtures.OtherHostPNG: &fstest.MapFile{Data: pngData},
|
||||
|
||||
// Invalid files
|
||||
fixtures.InvalidFile: &fstest.MapFile{Data: []byte("not a real image file content")},
|
||||
fixtures.EmptyFile: &fstest.MapFile{Data: []byte{}},
|
||||
fixtures.TextFile: &fstest.MapFile{Data: []byte("This is just text, not a PNG")},
|
||||
}
|
||||
|
||||
return mockFS, fixtures
|
||||
}
|
||||
|
||||
// generateTestJPEG creates a minimal valid JPEG image.
|
||||
func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
|
||||
t.Helper()
|
||||
|
||||
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||
for y := range height {
|
||||
for x := range width {
|
||||
img.Set(x, y, c)
|
||||
}
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to encode test JPEG: %v", err)
|
||||
}
|
||||
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
// generateTestPNG creates a minimal valid PNG image.
|
||||
func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
|
||||
t.Helper()
|
||||
|
||||
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||
for y := range height {
|
||||
for x := range width {
|
||||
img.Set(x, y, c)
|
||||
}
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
err := png.Encode(&buf, img)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to encode test PNG: %v", err)
|
||||
}
|
||||
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
// generateTestGIF creates a minimal valid GIF image.
|
||||
func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
|
||||
t.Helper()
|
||||
|
||||
img := image.NewPaletted(
|
||||
image.Rect(0, 0, width, height),
|
||||
[]color.Color{c, color.White},
|
||||
)
|
||||
for y := range height {
|
||||
for x := range width {
|
||||
img.SetColorIndex(x, y, 0)
|
||||
}
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
err := gif.Encode(&buf, img, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to encode test GIF: %v", err)
|
||||
}
|
||||
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
// testProxy is a Proxy with the database and the state directory of its
|
||||
// cache, through which a test reads and changes the cache behind the proxy.
|
||||
// registry is the metrics registry the proxy and its cache share.
|
||||
type testProxy struct {
|
||||
*Proxy
|
||||
|
||||
db *sql.DB
|
||||
stateDir string
|
||||
registry *prometheus.Registry
|
||||
}
|
||||
|
||||
// setupTestProxy creates a Proxy with mock fetcher for testing.
|
||||
func setupTestProxy(t *testing.T) (*testProxy, *TestFixtures) {
|
||||
t.Helper()
|
||||
|
||||
mockFS, fixtures := NewTestFS(t)
|
||||
|
||||
// Create temp directory for cache
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
// Create in-memory database
|
||||
db := setupTestDB(t)
|
||||
|
||||
registry := prometheus.NewRegistry()
|
||||
|
||||
cache, err := imgcache.NewCache(db, imgcache.Config{
|
||||
StateDir: tmpDir,
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: 1 << 30,
|
||||
}, registry)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
|
||||
proxy, err := New(&Config{
|
||||
Cache: cache,
|
||||
Fetcher: httpfetcher.NewMock(mockFS),
|
||||
Processor: imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 2,
|
||||
}),
|
||||
MaxResponseSize: 50 << 20,
|
||||
}, registry)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create proxy: %v", err)
|
||||
}
|
||||
|
||||
return &testProxy{
|
||||
Proxy: proxy,
|
||||
db: db,
|
||||
stateDir: tmpDir,
|
||||
registry: registry,
|
||||
}, fixtures
|
||||
}
|
||||
|
||||
// counterValues holds the values of the counters of a proxy and its cache.
|
||||
type counterValues struct {
|
||||
hitCount int64
|
||||
missCount int64
|
||||
upstreamFetchCount int64
|
||||
upstreamFetchBytes int64
|
||||
transformCount int64
|
||||
}
|
||||
|
||||
// readCounters reads the counters of svc and its cache from their metrics
|
||||
// registry.
|
||||
func readCounters(t *testing.T, svc *testProxy) counterValues {
|
||||
t.Helper()
|
||||
|
||||
families, err := svc.registry.Gather()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to gather the counters: %v", err)
|
||||
}
|
||||
|
||||
values := make(map[string]int64, len(families))
|
||||
|
||||
for _, family := range families {
|
||||
values[family.GetName()] = int64(family.GetMetric()[0].GetCounter().GetValue())
|
||||
}
|
||||
|
||||
return counterValues{
|
||||
hitCount: values["pixa_cache_hits_total"],
|
||||
missCount: values["pixa_cache_misses_total"],
|
||||
upstreamFetchCount: values["pixa_upstream_fetches_total"],
|
||||
upstreamFetchBytes: values["pixa_upstream_fetch_bytes_total"],
|
||||
transformCount: values["pixa_transforms_total"],
|
||||
}
|
||||
}
|
||||
|
||||
// sourceFilePath returns the path of the file of the source key in svc's
|
||||
// cache, as README.md gives it: <state_dir>/cache/sources/<ab>/<cd>/<key>.
|
||||
func sourceFilePath(svc *testProxy, key imagerequest.SourceKey) string {
|
||||
name := string(key)
|
||||
|
||||
return filepath.Join(svc.stateDir, "cache", "sources", name[0:2], name[2:4], name)
|
||||
}
|
||||
|
||||
// variantFilePath returns the path of the file of the variant key in svc's
|
||||
// cache, as README.md gives it: <state_dir>/cache/variants/<ab>/<cd>/<key>.
|
||||
func variantFilePath(svc *testProxy, key imagerequest.VariantKey) string {
|
||||
name := string(key)
|
||||
|
||||
return filepath.Join(svc.stateDir, "cache", "variants", name[0:2], name[2:4], name)
|
||||
}
|
||||
|
||||
// setVariantLastAccessed backdates the last access time of a tracked
|
||||
// variant.
|
||||
func setVariantLastAccessed(
|
||||
t *testing.T, db *sql.DB, key imagerequest.VariantKey, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
res, err := db.ExecContext(t.Context(),
|
||||
`UPDATE variant_content SET last_accessed_at = ? WHERE cache_key = ?`,
|
||||
when.UTC().Format(sqliteTimestampFormat), string(key),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set variant last_accessed_at: %v", err)
|
||||
}
|
||||
|
||||
affected, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read affected rows: %v", err)
|
||||
}
|
||||
|
||||
if affected != 1 {
|
||||
t.Fatalf("variant %s has no accounting row (affected=%d); "+
|
||||
"stores must track variants in the database", key, affected)
|
||||
}
|
||||
}
|
||||
|
||||
// setSourceLastAccessed backdates the last access time of a tracked
|
||||
// source content blob.
|
||||
func setSourceLastAccessed(
|
||||
t *testing.T, db *sql.DB, key imagerequest.SourceKey, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
res, err := db.ExecContext(t.Context(),
|
||||
`UPDATE source_content SET last_accessed_at = ? WHERE source_key = ?`,
|
||||
when.UTC().Format(sqliteTimestampFormat), string(key),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set source last_accessed_at: %v", err)
|
||||
}
|
||||
|
||||
affected, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read affected rows: %v", err)
|
||||
}
|
||||
|
||||
if affected != 1 {
|
||||
t.Fatalf("source %s has no accounting row (affected=%d)", key, affected)
|
||||
}
|
||||
}
|
||||
|
||||
// countRows returns the number of rows the given query yields.
|
||||
func countRows(t *testing.T, db *sql.DB, query string, args ...any) int {
|
||||
t.Helper()
|
||||
|
||||
var n int
|
||||
|
||||
err := db.QueryRowContext(t.Context(), query, args...).Scan(&n)
|
||||
if err != nil {
|
||||
t.Fatalf("count query %q failed: %v", query, err)
|
||||
}
|
||||
|
||||
return n
|
||||
}
|
||||
|
||||
// setupTestDB creates an in-memory SQLite database for testing
|
||||
// using the production schema.
|
||||
func setupTestDB(t *testing.T) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
db, err := sql.Open("sqlite", ":memory:")
|
||||
if err != nil {
|
||||
t.Fatalf("failed to open test db: %v", err)
|
||||
}
|
||||
|
||||
// Use the real production schema via migrations
|
||||
err = database.ApplyMigrations(context.Background(), db, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to apply migrations: %v", err)
|
||||
}
|
||||
|
||||
return db
|
||||
}
|
||||
@@ -1,23 +1,22 @@
|
||||
// Package imagerequest holds the image request type, its parts, its checks,
|
||||
// the parsing of the /v1/image/ path, the variant key, the source key and the
|
||||
// source's upstream URL. It also holds the image format type and the MIME type
|
||||
// of each format. It imports no other package of this repo, so every package
|
||||
// that handles an image request can import it.
|
||||
// the parsing of the /v1/image/ path, the variant key and the source key. It
|
||||
// imports no other package of this repo, so every package that handles an
|
||||
// image request can import it.
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/url"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
|
||||
var ErrInvalidFitMode = errors.New("invalid fit mode")
|
||||
|
||||
// Format is an image format. pixa reads and writes the formats from JPEG to
|
||||
// GIF below. In a request, orig and auto stand for one of them.
|
||||
// Format represents supported output image formats.
|
||||
type Format string
|
||||
|
||||
// Supported image formats.
|
||||
// Supported image output formats.
|
||||
const (
|
||||
FormatOriginal Format = "orig"
|
||||
FormatJPEG Format = "jpeg"
|
||||
@@ -33,30 +32,6 @@ const (
|
||||
FormatAuto Format = "auto"
|
||||
)
|
||||
|
||||
// MIMEType returns the MIME type of the format, such as image/jpeg for
|
||||
// FormatJPEG. orig and auto stand for another format, so they have none:
|
||||
// MIMEType returns "" for them.
|
||||
func (f Format) MIMEType() string {
|
||||
switch f {
|
||||
case FormatJPEG:
|
||||
return "image/jpeg"
|
||||
case FormatPNG:
|
||||
return "image/png"
|
||||
case FormatWebP:
|
||||
return "image/webp"
|
||||
case FormatAVIF:
|
||||
return "image/avif"
|
||||
case FormatJXL:
|
||||
return "image/jxl"
|
||||
case FormatGIF:
|
||||
return "image/gif"
|
||||
case FormatOriginal, FormatAuto:
|
||||
return ""
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// Size represents requested image dimensions
|
||||
type Size struct {
|
||||
Width int
|
||||
@@ -110,13 +85,19 @@ func ValidateImageRequest(req *Request) error {
|
||||
}
|
||||
|
||||
// Request represents a request for a processed image: the source to fetch
|
||||
// and the transform to apply to it. It holds exactly what the variant key
|
||||
// covers.
|
||||
// and the transform to apply to it.
|
||||
type Request struct {
|
||||
// Source is what to fetch
|
||||
Source Source
|
||||
// Transform is what to do to it
|
||||
Transform Transform
|
||||
// Signature is the HMAC signature for non-allowlisted hosts
|
||||
Signature string
|
||||
// Expires is when the URL expires: the exp of a signed URL, or the expiry
|
||||
// of an encrypted URL; the zero time if it has none
|
||||
Expires time.Time
|
||||
// AllowHTTP indicates whether HTTP (non-TLS) is allowed for this request
|
||||
AllowHTTP bool
|
||||
}
|
||||
|
||||
// Source is the image a request fetches from the origin.
|
||||
@@ -144,20 +125,23 @@ type Transform struct {
|
||||
FitMode Fit
|
||||
}
|
||||
|
||||
// URL returns the source's upstream URL, with the scheme http when allowHTTP
|
||||
// is true and https otherwise. net/url writes its path percent-encoded again,
|
||||
// so each source path is one upstream path. Its query is the source query as
|
||||
// it is, with the characters escapeQuery encodes percent-encoded.
|
||||
func (s Source) URL(allowHTTP bool) *url.URL {
|
||||
// SourceURL returns the full upstream URL to fetch: the source path
|
||||
// percent-encoded again, as net/url writes a path, so each source path is one
|
||||
// upstream path, and the source query as it is, with the characters
|
||||
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
|
||||
// true, otherwise https://.
|
||||
func (r *Request) SourceURL() string {
|
||||
scheme := "https"
|
||||
if allowHTTP {
|
||||
if r.AllowHTTP {
|
||||
scheme = "http"
|
||||
}
|
||||
|
||||
return &url.URL{
|
||||
sourceURL := url.URL{
|
||||
Scheme: scheme,
|
||||
Host: s.Host,
|
||||
Path: s.Path,
|
||||
RawQuery: escapeQuery(s.Query),
|
||||
Host: r.Source.Host,
|
||||
Path: r.Source.Path,
|
||||
RawQuery: escapeQuery(r.Source.Query),
|
||||
}
|
||||
|
||||
return sourceURL.String()
|
||||
}
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
package imagerequest
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestFormat_MIMEType checks the MIME type of each format pixa reads and
|
||||
// writes, and that orig and auto have none.
|
||||
func TestFormat_MIMEType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
format Format
|
||||
want string
|
||||
}{
|
||||
{FormatJPEG, "image/jpeg"},
|
||||
{FormatPNG, "image/png"},
|
||||
{FormatWebP, "image/webp"},
|
||||
{FormatAVIF, "image/avif"},
|
||||
{FormatJXL, "image/jxl"},
|
||||
{FormatGIF, "image/gif"},
|
||||
{FormatOriginal, ""},
|
||||
{FormatAuto, ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
got := tt.format.MIMEType()
|
||||
if got != tt.want {
|
||||
t.Errorf("%s MIMEType() = %q, want %q", tt.format, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,49 +2,57 @@ package imagerequest
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestSource_URL_DefaultHTTPS(t *testing.T) {
|
||||
func TestRequest_SourceURL_DefaultHTTPS(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
source := Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
Query: "v=2",
|
||||
req := &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
Query: "v=2",
|
||||
},
|
||||
}
|
||||
|
||||
got := source.URL(false).String()
|
||||
got := req.SourceURL()
|
||||
|
||||
want := "https://cdn.example.com/photos/cat.jpg?v=2"
|
||||
if got != want {
|
||||
t.Errorf("URL() = %q, want %q", got, want)
|
||||
t.Errorf("SourceURL() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSource_URL_AllowHTTP(t *testing.T) {
|
||||
func TestRequest_SourceURL_AllowHTTP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
source := Source{
|
||||
Host: "localhost:8080",
|
||||
Path: testPathCat,
|
||||
req := &Request{
|
||||
Source: Source{
|
||||
Host: "localhost:8080",
|
||||
Path: testPathCat,
|
||||
},
|
||||
AllowHTTP: true,
|
||||
}
|
||||
|
||||
got := source.URL(true).String()
|
||||
got := req.SourceURL()
|
||||
|
||||
want := "http://localhost:8080/photos/cat.jpg"
|
||||
if got != want {
|
||||
t.Errorf("URL() = %q, want %q", got, want)
|
||||
t.Errorf("SourceURL() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSource_URL_AllowHTTPFalse(t *testing.T) {
|
||||
func TestRequest_SourceURL_AllowHTTPFalse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
source := Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/img.jpg",
|
||||
req := &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/img.jpg",
|
||||
},
|
||||
AllowHTTP: false,
|
||||
}
|
||||
|
||||
got := source.URL(false).String()
|
||||
got := req.SourceURL()
|
||||
if got != "https://cdn.example.com/img.jpg" {
|
||||
t.Errorf("URL() = %q, want https scheme", got)
|
||||
t.Errorf("SourceURL() = %q, want https scheme", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -66,6 +66,23 @@ var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#'<>%]`)
|
||||
var hostRegex = regexp.MustCompile(
|
||||
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
|
||||
|
||||
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
|
||||
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
|
||||
// Examples:
|
||||
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
||||
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
||||
// - cdn.example.com/photos/cat.jpg/orig.png
|
||||
// - cdn.example.com/photos/cat.jpg/800x600
|
||||
func ParseImagePath(path string) (*Request, error) {
|
||||
// Strip leading slash if present (chi may include it)
|
||||
path = strings.TrimPrefix(path, "/")
|
||||
if path == "" {
|
||||
return nil, ErrMissingHost
|
||||
}
|
||||
|
||||
return parseImageComponents(path)
|
||||
}
|
||||
|
||||
// ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds
|
||||
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
|
||||
// for JPEG XL. A source query follows the source path after a "?", which the
|
||||
@@ -126,15 +143,15 @@ func parseImageComponents(remainder string) (*Request, error) {
|
||||
return nil, ErrMissingHost
|
||||
}
|
||||
|
||||
// Later steps put the host unparsed into the allowlist check, the cache key
|
||||
// and the upstream URL, so only the forms hostRegex describes are accepted
|
||||
// Only the forms hostRegex describes are accepted. A "?", "#" or "@" in
|
||||
// the host would make Go's URL parser, which the allowlist check and the
|
||||
// fetcher use, read a different host from the one allowlisted and cached
|
||||
hostMatch := hostRegex.FindStringSubmatch(host)
|
||||
if hostMatch == nil {
|
||||
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
|
||||
}
|
||||
|
||||
// The allowlist check and the fetcher drop the brackets, so brackets
|
||||
// around anything but an IPv6 address would give a host a second spelling
|
||||
// Go's URL parser refuses brackets around anything but an IPv6 address
|
||||
if inBrackets := hostMatch[1]; inBrackets != "" {
|
||||
addr, err := netip.ParseAddr(inBrackets)
|
||||
if err != nil || !addr.Is6() {
|
||||
@@ -147,8 +164,9 @@ func parseImageComponents(remainder string) (*Request, error) {
|
||||
// host
|
||||
host = strings.ToLower(host)
|
||||
|
||||
// A "#" starts a URL's fragment, so neither the source path nor the source
|
||||
// query may hold one
|
||||
// Go's URL parser, which the fetcher uses, reads a "#" as the start of a
|
||||
// fragment and drops the rest, so each text after it would be another
|
||||
// spelling, cached under its own keys, of one upstream request
|
||||
if strings.Contains(path, "#") {
|
||||
return nil, fmt.Errorf(
|
||||
`%w: the source path or query holds a "#"`, ErrInvalidPath)
|
||||
|
||||
@@ -192,16 +192,76 @@ func TestParseImageURL_Errors(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseImageURL_LowerCasesHost verifies that the host is lower-cased, so
|
||||
func TestParseImagePath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// ParseImagePath is for chi wildcard capture (no /v1/image/ prefix)
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want *Request
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "chi wildcard capture",
|
||||
input: "cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
want: &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "with leading slash from chi",
|
||||
input: "/cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
want: &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := ParseImagePath(tt.input)
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Errorf("ParseImagePath() error = %v, wantErr %v", err, tt.wantErr)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
assertRequest(t, got, tt.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseImagePath_LowerCasesHost verifies that the host is lower-cased, so
|
||||
// that every spelling of its letter case is the same host to the allowlist,
|
||||
// the upstream connection limits and the cache, and that the path keeps its
|
||||
// case.
|
||||
func TestParseImageURL_LowerCasesHost(t *testing.T) {
|
||||
func TestParseImagePath_LowerCasesHost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := ParseImageURL("/v1/image/CDN.Example.COM/Photos/Cat.jpg/800x600.webp")
|
||||
got, err := ParseImagePath("CDN.Example.COM/Photos/Cat.jpg/800x600.webp")
|
||||
if err != nil {
|
||||
t.Fatalf("ParseImageURL() error = %v", err)
|
||||
t.Fatalf("ParseImagePath() error = %v", err)
|
||||
}
|
||||
|
||||
t.Logf("parsed host %q, path %q", got.Source.Host, got.Source.Path)
|
||||
@@ -215,21 +275,21 @@ func TestParseImageURL_LowerCasesHost(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseImageURL_RefusesUserinfo verifies that a host holding "@" is
|
||||
// TestParseImagePath_RefusesUserinfo verifies that a host holding "@" is
|
||||
// refused: a URL reads what comes before the "@" as a user name and password
|
||||
// and the rest as the host.
|
||||
func TestParseImageURL_RefusesUserinfo(t *testing.T) {
|
||||
func TestParseImagePath_RefusesUserinfo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, input := range []string{
|
||||
"/v1/image/user@cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
"/v1/image/user:password@cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
"user@cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
"user:password@cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
} {
|
||||
got, err := ParseImageURL(input)
|
||||
t.Logf("ParseImageURL(%q) = %+v, error = %v", input, got, err)
|
||||
got, err := ParseImagePath(input)
|
||||
t.Logf("ParseImagePath(%q) = %+v, error = %v", input, got, err)
|
||||
|
||||
if !errors.Is(err, ErrInvalidHost) {
|
||||
t.Errorf("ParseImageURL(%q) error = %v, want %v",
|
||||
t.Errorf("ParseImagePath(%q) error = %v, want %v",
|
||||
input, err, ErrInvalidHost)
|
||||
}
|
||||
}
|
||||
@@ -255,10 +315,6 @@ func TestParseImageURL_PathTraversal(t *testing.T) {
|
||||
name: "parent in middle of path",
|
||||
input: "/v1/image/cdn.example.com/photos/../../../etc/passwd/800x600.jpeg",
|
||||
},
|
||||
{
|
||||
name: "parent directory in path",
|
||||
input: "/v1/image/cdn.example.com/photos/../secret/800x600.jpeg",
|
||||
},
|
||||
{
|
||||
name: "encoded parent directory",
|
||||
input: "/v1/image/cdn.example.com/photos/%2e%2e/secret/800x600.jpeg",
|
||||
@@ -301,6 +357,40 @@ func TestParseImageURL_PathTraversal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseImagePath_PathTraversal(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Test path traversal via ParseImagePath (chi wildcard)
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
}{
|
||||
{
|
||||
name: "parent directory in path",
|
||||
input: "cdn.example.com/photos/../secret/800x600.jpeg",
|
||||
},
|
||||
{
|
||||
name: "encoded traversal",
|
||||
input: "cdn.example.com/photos/%2e%2e/secret/800x600.jpeg",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := ParseImagePath(tt.input)
|
||||
if err == nil {
|
||||
t.Error("ParseImagePath() should reject path traversal attempts")
|
||||
}
|
||||
|
||||
if !errors.Is(err, ErrPathTraversal) {
|
||||
t.Errorf("ParseImagePath() error = %v, want ErrPathTraversal", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
|
||||
// a source query holding %25, the percent-escape of a literal "%", are
|
||||
// accepted as they are, although decoding them once more leaves a "%" that
|
||||
|
||||
+469
-233
@@ -1,10 +1,3 @@
|
||||
// Package imgcache is the image cache. It keeps:
|
||||
// - the source images fetched from upstream
|
||||
// - the variants made from them
|
||||
// - the negative cache: the source URLs whose fetch failed recently
|
||||
//
|
||||
// Each image is a file with a row in the database. Eviction removes the
|
||||
// least recently used images to keep the cache under its size limit.
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
@@ -15,18 +8,24 @@ import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
lru "github.com/hashicorp/golang-lru/v2"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// maxPendingAccessTimes is how many access times of variants served from
|
||||
// the cache, and how many of cached sources a miss used, may each wait to be
|
||||
// written, at about 80 bytes each. An access time handed over while that
|
||||
// many of its kind wait is dropped.
|
||||
const maxPendingAccessTimes = 10000
|
||||
// Cache errors.
|
||||
var (
|
||||
ErrCacheMiss = errors.New("cache miss")
|
||||
ErrNegativeCache = errors.New("negative cache hit")
|
||||
)
|
||||
|
||||
// metaCacheSize is how many variants' content types metaCache holds. A
|
||||
// variant not among them is served as before, reading its .meta file.
|
||||
const metaCacheSize = 10000
|
||||
|
||||
// cleanExpiredBatchSize is the most expired negative cache rows one
|
||||
// statement of CleanExpired deletes. Each batch is a statement of its
|
||||
@@ -34,22 +33,29 @@ const maxPendingAccessTimes = 10000
|
||||
// have expired.
|
||||
const cleanExpiredBatchSize = 1000
|
||||
|
||||
// DefaultNegativeTTL is how long pixad keeps a failed fetch in the negative
|
||||
// cache.
|
||||
const DefaultNegativeTTL = 5 * time.Minute
|
||||
|
||||
// Config holds cache configuration.
|
||||
type Config struct {
|
||||
// CacheConfig holds cache configuration.
|
||||
type CacheConfig struct {
|
||||
StateDir string
|
||||
NegativeTTL time.Duration
|
||||
|
||||
// MaxBytes is the disk cache size limit in bytes that eviction
|
||||
// enforces. Zero turns the disk cache off entirely: no cache
|
||||
// directories are created, lookups always miss, stores are no-ops,
|
||||
// and the eviction goroutine only removes the expired negative cache
|
||||
// rows.
|
||||
// enforces. Zero means no limit is enforced (no eviction).
|
||||
MaxBytes int64
|
||||
|
||||
// UseDefaultMaxBytes makes NewCache replace MaxBytes with the
|
||||
// default limit: 75% of the sum of the space free on the filesystem
|
||||
// holding the cache and the bytes the cache already holds, at least
|
||||
// DefaultCacheMaxBytesFloor. The config layer sets this when the
|
||||
// operator omits cache_max_bytes.
|
||||
UseDefaultMaxBytes bool
|
||||
|
||||
// DisableDiskCache turns the disk cache off entirely: no cache
|
||||
// directories are created, lookups always miss, stores are
|
||||
// no-ops, and the eviction goroutine only removes the expired
|
||||
// negative cache rows. The config layer sets this when the
|
||||
// operator configures cache_max_bytes: 0.
|
||||
DisableDiskCache bool
|
||||
|
||||
// Logger receives accounting and eviction log output. A nil
|
||||
// Logger means slog.Default().
|
||||
Logger *slog.Logger
|
||||
@@ -58,11 +64,15 @@ type Config struct {
|
||||
// Cache implements the caching layer for the image proxy.
|
||||
type Cache struct {
|
||||
db *sql.DB
|
||||
srcContent *FileStore // source images by source key
|
||||
variants *FileStore // processed variants by cache key
|
||||
config Config
|
||||
srcContent *ContentStorage // source images by source key
|
||||
variants *VariantStorage // processed variants by cache key
|
||||
config CacheConfig
|
||||
log *slog.Logger
|
||||
|
||||
// disabled means the disk cache is turned off entirely: lookups
|
||||
// always miss, stores are no-ops, and no eviction runs.
|
||||
disabled bool
|
||||
|
||||
// Eviction machinery. The channels are created in NewCache so
|
||||
// stores can signal write pressure without racing StartEviction.
|
||||
// evictionCancel, set by StartEviction, cancels the eviction
|
||||
@@ -71,27 +81,34 @@ type Cache struct {
|
||||
evictionDone chan struct{}
|
||||
evictionCancel context.CancelFunc
|
||||
|
||||
// pendingVariantAccessTimes holds the variants served from the cache,
|
||||
// and pendingSourceAccessTimes the cached sources a miss used, whose
|
||||
// access time is still to be written by the goroutine that
|
||||
// StartPendingAccessTimeWrites starts, so that no request waits for the
|
||||
// database to record it. As for eviction, the channels are created in
|
||||
// NewCache, so that requests can hand over access times before that
|
||||
// goroutine starts; pendingAccessTimesDone is closed when it returns.
|
||||
// pendingAccessTimesCancel, set by StartPendingAccessTimeWrites,
|
||||
// cancels its context, which tells it to finish.
|
||||
pendingVariantAccessTimes chan imagerequest.VariantKey
|
||||
pendingSourceAccessTimes chan imagerequest.SourceKey
|
||||
pendingAccessTimesDone chan struct{}
|
||||
pendingAccessTimesCancel context.CancelFunc
|
||||
// The pending counts: hits, misses, upstream fetches and transforms
|
||||
// whose write to the database missed its request's deadline, kept
|
||||
// here until they are written by the goroutine that
|
||||
// StartPendingCountWrites starts. As for eviction, the channels are
|
||||
// created in NewCache: pendingCountsAdded wakes that goroutine, and
|
||||
// pendingCountsDone is closed when it returns. pendingCountsCancel,
|
||||
// set by StartPendingCountWrites, cancels its context, which tells it
|
||||
// to finish.
|
||||
pendingHits atomic.Int64
|
||||
pendingMisses atomic.Int64
|
||||
pendingUpstreamFetches atomic.Int64
|
||||
pendingUpstreamFetchBytes atomic.Int64
|
||||
pendingTransforms atomic.Int64
|
||||
pendingCountsAdded chan struct{}
|
||||
pendingCountsDone chan struct{}
|
||||
pendingCountsCancel context.CancelFunc
|
||||
|
||||
// The counts of cache hits and misses, made and registered by
|
||||
// registerCounters (see counters.go)
|
||||
cacheHits prometheus.Counter
|
||||
cacheMisses prometheus.Counter
|
||||
// Held through each write of the pending counts, and by
|
||||
// readHitAndMissCounts to read them
|
||||
pendingCountsWriteMutex sync.Mutex
|
||||
|
||||
// metaCache holds the content types of the variants most recently
|
||||
// stored or served, so a hit does not read the variant's .meta file.
|
||||
// It never stands in for the variant file, which is always opened.
|
||||
metaCache *lru.Cache[imagerequest.VariantKey, string]
|
||||
|
||||
// reconciliationPageSize is the most rows one read of a content table
|
||||
// returns in the reconciliation pass. NewCache sets it to
|
||||
// returns in the reconciliation pass and in Stats. newCache sets it to
|
||||
// defaultReconciliationPageSize; tests set it smaller.
|
||||
reconciliationPageSize int
|
||||
|
||||
@@ -102,49 +119,54 @@ type Cache struct {
|
||||
reconciliationReadTestHook func(rows int)
|
||||
}
|
||||
|
||||
// NewCache creates a new cache instance and registers its counters on
|
||||
// metricsRegistry. pixad passes Prometheus' default registry, the one
|
||||
// /metrics serves. It makes no database call.
|
||||
func NewCache(
|
||||
db *sql.DB, config Config, metricsRegistry prometheus.Registerer,
|
||||
// NewCache creates a new cache instance.
|
||||
func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
|
||||
return newCache(db, config, defaultFreeSpaceProbe)
|
||||
}
|
||||
|
||||
// newCache is NewCache with the free-space probe passed in, so tests
|
||||
// can fake the free space the default limit is worked out from.
|
||||
func newCache(
|
||||
db *sql.DB, config CacheConfig, probe FreeSpaceProbeFunc,
|
||||
) (*Cache, error) {
|
||||
log := config.Logger
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
|
||||
metaCache, err := lru.New[imagerequest.VariantKey, string](metaCacheSize)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create variant content type cache: %w", err)
|
||||
}
|
||||
|
||||
c := &Cache{
|
||||
db: db,
|
||||
config: config,
|
||||
log: log,
|
||||
disabled: config.DisableDiskCache,
|
||||
evictionPressure: make(chan struct{}, 1),
|
||||
evictionDone: make(chan struct{}),
|
||||
metaCache: metaCache,
|
||||
|
||||
pendingVariantAccessTimes: make(chan imagerequest.VariantKey, maxPendingAccessTimes),
|
||||
pendingSourceAccessTimes: make(chan imagerequest.SourceKey, maxPendingAccessTimes),
|
||||
pendingAccessTimesDone: make(chan struct{}),
|
||||
pendingCountsAdded: make(chan struct{}, 1),
|
||||
pendingCountsDone: make(chan struct{}),
|
||||
|
||||
reconciliationPageSize: defaultReconciliationPageSize,
|
||||
}
|
||||
|
||||
err := c.registerCounters(metricsRegistry)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if config.MaxBytes == 0 {
|
||||
if c.disabled {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
srcContent, err := NewFileStore(
|
||||
srcContent, err := NewContentStorage(
|
||||
filepath.Join(config.StateDir, "cache", "sources"),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create source content storage: %w", err)
|
||||
}
|
||||
|
||||
variants, err := NewFileStore(
|
||||
filepath.Join(config.StateDir, "cache", "variants"),
|
||||
variants, err := NewVariantStorage(
|
||||
filepath.Join(config.StateDir, "cache", "variants"), log,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create variant storage: %w", err)
|
||||
@@ -153,137 +175,98 @@ func NewCache(
|
||||
c.srcContent = srcContent
|
||||
c.variants = variants
|
||||
|
||||
if config.UseDefaultMaxBytes {
|
||||
limit, err := c.computeDefaultMaxBytes(context.Background(), probe)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
c.config.MaxBytes = limit
|
||||
|
||||
log.Info("computed default cache size limit from free space and cache contents",
|
||||
"cache_max_bytes", limit,
|
||||
"cache_dir", filepath.Join(config.StateDir, "cache"),
|
||||
)
|
||||
}
|
||||
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Variant opens a cached variant and returns a reader, its size and its
|
||||
// content type, read from the variant file's first bytes. It opens that one
|
||||
// file and makes no database call. A variant it opens has its access time
|
||||
// handed to the background writer without waiting (see touchVariant). A
|
||||
// variant not in the cache, or any variant of a disabled cache, is
|
||||
// ErrNotFound; a variant file whose first bytes are not those of an output
|
||||
// format is another error.
|
||||
func (c *Cache) Variant(
|
||||
// LookupResult contains the result of a cache lookup.
|
||||
type LookupResult struct {
|
||||
Hit bool
|
||||
CacheKey imagerequest.VariantKey
|
||||
ContentType string
|
||||
SizeBytes int64
|
||||
CacheStatus CacheStatus
|
||||
}
|
||||
|
||||
// Lookup checks if a processed variant exists on disk. Hits touch the
|
||||
// variant's LRU timestamp; a disabled cache always misses.
|
||||
func (c *Cache) Lookup(
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (*LookupResult, error) {
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
|
||||
// Check variant storage directly - no DB needed for cache hits
|
||||
if !c.disabled && c.variants.Exists(cacheKey) {
|
||||
c.touchVariant(ctx, cacheKey)
|
||||
|
||||
return &LookupResult{
|
||||
Hit: true,
|
||||
CacheKey: cacheKey,
|
||||
CacheStatus: CacheHit,
|
||||
}, nil
|
||||
}
|
||||
|
||||
return &LookupResult{
|
||||
Hit: false,
|
||||
CacheKey: cacheKey,
|
||||
CacheStatus: CacheMiss,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// GetVariant returns a reader, size, and content type for a cached
|
||||
// variant. The content type comes from metaCache, or else from the
|
||||
// variant's .meta file and is then kept in metaCache. A variant with
|
||||
// no .meta file is served as application/octet-stream, which is not
|
||||
// kept.
|
||||
func (c *Cache) GetVariant(
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
if c.disabled {
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
file, size, err := c.variants.Open(string(cacheKey))
|
||||
if err != nil {
|
||||
return nil, 0, "", err
|
||||
contentType, known := c.metaCache.Get(cacheKey)
|
||||
if !known {
|
||||
return c.loadVariantWithMeta(cacheKey)
|
||||
}
|
||||
|
||||
format, err := readFormat(file)
|
||||
reader, size, err := c.variants.LoadWithSize(cacheKey)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
// The file is gone, e.g. deleted outside pixa
|
||||
c.metaCache.Remove(cacheKey)
|
||||
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
c.touchVariant(cacheKey)
|
||||
|
||||
return file, size, format.MIMEType(), nil
|
||||
return reader, size, contentType, nil
|
||||
}
|
||||
|
||||
// PutVariant stores a processed variant by its cache key and records
|
||||
// it in the size accounting. On a disabled cache it is a no-op. The
|
||||
// accounting insert is best-effort (the startup reconciliation pass
|
||||
// adopts any variant file that misses its accounting row).
|
||||
func (c *Cache) PutVariant(
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey, content io.Reader,
|
||||
) error {
|
||||
if c.config.MaxBytes == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
data, err := io.ReadAll(content)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to read content: %w", err)
|
||||
}
|
||||
|
||||
err = c.variants.Write(string(cacheKey), data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
size := int64(len(data))
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO variant_content (cache_key, size_bytes)
|
||||
VALUES (?, ?)
|
||||
ON CONFLICT(cache_key) DO UPDATE SET
|
||||
size_bytes = excluded.size_bytes,
|
||||
last_accessed_at = CURRENT_TIMESTAMP
|
||||
`, string(cacheKey), size)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to record variant in size accounting",
|
||||
"cache_key", cacheKey, "error", err)
|
||||
}
|
||||
|
||||
c.notifyWritePressure()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Source opens the cached source of src and returns a reader, its size in
|
||||
// bytes and its format, read from the file's first bytes. A source whose row
|
||||
// or file is missing is ErrNotFound, as is any source of a disabled cache; a
|
||||
// file whose first bytes are not those of a source format is another error.
|
||||
// A source it opens has its access time handed to the background writer
|
||||
// without waiting (see touchSourceContent).
|
||||
func (c *Cache) Source(
|
||||
ctx context.Context, src imagerequest.Source,
|
||||
) (io.ReadCloser, int64, imagerequest.Format, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
key := src.Key()
|
||||
|
||||
var rowExists bool
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT EXISTS(SELECT 1 FROM source_content WHERE source_key = ?)
|
||||
`, string(key)).Scan(&rowExists)
|
||||
if err != nil {
|
||||
return nil, 0, "", fmt.Errorf("failed to lookup source: %w", err)
|
||||
}
|
||||
|
||||
if !rowExists {
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
file, size, err := c.srcContent.Open(string(key))
|
||||
if err != nil {
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
format, err := readFormat(file)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
c.touchSourceContent(key)
|
||||
|
||||
return file, size, format, nil
|
||||
}
|
||||
|
||||
// PutSource stores fetched source content: it writes the file named by
|
||||
// StoreSource stores fetched source content: it writes the file named by
|
||||
// the source's key, then the source's row. When the row cannot be written
|
||||
// the file is removed and the store fails, as a file with no row is never
|
||||
// served or evicted; if that removal fails too, the next reconciliation
|
||||
// pass removes the file. On a disabled cache it is a no-op returning an
|
||||
// empty key.
|
||||
func (c *Cache) PutSource(
|
||||
func (c *Cache) StoreSource(
|
||||
ctx context.Context,
|
||||
src imagerequest.Source,
|
||||
req *imagerequest.Request,
|
||||
content io.Reader,
|
||||
result *httpfetcher.FetchResult,
|
||||
) (imagerequest.SourceKey, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
if c.disabled {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
@@ -292,24 +275,27 @@ func (c *Cache) PutSource(
|
||||
return "", fmt.Errorf("failed to read source content: %w", err)
|
||||
}
|
||||
|
||||
key := src.Key()
|
||||
key := req.Source.Key()
|
||||
|
||||
err = c.srcContent.Write(string(key), data)
|
||||
size, err := c.srcContent.Store(key, data)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to store source content: %w", err)
|
||||
}
|
||||
|
||||
size := int64(len(data))
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO source_content (source_key, size_bytes)
|
||||
VALUES (?, ?)
|
||||
INSERT INTO source_content
|
||||
(source_key, source_host, source_path, source_query,
|
||||
content_type, size_bytes)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(source_key) DO UPDATE SET
|
||||
content_type = excluded.content_type,
|
||||
size_bytes = excluded.size_bytes,
|
||||
fetched_at = CURRENT_TIMESTAMP,
|
||||
last_accessed_at = CURRENT_TIMESTAMP
|
||||
`, string(key), size)
|
||||
`, string(key), req.Source.Host, req.Source.Path, req.Source.Query,
|
||||
result.ContentType, size)
|
||||
if err != nil {
|
||||
deleteErr := c.srcContent.Delete(string(key))
|
||||
deleteErr := c.srcContent.Delete(key)
|
||||
if deleteErr != nil {
|
||||
c.log.Warn("failed to remove source file whose row was not written",
|
||||
"source_key", key, "error", deleteErr)
|
||||
@@ -323,43 +309,98 @@ func (c *Cache) PutSource(
|
||||
return key, nil
|
||||
}
|
||||
|
||||
// Failure reports whether src is in the negative cache. It only reads: a
|
||||
// row past its expiry is not a hit, and the eviction goroutine deletes it
|
||||
// with the other expired rows (see CleanExpired).
|
||||
func (c *Cache) Failure(
|
||||
ctx context.Context, src imagerequest.Source,
|
||||
) (bool, error) {
|
||||
var expiresAt time.Time
|
||||
// StoreVariant stores a processed variant by its cache key and records
|
||||
// it in the size accounting. On a disabled cache it is a no-op. The
|
||||
// accounting insert is best-effort (the startup reconciliation pass
|
||||
// adopts any variant file that misses its accounting row).
|
||||
func (c *Cache) StoreVariant(
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey,
|
||||
content io.Reader, contentType string,
|
||||
) error {
|
||||
if c.disabled {
|
||||
return nil
|
||||
}
|
||||
|
||||
size, err := c.variants.Store(cacheKey, content, contentType)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
c.metaCache.Add(cacheKey, contentType)
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(cache_key) DO UPDATE SET
|
||||
size_bytes = excluded.size_bytes,
|
||||
content_type = excluded.content_type,
|
||||
last_accessed_at = CURRENT_TIMESTAMP
|
||||
`, string(cacheKey), size, contentType)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to record variant in size accounting",
|
||||
"cache_key", cacheKey, "error", err)
|
||||
}
|
||||
|
||||
c.notifyWritePressure()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// LookupSource checks if we have cached source content for a request.
|
||||
// Returns the source key and content type if found, or empty values
|
||||
// if not; a row whose file is missing is not found. Hits touch the
|
||||
// source's LRU timestamp; a disabled cache always reports no cached
|
||||
// source.
|
||||
func (c *Cache) LookupSource(
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (imagerequest.SourceKey, string, error) {
|
||||
if c.disabled {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
key := req.Source.Key()
|
||||
|
||||
var contentType string
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT expires_at FROM negative_cache
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, src.Host, src.Path, src.Query).Scan(&expiresAt)
|
||||
SELECT content_type FROM source_content WHERE source_key = ?
|
||||
`, string(key)).Scan(&contentType)
|
||||
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return false, nil
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("failed to check negative cache: %w", err)
|
||||
return "", "", fmt.Errorf("failed to lookup source: %w", err)
|
||||
}
|
||||
|
||||
expired := time.Now().After(expiresAt)
|
||||
// Verify the content file exists
|
||||
if !c.srcContent.Exists(key) {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
return !expired, nil
|
||||
c.touchSourceContent(ctx, key)
|
||||
|
||||
return key, contentType, nil
|
||||
}
|
||||
|
||||
// PutFailure stores a negative cache entry for src, whose fetch failed.
|
||||
func (c *Cache) PutFailure(ctx context.Context, src imagerequest.Source) error {
|
||||
// StoreNegative stores a negative cache entry for a failed fetch.
|
||||
func (c *Cache) StoreNegative(
|
||||
ctx context.Context, req *imagerequest.Request, statusCode int, errMsg string,
|
||||
) error {
|
||||
expiresAt := time.Now().UTC().Add(c.config.NegativeTTL)
|
||||
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
INSERT INTO negative_cache
|
||||
(source_host, source_path, source_query, expires_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
(source_host, source_path, source_query, status_code,
|
||||
error_message, expires_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET
|
||||
status_code = excluded.status_code,
|
||||
error_message = excluded.error_message,
|
||||
fetched_at = CURRENT_TIMESTAMP,
|
||||
expires_at = excluded.expires_at
|
||||
`, src.Host, src.Path, src.Query, expiresAt)
|
||||
`, req.Source.Host, req.Source.Path, req.Source.Query, statusCode, errMsg, expiresAt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to insert negative cache: %w", err)
|
||||
}
|
||||
@@ -367,6 +408,18 @@ func (c *Cache) PutFailure(ctx context.Context, src imagerequest.Source) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetSourceContent returns a reader for cached source content by its key,
|
||||
// and the content's size in bytes.
|
||||
func (c *Cache) GetSourceContent(
|
||||
key imagerequest.SourceKey,
|
||||
) (io.ReadCloser, int64, error) {
|
||||
if c.disabled {
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
return c.srcContent.LoadWithSize(key)
|
||||
}
|
||||
|
||||
// CleanExpired removes the negative cache rows past their expiry, one
|
||||
// batch per statement, until none is left. Once ctx ends it starts no
|
||||
// further batch and returns ctx's error. The eviction goroutine calls it
|
||||
@@ -390,39 +443,236 @@ func (c *Cache) CleanExpired(ctx context.Context) error {
|
||||
}
|
||||
}
|
||||
|
||||
// IncrementStats counts a cache hit or miss. It only adds to the cache's
|
||||
// Prometheus counters in memory (see counters.go), so it never waits for the
|
||||
// database or fails.
|
||||
func (c *Cache) IncrementStats(hit bool) {
|
||||
// Stats returns cache statistics.
|
||||
func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
|
||||
var stats CacheStats
|
||||
|
||||
err := c.readHitAndMissCounts(ctx, &stats)
|
||||
if err != nil && !errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, fmt.Errorf("failed to get cache stats: %w", err)
|
||||
}
|
||||
|
||||
// Count and size the cached source images and processed variants from
|
||||
// their tables. A disabled cache holds none, whatever rows an earlier
|
||||
// run left.
|
||||
if !c.disabled {
|
||||
err = c.db.QueryRowContext(ctx, `
|
||||
SELECT (SELECT COUNT(*) FROM source_content)
|
||||
+ (SELECT COUNT(*) FROM variant_content)
|
||||
`).Scan(&stats.TotalItems)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to count cache items for stats", "error", err)
|
||||
}
|
||||
|
||||
stats.TotalSizeBytes, err = c.sumContentSizeBytes(ctx)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to sum cache size for stats", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Compute hit rate as a ratio
|
||||
if stats.HitCount+stats.MissCount > 0 {
|
||||
stats.HitRate = float64(stats.HitCount) / float64(stats.HitCount+stats.MissCount)
|
||||
}
|
||||
|
||||
return &stats, nil
|
||||
}
|
||||
|
||||
// IncrementStats counts a cache hit or miss, and an upstream fetch that read
|
||||
// fetchBytes bytes, as IncrementUpstreamFetch does. Like the other Increment
|
||||
// methods, it writes to the database with ctx's deadline but not its
|
||||
// cancellation, so an ended request is still counted but does not wait for
|
||||
// the database past its deadline; a count not written by then becomes a
|
||||
// pending count (see StartPendingCountWrites).
|
||||
func (c *Cache) IncrementStats(ctx context.Context, hit bool, fetchBytes int64) {
|
||||
countCtx, cancel := withoutCancelKeepingDeadline(ctx)
|
||||
defer cancel()
|
||||
|
||||
var err error
|
||||
|
||||
pendingCount := &c.pendingMisses
|
||||
|
||||
if hit {
|
||||
c.cacheHits.Inc()
|
||||
pendingCount = &c.pendingHits
|
||||
|
||||
_, err = c.db.ExecContext(countCtx, `
|
||||
UPDATE cache_stats
|
||||
SET hit_count = hit_count + 1,
|
||||
last_updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = 1
|
||||
`)
|
||||
} else {
|
||||
c.cacheMisses.Inc()
|
||||
_, err = c.db.ExecContext(countCtx, `
|
||||
UPDATE cache_stats
|
||||
SET miss_count = miss_count + 1,
|
||||
last_updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = 1
|
||||
`)
|
||||
}
|
||||
|
||||
switch {
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
c.addPendingCount(pendingCount, 1)
|
||||
case err != nil:
|
||||
c.log.Warn("failed to count cache hit or miss", "hit", hit, "error", err)
|
||||
}
|
||||
|
||||
c.IncrementUpstreamFetch(ctx, fetchBytes)
|
||||
}
|
||||
|
||||
// IncrementUpstreamFetch counts one upstream fetch that read fetchBytes bytes.
|
||||
// A fetch that read no bytes is not counted.
|
||||
func (c *Cache) IncrementUpstreamFetch(ctx context.Context, fetchBytes int64) {
|
||||
if fetchBytes <= 0 {
|
||||
return
|
||||
}
|
||||
|
||||
countCtx, cancel := withoutCancelKeepingDeadline(ctx)
|
||||
defer cancel()
|
||||
|
||||
_, err := c.db.ExecContext(countCtx, `
|
||||
UPDATE cache_stats
|
||||
SET upstream_fetch_count = upstream_fetch_count + 1,
|
||||
upstream_fetch_bytes = upstream_fetch_bytes + ?,
|
||||
last_updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = 1
|
||||
`, fetchBytes)
|
||||
|
||||
switch {
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
c.addPendingCount(&c.pendingUpstreamFetches, 1)
|
||||
c.addPendingCount(&c.pendingUpstreamFetchBytes, fetchBytes)
|
||||
case err != nil:
|
||||
c.log.Warn("failed to count upstream fetch",
|
||||
"fetch_bytes", fetchBytes, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// touchVariant hands the access time of a variant served from the cache to
|
||||
// the goroutine StartPendingAccessTimeWrites starts, which writes it in its
|
||||
// next batch. It never waits: while maxPendingAccessTimes access times of
|
||||
// variants wait to be written, this one is dropped, and the variant only
|
||||
// looks colder to eviction.
|
||||
func (c *Cache) touchVariant(cacheKey imagerequest.VariantKey) {
|
||||
select {
|
||||
case c.pendingVariantAccessTimes <- cacheKey:
|
||||
default:
|
||||
// IncrementTransformCount counts one image transcoded by the image processor.
|
||||
func (c *Cache) IncrementTransformCount(ctx context.Context) {
|
||||
countCtx, cancel := withoutCancelKeepingDeadline(ctx)
|
||||
defer cancel()
|
||||
|
||||
_, err := c.db.ExecContext(countCtx, `
|
||||
UPDATE cache_stats
|
||||
SET transform_count = transform_count + 1,
|
||||
last_updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = 1
|
||||
`)
|
||||
|
||||
switch {
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
c.addPendingCount(&c.pendingTransforms, 1)
|
||||
case err != nil:
|
||||
c.log.Warn("failed to count transform", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// touchSourceContent hands the access time of a cached source a miss used to
|
||||
// the goroutine StartPendingAccessTimeWrites starts, which writes it in its
|
||||
// next batch. It never waits: while maxPendingAccessTimes access times of
|
||||
// sources wait to be written, this one is dropped, and the source only looks
|
||||
// colder to eviction.
|
||||
func (c *Cache) touchSourceContent(key imagerequest.SourceKey) {
|
||||
select {
|
||||
case c.pendingSourceAccessTimes <- key:
|
||||
default:
|
||||
// readHitAndMissCounts sets the hit and miss counts of stats to those of the
|
||||
// cache_stats row plus the pending hits and misses (see
|
||||
// StartPendingCountWrites). It holds pendingCountsWriteMutex, so that no
|
||||
// write of the pending counts falls between the two reads.
|
||||
func (c *Cache) readHitAndMissCounts(ctx context.Context, stats *CacheStats) error {
|
||||
c.pendingCountsWriteMutex.Lock()
|
||||
defer c.pendingCountsWriteMutex.Unlock()
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT hit_count, miss_count
|
||||
FROM cache_stats WHERE id = 1
|
||||
`).Scan(&stats.HitCount, &stats.MissCount)
|
||||
|
||||
stats.HitCount += c.pendingHits.Load()
|
||||
stats.MissCount += c.pendingMisses.Load()
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// loadVariantWithMeta is GetVariant for a variant metaCache does not
|
||||
// hold: it reads the content type from the variant's .meta file and
|
||||
// keeps it in metaCache, unless a StoreVariant has put one there
|
||||
// meanwhile, as the store's is newer. A read that finds no .meta file,
|
||||
// as one can when the store's .meta write failed or an eviction removed
|
||||
// the file after the read opened the variant, serves
|
||||
// application/octet-stream and keeps nothing, so metaCache only ever
|
||||
// holds a type read from a .meta file or passed to StoreVariant.
|
||||
func (c *Cache) loadVariantWithMeta(
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
reader, size, contentType, err := c.variants.LoadWithMeta(cacheKey)
|
||||
if err != nil {
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
if contentType == "" {
|
||||
return reader, size, fallbackContentType, nil
|
||||
}
|
||||
|
||||
c.metaCache.ContainsOrAdd(cacheKey, contentType)
|
||||
|
||||
return reader, size, contentType, nil
|
||||
}
|
||||
|
||||
// touchVariant updates the LRU timestamp of a variant, best-effort:
|
||||
// a failed touch only makes the entry look colder to eviction.
|
||||
func (c *Cache) touchVariant(ctx context.Context, cacheKey imagerequest.VariantKey) {
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
UPDATE variant_content SET last_accessed_at = CURRENT_TIMESTAMP
|
||||
WHERE cache_key = ?
|
||||
`, string(cacheKey))
|
||||
if err != nil {
|
||||
c.log.Debug("failed to touch variant LRU timestamp",
|
||||
"cache_key", cacheKey, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// touchSourceContent updates the LRU timestamp of a cached source,
|
||||
// best-effort: a failed touch only makes the source look colder.
|
||||
func (c *Cache) touchSourceContent(ctx context.Context, key imagerequest.SourceKey) {
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
UPDATE source_content SET last_accessed_at = CURRENT_TIMESTAMP
|
||||
WHERE source_key = ?
|
||||
`, string(key))
|
||||
if err != nil {
|
||||
c.log.Debug("failed to touch source content LRU timestamp",
|
||||
"source_key", key, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// checkNegativeCache checks if a request is in the negative cache.
|
||||
func (c *Cache) checkNegativeCache(
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (bool, error) {
|
||||
var expiresAt time.Time
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT expires_at FROM negative_cache
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, req.Source.Host, req.Source.Path, req.Source.Query).Scan(&expiresAt)
|
||||
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("failed to check negative cache: %w", err)
|
||||
}
|
||||
|
||||
// Check if expired
|
||||
if time.Now().After(expiresAt) {
|
||||
// Clean up expired entry
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
DELETE FROM negative_cache
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, req.Source.Host, req.Source.Path, req.Source.Query)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to delete expired negative cache entry",
|
||||
"host", req.Source.Host, "path", req.Source.Path, "error", err)
|
||||
}
|
||||
|
||||
return false, nil
|
||||
}
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// cleanExpiredBatch deletes up to cleanExpiredBatchSize negative cache
|
||||
@@ -448,17 +698,3 @@ func (c *Cache) cleanExpiredBatch(ctx context.Context) (int64, error) {
|
||||
|
||||
return deleted, nil
|
||||
}
|
||||
|
||||
// readFormat reads the first bytes of a source or variant file, without
|
||||
// moving its offset, and returns the format they show. Sources and variants
|
||||
// are in the same six formats.
|
||||
func readFormat(content io.ReaderAt) (imagerequest.Format, error) {
|
||||
firstBytes := make([]byte, magic.MinMagicBytes)
|
||||
|
||||
_, err := content.ReadAt(firstBytes, 0)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read the first bytes: %w", err)
|
||||
}
|
||||
|
||||
return magic.DetectFormat(firstBytes)
|
||||
}
|
||||
|
||||
@@ -1,214 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image/color"
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// TestVariant_ContentTypeFromFirstBytes stores an image in each output
|
||||
// format as a variant and checks that Variant serves it whole with the
|
||||
// content type of its format, read from the file.
|
||||
func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
processor := imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
})
|
||||
source := generateTestJPEG(t, 40, 30, color.RGBA{200, 40, 40, 255})
|
||||
|
||||
for _, format := range []imagerequest.Format{
|
||||
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
|
||||
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
|
||||
} {
|
||||
result, err := processor.Process(t.Context(), bytes.NewReader(source),
|
||||
imagerequest.FormatJPEG, &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 20, Height: 15},
|
||||
Format: format,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("making a %s image: %v", format, err)
|
||||
}
|
||||
|
||||
image, err := io.ReadAll(result.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the %s image: %v", format, err)
|
||||
}
|
||||
|
||||
key := imagerequest.VariantKey("aabbccdd10" + string(format))
|
||||
|
||||
err = cache.PutVariant(t.Context(), key, bytes.NewReader(image))
|
||||
if err != nil {
|
||||
t.Fatalf("PutVariant(%s) error = %v", format, err)
|
||||
}
|
||||
|
||||
reader, size, contentType, err := cache.Variant(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Variant(%s) error = %v", format, err)
|
||||
}
|
||||
|
||||
served, err := io.ReadAll(reader)
|
||||
_ = reader.Close()
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("reading the %s variant: %v", format, err)
|
||||
}
|
||||
|
||||
t.Logf("%s: %d bytes served as %s", format, size, contentType)
|
||||
|
||||
if want := format.MIMEType(); contentType != want {
|
||||
t.Errorf("%s variant content type = %q, want %q", format, contentType, want)
|
||||
}
|
||||
|
||||
if size != int64(len(image)) || !bytes.Equal(served, image) {
|
||||
t.Errorf("%s variant served %d bytes, size %d, want the %d bytes stored",
|
||||
format, len(served), size, len(image))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestVariant_RefusesFileOfNoOutputFormat checks that Variant refuses
|
||||
// a variant file whose first bytes are not those of an output format, with
|
||||
// an error other than ErrNotFound, and that a variant with no file, or any
|
||||
// variant of a disabled cache, is ErrNotFound.
|
||||
func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
jpegImage := generateTestJPEG(t, 10, 10, color.RGBA{0, 0, 255, 255})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
content []byte
|
||||
wantErr error
|
||||
}{
|
||||
{"text", []byte("this is text, not an image"), magic.ErrUnknownFormat},
|
||||
{"SVG", []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
|
||||
magic.ErrUnknownFormat},
|
||||
{"JPEG cut to 10 bytes", jpegImage[:10], io.EOF},
|
||||
{"empty", nil, io.EOF},
|
||||
}
|
||||
|
||||
for i, tt := range tests {
|
||||
key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd20%02d", i))
|
||||
|
||||
err := cache.variants.Write(string(key), tt.content)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: storing the variant file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
reader, _, contentType, err := cache.Variant(key)
|
||||
t.Logf("%s: Variant() error = %v", tt.name, err)
|
||||
|
||||
if reader != nil || contentType != "" || !errors.Is(err, tt.wantErr) {
|
||||
t.Errorf("%s: Variant() = reader %v, content type %q, error %v; "+
|
||||
"want no reader and %v", tt.name, reader, contentType, err, tt.wantErr)
|
||||
}
|
||||
}
|
||||
|
||||
reader, _, _, err := cache.Variant(testVariantKeyFour)
|
||||
if reader != nil || !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Variant() of a variant with no file = reader %v, error %v; "+
|
||||
"want no reader and %v", reader, err, ErrNotFound)
|
||||
}
|
||||
|
||||
disabled, _ := newEvictionTestCache(t, 0)
|
||||
|
||||
reader, _, _, err = disabled.Variant(testVariantKeyOne)
|
||||
if reader != nil || !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Variant() on a disabled cache = reader %v, error %v; "+
|
||||
"want no reader and %v", reader, err, ErrNotFound)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTouchVariantAndTouchSourceContent_DropAccessTimesPastTheLimit hands one
|
||||
// more access time of a variant, and one more of a source, than may wait to
|
||||
// be written. touchVariant and touchSourceContent must return each time
|
||||
// without waiting, and the last of each must be dropped.
|
||||
func TestTouchVariantAndTouchSourceContent_DropAccessTimesPastTheLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
done := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
defer close(done)
|
||||
|
||||
for range maxPendingAccessTimes + 1 {
|
||||
cache.touchVariant(testVariantKeyOne)
|
||||
cache.touchSourceContent("aabbccdd3001")
|
||||
}
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(pendingAccessTimesWait):
|
||||
t.Fatal("touchVariant or touchSourceContent waited once the access " +
|
||||
"times reached the limit")
|
||||
}
|
||||
|
||||
if waiting := len(cache.pendingVariantAccessTimes); waiting != maxPendingAccessTimes {
|
||||
t.Errorf("%d access times of variants wait to be written, want %d",
|
||||
waiting, maxPendingAccessTimes)
|
||||
}
|
||||
|
||||
if waiting := len(cache.pendingSourceAccessTimes); waiting != maxPendingAccessTimes {
|
||||
t.Errorf("%d access times of sources wait to be written, want %d",
|
||||
waiting, maxPendingAccessTimes)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopPendingAccessTimeWrites_WritesPendingAccessTimes hands the access
|
||||
// time of one of two variants, three times, then stops the goroutine that
|
||||
// writes them, which never started. The stop must write that access time and
|
||||
// leave the other variant's.
|
||||
func TestStopPendingAccessTimeWrites_WritesPendingAccessTimes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
anHourAgo := time.Now().Add(-time.Hour)
|
||||
|
||||
for _, key := range []imagerequest.VariantKey{testVariantKeyOne, testVariantKeyTwo} {
|
||||
storeEvictionTestVariant(t, cache, key, []byte("variant bytes"))
|
||||
setVariantLastAccessed(t, cache, key, anHourAgo)
|
||||
}
|
||||
|
||||
for range 3 {
|
||||
cache.touchVariant(testVariantKeyOne)
|
||||
}
|
||||
|
||||
err := cache.StopPendingAccessTimeWrites(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("StopPendingAccessTimeWrites() error = %v", err)
|
||||
}
|
||||
|
||||
for key, wantTouched := range map[imagerequest.VariantKey]int{
|
||||
testVariantKeyOne: 1, testVariantKeyTwo: 0,
|
||||
} {
|
||||
touched := countRows(t, cache, `
|
||||
SELECT COUNT(*) FROM variant_content
|
||||
WHERE cache_key = ? AND last_accessed_at > ?
|
||||
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat))
|
||||
if touched != wantTouched {
|
||||
t.Errorf("variant %s: %d rows with a new access time, want %d",
|
||||
key, touched, wantTouched)
|
||||
}
|
||||
}
|
||||
|
||||
if waiting := len(cache.pendingVariantAccessTimes); waiting != 0 {
|
||||
t.Errorf("%d access times still wait to be written, want 0", waiting)
|
||||
}
|
||||
}
|
||||
@@ -1,14 +1,16 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func setupTestDB(t *testing.T) *sql.DB {
|
||||
@@ -23,17 +25,54 @@ func setupTestDB(t *testing.T) *sql.DB {
|
||||
schema := `
|
||||
CREATE TABLE source_content (
|
||||
source_key TEXT PRIMARY KEY,
|
||||
source_host TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_query TEXT NOT NULL DEFAULT '',
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE TABLE output_content (
|
||||
content_hash TEXT PRIMARY KEY,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE TABLE request_cache (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
cache_key TEXT NOT NULL UNIQUE,
|
||||
source_metadata_id INTEGER NOT NULL,
|
||||
output_hash TEXT NOT NULL,
|
||||
width INTEGER NOT NULL,
|
||||
height INTEGER NOT NULL,
|
||||
format TEXT NOT NULL,
|
||||
quality INTEGER NOT NULL DEFAULT 85,
|
||||
fit_mode TEXT NOT NULL DEFAULT 'cover',
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
access_count INTEGER NOT NULL DEFAULT 1
|
||||
);
|
||||
CREATE TABLE negative_cache (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_host TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_query TEXT NOT NULL DEFAULT '',
|
||||
status_code INTEGER NOT NULL,
|
||||
error_message TEXT,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at DATETIME NOT NULL,
|
||||
UNIQUE(source_host, source_path, source_query)
|
||||
);
|
||||
CREATE TABLE cache_stats (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
hit_count INTEGER NOT NULL DEFAULT 0,
|
||||
miss_count INTEGER NOT NULL DEFAULT 0,
|
||||
upstream_fetch_count INTEGER NOT NULL DEFAULT 0,
|
||||
upstream_fetch_bytes INTEGER NOT NULL DEFAULT 0,
|
||||
transform_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
INSERT INTO cache_stats (id) VALUES (1);
|
||||
`
|
||||
|
||||
_, err = db.ExecContext(t.Context(), schema)
|
||||
@@ -44,25 +83,407 @@ func setupTestDB(t *testing.T) *sql.DB {
|
||||
return db
|
||||
}
|
||||
|
||||
func setupTestCache(t *testing.T) *Cache {
|
||||
t.Helper()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
db := setupTestDB(t)
|
||||
|
||||
cache, err := NewCache(db, CacheConfig{
|
||||
StateDir: tmpDir,
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
|
||||
return cache
|
||||
}
|
||||
|
||||
func TestCache_LookupMiss(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
result, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() error = %v", err)
|
||||
}
|
||||
|
||||
if result.Hit {
|
||||
t.Error("Lookup() hit = true, want false for cache miss")
|
||||
}
|
||||
|
||||
if result.CacheStatus != CacheMiss {
|
||||
t.Errorf("Lookup() status = %v, want %v", result.CacheStatus, CacheMiss)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_StoreAndLookup(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store source content
|
||||
sourceContent := []byte("fake jpeg data")
|
||||
fetchResult := &httpfetcher.FetchResult{
|
||||
ContentType: testContentTypeJPEG,
|
||||
Headers: map[string][]string{testHeaderContentType: {testContentTypeJPEG}},
|
||||
}
|
||||
|
||||
sourceKey, err := cache.StoreSource(
|
||||
ctx, req, bytes.NewReader(sourceContent), fetchResult)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreSource() error = %v", err)
|
||||
}
|
||||
|
||||
if sourceKey == "" {
|
||||
t.Error("StoreSource() returned empty key")
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("fake webp data")
|
||||
|
||||
err = cache.StoreVariant(
|
||||
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
|
||||
if err != nil {
|
||||
t.Fatalf("StoreVariant() error = %v", err)
|
||||
}
|
||||
|
||||
// Now lookup should hit
|
||||
result, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() error = %v", err)
|
||||
}
|
||||
|
||||
if !result.Hit {
|
||||
t.Error("Lookup() hit = false, want true after store")
|
||||
}
|
||||
|
||||
if result.CacheStatus != CacheHit {
|
||||
t.Errorf("Lookup() status = %v, want %v", result.CacheStatus, CacheHit)
|
||||
}
|
||||
|
||||
if result.CacheKey == "" {
|
||||
t.Error("Lookup() returned empty cache key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_NegativeCache(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/notfound.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
},
|
||||
}
|
||||
|
||||
// Store negative cache entry
|
||||
err := cache.StoreNegative(ctx, req, 404, "not found")
|
||||
if err != nil {
|
||||
t.Fatalf("StoreNegative() error = %v", err)
|
||||
}
|
||||
|
||||
// Lookup should return miss (negative cache is checked at service level)
|
||||
result, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() error = %v", err)
|
||||
}
|
||||
|
||||
if result.Hit {
|
||||
t.Error("Lookup() hit = true, want false for negative cache entry")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_NegativeCacheExpiry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
db := setupTestDB(t)
|
||||
|
||||
// Very short negative TTL for testing
|
||||
cache, err := NewCache(db, CacheConfig{
|
||||
StateDir: tmpDir,
|
||||
NegativeTTL: 1 * time.Millisecond,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/expired.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
},
|
||||
}
|
||||
|
||||
// Store negative cache entry
|
||||
err = cache.StoreNegative(ctx, req, 404, "not found")
|
||||
if err != nil {
|
||||
t.Fatalf("StoreNegative() error = %v", err)
|
||||
}
|
||||
|
||||
// Wait for expiry
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
|
||||
// Lookup should return miss after expiry
|
||||
result, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() error = %v, want nil after expiry", err)
|
||||
}
|
||||
|
||||
if result.Hit {
|
||||
t.Error("Lookup() hit = true, want false after negative cache expiry")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_VariantLookup(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/variant.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("output data")
|
||||
|
||||
err := cache.StoreVariant(
|
||||
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
|
||||
if err != nil {
|
||||
t.Fatalf("StoreVariant() error = %v", err)
|
||||
}
|
||||
|
||||
// First lookup
|
||||
result1, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() first error = %v", err)
|
||||
}
|
||||
|
||||
if !result1.Hit {
|
||||
t.Error("Lookup() first hit = false")
|
||||
}
|
||||
|
||||
// Second lookup should also hit (from disk)
|
||||
result2, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() second error = %v", err)
|
||||
}
|
||||
|
||||
if !result2.Hit {
|
||||
t.Error("Lookup() second hit = false")
|
||||
}
|
||||
|
||||
if result1.CacheKey != result2.CacheKey {
|
||||
t.Error("Lookup() returned different cache keys")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_GetVariant_ReturnsContentType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/variantct.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("output webp data")
|
||||
|
||||
err := cache.StoreVariant(
|
||||
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
|
||||
if err != nil {
|
||||
t.Fatalf("StoreVariant() error = %v", err)
|
||||
}
|
||||
|
||||
// Lookup
|
||||
result, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() error = %v", err)
|
||||
}
|
||||
|
||||
if !result.Hit {
|
||||
t.Fatal("Lookup() hit = false, want true")
|
||||
}
|
||||
|
||||
// GetVariant should return content type
|
||||
reader, size, contentType, err := cache.GetVariant(result.CacheKey)
|
||||
if err != nil {
|
||||
t.Fatalf("GetVariant() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = reader.Close() }()
|
||||
|
||||
if contentType != "image/webp" {
|
||||
t.Errorf("GetVariant() ContentType = %q, want %q", contentType, "image/webp")
|
||||
}
|
||||
|
||||
if size != int64(len(outputContent)) {
|
||||
t.Errorf("GetVariant() size = %d, want %d", size, len(outputContent))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_GetVariant(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/output.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("the actual output content")
|
||||
|
||||
err := cache.StoreVariant(
|
||||
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
|
||||
if err != nil {
|
||||
t.Fatalf("StoreVariant() error = %v", err)
|
||||
}
|
||||
|
||||
// Lookup to get cache key
|
||||
result, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup() error = %v", err)
|
||||
}
|
||||
|
||||
// Get variant content
|
||||
reader, _, _, err := cache.GetVariant(result.CacheKey)
|
||||
if err != nil {
|
||||
t.Fatalf("GetVariant() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = reader.Close() }()
|
||||
|
||||
buf := make([]byte, 100)
|
||||
n, _ := reader.Read(buf)
|
||||
|
||||
if !bytes.Equal(buf[:n], outputContent) {
|
||||
t.Errorf("GetVariant() content = %q, want %q", buf[:n], outputContent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_Stats(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
// Increment some stats
|
||||
cache.IncrementStats(ctx, true, 0)
|
||||
cache.IncrementStats(ctx, true, 0)
|
||||
cache.IncrementStats(ctx, false, 1024)
|
||||
|
||||
stats, err := cache.Stats(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Stats() error = %v", err)
|
||||
}
|
||||
|
||||
if stats.HitCount != 2 {
|
||||
t.Errorf("Stats() HitCount = %d, want 2", stats.HitCount)
|
||||
}
|
||||
|
||||
if stats.MissCount != 1 {
|
||||
t.Errorf("Stats() MissCount = %d, want 1", stats.MissCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_CleanExpired(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
db := setupTestDB(t)
|
||||
|
||||
cache, _ := NewCache(db, Config{
|
||||
cache, _ := NewCache(db, CacheConfig{
|
||||
StateDir: tmpDir,
|
||||
NegativeTTL: 1 * time.Millisecond,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
})
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Insert expired negative cache entry directly
|
||||
_, err := db.ExecContext(ctx, `
|
||||
INSERT INTO negative_cache
|
||||
(source_host, source_path, source_query, expires_at)
|
||||
VALUES ('example.com', '/old.jpg', '', datetime('now', '-1 hour'))
|
||||
(source_host, source_path, source_query, status_code, expires_at)
|
||||
VALUES ('example.com', '/old.jpg', '', 404, datetime('now', '-1 hour'))
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert test data: %v", err)
|
||||
@@ -103,10 +524,9 @@ func TestCache_StorageDirectoriesCreated(t *testing.T) {
|
||||
tmpDir := t.TempDir()
|
||||
db := setupTestDB(t)
|
||||
|
||||
_, err := NewCache(db, Config{
|
||||
_, err := NewCache(db, CacheConfig{
|
||||
StateDir: tmpDir,
|
||||
MaxBytes: 1 << 30,
|
||||
}, prometheus.NewRegistry())
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"path/filepath"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// DefaultCacheMaxBytesFloor is the minimum computed default for the
|
||||
// cache_max_bytes setting: 500 MiB. The floor applies only to the
|
||||
// computed default (when the key is omitted from the configuration),
|
||||
// never to explicitly configured values.
|
||||
const DefaultCacheMaxBytesFloor int64 = 524288000
|
||||
|
||||
// freeSpaceFractionNumerator and freeSpaceFractionDenominator express
|
||||
// the 75% share used for the computed default limit as integer
|
||||
// arithmetic (dividing before multiplying avoids overflow).
|
||||
const (
|
||||
freeSpaceFractionNumerator uint64 = 3
|
||||
freeSpaceFractionDenominator uint64 = 4
|
||||
)
|
||||
|
||||
var errNegativeBlockSize = errors.New("statfs reported negative block size")
|
||||
|
||||
// FreeSpaceProbeFunc reports the number of free bytes available on the
|
||||
// filesystem containing path. It is a function type so tests can
|
||||
// inject a fake probe instead of depending on the host disk.
|
||||
type FreeSpaceProbeFunc func(path string) (uint64, error)
|
||||
|
||||
// defaultFreeSpaceProbe reports free filesystem bytes via statfs on
|
||||
// the given path, as available to unprivileged processes.
|
||||
func defaultFreeSpaceProbe(path string) (uint64, error) {
|
||||
var stat syscall.Statfs_t
|
||||
|
||||
err := syscall.Statfs(path, &stat)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
if stat.Bsize < 0 {
|
||||
return 0, fmt.Errorf("%w %d for %q", errNegativeBlockSize, stat.Bsize, path)
|
||||
}
|
||||
|
||||
blockSize := uint64(stat.Bsize)
|
||||
|
||||
return stat.Bavail * blockSize, nil
|
||||
}
|
||||
|
||||
// computeDefaultMaxBytes returns the default cache size limit: 75% of
|
||||
// the sum of the free bytes probe reports for <state_dir>/cache/ and
|
||||
// the bytes the cache already holds, with a floor of
|
||||
// DefaultCacheMaxBytesFloor. Counting what the cache holds keeps the
|
||||
// limit from shrinking as the cache fills.
|
||||
func (c *Cache) computeDefaultMaxBytes(
|
||||
ctx context.Context, probe FreeSpaceProbeFunc,
|
||||
) (int64, error) {
|
||||
cacheDir := filepath.Join(c.config.StateDir, "cache")
|
||||
|
||||
freeBytes, err := probe(cacheDir)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf(
|
||||
"default cache_max_bytes: cannot determine free space for %q: %w",
|
||||
cacheDir, err)
|
||||
}
|
||||
|
||||
usedBytes, err := c.UsageBytes(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
// Both terms are at most math.MaxInt64, so the sum cannot overflow.
|
||||
//nolint:gosec // G115: UsageBytes returns the total cache usage, never negative
|
||||
spaceBytes := min(freeBytes, math.MaxInt64) + uint64(usedBytes)
|
||||
|
||||
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
||||
computed = min(computed, math.MaxInt64)
|
||||
|
||||
limit := int64(computed)
|
||||
limit = max(limit, DefaultCacheMaxBytesFloor)
|
||||
|
||||
return limit, nil
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Static errors returned by the stub free-space probes below.
|
||||
var (
|
||||
errTestStatfsFailed = errors.New("statfs failed")
|
||||
errTestProbeNotExpected = errors.New("probe must not be called")
|
||||
)
|
||||
|
||||
// TestComputeDefaultMaxBytesCountsWhatTheCacheHolds verifies that the
|
||||
// default limit is 75% of the free space plus what the cache already
|
||||
// holds, so a cache filled to its limit keeps that limit across a
|
||||
// restart instead of shrinking to 75% of the space left free.
|
||||
func TestComputeDefaultMaxBytesCountsWhatTheCacheHolds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
// Empty cache, 4 GiB free -> 3 GiB default.
|
||||
got, err := cache.computeDefaultMaxBytes(t.Context(),
|
||||
func(string) (uint64, error) { return 4294967296, nil })
|
||||
if err != nil {
|
||||
t.Fatalf("computeDefaultMaxBytes returned error: %v", err)
|
||||
}
|
||||
|
||||
t.Logf("default for an empty cache with 4 GiB free: %d", got)
|
||||
|
||||
if got != 3221225472 {
|
||||
t.Errorf("default for an empty cache = %d, want 3221225472 (75%% of 4 GiB)",
|
||||
got)
|
||||
}
|
||||
|
||||
// The cache now holds those 3 GiB, which leaves 1 GiB free.
|
||||
_, err = cache.db.ExecContext(t.Context(),
|
||||
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
||||
VALUES (?, ?, ?)`,
|
||||
string(testVariantKeyOne), 3221225472, testContentTypeWebP,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert variant accounting row: %v", err)
|
||||
}
|
||||
|
||||
got, err = cache.computeDefaultMaxBytes(t.Context(),
|
||||
func(string) (uint64, error) { return 1073741824, nil })
|
||||
if err != nil {
|
||||
t.Fatalf("computeDefaultMaxBytes returned error: %v", err)
|
||||
}
|
||||
|
||||
t.Logf("default for a cache holding 3 GiB with 1 GiB free: %d", got)
|
||||
|
||||
if got != 3221225472 {
|
||||
t.Errorf("default for a cache holding 3 GiB with 1 GiB free = %d, "+
|
||||
"want 3221225472 (75%% of 1 GiB + 3 GiB)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputeDefaultMaxBytesAppliesFloor verifies that when 75% of the
|
||||
// free space plus what the cache holds is below 500 MiB, the default
|
||||
// is floored at DefaultCacheMaxBytesFloor.
|
||||
func TestComputeDefaultMaxBytesAppliesFloor(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
freeBytes uint64
|
||||
}{
|
||||
{name: "100 MiB free", freeBytes: 104857600},
|
||||
{name: "zero free", freeBytes: 0},
|
||||
{name: "just below floor threshold", freeBytes: 699050665},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
got, err := cache.computeDefaultMaxBytes(t.Context(),
|
||||
func(string) (uint64, error) { return tc.freeBytes, nil })
|
||||
if err != nil {
|
||||
t.Fatalf("computeDefaultMaxBytes returned error: %v", err)
|
||||
}
|
||||
|
||||
if got != DefaultCacheMaxBytesFloor {
|
||||
t.Errorf("computeDefaultMaxBytes = %d, want floor %d",
|
||||
got, DefaultCacheMaxBytesFloor)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputeDefaultMaxBytesPropagatesProbeError verifies that a
|
||||
// failing free-space probe produces an error naming cache_max_bytes,
|
||||
// instead of a silently wrong default.
|
||||
func TestComputeDefaultMaxBytesPropagatesProbeError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
_, err := cache.computeDefaultMaxBytes(t.Context(),
|
||||
func(string) (uint64, error) { return 0, errTestStatfsFailed })
|
||||
if err == nil {
|
||||
t.Fatal("probe failure must produce an error, got nil")
|
||||
}
|
||||
|
||||
t.Logf("got expected error: %v", err)
|
||||
|
||||
if !strings.Contains(err.Error(), "cache_max_bytes") {
|
||||
t.Errorf("error %q does not name cache_max_bytes", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewCacheComputesDefaultMaxBytesWhenAsked verifies that with
|
||||
// UseDefaultMaxBytes set, the cache's limit becomes the computed
|
||||
// default, and that the probe is pointed at <state_dir>/cache/, which
|
||||
// must be created first so statfs measures the right filesystem.
|
||||
func TestNewCacheComputesDefaultMaxBytesWhenAsked(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
stateDir := t.TempDir()
|
||||
wantCacheDir := filepath.Join(stateDir, "cache")
|
||||
|
||||
var probedPath string
|
||||
|
||||
// 4 GiB free -> 3 GiB default.
|
||||
probe := func(path string) (uint64, error) {
|
||||
probedPath = path
|
||||
|
||||
info, err := os.Stat(path)
|
||||
if err != nil || !info.IsDir() {
|
||||
t.Errorf("cache directory %q was not created before probing: info=%v err=%v",
|
||||
path, info, err)
|
||||
}
|
||||
|
||||
return 4294967296, nil
|
||||
}
|
||||
|
||||
cache, err := newCache(evictionTestDB(t), CacheConfig{
|
||||
StateDir: stateDir,
|
||||
UseDefaultMaxBytes: true,
|
||||
}, probe)
|
||||
if err != nil {
|
||||
t.Fatalf("newCache returned error: %v", err)
|
||||
}
|
||||
|
||||
if cache.config.MaxBytes != 3221225472 {
|
||||
t.Errorf("MaxBytes = %d, want computed default 3221225472",
|
||||
cache.config.MaxBytes)
|
||||
}
|
||||
|
||||
if probedPath != wantCacheDir {
|
||||
t.Errorf("free space probed at %q, want cache directory %q",
|
||||
probedPath, wantCacheDir)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewCacheKeepsExplicitMaxBytes verifies that without
|
||||
// UseDefaultMaxBytes the cache keeps MaxBytes exactly as given and
|
||||
// never consults the free-space probe.
|
||||
func TestNewCacheKeepsExplicitMaxBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
probe := func(string) (uint64, error) {
|
||||
t.Error("free-space probe must not be consulted for explicit values")
|
||||
|
||||
return 0, errTestProbeNotExpected
|
||||
}
|
||||
|
||||
cache, err := newCache(evictionTestDB(t), CacheConfig{
|
||||
StateDir: t.TempDir(),
|
||||
MaxBytes: 1024,
|
||||
}, probe)
|
||||
if err != nil {
|
||||
t.Fatalf("newCache returned error: %v", err)
|
||||
}
|
||||
|
||||
if cache.config.MaxBytes != 1024 {
|
||||
t.Errorf("MaxBytes = %d, want explicit 1024 (no floor, no recompute)",
|
||||
cache.config.MaxBytes)
|
||||
}
|
||||
}
|
||||
+45
-45
@@ -1,4 +1,4 @@
|
||||
package imageproxy
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -36,7 +36,7 @@ type heldFetcher struct {
|
||||
}
|
||||
|
||||
func (f *heldFetcher) Fetch(
|
||||
ctx context.Context, source imagerequest.Source,
|
||||
ctx context.Context, url string,
|
||||
) (*httpfetcher.FetchResult, error) {
|
||||
f.fetches.Add(1)
|
||||
|
||||
@@ -48,7 +48,7 @@ func (f *heldFetcher) Fetch(
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
|
||||
return f.upstream.Fetch(ctx, source)
|
||||
return f.upstream.Fetch(ctx, url)
|
||||
}
|
||||
|
||||
// releaseFetches lets every held fetch, and every later one, go on.
|
||||
@@ -56,15 +56,15 @@ func (f *heldFetcher) releaseFetches() {
|
||||
f.releaseOnce.Do(func() { close(f.release) })
|
||||
}
|
||||
|
||||
// setupHeldFetchProxy returns a test proxy whose fetches go through a
|
||||
// setupHeldFetchService returns a test service whose fetches go through a
|
||||
// heldFetcher. Its database is limited to one connection: each connection to
|
||||
// an in-memory SQLite database opens a new, empty one, so requests running at
|
||||
// once must share the connection that holds the schema.
|
||||
func setupHeldFetchProxy(t *testing.T) (*testProxy, *TestFixtures, *heldFetcher) {
|
||||
func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher) {
|
||||
t.Helper()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
svc.db.SetMaxOpenConns(1)
|
||||
svc, fixtures := SetupTestService(t)
|
||||
svc.cache.db.SetMaxOpenConns(1)
|
||||
|
||||
fetcher := &heldFetcher{
|
||||
upstream: svc.fetcher,
|
||||
@@ -79,8 +79,8 @@ func setupHeldFetchProxy(t *testing.T) (*testProxy, *TestFixtures, *heldFetcher)
|
||||
}
|
||||
|
||||
// photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the
|
||||
// given quality and fit mode. Each call returns a new request, which a test
|
||||
// may change.
|
||||
// given quality and fit mode. Each call returns a new request, as Get writes
|
||||
// to the request it is given.
|
||||
func photoVariant(
|
||||
fixtures *TestFixtures, quality int, fit imagerequest.Fit,
|
||||
) *imagerequest.Request {
|
||||
@@ -107,7 +107,7 @@ type getResult struct {
|
||||
// startGet calls Get in a goroutine of its own and delivers what it returned
|
||||
// on the channel.
|
||||
func startGet(
|
||||
ctx context.Context, svc *testProxy, req *imagerequest.Request,
|
||||
ctx context.Context, svc *Service, req *imagerequest.Request,
|
||||
) <-chan getResult {
|
||||
results := make(chan getResult, 1)
|
||||
|
||||
@@ -139,14 +139,14 @@ func jpegSize(data []byte) (int, int) {
|
||||
return config.Width, config.Height
|
||||
}
|
||||
|
||||
// TestProxy_Get_ConcurrentMissesShareOneFetch starts several requests for
|
||||
// TestService_Get_ConcurrentMissesShareOneFetch starts several requests for
|
||||
// one uncached variant while the first one's fetch is held. Between them they
|
||||
// must fetch the source once and transcode it once, every one must be answered
|
||||
// with the same 50x25 JPEG, and each must count one miss.
|
||||
func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
|
||||
func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
const requests = 8
|
||||
|
||||
@@ -183,7 +183,7 @@ func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
|
||||
t.Errorf("%d requests made %d upstream fetches, want 1", requests, fetches)
|
||||
}
|
||||
|
||||
// NewTestFS builds the same files the test proxy's fetcher serves.
|
||||
// NewTestFS builds the same files the test service's fetcher serves.
|
||||
testFS, _ := NewTestFS(t)
|
||||
|
||||
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
|
||||
@@ -191,20 +191,20 @@ func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := counterValues{0, requests, 1, int64(len(photo)), 1}
|
||||
want := cacheStatsCounters{0, requests, 1, int64(len(photo)), 1}
|
||||
|
||||
if got := readCounters(t, svc); got != want {
|
||||
if got := readCacheStatsCounters(t, svc.cache); got != want {
|
||||
t.Errorf("counters = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_ConcurrentVariantsStayApart requests three variants of the
|
||||
// TestService_Get_ConcurrentVariantsStayApart requests three variants of the
|
||||
// test photo at once that differ only in quality or fit. Each must be made by
|
||||
// a fetch and a transcode of its own, and each answer must be its own variant.
|
||||
func TestProxy_Get_ConcurrentVariantsStayApart(t *testing.T) {
|
||||
func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
lowQuality := startGet(t.Context(), svc,
|
||||
@@ -253,14 +253,14 @@ func TestProxy_Get_ConcurrentVariantsStayApart(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_WaiterStopsWhenItsContextEnds has a second request for a
|
||||
// TestService_Get_WaiterStopsWhenItsContextEnds has a second request for a
|
||||
// variant join the first one's held fetch, then ends the second request's
|
||||
// context. The second request must return at once with the context's error,
|
||||
// while the fetch is still held, and the first must still be answered.
|
||||
func TestProxy_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
|
||||
func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
@@ -298,15 +298,15 @@ func TestProxy_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_FirstRequestLeavingKeepsTheWork ends the context of the
|
||||
// TestService_Get_FirstRequestLeavingKeepsTheWork ends the context of the
|
||||
// request whose fetch is held, after a second request has joined it. The fetch
|
||||
// and transcode must go on and answer the second request. The first request
|
||||
// waits for its own work, as every request did before misses were shared, and
|
||||
// is answered too.
|
||||
func TestProxy_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
|
||||
func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
firstCtx, cancelFirst := context.WithCancel(t.Context())
|
||||
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
@@ -337,11 +337,11 @@ func TestProxy_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_ConcurrentMissesShareAFailure has several requests for an
|
||||
// TestService_Get_ConcurrentMissesShareAFailure has several requests for an
|
||||
// image that cannot be served arrive while its fetch is held: the one fetch
|
||||
// answers all of them with its error. The request after them is answered from
|
||||
// the negative cache, where the failure is kept.
|
||||
func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
|
||||
func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
@@ -362,7 +362,7 @@ func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
request := func() *imagerequest.Request {
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
@@ -398,12 +398,12 @@ func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_EndedRequestFetchesNothing checks that a request whose
|
||||
// TestService_Get_EndedRequestFetchesNothing checks that a request whose
|
||||
// context has already ended when it misses the cache starts no fetch.
|
||||
func TestProxy_Get_EndedRequestFetchesNothing(t *testing.T) {
|
||||
func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
cancel()
|
||||
@@ -418,14 +418,14 @@ func TestProxy_Get_EndedRequestFetchesNothing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_ReturnsByItsDeadline gives a request a deadline and holds
|
||||
// TestService_Get_ReturnsByItsDeadline gives a request a deadline and holds
|
||||
// its fetch until the fetch's context ends, as the fetcher does while it waits
|
||||
// for a free connection to the host. The request must return by its deadline
|
||||
// with the deadline's error.
|
||||
func TestProxy_Get_ReturnsByItsDeadline(t *testing.T) {
|
||||
func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, _ := setupHeldFetchProxy(t)
|
||||
svc, fixtures, _ := setupHeldFetchService(t)
|
||||
|
||||
const timeout = 200 * time.Millisecond
|
||||
|
||||
@@ -444,18 +444,18 @@ func TestProxy_Get_ReturnsByItsDeadline(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_WaiterGetsProcessingTimedOut has a request with no deadline
|
||||
// TestService_Get_WaiterGetsProcessingTimedOut has a request with no deadline
|
||||
// join the held fetch of a first request with a short one. With the only
|
||||
// processing slot held, the first request's transcode waits for it until the
|
||||
// first request's deadline ends, and stops then. The waiting request, whose
|
||||
// own context has not ended, must get ErrProcessingTimedOut, which the image
|
||||
// routes answer 503 as when pixa is busy.
|
||||
func TestProxy_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
|
||||
func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchProxy(t)
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
svc.processor = imageprocessor.New(
|
||||
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
|
||||
imageprocessor.Params{MaxConcurrentProcessing: 1},
|
||||
)
|
||||
|
||||
holdProcessingSlot(t, svc.processor)
|
||||
@@ -489,18 +489,18 @@ func TestProxy_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
|
||||
type panickingFetcher struct{}
|
||||
|
||||
func (panickingFetcher) Fetch(
|
||||
context.Context, imagerequest.Source,
|
||||
context.Context, string,
|
||||
) (*httpfetcher.FetchResult, error) {
|
||||
panic("upstream fetcher panicked")
|
||||
}
|
||||
|
||||
// TestProxy_Get_PanicBecomesAnError checks that a panic while a variant is
|
||||
// TestService_Get_PanicBecomesAnError checks that a panic while a variant is
|
||||
// being made reaches its request as an error naming the panic, instead of
|
||||
// being raised again.
|
||||
func TestProxy_Get_PanicBecomesAnError(t *testing.T) {
|
||||
func TestService_Get_PanicBecomesAnError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
svc, fixtures := SetupTestService(t)
|
||||
svc.fetcher = panickingFetcher{}
|
||||
|
||||
var err error
|
||||
@@ -520,13 +520,13 @@ func TestProxy_Get_PanicBecomesAnError(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_PanicIsReportedToSentry checks that a panic while a variant
|
||||
// TestService_Get_PanicIsReportedToSentry checks that a panic while a variant
|
||||
// is being made is reported through the Sentry hub on the request's context,
|
||||
// where the Sentry middleware puts one when sentry_dsn is set.
|
||||
func TestProxy_Get_PanicIsReportedToSentry(t *testing.T) {
|
||||
func TestService_Get_PanicIsReportedToSentry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
svc, fixtures := SetupTestService(t)
|
||||
svc.fetcher = panickingFetcher{}
|
||||
|
||||
transport := &sentry.MockTransport{}
|
||||
@@ -1,31 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
)
|
||||
|
||||
// registerCounters makes the cache's counters and registers them on
|
||||
// metricsRegistry. The counters are kept in memory, so adding to one waits
|
||||
// for nothing and makes no database call. It fails when metricsRegistry
|
||||
// already holds counters of the same names.
|
||||
func (c *Cache) registerCounters(metricsRegistry prometheus.Registerer) error {
|
||||
c.cacheHits = prometheus.NewCounter(prometheus.CounterOpts{
|
||||
Name: "pixa_cache_hits_total",
|
||||
Help: "Image requests served from the cache.",
|
||||
})
|
||||
c.cacheMisses = prometheus.NewCounter(prometheus.CounterOpts{
|
||||
Name: "pixa_cache_misses_total",
|
||||
Help: "Image requests not served from the cache.",
|
||||
})
|
||||
|
||||
for _, counter := range []prometheus.Counter{c.cacheHits, c.cacheMisses} {
|
||||
err := metricsRegistry.Register(counter)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to register the cache's counters: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -1,64 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||
)
|
||||
|
||||
// TestNewCache_RegistersItsCountersOnItsRegistry checks that NewCache
|
||||
// registers the cache's counters on the registry it is given, under the
|
||||
// names README.md gives them.
|
||||
func TestNewCache_RegistersItsCountersOnItsRegistry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
registry := prometheus.NewRegistry()
|
||||
|
||||
_, err := NewCache(evictionTestDB(t),
|
||||
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
|
||||
if err != nil {
|
||||
t.Fatalf("NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
names := []string{
|
||||
"pixa_cache_hits_total",
|
||||
"pixa_cache_misses_total",
|
||||
}
|
||||
|
||||
count, err := testutil.GatherAndCount(registry, names...)
|
||||
if err != nil {
|
||||
t.Fatalf("testutil.GatherAndCount() error = %v", err)
|
||||
}
|
||||
|
||||
if count != len(names) {
|
||||
t.Errorf("%d of the counters %v are on the registry, want %d",
|
||||
count, names, len(names))
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewCache_FailsWhenItsCountersCannotBeRegistered makes a second cache
|
||||
// on the registry of a first. Its counters have the same names, so the
|
||||
// registry refuses them, and NewCache must return that error.
|
||||
func TestNewCache_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
registry := prometheus.NewRegistry()
|
||||
|
||||
_, err := NewCache(evictionTestDB(t),
|
||||
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
|
||||
if err != nil {
|
||||
t.Fatalf("NewCache() of the first cache error = %v", err)
|
||||
}
|
||||
|
||||
_, err = NewCache(evictionTestDB(t),
|
||||
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
|
||||
t.Logf("NewCache() of the second cache error = %v", err)
|
||||
|
||||
var alreadyRegistered prometheus.AlreadyRegisteredError
|
||||
if !errors.As(err, &alreadyRegistered) {
|
||||
t.Errorf("NewCache() of the second cache error = %v, want %T",
|
||||
err, alreadyRegistered)
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
package imageproxy
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -3,6 +3,7 @@ package imgcache
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
@@ -48,29 +49,30 @@ const staleTempFileAge = time.Hour
|
||||
const sqliteTimestampLayout = "2006-01-02 15:04:05"
|
||||
|
||||
// tempFilePrefix is the prefix os.CreateTemp uses for in-flight cache
|
||||
// writes (the ".tmp-*" pattern in writeFileSynced).
|
||||
// writes (".tmp-*" patterns in the storage layer).
|
||||
const tempFilePrefix = ".tmp-"
|
||||
|
||||
// variantMetaSuffix is the sidecar suffix VariantStorage writes next
|
||||
// to each variant file.
|
||||
const variantMetaSuffix = ".meta"
|
||||
|
||||
// fallbackContentType is the content type given to a variant file that
|
||||
// has no readable .meta sidecar, when it is served or reconciled.
|
||||
const fallbackContentType = "application/octet-stream"
|
||||
|
||||
// UsageBytes returns the total number of bytes of cache content
|
||||
// tracked in the database, as the package function UsageBytes does, or 0
|
||||
// when the disk cache is off.
|
||||
// tracked in the database (source content blobs plus processed
|
||||
// variants). It reads the total the database keeps up to date as rows
|
||||
// are added and removed, so it neither scans the cache directories nor
|
||||
// sums the tables.
|
||||
func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
if c.disabled {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
return UsageBytes(ctx, c.db)
|
||||
}
|
||||
|
||||
// UsageBytes returns the total number of bytes of cache content tracked
|
||||
// in db (source content blobs plus processed variants). It reads the
|
||||
// total the database keeps up to date as rows are added and removed, so
|
||||
// it neither scans the cache directories nor sums the tables. pixad reads
|
||||
// it before it builds the cache, to work out the default cache_max_bytes.
|
||||
func UsageBytes(ctx context.Context, db *sql.DB) (int64, error) {
|
||||
var total int64
|
||||
|
||||
err := db.QueryRowContext(ctx,
|
||||
err := c.db.QueryRowContext(ctx,
|
||||
`SELECT total_size_bytes FROM cache_usage WHERE id = 1`,
|
||||
).Scan(&total)
|
||||
if err != nil {
|
||||
@@ -93,9 +95,9 @@ type evictionCandidate struct {
|
||||
|
||||
// EvictToLimit evicts least-recently-used cache entries until total
|
||||
// tracked usage is at or below the configured MaxBytes limit. It is a
|
||||
// no-op when the cache is disabled.
|
||||
// no-op when the cache is disabled or no limit is configured.
|
||||
func (c *Cache) EvictToLimit(ctx context.Context) error {
|
||||
if c.config.MaxBytes == 0 {
|
||||
if c.disabled || c.config.MaxBytes <= 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -296,7 +298,10 @@ func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, erro
|
||||
}
|
||||
|
||||
// evictVariant removes one variant: accounting row first, then the
|
||||
// file, so the database never references a deleted file.
|
||||
// content and .meta files, so the database never references a deleted
|
||||
// file. The metaCache entry goes before the files; a GetVariant that
|
||||
// read them just before may put it back, and the next GetVariant then
|
||||
// fails to open the file and removes it again.
|
||||
func (c *Cache) evictVariant(
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey,
|
||||
) error {
|
||||
@@ -306,7 +311,9 @@ func (c *Cache) evictVariant(
|
||||
return fmt.Errorf("failed to delete variant accounting row: %w", err)
|
||||
}
|
||||
|
||||
err = c.variants.Delete(string(cacheKey))
|
||||
c.metaCache.Remove(cacheKey)
|
||||
|
||||
err = c.variants.Delete(cacheKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -317,7 +324,7 @@ func (c *Cache) evictVariant(
|
||||
// evictSourceBlob removes one source: its row first, then its file, as
|
||||
// evictVariant does for a variant. It holds no lock: a store of the same
|
||||
// source in between can leave the source's row with no file, which
|
||||
// Source treats as a miss.
|
||||
// LookupSource treats as a miss.
|
||||
func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey) error {
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM source_content WHERE source_key = ?`, string(key))
|
||||
@@ -325,7 +332,7 @@ func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey)
|
||||
return fmt.Errorf("failed to delete source content row: %w", err)
|
||||
}
|
||||
|
||||
return c.srcContent.Delete(string(key))
|
||||
return c.srcContent.Delete(key)
|
||||
}
|
||||
|
||||
// notifyWritePressure wakes the background evictor after a store, so
|
||||
@@ -333,7 +340,7 @@ func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey)
|
||||
// storing request. The notification channel has capacity one and drops
|
||||
// when a wakeup is already pending.
|
||||
func (c *Cache) notifyWritePressure() {
|
||||
if c.config.MaxBytes == 0 {
|
||||
if c.disabled || c.config.MaxBytes <= 0 {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -479,7 +486,7 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
||||
// then on the first periodic eviction tick that comes at least ten times
|
||||
// as long after the last pass ended as that pass took, off the request
|
||||
// hot path: it adopts variant files that predate the accounting table
|
||||
// (or whose accounting insert failed, e.g. PutVariant's best-effort
|
||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||
// insert under transient DB contention), drops variant accounting rows
|
||||
// whose files are missing, removes source blob files the database does
|
||||
// not know, removes source and variant files that are empty or not the
|
||||
@@ -492,7 +499,7 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
||||
// Once ctx is cancelled, it stops at the next file or row and returns
|
||||
// ctx's error.
|
||||
func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||
if c.config.MaxBytes == 0 {
|
||||
if c.disabled {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -514,16 +521,12 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||
return c.reconcileUsageTotal(ctx)
|
||||
}
|
||||
|
||||
// reconcileFiles walks baseDir, one of the cache's storage directories,
|
||||
// sweeping stale temp files and passing each other file, with its name, to
|
||||
// reconcileFile. Once ctx is cancelled it stops at the next file and
|
||||
// returns ctx's error.
|
||||
func (c *Cache) reconcileFiles(
|
||||
ctx context.Context, baseDir string,
|
||||
reconcileFile func(path string, entry fs.DirEntry, name string) error,
|
||||
) error {
|
||||
// reconcileVariantFiles walks the variant storage directory, checking
|
||||
// each variant file against its accounting row (reconcileVariantFile)
|
||||
// and sweeping stale temp files.
|
||||
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||
return filepath.WalkDir(
|
||||
baseDir,
|
||||
c.variants.baseDir,
|
||||
func(path string, entry fs.DirEntry, err error) error {
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
@@ -541,21 +544,15 @@ func (c *Cache) reconcileFiles(
|
||||
return nil
|
||||
}
|
||||
|
||||
return reconcileFile(path, entry, name)
|
||||
if strings.HasSuffix(name, variantMetaSuffix) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return c.reconcileVariantFile(ctx, path, entry, imagerequest.VariantKey(name))
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
// reconcileVariantFiles walks the variant storage directory, checking
|
||||
// each variant file against its accounting row (reconcileVariantFile)
|
||||
// and sweeping stale temp files.
|
||||
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||
return c.reconcileFiles(ctx, c.variants.baseDir,
|
||||
func(path string, entry fs.DirEntry, name string) error {
|
||||
return c.reconcileVariantFile(ctx, path, entry, imagerequest.VariantKey(name))
|
||||
})
|
||||
}
|
||||
|
||||
// reconcileVariantFile checks one variant file against its accounting
|
||||
// row. A file without a row is adopted into the accounting, unless it is
|
||||
// empty. A file that is empty, or not the size its row records, as a
|
||||
@@ -577,7 +574,7 @@ func (c *Cache) reconcileVariantFile(
|
||||
|
||||
switch {
|
||||
case errors.Is(err, sql.ErrNoRows) && info.Size() > 0:
|
||||
return c.adoptVariantFile(ctx, info, cacheKey)
|
||||
return c.adoptVariantFile(ctx, path, info, cacheKey)
|
||||
case err != nil && !errors.Is(err, sql.ErrNoRows):
|
||||
return fmt.Errorf("failed to check variant accounting row: %w", err)
|
||||
case info.Size() > 0 && info.Size() == recordedSize:
|
||||
@@ -594,14 +591,16 @@ func (c *Cache) reconcileVariantFile(
|
||||
// adoptVariantFile inserts an accounting row for a variant file that
|
||||
// has none, using the file's size and modification time.
|
||||
func (c *Cache) adoptVariantFile(
|
||||
ctx context.Context, info fs.FileInfo, cacheKey imagerequest.VariantKey,
|
||||
ctx context.Context, path string, info fs.FileInfo, cacheKey imagerequest.VariantKey,
|
||||
) error {
|
||||
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
|
||||
contentType := c.variantContentTypeFromSidecar(path)
|
||||
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
INSERT INTO variant_content (cache_key, size_bytes, last_accessed_at)
|
||||
VALUES (?, ?, ?)
|
||||
`, string(cacheKey), info.Size(), modTime)
|
||||
INSERT INTO variant_content
|
||||
(cache_key, size_bytes, content_type, created_at, last_accessed_at)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
`, string(cacheKey), info.Size(), contentType, modTime, modTime)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to adopt variant file into accounting: %w", err)
|
||||
}
|
||||
@@ -612,6 +611,23 @@ func (c *Cache) adoptVariantFile(
|
||||
return nil
|
||||
}
|
||||
|
||||
// variantContentTypeFromSidecar reads the content type from a variant
|
||||
// .meta sidecar, falling back to application/octet-stream.
|
||||
func (c *Cache) variantContentTypeFromSidecar(variantPath string) string {
|
||||
//nolint:gosec // path from cache walk
|
||||
metaData, err := os.ReadFile(variantPath + variantMetaSuffix)
|
||||
if err != nil {
|
||||
return fallbackContentType
|
||||
}
|
||||
|
||||
var meta VariantMeta
|
||||
if json.Unmarshal(metaData, &meta) != nil || meta.ContentType == "" {
|
||||
return fallbackContentType
|
||||
}
|
||||
|
||||
return meta.ContentType
|
||||
}
|
||||
|
||||
// reconcileVariantRows drops accounting rows whose variant files are
|
||||
// missing, so the database never references deleted content.
|
||||
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||
@@ -636,12 +652,11 @@ func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
_, err := c.variants.Size(string(key))
|
||||
if err == nil {
|
||||
if c.variants.Exists(key) {
|
||||
continue
|
||||
}
|
||||
|
||||
_, err = c.db.ExecContext(ctx,
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
|
||||
@@ -704,10 +719,28 @@ func queryStringColumn[T ~string](
|
||||
// each blob file against its source_content row (reconcileSourceFile)
|
||||
// and sweeping stale temp files.
|
||||
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
||||
return c.reconcileFiles(ctx, c.srcContent.baseDir,
|
||||
func(path string, entry fs.DirEntry, name string) error {
|
||||
return filepath.WalkDir(
|
||||
c.srcContent.baseDir,
|
||||
func(path string, entry fs.DirEntry, err error) error {
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
|
||||
name := entry.Name()
|
||||
|
||||
if strings.HasPrefix(name, tempFilePrefix) {
|
||||
c.sweepStaleTempFile(path, entry)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return c.reconcileSourceFile(ctx, path, entry, imagerequest.SourceKey(name))
|
||||
})
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
// reconcileSourceFile checks one source blob file against its
|
||||
|
||||
@@ -14,9 +14,9 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"modernc.org/sqlite"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
@@ -66,12 +66,16 @@ func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
|
||||
tmpDir := t.TempDir()
|
||||
db := evictionTestDB(t)
|
||||
|
||||
// maxBytes zero turns the disk cache off, as cache_max_bytes: 0 does.
|
||||
cache, err := NewCache(db, Config{
|
||||
StateDir: tmpDir,
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: maxBytes,
|
||||
}, prometheus.NewRegistry())
|
||||
// maxBytes zero mirrors the production mapping of
|
||||
// cache_max_bytes: 0 (handlers sets DisableDiskCache); at the
|
||||
// CacheConfig layer itself a zero MaxBytes means "no limit" for
|
||||
// backwards compatibility with existing fixtures.
|
||||
cache, err := NewCache(db, CacheConfig{
|
||||
StateDir: tmpDir,
|
||||
NegativeTTL: 5 * time.Minute,
|
||||
MaxBytes: maxBytes,
|
||||
DisableDiskCache: maxBytes == 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
@@ -98,11 +102,20 @@ func storeEvictionTestSource(
|
||||
},
|
||||
}
|
||||
|
||||
key, err := cache.PutSource(
|
||||
context.Background(), req.Source, bytes.NewReader(content),
|
||||
result := &httpfetcher.FetchResult{
|
||||
StatusCode: 200,
|
||||
ContentType: testContentTypeJPEG,
|
||||
ContentLength: int64(len(content)),
|
||||
Headers: map[string][]string{
|
||||
testHeaderContentType: {testContentTypeJPEG},
|
||||
},
|
||||
}
|
||||
|
||||
key, err := cache.StoreSource(
|
||||
context.Background(), req, bytes.NewReader(content), result,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("PutSource(%s%s) failed: %v", host, path, err)
|
||||
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err)
|
||||
}
|
||||
|
||||
return key
|
||||
@@ -115,9 +128,9 @@ func storeEvictionTestVariant(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
err := cache.PutVariant(t.Context(), key, bytes.NewReader(content))
|
||||
err := cache.StoreVariant(t.Context(), key, bytes.NewReader(content), "image/webp")
|
||||
if err != nil {
|
||||
t.Fatalf("PutVariant(%s) failed: %v", key, err)
|
||||
t.Fatalf("StoreVariant(%s) failed: %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -209,8 +222,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
|
||||
t.Fatalf("failed to scan source_key: %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(key))
|
||||
if err != nil {
|
||||
if !cache.srcContent.Exists(imagerequest.SourceKey(key)) {
|
||||
t.Errorf("source_content references content %s but the file is missing",
|
||||
key)
|
||||
}
|
||||
@@ -237,8 +249,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
|
||||
t.Fatalf("failed to scan cache_key: %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.variants.keyToPath(key))
|
||||
if err != nil {
|
||||
if !cache.variants.Exists(imagerequest.VariantKey(key)) {
|
||||
t.Errorf("variant_content references key %s but the file is missing", key)
|
||||
}
|
||||
}
|
||||
@@ -336,8 +347,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(keys[0])))
|
||||
if err == nil {
|
||||
if cache.variants.Exists(keys[0]) {
|
||||
t.Errorf("least recently used variant %s must be evicted", keys[0])
|
||||
}
|
||||
|
||||
@@ -348,8 +358,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, key := range keys[1:] {
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(key)))
|
||||
if err != nil {
|
||||
if !cache.variants.Exists(key) {
|
||||
t.Errorf("more recently used variant %s must survive eviction", key)
|
||||
}
|
||||
}
|
||||
@@ -377,8 +386,7 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, key := range keys {
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(string(key)))
|
||||
if err != nil {
|
||||
if !cache.srcContent.Exists(key) {
|
||||
t.Errorf("source %s must not be evicted while usage is under the limit", key)
|
||||
}
|
||||
|
||||
@@ -389,8 +397,7 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(testVariantKeyOne)))
|
||||
if err != nil {
|
||||
if !cache.variants.Exists(testVariantKeyOne) {
|
||||
t.Error("variant must not be evicted while usage is under the limit")
|
||||
}
|
||||
|
||||
@@ -429,20 +436,27 @@ func assertDisabledCacheWritesAreNoOps(
|
||||
|
||||
ctx := t.Context()
|
||||
|
||||
err := cache.PutVariant(
|
||||
ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")),
|
||||
err := cache.StoreVariant(
|
||||
ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")), "image/webp",
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("PutVariant on disabled cache must be a no-op, got error: %v", err)
|
||||
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err)
|
||||
}
|
||||
|
||||
key, err := cache.PutSource(ctx, req.Source, bytes.NewReader([]byte("data")))
|
||||
result := &httpfetcher.FetchResult{
|
||||
StatusCode: 200,
|
||||
ContentType: testContentTypeJPEG,
|
||||
ContentLength: 4,
|
||||
Headers: map[string][]string{},
|
||||
}
|
||||
|
||||
key, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")), result)
|
||||
if err != nil {
|
||||
t.Fatalf("PutSource on disabled cache must be a no-op, got error: %v", err)
|
||||
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err)
|
||||
}
|
||||
|
||||
if key != "" {
|
||||
t.Errorf("PutSource on disabled cache returned key %q, want empty", key)
|
||||
t.Errorf("StoreSource on disabled cache returned key %q, want empty", key)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -455,13 +469,23 @@ func assertDisabledCacheReadsAlwaysMiss(
|
||||
|
||||
ctx := t.Context()
|
||||
|
||||
reader, _, _, err := cache.Source(ctx, req.Source)
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("Source on disabled cache error = %v, want %v", err, ErrNotFound)
|
||||
lookup, err := cache.Lookup(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup on disabled cache failed: %v", err)
|
||||
}
|
||||
|
||||
if reader != nil {
|
||||
t.Errorf("Source on disabled cache = reader %v, want none", reader)
|
||||
if lookup.Hit {
|
||||
t.Error("Lookup on disabled cache must always miss")
|
||||
}
|
||||
|
||||
key, srcType, err := cache.LookupSource(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("LookupSource on disabled cache failed: %v", err)
|
||||
}
|
||||
|
||||
if key != "" || srcType != "" {
|
||||
t.Errorf("LookupSource on disabled cache = (%q, %q), want empty",
|
||||
key, srcType)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -550,7 +574,7 @@ func TestEvictionRunsUnderWritePressure(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestEvictionRunsOnPeriodicSchedule writes three variant files straight
|
||||
// to disk, bypassing PutVariant, so they have no accounting rows and no
|
||||
// to disk, bypassing StoreVariant, so they have no accounting rows and no
|
||||
// write-pressure notification fires. Only a periodic reconciliation pass
|
||||
// can then adopt them, and only the eviction pass that follows it can
|
||||
// evict them.
|
||||
@@ -592,7 +616,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
||||
for i, key := range keys {
|
||||
content := bytes.Repeat([]byte{fills[i]}, 1000)
|
||||
|
||||
err = cache.variants.Write(string(key), content)
|
||||
_, err = cache.variants.Store(key, bytes.NewReader(content), "image/webp")
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store variant file: %v", err)
|
||||
}
|
||||
@@ -611,8 +635,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
||||
stored = 0
|
||||
|
||||
for _, key := range keys {
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(key)))
|
||||
if err == nil {
|
||||
if cache.variants.Exists(key) {
|
||||
stored++
|
||||
}
|
||||
}
|
||||
@@ -644,15 +667,18 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
|
||||
// feature existed) must be adopted into the accounting.
|
||||
untracked := bytes.Repeat([]byte{0x31}, 1000)
|
||||
|
||||
err := cache.variants.Write(string(testVariantKeyOne), untracked)
|
||||
_, err := cache.variants.Store(
|
||||
testVariantKeyOne, bytes.NewReader(untracked), "image/webp",
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store untracked variant file: %v", err)
|
||||
}
|
||||
|
||||
// An accounting row whose file is missing must be dropped.
|
||||
_, err = cache.db.ExecContext(t.Context(),
|
||||
`INSERT INTO variant_content (cache_key, size_bytes) VALUES (?, ?)`,
|
||||
"deadbeef0001", 700,
|
||||
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
||||
VALUES (?, ?, ?)`,
|
||||
"deadbeef0001", 700, "image/webp",
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert stale variant accounting row: %v", err)
|
||||
@@ -702,7 +728,7 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
|
||||
// TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves
|
||||
// reconciliation is not a one-shot startup-only pass: it must also run
|
||||
// on the periodic ticker, so a variant file that lands on disk with no
|
||||
// accounting row well after startup (e.g. because PutVariant's
|
||||
// accounting row well after startup (e.g. because StoreVariant's
|
||||
// best-effort accounting insert failed under transient contention, or
|
||||
// any other cause of an untracked file appearing during steady-state
|
||||
// operation) is still adopted into accounting eventually, rather than
|
||||
@@ -738,12 +764,14 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
|
||||
|
||||
// Simulate a variant whose accounting insert failed after the
|
||||
// process was already running and serving requests: the content
|
||||
// file is written directly, bypassing PutVariant's (and thus its
|
||||
// file is written directly, bypassing StoreVariant's (and thus its
|
||||
// accounting insert) entirely, exactly as would happen if that
|
||||
// insert had failed and only the file write had succeeded.
|
||||
untracked := bytes.Repeat([]byte{0x41}, 900)
|
||||
|
||||
err = cache.variants.Write("aabbccdd0099", untracked)
|
||||
_, err = cache.variants.Store(
|
||||
"aabbccdd0099", bytes.NewReader(untracked), "image/webp",
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store untracked variant file: %v", err)
|
||||
}
|
||||
@@ -1049,15 +1077,13 @@ func assertOnlyOldestSourceEvicted(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
_, err := os.Stat(cache.srcContent.keyToPath(string(keys[0])))
|
||||
if err == nil {
|
||||
if cache.srcContent.Exists(keys[0]) {
|
||||
t.Errorf("source %s, evicted when the context was cancelled, "+
|
||||
"is still on disk", keys[0])
|
||||
}
|
||||
|
||||
for _, key := range keys[1:] {
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(string(key)))
|
||||
if err != nil {
|
||||
if !cache.srcContent.Exists(key) {
|
||||
t.Errorf("source %s was evicted after the context was cancelled", key)
|
||||
}
|
||||
|
||||
|
||||
@@ -1,185 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Storage constants.
|
||||
const (
|
||||
// StorageDirPerm is the permission mode for storage directories.
|
||||
StorageDirPerm = 0750
|
||||
// StorageFilePerm is the permission mode for storage files.
|
||||
StorageFilePerm = 0600
|
||||
// MinHashLength is the minimum hash length for path splitting.
|
||||
MinHashLength = 4
|
||||
)
|
||||
|
||||
// Storage errors.
|
||||
var (
|
||||
ErrNotFound = errors.New("content not found")
|
||||
)
|
||||
|
||||
// FileStore is a directory of files, each named by a hex key. The file for
|
||||
// a key is at <basedir>/<ab>/<cd>/<key>, where <ab> and <cd> are the key's
|
||||
// first and second pairs of characters. The cache keeps its source images in
|
||||
// one FileStore and its variants in another.
|
||||
type FileStore struct {
|
||||
baseDir string
|
||||
}
|
||||
|
||||
// NewFileStore creates the directory baseDir if it is missing and returns
|
||||
// the FileStore of the files in it.
|
||||
func NewFileStore(baseDir string) (*FileStore, error) {
|
||||
err := os.MkdirAll(baseDir, StorageDirPerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create storage directory: %w", err)
|
||||
}
|
||||
|
||||
return &FileStore{baseDir: baseDir}, nil
|
||||
}
|
||||
|
||||
// Write writes data with writeFileSynced to the file for key, replacing any
|
||||
// file already there.
|
||||
func (s *FileStore) Write(key string, data []byte) error {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
return writeFileSynced(path, data)
|
||||
}
|
||||
|
||||
// Open opens the file for key and returns it with its size in bytes. A
|
||||
// missing file is ErrNotFound.
|
||||
func (s *FileStore) Open(key string) (*os.File, int64, error) {
|
||||
f, err := os.Open(s.keyToPath(key))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
return nil, 0, fmt.Errorf("failed to open content: %w", err)
|
||||
}
|
||||
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
|
||||
return nil, 0, fmt.Errorf("failed to stat content: %w", err)
|
||||
}
|
||||
|
||||
return f, stat.Size(), nil
|
||||
}
|
||||
|
||||
// Size returns the size in bytes of the file for key. A missing file is
|
||||
// ErrNotFound.
|
||||
func (s *FileStore) Size(key string) (int64, error) {
|
||||
stat, err := os.Stat(s.keyToPath(key))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return 0, ErrNotFound
|
||||
}
|
||||
|
||||
return 0, fmt.Errorf("failed to stat content: %w", err)
|
||||
}
|
||||
|
||||
return stat.Size(), nil
|
||||
}
|
||||
|
||||
// Delete removes the file for key. A missing file is not an error.
|
||||
func (s *FileStore) Delete(key string) error {
|
||||
err := os.Remove(s.keyToPath(key))
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to delete content: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeFileSynced writes data to path through a temp file in the same
|
||||
// directory: the temp file is synced to disk, renamed to path, and the
|
||||
// directory synced after, so that neither a concurrent reader nor a
|
||||
// crash finds a partial file under path. When the directory sync fails,
|
||||
// the file is removed from path, so that a failed write leaves no file
|
||||
// there.
|
||||
func writeFileSynced(path string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
|
||||
tmpPath := tmpFile.Name()
|
||||
|
||||
_, err = tmpFile.Write(data)
|
||||
if err != nil {
|
||||
_ = tmpFile.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to write temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmpFile.Sync()
|
||||
if err != nil {
|
||||
_ = tmpFile.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to sync temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmpFile.Close()
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to close temp file: %w", err)
|
||||
}
|
||||
|
||||
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||
}
|
||||
|
||||
err = syncDirectory(dir)
|
||||
if err != nil {
|
||||
_ = os.Remove(path)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// syncDirectory syncs the directory dir to disk, so that a file just
|
||||
// renamed into it is still there after a crash.
|
||||
func syncDirectory(dir string) error {
|
||||
d, err := os.Open(filepath.Clean(dir))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to open directory: %w", err)
|
||||
}
|
||||
|
||||
err = d.Sync()
|
||||
_ = d.Close()
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to sync directory: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
|
||||
func (s *FileStore) keyToPath(key string) string {
|
||||
if len(key) < MinHashLength {
|
||||
return filepath.Join(s.baseDir, key)
|
||||
}
|
||||
|
||||
return filepath.Join(s.baseDir, key[0:2], key[2:4], key)
|
||||
}
|
||||
@@ -1,184 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func TestFileStore_WriteAndOpen(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("hello world")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify file exists at expected path
|
||||
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
t.Errorf("File not at expected path %s: %v", expectedPath, err)
|
||||
}
|
||||
|
||||
// Open and verify size and content
|
||||
r, size, err := storage.Open(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Open() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = r.Close() }()
|
||||
|
||||
if size != int64(len(content)) {
|
||||
t.Errorf("Open() size = %d, want %d", size, len(content))
|
||||
}
|
||||
|
||||
loaded, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadAll() error = %v", err)
|
||||
}
|
||||
|
||||
if !bytes.Equal(loaded, content) {
|
||||
t.Errorf("Open() content = %q, want %q", loaded, content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_OpenNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
_, _, err = storage.Open("nonexistent")
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Open() error = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileStore_Size checks that Size returns the size of a written file,
|
||||
// and ErrNotFound for a key with no file.
|
||||
func TestFileStore_Size(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
storage, err := NewFileStore(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("the file's content")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
size, err := storage.Size(key)
|
||||
if err != nil || size != int64(len(content)) {
|
||||
t.Errorf("Size() = %d, error %v; want %d", size, err, len(content))
|
||||
}
|
||||
|
||||
_, err = storage.Size("nonexistent")
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Size() of a missing file error = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_Delete(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("to be deleted")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
path := storage.keyToPath(key)
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(path)
|
||||
if err != nil {
|
||||
t.Errorf("file missing before Delete(): %v", err)
|
||||
}
|
||||
|
||||
err = storage.Delete(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(path)
|
||||
if err == nil {
|
||||
t.Error("file still on disk after Delete()")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_DeleteNonexistent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
// Should not error
|
||||
err = storage.Delete("nonexistent")
|
||||
if err != nil {
|
||||
t.Errorf("Delete() error = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_KeyToPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
// Test by writing and verifying the resulting path structure
|
||||
content := []byte("test content for path verification")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
t.Errorf("File not at expected path %s: %v", expectedPath, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
// Package imgcache provides interfaces and types for the image caching proxy.
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// ImageResponse represents a processed image ready to serve
|
||||
type ImageResponse struct {
|
||||
// Content is the image data reader
|
||||
Content io.ReadCloser
|
||||
// ContentLength is the size in bytes (-1 if unknown)
|
||||
ContentLength int64
|
||||
// ContentType is the MIME type of the response
|
||||
ContentType string
|
||||
// ETag is the entity tag for caching
|
||||
ETag string
|
||||
// LastModified is when the content was last modified
|
||||
LastModified time.Time
|
||||
// CacheStatus indicates HIT, MISS, or STALE
|
||||
CacheStatus CacheStatus
|
||||
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
|
||||
FetchedBytes int64
|
||||
}
|
||||
|
||||
// CacheStatus indicates whether the response was served from cache.
|
||||
type CacheStatus string
|
||||
|
||||
// Cache status values for response headers.
|
||||
const (
|
||||
CacheHit CacheStatus = "HIT"
|
||||
CacheMiss CacheStatus = "MISS"
|
||||
CacheStale CacheStatus = "STALE"
|
||||
)
|
||||
|
||||
// ImageCache is the main interface for the image caching proxy
|
||||
type ImageCache interface {
|
||||
// Get retrieves a processed image, fetching and processing if necessary
|
||||
Get(ctx context.Context, req *imagerequest.Request) (*ImageResponse, error)
|
||||
|
||||
// Warm pre-fetches and caches an image without returning it
|
||||
Warm(ctx context.Context, req *imagerequest.Request) error
|
||||
|
||||
// Stats returns cache statistics
|
||||
Stats(ctx context.Context) (*CacheStats, error)
|
||||
}
|
||||
|
||||
// CacheStats contains cache statistics
|
||||
type CacheStats struct {
|
||||
// TotalItems is the number of cached source images plus processed
|
||||
// variants
|
||||
TotalItems int64
|
||||
// TotalSizeBytes is the total size of cached source images and
|
||||
// processed variants
|
||||
TotalSizeBytes int64
|
||||
// HitCount is the number of cache hits
|
||||
HitCount int64
|
||||
// MissCount is the number of cache misses
|
||||
MissCount int64
|
||||
// HitRate is HitCount / (HitCount + MissCount)
|
||||
HitRate float64
|
||||
}
|
||||
+16
-20
@@ -1,4 +1,4 @@
|
||||
package imageproxy
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"image/color"
|
||||
@@ -41,8 +41,7 @@ func holdProcessingSlot(
|
||||
input, feed := io.Pipe()
|
||||
|
||||
go func() {
|
||||
_, _ = proc.Process(t.Context(), input, imagerequest.FormatJPEG,
|
||||
&imagerequest.Transform{})
|
||||
_, _ = proc.Process(t.Context(), input, &imagerequest.Transform{})
|
||||
}()
|
||||
|
||||
_, err := feed.Write([]byte{0})
|
||||
@@ -56,19 +55,18 @@ func holdProcessingSlot(
|
||||
return release
|
||||
}
|
||||
|
||||
// TestProxy_Get_WaitsForSlotBeforeReadingCachedSource checks that a
|
||||
// TestService_Get_WaitsForSlotBeforeReadingCachedSource checks that a
|
||||
// request whose source is cached holds none of it while it waits for a
|
||||
// processing slot: past the first bytes, which give the source's format, it
|
||||
// reads the cached file only once it has a slot. With the only slot held, a
|
||||
// request for a new width of the cached 100x100 photo waits; the cached file
|
||||
// is then rewritten as a 100x50 image before the slot is freed, so the
|
||||
// request must answer with that image scaled to 40x20.
|
||||
func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
// processing slot: it reads the cached file only once it has a slot. With
|
||||
// the only slot held, a request for a new width of the cached 100x100 photo
|
||||
// waits; the cached file is then rewritten as a 100x50 image before the slot
|
||||
// is freed, so the request must answer with that image scaled to 40x20.
|
||||
func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
svc, fixtures := SetupTestService(t)
|
||||
svc.processor = imageprocessor.New(
|
||||
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
|
||||
imageprocessor.Params{MaxConcurrentProcessing: 1},
|
||||
)
|
||||
|
||||
// A first request caches the photo as a source.
|
||||
@@ -79,15 +77,13 @@ func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
cachedSource := widthOnlyRequest(fixtures, 50).Source
|
||||
|
||||
reader, _, _, err := svc.cache.Source(t.Context(), cachedSource)
|
||||
if err != nil {
|
||||
t.Fatalf("Source() error = %v; want the cached source", err)
|
||||
sourceKey, _, err := svc.cache.LookupSource(t.Context(),
|
||||
widthOnlyRequest(fixtures, 50))
|
||||
if err != nil || sourceKey == "" {
|
||||
t.Fatalf("LookupSource() = %q, %v; want the cached source",
|
||||
sourceKey, err)
|
||||
}
|
||||
|
||||
_ = reader.Close()
|
||||
|
||||
release := holdProcessingSlot(t, svc.processor)
|
||||
|
||||
var (
|
||||
@@ -107,7 +103,7 @@ func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
// before waiting, it would have read it by now.
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
err = os.WriteFile(sourceFilePath(svc, cachedSource.Key()),
|
||||
err = os.WriteFile(svc.cache.srcContent.keyToPath(sourceKey),
|
||||
generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to rewrite the cached source: %v", err)
|
||||
@@ -0,0 +1,361 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// webpRequest returns a request for a 100x100 WebP variant of path.
|
||||
func webpRequest(path string) *imagerequest.Request {
|
||||
return &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: path,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100, Height: 100},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// assertVariantServed checks that GetVariant serves key with the given
|
||||
// content and the image/webp content type storeEvictionTestVariant stores.
|
||||
func assertVariantServed(
|
||||
t *testing.T, cache *Cache, key imagerequest.VariantKey, content []byte,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
reader, size, contentType, err := cache.GetVariant(key)
|
||||
if err != nil {
|
||||
t.Fatalf("GetVariant(%s) error = %v", key, err)
|
||||
}
|
||||
|
||||
defer func() { _ = reader.Close() }()
|
||||
|
||||
got, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
t.Fatalf("reading variant %s: %v", key, err)
|
||||
}
|
||||
|
||||
if !bytes.Equal(got, content) {
|
||||
t.Errorf("GetVariant(%s) content = %q, want %q", key, got, content)
|
||||
}
|
||||
|
||||
if size != int64(len(content)) {
|
||||
t.Errorf("GetVariant(%s) size = %d, want %d", key, size, len(content))
|
||||
}
|
||||
|
||||
if contentType != testContentTypeWebP {
|
||||
t.Errorf("GetVariant(%s) content type = %q, want %q",
|
||||
key, contentType, testContentTypeWebP)
|
||||
}
|
||||
}
|
||||
|
||||
// assertVariantNotFound checks that GetVariant refuses key with
|
||||
// ErrNotFound.
|
||||
func assertVariantNotFound(
|
||||
t *testing.T, cache *Cache, key imagerequest.VariantKey,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
reader, _, _, err := cache.GetVariant(key)
|
||||
if err == nil {
|
||||
_ = reader.Close()
|
||||
}
|
||||
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("GetVariant(%s) error = %v, want ErrNotFound", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
// assertLookupMisses checks that Lookup reports request as a miss.
|
||||
func assertLookupMisses(
|
||||
t *testing.T, cache *Cache, request *imagerequest.Request,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
lookup, err := cache.Lookup(t.Context(), request)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup(%s) error = %v", request.Source.Path, err)
|
||||
}
|
||||
|
||||
if lookup.Hit {
|
||||
t.Errorf("Lookup(%s) is a hit, want a miss", request.Source.Path)
|
||||
}
|
||||
}
|
||||
|
||||
// renameFile renames the file at from to to.
|
||||
func renameFile(t *testing.T, from, to string) {
|
||||
t.Helper()
|
||||
|
||||
err := os.Rename(from, to)
|
||||
if err != nil {
|
||||
t.Fatalf("renaming %s: %v", from, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSecondHitDoesNotReadMetaFile checks that once a variant has been
|
||||
// stored or read, a hit takes its content type from memory: with the
|
||||
// .meta file deleted, GetVariant must still return the stored content
|
||||
// type rather than the application/octet-stream it uses without one.
|
||||
func TestSecondHitDoesNotReadMetaFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<20)
|
||||
content := []byte("webp variant bytes")
|
||||
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
|
||||
|
||||
// A second Cache on the same state directory starts with nothing in
|
||||
// memory, as pixad does after a restart, so its first read uses the
|
||||
// .meta file.
|
||||
restarted, err := NewCache(cache.db, cache.config)
|
||||
if err != nil {
|
||||
t.Fatalf("NewCache() error = %v", err)
|
||||
}
|
||||
|
||||
assertVariantServed(t, restarted, testVariantKeyOne, content)
|
||||
|
||||
err = os.Remove(cache.variants.keyToPath(testVariantKeyOne) + ".meta")
|
||||
if err != nil {
|
||||
t.Fatalf("removing .meta file: %v", err)
|
||||
}
|
||||
|
||||
assertVariantServed(t, cache, testVariantKeyOne, content)
|
||||
assertVariantServed(t, restarted, testVariantKeyOne, content)
|
||||
}
|
||||
|
||||
// TestReadDuringStoreKeepsStoredContentType checks that a GetVariant
|
||||
// which began before StoreVariant finished cannot replace the content
|
||||
// type the store kept in memory. Such a read can find the variant file
|
||||
// but not yet its .meta file, and so gets application/octet-stream. The
|
||||
// test deletes the .meta file after the store, then runs the part of
|
||||
// GetVariant that comes after its check of memory.
|
||||
func TestReadDuringStoreKeepsStoredContentType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<20)
|
||||
content := []byte("webp variant bytes")
|
||||
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
|
||||
|
||||
err := os.Remove(cache.variants.keyToPath(testVariantKeyOne) + ".meta")
|
||||
if err != nil {
|
||||
t.Fatalf("removing .meta file: %v", err)
|
||||
}
|
||||
|
||||
reader, _, contentType, err := cache.loadVariantWithMeta(testVariantKeyOne)
|
||||
if err != nil {
|
||||
t.Fatalf("loadVariantWithMeta(%s) error = %v", testVariantKeyOne, err)
|
||||
}
|
||||
|
||||
_ = reader.Close()
|
||||
|
||||
t.Logf("the read without a .meta file got content type %q", contentType)
|
||||
|
||||
assertVariantServed(t, cache, testVariantKeyOne, content)
|
||||
}
|
||||
|
||||
// TestReadOfOlderMetaFileKeepsStoredContentType checks that a read
|
||||
// which got its content type from a .meta file that StoreVariant had
|
||||
// not yet rewritten cannot replace the type the store kept in memory.
|
||||
// The test writes such a .meta file, with a different content type,
|
||||
// after the store, then runs the part of GetVariant that comes after
|
||||
// its check of memory.
|
||||
func TestReadOfOlderMetaFileKeepsStoredContentType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<20)
|
||||
content := []byte("webp variant bytes")
|
||||
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
|
||||
|
||||
olderMeta, err := json.Marshal(VariantMeta{
|
||||
ContentType: testContentTypeJPEG,
|
||||
Size: int64(len(content)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encoding .meta file: %v", err)
|
||||
}
|
||||
|
||||
metaPath := cache.variants.keyToPath(testVariantKeyOne) + ".meta"
|
||||
|
||||
err = os.WriteFile(metaPath, olderMeta, StorageFilePerm)
|
||||
if err != nil {
|
||||
t.Fatalf("writing .meta file: %v", err)
|
||||
}
|
||||
|
||||
reader, _, contentType, err := cache.loadVariantWithMeta(testVariantKeyOne)
|
||||
if err != nil {
|
||||
t.Fatalf("loadVariantWithMeta(%s) error = %v", testVariantKeyOne, err)
|
||||
}
|
||||
|
||||
_ = reader.Close()
|
||||
|
||||
if contentType != testContentTypeJPEG {
|
||||
t.Fatalf("loadVariantWithMeta(%s) content type = %q, want %q from the .meta file",
|
||||
testVariantKeyOne, contentType, testContentTypeJPEG)
|
||||
}
|
||||
|
||||
kept, _ := cache.metaCache.Get(testVariantKeyOne)
|
||||
if kept != testContentTypeWebP {
|
||||
t.Errorf("content type in memory = %q, want the stored %q",
|
||||
kept, testContentTypeWebP)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailedReadDuringStoreKeepsStoredContentType checks that a read
|
||||
// which found no .meta file cannot leave application/octet-stream in
|
||||
// memory, even when another read has removed the content type
|
||||
// StoreVariant kept there. In this order: the store; a read that finds
|
||||
// the variant in memory but cannot open its file, and so removes it
|
||||
// from memory; a read that opened the variant file before the store
|
||||
// wrote its .meta file. Later hits must get the stored content type.
|
||||
func TestFailedReadDuringStoreKeepsStoredContentType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<20)
|
||||
content := []byte("webp variant bytes")
|
||||
variantPath := cache.variants.keyToPath(testVariantKeyOne)
|
||||
metaPath := variantPath + ".meta"
|
||||
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
|
||||
|
||||
renameFile(t, variantPath, variantPath+".hidden")
|
||||
assertVariantNotFound(t, cache, testVariantKeyOne)
|
||||
renameFile(t, variantPath+".hidden", variantPath)
|
||||
|
||||
renameFile(t, metaPath, metaPath+".hidden")
|
||||
|
||||
reader, _, contentType, err := cache.GetVariant(testVariantKeyOne)
|
||||
if err != nil {
|
||||
t.Fatalf("GetVariant(%s) error = %v", testVariantKeyOne, err)
|
||||
}
|
||||
|
||||
_ = reader.Close()
|
||||
|
||||
t.Logf("the read without a .meta file got content type %q", contentType)
|
||||
|
||||
renameFile(t, metaPath+".hidden", metaPath)
|
||||
|
||||
assertVariantServed(t, cache, testVariantKeyOne, content)
|
||||
}
|
||||
|
||||
// TestEvictedVariantIsNotServed checks that a variant the evictor
|
||||
// removed is a miss and cannot be read, although it had been stored
|
||||
// and served before.
|
||||
func TestEvictedVariantIsNotServed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1500)
|
||||
|
||||
oldRequest := webpRequest("/old.jpg")
|
||||
newRequest := webpRequest("/new.jpg")
|
||||
oldKey := imagerequest.CacheKey(oldRequest)
|
||||
newKey := imagerequest.CacheKey(newRequest)
|
||||
oldContent := bytes.Repeat([]byte{0x01}, 1000)
|
||||
newContent := bytes.Repeat([]byte{0x02}, 1000)
|
||||
|
||||
storeEvictionTestVariant(t, cache, oldKey, oldContent)
|
||||
storeEvictionTestVariant(t, cache, newKey, newContent)
|
||||
assertVariantServed(t, cache, oldKey, oldContent)
|
||||
setVariantLastAccessed(t, cache, oldKey, time.Now().Add(-time.Hour))
|
||||
|
||||
err := cache.EvictToLimit(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("EvictToLimit() error = %v", err)
|
||||
}
|
||||
|
||||
assertLookupMisses(t, cache, oldRequest)
|
||||
assertVariantNotFound(t, cache, oldKey)
|
||||
assertVariantServed(t, cache, newKey, newContent)
|
||||
}
|
||||
|
||||
// TestVariantDeletedFromDiskIsNotServed checks that a variant whose
|
||||
// file was deleted by something other than the evictor cannot be read,
|
||||
// and is a miss afterwards.
|
||||
func TestVariantDeletedFromDiskIsNotServed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<20)
|
||||
|
||||
request := webpRequest("/deleted.jpg")
|
||||
key := imagerequest.CacheKey(request)
|
||||
content := []byte("webp variant bytes")
|
||||
|
||||
storeEvictionTestVariant(t, cache, key, content)
|
||||
assertVariantServed(t, cache, key, content)
|
||||
|
||||
err := os.Remove(cache.variants.keyToPath(key))
|
||||
if err != nil {
|
||||
t.Fatalf("removing variant file: %v", err)
|
||||
}
|
||||
|
||||
assertVariantNotFound(t, cache, key)
|
||||
assertLookupMisses(t, cache, request)
|
||||
}
|
||||
|
||||
// TestConcurrentVariantStoreReadAndEvict stores, reads and evicts
|
||||
// variants from several goroutines at once, for the race detector.
|
||||
func TestConcurrentVariantStoreReadAndEvict(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<20)
|
||||
ctx := t.Context()
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
for goroutine := range 8 {
|
||||
wg.Go(func() {
|
||||
key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd01%02d", goroutine))
|
||||
content := []byte(key)
|
||||
|
||||
for range 20 {
|
||||
err := cache.StoreVariant(
|
||||
ctx, key, bytes.NewReader(content), testContentTypeWebP)
|
||||
if err != nil {
|
||||
t.Errorf("StoreVariant(%s) error = %v", key, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
reader, _, contentType, err := cache.GetVariant(key)
|
||||
if err != nil {
|
||||
t.Errorf("GetVariant(%s) error = %v", key, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
_ = reader.Close()
|
||||
|
||||
if contentType != testContentTypeWebP {
|
||||
t.Errorf("GetVariant(%s) content type = %q, want %q",
|
||||
key, contentType, testContentTypeWebP)
|
||||
}
|
||||
|
||||
err = cache.evictVariant(ctx, key)
|
||||
if err != nil {
|
||||
t.Errorf("evictVariant(%s) error = %v", key, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
assertVariantNotFound(t, cache, key)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
package imgcache
|
||||
|
||||
import "time"
|
||||
|
||||
// CacheConfig defaults.
|
||||
const (
|
||||
DefaultNegativeTTL = 5 * time.Minute
|
||||
)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user