Compare commits
1
Commits
next
..
ef70bb6fa0
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
ef70bb6fa0 |
@@ -1,76 +1,75 @@
|
||||
# Repository Rules
|
||||
|
||||
Last Updated 2026-10-27
|
||||
Last Updated 2026-01-08
|
||||
|
||||
These rules MUST be followed at all times. This is very important.
|
||||
These rules MUST be followed at all times, it is very important.
|
||||
|
||||
- Never use `git add -A`. Stage specific changes for a deliberate commit. A
|
||||
commit holds one change. After each change, make a commit with a good one-line
|
||||
summary.
|
||||
- Never use `git add -A` - add specific changes to a deliberate commit. A commit
|
||||
should contain one change. After each change, make a commit with a good
|
||||
one-line summary.
|
||||
|
||||
- NEVER modify the linter config without asking first.
|
||||
|
||||
- NEVER modify a test to exclude special cases or otherwise make it pass without
|
||||
asking first. In almost all cases, change the code, NOT the test. You need
|
||||
explicit user approval to modify an existing test. If you think a test needs
|
||||
to change, make your case and ask for permission to proceed. Then stop. A bug
|
||||
fix may change an existing test without asking only when that test pins
|
||||
exactly the behavior the filed bug calls wrong. The change must then move the
|
||||
test to the fixed behavior and nothing else. Any other change to an existing
|
||||
test needs explicit user approval. Writing NEW tests needs no approval.
|
||||
- NEVER modify tests to exclude special cases or otherwise get them to pass
|
||||
without asking first. In almost all cases, the code should be changed, NOT the
|
||||
tests. If you think the test needs to be changed, make your case for that and
|
||||
ask for permission to proceed, then stop. You need explicit user approval to
|
||||
modify existing tests. A bug fix may change an existing test without asking
|
||||
when that test pins exactly the behavior the filed bug calls wrong, and the
|
||||
change moves it to the fixed behavior and nothing else; any other change to an
|
||||
existing test still needs explicit user approval. (You do not need user
|
||||
approval for writing NEW tests.)
|
||||
|
||||
- When linting, assume the linter config is CORRECT. Treat each item the linter
|
||||
reports as something that legitimately needs fixing in the code.
|
||||
- When linting, assume the linter config is CORRECT, and that each item output
|
||||
by the linter is something that legitimately needs fixing in the code.
|
||||
|
||||
- When running tests, use `make test`.
|
||||
|
||||
- Before committing, run `make check`. It runs `make test`, `make lint` and
|
||||
`make fmt-check`. Any issue it finds MUST be resolved before committing,
|
||||
unless you are explicitly told otherwise.
|
||||
- Before commits, run `make check`. This runs `make lint` and `make test` and
|
||||
`make check-fmt`. Any issues discovered MUST be resolved before committing
|
||||
unless explicitly told otherwise.
|
||||
|
||||
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate
|
||||
logging to the test, to make sure the test is written correctly. Commit that.
|
||||
Then work on the fix until the test passes, without changing the test further.
|
||||
Then commit the fix.
|
||||
logging to the test to ensure it is written correctly. Commit that. Then go
|
||||
about fixing the bug until the test passes (without modifying the test
|
||||
further). Then commit that.
|
||||
|
||||
- When adding a new feature, do the same: write a test first (test-driven
|
||||
development, TDD). The test does not need to be very complex. Commit the test,
|
||||
then commit the feature.
|
||||
- When adding a new feature, do the same - implement a test first (TDD). It
|
||||
doesn't have to be super complex. Commit the test, then commit the feature.
|
||||
|
||||
- When adding a new feature, use a feature branch. The feature branch reaches
|
||||
`next` through a PR that passes review. It is merged only when the feature is
|
||||
completely finished and the code is up to standards. The code is up to
|
||||
standards when it passes `make check`. The branch is deleted after the merge.
|
||||
Only the user merges `next` into `main`.
|
||||
- When adding a new feature, use a feature branch. When the feature is
|
||||
completely finished and the code is up to standards (passes `make check`) then
|
||||
and only then can the feature branch be merged into `main` and the branch
|
||||
deleted.
|
||||
|
||||
- Write `godoc` comments for all exported types and functions as you go along.
|
||||
- Write godoc documentation comments for all exported types and functions as you
|
||||
go along.
|
||||
|
||||
- ALWAYS be consistent in naming. If you name something one way in one place,
|
||||
use the EXACT SAME name in every other place.
|
||||
- ALWAYS be consistent in naming. If you name something one thing in one place,
|
||||
name it the EXACT SAME THING in another place.
|
||||
|
||||
- Be descriptive and specific in naming. `al` is bad; `SourceHostAllowlist` is
|
||||
- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is
|
||||
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
|
||||
|
||||
- This is not prototype or teaching code. It is designed for production. Any
|
||||
security issue (such as denial of service) or other web vulnerability is a P1
|
||||
bug. It must be added at the top of `TODO.md`.
|
||||
- This is not prototype or teaching code - this is designed for production. Any
|
||||
security issues (such as denial of service) or other web vulnerabilities are
|
||||
P1 bugs and must be added to TODO.md at the top.
|
||||
|
||||
- Do not stub an implementation unless you are specifically instructed to. This
|
||||
is production code, and we need working implementations.
|
||||
- As this is production code, no stubbing of implementations unless specifically
|
||||
instructed. We need working implementations.
|
||||
|
||||
- NEVER silently fall back to a different setting when a user's parameter
|
||||
explicitly specifies a value. For example, if a user requests `format=webp`
|
||||
and WebP encoding is not supported, return an error. Do NOT silently output
|
||||
PNG instead. If a user specifies `fit=invalid` and that fit mode does not
|
||||
exist, return an error. Do NOT silently default to `cover`. A silent fallback
|
||||
violates the principle of least surprise and masks bugs. Defaults are
|
||||
acceptable only for OMITTED parameters, never for INVALID explicit values.
|
||||
explicitly specifies a value. If a user requests format=webp and WebP encoding
|
||||
is not supported, return an error - do NOT silently output PNG instead. If a
|
||||
user specifies fit=invalid and that fit mode doesn't exist, return an error -
|
||||
do NOT silently default to "cover". Silent fallbacks violate the principle of
|
||||
least surprise and mask bugs. The only acceptable defaults are for OMITTED
|
||||
parameters, never for INVALID explicit values.
|
||||
|
||||
- Avoid vendoring dependencies unless specifically instructed to. NEVER commit
|
||||
the `vendor/` directory. NEVER commit compiled binaries. If such a directory
|
||||
or file exists and `.gitignore` does not list it, add it to `.gitignore`.
|
||||
Commit that `.gitignore` change. Keep the entire git repository, history
|
||||
included, under 20 MiB. Go over 20 MiB only when you specifically must commit
|
||||
a large file, such as a media file used as a test fixture. Only OUR source
|
||||
code and its immediately supporting files, such as test examples, go into the
|
||||
repository and its history.
|
||||
- Avoid vendoring deps unless specifically instructed to. NEVER commit the
|
||||
vendor directory, NEVER commit compiled binaries. If these directories or
|
||||
files exist, add them to .gitignore (and commit the .gitignore) if they are
|
||||
not already in there. Keep the entire git repository (with history) small -
|
||||
under 20MiB, unless you specifically must commit larger files (e.g. test
|
||||
fixture example media files). Only OUR source code and immediately supporting
|
||||
files (such as test examples) goes into the repo/history.
|
||||
|
||||
@@ -133,7 +133,7 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
||||
filesystem's free space and what the cache holds, and at least 500 MiB, so
|
||||
the cache can grow to three quarters or more of the space they all have
|
||||
left, and to all of it when that is 500 MiB or less.
|
||||
- the rest are in the list under Configuration below
|
||||
- the rest are in the table under Configuration below
|
||||
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
|
||||
the images it processes at once, plus `upstream_connections` times
|
||||
`upstream_max_response_size` for the source images it has fetched that wait
|
||||
@@ -178,100 +178,46 @@ allowlisted source hosts for open access.
|
||||
|
||||
## Design
|
||||
|
||||
### Concepts
|
||||
|
||||
These terms are the names the code uses:
|
||||
|
||||
- **Source**: the image at one upstream URL, named by its host, path and query.
|
||||
- **Source key**: the SHA-256 of a source's host, path and query, which names
|
||||
the source's file and its row in the database.
|
||||
- **Transform**: the output size, format, quality and fit an image request asks
|
||||
for; with the format `auto`, pixa chooses the output format from the `Accept`
|
||||
header.
|
||||
- **Image request**: a source and a transform, which each image route reads from
|
||||
its URL.
|
||||
- **Variant**: the image pixa serves for an image request, made from its source
|
||||
with its transform.
|
||||
- **Variant key**: the SHA-256 of a variant's source, size, format, quality and
|
||||
fit, which names the variant's file and its row in the database, and gives its
|
||||
`ETag`.
|
||||
- **Signed URL**: a `/v1/image/` URL whose `sig` signs its image request and its
|
||||
expiry `exp` with `signing_key`.
|
||||
- **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any
|
||||
expiry, encrypted with a key derived from `signing_key`.
|
||||
- **Login session**: an encrypted cookie that keeps a browser logged in to the
|
||||
generator page for 30 days, or until a logout replaces the random value pixa
|
||||
keeps in the `logout` table (see Encrypted URLs).
|
||||
- **Negative cache**: the source URLs pixa does not fetch again for 5 minutes,
|
||||
after one of these failed fetches:
|
||||
- the upstream answered with an error status
|
||||
- the body's first bytes were not those of the format its `Content-Type`
|
||||
names
|
||||
- **Image cache**: the part of pixa that keeps the sources, the variants and the
|
||||
negative cache (see Storage).
|
||||
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
|
||||
sources and variants may reach together.
|
||||
- **Image proxy**: the part of pixa that serves a variant from the image cache
|
||||
when it can, and otherwise makes it from the source.
|
||||
|
||||
### Storage
|
||||
|
||||
- **Sources**: `<state_dir>/cache/sources/<ab>/<cd>/<source key>`, one file for
|
||||
each source URL
|
||||
- **Source images**:
|
||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one
|
||||
file for each source URL
|
||||
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
||||
- **Variants**: `<state_dir>/cache/variants/<ab>/<cd>/<variant key>`
|
||||
- **Transformed images**:
|
||||
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`
|
||||
|
||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||
name.
|
||||
|
||||
The image cache keeps these tables in the database:
|
||||
Each source image and each transformed image has one row in the database,
|
||||
holding its size and when it was last used. Two source URLs that serve the same
|
||||
bytes are stored twice, once under each URL. A source image is stored by writing
|
||||
its file and then its row; when the row cannot be written, the file is removed
|
||||
again.
|
||||
|
||||
- `source_content`: a row for each source, by source key, with its size and when
|
||||
it was last used
|
||||
- `variant_content`: a row for each variant, by variant key (column
|
||||
`cache_key`), with its size and when it was last used
|
||||
- `cache_usage`: the total size of the cached sources and variants, which
|
||||
eviction reads
|
||||
- `negative_cache`: a row for each source URL in the negative cache, with when
|
||||
it expires
|
||||
|
||||
pixa reads a cached source image's format from its file's first bytes. When they
|
||||
match none of the source formats, pixa fetches the source again. Two source URLs
|
||||
that serve the same bytes are stored twice, once under each URL. A source image
|
||||
is stored by writing its file and then its row; when the row cannot be written,
|
||||
the file is removed again.
|
||||
|
||||
Each source and variant file is written under a temporary name and synced to
|
||||
disk before it is renamed into place. So a crash never leaves part of a file
|
||||
under its own name. When pixa starts, and every five minutes after, it removes
|
||||
each source or variant whose file is empty, or not the size its row records. It
|
||||
logs a warning naming the file. A later request for that image fetches the
|
||||
source or makes the variant again. Looking for those files reads the database
|
||||
once for every cached file, so pixa skips each five-minute look that would start
|
||||
less than ten times as long after the last look ended as that look took. After a
|
||||
look that took one minute, the next one starts 10 to 15 minutes after that look
|
||||
ended.
|
||||
|
||||
Each look also drops the rows of variants whose files are missing. When pixa
|
||||
cannot read a variant file's size for another reason, it keeps the variant's
|
||||
row. It logs a warning naming the variant key and the error.
|
||||
|
||||
During a look, pixa skips a directory it cannot list and a file whose size it
|
||||
cannot read. It logs a warning naming the path and the error, and goes on with
|
||||
the rest of the look. A file renamed or removed after pixa listed its directory
|
||||
gets no warning.
|
||||
Source and transformed image files are written under a temporary name, synced to
|
||||
disk, and only then renamed into place, so a crash leaves no file under its own
|
||||
name with part of its data. When pixa starts, and every five minutes after, it
|
||||
removes each cached source or transformed image whose file is empty, or not the
|
||||
size the database records for it, with a warning naming the file; the next
|
||||
request for it fetches the source or converts the image again. Looking for those
|
||||
files reads the database once for every cached file, so pixa skips each
|
||||
five-minute look that would start less than ten times as long after the last
|
||||
look ended as that look took. After a look that took one minute, the next one
|
||||
starts 10 to 15 minutes after that look ended.
|
||||
|
||||
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||
Test). Toward it, a cache hit opens the variant's file and nothing else, and
|
||||
makes no database call. The content type comes from the file's first bytes,
|
||||
which are those of the format pixa wrote it in; a file whose first bytes are not
|
||||
those of a format pixa writes is not served, and the image is converted again. A
|
||||
hit hands the time it was served to a background writer and does not wait for
|
||||
the writer. Eviction orders by that time. The hit is counted in memory. A miss
|
||||
that uses a cached source image also hands that writer the time it used the
|
||||
image. Once a second that writer writes the waiting times in one transaction. Up
|
||||
to 10,000 access times of variants, and as many of sources, can wait to be
|
||||
written. A time handed over beyond that is dropped. Its image then only looks
|
||||
Test). Toward it, a cache hit opens the transformed image's file and nothing
|
||||
else, and makes no database call. The content type comes from the file's first
|
||||
bytes, which are those of the format pixa wrote it in; a file whose first bytes
|
||||
are not those of a format pixa writes is not served, and the image is converted
|
||||
again. A hit hands the time it was served to a background writer and does not
|
||||
wait for the writer. Eviction orders by that time. The hit is counted in memory.
|
||||
A miss that uses a cached source image also hands that writer the time it used
|
||||
the image. Once a second that writer writes the waiting times in one
|
||||
transaction. While 10,000 times of transformed images, or 10,000 of source
|
||||
images, wait to be written, a further one is dropped, and that image only looks
|
||||
less recently used to eviction. `script/bench` measures how many hits pixa
|
||||
serves per second (see Load Test).
|
||||
|
||||
@@ -319,8 +265,6 @@ path under `/v1/` answers 200, in maintenance mode too.
|
||||
- 403 when the upstream host, or a host it redirects to, has an address in a
|
||||
blocked network (see `blocked_networks`).
|
||||
- 429 over `image_requests_per_minute` (below).
|
||||
- 499, with no body, when the client goes away before its image is ready,
|
||||
unless its request is the one making the image (below).
|
||||
- 415 when the upstream answers with a `Content-Type` that is not one of the
|
||||
source formats (below).
|
||||
- 415 when the upstream sends a body whose first bytes are not those of the
|
||||
@@ -352,7 +296,6 @@ path under `/v1/` answers 200, in maintenance mode too.
|
||||
- 403
|
||||
- 415
|
||||
- 429
|
||||
- 499, with no body
|
||||
- 502
|
||||
- 503
|
||||
- 504
|
||||
@@ -734,85 +677,54 @@ refused with 400.
|
||||
Every setting can be given as an environment variable, in a YAML config file
|
||||
(`--config`), or both. A variable present in the environment wins over the file,
|
||||
even when it is empty, and the file wins over the built-in default. The one
|
||||
exception is a variable set in the file's `env:` section. pixa sets that
|
||||
variable while loading the file, so the variable wins over the environment pixa
|
||||
started with. It also wins over the file's key for the same setting. A
|
||||
variable's value is parsed as the same text in the file would be. A setting that
|
||||
holds a list takes comma-separated entries, with the spaces around each trimmed.
|
||||
An empty variable is an empty list. A value that does not parse or is invalid
|
||||
aborts startup, naming the variable. pixa refuses to start when a variable
|
||||
starts with `PIXA_` but is not in the list below, such as a misspelled one or
|
||||
`PIXA_PORT`. The error names the variable, as it names an unknown config key.
|
||||
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
|
||||
`--config`. The variables set by the file's `env:` section are checked the same
|
||||
exception is a variable named in the file's `env:` section: it is set while the
|
||||
file loads, so it overrides both the environment the process was started with
|
||||
and the file's own key. A variable's value is parsed as the same text in the
|
||||
file would be. The three lists take comma-separated entries, with the spaces
|
||||
around each trimmed; an empty variable is an empty list. A value that does not
|
||||
parse or is invalid aborts startup, naming the variable. A variable whose name
|
||||
starts with `PIXA_` but is not in the table below, such as a misspelled one or
|
||||
`PIXA_PORT`, aborts startup naming it, as an unknown config key does. The one
|
||||
other accepted name is `PIXA_CONFIG_PATH`, the config file's path (like
|
||||
`--config`). The variables set by the file's `env:` section are checked the same
|
||||
way.
|
||||
|
||||
pixa reads at most one config file. It reads the file given with `--config` (or
|
||||
`-c`), or else the one `PIXA_CONFIG_PATH` names. pixa refuses to start when that
|
||||
file does not exist, cannot be read or does not parse. Without `--config` or
|
||||
`PIXA_CONFIG_PATH`, pixa looks in `/etc/pixa/`, then in `~/.config/pixa/`, then
|
||||
in the working directory. In each directory it looks for `config.yml`, then
|
||||
`config.yaml`. It reads the first file it finds. It skips a missing file without
|
||||
a message. pixa refuses to start when a file it finds cannot be read or does not
|
||||
parse, and the error names the file. It also refuses to start when it may not
|
||||
enter one of these directories, whether or not the file is there, since it
|
||||
cannot tell. With no file, pixa uses the environment and the defaults.
|
||||
pixa reads at most one config file: the one given with `--config` (or `-c`),
|
||||
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that
|
||||
pixa finds: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`,
|
||||
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml` and
|
||||
`config.yaml` in the working directory. A named file that does not exist, cannot
|
||||
be read or does not parse aborts startup. Of the files pixa looks for on its
|
||||
own, only one that does not exist is passed over, without a message. One that
|
||||
pixa cannot read or parse aborts startup, naming the file. So does one in a
|
||||
directory pixa may not enter, whether or not it is there, since pixa cannot
|
||||
tell. With no file, pixa uses the environment and the defaults.
|
||||
|
||||
Each item below is one setting, named by its config key:
|
||||
|
||||
- `signing_key`: variable `PIXA_SIGNING_KEY`, required. The secret for signed
|
||||
URLs, encrypted URLs and login. It must be 32 characters or longer.
|
||||
- `port`: variable `PORT`, default `8080`. The port to listen on.
|
||||
- `state_dir`: variable `PIXA_STATE_DIR`, default `/var/lib/pixa`. The directory
|
||||
for the database and the disk cache.
|
||||
- `db_url`: variable `PIXA_DB_URL`, default `state.sqlite3` in the state
|
||||
directory. The URL of the SQLite database.
|
||||
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default 75% of the sum of
|
||||
the free space and what the cache holds. The disk cache limit in bytes. `0`
|
||||
turns the disk cache off.
|
||||
- `allowlist_hosts`: variable `PIXA_ALLOWLIST_HOSTS`, default none. The upstream
|
||||
hosts whose images pixa serves without a signature.
|
||||
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. The
|
||||
image routes refuse a request with 403 when its `Referer` header names one of
|
||||
these hosts.
|
||||
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. The CIDR
|
||||
ranges pixa never fetches from, on top of the built-in ones.
|
||||
- `trusted_proxies`: variable `PIXA_TRUSTED_PROXIES`, default the RFC 1918
|
||||
ranges. The CIDR ranges of the proxies whose `X-Forwarded-For` header pixa
|
||||
believes.
|
||||
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. Allows plain-HTTP
|
||||
upstream hosts. It is for testing only.
|
||||
- `upstream_connections_per_host`: variable
|
||||
`PIXA_UPSTREAM_CONNECTIONS_PER_HOST`, default `20`. The most connections to
|
||||
one upstream host at once.
|
||||
- `upstream_connections`: variable `PIXA_UPSTREAM_CONNECTIONS`, default `64`.
|
||||
The most connections to all upstream hosts together at once.
|
||||
- `max_concurrent_processing`: variable `PIXA_MAX_CONCURRENT_PROCESSING`,
|
||||
default half the CPUs, at least 1. The most images processed at once.
|
||||
- `image_requests_per_minute`: variable `PIXA_IMAGE_REQUESTS_PER_MINUTE`,
|
||||
default `600`. How many image requests one client may send per minute. `0`
|
||||
turns the limit off.
|
||||
- `upstream_fetch_timeout`: variable `PIXA_UPSTREAM_FETCH_TIMEOUT`, default
|
||||
`30s`. The time allowed for one fetch from an upstream host.
|
||||
- `upstream_max_response_size`: variable `PIXA_UPSTREAM_MAX_RESPONSE_SIZE`,
|
||||
default 50 MiB. The largest upstream response accepted, in bytes.
|
||||
- `downstream_timeout`: variable `PIXA_DOWNSTREAM_TIMEOUT`, default `60s`. The
|
||||
time allowed for answering one client request.
|
||||
- `access_control_allow_origin`: variable `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN`,
|
||||
default `*`. The CORS origin allowed to read image responses. It is `*` or one
|
||||
origin.
|
||||
- `metrics.username`: variable `PIXA_METRICS_USERNAME`, default empty. The
|
||||
username for `/metrics`. pixa serves `/metrics` only when the username and the
|
||||
password are both set.
|
||||
- `metrics.password`: variable `PIXA_METRICS_PASSWORD`, default empty. The
|
||||
password for `/metrics`. Set it together with the username.
|
||||
- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. The Sentry DSN for
|
||||
error reporting. An empty DSN turns error reporting off.
|
||||
- `debug`: variable `PIXA_DEBUG`, default `false`. Turns on debug logging and
|
||||
plain-HTTP local development.
|
||||
- `maintenance_mode`: variable `PIXA_MAINTENANCE_MODE`, default `false`. While
|
||||
`true`, pixa answers image requests with 503. The health check still
|
||||
answers 200.
|
||||
| Variable | Config key | Meaning |
|
||||
| ------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| `PIXA_SIGNING_KEY` | `signing_key` | Required: secret for signed and encrypted URLs and login, 32+ characters |
|
||||
| `PORT` | `port` | Port to listen on; default `8080` |
|
||||
| `PIXA_STATE_DIR` | `state_dir` | Directory for the database and the disk cache; default `/var/lib/pixa` |
|
||||
| `PIXA_DB_URL` | `db_url` | SQLite database URL; default `state.sqlite3` in the state directory |
|
||||
| `PIXA_CACHE_MAX_BYTES` | `cache_max_bytes` | Disk cache limit in bytes; `0` disables it; default 75% of (free + cached) |
|
||||
| `PIXA_ALLOWLIST_HOSTS` | `allowlist_hosts` | Upstream hosts served without a signature |
|
||||
| `PIXA_REFERER_BLOCKLIST` | `referer_blocklist` | Hosts whose pages the image routes refuse with 403, by `Referer` |
|
||||
| `PIXA_BLOCKED_NETWORKS` | `blocked_networks` | CIDR ranges never fetched from, on top of the built-in ones |
|
||||
| `PIXA_TRUSTED_PROXIES` | `trusted_proxies` | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 |
|
||||
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
||||
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
|
||||
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
|
||||
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 |
|
||||
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
|
||||
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
|
||||
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
|
||||
| `PIXA_DOWNSTREAM_TIMEOUT` | `downstream_timeout` | Time allowed for answering one client request; default `60s` |
|
||||
| `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN` | `access_control_allow_origin` | CORS origin allowed to read image responses: `*` or one origin; default `*` |
|
||||
| `PIXA_METRICS_USERNAME` | `metrics.username` | Username for `/metrics`, which is served only when both are set |
|
||||
| `PIXA_METRICS_PASSWORD` | `metrics.password` | Password for `/metrics`; set together with the username |
|
||||
| `PIXA_SENTRY_DSN` | `sentry_dsn` | Sentry DSN for error reporting; empty disables it |
|
||||
| `PIXA_DEBUG` | `debug` | Debug logging and plain-HTTP local development; default `false` |
|
||||
| `PIXA_MAINTENANCE_MODE` | `maintenance_mode` | Answer image requests with 503; the health check stays 200; default `false` |
|
||||
|
||||
Key settings in more detail:
|
||||
|
||||
@@ -936,25 +848,9 @@ See `configs/config.example.yml` for all options with defaults.
|
||||
|
||||
### Architecture
|
||||
|
||||
- **Dependency injection**: Uber fx. `internal/app` puts every part of pixad in
|
||||
one fx graph. `internal/providers` builds the image cache, the image proxy and
|
||||
the other parts that serve images and logins. Each setting's default is
|
||||
declared once, in `internal/config`.
|
||||
- **Packages**: these packages under `internal/` work with the concepts above:
|
||||
- `handlers`: the routes, which read the image request from the URL, check
|
||||
the signed or encrypted URL, and make the login session
|
||||
- `imagerequest`: the image request, with the source key and the variant key
|
||||
- `imageproxy`: the image proxy
|
||||
- `imgcache`: the image cache, which keeps its sources in one `FileStore`
|
||||
and its variants in another
|
||||
- `httpfetcher`: fetching sources from upstream hosts
|
||||
- `magic`: reading an image's format from its first bytes
|
||||
- `imageprocessor`: making a variant from a source with libvips
|
||||
- `signature`: checking the `sig` of a signed URL
|
||||
- `encurl`: making and reading the tokens of encrypted URLs
|
||||
- `seal`: the keys derived from `signing_key`, and the encryption of those
|
||||
tokens
|
||||
- `session`: the login session
|
||||
- **Dependency injection**: Uber fx. `internal/app` builds every part of pixad
|
||||
from the config in one fx graph. Each setting's default is declared once, in
|
||||
`internal/config`.
|
||||
- **HTTP router**: go-chi
|
||||
- **Image processing**: govips (CGO wrapper for libvips)
|
||||
- **Database**: SQLite via modernc.org/sqlite
|
||||
|
||||
@@ -30,107 +30,6 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-27 the reconciliation pass skips a directory it cannot list and a file
|
||||
whose info it cannot read, and goes on (closes #343). It stopped before its
|
||||
later checks at a directory it could not list, or at a source or variant file
|
||||
whose info it could not read.
|
||||
- It logs the warning "skipped unreadable entry in cache directory", with
|
||||
the path and the error.
|
||||
- A file renamed or removed after its directory was listed gets no warning.
|
||||
- `reconcileFiles` reads each file's info once, before the temp-file check,
|
||||
and hands it to `sweepStaleTempFile`, `reconcileVariantFile` or
|
||||
`reconcileSourceFile`.
|
||||
- A cancelled context still stops the walk with its error.
|
||||
- 2026-10-26 the reconciliation pass drops a variant's accounting row only when
|
||||
its file is missing (closes #340). Any other error reading the file's size
|
||||
keeps the row. It dropped the row on any error and logged the file as missing.
|
||||
- The kept row gets the warning "kept accounting row for unreadable variant
|
||||
file", with the variant key and the error.
|
||||
- The pass checks for `ErrNotFound`, which `FileStore.Size` returns for a
|
||||
missing file.
|
||||
- 2026-10-26 on both image routes, a request is answered 499 and logged at info
|
||||
level when its client goes away before its image is ready, unless it is the
|
||||
request making the image (closes #339). It was answered 500 and logged at
|
||||
error level, as pixa's own failure.
|
||||
- The line is "client went away before the image was ready". It carries the
|
||||
`request_id`, the host and the path.
|
||||
- The 499 has no body, since nobody reads it.
|
||||
- `serveImage` answers it before calling `respondImageError`, which cannot
|
||||
see the request's context.
|
||||
- An error wrapping `context.Canceled` while the request's context is live
|
||||
is still answered 500 and logged at error level.
|
||||
- 2026-10-26 `README.md`'s Design section starts with a Concepts list (closes
|
||||
#321). Each concept gets one sentence, under the name the code gives it.
|
||||
- Storage calls the transformed images variants, in shorter sentences.
|
||||
- Storage's paths end in the source key and the variant key.
|
||||
- Storage lists the image cache's tables in the database.
|
||||
- Architecture names `internal/providers`, and lists the packages that work
|
||||
with the concepts.
|
||||
- 2026-10-26 one table in `internal/config` lists every setting once (closes
|
||||
#320). `Config.settings` gives each setting's key, its environment variable
|
||||
and the function that loads it.
|
||||
- Most rows load through `field`, whose arguments give the setting's type,
|
||||
its default and its check.
|
||||
- `db_url` and `cache_max_bytes` have their own load functions. The default
|
||||
of `db_url` comes from `state_dir`, and `cache_max_bytes` records whether
|
||||
it is set.
|
||||
- Loading, the unknown-key check, the unknown-variable check and the names
|
||||
in error messages all read the table.
|
||||
- `allowlist_hosts` loads as `referer_blocklist` does, through
|
||||
`parseHostList`.
|
||||
- The metrics username and password are checked together once the table has
|
||||
loaded.
|
||||
- When a config holds more than one invalid value, the error names the one
|
||||
that comes first in the table.
|
||||
- `README.md` lists the settings one item each, where it had a table. A test
|
||||
checks that this list and `configs/config.example.yml` name exactly the
|
||||
keys of the table.
|
||||
- The key constants moved to the tests, their only users.
|
||||
- Removed:
|
||||
- `isKnownConfigKey`, `envVarNames`, `strictLoader` and `validate`
|
||||
- `validateAllowlistHostsValue` and `getStringSlice`, with the three
|
||||
tests of `getStringSlice`
|
||||
- 2026-10-26 a fetched image's format is detected from its bytes once, as an
|
||||
`imagerequest.Format` (closes #319). The upstream `Content-Type` check in the
|
||||
fetcher stays as the filter before downloading.
|
||||
- `magic.DetectFormat` returns an `imagerequest.Format`, and
|
||||
`magic.ValidateMagicBytes` returns the format it checked against the
|
||||
`Content-Type`.
|
||||
- `Format.MIMEType` gives each format's MIME type. The fetcher, the mock
|
||||
fetcher, the cache's read of a variant's type and the handler's `Accept`
|
||||
matching use it.
|
||||
- `Cache.Source` returns a source's format, read from its file's first bytes
|
||||
as `Cache.Variant` reads a variant's. A file in no source format is an
|
||||
error, and the proxy fetches the source again.
|
||||
- The proxy hands the format to `ImageProcessor.Process`. The processor
|
||||
refuses input that libvips would load as another format, as before, with
|
||||
`ErrUnsupportedInputFormat`. It refuses a format that is not one of the
|
||||
six source formats the same way.
|
||||
- `internal/magic` no longer detects SVG, which pixa does not read. SVG data
|
||||
is `ErrUnknownFormat`.
|
||||
- Removed:
|
||||
- `magic.MIMEType`, `magic.ImageFormat` and `magic.MIMEToImageFormat`
|
||||
- `detectFormat`, `formatFromString`, `FormatToMIME` and
|
||||
`Result.InputFormat` of the processor
|
||||
- the fetcher's MIME constants of the six formats
|
||||
- `errNotAnOutputFormat` of the cache
|
||||
- 2026-10-26 the cache keeps its source images in one `imgcache.FileStore` and
|
||||
its variants in another (closes #318). A `FileStore` is a directory of files,
|
||||
each named by a hex key, at `<ab>/<cd>/<key>`.
|
||||
- `ContentStorage` and `VariantStorage` are gone.
|
||||
- `FileStore` has these methods:
|
||||
- `Write`, which writes a file under a temp name, syncs it and renames
|
||||
it into place
|
||||
- `Open`, which returns the file and its size
|
||||
- `Size`
|
||||
- `Delete`
|
||||
- `FileStore` has no `Exists`. The reconciliation pass checks that a
|
||||
variant's file is there with `Size`. The tests look for the file on disk.
|
||||
- `Cache.Variant` reads the content type from the variant file's first
|
||||
bytes, as `VariantStorage.LoadWithContentType` did. A cache hit still
|
||||
makes no database call.
|
||||
- `storage.go` is now `filestore.go`. `internal/imgcache/testdata/red.avif`,
|
||||
which no test used, is gone.
|
||||
- 2026-10-26 `internal/imgcache` holds the cache only (closes #317). The new
|
||||
package `internal/imageproxy` fetches, transforms, caches and serves images.
|
||||
It imports `imgcache`, and `imgcache` does not import it.
|
||||
|
||||
+586
-345
File diff suppressed because it is too large
Load Diff
@@ -1,12 +1,97 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.eeqj.de/sneak/smartconfig"
|
||||
)
|
||||
|
||||
// writeTestConfig writes yamlContent to a temp config file and returns
|
||||
// the file path.
|
||||
func writeTestConfig(t *testing.T, yamlContent string) string {
|
||||
t.Helper()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
configPath := filepath.Join(tmpDir, "config.yml")
|
||||
|
||||
err := os.WriteFile(configPath, []byte(yamlContent), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to write test config: %v", err)
|
||||
}
|
||||
|
||||
return configPath
|
||||
}
|
||||
|
||||
// checkAllowlistHosts loads the config at configPath and asserts that
|
||||
// getStringSlice returns the three expected hosts.
|
||||
func checkAllowlistHosts(t *testing.T, configPath string) {
|
||||
t.Helper()
|
||||
|
||||
sc, err := loadTestConfig(configPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to load config: %v", err)
|
||||
}
|
||||
|
||||
hosts := getStringSlice(sc)
|
||||
|
||||
if len(hosts) != 3 {
|
||||
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts)
|
||||
}
|
||||
|
||||
expected := []string{"static.sneak.cloud", "sneak.berlin", testHostS3}
|
||||
for i, want := range expected {
|
||||
if i >= len(hosts) {
|
||||
t.Errorf("missing host at index %d: want %q", i, want)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if hosts[i] != want {
|
||||
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetStringSlice_YAMLList(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
yamlContent := `
|
||||
allowlist_hosts:
|
||||
- static.sneak.cloud
|
||||
- sneak.berlin
|
||||
- s3.sneak.cloud
|
||||
`
|
||||
|
||||
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
|
||||
}
|
||||
|
||||
func TestGetStringSlice_CommaSeparated(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Backwards compatibility with comma-separated string values.
|
||||
yamlContent := `allowlist_hosts: "static.sneak.cloud, sneak.berlin, s3.sneak.cloud"`
|
||||
|
||||
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
|
||||
}
|
||||
|
||||
func TestGetStringSlice_Empty(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
configPath := writeTestConfig(t, `port: 8080`)
|
||||
|
||||
sc, err := loadTestConfig(configPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to load config: %v", err)
|
||||
}
|
||||
|
||||
hosts := getStringSlice(sc)
|
||||
if len(hosts) != 0 {
|
||||
t.Errorf("expected nil or empty slice, got %v", hosts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestExampleConfigAllowlistsNoHost loads configs/config.example.yml
|
||||
// with a real signing key, as Getting Started in README.md has it copied
|
||||
// and edited, and checks that it allowlists no host: every image on an
|
||||
|
||||
@@ -15,36 +15,8 @@ import (
|
||||
_ "modernc.org/sqlite" // SQLite driver registration
|
||||
)
|
||||
|
||||
// The config keys the tests name in the errors they expect.
|
||||
const (
|
||||
keyDebug = "debug"
|
||||
keyMaintenanceMode = "maintenance_mode"
|
||||
keyPort = "port"
|
||||
keyStateDir = "state_dir"
|
||||
keySentryDSN = "sentry_dsn"
|
||||
keyDBURL = "db_url"
|
||||
keyMetrics = "metrics"
|
||||
keyMetricsUsername = "metrics.username"
|
||||
keyMetricsPassword = "metrics.password"
|
||||
keySigningKey = "signing_key"
|
||||
keyAllowlistHosts = "allowlist_hosts"
|
||||
keyRefererBlocklist = "referer_blocklist"
|
||||
keyAllowHTTP = "allow_http"
|
||||
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
|
||||
keyUpstreamConnections = "upstream_connections"
|
||||
keyMaxConcurrentProcessing = "max_concurrent_processing"
|
||||
keyCacheMaxBytes = "cache_max_bytes"
|
||||
keyBlockedNetworks = "blocked_networks"
|
||||
keyTrustedProxies = "trusted_proxies"
|
||||
keyAccessControlAllowOrigin = "access_control_allow_origin"
|
||||
keyUpstreamFetchTimeout = "upstream_fetch_timeout"
|
||||
keyUpstreamMaxResponseSize = "upstream_max_response_size"
|
||||
keyDownstreamTimeout = "downstream_timeout"
|
||||
keyImageRequestsPerMinute = "image_requests_per_minute"
|
||||
)
|
||||
|
||||
// validTestSigningKey is a 32-character signing key that satisfies the
|
||||
// minimum length requirement in validateSigningKey.
|
||||
// minimum length requirement in validate().
|
||||
const validTestSigningKey = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
// signingKeyLine is a valid signing_key config line used as the base of
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestReadmeNamesEverySetting checks that the list of settings in
|
||||
// README.md names exactly the keys of the settings table, one item each.
|
||||
// The list is the first one in the Configuration section, and it ends at
|
||||
// the first blank line. Each item starts with its key in backticks.
|
||||
func TestReadmeNamesEverySetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
readme, err := os.ReadFile(filepath.Join("..", "..", "README.md"))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read README.md: %v", err)
|
||||
}
|
||||
|
||||
_, section, found := strings.Cut(string(readme), "\n### Configuration\n")
|
||||
if !found {
|
||||
t.Fatal("README.md has no Configuration section")
|
||||
}
|
||||
|
||||
// The section ends at the next heading of its level or above.
|
||||
section, _, _ = strings.Cut(section, "\n## ")
|
||||
section, _, _ = strings.Cut(section, "\n### ")
|
||||
|
||||
_, list, found := strings.Cut(section, "\n- ")
|
||||
if !found {
|
||||
t.Fatal("README.md's Configuration section has no list")
|
||||
}
|
||||
|
||||
list, _, _ = strings.Cut("- "+list, "\n\n")
|
||||
|
||||
itemKey := regexp.MustCompile("^- `([^`]+)`")
|
||||
named := map[string]bool{}
|
||||
|
||||
for line := range strings.SplitSeq(list, "\n") {
|
||||
// A line that does not start with "- " goes on with the item above.
|
||||
if !strings.HasPrefix(line, "- ") {
|
||||
continue
|
||||
}
|
||||
|
||||
match := itemKey.FindStringSubmatch(line)
|
||||
if match == nil {
|
||||
t.Errorf("README.md's list of settings has an item "+
|
||||
"that does not start with a key in backticks: %s", line)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if named[match[1]] {
|
||||
t.Errorf("README.md's list of settings names %s twice", match[1])
|
||||
}
|
||||
|
||||
named[match[1]] = true
|
||||
}
|
||||
|
||||
for _, s := range new(Config).settings() {
|
||||
if !named[s.key] {
|
||||
t.Errorf("README.md's list of settings does not name %s", s.key)
|
||||
}
|
||||
|
||||
delete(named, s.key)
|
||||
}
|
||||
|
||||
for key := range named {
|
||||
t.Errorf("README.md's list of settings names %s, "+
|
||||
"which is not a setting", key)
|
||||
}
|
||||
}
|
||||
|
||||
// TestExampleConfigNamesEverySetting checks that
|
||||
// configs/config.example.yml names exactly the keys of the settings
|
||||
// table, each one set or commented out with "# ". A key line holds the
|
||||
// key, a colon and at most a one-word value, which tells it from a
|
||||
// comment line that starts with a word and a colon. An indented key is
|
||||
// inside the map of the key above it, as username is inside metrics.
|
||||
func TestExampleConfigNamesEverySetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const examplePath = "configs/config.example.yml"
|
||||
|
||||
example, err := os.ReadFile(filepath.Join("..", "..", examplePath))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read %s: %v", examplePath, err)
|
||||
}
|
||||
|
||||
keyLine := regexp.MustCompile(`(?m)^(?:# )?( *)([a-z_]+):(?: \S+)?$`)
|
||||
named := map[string]bool{}
|
||||
|
||||
var mapKey string
|
||||
|
||||
for _, match := range keyLine.FindAllStringSubmatch(string(example), -1) {
|
||||
indent, key := match[1], match[2]
|
||||
if indent == "" {
|
||||
mapKey = key
|
||||
named[key] = true
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
// The key holding the map is not itself a setting.
|
||||
delete(named, mapKey)
|
||||
|
||||
named[mapKey+"."+key] = true
|
||||
}
|
||||
|
||||
// The env section sets environment variables; it is not a setting.
|
||||
delete(named, "env")
|
||||
|
||||
for _, s := range new(Config).settings() {
|
||||
if !named[s.key] {
|
||||
t.Errorf("%s does not name %s", examplePath, s.key)
|
||||
}
|
||||
|
||||
delete(named, s.key)
|
||||
}
|
||||
|
||||
for key := range named {
|
||||
t.Errorf("%s names %s, which is not a setting", examplePath, key)
|
||||
}
|
||||
}
|
||||
@@ -138,8 +138,7 @@ func TestPixaPortAbortsStartupPointingToPort(t *testing.T) {
|
||||
|
||||
// TestSettingVariablesAndConfigPathAreAccepted checks that every
|
||||
// setting's variable and PIXA_CONFIG_PATH pass the check for unknown
|
||||
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the
|
||||
// settings table.
|
||||
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the list.
|
||||
func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
|
||||
// A config file's env section loaded by another test can leave a
|
||||
// PIXA_ variable set for the whole process, so every one is unset
|
||||
@@ -160,8 +159,8 @@ func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
|
||||
|
||||
t.Setenv("PIXA_CONFIG_PATH", "/etc/pixa/config.yml")
|
||||
|
||||
for _, s := range new(Config).settings() {
|
||||
t.Setenv(s.envVar, "")
|
||||
for _, name := range envVarNames() {
|
||||
t.Setenv(name, "")
|
||||
}
|
||||
|
||||
err := validateKnownEnvVars()
|
||||
|
||||
@@ -1,177 +0,0 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// heldFetcher sends on started when a fetch begins, and holds the fetch until
|
||||
// release is closed. It then fails the fetch as an upstream error.
|
||||
type heldFetcher struct {
|
||||
started chan struct{}
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
// Fetch sends on f.started, waits for f.release to be closed, and returns
|
||||
// httpfetcher.ErrUpstreamError.
|
||||
func (f heldFetcher) Fetch(
|
||||
context.Context, imagerequest.Source,
|
||||
) (*httpfetcher.FetchResult, error) {
|
||||
f.started <- struct{}{}
|
||||
|
||||
<-f.release
|
||||
|
||||
return nil, httpfetcher.ErrUpstreamError
|
||||
}
|
||||
|
||||
// readLogLines decodes the JSON log lines in logs.
|
||||
func readLogLines(t *testing.T, logs *bytes.Buffer) []map[string]any {
|
||||
t.Helper()
|
||||
|
||||
t.Logf("logged:\n%s", logs.String())
|
||||
|
||||
var lines []map[string]any
|
||||
|
||||
dec := json.NewDecoder(logs)
|
||||
for dec.More() {
|
||||
var line map[string]any
|
||||
|
||||
err := dec.Decode(&line)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding log line: %v", err)
|
||||
}
|
||||
|
||||
lines = append(lines, line)
|
||||
}
|
||||
|
||||
return lines
|
||||
}
|
||||
|
||||
// TestImageRoutes_ClientGoneWhileWaitingAnswers499 checks that both image
|
||||
// routes answer 499 to a request whose client goes away while it waits for an
|
||||
// image another request is making. A first request's fetch is held, so the
|
||||
// image is being made, and a second request for the same image waits for it
|
||||
// until its client goes away. The second request must be logged once at info
|
||||
// level with its request ID and the source host and path. Nothing may be
|
||||
// logged at error level.
|
||||
func TestImageRoutes_ClientGoneWhileWaitingAnswers499(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const requestID = "test-request-id"
|
||||
|
||||
for route, imageURL := range photoURLsByRoute() {
|
||||
t.Run(route, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var logs bytes.Buffer
|
||||
|
||||
fetcher := heldFetcher{started: make(chan struct{}), release: make(chan struct{})}
|
||||
h, routes := newRoutesFetchingThrough(t, fetcher,
|
||||
slog.New(slog.NewJSONHandler(&logs, nil)))
|
||||
target := imageURL(t, h)
|
||||
|
||||
firstDone := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
defer close(firstDone)
|
||||
|
||||
sendGet(t, routes, target)
|
||||
}()
|
||||
|
||||
<-fetcher.started
|
||||
|
||||
ctx, cancel := context.WithCancel(context.WithValue(t.Context(),
|
||||
middleware.RequestIDKey, requestID))
|
||||
rec := httptest.NewRecorder()
|
||||
secondDone := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
defer close(secondDone)
|
||||
|
||||
routes.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
ctx, http.MethodGet, target, nil))
|
||||
}()
|
||||
|
||||
// Give the second request time to start waiting for the first
|
||||
// request's image. The fetch is still held, so the second request
|
||||
// can end only because its client went away
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
cancel()
|
||||
<-secondDone
|
||||
|
||||
close(fetcher.release)
|
||||
<-firstDone
|
||||
|
||||
if rec.Code != 499 {
|
||||
t.Errorf("status = %d, want 499", rec.Code)
|
||||
}
|
||||
|
||||
var clientGoneLines int
|
||||
|
||||
for _, line := range readLogLines(t, &logs) {
|
||||
if line["level"] == slog.LevelError.String() {
|
||||
t.Errorf("logged at error level: %v", line["msg"])
|
||||
}
|
||||
|
||||
if line["level"] == slog.LevelInfo.String() &&
|
||||
line["msg"] == "client went away before the image was ready" &&
|
||||
line["request_id"] == requestID &&
|
||||
line["host"] == signedHost &&
|
||||
line["path"] == photoPath {
|
||||
clientGoneLines++
|
||||
}
|
||||
}
|
||||
|
||||
if clientGoneLines != 1 {
|
||||
t.Errorf("info line naming the request logged %d times, want 1",
|
||||
clientGoneLines)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageRoutes_CanceledWithLiveRequestAnswers500 checks that both image
|
||||
// routes answer 500, logged as "failed to get image" at error level, for an
|
||||
// image service error that wraps context.Canceled while the request's own
|
||||
// context is live. Its client has not gone away, so the cancellation is
|
||||
// pixa's own failure.
|
||||
func TestImageRoutes_CanceledWithLiveRequestAnswers500(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for route, imageURL := range photoURLsByRoute() {
|
||||
t.Run(route, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var logs bytes.Buffer
|
||||
|
||||
h, routes := newRoutesFetchingThrough(t,
|
||||
failingFetcher{err: context.Canceled},
|
||||
slog.New(slog.NewJSONHandler(&logs, nil)))
|
||||
|
||||
rec := sendGet(t, routes, imageURL(t, h))
|
||||
if rec.Code != http.StatusInternalServerError {
|
||||
t.Errorf("status = %d, want %d",
|
||||
rec.Code, http.StatusInternalServerError)
|
||||
}
|
||||
|
||||
if !slices.ContainsFunc(readLogLines(t, &logs),
|
||||
func(line map[string]any) bool {
|
||||
return line["level"] == slog.LevelError.String() &&
|
||||
line["msg"] == "failed to get image"
|
||||
}) {
|
||||
t.Error(`no "failed to get image" line logged at error level`)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -65,20 +65,20 @@ func formatForAccept(accept string) (imagerequest.Format, error) {
|
||||
return imagerequest.FormatJPEG, nil
|
||||
}
|
||||
|
||||
if qualities[imagerequest.FormatJXL.MIMEType()] > 0 {
|
||||
if qualities["image/jxl"] > 0 {
|
||||
return imagerequest.FormatJXL, nil
|
||||
}
|
||||
|
||||
if qualities[imagerequest.FormatAVIF.MIMEType()] > 0 {
|
||||
if qualities["image/avif"] > 0 {
|
||||
return imagerequest.FormatAVIF, nil
|
||||
}
|
||||
|
||||
if qualities[imagerequest.FormatWebP.MIMEType()] > 0 {
|
||||
if qualities["image/webp"] > 0 {
|
||||
return imagerequest.FormatWebP, nil
|
||||
}
|
||||
|
||||
// For JPEG, the most specific entry the header has decides
|
||||
quality, named := qualities[imagerequest.FormatJPEG.MIMEType()]
|
||||
quality, named := qualities["image/jpeg"]
|
||||
if !named {
|
||||
quality, named = qualities["image/*"]
|
||||
}
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -20,12 +19,6 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// statusClientClosedRequest is the status nginx calls "client closed
|
||||
// request". The image routes answer it when the client goes away before its
|
||||
// image is ready, unless its request is the one making the image. Nobody reads
|
||||
// it; it is for logs and metrics.
|
||||
const statusClientClosedRequest = 499
|
||||
|
||||
// HandleImage handles the main image proxy route:
|
||||
// /v1/image/<host>/<path>/<width>x<height>.<format>, or with no format
|
||||
// /v1/image/<host>/<path>/<width>x<height>
|
||||
@@ -352,8 +345,7 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
|
||||
// serveImage serves the image that req asks for. Both image routes call it
|
||||
// once they have the request and its URL's expiry, which ends the
|
||||
// Cache-Control max-age. It chooses the format for the format auto, answers
|
||||
// If-None-Match with 304 and HEAD with the headers only. It answers 499 when
|
||||
// the image service stopped because the client went away, and any other error
|
||||
// If-None-Match with 304 and HEAD with the headers only, and answers an error
|
||||
// of the image service with respondImageError.
|
||||
func (s *Handlers) serveImage(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
@@ -370,21 +362,6 @@ func (s *Handlers) serveImage(
|
||||
startTime := time.Now()
|
||||
|
||||
resp, err := s.imgSvc.Get(r.Context(), req)
|
||||
|
||||
// The client went away before its image was ready, which is not pixa's
|
||||
// failure. A cancellation while the request's context is live is pixa's,
|
||||
// and respondImageError answers it 500.
|
||||
if errors.Is(err, context.Canceled) && r.Context().Err() != nil {
|
||||
s.log.Info("client went away before the image was ready",
|
||||
"request_id", middleware.GetReqID(r.Context()),
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
)
|
||||
w.WriteHeader(statusClientClosedRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
s.respondImageError(w, req, err)
|
||||
|
||||
|
||||
@@ -16,8 +16,8 @@ import (
|
||||
// TestUnsupportedSourceFormatAnswers415 checks that respondImageError, which
|
||||
// answers the errors of both image routes, answers 415 with a clear error when
|
||||
// the upstream answers with a content type pixa does not accept, such as SVG,
|
||||
// and when libvips does not read a source as the format its first bytes
|
||||
// show, wrapped as the service wraps them.
|
||||
// and when orig is asked of a source in a format the processor cannot name,
|
||||
// wrapped as the service wraps them.
|
||||
func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -29,7 +29,7 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
|
||||
for _, err := range []error{
|
||||
fmt.Errorf("upstream fetch failed: %w: image/svg+xml",
|
||||
httpfetcher.ErrInvalidContentType),
|
||||
fmt.Errorf("image processing failed: %w: libvips does not read the input as avif",
|
||||
fmt.Errorf("image processing failed: %w: unknown",
|
||||
imageprocessor.ErrUnsupportedInputFormat),
|
||||
} {
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
@@ -83,7 +83,7 @@ func startUpstream(t *testing.T) *httptest.Server {
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/image", func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
_, _ = io.WriteString(w, imagePayload)
|
||||
})
|
||||
mux.HandleFunc("/status/500", func(w http.ResponseWriter, _ *http.Request) {
|
||||
@@ -358,7 +358,7 @@ func TestFetchContentLengthBodyAtMaxResponseSize(t *testing.T) {
|
||||
|
||||
res := fetchWithMaxResponseSize(t, maxResponseSize,
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
w.Header().Set("Content-Length", strconv.Itoa(tc.size))
|
||||
_, _ = io.WriteString(w, body)
|
||||
})
|
||||
@@ -406,7 +406,7 @@ func TestFetchChunkedBodyAtMaxResponseSize(t *testing.T) {
|
||||
|
||||
res := fetchWithMaxResponseSize(t, maxResponseSize,
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
_, _ = io.WriteString(w, body)
|
||||
// With no Content-Length, flushing sends the body
|
||||
// chunked; its end is sent when the handler returns.
|
||||
|
||||
@@ -38,6 +38,18 @@ const userAgent = "pixa/1.0"
|
||||
// MaxConnections fetches are already in progress.
|
||||
const ConnectionWaitTimeout = 10 * time.Second
|
||||
|
||||
// MIME content types.
|
||||
const (
|
||||
contentTypeJPEG = "image/jpeg"
|
||||
contentTypePNG = "image/png"
|
||||
contentTypeGIF = "image/gif"
|
||||
contentTypeWebP = "image/webp"
|
||||
contentTypeAVIF = "image/avif"
|
||||
contentTypeJXL = "image/jxl"
|
||||
contentTypeSVG = "image/svg+xml"
|
||||
contentTypeOctetStream = "application/octet-stream"
|
||||
)
|
||||
|
||||
// Loopback addresses blocked by SSRF protection.
|
||||
const (
|
||||
localhostIPv4 = "127.0.0.1"
|
||||
@@ -109,16 +121,15 @@ type FetchResult struct {
|
||||
}
|
||||
|
||||
// allowedContentTypes returns the content types an upstream response may
|
||||
// have, which Fetch also sends as its Accept header: the MIME types of the
|
||||
// formats pixa reads.
|
||||
// have, which Fetch also sends as its Accept header.
|
||||
func allowedContentTypes() []string {
|
||||
return []string{
|
||||
imagerequest.FormatJPEG.MIMEType(),
|
||||
imagerequest.FormatPNG.MIMEType(),
|
||||
imagerequest.FormatGIF.MIMEType(),
|
||||
imagerequest.FormatWebP.MIMEType(),
|
||||
imagerequest.FormatAVIF.MIMEType(),
|
||||
imagerequest.FormatJXL.MIMEType(),
|
||||
contentTypeJPEG,
|
||||
contentTypePNG,
|
||||
contentTypeGIF,
|
||||
contentTypeWebP,
|
||||
contentTypeAVIF,
|
||||
contentTypeJXL,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -30,9 +30,9 @@ func TestIsAllowedContentType(t *testing.T) {
|
||||
contentType string
|
||||
want bool
|
||||
}{
|
||||
{imagerequest.FormatJPEG.MIMEType(), true},
|
||||
{imagerequest.FormatPNG.MIMEType(), true},
|
||||
{imagerequest.FormatWebP.MIMEType(), true},
|
||||
{contentTypeJPEG, true},
|
||||
{contentTypePNG, true},
|
||||
{contentTypeWebP, true},
|
||||
{"image/jpeg; charset=utf-8", true},
|
||||
{"IMAGE/JPEG", true},
|
||||
{"text/html", false},
|
||||
@@ -217,7 +217,7 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
if result.ContentType != imagerequest.FormatJPEG.MIMEType() {
|
||||
if result.ContentType != contentTypeJPEG {
|
||||
t.Errorf("ContentType = %q, want image/jpeg", result.ContentType)
|
||||
}
|
||||
|
||||
@@ -272,13 +272,13 @@ func TestDetectContentTypeFromPath(t *testing.T) {
|
||||
path string
|
||||
want string
|
||||
}{
|
||||
{"foo/bar.jpg", imagerequest.FormatJPEG.MIMEType()},
|
||||
{"foo/bar.JPG", imagerequest.FormatJPEG.MIMEType()},
|
||||
{"foo/bar.jpeg", imagerequest.FormatJPEG.MIMEType()},
|
||||
{"foo/bar.png", imagerequest.FormatPNG.MIMEType()},
|
||||
{"foo/bar.gif", imagerequest.FormatGIF.MIMEType()},
|
||||
{"foo/bar.webp", imagerequest.FormatWebP.MIMEType()},
|
||||
{"foo/bar.avif", imagerequest.FormatAVIF.MIMEType()},
|
||||
{"foo/bar.jpg", contentTypeJPEG},
|
||||
{"foo/bar.JPG", contentTypeJPEG},
|
||||
{"foo/bar.jpeg", contentTypeJPEG},
|
||||
{"foo/bar.png", contentTypePNG},
|
||||
{"foo/bar.gif", contentTypeGIF},
|
||||
{"foo/bar.webp", contentTypeWebP},
|
||||
{"foo/bar.avif", contentTypeAVIF},
|
||||
{"foo/bar.svg", contentTypeSVG},
|
||||
{"foo/bar.bin", contentTypeOctetStream},
|
||||
{"foo/bar", contentTypeOctetStream},
|
||||
|
||||
@@ -66,29 +66,23 @@ func (m *MockFetcher) Fetch(
|
||||
}, nil
|
||||
}
|
||||
|
||||
// The content types the mock fetcher gives files in no format pixa reads.
|
||||
const (
|
||||
contentTypeSVG = "image/svg+xml"
|
||||
contentTypeOctetStream = "application/octet-stream"
|
||||
)
|
||||
|
||||
// detectContentTypeFromPath returns the MIME type based on file extension.
|
||||
func detectContentTypeFromPath(path string) string {
|
||||
path = strings.ToLower(path)
|
||||
|
||||
switch {
|
||||
case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"):
|
||||
return imagerequest.FormatJPEG.MIMEType()
|
||||
return contentTypeJPEG
|
||||
case strings.HasSuffix(path, ".png"):
|
||||
return imagerequest.FormatPNG.MIMEType()
|
||||
return contentTypePNG
|
||||
case strings.HasSuffix(path, ".gif"):
|
||||
return imagerequest.FormatGIF.MIMEType()
|
||||
return contentTypeGIF
|
||||
case strings.HasSuffix(path, ".webp"):
|
||||
return imagerequest.FormatWebP.MIMEType()
|
||||
return contentTypeWebP
|
||||
case strings.HasSuffix(path, ".avif"):
|
||||
return imagerequest.FormatAVIF.MIMEType()
|
||||
return contentTypeAVIF
|
||||
case strings.HasSuffix(path, ".jxl"):
|
||||
return imagerequest.FormatJXL.MIMEType()
|
||||
return contentTypeJXL
|
||||
case strings.HasSuffix(path, ".svg"):
|
||||
return contentTypeSVG
|
||||
default:
|
||||
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestFetchSendsRequestID verifies that a fetch sends the ID of the request
|
||||
@@ -26,7 +25,7 @@ func TestFetchSendsRequestID(t *testing.T) {
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
received <- r.Header.Get("X-Request-Id")
|
||||
|
||||
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
|
||||
w.Header().Set("Content-Type", contentTypeJPEG)
|
||||
_, _ = io.WriteString(w, imagePayload)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
@@ -39,31 +39,30 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input []byte
|
||||
sourceFormat imagerequest.Format
|
||||
format imagerequest.Format
|
||||
width, height int
|
||||
fit imagerequest.Fit
|
||||
refused bool
|
||||
}{
|
||||
{"AVIF at the limit", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 4096, 4096, imagerequest.FitCover, false},
|
||||
{"AVIF above the limit", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 4096, 4097, imagerequest.FitCover, true},
|
||||
{"WebP above the AVIF limit", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatWebP, 4096, 4097, imagerequest.FitCover, false},
|
||||
{"AVIF at the limit", cutOff, imagerequest.FormatAVIF,
|
||||
4096, 4096, imagerequest.FitCover, false},
|
||||
{"AVIF above the limit", cutOff, imagerequest.FormatAVIF,
|
||||
4096, 4097, imagerequest.FitCover, true},
|
||||
{"WebP above the AVIF limit", cutOff, imagerequest.FormatWebP,
|
||||
4096, 4097, imagerequest.FitCover, false},
|
||||
// FitInside does not enlarge, so the output stays 64x32
|
||||
{"AVIF inside a larger size", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 8192, 8192, imagerequest.FitInside, false},
|
||||
{"AVIF inside a larger size", cutOff, imagerequest.FormatAVIF,
|
||||
8192, 8192, imagerequest.FitInside, false},
|
||||
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
|
||||
{"AVIF outside a smaller size", cutOff, imagerequest.FormatPNG,
|
||||
imagerequest.FormatAVIF, 8192, 1, imagerequest.FitOutside, true},
|
||||
{"AVIF outside a smaller size", cutOff, imagerequest.FormatAVIF,
|
||||
8192, 1, imagerequest.FitOutside, true},
|
||||
// With no size, the output keeps the source's 4097x4096
|
||||
{"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF,
|
||||
imagerequest.FormatAVIF, 0, 0, imagerequest.FitCover, true},
|
||||
{"orig of an AVIF source", largeAVIF, imagerequest.FormatAVIF,
|
||||
imagerequest.FormatOriginal, 0, 0, imagerequest.FitCover, true},
|
||||
{"JPEG of the source's size", largeAVIF, imagerequest.FormatAVIF,
|
||||
imagerequest.FormatJPEG, 0, 0, imagerequest.FitCover, false},
|
||||
0, 0, imagerequest.FitCover, true},
|
||||
{"orig of an AVIF source", largeAVIF, imagerequest.FormatOriginal,
|
||||
0, 0, imagerequest.FitCover, true},
|
||||
{"JPEG of the source's size", largeAVIF, imagerequest.FormatJPEG,
|
||||
0, 0, imagerequest.FitCover, false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -73,7 +72,7 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||
_, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(), bytes.NewReader(tt.input), tt.sourceFormat,
|
||||
}).Process(t.Context(), bytes.NewReader(tt.input),
|
||||
&imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
|
||||
Format: tt.format,
|
||||
|
||||
@@ -42,7 +42,7 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
@@ -111,7 +111,7 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
|
||||
@@ -7,8 +7,6 @@ import (
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// readerAfterDeadline is a Process input whose first Read waits until done
|
||||
@@ -88,8 +86,7 @@ func checkProcessStopsAtDeadline(
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
|
||||
result, err := proc.Process(ctx, input, imagerequest.FormatJPEG,
|
||||
smallJPEGTransform())
|
||||
result, err := proc.Process(ctx, input, smallJPEGTransform())
|
||||
if result != nil {
|
||||
t.Logf("Process() made a %dx%d image after its deadline passed",
|
||||
result.Width, result.Height)
|
||||
|
||||
@@ -19,7 +19,7 @@ func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)),
|
||||
imagerequest.FormatJPEG, &imagerequest.Transform{},
|
||||
&imagerequest.Transform{},
|
||||
)
|
||||
if !errors.Is(err, ErrUnsupportedOutputFormat) {
|
||||
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
|
||||
|
||||
@@ -11,18 +11,17 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// processedSize runs input, an image in sourceFormat, through Process and
|
||||
// returns the output's size in bytes.
|
||||
// processedSize runs input through Process and returns the output's size in
|
||||
// bytes.
|
||||
func processedSize(
|
||||
t *testing.T, input []byte, sourceFormat imagerequest.Format,
|
||||
transform *imagerequest.Transform,
|
||||
t *testing.T, input []byte, transform *imagerequest.Transform,
|
||||
) int64 {
|
||||
t.Helper()
|
||||
|
||||
result, err := New(Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(), bytes.NewReader(input), sourceFormat, transform)
|
||||
}).Process(t.Context(), bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -75,7 +74,7 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
size := processedSize(t, encodePNG(t, flat), imagerequest.FormatPNG,
|
||||
size := processedSize(t, encodePNG(t, flat),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatPNG})
|
||||
|
||||
const rawBytes = width * height * 3
|
||||
@@ -93,7 +92,7 @@ func TestImageProcessor_WebPAtEffort2(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, imagerequest.FormatJPEG,
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85})
|
||||
|
||||
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
|
||||
@@ -135,7 +134,7 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
|
||||
|
||||
input := encodePNG(t, source)
|
||||
|
||||
size := processedSize(t, input, imagerequest.FormatPNG,
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85})
|
||||
|
||||
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
|
||||
@@ -162,7 +161,7 @@ func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, imagerequest.FormatJPEG,
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85})
|
||||
|
||||
params := vips.NewJxlExportParams()
|
||||
|
||||
@@ -40,7 +40,7 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
@@ -113,7 +113,7 @@ func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatPNG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatPNG,
|
||||
@@ -240,7 +240,7 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatPNG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatPNG,
|
||||
|
||||
@@ -45,7 +45,7 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
@@ -102,7 +102,7 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
|
||||
@@ -70,6 +70,8 @@ type Result struct {
|
||||
InputWidth int
|
||||
// InputHeight is the original image height before processing.
|
||||
InputHeight int
|
||||
// InputFormat is the detected input format (e.g., "jpeg", "png").
|
||||
InputFormat string
|
||||
}
|
||||
|
||||
// MaxInputDimension is the maximum allowed width or height for input images.
|
||||
@@ -111,10 +113,9 @@ var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
|
||||
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
|
||||
|
||||
// ErrUnsupportedInputFormat is returned, before the input is decoded, when
|
||||
// libvips reads the input's first bytes as another format than its source
|
||||
// format, or as none. An SVG whose first bytes pass the first-bytes check for
|
||||
// AVIF is one example. A source format other than the six pixa reads is also
|
||||
// refused with it.
|
||||
// libvips reads the input's first bytes as another format than the source
|
||||
// formats, or as none: for example an SVG whose first bytes pass the
|
||||
// first-bytes check for AVIF.
|
||||
var ErrUnsupportedInputFormat = errors.New("unsupported input format")
|
||||
|
||||
// ErrTooManyImages is returned when MaxConcurrentProcessing images are being
|
||||
@@ -156,10 +157,9 @@ func New(params Params) *ImageProcessor {
|
||||
}
|
||||
}
|
||||
|
||||
// Process transforms the image read from input according to transform.
|
||||
// sourceFormat is the input's format, which magic detected from its first
|
||||
// bytes. When MaxConcurrentProcessing images are already being processed,
|
||||
// it waits up to ProcessingWaitTimeout for one to finish, then fails with
|
||||
// Process transforms an image according to transform. When
|
||||
// MaxConcurrentProcessing images are already being processed, it waits up
|
||||
// to ProcessingWaitTimeout for one to finish, then fails with
|
||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
|
||||
// no way to stop libvips once an encode has begun, so an encode keeps its
|
||||
@@ -168,7 +168,6 @@ func New(params Params) *ImageProcessor {
|
||||
func (p *ImageProcessor) Process(
|
||||
ctx context.Context,
|
||||
input io.Reader,
|
||||
sourceFormat imagerequest.Format,
|
||||
transform *imagerequest.Transform,
|
||||
) (*Result, error) {
|
||||
release, err := p.acquireSlot(ctx)
|
||||
@@ -183,9 +182,13 @@ func (p *ImageProcessor) Process(
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Input libvips would not load as sourceFormat is refused before it is
|
||||
// decoded
|
||||
err = checkInputFormat(data, sourceFormat)
|
||||
// govips picks the libvips loader from the input's first bytes with
|
||||
// DetermineImageType. Input whose first bytes it reads as no source
|
||||
// format, such as SVG, which librsvg would render, is refused before
|
||||
// libvips reads it.
|
||||
inputFormat := p.detectFormat(vips.DetermineImageType(data))
|
||||
|
||||
sourceFormat, err := p.formatFromString(inputFormat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -251,11 +254,12 @@ func (p *ImageProcessor) Process(
|
||||
return &Result{
|
||||
Content: io.NopCloser(bytes.NewReader(output)),
|
||||
ContentLength: int64(len(output)),
|
||||
ContentType: outputFormat.MIMEType(),
|
||||
ContentType: FormatToMIME(outputFormat),
|
||||
Width: img.Width(),
|
||||
Height: img.Height(),
|
||||
InputWidth: origWidth,
|
||||
InputHeight: origHeight,
|
||||
InputFormat: inputFormat,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -290,6 +294,38 @@ func targetDimensions(
|
||||
return width, height, nil
|
||||
}
|
||||
|
||||
// MIME types for the supported image formats.
|
||||
const (
|
||||
mimeJPEG = "image/jpeg"
|
||||
mimePNG = "image/png"
|
||||
mimeGIF = "image/gif"
|
||||
mimeWebP = "image/webp"
|
||||
mimeAVIF = "image/avif"
|
||||
mimeJXL = "image/jxl"
|
||||
)
|
||||
|
||||
// FormatToMIME converts a Format to its MIME type string.
|
||||
func FormatToMIME(format imagerequest.Format) string {
|
||||
switch format {
|
||||
case imagerequest.FormatJPEG:
|
||||
return mimeJPEG
|
||||
case imagerequest.FormatPNG:
|
||||
return mimePNG
|
||||
case imagerequest.FormatWebP:
|
||||
return mimeWebP
|
||||
case imagerequest.FormatGIF:
|
||||
return mimeGIF
|
||||
case imagerequest.FormatAVIF:
|
||||
return mimeAVIF
|
||||
case imagerequest.FormatJXL:
|
||||
return mimeJXL
|
||||
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
|
||||
return "application/octet-stream"
|
||||
default:
|
||||
return "application/octet-stream"
|
||||
}
|
||||
}
|
||||
|
||||
// WaitForProcessing waits until no image is being processed, or until ctx
|
||||
// ends, and returns how many images were still being processed then. It
|
||||
// waits by taking each slot in processingSemaphore as it frees up until it
|
||||
@@ -358,49 +394,27 @@ func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// checkInputFormat returns ErrUnsupportedInputFormat unless libvips would
|
||||
// load data with the loader of sourceFormat. govips picks the libvips loader
|
||||
// from the input's first bytes with DetermineImageType. It refuses an SVG
|
||||
// whose first bytes pass the first-bytes check for AVIF, which librsvg would
|
||||
// otherwise render.
|
||||
func checkInputFormat(data []byte, sourceFormat imagerequest.Format) error {
|
||||
// A format with no loader, such as orig or auto, has the type govips also
|
||||
// gives input it cannot read, which the check below would let through.
|
||||
sourceType := vipsImageType(sourceFormat)
|
||||
if sourceType == vips.ImageTypeUnknown {
|
||||
return fmt.Errorf("%w: %q is not a source format",
|
||||
ErrUnsupportedInputFormat, sourceFormat)
|
||||
}
|
||||
|
||||
if vips.DetermineImageType(data) != sourceType {
|
||||
return fmt.Errorf("%w: libvips does not read the input as %s",
|
||||
ErrUnsupportedInputFormat, sourceFormat)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// vipsImageType returns the govips image type whose libvips loader reads
|
||||
// format. orig and auto stand for another format, so they have
|
||||
// vips.ImageTypeUnknown, which govips has no loader for.
|
||||
func vipsImageType(format imagerequest.Format) vips.ImageType {
|
||||
switch format {
|
||||
case imagerequest.FormatJPEG:
|
||||
return vips.ImageTypeJPEG
|
||||
case imagerequest.FormatPNG:
|
||||
return vips.ImageTypePNG
|
||||
case imagerequest.FormatGIF:
|
||||
return vips.ImageTypeGIF
|
||||
case imagerequest.FormatWebP:
|
||||
return vips.ImageTypeWEBP
|
||||
case imagerequest.FormatAVIF:
|
||||
return vips.ImageTypeAVIF
|
||||
case imagerequest.FormatJXL:
|
||||
return vips.ImageTypeJXL
|
||||
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
|
||||
return vips.ImageTypeUnknown
|
||||
// detectFormat returns the format string for a govips image type.
|
||||
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
switch imageType {
|
||||
case vips.ImageTypeJPEG:
|
||||
return "jpeg"
|
||||
case vips.ImageTypePNG:
|
||||
return "png"
|
||||
case vips.ImageTypeGIF:
|
||||
return "gif"
|
||||
case vips.ImageTypeWEBP:
|
||||
return "webp"
|
||||
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
|
||||
return string(imagerequest.FormatAVIF)
|
||||
case vips.ImageTypeJXL:
|
||||
return string(imagerequest.FormatJXL)
|
||||
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
|
||||
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
|
||||
vips.ImageTypeJP2K:
|
||||
return "unknown"
|
||||
default:
|
||||
return vips.ImageTypeUnknown
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -754,3 +768,26 @@ func jxlDistance(quality int) float64 {
|
||||
|
||||
return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0
|
||||
}
|
||||
|
||||
// formatFromString converts a format string from detectFormat to Format. A
|
||||
// format detectFormat could not name is ErrUnsupportedInputFormat.
|
||||
func (p *ImageProcessor) formatFromString(
|
||||
format string,
|
||||
) (imagerequest.Format, error) {
|
||||
switch format {
|
||||
case "jpeg":
|
||||
return imagerequest.FormatJPEG, nil
|
||||
case "png":
|
||||
return imagerequest.FormatPNG, nil
|
||||
case "gif":
|
||||
return imagerequest.FormatGIF, nil
|
||||
case "webp":
|
||||
return imagerequest.FormatWebP, nil
|
||||
case string(imagerequest.FormatAVIF):
|
||||
return imagerequest.FormatAVIF, nil
|
||||
case string(imagerequest.FormatJXL):
|
||||
return imagerequest.FormatJXL, nil
|
||||
default:
|
||||
return "", fmt.Errorf("%w: %s", ErrUnsupportedInputFormat, format)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -94,23 +94,23 @@ func isAVIF(data []byte) bool {
|
||||
// detectMIME is a minimal magic-byte detector for test assertions.
|
||||
func detectMIME(data []byte) string {
|
||||
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
|
||||
return imagerequest.FormatJPEG.MIMEType()
|
||||
return mimeJPEG
|
||||
}
|
||||
|
||||
if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" {
|
||||
return imagerequest.FormatPNG.MIMEType()
|
||||
return mimePNG
|
||||
}
|
||||
|
||||
if len(data) >= 4 && string(data[:4]) == "GIF8" {
|
||||
return imagerequest.FormatGIF.MIMEType()
|
||||
return mimeGIF
|
||||
}
|
||||
|
||||
if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" {
|
||||
return imagerequest.FormatWebP.MIMEType()
|
||||
return mimeWebP
|
||||
}
|
||||
|
||||
if isAVIF(data) {
|
||||
return imagerequest.FormatAVIF.MIMEType()
|
||||
return mimeAVIF
|
||||
}
|
||||
|
||||
return ""
|
||||
@@ -131,8 +131,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -158,7 +157,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
|
||||
}
|
||||
|
||||
mime := detectMIME(data)
|
||||
if mime != imagerequest.FormatJPEG.MIMEType() {
|
||||
if mime != mimeJPEG {
|
||||
t.Errorf("Output format = %v, want image/jpeg", mime)
|
||||
}
|
||||
}
|
||||
@@ -177,8 +176,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -191,7 +189,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
|
||||
}
|
||||
|
||||
mime := detectMIME(data)
|
||||
if mime != imagerequest.FormatPNG.MIMEType() {
|
||||
if mime != mimePNG {
|
||||
t.Errorf("Output format = %v, want image/png", mime)
|
||||
}
|
||||
}
|
||||
@@ -215,8 +213,7 @@ func processAndCheckSize(
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -256,8 +253,7 @@ func TestImageProcessor_FitContain(t *testing.T) {
|
||||
FitMode: imagerequest.FitContain,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -300,8 +296,7 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatPNG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -346,8 +341,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err == nil {
|
||||
t.Error("Process() should reject oversized input images")
|
||||
}
|
||||
@@ -375,8 +369,7 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf(
|
||||
"Process() should accept images at MaxInputDimension, got error: %v",
|
||||
@@ -406,8 +399,7 @@ func encodeAndCheck(
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil", err)
|
||||
}
|
||||
@@ -438,7 +430,7 @@ func encodeAndCheck(
|
||||
func TestImageProcessor_EncodeWebP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
encodeAndCheck(t, imagerequest.FormatWebP, 80, imagerequest.FormatWebP.MIMEType())
|
||||
encodeAndCheck(t, imagerequest.FormatWebP, 80, mimeWebP)
|
||||
}
|
||||
|
||||
func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
@@ -461,8 +453,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatAVIF, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
|
||||
}
|
||||
@@ -476,7 +467,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
}
|
||||
|
||||
mime := detectMIME(data)
|
||||
if mime != imagerequest.FormatJPEG.MIMEType() {
|
||||
if mime != mimeJPEG {
|
||||
t.Errorf("Output format = %v, want image/jpeg", mime)
|
||||
}
|
||||
}
|
||||
@@ -503,8 +494,7 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err == nil {
|
||||
t.Fatal("Process() should reject input exceeding maxInputBytes")
|
||||
}
|
||||
@@ -531,8 +521,7 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, transform)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil", err)
|
||||
}
|
||||
@@ -543,11 +532,11 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
|
||||
func TestImageProcessor_EncodeAVIF(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
encodeAndCheck(t, imagerequest.FormatAVIF, 85, imagerequest.FormatAVIF.MIMEType())
|
||||
encodeAndCheck(t, imagerequest.FormatAVIF, 85, mimeAVIF)
|
||||
}
|
||||
|
||||
// processAndDecode runs input, a JPEG, through Process and decodes the output
|
||||
// with vips, so a test can inspect the image a client would receive.
|
||||
// processAndDecode runs input through Process and decodes the output with
|
||||
// vips, so a test can inspect the image a client would receive.
|
||||
func processAndDecode(
|
||||
t *testing.T, input []byte, transform *imagerequest.Transform,
|
||||
) *vips.ImageRef {
|
||||
@@ -557,8 +546,7 @@ func processAndDecode(
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
context.Background(), bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
transform,
|
||||
context.Background(), bytes.NewReader(input), transform,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
|
||||
@@ -25,8 +25,7 @@ func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)),
|
||||
imagerequest.FormatJPEG, transform,
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), transform,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
@@ -72,7 +71,7 @@ func TestImageProcessor_JPEGXLQuality(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(t.Context(),
|
||||
bytes.NewReader(input), imagerequest.FormatJPEG,
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() at quality %d error = %v", quality, err)
|
||||
|
||||
@@ -93,8 +93,7 @@ func processInBackground(
|
||||
proc *ImageProcessor, reader *gatedReader, results chan<- error,
|
||||
) {
|
||||
go func() {
|
||||
result, err := proc.Process(context.Background(), reader,
|
||||
imagerequest.FormatJPEG, smallJPEGTransform())
|
||||
result, err := proc.Process(context.Background(), reader, smallJPEGTransform())
|
||||
if err == nil {
|
||||
_ = result.Content.Close()
|
||||
}
|
||||
@@ -207,7 +206,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
|
||||
start := time.Now()
|
||||
|
||||
_, err := proc.Process(context.Background(), bytes.NewReader(input),
|
||||
imagerequest.FormatJPEG, smallJPEGTransform())
|
||||
smallJPEGTransform())
|
||||
if !errors.Is(err, ErrTooManyImages) {
|
||||
t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
|
||||
}
|
||||
@@ -275,8 +274,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
|
||||
proc.processingWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
_, err := proc.Process(context.Background(), tc.input,
|
||||
imagerequest.FormatJPEG, tc.transform)
|
||||
_, err := proc.Process(context.Background(), tc.input, tc.transform)
|
||||
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
|
||||
t.Fatalf("Process() error = %v, want %v", err, tc.want)
|
||||
}
|
||||
@@ -286,7 +284,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
}
|
||||
|
||||
result, err := proc.Process(context.Background(), bytes.NewReader(valid),
|
||||
imagerequest.FormatJPEG, smallJPEGTransform())
|
||||
smallJPEGTransform())
|
||||
if err != nil {
|
||||
t.Fatalf("Process() after the error = %v, want nil", err)
|
||||
}
|
||||
|
||||
@@ -10,9 +10,8 @@ import (
|
||||
)
|
||||
|
||||
// TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF
|
||||
// input is refused with ErrUnsupportedInputFormat before it is decoded, and
|
||||
// not saved as JPEG. TIFF is not a source format, and the test passes the
|
||||
// input to Process as a JPEG.
|
||||
// input, which is not a source format, is refused with
|
||||
// ErrUnsupportedInputFormat before it is decoded, and not saved as JPEG.
|
||||
func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -32,7 +31,7 @@ func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
}).Process(
|
||||
t.Context(), bytes.NewReader(tiff), imagerequest.FormatJPEG,
|
||||
t.Context(), bytes.NewReader(tiff),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatOriginal},
|
||||
)
|
||||
if result != nil {
|
||||
|
||||
@@ -2,7 +2,6 @@ package imageprocessor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"io"
|
||||
"testing"
|
||||
|
||||
@@ -22,7 +21,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
|
||||
} {
|
||||
saved, err := proc.Process(t.Context(),
|
||||
bytes.NewReader(createTestPNG(t, 20, 20)), imagerequest.FormatPNG,
|
||||
bytes.NewReader(createTestPNG(t, 20, 20)),
|
||||
&imagerequest.Transform{Format: format})
|
||||
if err != nil {
|
||||
t.Fatalf("saving the test image as %s: %v", format, err)
|
||||
@@ -33,7 +32,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
t.Fatalf("reading the %s image: %v", format, err)
|
||||
}
|
||||
|
||||
result, err := proc.Process(t.Context(), bytes.NewReader(source), format,
|
||||
result, err := proc.Process(t.Context(), bytes.NewReader(source),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatOriginal})
|
||||
if err != nil {
|
||||
t.Errorf("orig of the %s image: %v", format, err)
|
||||
@@ -43,33 +42,9 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
|
||||
t.Logf("orig of the %s image: %s", format, result.ContentType)
|
||||
|
||||
if result.ContentType != format.MIMEType() {
|
||||
if result.ContentType != FormatToMIME(format) {
|
||||
t.Errorf("orig of the %s image is %s, want %s",
|
||||
format, result.ContentType, format.MIMEType())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_NotASourceFormatRefused verifies that a source format
|
||||
// other than the six, such as orig, auto or none, is refused with
|
||||
// ErrUnsupportedInputFormat. The input is text, which libvips reads as no
|
||||
// format.
|
||||
func TestImageProcessor_NotASourceFormatRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
|
||||
|
||||
for _, sourceFormat := range []imagerequest.Format{
|
||||
imagerequest.FormatOriginal, imagerequest.FormatAuto, "",
|
||||
} {
|
||||
result, err := proc.Process(t.Context(),
|
||||
bytes.NewReader([]byte("this is text, not an image")), sourceFormat,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJPEG})
|
||||
t.Logf("source format %q: error = %v", sourceFormat, err)
|
||||
|
||||
if result != nil || !errors.Is(err, ErrUnsupportedInputFormat) {
|
||||
t.Errorf("Process() with source format %q = %v, error %v; want %v",
|
||||
sourceFormat, result, err, ErrUnsupportedInputFormat)
|
||||
format, result.ContentType, FormatToMIME(format))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -303,22 +303,21 @@ func withoutCancelKeepingDeadline(
|
||||
return context.WithDeadline(context.WithoutCancel(ctx), deadline)
|
||||
}
|
||||
|
||||
// loadCachedSource opens src's cached source, reading only the first bytes,
|
||||
// which give its format, and returns it with its size and its format. It
|
||||
// returns nil if the source is not in the cache, cannot be read, is in no
|
||||
// source format or exceeds maxResponseSize.
|
||||
// loadCachedSource opens src's cached source, without reading it, and
|
||||
// returns it with its size; nil if it is not in the cache, cannot be read,
|
||||
// is empty or exceeds maxResponseSize.
|
||||
func (p *Proxy) loadCachedSource(
|
||||
ctx context.Context, src imagerequest.Source,
|
||||
) (io.ReadCloser, int64, imagerequest.Format) {
|
||||
reader, size, format, err := p.cache.Source(ctx, src)
|
||||
) (io.ReadCloser, int64) {
|
||||
reader, size, err := p.cache.Source(ctx, src)
|
||||
if errors.Is(err, imgcache.ErrNotFound) {
|
||||
return nil, 0, ""
|
||||
return nil, 0
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
p.log.Warn("failed to load cached source, fetching", "error", err)
|
||||
|
||||
return nil, 0, ""
|
||||
return nil, 0
|
||||
}
|
||||
|
||||
p.log.Debug("using cached source", "source_key", src.Key())
|
||||
@@ -331,10 +330,16 @@ func (p *Proxy) loadCachedSource(
|
||||
"max_bytes", p.maxResponseSize,
|
||||
)
|
||||
|
||||
return nil, 0, ""
|
||||
return nil, 0
|
||||
}
|
||||
|
||||
return reader, size, format
|
||||
if size == 0 {
|
||||
_ = reader.Close()
|
||||
|
||||
return nil, 0
|
||||
}
|
||||
|
||||
return reader, size
|
||||
}
|
||||
|
||||
// processFromSourceOrFetch processes an image, using cached source content
|
||||
@@ -344,7 +349,7 @@ func (p *Proxy) processFromSourceOrFetch(
|
||||
req *imagerequest.Request,
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (*processedVariant, error) {
|
||||
source, sourceSize, sourceFormat := p.loadCachedSource(ctx, req.Source)
|
||||
source, sourceSize := p.loadCachedSource(ctx, req.Source)
|
||||
|
||||
// Fetch from upstream if we don't have source data or it's empty
|
||||
if source == nil {
|
||||
@@ -356,7 +361,7 @@ func (p *Proxy) processFromSourceOrFetch(
|
||||
// Process using cached source; nothing was fetched from upstream. The
|
||||
// image processor reads the source only once it has a processing slot,
|
||||
// so a request waiting for one holds none of it in memory.
|
||||
return p.processAndStore(ctx, req, cacheKey, source, sourceFormat, sourceSize)
|
||||
return p.processAndStore(ctx, req, cacheKey, source, sourceSize)
|
||||
}
|
||||
|
||||
// fetchAndProcess fetches from upstream, processes, and caches the result.
|
||||
@@ -437,11 +442,10 @@ func (p *Proxy) fetchAndProcess(
|
||||
"cipher", fetchResult.TLSCipherSuite,
|
||||
)
|
||||
|
||||
// The source's format, detected from its first bytes, which must match
|
||||
// its content type. A body that does not is the upstream's error, so it
|
||||
// gets a negative cache entry, as an upstream error status does.
|
||||
sourceFormat, err := magic.ValidateMagicBytes(sourceData,
|
||||
fetchResult.ContentType)
|
||||
// Validate magic bytes match content type. A body that does not is the
|
||||
// upstream's error, so it gets a negative cache entry, as an upstream
|
||||
// error status does.
|
||||
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType)
|
||||
if err != nil {
|
||||
p.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
|
||||
"host", req.Source.Host, "path", req.Source.Path,
|
||||
@@ -459,7 +463,7 @@ func (p *Proxy) fetchAndProcess(
|
||||
}
|
||||
|
||||
return p.processAndStore(
|
||||
ctx, req, cacheKey, bytes.NewReader(sourceData), sourceFormat, fetchBytes,
|
||||
ctx, req, cacheKey, bytes.NewReader(sourceData), fetchBytes,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -493,14 +497,13 @@ type processedVariant struct {
|
||||
fetchedBytes int64
|
||||
}
|
||||
|
||||
// processAndStore processes the image read from source, which is in
|
||||
// sourceFormat, and stores the result.
|
||||
// processAndStore processes the image read from source and stores the
|
||||
// result.
|
||||
func (p *Proxy) processAndStore(
|
||||
ctx context.Context,
|
||||
req *imagerequest.Request,
|
||||
cacheKey imagerequest.VariantKey,
|
||||
source io.Reader,
|
||||
sourceFormat imagerequest.Format,
|
||||
fetchBytes int64,
|
||||
) (*processedVariant, error) {
|
||||
// Process the image
|
||||
@@ -508,8 +511,7 @@ func (p *Proxy) processAndStore(
|
||||
|
||||
// ctx has the deadline of the request doing the processing, which can
|
||||
// end while a request waiting for its image has time left
|
||||
processResult, err := p.processor.Process(ctx, source, sourceFormat,
|
||||
&req.Transform)
|
||||
processResult, err := p.processor.Process(ctx, source, &req.Transform)
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
|
||||
}
|
||||
@@ -544,7 +546,7 @@ func (p *Proxy) processAndStore(
|
||||
"request_id", middleware.GetReqID(ctx),
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"src_format", sourceFormat,
|
||||
"src_format", processResult.InputFormat,
|
||||
"dst_format", req.Transform.Format,
|
||||
"src_bytes", fetchBytes,
|
||||
"dst_bytes", outputSize,
|
||||
|
||||
@@ -191,14 +191,23 @@ func TestProxy_Get_FormatConversion(t *testing.T) {
|
||||
t.Fatalf("failed to read response: %v", err)
|
||||
}
|
||||
|
||||
detectedFormat, err := magic.DetectFormat(data)
|
||||
detectedMIME, err := magic.DetectFormat(data)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to detect format: %v", err)
|
||||
}
|
||||
|
||||
if detectedFormat.MIMEType() != tt.wantMIME {
|
||||
t.Errorf("detected format = %q, want the format of %s",
|
||||
detectedFormat, tt.wantMIME)
|
||||
expectedFormat, ok := magic.MIMEToImageFormat(tt.wantMIME)
|
||||
if !ok {
|
||||
t.Fatalf("unknown format for MIME type: %s", tt.wantMIME)
|
||||
}
|
||||
|
||||
detectedFormat, ok := magic.MIMEToImageFormat(string(detectedMIME))
|
||||
if !ok {
|
||||
t.Fatalf("unknown format for detected MIME type: %s", detectedMIME)
|
||||
}
|
||||
|
||||
if detectedFormat != expectedFormat {
|
||||
t.Errorf("detected format = %q, want %q", detectedFormat, expectedFormat)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -41,8 +41,7 @@ func holdProcessingSlot(
|
||||
input, feed := io.Pipe()
|
||||
|
||||
go func() {
|
||||
_, _ = proc.Process(t.Context(), input, imagerequest.FormatJPEG,
|
||||
&imagerequest.Transform{})
|
||||
_, _ = proc.Process(t.Context(), input, &imagerequest.Transform{})
|
||||
}()
|
||||
|
||||
_, err := feed.Write([]byte{0})
|
||||
@@ -58,11 +57,10 @@ func holdProcessingSlot(
|
||||
|
||||
// TestProxy_Get_WaitsForSlotBeforeReadingCachedSource checks that a
|
||||
// request whose source is cached holds none of it while it waits for a
|
||||
// processing slot: past the first bytes, which give the source's format, it
|
||||
// reads the cached file only once it has a slot. With the only slot held, a
|
||||
// request for a new width of the cached 100x100 photo waits; the cached file
|
||||
// is then rewritten as a 100x50 image before the slot is freed, so the
|
||||
// request must answer with that image scaled to 40x20.
|
||||
// processing slot: it reads the cached file only once it has a slot. With
|
||||
// the only slot held, a request for a new width of the cached 100x100 photo
|
||||
// waits; the cached file is then rewritten as a 100x50 image before the slot
|
||||
// is freed, so the request must answer with that image scaled to 40x20.
|
||||
func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -81,7 +79,7 @@ func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
|
||||
cachedSource := widthOnlyRequest(fixtures, 50).Source
|
||||
|
||||
reader, _, _, err := svc.cache.Source(t.Context(), cachedSource)
|
||||
reader, _, err := svc.cache.Source(t.Context(), cachedSource)
|
||||
if err != nil {
|
||||
t.Fatalf("Source() error = %v; want the cached source", err)
|
||||
}
|
||||
|
||||
@@ -55,7 +55,7 @@ func TestProxy_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
|
||||
|
||||
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
|
||||
|
||||
reader, _, _, err := svc.cache.Source(t.Context(), source)
|
||||
reader, _, err := svc.cache.Source(t.Context(), source)
|
||||
if err != nil {
|
||||
t.Fatalf("Source() error = %v, want the cached source", err)
|
||||
}
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// getImage calls Get for req and returns the image it answered with and
|
||||
@@ -50,7 +49,7 @@ func cachedSourcePath(
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
reader, _, _, err := svc.cache.Source(t.Context(), req.Source)
|
||||
reader, _, err := svc.cache.Source(t.Context(), req.Source)
|
||||
if err != nil {
|
||||
t.Fatalf("Source() error = %v; want the cached source", err)
|
||||
}
|
||||
@@ -105,40 +104,6 @@ func TestProxy_Get_FetchesAgainForRemovedCachedSource(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_FetchesAgainSourceOfNoSourceFormat replaces the cached
|
||||
// source's file with text. The next request for a new size of the source
|
||||
// must fetch it again and cache it whole.
|
||||
func TestProxy_Get_FetchesAgainSourceOfNoSourceFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := setupTestProxy(t)
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
wholeSize := fileSize(t, path)
|
||||
|
||||
err := os.WriteFile(path, []byte("this is text, not an image"),
|
||||
imgcache.StorageFilePerm)
|
||||
if err != nil {
|
||||
t.Fatalf("replacing the source's file: %v", err)
|
||||
}
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
|
||||
fetches := readCounters(t, svc).upstreamFetchCount
|
||||
t.Logf("upstream fetches = %d", fetches)
|
||||
|
||||
if fetches != 2 {
|
||||
t.Errorf("upstream fetches = %d, want 2", fetches)
|
||||
}
|
||||
|
||||
size := fileSize(t, path)
|
||||
if size != wholeSize {
|
||||
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProxy_Get_ConvertsAgainForRemovedCachedVariant caches a variant of
|
||||
// the photo and removes its row and file, as the reconciliation pass does
|
||||
// when the file is empty or cut short. internal/imgcache tests that the
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
// Package imagerequest holds the image request type, its parts, its checks,
|
||||
// the parsing of the /v1/image/ path, the variant key, the source key and the
|
||||
// source's upstream URL. It also holds the image format type and the MIME type
|
||||
// of each format. It imports no other package of this repo, so every package
|
||||
// that handles an image request can import it.
|
||||
// source's upstream URL. It imports no other package of this repo, so every
|
||||
// package that handles an image request can import it.
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
@@ -13,11 +12,10 @@ import (
|
||||
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
|
||||
var ErrInvalidFitMode = errors.New("invalid fit mode")
|
||||
|
||||
// Format is an image format. pixa reads and writes the formats from JPEG to
|
||||
// GIF below. In a request, orig and auto stand for one of them.
|
||||
// Format represents supported output image formats.
|
||||
type Format string
|
||||
|
||||
// Supported image formats.
|
||||
// Supported image output formats.
|
||||
const (
|
||||
FormatOriginal Format = "orig"
|
||||
FormatJPEG Format = "jpeg"
|
||||
@@ -33,30 +31,6 @@ const (
|
||||
FormatAuto Format = "auto"
|
||||
)
|
||||
|
||||
// MIMEType returns the MIME type of the format, such as image/jpeg for
|
||||
// FormatJPEG. orig and auto stand for another format, so they have none:
|
||||
// MIMEType returns "" for them.
|
||||
func (f Format) MIMEType() string {
|
||||
switch f {
|
||||
case FormatJPEG:
|
||||
return "image/jpeg"
|
||||
case FormatPNG:
|
||||
return "image/png"
|
||||
case FormatWebP:
|
||||
return "image/webp"
|
||||
case FormatAVIF:
|
||||
return "image/avif"
|
||||
case FormatJXL:
|
||||
return "image/jxl"
|
||||
case FormatGIF:
|
||||
return "image/gif"
|
||||
case FormatOriginal, FormatAuto:
|
||||
return ""
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// Size represents requested image dimensions
|
||||
type Size struct {
|
||||
Width int
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
package imagerequest
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestFormat_MIMEType checks the MIME type of each format pixa reads and
|
||||
// writes, and that orig and auto have none.
|
||||
func TestFormat_MIMEType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
format Format
|
||||
want string
|
||||
}{
|
||||
{FormatJPEG, "image/jpeg"},
|
||||
{FormatPNG, "image/png"},
|
||||
{FormatWebP, "image/webp"},
|
||||
{FormatAVIF, "image/avif"},
|
||||
{FormatJXL, "image/jxl"},
|
||||
{FormatGIF, "image/gif"},
|
||||
{FormatOriginal, ""},
|
||||
{FormatAuto, ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
got := tt.format.MIMEType()
|
||||
if got != tt.want {
|
||||
t.Errorf("%s MIMEType() = %q, want %q", tt.format, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
+20
-60
@@ -19,7 +19,6 @@ import (
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// maxPendingAccessTimes is how many access times of variants served from
|
||||
@@ -58,8 +57,8 @@ type Config struct {
|
||||
// Cache implements the caching layer for the image proxy.
|
||||
type Cache struct {
|
||||
db *sql.DB
|
||||
srcContent *FileStore // source images by source key
|
||||
variants *FileStore // processed variants by cache key
|
||||
srcContent *ContentStorage // source images by source key
|
||||
variants *VariantStorage // processed variants by cache key
|
||||
config Config
|
||||
log *slog.Logger
|
||||
|
||||
@@ -136,14 +135,14 @@ func NewCache(
|
||||
return c, nil
|
||||
}
|
||||
|
||||
srcContent, err := NewFileStore(
|
||||
srcContent, err := NewContentStorage(
|
||||
filepath.Join(config.StateDir, "cache", "sources"),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create source content storage: %w", err)
|
||||
}
|
||||
|
||||
variants, err := NewFileStore(
|
||||
variants, err := NewVariantStorage(
|
||||
filepath.Join(config.StateDir, "cache", "variants"),
|
||||
)
|
||||
if err != nil {
|
||||
@@ -170,21 +169,14 @@ func (c *Cache) Variant(
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
file, size, err := c.variants.Open(string(cacheKey))
|
||||
reader, size, contentType, err := c.variants.LoadWithContentType(cacheKey)
|
||||
if err != nil {
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
format, err := readFormat(file)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
c.touchVariant(cacheKey)
|
||||
|
||||
return file, size, format.MIMEType(), nil
|
||||
return reader, size, contentType, nil
|
||||
}
|
||||
|
||||
// PutVariant stores a processed variant by its cache key and records
|
||||
@@ -198,18 +190,11 @@ func (c *Cache) PutVariant(
|
||||
return nil
|
||||
}
|
||||
|
||||
data, err := io.ReadAll(content)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to read content: %w", err)
|
||||
}
|
||||
|
||||
err = c.variants.Write(string(cacheKey), data)
|
||||
size, err := c.variants.Store(cacheKey, content)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
size := int64(len(data))
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO variant_content (cache_key, size_bytes)
|
||||
VALUES (?, ?)
|
||||
@@ -227,17 +212,15 @@ func (c *Cache) PutVariant(
|
||||
return nil
|
||||
}
|
||||
|
||||
// Source opens the cached source of src and returns a reader, its size in
|
||||
// bytes and its format, read from the file's first bytes. A source whose row
|
||||
// or file is missing is ErrNotFound, as is any source of a disabled cache; a
|
||||
// file whose first bytes are not those of a source format is another error.
|
||||
// A source it opens has its access time handed to the background writer
|
||||
// without waiting (see touchSourceContent).
|
||||
// Source opens the cached source of src and returns a reader and its size
|
||||
// in bytes. A source whose row or file is missing is ErrNotFound, as is any
|
||||
// source of a disabled cache. A source it opens has its access time handed
|
||||
// to the background writer without waiting (see touchSourceContent).
|
||||
func (c *Cache) Source(
|
||||
ctx context.Context, src imagerequest.Source,
|
||||
) (io.ReadCloser, int64, imagerequest.Format, error) {
|
||||
) (io.ReadCloser, int64, error) {
|
||||
if c.config.MaxBytes == 0 {
|
||||
return nil, 0, "", ErrNotFound
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
key := src.Key()
|
||||
@@ -248,28 +231,21 @@ func (c *Cache) Source(
|
||||
SELECT EXISTS(SELECT 1 FROM source_content WHERE source_key = ?)
|
||||
`, string(key)).Scan(&rowExists)
|
||||
if err != nil {
|
||||
return nil, 0, "", fmt.Errorf("failed to lookup source: %w", err)
|
||||
return nil, 0, fmt.Errorf("failed to lookup source: %w", err)
|
||||
}
|
||||
|
||||
if !rowExists {
|
||||
return nil, 0, "", ErrNotFound
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
file, size, err := c.srcContent.Open(string(key))
|
||||
reader, size, err := c.srcContent.LoadWithSize(key)
|
||||
if err != nil {
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
format, err := readFormat(file)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return nil, 0, "", err
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
c.touchSourceContent(key)
|
||||
|
||||
return file, size, format, nil
|
||||
return reader, size, nil
|
||||
}
|
||||
|
||||
// PutSource stores fetched source content: it writes the file named by
|
||||
@@ -294,13 +270,11 @@ func (c *Cache) PutSource(
|
||||
|
||||
key := src.Key()
|
||||
|
||||
err = c.srcContent.Write(string(key), data)
|
||||
size, err := c.srcContent.Store(key, data)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to store source content: %w", err)
|
||||
}
|
||||
|
||||
size := int64(len(data))
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO source_content (source_key, size_bytes)
|
||||
VALUES (?, ?)
|
||||
@@ -309,7 +283,7 @@ func (c *Cache) PutSource(
|
||||
last_accessed_at = CURRENT_TIMESTAMP
|
||||
`, string(key), size)
|
||||
if err != nil {
|
||||
deleteErr := c.srcContent.Delete(string(key))
|
||||
deleteErr := c.srcContent.Delete(key)
|
||||
if deleteErr != nil {
|
||||
c.log.Warn("failed to remove source file whose row was not written",
|
||||
"source_key", key, "error", deleteErr)
|
||||
@@ -448,17 +422,3 @@ func (c *Cache) cleanExpiredBatch(ctx context.Context) (int64, error) {
|
||||
|
||||
return deleted, nil
|
||||
}
|
||||
|
||||
// readFormat reads the first bytes of a source or variant file, without
|
||||
// moving its offset, and returns the format they show. Sources and variants
|
||||
// are in the same six formats.
|
||||
func readFormat(content io.ReaderAt) (imagerequest.Format, error) {
|
||||
firstBytes := make([]byte, magic.MinMagicBytes)
|
||||
|
||||
_, err := content.ReadAt(firstBytes, 0)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read the first bytes: %w", err)
|
||||
}
|
||||
|
||||
return magic.DetectFormat(firstBytes)
|
||||
}
|
||||
|
||||
@@ -32,7 +32,7 @@ func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
|
||||
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
|
||||
} {
|
||||
result, err := processor.Process(t.Context(), bytes.NewReader(source),
|
||||
imagerequest.FormatJPEG, &imagerequest.Transform{
|
||||
&imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 20, Height: 15},
|
||||
Format: format,
|
||||
Quality: 85,
|
||||
@@ -68,7 +68,7 @@ func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
|
||||
|
||||
t.Logf("%s: %d bytes served as %s", format, size, contentType)
|
||||
|
||||
if want := format.MIMEType(); contentType != want {
|
||||
if want := imageprocessor.FormatToMIME(format); contentType != want {
|
||||
t.Errorf("%s variant content type = %q, want %q", format, contentType, want)
|
||||
}
|
||||
|
||||
@@ -96,7 +96,7 @@ func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
|
||||
}{
|
||||
{"text", []byte("this is text, not an image"), magic.ErrUnknownFormat},
|
||||
{"SVG", []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
|
||||
magic.ErrUnknownFormat},
|
||||
errNotAnOutputFormat},
|
||||
{"JPEG cut to 10 bytes", jpegImage[:10], io.EOF},
|
||||
{"empty", nil, io.EOF},
|
||||
}
|
||||
@@ -104,7 +104,7 @@ func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
|
||||
for i, tt := range tests {
|
||||
key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd20%02d", i))
|
||||
|
||||
err := cache.variants.Write(string(key), tt.content)
|
||||
_, err := cache.variants.Store(key, bytes.NewReader(tt.content))
|
||||
if err != nil {
|
||||
t.Fatalf("%s: storing the variant file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
@@ -48,7 +48,7 @@ const staleTempFileAge = time.Hour
|
||||
const sqliteTimestampLayout = "2006-01-02 15:04:05"
|
||||
|
||||
// tempFilePrefix is the prefix os.CreateTemp uses for in-flight cache
|
||||
// writes (the ".tmp-*" pattern in writeFileSynced).
|
||||
// writes (".tmp-*" patterns in the storage layer).
|
||||
const tempFilePrefix = ".tmp-"
|
||||
|
||||
// UsageBytes returns the total number of bytes of cache content
|
||||
@@ -306,7 +306,7 @@ func (c *Cache) evictVariant(
|
||||
return fmt.Errorf("failed to delete variant accounting row: %w", err)
|
||||
}
|
||||
|
||||
err = c.variants.Delete(string(cacheKey))
|
||||
err = c.variants.Delete(cacheKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -325,7 +325,7 @@ func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey)
|
||||
return fmt.Errorf("failed to delete source content row: %w", err)
|
||||
}
|
||||
|
||||
return c.srcContent.Delete(string(key))
|
||||
return c.srcContent.Delete(key)
|
||||
}
|
||||
|
||||
// notifyWritePressure wakes the background evictor after a store, so
|
||||
@@ -515,15 +515,12 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||
}
|
||||
|
||||
// reconcileFiles walks baseDir, one of the cache's storage directories,
|
||||
// sweeping stale temp files and passing each other file, with its info and
|
||||
// name, to reconcileFile. It skips a directory it cannot list and a file
|
||||
// whose info it cannot read, so one such entry does not stop the pass. It
|
||||
// logs a warning naming the path and the error. A file renamed or removed
|
||||
// after its directory was listed is skipped with no warning. Once ctx is
|
||||
// cancelled it stops at the next file and returns ctx's error.
|
||||
// sweeping stale temp files and passing each other file, with its name, to
|
||||
// reconcileFile. Once ctx is cancelled it stops at the next file and
|
||||
// returns ctx's error.
|
||||
func (c *Cache) reconcileFiles(
|
||||
ctx context.Context, baseDir string,
|
||||
reconcileFile func(path string, info fs.FileInfo, name string) error,
|
||||
reconcileFile func(path string, entry fs.DirEntry, name string) error,
|
||||
) error {
|
||||
return filepath.WalkDir(
|
||||
baseDir,
|
||||
@@ -532,38 +529,19 @@ func (c *Cache) reconcileFiles(
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
c.log.Warn("skipped unreadable entry in cache directory",
|
||||
"path", path, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if entry.IsDir() {
|
||||
return nil
|
||||
}
|
||||
|
||||
info, err := entry.Info()
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
c.log.Warn("skipped unreadable entry in cache directory",
|
||||
"path", path, "error", err)
|
||||
|
||||
return nil
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
|
||||
name := entry.Name()
|
||||
|
||||
if strings.HasPrefix(name, tempFilePrefix) {
|
||||
c.sweepStaleTempFile(path, info)
|
||||
c.sweepStaleTempFile(path, entry)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return reconcileFile(path, info, name)
|
||||
return reconcileFile(path, entry, name)
|
||||
},
|
||||
)
|
||||
}
|
||||
@@ -573,8 +551,8 @@ func (c *Cache) reconcileFiles(
|
||||
// and sweeping stale temp files.
|
||||
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||
return c.reconcileFiles(ctx, c.variants.baseDir,
|
||||
func(path string, info fs.FileInfo, name string) error {
|
||||
return c.reconcileVariantFile(ctx, path, info, imagerequest.VariantKey(name))
|
||||
func(path string, entry fs.DirEntry, name string) error {
|
||||
return c.reconcileVariantFile(ctx, path, entry, imagerequest.VariantKey(name))
|
||||
})
|
||||
}
|
||||
|
||||
@@ -584,11 +562,16 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||
// crash or a restored snapshot can leave it, is removed with its row and
|
||||
// a warning naming it, so the variant is converted again.
|
||||
func (c *Cache) reconcileVariantFile(
|
||||
ctx context.Context, path string, info fs.FileInfo, cacheKey imagerequest.VariantKey,
|
||||
ctx context.Context, path string, entry fs.DirEntry, cacheKey imagerequest.VariantKey,
|
||||
) error {
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to stat variant file: %w", err)
|
||||
}
|
||||
|
||||
var recordedSize int64
|
||||
|
||||
err := c.db.QueryRowContext(ctx,
|
||||
err = c.db.QueryRowContext(ctx,
|
||||
`SELECT size_bytes FROM variant_content WHERE cache_key = ?`, string(cacheKey),
|
||||
).Scan(&recordedSize)
|
||||
|
||||
@@ -630,9 +613,7 @@ func (c *Cache) adoptVariantFile(
|
||||
}
|
||||
|
||||
// reconcileVariantRows drops accounting rows whose variant files are
|
||||
// missing, so the database never references deleted content. A file whose
|
||||
// size cannot be read for another reason may still be on disk, so its row
|
||||
// stays, with a warning naming its key and the error.
|
||||
// missing, so the database never references deleted content.
|
||||
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||
var after imagerequest.VariantKey
|
||||
|
||||
@@ -655,19 +636,11 @@ func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
_, err := c.variants.Size(string(key))
|
||||
if err == nil {
|
||||
if c.variants.Exists(key) {
|
||||
continue
|
||||
}
|
||||
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
c.log.Warn("kept accounting row for unreadable variant file",
|
||||
"cache_key", key, "error", err)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
_, err = c.db.ExecContext(ctx,
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
|
||||
@@ -731,8 +704,8 @@ func queryStringColumn[T ~string](
|
||||
// and sweeping stale temp files.
|
||||
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
||||
return c.reconcileFiles(ctx, c.srcContent.baseDir,
|
||||
func(path string, info fs.FileInfo, name string) error {
|
||||
return c.reconcileSourceFile(ctx, path, info, imagerequest.SourceKey(name))
|
||||
func(path string, entry fs.DirEntry, name string) error {
|
||||
return c.reconcileSourceFile(ctx, path, entry, imagerequest.SourceKey(name))
|
||||
})
|
||||
}
|
||||
|
||||
@@ -743,11 +716,16 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
||||
// is removed with its row and a warning naming it, so the source is
|
||||
// fetched again.
|
||||
func (c *Cache) reconcileSourceFile(
|
||||
ctx context.Context, path string, info fs.FileInfo, key imagerequest.SourceKey,
|
||||
ctx context.Context, path string, entry fs.DirEntry, key imagerequest.SourceKey,
|
||||
) error {
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to stat source content file: %w", err)
|
||||
}
|
||||
|
||||
var recordedSize int64
|
||||
|
||||
err := c.db.QueryRowContext(ctx,
|
||||
err = c.db.QueryRowContext(ctx,
|
||||
`SELECT size_bytes FROM source_content WHERE source_key = ?`,
|
||||
string(key),
|
||||
).Scan(&recordedSize)
|
||||
@@ -912,12 +890,17 @@ func (c *Cache) correctUsageTotal(
|
||||
|
||||
// sweepStaleTempFile removes a temp file left behind by a crashed
|
||||
// write once it is old enough that no in-flight store can own it.
|
||||
func (c *Cache) sweepStaleTempFile(path string, info fs.FileInfo) {
|
||||
func (c *Cache) sweepStaleTempFile(path string, entry fs.DirEntry) {
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Since(info.ModTime()) < staleTempFileAge {
|
||||
return
|
||||
}
|
||||
|
||||
err := os.Remove(path)
|
||||
err = os.Remove(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
c.log.Warn("failed to remove stale temp file", "path", path, "error", err)
|
||||
|
||||
|
||||
@@ -9,10 +9,8 @@ import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -211,8 +209,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
|
||||
t.Fatalf("failed to scan source_key: %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(key))
|
||||
if err != nil {
|
||||
if !cache.srcContent.Exists(imagerequest.SourceKey(key)) {
|
||||
t.Errorf("source_content references content %s but the file is missing",
|
||||
key)
|
||||
}
|
||||
@@ -239,8 +236,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
|
||||
t.Fatalf("failed to scan cache_key: %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.variants.keyToPath(key))
|
||||
if err != nil {
|
||||
if !cache.variants.Exists(imagerequest.VariantKey(key)) {
|
||||
t.Errorf("variant_content references key %s but the file is missing", key)
|
||||
}
|
||||
}
|
||||
@@ -338,8 +334,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(keys[0])))
|
||||
if err == nil {
|
||||
if cache.variants.Exists(keys[0]) {
|
||||
t.Errorf("least recently used variant %s must be evicted", keys[0])
|
||||
}
|
||||
|
||||
@@ -350,8 +345,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, key := range keys[1:] {
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(key)))
|
||||
if err != nil {
|
||||
if !cache.variants.Exists(key) {
|
||||
t.Errorf("more recently used variant %s must survive eviction", key)
|
||||
}
|
||||
}
|
||||
@@ -379,8 +373,7 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, key := range keys {
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(string(key)))
|
||||
if err != nil {
|
||||
if !cache.srcContent.Exists(key) {
|
||||
t.Errorf("source %s must not be evicted while usage is under the limit", key)
|
||||
}
|
||||
|
||||
@@ -391,8 +384,7 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(testVariantKeyOne)))
|
||||
if err != nil {
|
||||
if !cache.variants.Exists(testVariantKeyOne) {
|
||||
t.Error("variant must not be evicted while usage is under the limit")
|
||||
}
|
||||
|
||||
@@ -457,7 +449,7 @@ func assertDisabledCacheReadsAlwaysMiss(
|
||||
|
||||
ctx := t.Context()
|
||||
|
||||
reader, _, _, err := cache.Source(ctx, req.Source)
|
||||
reader, _, err := cache.Source(ctx, req.Source)
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("Source on disabled cache error = %v, want %v", err, ErrNotFound)
|
||||
}
|
||||
@@ -594,7 +586,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
||||
for i, key := range keys {
|
||||
content := bytes.Repeat([]byte{fills[i]}, 1000)
|
||||
|
||||
err = cache.variants.Write(string(key), content)
|
||||
_, err = cache.variants.Store(key, bytes.NewReader(content))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store variant file: %v", err)
|
||||
}
|
||||
@@ -613,8 +605,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
||||
stored = 0
|
||||
|
||||
for _, key := range keys {
|
||||
_, err = os.Stat(cache.variants.keyToPath(string(key)))
|
||||
if err == nil {
|
||||
if cache.variants.Exists(key) {
|
||||
stored++
|
||||
}
|
||||
}
|
||||
@@ -646,7 +637,9 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
|
||||
// feature existed) must be adopted into the accounting.
|
||||
untracked := bytes.Repeat([]byte{0x31}, 1000)
|
||||
|
||||
err := cache.variants.Write(string(testVariantKeyOne), untracked)
|
||||
_, err := cache.variants.Store(
|
||||
testVariantKeyOne, bytes.NewReader(untracked),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store untracked variant file: %v", err)
|
||||
}
|
||||
@@ -701,291 +694,6 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// logLineNaming returns the first line of logs, written by slog's JSON
|
||||
// handler, that is at level and names the variant key, or "" if none is.
|
||||
func logLineNaming(logs, level string, key imagerequest.VariantKey) string {
|
||||
for line := range strings.Lines(logs) {
|
||||
if strings.Contains(line, `"level":"`+level+`"`) &&
|
||||
strings.Contains(line, `"cache_key":"`+string(key)+`"`) {
|
||||
return line
|
||||
}
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// TestReconcileVariantRowsDropsOnlyRowsOfMissingFiles gives
|
||||
// reconcileVariantRows two tracked variants. The first one's file is a
|
||||
// symbolic link to itself, so its size cannot be read. Its row must stay,
|
||||
// with a warning naming its key and the error. The second one has no file.
|
||||
// Its row must go, with the info line for a missing file.
|
||||
func TestReconcileVariantRowsDropsOnlyRowsOfMissingFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyOne, []byte("variant one"))
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyTwo, []byte("variant two"))
|
||||
|
||||
unreadablePath := cache.variants.keyToPath(string(testVariantKeyOne))
|
||||
|
||||
err := os.Remove(unreadablePath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to remove the first variant's file: %v", err)
|
||||
}
|
||||
|
||||
err = os.Symlink(unreadablePath, unreadablePath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to link the first variant's path to itself: %v", err)
|
||||
}
|
||||
|
||||
_, statErr := os.Stat(unreadablePath)
|
||||
if statErr == nil || errors.Is(statErr, fs.ErrNotExist) {
|
||||
t.Fatalf("os.Stat() of the first variant: error = %v, "+
|
||||
"want an error other than a missing file", statErr)
|
||||
}
|
||||
|
||||
err = os.Remove(cache.variants.keyToPath(string(testVariantKeyTwo)))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to remove the second variant's file: %v", err)
|
||||
}
|
||||
|
||||
var logBuf bytes.Buffer
|
||||
|
||||
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
|
||||
|
||||
err = cache.reconcileVariantRows(t.Context())
|
||||
t.Logf("reconcileVariantRows() error = %v, log: %s", err, logBuf.String())
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("reconcileVariantRows() error = %v", err)
|
||||
}
|
||||
|
||||
rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
|
||||
|
||||
if n := countRows(t, cache, rowQuery, string(testVariantKeyOne)); n != 1 {
|
||||
t.Errorf("the variant whose size cannot be read has %d rows, want 1", n)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache, rowQuery, string(testVariantKeyTwo)); n != 0 {
|
||||
t.Errorf("the variant with no file has %d rows, want 0", n)
|
||||
}
|
||||
|
||||
logs := logBuf.String()
|
||||
|
||||
warning := logLineNaming(logs, "WARN", testVariantKeyOne)
|
||||
if !strings.Contains(warning, statErr.Error()) {
|
||||
t.Errorf("no warning names %s and the error %q", testVariantKeyOne, statErr)
|
||||
}
|
||||
|
||||
if line := logLineNaming(logs, "INFO", testVariantKeyOne); line != "" {
|
||||
t.Errorf("the variant whose size cannot be read was logged at info: %s", line)
|
||||
}
|
||||
|
||||
dropped := logLineNaming(logs, "INFO", testVariantKeyTwo)
|
||||
if !strings.Contains(dropped, "dropped accounting row for missing variant file") {
|
||||
t.Errorf("no info line says the file of %s is missing", testVariantKeyTwo)
|
||||
}
|
||||
}
|
||||
|
||||
// openDirectoryNearPathLimit makes a chain of directories inside dir, each
|
||||
// named with 255 zeros, the longest name Linux allows. Linux refuses a path
|
||||
// of 4096 bytes or more, and the chain goes as deep as it can below that.
|
||||
// It returns the chain's deepest directory as an os.Root, closed when the
|
||||
// test ends. An entry made through it with a 255-byte name has a path even
|
||||
// the root user cannot read. The os.Root can make that entry because it
|
||||
// names the entry only by its name, not by the whole path.
|
||||
func openDirectoryNearPathLimit(t *testing.T, dir string) *os.Root {
|
||||
t.Helper()
|
||||
|
||||
name := strings.Repeat("0", 255)
|
||||
deepest := dir
|
||||
|
||||
for len(filepath.Join(deepest, name)) < 4096 {
|
||||
deepest = filepath.Join(deepest, name)
|
||||
}
|
||||
|
||||
err := os.MkdirAll(deepest, StorageDirPerm)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to make a chain of directories: %v", err)
|
||||
}
|
||||
|
||||
root, err := os.OpenRoot(deepest)
|
||||
if err != nil {
|
||||
t.Fatalf("os.OpenRoot(%s) error = %v", deepest, err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = root.Close() })
|
||||
|
||||
return root
|
||||
}
|
||||
|
||||
// makeEntriesTooLongToRead makes a new directory and a new file in the
|
||||
// deepest directory of the chain that openDirectoryNearPathLimit makes
|
||||
// inside dir. Even the root user can then neither list the new directory
|
||||
// nor read the new file's info. It returns the paths of the new directory
|
||||
// and the new file.
|
||||
func makeEntriesTooLongToRead(t *testing.T, dir string) []string {
|
||||
t.Helper()
|
||||
|
||||
deepest := openDirectoryNearPathLimit(t, dir)
|
||||
dirName := strings.Repeat("0", 255)
|
||||
|
||||
err := deepest.Mkdir(dirName, StorageDirPerm)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to make a directory too deep to list: %v", err)
|
||||
}
|
||||
|
||||
fileName := strings.Repeat("1", 255)
|
||||
|
||||
err = deepest.WriteFile(fileName, []byte("unreadable"), StorageFilePerm)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to make a file too deep to read: %v", err)
|
||||
}
|
||||
|
||||
return []string{
|
||||
filepath.Join(deepest.Name(), dirName),
|
||||
filepath.Join(deepest.Name(), fileName),
|
||||
}
|
||||
}
|
||||
|
||||
// makeTempFileTooLongToRead makes a temp file in the deepest directory of
|
||||
// the chain that openDirectoryNearPathLimit makes inside dir. Even the
|
||||
// root user then cannot read the temp file's info. It returns the temp
|
||||
// file's path.
|
||||
func makeTempFileTooLongToRead(t *testing.T, dir string) string {
|
||||
t.Helper()
|
||||
|
||||
deepest := openDirectoryNearPathLimit(t, dir)
|
||||
name := tempFilePrefix + strings.Repeat("1", 255-len(tempFilePrefix))
|
||||
|
||||
err := deepest.WriteFile(name, []byte("partial"), StorageFilePerm)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to make a temp file too deep to read: %v", err)
|
||||
}
|
||||
|
||||
return filepath.Join(deepest.Name(), name)
|
||||
}
|
||||
|
||||
// assertSkippedEntryWarning checks that logs hold the warning "skipped
|
||||
// unreadable entry in cache directory" naming path, with the error
|
||||
// ENAMETOOLONG for that path.
|
||||
func assertSkippedEntryWarning(t *testing.T, logs, path string) {
|
||||
t.Helper()
|
||||
|
||||
for line := range strings.Lines(logs) {
|
||||
if strings.Contains(line, `"level":"WARN"`) &&
|
||||
strings.Contains(line, `"msg":"skipped unreadable entry in cache directory"`) &&
|
||||
strings.Contains(line, `"path":"`+path+`"`) &&
|
||||
strings.Contains(line, path+": "+syscall.ENAMETOOLONG.Error()) {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
t.Errorf("no warning %q names %s with the error %q",
|
||||
"skipped unreadable entry in cache directory", path, syscall.ENAMETOOLONG)
|
||||
}
|
||||
|
||||
// TestReconciliationGoesOnPastEntriesItCannotRead puts, in each storage
|
||||
// directory, a directory the pass cannot list and a file whose info it
|
||||
// cannot read. The pass walks each directory in name order, so it reaches
|
||||
// them before the cache's files, whose keys start with a letter. It must
|
||||
// log a warning naming each one's path and the error, and still do the
|
||||
// rest of its work:
|
||||
// - adopt a variant file that has no row
|
||||
// - drop the row of a variant whose file is missing
|
||||
// - remove a source file that has no row
|
||||
// - correct a wrong total cache usage
|
||||
func TestReconciliationGoesOnPastEntriesItCannotRead(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
ctx := t.Context()
|
||||
|
||||
untrackedVariant := []byte("variant with no row")
|
||||
|
||||
err := cache.variants.Write(string(testVariantKeyOne), untrackedVariant)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to write a variant file with no row: %v", err)
|
||||
}
|
||||
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyTwo, []byte("variant two"))
|
||||
|
||||
err = os.Remove(cache.variants.keyToPath(string(testVariantKeyTwo)))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to remove the second variant's file: %v", err)
|
||||
}
|
||||
|
||||
untrackedSourceKey := "ffeeddcc0001"
|
||||
|
||||
err = cache.srcContent.Write(untrackedSourceKey, []byte("source with no row"))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to write a source file with no row: %v", err)
|
||||
}
|
||||
|
||||
_, err = cache.db.ExecContext(ctx,
|
||||
`UPDATE cache_usage SET total_size_bytes = 1 WHERE id = 1`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set a wrong total: %v", err)
|
||||
}
|
||||
|
||||
unreadable := slices.Concat(
|
||||
makeEntriesTooLongToRead(t, cache.variants.baseDir),
|
||||
makeEntriesTooLongToRead(t, cache.srcContent.baseDir),
|
||||
)
|
||||
|
||||
logs := reconcileAsAtStartup(t, cache)
|
||||
|
||||
for _, path := range unreadable {
|
||||
assertSkippedEntryWarning(t, logs, path)
|
||||
}
|
||||
|
||||
rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
|
||||
|
||||
if n := countRows(t, cache, rowQuery, string(testVariantKeyOne)); n != 1 {
|
||||
t.Errorf("the variant file with no row got %d rows, want 1", n)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache, rowQuery, string(testVariantKeyTwo)); n != 0 {
|
||||
t.Errorf("the variant with no file kept %d rows, want 0", n)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(untrackedSourceKey))
|
||||
if err == nil {
|
||||
t.Error("the source file with no row is still on disk")
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes() error = %v", err)
|
||||
}
|
||||
|
||||
if usage != int64(len(untrackedVariant)) {
|
||||
t.Errorf("UsageBytes() = %d, want %d, the size of the adopted variant",
|
||||
usage, len(untrackedVariant))
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead puts, in each
|
||||
// storage directory, a temp file whose info the pass cannot read. The
|
||||
// pass must log a warning naming each one's path and the error, as it
|
||||
// does for any other file whose info it cannot read.
|
||||
func TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
unreadable := []string{
|
||||
makeTempFileTooLongToRead(t, cache.variants.baseDir),
|
||||
makeTempFileTooLongToRead(t, cache.srcContent.baseDir),
|
||||
}
|
||||
|
||||
logs := reconcileAsAtStartup(t, cache)
|
||||
|
||||
for _, path := range unreadable {
|
||||
assertSkippedEntryWarning(t, logs, path)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves
|
||||
// reconciliation is not a one-shot startup-only pass: it must also run
|
||||
// on the periodic ticker, so a variant file that lands on disk with no
|
||||
@@ -1030,7 +738,9 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
|
||||
// insert had failed and only the file write had succeeded.
|
||||
untracked := bytes.Repeat([]byte{0x41}, 900)
|
||||
|
||||
err = cache.variants.Write("aabbccdd0099", untracked)
|
||||
_, err = cache.variants.Store(
|
||||
"aabbccdd0099", bytes.NewReader(untracked),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store untracked variant file: %v", err)
|
||||
}
|
||||
@@ -1336,15 +1046,13 @@ func assertOnlyOldestSourceEvicted(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
_, err := os.Stat(cache.srcContent.keyToPath(string(keys[0])))
|
||||
if err == nil {
|
||||
if cache.srcContent.Exists(keys[0]) {
|
||||
t.Errorf("source %s, evicted when the context was cancelled, "+
|
||||
"is still on disk", keys[0])
|
||||
}
|
||||
|
||||
for _, key := range keys[1:] {
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(string(key)))
|
||||
if err != nil {
|
||||
if !cache.srcContent.Exists(key) {
|
||||
t.Errorf("source %s was evicted after the context was cancelled", key)
|
||||
}
|
||||
|
||||
|
||||
@@ -1,185 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Storage constants.
|
||||
const (
|
||||
// StorageDirPerm is the permission mode for storage directories.
|
||||
StorageDirPerm = 0750
|
||||
// StorageFilePerm is the permission mode for storage files.
|
||||
StorageFilePerm = 0600
|
||||
// MinHashLength is the minimum hash length for path splitting.
|
||||
MinHashLength = 4
|
||||
)
|
||||
|
||||
// Storage errors.
|
||||
var (
|
||||
ErrNotFound = errors.New("content not found")
|
||||
)
|
||||
|
||||
// FileStore is a directory of files, each named by a hex key. The file for
|
||||
// a key is at <basedir>/<ab>/<cd>/<key>, where <ab> and <cd> are the key's
|
||||
// first and second pairs of characters. The cache keeps its source images in
|
||||
// one FileStore and its variants in another.
|
||||
type FileStore struct {
|
||||
baseDir string
|
||||
}
|
||||
|
||||
// NewFileStore creates the directory baseDir if it is missing and returns
|
||||
// the FileStore of the files in it.
|
||||
func NewFileStore(baseDir string) (*FileStore, error) {
|
||||
err := os.MkdirAll(baseDir, StorageDirPerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create storage directory: %w", err)
|
||||
}
|
||||
|
||||
return &FileStore{baseDir: baseDir}, nil
|
||||
}
|
||||
|
||||
// Write writes data with writeFileSynced to the file for key, replacing any
|
||||
// file already there.
|
||||
func (s *FileStore) Write(key string, data []byte) error {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
return writeFileSynced(path, data)
|
||||
}
|
||||
|
||||
// Open opens the file for key and returns it with its size in bytes. A
|
||||
// missing file is ErrNotFound.
|
||||
func (s *FileStore) Open(key string) (*os.File, int64, error) {
|
||||
f, err := os.Open(s.keyToPath(key))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
return nil, 0, fmt.Errorf("failed to open content: %w", err)
|
||||
}
|
||||
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
|
||||
return nil, 0, fmt.Errorf("failed to stat content: %w", err)
|
||||
}
|
||||
|
||||
return f, stat.Size(), nil
|
||||
}
|
||||
|
||||
// Size returns the size in bytes of the file for key. A missing file is
|
||||
// ErrNotFound.
|
||||
func (s *FileStore) Size(key string) (int64, error) {
|
||||
stat, err := os.Stat(s.keyToPath(key))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return 0, ErrNotFound
|
||||
}
|
||||
|
||||
return 0, fmt.Errorf("failed to stat content: %w", err)
|
||||
}
|
||||
|
||||
return stat.Size(), nil
|
||||
}
|
||||
|
||||
// Delete removes the file for key. A missing file is not an error.
|
||||
func (s *FileStore) Delete(key string) error {
|
||||
err := os.Remove(s.keyToPath(key))
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to delete content: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeFileSynced writes data to path through a temp file in the same
|
||||
// directory: the temp file is synced to disk, renamed to path, and the
|
||||
// directory synced after, so that neither a concurrent reader nor a
|
||||
// crash finds a partial file under path. When the directory sync fails,
|
||||
// the file is removed from path, so that a failed write leaves no file
|
||||
// there.
|
||||
func writeFileSynced(path string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
|
||||
tmpPath := tmpFile.Name()
|
||||
|
||||
_, err = tmpFile.Write(data)
|
||||
if err != nil {
|
||||
_ = tmpFile.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to write temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmpFile.Sync()
|
||||
if err != nil {
|
||||
_ = tmpFile.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to sync temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmpFile.Close()
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to close temp file: %w", err)
|
||||
}
|
||||
|
||||
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||
}
|
||||
|
||||
err = syncDirectory(dir)
|
||||
if err != nil {
|
||||
_ = os.Remove(path)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// syncDirectory syncs the directory dir to disk, so that a file just
|
||||
// renamed into it is still there after a crash.
|
||||
func syncDirectory(dir string) error {
|
||||
d, err := os.Open(filepath.Clean(dir))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to open directory: %w", err)
|
||||
}
|
||||
|
||||
err = d.Sync()
|
||||
_ = d.Close()
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to sync directory: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
|
||||
func (s *FileStore) keyToPath(key string) string {
|
||||
if len(key) < MinHashLength {
|
||||
return filepath.Join(s.baseDir, key)
|
||||
}
|
||||
|
||||
return filepath.Join(s.baseDir, key[0:2], key[2:4], key)
|
||||
}
|
||||
@@ -1,184 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func TestFileStore_WriteAndOpen(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("hello world")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify file exists at expected path
|
||||
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
t.Errorf("File not at expected path %s: %v", expectedPath, err)
|
||||
}
|
||||
|
||||
// Open and verify size and content
|
||||
r, size, err := storage.Open(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Open() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = r.Close() }()
|
||||
|
||||
if size != int64(len(content)) {
|
||||
t.Errorf("Open() size = %d, want %d", size, len(content))
|
||||
}
|
||||
|
||||
loaded, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadAll() error = %v", err)
|
||||
}
|
||||
|
||||
if !bytes.Equal(loaded, content) {
|
||||
t.Errorf("Open() content = %q, want %q", loaded, content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_OpenNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
_, _, err = storage.Open("nonexistent")
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Open() error = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileStore_Size checks that Size returns the size of a written file,
|
||||
// and ErrNotFound for a key with no file.
|
||||
func TestFileStore_Size(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
storage, err := NewFileStore(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("the file's content")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
size, err := storage.Size(key)
|
||||
if err != nil || size != int64(len(content)) {
|
||||
t.Errorf("Size() = %d, error %v; want %d", size, err, len(content))
|
||||
}
|
||||
|
||||
_, err = storage.Size("nonexistent")
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Size() of a missing file error = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_Delete(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("to be deleted")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
path := storage.keyToPath(key)
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(path)
|
||||
if err != nil {
|
||||
t.Errorf("file missing before Delete(): %v", err)
|
||||
}
|
||||
|
||||
err = storage.Delete(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(path)
|
||||
if err == nil {
|
||||
t.Error("file still on disk after Delete()")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_DeleteNonexistent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
// Should not error
|
||||
err = storage.Delete("nonexistent")
|
||||
if err != nil {
|
||||
t.Errorf("Delete() error = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileStore_KeyToPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewFileStore(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
}
|
||||
|
||||
// Test by writing and verifying the resulting path structure
|
||||
content := []byte("test content for path verification")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
|
||||
err = storage.Write(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
}
|
||||
|
||||
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
t.Errorf("File not at expected path %s: %v", expectedPath, err)
|
||||
}
|
||||
}
|
||||
@@ -67,8 +67,8 @@ func TestReconciliationRemovesDamagedFiles(t *testing.T) {
|
||||
sourceKey := storeEvictionTestSource(t, cache, testHostCDN, testPathCat, content)
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
|
||||
|
||||
sourcePath := cache.srcContent.keyToPath(string(sourceKey))
|
||||
variantPath := cache.variants.keyToPath(string(testVariantKeyOne))
|
||||
sourcePath := cache.srcContent.keyToPath(sourceKey)
|
||||
variantPath := cache.variants.keyToPath(testVariantKeyOne)
|
||||
|
||||
for _, path := range []string{sourcePath, variantPath} {
|
||||
err := os.Truncate(path, size)
|
||||
@@ -82,13 +82,11 @@ func TestReconciliationRemovesDamagedFiles(t *testing.T) {
|
||||
assertWarningNames(t, logs, sourcePath)
|
||||
assertWarningNames(t, logs, variantPath)
|
||||
|
||||
_, err := os.Stat(sourcePath)
|
||||
if err == nil {
|
||||
if cache.srcContent.Exists(sourceKey) {
|
||||
t.Error("the damaged source file is still on disk")
|
||||
}
|
||||
|
||||
_, err = os.Stat(variantPath)
|
||||
if err == nil {
|
||||
if cache.variants.Exists(testVariantKeyOne) {
|
||||
t.Error("the damaged variant file is still on disk")
|
||||
}
|
||||
|
||||
@@ -118,19 +116,16 @@ func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
err := cache.variants.Write(string(testVariantKeyOne), nil)
|
||||
_, err := cache.variants.Store(testVariantKeyOne, bytes.NewReader(nil))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store an empty variant file: %v", err)
|
||||
}
|
||||
|
||||
variantPath := cache.variants.keyToPath(string(testVariantKeyOne))
|
||||
|
||||
logs := reconcileAsAtStartup(t, cache)
|
||||
|
||||
assertWarningNames(t, logs, variantPath)
|
||||
assertWarningNames(t, logs, cache.variants.keyToPath(testVariantKeyOne))
|
||||
|
||||
_, err = os.Stat(variantPath)
|
||||
if err == nil {
|
||||
if cache.variants.Exists(testVariantKeyOne) {
|
||||
t.Error("the empty variant file is still on disk")
|
||||
}
|
||||
|
||||
@@ -142,19 +137,19 @@ func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileStore_WriteReplacesFileOfAnotherSize checks that a write over a
|
||||
// file of another size at the source's path, such as a partial file a crash
|
||||
// left, replaces it with the content.
|
||||
func TestFileStore_WriteReplacesFileOfAnotherSize(t *testing.T) {
|
||||
// TestContentStorage_StoreReplacesFileOfAnotherSize checks that a
|
||||
// store over a file of another size at the source's path, such as a
|
||||
// partial file a crash left, replaces it with the content.
|
||||
func TestContentStorage_StoreReplacesFileOfAnotherSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
storage, err := NewFileStore(t.TempDir())
|
||||
storage, err := NewContentStorage(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStore() error = %v", err)
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("the whole source image")
|
||||
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
path := storage.keyToPath(key)
|
||||
|
||||
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
@@ -167,14 +162,14 @@ func TestFileStore_WriteReplacesFileOfAnotherSize(t *testing.T) {
|
||||
t.Fatalf("writing a partial file: %v", err)
|
||||
}
|
||||
|
||||
err = storage.Write(key, content)
|
||||
_, err = storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Write() error = %v", err)
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
stored, _, err := storage.Open(key)
|
||||
stored, _, err := storage.LoadWithSize(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Open() error = %v", err)
|
||||
t.Fatalf("LoadWithSize() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = stored.Close() }()
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"image/color"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
@@ -205,8 +204,8 @@ func TestSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
content := generateTestJPEG(t, 10, 10, color.RGBA{0, 0, 255, 255})
|
||||
key := storeEvictionTestSource(t, cache, "example.com", "/photo.jpg", content)
|
||||
key := storeEvictionTestSource(t, cache, "example.com", "/photo.jpg",
|
||||
[]byte("source bytes"))
|
||||
anHourAgo := time.Now().Add(-time.Hour)
|
||||
setSourceLastAccessed(t, cache, key, anHourAgo)
|
||||
|
||||
@@ -216,7 +215,7 @@ func TestSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
|
||||
`
|
||||
anHourAgoInDatabase := anHourAgo.UTC().Format(sqliteTimestampFormat)
|
||||
|
||||
reader, _, _, err := cache.Source(t.Context(),
|
||||
reader, _, err := cache.Source(t.Context(),
|
||||
imagerequest.Source{Host: "example.com", Path: "/photo.jpg"})
|
||||
if err != nil {
|
||||
t.Fatalf("Source() error = %v; want the source of %q", err, key)
|
||||
@@ -225,7 +224,7 @@ func TestSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
|
||||
found, err := io.ReadAll(reader)
|
||||
_ = reader.Close()
|
||||
|
||||
if err != nil || !bytes.Equal(found, content) {
|
||||
if err != nil || string(found) != "source bytes" {
|
||||
t.Fatalf("Source() reads %q, %v; want the source of %q", found, err, key)
|
||||
}
|
||||
|
||||
|
||||
@@ -2,17 +2,12 @@ package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image/color"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// TestPutSource_StoresIdenticalBytesOfTwoURLsTwice stores the same bytes
|
||||
@@ -69,7 +64,7 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
content := generateTestJPEG(t, 20, 20, color.RGBA{0x5B, 0x5B, 0x5B, 255})
|
||||
content := bytes.Repeat([]byte{0x5B}, 400)
|
||||
|
||||
evicted := storeEvictionTestSource(t, cache, testHostCDN, "/evicted.jpg", content)
|
||||
storeEvictionTestSource(t, cache, testHostCDN, "/kept.jpg", content)
|
||||
@@ -79,8 +74,7 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
|
||||
t.Fatalf("evictSourceBlob() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(string(evicted)))
|
||||
if err == nil {
|
||||
if cache.srcContent.Exists(evicted) {
|
||||
t.Error("the evicted source's file is still on disk")
|
||||
}
|
||||
|
||||
@@ -90,7 +84,7 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
|
||||
t.Errorf("the evicted source still has %d source_content rows", n)
|
||||
}
|
||||
|
||||
reader, _, _, err := cache.Source(t.Context(),
|
||||
reader, _, err := cache.Source(t.Context(),
|
||||
imagerequest.Source{Host: testHostCDN, Path: "/kept.jpg"})
|
||||
t.Logf("Source() of the kept source: error %v", err)
|
||||
|
||||
@@ -135,104 +129,7 @@ func TestPutSource_FailedRowWriteLeavesNoFile(t *testing.T) {
|
||||
t.Error("PutSource() error = nil, want the failed row write")
|
||||
}
|
||||
|
||||
_, err = os.Stat(cache.srcContent.keyToPath(string(req.Source.Key())))
|
||||
if err == nil {
|
||||
if cache.srcContent.Exists(req.Source.Key()) {
|
||||
t.Error("the source's file is on disk although its row was not written")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSource_FormatFromFirstBytes stores an image in each source format
|
||||
// under one source URL, in turn. After each store, Source must return that
|
||||
// image whole, with its format read from the file.
|
||||
func TestSource_FormatFromFirstBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
processor := imageprocessor.New(imageprocessor.Params{
|
||||
MaxInputBytes: 50 << 20,
|
||||
MaxConcurrentProcessing: 1,
|
||||
})
|
||||
jpegImage := generateTestJPEG(t, 40, 30, color.RGBA{200, 40, 40, 255})
|
||||
src := imagerequest.Source{Host: testHostCDN, Path: testPathCat}
|
||||
|
||||
for _, format := range []imagerequest.Format{
|
||||
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
|
||||
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
|
||||
} {
|
||||
result, err := processor.Process(t.Context(), bytes.NewReader(jpegImage),
|
||||
imagerequest.FormatJPEG, &imagerequest.Transform{Format: format})
|
||||
if err != nil {
|
||||
t.Fatalf("making a %s image: %v", format, err)
|
||||
}
|
||||
|
||||
image, err := io.ReadAll(result.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the %s image: %v", format, err)
|
||||
}
|
||||
|
||||
_, err = cache.PutSource(t.Context(), src, bytes.NewReader(image))
|
||||
if err != nil {
|
||||
t.Fatalf("PutSource(%s) error = %v", format, err)
|
||||
}
|
||||
|
||||
reader, size, got, err := cache.Source(t.Context(), src)
|
||||
if err != nil {
|
||||
t.Fatalf("Source() of the %s image error = %v", format, err)
|
||||
}
|
||||
|
||||
served, err := io.ReadAll(reader)
|
||||
_ = reader.Close()
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("reading the %s source: %v", format, err)
|
||||
}
|
||||
|
||||
t.Logf("%s: %d bytes read as %s", format, size, got)
|
||||
|
||||
if got != format {
|
||||
t.Errorf("Source() of the %s image format = %q, want %q",
|
||||
format, got, format)
|
||||
}
|
||||
|
||||
if size != int64(len(image)) || !bytes.Equal(served, image) {
|
||||
t.Errorf("%s source read as %d bytes, size %d, want the %d bytes stored",
|
||||
format, len(served), size, len(image))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSource_RefusesFileOfNoSourceFormat checks that Source refuses a
|
||||
// cached source whose file's first bytes are not those of a source format,
|
||||
// with an error other than ErrNotFound.
|
||||
func TestSource_RefusesFileOfNoSourceFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
jpegImage := generateTestJPEG(t, 10, 10, color.RGBA{0, 0, 255, 255})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
content []byte
|
||||
wantErr error
|
||||
}{
|
||||
{"text", []byte("this is text, not an image"), magic.ErrUnknownFormat},
|
||||
{"SVG", []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
|
||||
magic.ErrUnknownFormat},
|
||||
{"JPEG cut to 10 bytes", jpegImage[:10], io.EOF},
|
||||
{"empty file", nil, io.EOF},
|
||||
}
|
||||
|
||||
for i, tt := range tests {
|
||||
path := fmt.Sprintf("/refused%d.jpg", i)
|
||||
storeEvictionTestSource(t, cache, testHostCDN, path, tt.content)
|
||||
|
||||
reader, _, format, err := cache.Source(t.Context(),
|
||||
imagerequest.Source{Host: testHostCDN, Path: path})
|
||||
t.Logf("%s: Source() error = %v", tt.name, err)
|
||||
|
||||
if reader != nil || format != "" || !errors.Is(err, tt.wantErr) {
|
||||
t.Errorf("%s: Source() = reader %v, format %q, error %v; "+
|
||||
"want no reader and %v", tt.name, reader, format, err, tt.wantErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
// Storage constants.
|
||||
const (
|
||||
// StorageDirPerm is the permission mode for storage directories.
|
||||
StorageDirPerm = 0750
|
||||
// StorageFilePerm is the permission mode for storage files.
|
||||
StorageFilePerm = 0600
|
||||
// MinHashLength is the minimum hash length for path splitting.
|
||||
MinHashLength = 4
|
||||
)
|
||||
|
||||
// Storage errors.
|
||||
var (
|
||||
ErrNotFound = errors.New("content not found")
|
||||
)
|
||||
|
||||
// ContentStorage handles source image file storage, one file per source
|
||||
// URL. Files are stored at: <basedir>/<ab>/<cd>/<source key>
|
||||
type ContentStorage struct {
|
||||
baseDir string
|
||||
}
|
||||
|
||||
// NewContentStorage creates a new content storage at the given base directory.
|
||||
func NewContentStorage(baseDir string) (*ContentStorage, error) {
|
||||
err := os.MkdirAll(baseDir, StorageDirPerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create storage directory: %w", err)
|
||||
}
|
||||
|
||||
return &ContentStorage{baseDir: baseDir}, nil
|
||||
}
|
||||
|
||||
// Store writes data with writeFileSynced to the path derived from key,
|
||||
// replacing any file already there, and returns its size.
|
||||
func (s *ContentStorage) Store(
|
||||
key imagerequest.SourceKey, data []byte,
|
||||
) (int64, error) {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
err = writeFileSynced(path, data)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return int64(len(data)), nil
|
||||
}
|
||||
|
||||
// LoadWithSize returns a reader and file size for the content with the given key.
|
||||
func (s *ContentStorage) LoadWithSize(
|
||||
key imagerequest.SourceKey,
|
||||
) (io.ReadCloser, int64, error) {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from source key
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
return nil, 0, fmt.Errorf("failed to open content: %w", err)
|
||||
}
|
||||
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
|
||||
return nil, 0, fmt.Errorf("failed to stat content: %w", err)
|
||||
}
|
||||
|
||||
return f, stat.Size(), nil
|
||||
}
|
||||
|
||||
// Delete removes content with the given key.
|
||||
func (s *ContentStorage) Delete(key imagerequest.SourceKey) error {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.Remove(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to delete content: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Exists checks if content with the given key exists.
|
||||
func (s *ContentStorage) Exists(key imagerequest.SourceKey) bool {
|
||||
path := s.keyToPath(key)
|
||||
_, err := os.Stat(path)
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// writeFileSynced writes data to path through a temp file in the same
|
||||
// directory: the temp file is synced to disk, renamed to path, and the
|
||||
// directory synced after, so that neither a concurrent reader nor a
|
||||
// crash finds a partial file under path. When the directory sync fails,
|
||||
// the file is removed from path, so that a failed write leaves no file
|
||||
// there.
|
||||
func writeFileSynced(path string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
|
||||
tmpPath := tmpFile.Name()
|
||||
|
||||
_, err = tmpFile.Write(data)
|
||||
if err != nil {
|
||||
_ = tmpFile.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to write temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmpFile.Sync()
|
||||
if err != nil {
|
||||
_ = tmpFile.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to sync temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmpFile.Close()
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to close temp file: %w", err)
|
||||
}
|
||||
|
||||
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||
}
|
||||
|
||||
err = syncDirectory(dir)
|
||||
if err != nil {
|
||||
_ = os.Remove(path)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// syncDirectory syncs the directory dir to disk, so that a file just
|
||||
// renamed into it is still there after a crash.
|
||||
func syncDirectory(dir string) error {
|
||||
d, err := os.Open(filepath.Clean(dir))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to open directory: %w", err)
|
||||
}
|
||||
|
||||
err = d.Sync()
|
||||
_ = d.Close()
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to sync directory: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
|
||||
func (s *ContentStorage) keyToPath(key imagerequest.SourceKey) string {
|
||||
k := string(key)
|
||||
if len(k) < MinHashLength {
|
||||
return filepath.Clean(filepath.Join(s.baseDir, k))
|
||||
}
|
||||
|
||||
return filepath.Clean(filepath.Join(s.baseDir, k[0:2], k[2:4], k))
|
||||
}
|
||||
|
||||
// VariantStorage handles key-based file storage for processed image variants.
|
||||
// Files are stored at: <basedir>/<ab>/<cd>/<cache_key>
|
||||
type VariantStorage struct {
|
||||
baseDir string
|
||||
}
|
||||
|
||||
// errNotAnOutputFormat is returned for a variant file whose first bytes are
|
||||
// those of a format pixa reads but never writes, such as SVG.
|
||||
var errNotAnOutputFormat = errors.New("variant file is not in an output format")
|
||||
|
||||
// NewVariantStorage creates a new variant storage at the given base directory.
|
||||
func NewVariantStorage(baseDir string) (*VariantStorage, error) {
|
||||
err := os.MkdirAll(baseDir, StorageDirPerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create variant storage directory: %w", err)
|
||||
}
|
||||
|
||||
return &VariantStorage{baseDir: baseDir}, nil
|
||||
}
|
||||
|
||||
// Store writes content to storage at the given key with writeFileSynced.
|
||||
func (s *VariantStorage) Store(
|
||||
key imagerequest.VariantKey, r io.Reader,
|
||||
) (int64, error) {
|
||||
data, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to read content: %w", err)
|
||||
}
|
||||
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
err = writeFileSynced(path, data)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return int64(len(data)), nil
|
||||
}
|
||||
|
||||
// LoadWithContentType opens the content at the given key and returns a
|
||||
// reader, the file size and the content type, which is read from the
|
||||
// file's first bytes. A missing file is ErrNotFound. A file whose first
|
||||
// bytes are not those of an output format, an empty one included, is
|
||||
// closed again and the reason returned.
|
||||
func (s *VariantStorage) LoadWithContentType(
|
||||
key imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from cache key
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
|
||||
return nil, 0, "", fmt.Errorf("failed to open content: %w", err)
|
||||
}
|
||||
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
|
||||
return nil, 0, "", fmt.Errorf("failed to stat content: %w", err)
|
||||
}
|
||||
|
||||
contentType, err := outputContentType(f)
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
|
||||
return nil, 0, "", err
|
||||
}
|
||||
|
||||
return f, stat.Size(), contentType, nil
|
||||
}
|
||||
|
||||
// outputContentType reads the first bytes of content, without moving its
|
||||
// offset, and returns the content type of the output format they show.
|
||||
func outputContentType(content io.ReaderAt) (string, error) {
|
||||
firstBytes := make([]byte, magic.MinMagicBytes)
|
||||
|
||||
_, err := content.ReadAt(firstBytes, 0)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read the first bytes: %w", err)
|
||||
}
|
||||
|
||||
detected, err := magic.DetectFormat(firstBytes)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
_, isOutputFormat := magic.MIMEToImageFormat(string(detected))
|
||||
if !isOutputFormat {
|
||||
return "", fmt.Errorf("%w: %s", errNotAnOutputFormat, detected)
|
||||
}
|
||||
|
||||
return string(detected), nil
|
||||
}
|
||||
|
||||
// Exists checks if content exists at the given key.
|
||||
func (s *VariantStorage) Exists(key imagerequest.VariantKey) bool {
|
||||
path := s.keyToPath(key)
|
||||
_, err := os.Stat(path)
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// Delete removes the content at the given key. A missing file is not an
|
||||
// error.
|
||||
func (s *VariantStorage) Delete(key imagerequest.VariantKey) error {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.Remove(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to delete content: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
|
||||
func (s *VariantStorage) keyToPath(key imagerequest.VariantKey) string {
|
||||
k := string(key)
|
||||
if len(k) < MinHashLength {
|
||||
return filepath.Join(s.baseDir, k)
|
||||
}
|
||||
|
||||
return filepath.Join(s.baseDir, k[0:2], k[2:4], k)
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func TestContentStorage_StoreAndLoadWithSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewContentStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("hello world")
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
|
||||
size, err := storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
if size != int64(len(content)) {
|
||||
t.Errorf("Store() size = %d, want %d", size, len(content))
|
||||
}
|
||||
|
||||
// Verify file exists at expected path
|
||||
keyStr := string(key)
|
||||
|
||||
expectedPath := filepath.Join(tmpDir, keyStr[0:2], keyStr[2:4], keyStr)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
t.Errorf("File not at expected path %s: %v", expectedPath, err)
|
||||
}
|
||||
|
||||
// Load and verify content
|
||||
r, _, err := storage.LoadWithSize(key)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadWithSize() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = r.Close() }()
|
||||
|
||||
loaded, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadAll() error = %v", err)
|
||||
}
|
||||
|
||||
if !bytes.Equal(loaded, content) {
|
||||
t.Errorf("LoadWithSize() content = %q, want %q", loaded, content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContentStorage_LoadWithSizeNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewContentStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
_, _, err = storage.LoadWithSize(imagerequest.SourceKey("nonexistent"))
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("LoadWithSize() error = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContentStorage_Delete(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewContentStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("to be deleted")
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
|
||||
_, err = storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
if !storage.Exists(key) {
|
||||
t.Error("Exists() = false, want true")
|
||||
}
|
||||
|
||||
err = storage.Delete(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete() error = %v", err)
|
||||
}
|
||||
|
||||
if storage.Exists(key) {
|
||||
t.Error("Exists() = true after delete, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestContentStorage_DeleteNonexistent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewContentStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
// Should not error
|
||||
err = storage.Delete(imagerequest.SourceKey("nonexistent"))
|
||||
if err != nil {
|
||||
t.Errorf("Delete() error = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContentStorage_KeyToPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewContentStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
// Test by storing and verifying the resulting path structure
|
||||
content := []byte("test content for path verification")
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
|
||||
_, err = storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
keyStr := string(key)
|
||||
|
||||
expectedPath := filepath.Join(tmpDir, keyStr[0:2], keyStr[2:4], keyStr)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
t.Errorf("File not at expected path %s: %v", expectedPath, err)
|
||||
}
|
||||
}
|
||||
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 281 B |
@@ -1,17 +1,12 @@
|
||||
package magic
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
import "testing"
|
||||
|
||||
// testMIMEJXL is the MIME type of JPEG XL.
|
||||
const testMIMEJXL = "image/jxl"
|
||||
|
||||
// TestDetectFormatJPEGXL verifies that both JPEG XL signatures, that of a
|
||||
// bare codestream and that of the container, are detected as JPEG XL and
|
||||
// match the MIME type image/jxl.
|
||||
// bare codestream and that of the container, are detected as image/jxl.
|
||||
func TestDetectFormatJPEGXL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -30,15 +25,25 @@ func TestDetectFormatJPEGXL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := DetectFormat(tt.data)
|
||||
if err != nil || got != imagerequest.FormatJXL {
|
||||
t.Errorf("DetectFormat() = %q, %v, want %q",
|
||||
got, err, imagerequest.FormatJXL)
|
||||
if err != nil || got != MIMETypeJXL {
|
||||
t.Errorf("DetectFormat() = %q, %v, want %q", got, err, testMIMEJXL)
|
||||
}
|
||||
|
||||
_, err = ValidateMagicBytes(tt.data, testMIMEJXL)
|
||||
err = ValidateMagicBytes(tt.data, testMIMEJXL)
|
||||
if err != nil {
|
||||
t.Errorf("ValidateMagicBytes(%q) error = %v", testMIMEJXL, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestJPEGXLMIMEType verifies that image/jxl maps to the format jxl.
|
||||
func TestJPEGXLMIMEType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
format, ok := MIMEToImageFormat(testMIMEJXL)
|
||||
if format != FormatJXL || !ok {
|
||||
t.Errorf("MIMEToImageFormat(%q) = %q, %v, want %q, true",
|
||||
testMIMEJXL, format, ok, FormatJXL)
|
||||
}
|
||||
}
|
||||
|
||||
+98
-21
@@ -6,8 +6,6 @@ import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Magic byte errors.
|
||||
@@ -17,6 +15,34 @@ var (
|
||||
ErrNotEnoughData = errors.New("not enough data to detect format")
|
||||
)
|
||||
|
||||
// MIMEType represents a supported MIME type for input images.
|
||||
type MIMEType string
|
||||
|
||||
// Supported input MIME types.
|
||||
const (
|
||||
MIMETypeJPEG = MIMEType("image/jpeg")
|
||||
MIMETypePNG = MIMEType("image/png")
|
||||
MIMETypeWebP = MIMEType("image/webp")
|
||||
MIMETypeGIF = MIMEType("image/gif")
|
||||
MIMETypeAVIF = MIMEType("image/avif")
|
||||
MIMETypeJXL = MIMEType("image/jxl")
|
||||
MIMETypeSVG = MIMEType("image/svg+xml")
|
||||
)
|
||||
|
||||
// ImageFormat represents supported output image formats.
|
||||
// This mirrors the type in imgcache to avoid circular imports.
|
||||
type ImageFormat string
|
||||
|
||||
// Supported image output formats.
|
||||
const (
|
||||
FormatJPEG ImageFormat = "jpeg"
|
||||
FormatPNG ImageFormat = "png"
|
||||
FormatWebP ImageFormat = "webp"
|
||||
FormatAVIF ImageFormat = "avif"
|
||||
FormatJXL ImageFormat = "jxl"
|
||||
FormatGIF ImageFormat = "gif"
|
||||
)
|
||||
|
||||
// MinMagicBytes is the minimum number of bytes needed to detect format.
|
||||
const MinMagicBytes = 12
|
||||
|
||||
@@ -52,32 +78,32 @@ var (
|
||||
avisBrand = []byte{0x61, 0x76, 0x69, 0x73} // "avis" (AVIF sequence)
|
||||
)
|
||||
|
||||
// DetectFormat detects the image format from magic bytes. Data in none of
|
||||
// the formats pixa reads, SVG included, is ErrUnknownFormat.
|
||||
func DetectFormat(data []byte) (imagerequest.Format, error) {
|
||||
// DetectFormat detects the image format from magic bytes.
|
||||
// Returns the MIME type and nil error on success.
|
||||
func DetectFormat(data []byte) (MIMEType, error) {
|
||||
if len(data) < MinMagicBytes {
|
||||
return "", ErrNotEnoughData
|
||||
}
|
||||
|
||||
// Check JPEG (FFD8FF)
|
||||
if bytes.HasPrefix(data, magicJPEG) {
|
||||
return imagerequest.FormatJPEG, nil
|
||||
return MIMETypeJPEG, nil
|
||||
}
|
||||
|
||||
// Check PNG (89504E47 0D0A1A0A)
|
||||
if bytes.HasPrefix(data, magicPNG) {
|
||||
return imagerequest.FormatPNG, nil
|
||||
return MIMETypePNG, nil
|
||||
}
|
||||
|
||||
// Check GIF (GIF87a or GIF89a)
|
||||
if bytes.HasPrefix(data, magicGIF) {
|
||||
return imagerequest.FormatGIF, nil
|
||||
return MIMETypeGIF, nil
|
||||
}
|
||||
|
||||
// Check WebP (RIFF....WEBP)
|
||||
if bytes.HasPrefix(data, magicWebP) && len(data) >= 12 {
|
||||
if bytes.Equal(data[8:12], webpIdent) {
|
||||
return imagerequest.FormatWebP, nil
|
||||
return MIMETypeWebP, nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,39 +112,66 @@ func DetectFormat(data []byte) (imagerequest.Format, error) {
|
||||
if len(data) >= 12 && bytes.Equal(data[4:8], magicFtyp) {
|
||||
brand := data[8:12]
|
||||
if bytes.Equal(brand, avifBrand) || bytes.Equal(brand, avisBrand) {
|
||||
return imagerequest.FormatAVIF, nil
|
||||
return MIMETypeAVIF, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Check JPEG XL (FF0A, or the 12-byte container signature)
|
||||
if bytes.HasPrefix(data, magicJXLCodestream) ||
|
||||
bytes.HasPrefix(data, magicJXLContainer) {
|
||||
return imagerequest.FormatJXL, nil
|
||||
return MIMETypeJXL, nil
|
||||
}
|
||||
|
||||
// Check SVG - look for XML declaration or SVG tag
|
||||
if detectSVG(data) {
|
||||
return MIMETypeSVG, nil
|
||||
}
|
||||
|
||||
return "", ErrUnknownFormat
|
||||
}
|
||||
|
||||
// ValidateMagicBytes detects the format of data from its magic bytes, and
|
||||
// returns it when it is the format the declared MIME type names. Data in
|
||||
// another format is ErrMagicByteMismatch.
|
||||
func ValidateMagicBytes(
|
||||
data []byte, declaredType string,
|
||||
) (imagerequest.Format, error) {
|
||||
// detectSVG checks if data appears to be SVG content.
|
||||
func detectSVG(data []byte) bool {
|
||||
// Skip BOM if present
|
||||
content := skipBOM(data)
|
||||
|
||||
// Convert to string for easier pattern matching
|
||||
s := strings.ToLower(string(content))
|
||||
|
||||
// Skip leading whitespace
|
||||
s = strings.TrimSpace(s)
|
||||
|
||||
// Check for XML declaration or SVG element
|
||||
return strings.HasPrefix(s, "<?xml") ||
|
||||
strings.HasPrefix(s, "<svg") ||
|
||||
strings.HasPrefix(s, "<!doctype svg")
|
||||
}
|
||||
|
||||
// skipBOM removes UTF-8 BOM if present.
|
||||
func skipBOM(data []byte) []byte {
|
||||
if len(data) >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF {
|
||||
return data[3:]
|
||||
}
|
||||
|
||||
return data
|
||||
}
|
||||
|
||||
// ValidateMagicBytes validates that the content matches the declared MIME type.
|
||||
func ValidateMagicBytes(data []byte, declaredType string) error {
|
||||
detected, err := DetectFormat(data)
|
||||
if err != nil {
|
||||
return "", err
|
||||
return err
|
||||
}
|
||||
|
||||
// Normalize the declared type (remove parameters like charset)
|
||||
normalizedDeclared := normalizeMIMEType(declaredType)
|
||||
|
||||
// Check if they match
|
||||
if detected.MIMEType() != normalizedDeclared {
|
||||
return "", ErrMagicByteMismatch
|
||||
if string(detected) != normalizedDeclared {
|
||||
return ErrMagicByteMismatch
|
||||
}
|
||||
|
||||
return detected, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// normalizeMIMEType extracts just the media type, removing parameters.
|
||||
@@ -130,3 +183,27 @@ func normalizeMIMEType(mimeType string) string {
|
||||
|
||||
return strings.TrimSpace(strings.ToLower(mimeType))
|
||||
}
|
||||
|
||||
// MIMEToImageFormat converts a MIME type to an ImageFormat.
|
||||
func MIMEToImageFormat(mimeType string) (ImageFormat, bool) {
|
||||
normalized := normalizeMIMEType(mimeType)
|
||||
switch MIMEType(normalized) {
|
||||
case MIMETypeJPEG:
|
||||
return FormatJPEG, true
|
||||
case MIMETypePNG:
|
||||
return FormatPNG, true
|
||||
case MIMETypeWebP:
|
||||
return FormatWebP, true
|
||||
case MIMETypeGIF:
|
||||
return FormatGIF, true
|
||||
case MIMETypeAVIF:
|
||||
return FormatAVIF, true
|
||||
case MIMETypeJXL:
|
||||
return FormatJXL, true
|
||||
case MIMETypeSVG:
|
||||
// SVG has no corresponding output format.
|
||||
return "", false
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
package magic
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Shared test fixture strings.
|
||||
@@ -13,6 +12,9 @@ const (
|
||||
testMIMEJPEG = "image/jpeg"
|
||||
testMIMEJPEGParams = "image/jpeg; charset=utf-8"
|
||||
testMIMEPNG = "image/png"
|
||||
testMIMEWebP = "image/webp"
|
||||
testMIMEGIF = "image/gif"
|
||||
testMIMEAVIF = "image/avif"
|
||||
)
|
||||
|
||||
// pad appends zero bytes so data is comfortably above MinMagicBytes.
|
||||
@@ -34,22 +36,37 @@ func TestDetectFormat(t *testing.T) {
|
||||
avis := pad(0x00, 0x00, 0x00, 0x1C, 0x66, 0x74, 0x79, 0x70, 0x61, 0x76, 0x69, 0x73)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
data []byte
|
||||
wantFormat imagerequest.Format
|
||||
wantErr error
|
||||
name string
|
||||
data []byte
|
||||
wantMIME MIMEType
|
||||
wantErr error
|
||||
}{
|
||||
{name: "JPEG", data: jpeg, wantFormat: imagerequest.FormatJPEG},
|
||||
{name: "PNG", data: png, wantFormat: imagerequest.FormatPNG},
|
||||
{name: "GIF87a", data: gif87a, wantFormat: imagerequest.FormatGIF},
|
||||
{name: "GIF89a", data: gif89a, wantFormat: imagerequest.FormatGIF},
|
||||
{name: "WebP", data: webp, wantFormat: imagerequest.FormatWebP},
|
||||
{name: "AVIF", data: avif, wantFormat: imagerequest.FormatAVIF},
|
||||
{name: "AVIF sequence", data: avis, wantFormat: imagerequest.FormatAVIF},
|
||||
{name: "JPEG", data: jpeg, wantMIME: MIMETypeJPEG},
|
||||
{name: "PNG", data: png, wantMIME: MIMETypePNG},
|
||||
{name: "GIF87a", data: gif87a, wantMIME: MIMETypeGIF},
|
||||
{name: "GIF89a", data: gif89a, wantMIME: MIMETypeGIF},
|
||||
{name: "WebP", data: webp, wantMIME: MIMETypeWebP},
|
||||
{name: "AVIF", data: avif, wantMIME: MIMETypeAVIF},
|
||||
{name: "AVIF sequence", data: avis, wantMIME: MIMETypeAVIF},
|
||||
{
|
||||
name: "SVG",
|
||||
data: []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
|
||||
wantErr: ErrUnknownFormat,
|
||||
name: "SVG with XML declaration",
|
||||
data: []byte(`<?xml version="1.0"?><svg></svg>`),
|
||||
wantMIME: MIMETypeSVG,
|
||||
},
|
||||
{
|
||||
name: "SVG without declaration",
|
||||
data: []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
|
||||
wantMIME: MIMETypeSVG,
|
||||
},
|
||||
{
|
||||
name: "SVG with whitespace",
|
||||
data: []byte(` <?xml version="1.0"?><svg></svg>`),
|
||||
wantMIME: MIMETypeSVG,
|
||||
},
|
||||
{
|
||||
name: "SVG with BOM",
|
||||
data: append([]byte{0xEF, 0xBB, 0xBF}, []byte(`<svg></svg>`)...),
|
||||
wantMIME: MIMETypeSVG,
|
||||
},
|
||||
{
|
||||
name: "unknown format",
|
||||
@@ -71,8 +88,8 @@ func TestDetectFormat(t *testing.T) {
|
||||
return
|
||||
}
|
||||
|
||||
if got != tt.wantFormat {
|
||||
t.Errorf("DetectFormat() = %v, want %v", got, tt.wantFormat)
|
||||
if got != tt.wantMIME {
|
||||
t.Errorf("DetectFormat() = %v, want %v", got, tt.wantMIME)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -88,28 +105,24 @@ func TestValidateMagicBytes(t *testing.T) {
|
||||
name string
|
||||
data []byte
|
||||
declaredType string
|
||||
wantFormat imagerequest.Format
|
||||
wantErr error
|
||||
}{
|
||||
{
|
||||
name: "matching JPEG",
|
||||
data: jpegData,
|
||||
declaredType: testMIMEJPEG,
|
||||
wantFormat: imagerequest.FormatJPEG,
|
||||
wantErr: nil,
|
||||
},
|
||||
{
|
||||
name: "matching JPEG with params",
|
||||
data: jpegData,
|
||||
declaredType: testMIMEJPEGParams,
|
||||
wantFormat: imagerequest.FormatJPEG,
|
||||
wantErr: nil,
|
||||
},
|
||||
{
|
||||
name: "matching PNG",
|
||||
data: pngData,
|
||||
declaredType: testMIMEPNG,
|
||||
wantFormat: imagerequest.FormatPNG,
|
||||
wantErr: nil,
|
||||
},
|
||||
{
|
||||
@@ -130,14 +143,45 @@ func TestValidateMagicBytes(t *testing.T) {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := ValidateMagicBytes(tt.data, tt.declaredType)
|
||||
err := ValidateMagicBytes(tt.data, tt.declaredType)
|
||||
|
||||
if !errors.Is(err, tt.wantErr) {
|
||||
t.Errorf("ValidateMagicBytes() error = %v, wantErr %v", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMIMEToImageFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
mimeType string
|
||||
wantFormat ImageFormat
|
||||
wantOk bool
|
||||
}{
|
||||
{testMIMEJPEG, FormatJPEG, true},
|
||||
{testMIMEPNG, FormatPNG, true},
|
||||
{testMIMEWebP, FormatWebP, true},
|
||||
{testMIMEGIF, FormatGIF, true},
|
||||
{testMIMEAVIF, FormatAVIF, true},
|
||||
{"image/svg+xml", "", false}, // SVG doesn't convert to ImageFormat
|
||||
{"image/tiff", "", false},
|
||||
{"text/plain", "", false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.mimeType, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, ok := MIMEToImageFormat(tt.mimeType)
|
||||
|
||||
if ok != tt.wantOk {
|
||||
t.Errorf("MIMEToImageFormat(%q) ok = %v, want %v", tt.mimeType, ok, tt.wantOk)
|
||||
}
|
||||
|
||||
if got != tt.wantFormat {
|
||||
t.Errorf("ValidateMagicBytes() = %q, want %q", got, tt.wantFormat)
|
||||
t.Errorf("MIMEToImageFormat(%q) = %v, want %v", tt.mimeType, got, tt.wantFormat)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -170,6 +214,109 @@ func TestNormalizeMIMEType(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectSVG(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
data string
|
||||
want bool
|
||||
}{
|
||||
{"xml declaration", `<?xml version="1.0"?><svg></svg>`, true},
|
||||
{"svg element", `<svg xmlns="http://www.w3.org/2000/svg"></svg>`, true},
|
||||
{
|
||||
"doctype",
|
||||
`<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" ` +
|
||||
`"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">`,
|
||||
true,
|
||||
},
|
||||
{"with whitespace", `
|
||||
<?xml version="1.0"?><svg></svg>`, true},
|
||||
{"uppercase", `<SVG></SVG>`, true},
|
||||
{"not svg", `<html></html>`, false},
|
||||
{"random text", `hello world`, false},
|
||||
{testNameEmpty, ``, false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := detectSVG([]byte(tt.data))
|
||||
|
||||
if got != tt.want {
|
||||
t.Errorf("detectSVG(%q) = %v, want %v", tt.data, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSkipBOM(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
data []byte
|
||||
want []byte
|
||||
}{
|
||||
{"with BOM", []byte{0xEF, 0xBB, 0xBF, 'h', 'e', 'l', 'l', 'o'}, []byte("hello")},
|
||||
{"without BOM", []byte("hello"), []byte("hello")},
|
||||
{testNameEmpty, []byte{}, []byte{}},
|
||||
{"only BOM", []byte{0xEF, 0xBB, 0xBF}, []byte{}},
|
||||
{"partial BOM", []byte{0xEF, 0xBB}, []byte{0xEF, 0xBB}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := skipBOM(tt.data)
|
||||
|
||||
if !bytes.Equal(got, tt.want) {
|
||||
t.Errorf("skipBOM() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRealWorldSVGPatterns(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Test various real-world SVG patterns
|
||||
svgPatterns := []string{
|
||||
`<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
|
||||
<circle cx="50" cy="50" r="40"/>
|
||||
</svg>`,
|
||||
`<svg viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M12 2L2 7l10 5 10-5-10-5z"/>
|
||||
</svg>`,
|
||||
`<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" ` +
|
||||
`"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">` + `
|
||||
<svg xmlns="http://www.w3.org/2000/svg">
|
||||
</svg>`,
|
||||
}
|
||||
|
||||
for i, pattern := range svgPatterns {
|
||||
data := []byte(pattern)
|
||||
if len(data) < MinMagicBytes {
|
||||
// Pad short SVGs for detection
|
||||
data = append(data, make([]byte, MinMagicBytes-len(data))...)
|
||||
}
|
||||
|
||||
got, err := DetectFormat(data)
|
||||
if err != nil {
|
||||
t.Errorf("Pattern %d: DetectFormat() error = %v", i, err)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if got != MIMETypeSVG {
|
||||
t.Errorf("Pattern %d: DetectFormat() = %v, want %v", i, got, MIMETypeSVG)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectFormatRIFFNotWebP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user