Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
47a2c2360a | ||
|
|
84a86274cd |
@@ -323,34 +323,25 @@ first `/`, the size and format after the last `/`, and between them the source
|
||||
path up to the first `?` and the source query after it. So the `?` before the
|
||||
query, any `?` in the query, and every `%` in the source path or query must be
|
||||
percent-encoded, as `%3F` and `%25`. Any other character may be written as
|
||||
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. Neither
|
||||
the source path nor the source query may hold a `#`, and an image URL that
|
||||
writes one as `%23` is refused with 400. For example,
|
||||
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. For
|
||||
example,
|
||||
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
|
||||
800x600 WebP is
|
||||
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
|
||||
signed as shown under Signature Specification.
|
||||
|
||||
The source path is the path with its percent-escapes decoded, and pixa fetches
|
||||
it percent-encoded again: every character other than letters, digits, `-`, `.`,
|
||||
`_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@` is sent as a
|
||||
percent-escape. So `my%20cat.jpg` in the image URL is the source path
|
||||
`/my cat.jpg`, fetched as `/my%20cat.jpg`; `my%2520cat.jpg` is the source path
|
||||
`/my%20cat.jpg`, fetched as `/my%2520cat.jpg`; and `100%25.png` is fetched as
|
||||
`/100%25.png`. The source query is the source URL's query as that URL writes it,
|
||||
its percent-escapes included, and is sent as it is, except that the characters a
|
||||
browser percent-encodes in a query (a space, `"`, `'`, `<`, `>`, control
|
||||
characters and characters outside ASCII) and a `%` that does not begin a
|
||||
percent-escape are percent-encoded. pixa signs and caches the query in that
|
||||
form, so `img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are
|
||||
the same source query, `name=my%20cat`.
|
||||
|
||||
A size with one side `0` has that side scaled to keep the source's proportions,
|
||||
and is refused with 400 when that side would come out above 8192, the most
|
||||
either side may be; so is a size with `fit=outside`, which scales the image to
|
||||
cover the size and so past it on one side, when that side would come out
|
||||
above 8192.
|
||||
|
||||
An AVIF image of more than 16,777,216 pixels, the number 4096x4096 and 8192x2048
|
||||
have, is refused with 400 before anything is decoded, as its encode could take
|
||||
more than 40 seconds (see below). That includes `orig` of an AVIF source, and
|
||||
`auto` when it chooses AVIF, so a client whose `Accept` names `image/avif` but
|
||||
not `image/jxl` is refused such a size.
|
||||
|
||||
The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
|
||||
`Content-Type` must name its format, and the image's first bytes must match it.
|
||||
An upstream answering with any other `Content-Type`, SVG included, is refused
|
||||
@@ -400,14 +391,18 @@ transcode that stops this way is answered 503 with the error
|
||||
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
|
||||
encode keeps its slot until it finishes, which can be after
|
||||
`downstream_timeout`. The limit of 8192 on each side of the source and of the
|
||||
output bounds how long that takes. With one libvips thread, as pixad runs it,
|
||||
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
|
||||
16 as GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the slowest of 8 or 16
|
||||
bits per sample, with or without an alpha channel. WebP, AVIF and JPEG XL are
|
||||
that slow only with an alpha channel: 8-bit RGB takes about 44, 53 and 28
|
||||
seconds. Decoding such a source adds up to about 11 seconds. A WebP, AVIF or
|
||||
JPEG XL encode can therefore hold its slot past the default `downstream_timeout`
|
||||
of 60 seconds.
|
||||
output, and for AVIF the limit of 16,777,216 pixels, bound how long that takes.
|
||||
With one libvips thread, as pixad runs it, encoding 8192x8192 random pixels
|
||||
takes up to about 1 second as JPEG, 15 as PNG and 16 as GIF, the slowest of 8 or
|
||||
16 bits per sample, with or without an alpha channel. The WebP and JPEG XL
|
||||
encoders run below libvips' default effort, and AVIF at the lowest effort govips
|
||||
can set, so that no encode takes more than 40 seconds of the default
|
||||
`downstream_timeout` of 60: at any quality and 8 or 16 bits per sample,
|
||||
8192x8192 random pixels with an alpha channel take up to about 23 seconds as
|
||||
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
|
||||
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
|
||||
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
|
||||
up to about 11 seconds.
|
||||
|
||||
The login form (`POST /`) is limited to 5 attempts per minute per client
|
||||
address, counting an IPv6 client by its /64; an attempt over the limit is
|
||||
@@ -459,13 +454,8 @@ nor change what it asks for.
|
||||
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
||||
height both empty or `0` keep the original size; if only one of them is empty
|
||||
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
||||
scaled side would come out above 8192 answers 400, and so does a source URL
|
||||
whose path holds a `..` segment. So does a source URL that the image URL
|
||||
would fetch written another way, and the page then shows the URL pixa would
|
||||
fetch: one with a user name and password, which pixa does not send, or one
|
||||
whose path holds `%2F`, fetched as `/`, or `(`, fetched as `%28`, as pixa
|
||||
keeps the source path decoded and fetches it percent-encoded again (see
|
||||
Routes).
|
||||
scaled side would come out above 8192 answers 400, and so does an AVIF URL of
|
||||
more than 16,777,216 pixels (see Routes).
|
||||
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
||||
it expires. `<host>` is the host the page was opened on, and the URL starts
|
||||
with `http` instead while `debug` is on. The name after the token is ignored
|
||||
@@ -473,11 +463,9 @@ nor change what it asks for.
|
||||
`jxl` for a form with no format.
|
||||
|
||||
The token holds the source's host, path and query and the size, format, quality,
|
||||
fit and expiry, encrypted with a key derived from `signing_key`. The host is
|
||||
kept in lower case and an empty path as `/`, as for `/v1/image/`, so
|
||||
`https://CDN.Example.com` and `https://cdn.example.com/` are one source image,
|
||||
fetched and cached once. The source URL's scheme is not kept: the image is
|
||||
fetched like any other (see Routes), and the blocked networks still apply.
|
||||
fit and expiry, encrypted with a key derived from `signing_key`. The source
|
||||
URL's scheme is not kept: the image is fetched like any other (see Routes), and
|
||||
the blocked networks still apply.
|
||||
|
||||
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
||||
The expiry is fixed in the token when the URL is made and cannot be changed or
|
||||
@@ -529,20 +517,13 @@ Where:
|
||||
- `host` — source host as the URL writes it up to the first `/`, in lower case,
|
||||
keeping an IPv6 address's brackets and any `:` and port after it (e.g.
|
||||
`cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
|
||||
- `path` — source path, with its percent-escapes decoded as under Routes (e.g.
|
||||
`/photos/cat.jpg`, or `/photos/my cat.jpg` for `/photos/my%20cat.jpg`), with
|
||||
each `%` written as `%25` and each `:` as `%3A`, so `/photos/a:b.jpg` is
|
||||
signed as `/photos/a%3Ab.jpg`
|
||||
- `query` — source query string as under Routes, empty string if none, with `%`
|
||||
and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no
|
||||
query and path `/a` with query `b:` are signed differently
|
||||
- `width` — requested width in pixels, or `0`: the width is then worked out from
|
||||
the source's proportions and the requested height, rounded down but at least
|
||||
1, or is the source's own width when the height is `0` too (the `orig` size)
|
||||
- `height` — requested height in pixels, or `0`: the height is then worked out
|
||||
from the source's proportions and the requested width, rounded down but at
|
||||
least 1, or is the source's own height when the width is `0` too (the `orig`
|
||||
size)
|
||||
- `path` — source path (e.g. `/photos/cat.jpg`), with each `%` written as `%25`
|
||||
and each `:` as `%3A`, so `/photos/a:b.jpg` is signed as `/photos/a%3Ab.jpg`
|
||||
- `query` — source query string, empty string if none, with `%` and `:` written
|
||||
as in `path`; neither then holds a `:`, so path `/a:b` with no query and path
|
||||
`/a` with query `b:` are signed differently
|
||||
- `width` — requested width in pixels, `0` for original
|
||||
- `height` — requested height in pixels, `0` for original
|
||||
- `format` — output format, one of those listed under Routes, with `original`
|
||||
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
|
||||
format has the signature of the same URL ending in `.jxl`; `auto` is signed as
|
||||
@@ -555,10 +536,7 @@ Where:
|
||||
with 400
|
||||
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
||||
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
||||
anything else, an empty `fit=` included, is refused with 400. `outside` scales
|
||||
the image to cover the requested size, keeping its proportions, so it can come
|
||||
out larger than that size on one side: an 8192x1 source at `100x0`, which is
|
||||
100x1 by its proportions, comes out at 8192x1
|
||||
anything else, an empty `fit=` included, is refused with 400
|
||||
|
||||
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
||||
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
||||
@@ -608,11 +586,7 @@ pixa lower-cases the host of a `/v1/image/` URL before anything else uses it, so
|
||||
`CDN.Example.com` is matched, fetched, cached and counted against
|
||||
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
|
||||
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
|
||||
even when the host is allowlisted. The host must be a host name or an IPv4
|
||||
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
|
||||
brackets, either one optionally followed by `:` and a port. Any other host, such
|
||||
as one holding `@`, or a `?`, `#`, `%`, `\` or space written percent-encoded, is
|
||||
refused with 400.
|
||||
even when the host is allowlisted. A host holding `@` is refused with 400.
|
||||
|
||||
### Configuration
|
||||
|
||||
@@ -694,10 +668,7 @@ Key settings in more detail:
|
||||
and generator pages. Default: empty
|
||||
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
|
||||
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
|
||||
benchmark, NAT64, and the like); an invalid CIDR aborts startup. An IPv6
|
||||
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
|
||||
address it holds, so an IPv6 range such as `::/0` refuses neither it nor a
|
||||
plain IPv4 address
|
||||
benchmark, NAT64, and the like); an invalid CIDR aborts startup
|
||||
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
|
||||
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
|
||||
of these ranges; the logged and login-recorded client address is then the
|
||||
@@ -728,14 +699,11 @@ Key settings in more detail:
|
||||
- `signing_key` — HMAC secret for URL signatures
|
||||
- `db_url` — the SQLite database to open; omitted, it is
|
||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
||||
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
|
||||
`#` in the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`
|
||||
or one starting with `//` it starts `file:////`, so SQLite does not read the
|
||||
first name in the path as a host. pixa opens one connection to it, so its own
|
||||
reads and writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to
|
||||
any `db_url`, so a write that finds another program writing to the file waits
|
||||
up to five seconds for it instead of failing. WAL mode comes only from the
|
||||
URL: keep `_pragma=journal_mode(WAL)` in one you set
|
||||
database in WAL mode. pixa opens one connection to it, so its own reads and
|
||||
writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to any
|
||||
`db_url`, so a write that finds another program writing to the file waits up
|
||||
to five seconds for it instead of failing. WAL mode comes only from the URL:
|
||||
keep `_pragma=journal_mode(WAL)` in one you set
|
||||
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
||||
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
||||
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
||||
|
||||
@@ -30,100 +30,18 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
|
||||
addresses refuses the local-use NAT64 range (`64:ff9b:1::/48`, RFC 8215) as it
|
||||
refuses `64:ff9b::/96`, and refuses an IPv6 address in the IPv4-compatible
|
||||
form (`::/96`) when `blocked_networks` contains either the address as written
|
||||
or the IPv4 address it holds (closes #299). A source host such as
|
||||
`[64:ff9b:1::a00:1]` was fetched, which on a network that runs NAT64 on that
|
||||
range reaches the IPv4 address it holds, here `10.0.0.1`; and with
|
||||
`203.0.113.0/24` in `blocked_networks`, `[::ffff:203.0.113.5]` was refused but
|
||||
`[::203.0.113.5]` was fetched. Both now answer 403. The mapped form
|
||||
(`::ffff:a.b.c.d`) is still matched only as the IPv4 address it holds: Go
|
||||
keeps a plain IPv4 address in that form, and an IPv6 entry such as `::/0` must
|
||||
not refuse every IPv4 address.
|
||||
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
|
||||
addresses judges an IPv6 address in the IPv4-compatible form (`::/96`), such
|
||||
as `::127.0.0.1` or `::10.0.0.1`, by the IPv4 address in its last four bytes
|
||||
(closes #297). It read only the mapped form (`::ffff:a.b.c.d`) as IPv4, so a
|
||||
source host such as `[::127.0.0.1]` was checked as a public IPv6 address and
|
||||
fetched, which on a network that routes it reaches the internal IPv4 host; it
|
||||
is now refused with 403. The NAT64 form (`64:ff9b::/96`) was already refused
|
||||
as a whole, as one of the built-in blocked ranges, and still is.
|
||||
- 2026-10-16 a source host that is an IPv6 address in brackets with no port,
|
||||
such as `[2001:db8::a]`, is fetched like one with a port, through `/v1/image/`
|
||||
and through an encrypted URL (closes #295). The fetcher's check of the source
|
||||
URL kept the brackets when no port followed them, could not look the host up,
|
||||
and the request answered 500. It now checks the address without brackets, so a
|
||||
private one is refused with 403, as with a port.
|
||||
- 2026-10-16 a `state_dir` whose name holds `%`, `?` or `#` keeps its database
|
||||
at `state.sqlite3` in that directory (closes #290): the `db_url` derived from
|
||||
it put the name into a `file:` URL as it was, so SQLite read `%41` as `A`, a
|
||||
`?` or `#` ended the path early, and the journal mode parameter was lost. The
|
||||
derived `db_url` is now built with Go's `net/url`, which writes those
|
||||
characters as `%25`, `%3F` and `%23`. For a `state_dir` of `/` or one starting
|
||||
with `//` it starts `file:////`, so SQLite no longer reads the first name in
|
||||
the path as a host.
|
||||
- 2026-10-16 a size with one side `0` keeps the side computed from the source's
|
||||
proportions at 1 or more (closes #283): it was rounded down, so for a thin
|
||||
source it came out at 0, and libvips then made the size square. An 8192x1
|
||||
source at `100x0` came out at 100x100, and at `1300x0` with `fit=cover` made
|
||||
libvips ask for about 39 GB at once; an 8192x2 source at `100x0` answered 500.
|
||||
`fit=contain` and `fit=inside` work both sides out again from the smaller of
|
||||
the two scales and `fit=outside` from the larger, and each keeps both sides at
|
||||
1 or more too, so a 1x8192 source at `0x100` with `fit=contain`, and a 49x8192
|
||||
source at `0x100` or a 49x49 source at `1x1` with `fit=outside`, no longer
|
||||
answer 500. An 8192x1 source at `1300x0` now comes out at 1300x1, and the
|
||||
process running it alone peaked at about 34 MiB resident. With `fit=outside`
|
||||
the image is scaled to cover the size, so that source at `100x0` comes out at
|
||||
8192x1. `README.md` now says this under Signature Specification, and says how
|
||||
a `0` side is worked out where it said `0` meant the original.
|
||||
- 2026-10-16 the generator page lower-cases the source host and writes an empty
|
||||
source path as `/` before it makes the encrypted URL, as `/v1/image/` does
|
||||
(closes #293). A host written in upper and in lower case, or a source URL with
|
||||
no path and with the path `/`, made two cache keys and two upstream requests
|
||||
for one image.
|
||||
- 2026-10-16 the source URL a request fetches is built with `net/url` from the
|
||||
source host, path and query, not joined as text (closes #288). The source
|
||||
path, decoded, is fetched percent-encoded again, so `100%25.png` fetches
|
||||
`/100%25.png` where it answered 401, and `my%20cat.png` and `my%2520cat.png`
|
||||
fetch different paths where both fetched `/my%20cat.png` under two cache keys.
|
||||
The source query stays as the source URL writes it; the characters a browser
|
||||
percent-encodes in a query, such as `'` or a space, which was sent raw and
|
||||
answered 502, are percent-encoded when the image URL is read, so the query is
|
||||
signed and cached as it is sent. The generator page answers 400, showing the
|
||||
URL pixa would fetch, for a source URL its image URL would fetch written
|
||||
another way, such as with `%2F` in the path, whatever else the path holds, or
|
||||
with a user name and password. It also answers 400 for a source path holding a
|
||||
`..` segment, as `/v1/image/` refuses one.
|
||||
- 2026-10-15 a thin source requested with `fit=cover` no longer makes libvips
|
||||
build a huge image (closes #282): libvips' thumbnail scaled the whole source
|
||||
to cover the requested size and copied that into memory before it cut out the
|
||||
centre, so an 8192x64 source at 8192x8192 made it allocate about 24 GiB, and
|
||||
an 8192x1 source answered 500. The image processor now first cuts the centre
|
||||
of the source, widened to whole source pixels, and thumbnail scales and cuts
|
||||
only that, by the same amount on both sides as before. What it copies into
|
||||
memory is then larger than the output by less than one scaled source pixel on
|
||||
each side. With the runtime image's libvips and JPEG output at 8192x8192,
|
||||
resident memory peaked at about 242 MiB for the 8192x64 source and 431 MiB for
|
||||
the 8192x1 one. The output can sit a fraction of an output pixel from where it
|
||||
was.
|
||||
- 2026-10-15 a `/v1/image/` URL whose source path or source query holds a `#`,
|
||||
written as `%23`, is refused with 400 (closes #286): the source URL built from
|
||||
it ended at the `#` when fetched, so each text after it fetched the same
|
||||
upstream image, without a signature for an allowlisted host, and cached it
|
||||
under keys of its own. `README.md` says neither may hold a `#`.
|
||||
- 2026-10-15 the host of a `/v1/image/` URL must be a host name or an IPv4
|
||||
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
|
||||
brackets, either one optionally followed by `:` and a port, and any other host
|
||||
is refused with 400 (closes #275): only `@` was refused, so a `?` written as
|
||||
`%3F` gave the host `cdn.example.com?x=1`, which Go's URL parser reads as
|
||||
`cdn.example.com` with a query. Such a URL passed the allowlist, and each text
|
||||
after the `?` fetched the host's root path and cached the answer under keys of
|
||||
its own. A `#`, `%`, `\` or space written percent-encoded is refused the same
|
||||
way, and so is an IPv6 address without brackets, which needed a signature
|
||||
before. Underscores are kept because `allowlist_hosts` accepts host names
|
||||
holding them.
|
||||
- 2026-10-15 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34
|
||||
seconds (closes #272): with one libvips thread, 8192x8192 random pixels with
|
||||
an alpha channel took about 61 seconds to encode as WebP, 96 as AVIF and 81 as
|
||||
JPEG XL, so with the default `downstream_timeout` of 60 seconds such an image
|
||||
was never delivered, yet its encode held a processing slot to the end. WebP is
|
||||
now saved at effort 2 and JPEG XL at effort 4, which take up to about 23 and
|
||||
27 seconds. AVIF was already at the lowest effort govips can set, so an AVIF
|
||||
output of more than 16,777,216 pixels is refused with 400 before anything is
|
||||
decoded; 4096x4096, 8192x2048 or 2048x8192 takes up to about 34 seconds. Each
|
||||
time is the slowest at any quality. Resizing with `fit=cover` already decodes
|
||||
the input, so the output's dimensions are worked out before resizing.
|
||||
`README.md` gives the limit and the new times.
|
||||
- 2026-10-15 the README says how a source URL with a query is written into an
|
||||
image URL, and the image route accepts it (closes #255): the query follows the
|
||||
source path after `%3F`. pixa now reads the image URL's path decoded once, so
|
||||
|
||||
@@ -338,7 +338,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
||||
settingName(keyDBURL), errValueEmpty)
|
||||
}
|
||||
|
||||
c.DBURL = defaultDBURL(c.StateDir)
|
||||
// The driver sets the journal mode only through a _pragma parameter.
|
||||
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_pragma=journal_mode(WAL)", c.StateDir)
|
||||
}
|
||||
|
||||
if loader.err != nil {
|
||||
@@ -353,30 +354,6 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// defaultDBURL returns the db_url used when db_url is omitted: the file
|
||||
// state.sqlite3 in stateDir, in WAL mode, which the driver sets only
|
||||
// through a _pragma parameter. String() percent-encodes any %, ? or # in
|
||||
// the directory name, which SQLite decodes back, so the URL names exactly
|
||||
// this file. OmitHost writes file:/var/lib/pixa/... instead of
|
||||
// file:///var/lib/pixa/..., and keeps a relative stateDir from being read
|
||||
// as a host name. The path written into the URL is stateDir followed by
|
||||
// /state.sqlite3; when it starts with //, as it does for a stateDir of /
|
||||
// or one starting with //, the empty host is kept, so the URL starts
|
||||
// file://// and SQLite does not read the first name in the path as a
|
||||
// host.
|
||||
func defaultDBURL(stateDir string) string {
|
||||
dbPath := stateDir + "/state.sqlite3"
|
||||
|
||||
dbURL := url.URL{
|
||||
Scheme: "file",
|
||||
OmitHost: !strings.HasPrefix(dbPath, "//"),
|
||||
Path: dbPath,
|
||||
RawQuery: "_pragma=journal_mode(WAL)",
|
||||
}
|
||||
|
||||
return dbURL.String()
|
||||
}
|
||||
|
||||
// validateKnownKeys rejects configuration files containing keys the
|
||||
// application does not understand, so typos fail at startup instead of
|
||||
// being silently ignored, and rejects keys that are explicitly set to
|
||||
|
||||
@@ -135,104 +135,6 @@ func TestDefaultDBURLOpensTheDatabaseInWALMode(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDerivedDBURLOpensStateDirHoldingURLCharacters opens the db_url
|
||||
// derived from a state_dir whose name holds %41, ? and #, and checks that
|
||||
// the database is state.sqlite3 in that directory, in WAL mode. Put into
|
||||
// a file: URL as it is, the name would lose its meaning: %41 is read as
|
||||
// A, and the path ends at the ?, which also loses the journal mode.
|
||||
func TestDerivedDBURLOpensStateDirHoldingURLCharacters(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
stateDir := filepath.Join(t.TempDir(), "a%41b?c#d")
|
||||
|
||||
err := os.Mkdir(stateDir, 0o750)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create %q: %v", stateDir, err)
|
||||
}
|
||||
|
||||
// Quoted so YAML takes the whole name as it is.
|
||||
c, err := configFromYAML(t, signingKeyLine+"state_dir: '"+stateDir+"'\n")
|
||||
if err != nil {
|
||||
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
|
||||
}
|
||||
|
||||
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
|
||||
|
||||
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(stateDir, "state.sqlite3"))
|
||||
}
|
||||
|
||||
// TestDerivedDBURLOpensStateDirStartingWithTwoSlashes opens the db_url
|
||||
// derived from a state_dir that starts with //, and checks that the
|
||||
// database is state.sqlite3 in that directory, in WAL mode. Written right
|
||||
// after file:, such a name gives a URL starting file://, and SQLite reads
|
||||
// the first directory name as a host and refuses to open it.
|
||||
func TestDerivedDBURLOpensStateDirStartingWithTwoSlashes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
stateDir := "/" + dir
|
||||
|
||||
c, err := configFromYAML(t, signingKeyLine+"state_dir: "+stateDir+"\n")
|
||||
if err != nil {
|
||||
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
|
||||
}
|
||||
|
||||
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
|
||||
|
||||
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(dir, "state.sqlite3"))
|
||||
}
|
||||
|
||||
// TestDerivedDBURLForRootStateDir checks the db_url derived from a
|
||||
// state_dir of /. The path written into the URL is then
|
||||
// //state.sqlite3, and written right after file: it gives a URL starting
|
||||
// file://, where SQLite reads state.sqlite3 as a host and refuses to open
|
||||
// it. The test only checks the URL, since it cannot write to /.
|
||||
func TestDerivedDBURLForRootStateDir(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c, err := configFromYAML(t, signingKeyLine+"state_dir: /\n")
|
||||
if err != nil {
|
||||
t.Fatalf("config with state_dir / should be valid, got: %v", err)
|
||||
}
|
||||
|
||||
t.Logf("state_dir / gives db_url %q", c.DBURL)
|
||||
|
||||
wantDBURL := "file:////state.sqlite3?_pragma=journal_mode(WAL)"
|
||||
if c.DBURL != wantDBURL {
|
||||
t.Errorf("DBURL = %q, want %q", c.DBURL, wantDBURL)
|
||||
}
|
||||
}
|
||||
|
||||
// checkDBURLOpensFileInWALMode opens dbURL with the SQLite driver pixad
|
||||
// uses, and checks that the database is in WAL mode and is the file
|
||||
// dbFile.
|
||||
func checkDBURLOpensFileInWALMode(t *testing.T, dbURL, dbFile string) {
|
||||
t.Helper()
|
||||
|
||||
db, err := sql.Open("sqlite", dbURL)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to open %q: %v", dbURL, err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
var journalMode string
|
||||
|
||||
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&journalMode)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read the journal mode of %q: %v", dbURL, err)
|
||||
}
|
||||
|
||||
if journalMode != "wal" {
|
||||
t.Errorf("journal mode of %q = %q, want wal", dbURL, journalMode)
|
||||
}
|
||||
|
||||
_, err = os.Stat(dbFile)
|
||||
if err != nil {
|
||||
t.Errorf("opening %q did not create %q: %v", dbURL, dbFile, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExplicitValidValuesAreUsed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -9,9 +9,7 @@ import (
|
||||
"math"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/clientip"
|
||||
@@ -127,7 +125,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
sourceURL := r.FormValue("url")
|
||||
|
||||
// Validate source URL
|
||||
parsed, err := parseSourceURL(sourceURL)
|
||||
parsed, err := url.Parse(sourceURL)
|
||||
if err != nil || parsed.Host == "" {
|
||||
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
|
||||
http.StatusBadRequest)
|
||||
@@ -135,16 +133,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
// /v1/image/ refuses a ".." segment in the source path, and so does
|
||||
// the generator. The path is decoded, so %2e%2e counts too
|
||||
if slices.Contains(strings.Split(parsed.Path, "/"), "..") {
|
||||
s.renderGeneratorWithForm(w, r,
|
||||
"Source URL path cannot hold a .. segment", r.Form,
|
||||
http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
|
||||
if err != nil {
|
||||
s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
|
||||
@@ -153,19 +141,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
// The image URL must fetch the source URL as it was written
|
||||
sourceRequest := payload.ToImageRequest()
|
||||
sourceRequest.AllowHTTP = s.config.AllowHTTP
|
||||
fetchURL := sourceRequest.SourceURL()
|
||||
|
||||
if !fetchedAsWritten(parsed, fetchURL) {
|
||||
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
|
||||
"written: pixa would fetch "+fetchURL,
|
||||
r.Form, http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Generate encrypted token
|
||||
token, err := s.encGen.Generate(payload)
|
||||
if err != nil {
|
||||
@@ -198,48 +173,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// parseSourceURL parses the source URL entered on the generator page with its
|
||||
// host lower-cased and an empty path written as "/", as /v1/image/ reads a
|
||||
// source URL, so that each spelling of one source URL makes one cache key and
|
||||
// one upstream request.
|
||||
func parseSourceURL(sourceURL string) (*url.URL, error) {
|
||||
parsed, err := url.Parse(sourceURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
parsed.Host = strings.ToLower(parsed.Host)
|
||||
|
||||
if parsed.Path == "" {
|
||||
parsed.Path = "/"
|
||||
}
|
||||
|
||||
return parsed, nil
|
||||
}
|
||||
|
||||
// fetchedAsWritten reports whether fetchURL, the URL pixa fetches for an
|
||||
// image URL made from the source URL source, is source as it was written,
|
||||
// apart from the scheme, which the token does not keep. It is not when source
|
||||
// holds a user name and password, which the token does not keep either, or a
|
||||
// "?" with no query after it. The token keeps the source path decoded and
|
||||
// pixa fetches it percent-encoded again, so a path written another way, such
|
||||
// as with %2F, is fetched as another path. url.URL keeps the path as written
|
||||
// in RawPath when it is not the decoded path percent-encoded again.
|
||||
// EscapedPath ignores RawPath when it also holds a character net/url encodes,
|
||||
// such as a space, so RawPath is compared itself.
|
||||
func fetchedAsWritten(source *url.URL, fetchURL string) bool {
|
||||
writtenPath := source.RawPath
|
||||
if writtenPath == "" {
|
||||
writtenPath = source.EscapedPath()
|
||||
}
|
||||
|
||||
fetched, err := url.Parse(fetchURL)
|
||||
|
||||
return err == nil && source.User == nil && !source.ForceQuery &&
|
||||
fetched.EscapedPath() == writtenPath &&
|
||||
fetched.RawQuery == source.RawQuery
|
||||
}
|
||||
|
||||
// buildGeneratePayload parses the numeric form fields and assembles the
|
||||
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
|
||||
// non-numeric or out-of-range width, height, quality or ttl, or an
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
|
||||
// handleImageError, which answer the errors of the two image routes, answer
|
||||
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
|
||||
// output above it, wrapped as the service wraps it. The request's source only
|
||||
// goes into a log line, so it is left empty.
|
||||
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||
err := fmt.Errorf("image processing failed: %w: 4097x4096",
|
||||
imageprocessor.ErrAVIFOutputTooLarge)
|
||||
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
|
||||
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
|
||||
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
for name, rec := range map[string]*httptest.ResponseRecorder{
|
||||
"respondImageError": plain, "handleImageError": encrypted,
|
||||
} {
|
||||
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status = %d, want %d",
|
||||
name, rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), wantBody) {
|
||||
t.Errorf("%s: body = %q, want it to say %q",
|
||||
name, rec.Body.String(), wantBody)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -264,6 +264,14 @@ func (s *Handlers) respondImageError(
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
|
||||
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||
http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||
}
|
||||
|
||||
|
||||
@@ -264,127 +264,3 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_InvalidHost_Returns400 sends image URLs whose host no URL
|
||||
// host can be, and checks that each is refused with 400 naming the host as
|
||||
// decoded: allowlistedHost with a character after it that a URL host cannot
|
||||
// hold, percent-encoded, an IPv6 address without brackets, and brackets
|
||||
// around something that is not an IPv6 address. A "?" or "#" after
|
||||
// allowlistedHost would otherwise end the host when the source URL is read,
|
||||
// so the request would pass the allowlist and be fetched and cached under a
|
||||
// host that is not the one fetched.
|
||||
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
photos := newPhotoFetcher(t, allowlistedHost)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
writtenHost string
|
||||
decodedHost string
|
||||
}{
|
||||
{"? written as %3F", allowlistedHost + "%3Fx=1", allowlistedHost + "?x=1"},
|
||||
{"# written as %23", allowlistedHost + "%23x", allowlistedHost + "#x"},
|
||||
{"% written as %25", allowlistedHost + "%253F", allowlistedHost + "%3F"},
|
||||
{`\ written as %5C`, allowlistedHost + "%5Cx", allowlistedHost + `\x`},
|
||||
{"space written as %20", allowlistedHost + "%20", allowlistedHost + " "},
|
||||
{"IPv6 address without brackets", "2001:db8::1", "2001:db8::1"},
|
||||
{"IPv4 address in brackets", "[203.0.113.5]", "[203.0.113.5]"},
|
||||
{"no address in brackets", "[abc]", "[abc]"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rec := sendGet(t, newImageRoute(t, photos), photoURL(tt.writtenHost))
|
||||
checkErrorBody(t, rec, http.StatusBadRequest,
|
||||
fmt.Sprintf("invalid image URL: invalid source host: %q",
|
||||
tt.decodedHost))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_HostForms_Served checks that the image route serves, with
|
||||
// a valid signature, the image at photoPath on a host name, an IPv4 address,
|
||||
// an IPv6 address in brackets, and a host followed by ":" with or without a
|
||||
// port.
|
||||
func TestHandleImage_HostForms_Served(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
hosts := []string{
|
||||
allowlistedHost,
|
||||
"img_1.example.com",
|
||||
"203.0.113.5",
|
||||
"[2001:db8::1]",
|
||||
allowlistedHost + ":8443",
|
||||
"[2001:db8::1]:8443",
|
||||
allowlistedHost + ":",
|
||||
}
|
||||
|
||||
photos := newPhotoFetcher(t, hosts...)
|
||||
exp := time.Now().Add(time.Hour)
|
||||
|
||||
for _, host := range hosts {
|
||||
t.Run(host, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
sig := photoSignature(testSigningKey, host, exp)
|
||||
|
||||
rec := sendGet(t, newImageRoute(t, photos),
|
||||
photoURLWithSig(host, sig, exp))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Errorf("status = %d, want %d: %s",
|
||||
rec.Code, http.StatusOK, rec.Body)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
|
||||
// source path or source query holds a "#", written as %23, and checks that
|
||||
// each is refused with 400, the one signed for that source path included. The
|
||||
// source URL would otherwise end at the "#" when it is fetched, so each text
|
||||
// after it would fetch the image at photoPath and cache it under keys of its
|
||||
// own.
|
||||
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
|
||||
exp := time.Now().Add(time.Hour)
|
||||
|
||||
sig := signature.New(testSigningKey).Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath + "#x",
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: exp,
|
||||
})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
target string
|
||||
}{
|
||||
{"# in the source path",
|
||||
"/v1/image/" + allowlistedHost + photoPath + "%23x/50x50.jpeg"},
|
||||
{"# in the source query",
|
||||
"/v1/image/" + allowlistedHost + photoPath + "%3Fv=1%23x/50x50.jpeg"},
|
||||
{"# in the source path, signed",
|
||||
fmt.Sprintf("/v1/image/%s%s%%23x/50x50.jpeg?sig=%s&exp=%d",
|
||||
signedHost, photoPath, sig, exp.Unix())},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rec := sendGet(t, newImageRoute(t, photos), tt.target)
|
||||
checkErrorBody(t, rec, http.StatusBadRequest,
|
||||
`invalid image URL: invalid image path: `+
|
||||
`the source path or query holds a "#"`)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -170,6 +170,10 @@ func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
|
||||
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
|
||||
"computed from its proportions is above %d",
|
||||
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
|
||||
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
|
||||
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||
http.StatusBadRequest)
|
||||
default:
|
||||
s.log.Error("image request failed", "error", err)
|
||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||
|
||||
@@ -59,17 +59,15 @@ const (
|
||||
// builtinBlockedPrefixes are internal or special-use ranges that Go's
|
||||
// net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not
|
||||
// already cover. They are always blocked, in addition to any
|
||||
// operator-supplied networks. IPv4-mapped and IPv4-compatible IPv6
|
||||
// addresses are read as the IPv4 address they hold before matching, so
|
||||
// these IPv4 ranges are caught in all three forms.
|
||||
// operator-supplied networks. IPv4-mapped IPv6 addresses are unmapped
|
||||
// before matching, so these IPv4 ranges are caught in both forms.
|
||||
//
|
||||
//nolint:gochecknoglobals // immutable built-in blocklist
|
||||
var builtinBlockedPrefixes = []netip.Prefix{
|
||||
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT
|
||||
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
|
||||
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
|
||||
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
|
||||
netip.MustParsePrefix("64:ff9b:1::/48"), // RFC 8215 local-use NAT64 (maps onto IPv4)
|
||||
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT
|
||||
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
|
||||
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
|
||||
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
|
||||
}
|
||||
|
||||
// Fetcher errors.
|
||||
@@ -475,18 +473,18 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
|
||||
return ErrUnsupportedScheme
|
||||
}
|
||||
|
||||
parsedURL, err := neturl.Parse(rawURL)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
|
||||
}
|
||||
|
||||
// Hostname drops any port, and the brackets of an IPv6 address whether
|
||||
// or not a port follows them
|
||||
host := parsedURL.Hostname()
|
||||
// Parse to extract host
|
||||
host := extractHost(rawURL)
|
||||
if host == "" {
|
||||
return ErrInvalidHost
|
||||
}
|
||||
|
||||
// Remove port if present
|
||||
h, _, err := net.SplitHostPort(host)
|
||||
if err == nil {
|
||||
host = h
|
||||
}
|
||||
|
||||
// Block obvious localhost patterns
|
||||
if isLocalhost(host) {
|
||||
return ErrSSRFBlocked
|
||||
@@ -545,9 +543,6 @@ func isPrivateIP(ip net.IP) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// An IPv4-compatible IPv6 address is judged as the IPv4 address it holds
|
||||
ip = ipv4CompatibleToIPv4(ip)
|
||||
|
||||
// Check for loopback
|
||||
if ip.IsLoopback() {
|
||||
return true
|
||||
@@ -599,12 +594,7 @@ func isPrivateIP(ip net.IP) bool {
|
||||
}
|
||||
|
||||
// isBlockedIP reports whether ip is refused, either by the built-in
|
||||
// internal-range check or by one of the operator-supplied prefixes. An
|
||||
// IPv4-compatible address (in ::/96) is refused when a prefix contains the
|
||||
// address as written or the IPv4 address it holds. An IPv4-mapped address (in
|
||||
// ::ffff:0:0/96) is matched only as the IPv4 address it holds: Go keeps a
|
||||
// plain IPv4 address in that form too, and an IPv6 prefix such as ::/0 must
|
||||
// not refuse every IPv4 address.
|
||||
// internal-range check or by one of the operator-supplied prefixes.
|
||||
func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
|
||||
if isPrivateIP(ip) {
|
||||
return true
|
||||
@@ -617,28 +607,11 @@ func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
|
||||
|
||||
addr = addr.Unmap()
|
||||
|
||||
// ip read as IPv4 where it is or holds an IPv4 address, otherwise the
|
||||
// zero netip.Addr, which no prefix contains. Matching asIPv4 as well as
|
||||
// addr refuses more only for an IPv4-compatible address.
|
||||
asIPv4, _ := netip.AddrFromSlice(ipv4CompatibleToIPv4(ip).To4())
|
||||
|
||||
return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool {
|
||||
return prefix.Contains(addr) || prefix.Contains(asIPv4)
|
||||
return prefix.Contains(addr)
|
||||
})
|
||||
}
|
||||
|
||||
// ipv4CompatibleToIPv4 returns the IPv4 address that an IPv4-compatible IPv6
|
||||
// address (in ::/96, such as ::127.0.0.1) holds in its last four bytes, and
|
||||
// any other address unchanged.
|
||||
func ipv4CompatibleToIPv4(ip net.IP) net.IP {
|
||||
addr, ok := netip.AddrFromSlice(ip)
|
||||
if ok && netip.MustParsePrefix("::/96").Contains(addr) {
|
||||
return ip[12:]
|
||||
}
|
||||
|
||||
return ip
|
||||
}
|
||||
|
||||
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges
|
||||
// before connecting. New wraps dialSSRFSafe with the operator-supplied
|
||||
// blocklist; this entry point enforces the built-in ranges alone.
|
||||
|
||||
@@ -186,44 +186,6 @@ func TestIsPrivateIP(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestIsPrivateIPJudgesEmbeddedIPv4Address checks IPv6 addresses that hold an
|
||||
// IPv4 address in their last four bytes: the IPv4-compatible form (::/96) and
|
||||
// the NAT64 form (64:ff9b::/96). One holding a loopback, private or
|
||||
// link-local IPv4 address is refused as that address is, and one holding a
|
||||
// public IPv4 address in the IPv4-compatible form passes.
|
||||
func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
ip string
|
||||
want bool
|
||||
}{
|
||||
{"::127.0.0.1", true}, // loopback
|
||||
{"::10.0.0.1", true}, // private
|
||||
{"::169.254.169.254", true}, // link-local
|
||||
{"64:ff9b::7f00:1", true}, // NAT64 form of 127.0.0.1
|
||||
{"::8.8.8.8", false}, // public
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.ip, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ip := net.ParseIP(tc.ip)
|
||||
if ip == nil {
|
||||
t.Fatalf("failed to parse IP %q", tc.ip)
|
||||
}
|
||||
|
||||
got := isPrivateIP(ip)
|
||||
t.Logf("isPrivateIP(%s) = %v", ip, got)
|
||||
|
||||
if got != tc.want {
|
||||
t.Errorf("isPrivateIP(%q) = %v, want %v", tc.ip, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
package imageprocessor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
|
||||
// of more than MaxAVIFOutputPixels is refused with ErrAVIFOutputTooLarge
|
||||
// before anything is decoded, whatever size and fit give it, and that an AVIF
|
||||
// output of at most that many pixels, or one in another format, is not. The
|
||||
// input is a 64x32 PNG cut off in its pixel data: libvips reads its size from
|
||||
// its header but fails when it decodes it, so a request that gets that failure
|
||||
// was not refused, and a refusal came before any decoding.
|
||||
func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
whole := createTestPNG(t, 64, 32)
|
||||
cutOff := whole[:len(whole)/2]
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
format Format
|
||||
width, height int
|
||||
fit FitMode
|
||||
refused bool
|
||||
}{
|
||||
{"AVIF at the limit", FormatAVIF, 4096, 4096, FitCover, false},
|
||||
{"AVIF above the limit", FormatAVIF, 4096, 4097, FitCover, true},
|
||||
{"WebP above the AVIF limit", FormatWebP, 4096, 4097, FitCover, false},
|
||||
// FitInside does not enlarge, so the output stays 64x32
|
||||
{"AVIF inside a larger size", FormatAVIF, 8192, 8192, FitInside, false},
|
||||
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
|
||||
{"AVIF outside a smaller size", FormatAVIF, 8192, 1, FitOutside, true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(cutOff),
|
||||
&Request{
|
||||
Size: Size{Width: tt.width, Height: tt.height},
|
||||
Format: tt.format,
|
||||
FitMode: tt.fit,
|
||||
})
|
||||
t.Logf("Process() error = %v", err)
|
||||
|
||||
if tt.refused {
|
||||
if !errors.Is(err, ErrAVIFOutputTooLarge) {
|
||||
t.Errorf("Process() error = %v, want %v", err, ErrAVIFOutputTooLarge)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if err == nil || errors.Is(err, ErrAVIFOutputTooLarge) {
|
||||
t.Errorf("Process() error = %v, want the failure to decode "+
|
||||
"the cut-off input", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -50,69 +50,3 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_ComputedSideIsAtLeastOne verifies that, for a size with
|
||||
// one side 0, the side computed from the input's proportions is 1 where it
|
||||
// would round down to 0, and that Process makes that size in every fit mode
|
||||
// but FitOutside, which scales the input to cover the size and so can make
|
||||
// it longer. Each case checks the size targetDimensions gives before it
|
||||
// calls Process: with that side at 0, an 8192x1 input at 1300x0 makes
|
||||
// libvips ask for about 39 GB with FitCover.
|
||||
func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
wantWidth, wantHeight int
|
||||
}{
|
||||
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
|
||||
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
|
||||
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
|
||||
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
|
||||
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
|
||||
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
|
||||
}
|
||||
|
||||
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
|
||||
|
||||
for _, tt := range tests {
|
||||
for _, fit := range fits {
|
||||
t.Run(tt.name+" "+string(fit), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
width, height, err := targetDimensions(tt.size,
|
||||
tt.inputWidth, tt.inputHeight)
|
||||
if err != nil || width != tt.wantWidth || height != tt.wantHeight {
|
||||
t.Fatalf("targetDimensions() = %dx%d, %v, want %dx%d",
|
||||
width, height, err, tt.wantWidth, tt.wantHeight)
|
||||
}
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
|
||||
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
|
||||
if fit == FitOutside {
|
||||
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
|
||||
}
|
||||
|
||||
if !ok {
|
||||
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
|
||||
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
|
||||
FitOutside)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -77,10 +77,10 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_WebPAtDefaultEffort verifies that WebP is saved at
|
||||
// libvips' default effort, 4: the output is the size of the same image saved
|
||||
// at that effort.
|
||||
func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
|
||||
// TestImageProcessor_WebPAtEffort2 verifies that WebP is saved at effort 2,
|
||||
// not libvips' default of 4: the output is the size of the same image saved
|
||||
// at effort 2.
|
||||
func TestImageProcessor_WebPAtEffort2(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
@@ -90,14 +90,14 @@ func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
|
||||
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
|
||||
StripMetadata: true,
|
||||
Quality: 85,
|
||||
ReductionEffort: 4,
|
||||
ReductionEffort: 2,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("ExportWebp() error = %v", err)
|
||||
}
|
||||
|
||||
if size != int64(len(want)) {
|
||||
t.Errorf("WebP output is %d bytes, want %d, the size at effort 4",
|
||||
t.Errorf("WebP output is %d bytes, want %d, the size at effort 2",
|
||||
size, len(want))
|
||||
}
|
||||
}
|
||||
@@ -143,3 +143,29 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
|
||||
"and 8 bits", size, len(want))
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_JPEGXLAtEffort4 verifies that JPEG XL is saved at
|
||||
// effort 4, not libvips' default of 7: the output is the size of the same
|
||||
// image saved at effort 4, with the distance exportJXL gives quality 85.
|
||||
func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, &Request{Format: FormatJXL, Quality: 85})
|
||||
|
||||
params := vips.NewJxlExportParams()
|
||||
params.Quality = 85
|
||||
params.Distance = jxlDistance(85)
|
||||
params.Effort = 4
|
||||
|
||||
want, _, err := decode(t, input).ExportJxl(params)
|
||||
if err != nil {
|
||||
t.Fatalf("ExportJxl() error = %v", err)
|
||||
}
|
||||
|
||||
if size != int64(len(want)) {
|
||||
t.Errorf("JPEG XL output is %d bytes, want %d, the size at effort 4",
|
||||
size, len(want))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,268 +0,0 @@
|
||||
package imageprocessor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/png"
|
||||
"math"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
|
||||
// a thin source at the requested size. An 8192x1 source scaled to cover
|
||||
// 8192x8192 is 8192 times as long as the output on its long side. The centre
|
||||
// of an 8192x1 source with the proportions of 1x8192 is 1/8192 of a pixel
|
||||
// wide. Each case is also turned on its side.
|
||||
func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
}{
|
||||
{"8192x1 at 8192x8192", 8192, 1, Size{Width: 8192, Height: 8192}},
|
||||
{"8192x1 at 1x8192", 8192, 1, Size{Width: 1, Height: 8192}},
|
||||
{"1x8192 at 8192x8192", 1, 8192, Size{Width: 8192, Height: 8192}},
|
||||
{"1x8192 at 8192x1", 1, 8192, Size{Width: 8192, Height: 1}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
|
||||
if result.Width != tt.size.Width || result.Height != tt.size.Height {
|
||||
t.Errorf("Process() made %dx%d, want %dx%d",
|
||||
result.Width, result.Height, tt.size.Width, tt.size.Height)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// createBandsPNG creates a PNG of three equal bands, red, green and blue,
|
||||
// side by side when it is wider than tall and top to bottom otherwise.
|
||||
func createBandsPNG(t *testing.T, width, height int) []byte {
|
||||
t.Helper()
|
||||
|
||||
bands := []color.RGBA{
|
||||
{R: 255, A: 255},
|
||||
{G: 255, A: 255},
|
||||
{B: 255, A: 255},
|
||||
}
|
||||
|
||||
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||
for y := range height {
|
||||
for x := range width {
|
||||
band := 3 * y / height
|
||||
if width > height {
|
||||
band = 3 * x / width
|
||||
}
|
||||
|
||||
img.Set(x, y, bands[band])
|
||||
}
|
||||
}
|
||||
|
||||
return encodePNG(t, img)
|
||||
}
|
||||
|
||||
// TestImageProcessor_FitCoverCropsTheCentre verifies that FitCover keeps the
|
||||
// centre of the source: a 300x100 source of red, green and blue bands side
|
||||
// by side, or a 100x300 one with them top to bottom, comes out at 50x50 as
|
||||
// its green band.
|
||||
func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
}{
|
||||
{"bands side by side", 300, 100},
|
||||
{"bands top to bottom", 100, 300},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createBandsPNG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatPNG,
|
||||
FitMode: FitCover,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
output, err := png.Decode(result.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to decode output: %v", err)
|
||||
}
|
||||
|
||||
bounds := output.Bounds()
|
||||
t.Logf("Process() made %dx%d", bounds.Dx(), bounds.Dy())
|
||||
|
||||
if bounds.Dx() != 50 || bounds.Dy() != 50 {
|
||||
t.Fatalf("Process() made %dx%d, want 50x50",
|
||||
bounds.Dx(), bounds.Dy())
|
||||
}
|
||||
|
||||
// Resampling may blend a little of the bands beside the
|
||||
// green one into the output's edges.
|
||||
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
|
||||
for x := bounds.Min.X; x < bounds.Max.X; x++ {
|
||||
r, g, b, _ := output.At(x, y).RGBA()
|
||||
if r>>8 > 64 || g>>8 < 192 || b>>8 > 64 {
|
||||
t.Fatalf("output pixel %d,%d is %d,%d,%d, want green",
|
||||
x, y, r>>8, g>>8, b>>8)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The greys of createSquarePNG, and the grey halfway between them.
|
||||
const (
|
||||
backgroundGrey = 192
|
||||
squareGrey = 64
|
||||
halfwayGrey = (backgroundGrey + squareGrey) / 2
|
||||
)
|
||||
|
||||
// createSquarePNG creates a grey PNG side pixels square with a darker grey
|
||||
// square squareSide pixels wide in its centre.
|
||||
func createSquarePNG(t *testing.T, side, squareSide int) []byte {
|
||||
t.Helper()
|
||||
|
||||
first := (side - squareSide) / 2
|
||||
last := first + squareSide - 1
|
||||
|
||||
img := image.NewGray(image.Rect(0, 0, side, side))
|
||||
for y := range side {
|
||||
for x := range side {
|
||||
grey := uint8(backgroundGrey)
|
||||
if x >= first && x <= last && y >= first && y <= last {
|
||||
grey = squareGrey
|
||||
}
|
||||
|
||||
img.SetGray(x, y, color.Gray{Y: grey})
|
||||
}
|
||||
}
|
||||
|
||||
return encodePNG(t, img)
|
||||
}
|
||||
|
||||
// squareLength returns how long the square is along greys, the greys of a
|
||||
// row or column of the output that crosses it: the distance between the
|
||||
// first and the last point where greys passes halfwayGrey, or 0 when there
|
||||
// is no such point. Resampling blurs the square's edges over several
|
||||
// pixels, and each point is placed between the two pixels on either side
|
||||
// of it in proportion to their greys, so the length is measured to a
|
||||
// fraction of a pixel.
|
||||
func squareLength(greys []float64) float64 {
|
||||
first, last := -1.0, -1.0
|
||||
|
||||
for i := 1; i < len(greys); i++ {
|
||||
before, after := greys[i-1], greys[i]
|
||||
if (before > halfwayGrey) == (after > halfwayGrey) {
|
||||
continue
|
||||
}
|
||||
|
||||
point := float64(i-1) + (before-halfwayGrey)/(before-after)
|
||||
if first < 0 {
|
||||
first = point
|
||||
}
|
||||
|
||||
last = point
|
||||
}
|
||||
|
||||
return last - first
|
||||
}
|
||||
|
||||
// TestImageProcessor_FitCoverKeepsProportions verifies that FitCover scales
|
||||
// both sides of the source by the same amount: a square in the centre of
|
||||
// the source comes out square. Its width and height are measured along the
|
||||
// output's middle row and column.
|
||||
func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
side, squareSide int
|
||||
size Size
|
||||
}{
|
||||
{"16x16 at 100x60", 16, 8, Size{Width: 100, Height: 60}},
|
||||
{"16x16 at 60x100", 16, 8, Size{Width: 60, Height: 100}},
|
||||
{"10x10 at 100x35", 10, 2, Size{Width: 100, Height: 35}},
|
||||
{"10x10 at 35x100", 10, 2, Size{Width: 35, Height: 100}},
|
||||
{"64x64 at 300x100", 64, 16, Size{Width: 300, Height: 100}},
|
||||
{"64x64 at 100x300", 64, 16, Size{Width: 100, Height: 300}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createSquarePNG(t, tt.side, tt.squareSide)
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatPNG, FitMode: FitCover})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
output, err := png.Decode(result.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to decode output: %v", err)
|
||||
}
|
||||
|
||||
bounds := output.Bounds()
|
||||
if bounds.Dx() != tt.size.Width || bounds.Dy() != tt.size.Height {
|
||||
t.Fatalf("Process() made %dx%d, want %dx%d",
|
||||
bounds.Dx(), bounds.Dy(), tt.size.Width, tt.size.Height)
|
||||
}
|
||||
|
||||
row := make([]float64, bounds.Dx())
|
||||
for x := range row {
|
||||
r, _, _, _ := output.At(x, bounds.Dy()/2).RGBA()
|
||||
row[x] = float64(r >> 8)
|
||||
}
|
||||
|
||||
column := make([]float64, bounds.Dy())
|
||||
for y := range column {
|
||||
r, _, _, _ := output.At(bounds.Dx()/2, y).RGBA()
|
||||
column[y] = float64(r >> 8)
|
||||
}
|
||||
|
||||
width, height := squareLength(row), squareLength(column)
|
||||
t.Logf("square is %.2f wide and %.2f high", width, height)
|
||||
|
||||
if width == 0 || math.Abs(width-height) > 0.5 {
|
||||
t.Errorf("square is %.2f wide and %.2f high, want it square",
|
||||
width, height)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -60,45 +60,3 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_FitOutsideSideIsAtLeastOne verifies that FitOutside
|
||||
// keeps each side at 1 or more when it works both sides out again from the
|
||||
// larger scale. A side of 49 scaled to 1 is 49 times 1/49, which in floating
|
||||
// point is just under 1 and so rounded down to 0, a width libvips refuses.
|
||||
func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
wantWidth, wantHeight int
|
||||
}{
|
||||
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
|
||||
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
|
||||
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
|
||||
t.Errorf("Process() made %dx%d, want %dx%d",
|
||||
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -142,6 +142,19 @@ const MaxOutputDimension = 8192
|
||||
// requested size.
|
||||
var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum")
|
||||
|
||||
// MaxAVIFOutputPixels is the most pixels an AVIF output may have, so that an
|
||||
// encode takes at most 40 seconds of the default downstream_timeout of 60
|
||||
// seconds; AVIF is already at the lowest effort govips can set (avifEffort).
|
||||
// With one thread, as pixad runs libvips, saving 4096x4096, 8192x2048 or
|
||||
// 2048x8192 random pixels with an alpha channel takes up to about 34
|
||||
// seconds, the slowest at any quality and at 8 or 16 bits per sample, and
|
||||
// 5120x4096 at 8 bits already takes about 36.
|
||||
const MaxAVIFOutputPixels = 4096 * 4096
|
||||
|
||||
// ErrAVIFOutputTooLarge is returned, before anything is decoded, when an AVIF
|
||||
// output would have more than MaxAVIFOutputPixels pixels.
|
||||
var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
|
||||
|
||||
// ErrUnsupportedOutputFormat is returned when the requested output format is
|
||||
// not supported.
|
||||
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
|
||||
@@ -210,7 +223,8 @@ func New(params Params) *ImageProcessor {
|
||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
|
||||
// no way to stop libvips once an encode has begun, so an encode keeps its
|
||||
// slot until it finishes.
|
||||
// slot until it finishes. An AVIF output of more than MaxAVIFOutputPixels is
|
||||
// refused with ErrAVIFOutputTooLarge before anything is decoded.
|
||||
func (p *ImageProcessor) Process(
|
||||
ctx context.Context,
|
||||
input io.Reader,
|
||||
@@ -223,18 +237,9 @@ func (p *ImageProcessor) Process(
|
||||
|
||||
defer release()
|
||||
|
||||
// Read input with a size limit to prevent unbounded memory consumption.
|
||||
// We read at most maxInputBytes+1 so we can detect if the input exceeds
|
||||
// the limit without consuming additional memory.
|
||||
limited := io.LimitReader(input, p.maxInputBytes+1)
|
||||
|
||||
data, err := io.ReadAll(limited)
|
||||
data, err := p.readInput(input)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read input: %w", err)
|
||||
}
|
||||
|
||||
if int64(len(data)) > p.maxInputBytes {
|
||||
return nil, ErrInputDataTooLarge
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// govips picks the libvips loader from the input's first bytes with
|
||||
@@ -277,20 +282,28 @@ func (p *ImageProcessor) Process(
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Resize if needed
|
||||
if targetWidth != origWidth || targetHeight != origHeight {
|
||||
err := p.resize(img, targetWidth, targetHeight, req.FitMode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to resize: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// orig is the source's own format; encode refuses an empty format
|
||||
outputFormat := req.Format
|
||||
if outputFormat == FormatOriginal {
|
||||
outputFormat = sourceFormat
|
||||
}
|
||||
|
||||
// Determine output dimensions; too large ones are refused before
|
||||
// resizing, which with FitCover already decodes the input
|
||||
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
|
||||
targetWidth, targetHeight, req.FitMode, outputFormat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Resize if needed
|
||||
if targetWidth != origWidth || targetHeight != origHeight {
|
||||
err := p.resize(img, outputWidth, outputHeight, req.FitMode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to resize: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Encode to target format
|
||||
output, err := p.encode(ctx, img, outputFormat, req.Quality)
|
||||
if err != nil {
|
||||
@@ -311,10 +324,8 @@ func (p *ImageProcessor) Process(
|
||||
|
||||
// targetDimensions calculates the output dimensions for a requested size,
|
||||
// scaling proportionally when only one dimension is given and keeping the
|
||||
// original dimensions when both are zero. A side scaled proportionally is
|
||||
// rounded down but kept at 1 or more, as for a thin image it would round
|
||||
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
|
||||
// which is ErrOutputTooLarge.
|
||||
// original dimensions when both are zero. A side scaled proportionally can
|
||||
// come out above MaxOutputDimension, which is ErrOutputTooLarge.
|
||||
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
||||
width, height := size.Width, size.Height
|
||||
|
||||
@@ -324,10 +335,10 @@ func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
||||
return origWidth, origHeight, nil
|
||||
case width == 0:
|
||||
// Only height specified: calculate width proportionally
|
||||
width = max(1, origWidth*height/origHeight)
|
||||
width = origWidth * height / origHeight
|
||||
case height == 0:
|
||||
// Only width specified: calculate height proportionally
|
||||
height = max(1, origHeight*width/origWidth)
|
||||
height = origHeight * width / origWidth
|
||||
}
|
||||
|
||||
if width > MaxOutputDimension || height > MaxOutputDimension {
|
||||
@@ -445,6 +456,23 @@ func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) {
|
||||
return release, nil
|
||||
}
|
||||
|
||||
// readInput reads input with a size limit to prevent unbounded memory
|
||||
// consumption, refusing more than maxInputBytes with ErrInputDataTooLarge. It
|
||||
// reads at most maxInputBytes+1 so it can detect if the input exceeds the
|
||||
// limit without consuming additional memory.
|
||||
func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
|
||||
data, err := io.ReadAll(io.LimitReader(input, p.maxInputBytes+1))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read input: %w", err)
|
||||
}
|
||||
|
||||
if int64(len(data)) > p.maxInputBytes {
|
||||
return nil, ErrInputDataTooLarge
|
||||
}
|
||||
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// detectFormat returns the format string for a govips image type.
|
||||
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
switch imageType {
|
||||
@@ -469,107 +497,89 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
}
|
||||
}
|
||||
|
||||
// resize resizes the image according to the fit mode. With FitOutside one
|
||||
// side can come out larger than requested; when it would be above
|
||||
// MaxOutputDimension, resize returns ErrOutputTooLarge without resizing.
|
||||
// outputDimensions returns the dimensions resize gives an image of imgWidth x
|
||||
// imgHeight for the target dimensions and fit mode, without resizing it, and
|
||||
// refuses an output too large for format. With FitOutside one side can come
|
||||
// out larger than the target; when it would be above MaxOutputDimension,
|
||||
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
|
||||
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
|
||||
func outputDimensions(
|
||||
imgWidth, imgHeight, width, height int, fit FitMode, format Format,
|
||||
) (int, int, error) {
|
||||
scaleW := float64(width) / float64(imgWidth)
|
||||
scaleH := float64(height) / float64(imgHeight)
|
||||
outputWidth, outputHeight := width, height
|
||||
|
||||
switch fit {
|
||||
case FitCover, FitFill, "":
|
||||
// Cover crops, and fill stretches, to exactly the target dimensions
|
||||
|
||||
case FitContain:
|
||||
// Fit within the target dimensions, maintaining aspect ratio
|
||||
scale := min(scaleW, scaleH)
|
||||
outputWidth = int(float64(imgWidth) * scale)
|
||||
outputHeight = int(float64(imgHeight) * scale)
|
||||
|
||||
case FitInside:
|
||||
// Same as contain, but only shrink
|
||||
scale := min(scaleW, scaleH, 1)
|
||||
outputWidth = int(float64(imgWidth) * scale)
|
||||
outputHeight = int(float64(imgHeight) * scale)
|
||||
|
||||
case FitOutside:
|
||||
// The smallest dimension fits, the other may exceed the target
|
||||
scale := max(scaleW, scaleH)
|
||||
outputWidth = int(float64(imgWidth) * scale)
|
||||
outputHeight = int(float64(imgHeight) * scale)
|
||||
|
||||
default:
|
||||
return 0, 0, fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||
}
|
||||
|
||||
// The side FitOutside scales past the target is held to the maximum too
|
||||
if max(outputWidth, outputHeight) > MaxOutputDimension {
|
||||
return 0, 0, fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
|
||||
outputWidth, outputHeight, MaxOutputDimension)
|
||||
}
|
||||
|
||||
if format == FormatAVIF && outputWidth*outputHeight > MaxAVIFOutputPixels {
|
||||
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
|
||||
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
|
||||
}
|
||||
|
||||
return outputWidth, outputHeight, nil
|
||||
}
|
||||
|
||||
// resize resizes the image to width x height, the dimensions
|
||||
// outputDimensions gives for the fit mode.
|
||||
func (p *ImageProcessor) resize(
|
||||
img *vips.ImageRef, width, height int, fit FitMode,
|
||||
) error {
|
||||
switch fit {
|
||||
case FitCover, "":
|
||||
// Crop the centre and resize to fill exact dimensions (default)
|
||||
return cropCentreAndResize(img, width, height)
|
||||
|
||||
case FitContain:
|
||||
// Resize to fit within dimensions, maintaining aspect ratio. Scaled
|
||||
// by the smaller scale, the short side of a thin image can round
|
||||
// down to 0, which libvips refuses as a width, so each side is at
|
||||
// least 1.
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := min(scaleW, scaleH)
|
||||
newW := max(1, int(float64(imgW)*scale))
|
||||
newH := max(1, int(float64(imgH)*scale))
|
||||
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
// Resize and crop to fill exact dimensions (default)
|
||||
return img.Thumbnail(width, height, vips.InterestingCentre)
|
||||
|
||||
case FitFill:
|
||||
// Resize to exact dimensions (may distort)
|
||||
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
|
||||
|
||||
case FitInside:
|
||||
// Same as contain, but only shrink
|
||||
if img.Width() <= width && img.Height() <= height {
|
||||
return nil // Already fits
|
||||
// outputDimensions keeps the dimensions of an image that already fits
|
||||
if img.Width() == width && img.Height() == height {
|
||||
return nil
|
||||
}
|
||||
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := min(scaleW, scaleH)
|
||||
newW := max(1, int(float64(imgW)*scale))
|
||||
newH := max(1, int(float64(imgH)*scale))
|
||||
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
|
||||
case FitOutside:
|
||||
// Resize so smallest dimension fits, may exceed target on other
|
||||
// dimension. A side scaled to 1 can come out just under 1 in
|
||||
// floating point, as 49 times 1/49 does, and round down to 0, which
|
||||
// libvips refuses as a width, so each side is at least 1.
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := max(scaleW, scaleH)
|
||||
newW := max(1, int(float64(imgW)*scale))
|
||||
newH := max(1, int(float64(imgH)*scale))
|
||||
|
||||
// The dimension that exceeds the target is held to the maximum too
|
||||
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
||||
return fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
|
||||
newW, newH, MaxOutputDimension)
|
||||
}
|
||||
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
case FitContain, FitOutside:
|
||||
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||
|
||||
default:
|
||||
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||
}
|
||||
}
|
||||
|
||||
// cropCentreAndResize scales img by the same amount on both sides, the
|
||||
// least that makes it cover width x height, and cuts the centre width x
|
||||
// height out of it, as libvips' thumbnail does with InterestingCentre.
|
||||
// thumbnail alone scales the whole image and copies the result into memory
|
||||
// before it cuts, and for a thin image that is far larger than the image or
|
||||
// the output: 8192x64 at 8192x8192 is 1048576x8192, about 24 GiB. So the
|
||||
// centre is cut from img first, widened to whole pixels of img, which adds
|
||||
// less than one pixel of img on each side, and thumbnail scales and cuts
|
||||
// only that.
|
||||
func cropCentreAndResize(img *vips.ImageRef, width, height int) error {
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
|
||||
// An image wider than the requested proportions keeps its height and
|
||||
// loses width down to imgH*width/height, any other keeps its width and
|
||||
// loses height down to imgW*height/width. Half of what is lost comes
|
||||
// off each side, rounded down to whole pixels, so the part kept holds
|
||||
// all of the centre.
|
||||
left, top := 0, 0
|
||||
if imgW*height > imgH*width {
|
||||
left = (imgW*height - imgH*width) / (2 * height) //nolint:mnd // each side
|
||||
} else {
|
||||
top = (imgH*width - imgW*height) / (2 * width) //nolint:mnd // each side
|
||||
}
|
||||
|
||||
err := img.ExtractArea(left, top, imgW-2*left, imgH-2*top)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return img.Thumbnail(width, height, vips.InterestingCentre)
|
||||
}
|
||||
|
||||
const defaultQuality = 85
|
||||
|
||||
// encode encodes an image to the specified format. govips gives pixa no way
|
||||
@@ -667,11 +677,16 @@ func exportGIF(img *vips.ImageRef) ([]byte, error) {
|
||||
return output, err
|
||||
}
|
||||
|
||||
// webpEffort is libvips' default WebP effort, from 0 (fastest) to 6.
|
||||
const webpEffort = 4
|
||||
// webpEffort is the WebP effort, from 0 (fastest) to 6. It is below libvips'
|
||||
// default of 4 so that an encode takes at most 40 seconds of the default
|
||||
// downstream_timeout of 60 seconds. With one thread, as pixad runs libvips,
|
||||
// saving an 8192x8192 image of random pixels with an alpha channel takes up
|
||||
// to about 23 seconds at 2, and up to about 16 without one, at any quality;
|
||||
// with an alpha channel, 3 takes about 50 seconds and 4 about 61.
|
||||
const webpEffort = 2
|
||||
|
||||
// exportWebP encodes img as lossy WebP at quality and libvips' default
|
||||
// effort, without metadata.
|
||||
// exportWebP encodes img as lossy WebP at quality and webpEffort, without
|
||||
// metadata.
|
||||
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
output, _, err := img.ExportWebp(&vips.WebpExportParams{
|
||||
StripMetadata: true,
|
||||
@@ -683,12 +698,13 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
}
|
||||
|
||||
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
|
||||
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51
|
||||
// seconds to save an 8192x8192 RGB image of random pixels, and about 110
|
||||
// with an alpha channel, the worst case, against the default
|
||||
// downstream_timeout of 60 seconds. On an image of milder noise, which 1
|
||||
// saves in about 12 seconds, 2 takes nearly a minute and libvips' default,
|
||||
// 4, takes minutes.
|
||||
// govips can set. With one thread, as pixad runs libvips, 1 takes about 52
|
||||
// seconds to save an 8192x8192 RGB image of random pixels at quality 85,
|
||||
// and about 96 with an alpha channel, against the default
|
||||
// downstream_timeout of 60 seconds, so an AVIF output is held to
|
||||
// MaxAVIFOutputPixels. On an image of milder noise, which 1 saves in about
|
||||
// 12 seconds, 2 takes nearly a minute and libvips' default, 4, takes
|
||||
// minutes.
|
||||
const avifEffort = 1
|
||||
|
||||
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
|
||||
@@ -715,10 +731,18 @@ func exportAVIF(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
// JPEG that names none.
|
||||
const jxlResolution = 72 / 25.4
|
||||
|
||||
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort
|
||||
// and without metadata. govips sends libvips a distance, the JPEG XL
|
||||
// encoder's own measure of quality, along with the quality, and libvips then
|
||||
// uses the distance alone, so the quality is also given as a distance.
|
||||
// jxlEffort is the JPEG XL effort, from 1 (fastest) to 9. It is below
|
||||
// libvips' default of 7 so that an encode takes at most 40 seconds of the
|
||||
// default downstream_timeout of 60 seconds. With one thread, as pixad runs
|
||||
// libvips, saving an 8192x8192 image of random pixels with an alpha channel
|
||||
// takes up to about 27 seconds at 4, and up to about 13 without one, at any
|
||||
// quality; with an alpha channel, 5 takes about 43 seconds and 7 about 81.
|
||||
const jxlEffort = 4
|
||||
|
||||
// exportJXL encodes img as JPEG XL at quality and jxlEffort, without
|
||||
// metadata. govips sends libvips a distance, the JPEG XL encoder's own
|
||||
// measure of quality, along with the quality, and libvips then uses the
|
||||
// distance alone, so the quality is also given as a distance.
|
||||
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
|
||||
// PNG, but cannot save one as JPEG XL. encode has already converted
|
||||
@@ -757,6 +781,7 @@ func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
params := vips.NewJxlExportParams()
|
||||
params.Quality = quality
|
||||
params.Distance = jxlDistance(quality)
|
||||
params.Effort = jxlEffort
|
||||
|
||||
output, _, err := toSave.ExportJxl(params)
|
||||
if err != nil {
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/url"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -87,11 +86,9 @@ func ValidateImageRequest(req *ImageRequest) error {
|
||||
type ImageRequest struct {
|
||||
// SourceHost is the origin host (e.g., "cdn.example.com")
|
||||
SourceHost string
|
||||
// SourcePath is the path on the origin with its percent-escapes decoded,
|
||||
// as url.URL's Path holds it (e.g., "/photos/my cat.jpg")
|
||||
// SourcePath is the path on the origin (e.g., "/photos/cat.jpg")
|
||||
SourcePath string
|
||||
// SourceQuery is the optional query string for the origin URL as the URL
|
||||
// writes it, percent-escapes included, as url.URL's RawQuery holds it
|
||||
// SourceQuery is the optional query string for the origin URL
|
||||
SourceQuery string
|
||||
// Size is the requested output dimensions
|
||||
Size Size
|
||||
@@ -110,25 +107,20 @@ type ImageRequest struct {
|
||||
AllowHTTP bool
|
||||
}
|
||||
|
||||
// SourceURL returns the full upstream URL to fetch: the source path
|
||||
// percent-encoded again, as net/url writes a path, so each source path is one
|
||||
// upstream path, and the source query as it is, with the characters
|
||||
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
|
||||
// true, otherwise https://.
|
||||
// SourceURL returns the full upstream URL to fetch.
|
||||
// Uses http:// scheme when AllowHTTP is true, otherwise https://.
|
||||
func (r *ImageRequest) SourceURL() string {
|
||||
scheme := "https"
|
||||
if r.AllowHTTP {
|
||||
scheme = "http"
|
||||
}
|
||||
|
||||
sourceURL := url.URL{
|
||||
Scheme: scheme,
|
||||
Host: r.SourceHost,
|
||||
Path: r.SourcePath,
|
||||
RawQuery: escapeQuery(r.SourceQuery),
|
||||
url := scheme + "://" + r.SourceHost + r.SourcePath
|
||||
if r.SourceQuery != "" {
|
||||
url += "?" + r.SourceQuery
|
||||
}
|
||||
|
||||
return sourceURL.String()
|
||||
return url
|
||||
}
|
||||
|
||||
// ImageResponse represents a processed image ready to serve
|
||||
|
||||
@@ -3,7 +3,6 @@ package imgcache
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strconv"
|
||||
@@ -29,9 +28,7 @@ var (
|
||||
const MaxDimension = imageprocessor.MaxOutputDimension
|
||||
|
||||
// ValidateDimension checks one requested width or height; name ("width" or
|
||||
// "height") appears in the error. 0 is valid: that side is then worked out from
|
||||
// the source's proportions and the other requested side, rounded down but at
|
||||
// least 1, or is the source's own side when the other side is 0 too.
|
||||
// "height") appears in the error. 0 means "original size" and is valid.
|
||||
func ValidateDimension(name string, value int) error {
|
||||
if value < 0 {
|
||||
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
|
||||
@@ -53,21 +50,6 @@ var sizeFormatRegex = regexp.MustCompile(
|
||||
// percentEscapeRegex matches one percent-escape, such as "%2e"
|
||||
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
|
||||
|
||||
// queryEscapeRegex matches a percent-escape, or one character that a browser
|
||||
// percent-encodes in the query of an http or https URL: one outside the
|
||||
// printable ASCII characters "!" to "~", such as a space, a control character
|
||||
// or a character outside ASCII, or one of `"#'<>`. A "%" that does not begin a
|
||||
// percent-escape matches too.
|
||||
var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#'<>%]`)
|
||||
|
||||
// hostRegex matches the forms the host of an image URL may take: a host name
|
||||
// or an IPv4 address (letters, digits, "-", "_" and "."), or an IPv6 address
|
||||
// in brackets (hex digits, ":" and "."), either one optionally followed by ":"
|
||||
// and a port, which may be empty. Its one group holds the text in brackets,
|
||||
// which parseImageComponents also checks is an IPv6 address.
|
||||
var hostRegex = regexp.MustCompile(
|
||||
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
|
||||
|
||||
// ParsedURL contains the parsed components of an image proxy URL.
|
||||
type ParsedURL struct {
|
||||
// Host is the source origin host (e.g., "cdn.example.com")
|
||||
@@ -159,20 +141,10 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||
return nil, ErrMissingHost
|
||||
}
|
||||
|
||||
// Only the forms hostRegex describes are accepted. A "?", "#" or "@" in
|
||||
// the host would make Go's URL parser, which the allowlist check and the
|
||||
// fetcher use, read a different host from the one allowlisted and cached
|
||||
hostMatch := hostRegex.FindStringSubmatch(host)
|
||||
if hostMatch == nil {
|
||||
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
|
||||
}
|
||||
|
||||
// Go's URL parser refuses brackets around anything but an IPv6 address
|
||||
if inBrackets := hostMatch[1]; inBrackets != "" {
|
||||
addr, err := netip.ParseAddr(inBrackets)
|
||||
if err != nil || !addr.Is6() {
|
||||
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
|
||||
}
|
||||
// A URL reads what comes before "@" as a user name and password, and the
|
||||
// rest as the host
|
||||
if strings.Contains(host, "@") {
|
||||
return nil, fmt.Errorf("%w: %q holds @", ErrInvalidHost, host)
|
||||
}
|
||||
|
||||
// Host names do not depend on letter case, so the allowlist, the upstream
|
||||
@@ -180,20 +152,9 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||
// host
|
||||
host = strings.ToLower(host)
|
||||
|
||||
// Go's URL parser, which the fetcher uses, reads a "#" as the start of a
|
||||
// fragment and drops the rest, so each text after it would be another
|
||||
// spelling, cached under its own keys, of one upstream request
|
||||
if strings.Contains(path, "#") {
|
||||
return nil, fmt.Errorf(
|
||||
`%w: the source path or query holds a "#"`, ErrInvalidPath)
|
||||
}
|
||||
|
||||
// Extract query string if present in path. It is the source URL's query
|
||||
// as that URL writes it. A character a URL's query cannot hold as it is,
|
||||
// such as a space written %20 in the image URL, is percent-encoded here,
|
||||
// so the query is signed and cached as it is sent upstream
|
||||
// Extract query string if present in path
|
||||
if qIndex := strings.Index(path, "?"); qIndex != -1 {
|
||||
query = escapeQuery(path[qIndex+1:])
|
||||
query = path[qIndex+1:]
|
||||
path = path[:qIndex]
|
||||
}
|
||||
|
||||
@@ -282,20 +243,6 @@ func decodePercentEscapes(s string) string {
|
||||
})
|
||||
}
|
||||
|
||||
// escapeQuery returns a source query as a URL's query holds it: each
|
||||
// character that queryEscapeRegex names, such as a space, percent-encoded,
|
||||
// and the rest, the query's percent-escapes included, kept as they are. The
|
||||
// result is unchanged by a second escapeQuery.
|
||||
func escapeQuery(query string) string {
|
||||
return queryEscapeRegex.ReplaceAllStringFunc(query, func(match string) string {
|
||||
if percentEscapeRegex.MatchString(match) {
|
||||
return match
|
||||
}
|
||||
|
||||
return url.PathEscape(match)
|
||||
})
|
||||
}
|
||||
|
||||
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
||||
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
||||
matches := sizeFormatRegex.FindStringSubmatch(s)
|
||||
|
||||
@@ -14,12 +14,10 @@ import (
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxtest"
|
||||
@@ -185,57 +183,6 @@ func startImageProxy(
|
||||
return s, db.DB(), stateDir
|
||||
}
|
||||
|
||||
// startImageProxyWithBlockedNetworks starts the components pixad's fx app
|
||||
// builds, from a config with a fresh state directory and blockedNetworks as
|
||||
// blocked_networks. Unlike startImageProxy it provides no upstream fetcher, so
|
||||
// the handlers build the one pixad uses, which checks every address it
|
||||
// connects to against the built-in blocked ranges and blocked_networks. It
|
||||
// returns the server with its routes.
|
||||
func startImageProxyWithBlockedNetworks(
|
||||
t *testing.T, blockedNetworks []netip.Prefix,
|
||||
) *Server {
|
||||
t.Helper()
|
||||
|
||||
stateDir := t.TempDir()
|
||||
cfg := &config.Config{
|
||||
SigningKey: testSigningKey,
|
||||
StateDir: stateDir,
|
||||
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||
BlockedNetworks: blockedNetworks,
|
||||
// A fetch that gets past the checks and finds nothing at the address
|
||||
// fails within this time at the latest.
|
||||
UpstreamFetchTimeout: 2 * time.Second,
|
||||
// With no connection slots, the fetch would fail before connecting.
|
||||
UpstreamConnections: config.DefaultUpstreamConnections,
|
||||
DownstreamTimeout: config.DefaultDownstreamTimeout,
|
||||
}
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
mw *middleware.Middleware
|
||||
)
|
||||
|
||||
app := fxtest.New(t,
|
||||
fx.Supply(cfg),
|
||||
fx.Provide(
|
||||
globals.New,
|
||||
logger.New,
|
||||
database.New,
|
||||
healthcheck.New,
|
||||
handlers.New,
|
||||
middleware.New,
|
||||
),
|
||||
fx.Populate(&h, &mw),
|
||||
)
|
||||
app.RequireStart()
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
s := &Server{config: cfg, mw: mw, h: h}
|
||||
s.SetupRoutes()
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// getImage sends a GET for target to s and fails unless it answers 200.
|
||||
func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
|
||||
@@ -1,665 +0,0 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// percentSignPath is a source path holding a "%", which a URL writes %25.
|
||||
const percentSignPath = "/100%25.png"
|
||||
|
||||
// generateFormPattern extracts the CSRF token rendered into the generator
|
||||
// page's form.
|
||||
var generateFormPattern = regexp.MustCompile(
|
||||
`(?s)action="/generate".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
|
||||
|
||||
// generatedURLPattern extracts the path of the encrypted URL the generator
|
||||
// page shows.
|
||||
var generatedURLPattern = regexp.MustCompile(`value="[^"]*(/v1/e/[^"]+)"`)
|
||||
|
||||
// recordingUpstream is a test upstream that answers every request with a PNG
|
||||
// and records the request target, the path and query as sent, of each request
|
||||
// it receives.
|
||||
type recordingUpstream struct {
|
||||
*httptest.Server
|
||||
|
||||
mu sync.Mutex
|
||||
received []string
|
||||
}
|
||||
|
||||
// startRecordingUpstream starts a recordingUpstream, which is closed when the
|
||||
// test ends.
|
||||
func startRecordingUpstream(t *testing.T) *recordingUpstream {
|
||||
t.Helper()
|
||||
|
||||
source := encodeTestPNG(t, 64, 48)
|
||||
upstream := &recordingUpstream{}
|
||||
|
||||
upstream.Server = httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
upstream.mu.Lock()
|
||||
upstream.received = append(upstream.received, r.RequestURI)
|
||||
upstream.mu.Unlock()
|
||||
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
_, _ = w.Write(source)
|
||||
}))
|
||||
t.Cleanup(upstream.Close)
|
||||
|
||||
return upstream
|
||||
}
|
||||
|
||||
// requestTargets returns the request targets the upstream has received, in
|
||||
// the order it received them.
|
||||
func (u *recordingUpstream) requestTargets() []string {
|
||||
u.mu.Lock()
|
||||
defer u.mu.Unlock()
|
||||
|
||||
return slices.Clone(u.received)
|
||||
}
|
||||
|
||||
// pageOrigin is the origin of the login and generator pages below, opened
|
||||
// over HTTPS; httptest gives every request the host example.com.
|
||||
const pageOrigin = "https://example.com"
|
||||
|
||||
// openRoot sends GET / to s over HTTPS, with cookie when it is not nil, and
|
||||
// returns the page: the login page, or the generator page for a login session.
|
||||
func openRoot(
|
||||
t *testing.T, s *Server, cookie *http.Cookie,
|
||||
) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||
pageOrigin+"/", nil)
|
||||
if cookie != nil {
|
||||
req.AddCookie(cookie)
|
||||
}
|
||||
|
||||
page := httptest.NewRecorder()
|
||||
s.ServeHTTP(page, req)
|
||||
|
||||
return page
|
||||
}
|
||||
|
||||
// submitForm posts form to path on s as a browser submits it from page over
|
||||
// HTTPS: with the CSRF token tokenPattern finds in page, the cookies page set,
|
||||
// cookie when it is not nil, and a Referer naming the page, which pixa needs
|
||||
// while debug is off, as it is in startImageProxy's config.
|
||||
func submitForm(
|
||||
t *testing.T, s *Server, page *httptest.ResponseRecorder,
|
||||
tokenPattern *regexp.Regexp, path string, form url.Values,
|
||||
cookie *http.Cookie,
|
||||
) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
|
||||
match := tokenPattern.FindStringSubmatch(page.Body.String())
|
||||
if match == nil {
|
||||
t.Fatalf("the page has no form for %s with a CSRF token", path)
|
||||
}
|
||||
|
||||
form.Set("gorilla.csrf.Token", match[1])
|
||||
|
||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodPost,
|
||||
pageOrigin+path, strings.NewReader(form.Encode()))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
req.Header.Set("Referer", pageOrigin+"/")
|
||||
|
||||
for _, c := range page.Result().Cookies() {
|
||||
req.AddCookie(c)
|
||||
}
|
||||
|
||||
if cookie != nil {
|
||||
req.AddCookie(cookie)
|
||||
}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, req)
|
||||
|
||||
return rec
|
||||
}
|
||||
|
||||
// postGenerate logs in to s with the signing key, submits the generator page's
|
||||
// form with sourceURL, and returns the response.
|
||||
func postGenerate(
|
||||
t *testing.T, s *Server, sourceURL string,
|
||||
) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
|
||||
login := submitForm(t, s, openRoot(t, s, nil), csrfFieldPattern, "/",
|
||||
url.Values{"key": {testSigningKey}}, nil)
|
||||
|
||||
var sessionCookie *http.Cookie
|
||||
|
||||
for _, c := range login.Result().Cookies() {
|
||||
if c.Name == session.CookieName {
|
||||
sessionCookie = c
|
||||
}
|
||||
}
|
||||
|
||||
if sessionCookie == nil {
|
||||
t.Fatalf("logging in answered %d and set no session cookie", login.Code)
|
||||
}
|
||||
|
||||
rec := submitForm(t, s, openRoot(t, s, sessionCookie), generateFormPattern,
|
||||
"/generate", url.Values{"url": {sourceURL}}, sessionCookie)
|
||||
t.Logf("POST /generate with url %s: %d", sourceURL, rec.Code)
|
||||
|
||||
return rec
|
||||
}
|
||||
|
||||
// TestImageURLFetchesItsSourcePathAndQuery requests image URLs whose source
|
||||
// query holds a space, written percent-encoded, or a "'", and whose source
|
||||
// path holds a "%", written percent-encoded, and checks the one request the
|
||||
// upstream receives: the source path percent-encoded again, and the source
|
||||
// query with the space written %20 and the "'" written %27, as a browser
|
||||
// writes them in a query.
|
||||
func TestImageURLFetchesItsSourcePathAndQuery(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
// pathAndQuery is the source path and query as the image URL writes
|
||||
// them
|
||||
pathAndQuery string
|
||||
// wantRequest is the path and query the upstream must receive
|
||||
wantRequest string
|
||||
}{
|
||||
{"space in the query", "/photo.png%3Fname=my%20cat",
|
||||
"/photo.png?name=my%20cat"},
|
||||
{"apostrophe in the query", "/photo.png%3Fname=it's",
|
||||
"/photo.png?name=it%27s"},
|
||||
{"percent sign in the path", percentSignPath, percentSignPath},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
getImage(t, s, "/v1/image/"+upstreamHost+tt.pathAndQuery+"/orig.png")
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
if !slices.Equal(got, []string{tt.wantRequest}) {
|
||||
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart requests twice each
|
||||
// of the source paths /my cat.png and /my%20cat.png, which the image URL
|
||||
// writes my%20cat.png and my%2520cat.png. Each is fetched once, as a request
|
||||
// of its own, and its second request is served from the cache.
|
||||
func TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
imageURLs := []string{
|
||||
"/v1/image/" + upstreamHost + "/my%20cat.png/orig.png",
|
||||
"/v1/image/" + upstreamHost + "/my%2520cat.png/orig.png",
|
||||
}
|
||||
|
||||
for _, imageURL := range imageURLs {
|
||||
getImage(t, s, imageURL)
|
||||
}
|
||||
|
||||
for _, imageURL := range imageURLs {
|
||||
cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache")
|
||||
if cacheStatus != string(imgcache.CacheHit) {
|
||||
t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT",
|
||||
imageURL, cacheStatus)
|
||||
}
|
||||
}
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
want := []string{"/my%20cat.png", "/my%2520cat.png"}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("upstream received %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest requests the source query
|
||||
// name=my%20cat written as the README says, name=my%2520cat, and with the
|
||||
// space written %20, name=my%20cat. Both are one source query: the upstream
|
||||
// receives one request, and the second image URL is served from the cache.
|
||||
func TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
getImage(t, s, "/v1/image/"+upstreamHost+"/photo.png%3Fname=my%2520cat/orig.png")
|
||||
|
||||
second := getImage(t, s,
|
||||
"/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png")
|
||||
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
|
||||
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
||||
}
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
want := []string{"/photo.png?name=my%20cat"}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("upstream received %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGeneratedURLFetchesTheSourceURLAsEntered makes an encrypted URL on the
|
||||
// generator page for source URLs holding %20, %25 and %23 in the path or the
|
||||
// query, requests it, and checks that the upstream receives the path and
|
||||
// query exactly as entered.
|
||||
func TestGeneratedURLFetchesTheSourceURLAsEntered(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// The names hold no "%": each test's state directory is named after it
|
||||
// and goes into the database URL, which decodes percent-escapes
|
||||
tests := []struct {
|
||||
name string
|
||||
pathAndQuery string
|
||||
}{
|
||||
{"space in the path", "/my%20cat.png"},
|
||||
{"percent sign in the path", percentSignPath},
|
||||
{"number sign in the path", "/a%23b.png"},
|
||||
{"all three in the query", "/photo.png?name=my%20cat%25%23"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
rec := postGenerate(t, s, "https://"+upstreamHost+tt.pathAndQuery)
|
||||
|
||||
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
||||
if rec.Code != http.StatusOK || match == nil {
|
||||
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
|
||||
rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
getImage(t, s, match[1])
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
if !slices.Equal(got, []string{tt.pathAndQuery}) {
|
||||
t.Errorf("upstream received %q, want [%q]", got, tt.pathAndQuery)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL enters two
|
||||
// spellings of one source URL on the generator page, requests the encrypted
|
||||
// URL made for each, and checks that the upstream receives one request and
|
||||
// that the second encrypted URL is served from the cache. One pair writes the
|
||||
// host in upper and in lower case, which /v1/image/ lower-cases. The host is
|
||||
// an IPv6 address, whose hex digits have letter case, because the fetcher
|
||||
// would look a host name up in DNS. It has a port, as the fetcher refuses an
|
||||
// IPv6 address written without one. The other pair is a source URL with no
|
||||
// path and with the path "/", both of which fetch "/".
|
||||
func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
// sourceURLs are the two spellings, after "https://"
|
||||
sourceURLs []string
|
||||
// wantRequest is the path the upstream must receive
|
||||
wantRequest string
|
||||
}{
|
||||
{"host in upper and in lower case",
|
||||
[]string{"[2001:DB8::A]:8443/a.png", "[2001:db8::a]:8443/a.png"},
|
||||
"/a.png"},
|
||||
{"no path and the path written as a slash",
|
||||
[]string{upstreamHost, upstreamHost + "/"}, "/"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
var imageURLs []string
|
||||
|
||||
for _, sourceURL := range tt.sourceURLs {
|
||||
rec := postGenerate(t, s, "https://"+sourceURL)
|
||||
|
||||
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
||||
if rec.Code != http.StatusOK || match == nil {
|
||||
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
|
||||
rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
imageURLs = append(imageURLs, match[1])
|
||||
}
|
||||
|
||||
getImage(t, s, imageURLs[0])
|
||||
|
||||
second := getImage(t, s, imageURLs[1])
|
||||
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
|
||||
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
||||
}
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
if !slices.Equal(got, []string{tt.wantRequest}) {
|
||||
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGeneratorRefusesSourceURLItCannotFetchAsWritten submits source URLs the
|
||||
// generator page must make no encrypted URL for, and checks that it answers
|
||||
// 400 and shows why. pixa keeps the source path decoded and fetches it
|
||||
// percent-encoded again, so a path holding %2F would be fetched with a "/"
|
||||
// there, whatever else the path holds. pixa sends no user name or password,
|
||||
// and no "?" with no query after it. For these the page shows the URL pixa
|
||||
// would fetch. A source path holding a ".." segment is refused, as
|
||||
// /v1/image/ refuses it.
|
||||
func TestGeneratorRefusesSourceURLItCannotFetchAsWritten(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// The test config sets allow_http, so pixa would fetch over plain HTTP
|
||||
const fetchedFrom = "http://" + upstreamHost
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
// sourceURL is the source URL after "https://"
|
||||
sourceURL string
|
||||
// wantShown is what the page must show: the URL pixa would fetch, or
|
||||
// why it makes no encrypted URL
|
||||
wantShown string
|
||||
}{
|
||||
{"encoded slash", upstreamHost + "/a%2Fb.png",
|
||||
fetchedFrom + "/a/b.png"},
|
||||
{"encoded slash and a space", upstreamHost + "/a%2Fb c.png",
|
||||
fetchedFrom + "/a/b%20c.png"},
|
||||
{"encoded slash and a bar", upstreamHost + "/a%2Fb|c.png",
|
||||
fetchedFrom + "/a/b%7Cc.png"},
|
||||
{"encoded slash and a letter outside ASCII", upstreamHost + "/café%2Fx.png",
|
||||
fetchedFrom + "/caf%C3%A9/x.png"},
|
||||
{"user name and password", "user:pass@" + upstreamHost + "/a.png",
|
||||
fetchedFrom + "/a.png"},
|
||||
{"question mark with no query", upstreamHost + "/a.png?",
|
||||
fetchedFrom + "/a.png"},
|
||||
{"two-dot segment", upstreamHost + "/a/../b.png",
|
||||
"cannot hold a .. segment"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startImageProxy(t, startRecordingUpstream(t).Server)
|
||||
|
||||
rec := postGenerate(t, s, "https://"+tt.sourceURL)
|
||||
body := rec.Body.String()
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
|
||||
if strings.Contains(body, "/v1/e/") {
|
||||
t.Error("the page shows an encrypted URL")
|
||||
}
|
||||
|
||||
if !strings.Contains(body, tt.wantShown) {
|
||||
t.Errorf("the page does not show %s; body %s", tt.wantShown, body)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ipv6SourcePath is the source path of the image URLs signedImageURL and
|
||||
// encryptedImageURL make.
|
||||
const ipv6SourcePath = "/ipv6.png"
|
||||
|
||||
// TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort requests ipv6SourcePath
|
||||
// on an IPv6 source host, written with and without a port, through a
|
||||
// /v1/image/ URL with a signature and through an encrypted URL. The public
|
||||
// address 2001:db8::a, a documentation address (RFC 3849) the fetcher accepts
|
||||
// as public, answers 200, and the upstream receives the request for
|
||||
// ipv6SourcePath. The private address fd00::1 answers 403, and the upstream
|
||||
// receives no request: the test's fetcher connects every address to the test
|
||||
// upstream, so only the fetcher's check of the source URL can refuse it.
|
||||
func TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
sourceHost string
|
||||
wantStatus int
|
||||
// wantRequests are the request targets the upstream must receive
|
||||
wantRequests []string
|
||||
}{
|
||||
{"public address without a port", "[2001:db8::a]",
|
||||
http.StatusOK, []string{ipv6SourcePath}},
|
||||
{"public address with a port", "[2001:db8::a]:8443",
|
||||
http.StatusOK, []string{ipv6SourcePath}},
|
||||
{"private address without a port", "[fd00::1]",
|
||||
http.StatusForbidden, nil},
|
||||
{"private address with a port", "[fd00::1]:8443",
|
||||
http.StatusForbidden, nil},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
imageURLs := map[string]string{
|
||||
"image URL with a signature": signedImageURL(t, tt.sourceHost),
|
||||
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
|
||||
}
|
||||
|
||||
for route, imageURL := range imageURLs {
|
||||
t.Run(tt.name+" "+route, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, imageURL, nil))
|
||||
t.Logf("GET %s: %d", imageURL, rec.Code)
|
||||
|
||||
if rec.Code != tt.wantStatus {
|
||||
t.Errorf("status = %d, want %d; body %s",
|
||||
rec.Code, tt.wantStatus, rec.Body.String())
|
||||
}
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
if !slices.Equal(got, tt.wantRequests) {
|
||||
t.Errorf("upstream received %q, want %q", got, tt.wantRequests)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// signedImageURL returns the /v1/image/ URL of ipv6SourcePath on sourceHost at
|
||||
// its original size as PNG, with a signature made with the test signing key,
|
||||
// which a host not on the allowlist needs. The URL has no q or fit, so it is
|
||||
// signed with the quality 85 and the fit cover.
|
||||
func signedImageURL(t *testing.T, sourceHost string) string {
|
||||
t.Helper()
|
||||
|
||||
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
|
||||
&signature.Request{
|
||||
SourceHost: sourceHost,
|
||||
SourcePath: ipv6SourcePath,
|
||||
Format: string(imgcache.FormatPNG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
}, time.Hour)
|
||||
|
||||
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
|
||||
}
|
||||
|
||||
// encryptedImageURL returns the encrypted URL of ipv6SourcePath on sourceHost
|
||||
// at its original size as PNG, made with the test signing key.
|
||||
func encryptedImageURL(t *testing.T, sourceHost string) string {
|
||||
t.Helper()
|
||||
|
||||
generator, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("encurl.NewGenerator: %v", err)
|
||||
}
|
||||
|
||||
token, err := generator.Generate(&encurl.Payload{
|
||||
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
|
||||
Format: imgcache.FormatPNG,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encurl Generate: %v", err)
|
||||
}
|
||||
|
||||
return "/v1/e/" + token + "/img.png"
|
||||
}
|
||||
|
||||
// TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address requests
|
||||
// ipv6SourcePath on IPv6 source hosts that hold a loopback or private IPv4
|
||||
// address in their last four bytes, in the IPv4-compatible form (::/96) or the
|
||||
// NAT64 form (64:ff9b::/96), through a /v1/image/ URL with a signature and
|
||||
// through an encrypted URL. Each answers 403, and the upstream receives no
|
||||
// request: the test's fetcher connects every address to the test upstream, so
|
||||
// only the fetcher's check of the source URL can refuse it.
|
||||
func TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
sourceHosts := []string{
|
||||
"[::127.0.0.1]", // IPv4-compatible form of a loopback address
|
||||
"[::10.0.0.1]", // IPv4-compatible form of a private address
|
||||
"[64:ff9b::7f00:1]", // NAT64 form of 127.0.0.1
|
||||
}
|
||||
|
||||
for _, sourceHost := range sourceHosts {
|
||||
imageURLs := []string{
|
||||
signedImageURL(t, sourceHost),
|
||||
encryptedImageURL(t, sourceHost),
|
||||
}
|
||||
|
||||
for _, imageURL := range imageURLs {
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, imageURL, nil))
|
||||
t.Logf("GET %s (source host %s): %d", imageURL, sourceHost, rec.Code)
|
||||
|
||||
if rec.Code != http.StatusForbidden {
|
||||
t.Errorf("source host %s: status = %d, want %d; body %s",
|
||||
sourceHost, rec.Code, http.StatusForbidden, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if got := upstream.requestTargets(); len(got) != 0 {
|
||||
t.Errorf("upstream received %q, want no request", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageRoutesCheckIPv4AddressInIPv6SourceHost requests ipv6SourcePath on
|
||||
// source hosts that hold an IPv4 address, through a /v1/image/ URL with a
|
||||
// signature and through an encrypted URL, with the fetcher pixad uses.
|
||||
// [64:ff9b:1::a00:1], in the local-use NAT64 range (64:ff9b:1::/48, RFC 8215),
|
||||
// answers 403. [::203.0.113.5] holds the public address 203.0.113.5 in the
|
||||
// IPv4-compatible form (::/96) and answers 403 when blocked_networks lists
|
||||
// 203.0.113.0/24 or ::/96. [::ffff:203.0.113.5] holds it in the IPv4-mapped
|
||||
// form (::ffff:0:0/96) and answers 403 when blocked_networks lists
|
||||
// 203.0.113.0/24. Go keeps the plain address 203.0.113.5 in the mapped form
|
||||
// too, and an IPv6 range must not refuse it: with ::/0 listed it is not
|
||||
// refused. Neither is [::203.0.113.5] with blocked_networks empty. Where a
|
||||
// host is not refused, the fetcher tries to connect to it, finds nothing
|
||||
// there, and the request fails with another status.
|
||||
func TestImageRoutesCheckIPv4AddressInIPv6SourceHost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
ipv4CompatibleHost = "[::203.0.113.5]"
|
||||
ipv4MappedHost = "[::ffff:203.0.113.5]"
|
||||
)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
sourceHost string
|
||||
blockedNetworks []netip.Prefix
|
||||
wantForbidden bool
|
||||
}{
|
||||
{"local-use NAT64 form of 10.0.0.1", "[64:ff9b:1::a00:1]", nil, true},
|
||||
{"IPv4-compatible form of an address in blocked_networks",
|
||||
ipv4CompatibleHost,
|
||||
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
|
||||
{"IPv4-compatible form with ::/96 in blocked_networks",
|
||||
ipv4CompatibleHost,
|
||||
[]netip.Prefix{netip.MustParsePrefix("::/96")}, true},
|
||||
{"IPv4-mapped form of an address in blocked_networks",
|
||||
ipv4MappedHost,
|
||||
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
|
||||
{"plain IPv4 address with ::/0 in blocked_networks",
|
||||
"203.0.113.5",
|
||||
[]netip.Prefix{netip.MustParsePrefix("::/0")}, false},
|
||||
{"IPv4-compatible form of an address with blocked_networks empty",
|
||||
ipv4CompatibleHost, nil, false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
imageURLs := map[string]string{
|
||||
"image URL with a signature": signedImageURL(t, tt.sourceHost),
|
||||
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
|
||||
}
|
||||
|
||||
for route, imageURL := range imageURLs {
|
||||
t.Run(tt.name+" "+route, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startImageProxyWithBlockedNetworks(t, tt.blockedNetworks)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, imageURL, nil))
|
||||
t.Logf("GET %s: %d", imageURL, rec.Code)
|
||||
|
||||
forbidden := rec.Code == http.StatusForbidden
|
||||
if tt.wantForbidden && !forbidden {
|
||||
t.Errorf("status = %d, want %d; body %s",
|
||||
rec.Code, http.StatusForbidden, rec.Body.String())
|
||||
}
|
||||
|
||||
if !tt.wantForbidden && forbidden {
|
||||
t.Errorf("status = %d, want any other status; body %s",
|
||||
rec.Code, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user