Compare commits

..
1 Commits
Author SHA1 Message Date
sneak c0560cae4d Support SVG sources (closes #68)
check / check (push) Canceled after 0s
The fetcher accepts image/svg+xml again and internal/magic detects SVG.
A raster format renders the SVG at the least size that covers the
output, held to 8192 on a side, through libvips' thumbnail, which
renders at any scale, so an enlarged SVG stays sharp. An SVG's own size
is not limited; one with a side more than 8192 times the other is
ErrSVGTooThin, answered 415. orig of an SVG at the orig size is the SVG
unchanged, cached as a variant; at any other size it is
ErrSVGNotOriginalSize, answered 400. The cache reads up to 1024 first
bytes of a file to find its format. Render time and memory grow with
what the SVG draws: P1 in TODO.md. Tests that pinned SVG as refused now
expect it to be read.

Model: opus-5-5
2026-11-11 10:08:22 +00:00
6 changed files with 199 additions and 65 deletions

No files matched your search

+22 -14
View File
@@ -147,7 +147,7 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1 times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
use. use. Rendering an SVG source can need more, depending on what it draws.
- **Temporary files:** while pixa processes a large source, libvips writes its - **Temporary files:** while pixa processes a large source, libvips writes its
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
written, so a listing of `/tmp` does not show it, but it keeps its space, on written, so a listing of `/tmp` does not show it, but it keeps its space, on
@@ -328,6 +328,8 @@ path under `/v1/` answers 200, in maintenance mode too.
format its `Content-Type` names. format its `Content-Type` names.
- 415 when the upstream sends an image whose first bytes libvips reads as - 415 when the upstream sends an image whose first bytes libvips reads as
another format. another format.
- 415 for an SVG source with one side more than 8192 times the other, in any
format but `orig`.
- 502 when the upstream answers with an error status. - 502 when the upstream answers with an error status.
- 502 for 5 minutes after that for the same source URL, when pixa has - 502 for 5 minutes after that for the same source URL, when pixa has
neither the image nor its source image in its cache. neither the image nor its source image in its cache.
@@ -472,9 +474,11 @@ format, such as an SVG whose first bytes pass the check for AVIF, is refused
with 415 too. with 415 too.
For any format but `orig`, pixa renders an SVG source at the requested size, so For any format but `orig`, pixa renders an SVG source at the requested size, so
an enlarged SVG stays sharp. At the `orig` size it is rendered at its own size, an enlarged SVG stays sharp. At the `orig` size it is rendered at its own size.
from the `width` and `height` of its root element. Rendering loads no file or The `orig` size of an SVG above 8192 on a side is refused with 400. An SVG with
URL that the SVG refers to. one side more than 8192 times the other is refused with 415 in those formats:
rendered with no side above 8192 pixels, its shorter side would be under 1
pixel. Rendering loads no file or URL that the SVG refers to.
`orig` of an SVG source is the SVG itself, unchanged, and is cached like any `orig` of an SVG source is the SVG itself, unchanged, and is cached like any
other image. It is given only at the `orig` size, since pixa does not resize an other image. It is given only at the `orig` size, since pixa does not resize an
@@ -536,7 +540,9 @@ can set, so that no encode takes more than 40 seconds of the default
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
up to about 11 seconds. up to about 11 seconds. These limits do not bound rendering an SVG source, which
happens within the encode: its time and memory grow with what the SVG draws,
such as filter effects.
The login form (`POST /`) is limited to 5 attempts per minute per client The login form (`POST /`) is limited to 5 attempts per minute per client
address, counting an IPv6 client by its /64; an attempt over the limit is address, counting an IPv6 client by its /64; an attempt over the limit is
@@ -618,22 +624,24 @@ session.
### Image Metadata ### Image Metadata
pixa decodes and re-encodes every image it serves, and removes all metadata from pixa decodes and re-encodes every image it serves, except `orig` of an SVG
the output: EXIF (GPS position, camera make, model and serial number, capture source, which is the SVG unchanged. It removes all metadata from every image it
re-encodes: EXIF (GPS position, camera make, model and serial number, capture
time, embedded thumbnail), XMP, IPTC and the ICC colour profile. This cannot be time, embedded thumbnail), XMP, IPTC and the ICC colour profile. This cannot be
turned off. The one exception is `orig` of an SVG source, which is the SVG turned off.
unchanged, its metadata included.
- The `orig` format means the source's own format, not the source's bytes: an - For a source in any format but SVG, the `orig` format means the source's own
`orig` image is re-encoded and stripped like any other. format, not the source's bytes: an `orig` image is re-encoded and stripped
like any other. For an SVG source, `orig` is the SVG's own bytes, its metadata
included.
- An image with an EXIF orientation is turned upright first, so it displays the - An image with an EXIF orientation is turned upright first, so it displays the
same without the tag; a requested size applies to the upright image. same without the tag; a requested size applies to the upright image.
- An image with an ICC profile is converted to sRGB first, since clients show an - An image with an ICC profile is converted to sRGB first, since clients show an
image with no profile as sRGB. Colours outside sRGB, such as the most image with no profile as sRGB. Colours outside sRGB, such as the most
saturated ones in a Display P3 photo, are clipped. saturated ones in a Display P3 photo, are clipped.
- The one exception is JPEG XL with libvips 8.16 and later: libvips then writes - JPEG XL is an exception with libvips 8.16 and later: libvips then writes an
an EXIF block of its own into the image, as govips cannot ask libvips to leave EXIF block of its own into the image, as govips cannot ask libvips to leave it
it out. It holds the image's size and otherwise fixed values, such as an out. It holds the image's size and otherwise fixed values, such as an
orientation of 1 and a resolution of 72 dpi, and nothing from the source. orientation of 1 and a resolution of 72 dpi, and nothing from the source.
### Source Hosts ### Source Hosts
+10 -1
View File
@@ -26,13 +26,20 @@ resolved. The disk cache is now size-bounded with LRU eviction
# Next Step # Next Step
P2: security: per-origin rate limiting P1: security: bound the time and memory of rendering an SVG source. Both grow
with what the SVG draws, such as filter effects, and govips cannot stop or limit
a render (https://git.eeqj.de/sneak/pixa/issues/68).
# Completed Steps # Completed Steps
- 2026-11-11 SVG is a source format again (closes #68), by the owner's ruling. - 2026-11-11 SVG is a source format again (closes #68), by the owner's ruling.
- The fetcher accepts `image/svg+xml`, and `internal/magic` detects SVG. - The fetcher accepts `image/svg+xml`, and `internal/magic` detects SVG.
- A raster format renders the SVG at the output size, so it stays sharp. - A raster format renders the SVG at the output size, so it stays sharp.
libvips' thumbnail renders it, at any scale.
- An SVG's own size is not limited. The output checks limit the size it is
rendered at, and no side is rendered above 8192.
- An SVG with one side more than 8192 times the other is `ErrSVGTooThin`,
answered 415.
- `orig` of an SVG at the `orig` size is the SVG unchanged, cached as a - `orig` of an SVG at the `orig` size is the SVG unchanged, cached as a
variant. At any other size it is `ErrSVGNotOriginalSize`, answered 400. variant. At any other size it is `ErrSVGNotOriginalSize`, answered 400.
- The cache reads up to 1024 first bytes of a file to find its format. - The cache reads up to 1024 first bytes of a file to find its format.
@@ -1403,6 +1410,8 @@ P2: security: per-origin rate limiting
# Future Steps # Future Steps
- P2: security
- per-origin rate limiting
- P2: HTTP response handling - P2: HTTP response handling
- Last-Modified headers - Last-Modified headers
- P2: configuration - P2: configuration
+4
View File
@@ -267,6 +267,10 @@ func (s *Handlers) respondImageError(
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat): errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
status = http.StatusUnsupportedMediaType status = http.StatusUnsupportedMediaType
message = "unsupported source format" message = "unsupported source format"
case errors.Is(err, imageprocessor.ErrSVGTooThin):
status = http.StatusUnsupportedMediaType
message = fmt.Sprintf("SVG source too thin to render: one side is "+
"more than %d times the other", imageprocessor.MaxInputDimension)
case errors.Is(err, imageprocessor.ErrOutputTooLarge): case errors.Is(err, imageprocessor.ErrOutputTooLarge):
status = http.StatusBadRequest status = http.StatusBadRequest
message = fmt.Sprintf("size too large for the source: the side "+ message = fmt.Sprintf("size too large for the source: the side "+
@@ -47,3 +47,30 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
} }
} }
} }
// TestSVGTooThinAnswers415 checks that respondImageError answers 415 with a
// clear error for an SVG source too thin to render, wrapped as the service
// wraps it.
func TestSVGTooThinAnswers415(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imagerequest.Request{
Source: imagerequest.Source{Host: "svg.example.com", Path: "/a.svg"},
}
err := fmt.Errorf("image processing failed: %w: 81930x10",
imageprocessor.ErrSVGTooThin)
rec := httptest.NewRecorder()
h.respondImageError(rec, req, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("status = %d, want %d", rec.Code,
http.StatusUnsupportedMediaType)
}
if !strings.Contains(rec.Body.String(), "SVG source too thin to render") {
t.Errorf("body = %q, want the SVG too thin error", rec.Body.String())
}
}
+43 -28
View File
@@ -124,6 +124,11 @@ var ErrUnsupportedInputFormat = errors.New("unsupported input format")
var ErrSVGNotOriginalSize = errors.New( var ErrSVGNotOriginalSize = errors.New(
"an SVG is returned as SVG only at the orig size") "an SVG is returned as SVG only at the orig size")
// ErrSVGTooThin is returned, before rendering, when an SVG source has one
// side more than MaxInputDimension times the other. Rendered with its longer
// side at most MaxInputDimension, its shorter side would be under 1 pixel.
var ErrSVGTooThin = errors.New("SVG source too thin to render")
// ErrTooManyImages is returned when MaxConcurrentProcessing images are being // ErrTooManyImages is returned when MaxConcurrentProcessing images are being
// processed and none finishes within ProcessingWaitTimeout. // processed and none finishes within ProcessingWaitTimeout.
var ErrTooManyImages = errors.New("too many images being processed at once") var ErrTooManyImages = errors.New("too many images being processed at once")
@@ -199,7 +204,7 @@ func (p *ImageProcessor) Process(
return nil, err return nil, err
} }
img, err := loadImage(data) img, err := loadImage(data, sourceFormat)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -229,8 +234,8 @@ func (p *ImageProcessor) Process(
// An SVG is rendered again at about the output size, to stay sharp // An SVG is rendered again at about the output size, to stay sharp
if sourceFormat == imagerequest.FormatSVG { if sourceFormat == imagerequest.FormatSVG {
rendered, err := renderSVG(data, rendered, err := renderSVG(data, origWidth, origHeight,
svgDensity(origWidth, origHeight, outputWidth, outputHeight)) outputWidth, outputHeight)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -312,8 +317,11 @@ func resolveOutputFormat(
// loadImage decodes data and turns the image upright: encode strips the EXIF // loadImage decodes data and turns the image upright: encode strips the EXIF
// orientation tag, and sizes must be worked out on the upright image. An // orientation tag, and sizes must be worked out on the upright image. An
// image with a side above MaxInputDimension is ErrInputTooLarge, which guards // image with a side above MaxInputDimension is ErrInputTooLarge, which guards
// against decompression bombs. // against decompression bombs. An SVG is not checked: only its header has
func loadImage(data []byte) (*vips.ImageRef, error) { // been read, and the output checks limit the size it is rendered at.
func loadImage(
data []byte, sourceFormat imagerequest.Format,
) (*vips.ImageRef, error) {
img, err := vips.NewImageFromBuffer(data) img, err := vips.NewImageFromBuffer(data)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to decode image: %w", err) return nil, fmt.Errorf("failed to decode image: %w", err)
@@ -326,7 +334,8 @@ func loadImage(data []byte) (*vips.ImageRef, error) {
return nil, fmt.Errorf("failed to auto-rotate: %w", err) return nil, fmt.Errorf("failed to auto-rotate: %w", err)
} }
if img.Width() > MaxInputDimension || img.Height() > MaxInputDimension { if sourceFormat != imagerequest.FormatSVG &&
(img.Width() > MaxInputDimension || img.Height() > MaxInputDimension) {
img.Close() img.Close()
return nil, ErrInputTooLarge return nil, ErrInputTooLarge
@@ -357,32 +366,38 @@ func svgAsItIs(
}, nil }, nil
} }
// svgDefaultDensity is the density, in dots per inch, at which libvips // svgRenderSize returns the least size in the proportions of width x height,
// renders an SVG unless told otherwise. At it, an SVG of width="64" is 64 // an SVG's own size, that covers outputWidth x outputHeight, so that resize
// pixels wide. // hardly changes it. Neither side is above MaxInputDimension.
const svgDefaultDensity = 72 func svgRenderSize(width, height, outputWidth, outputHeight int) (int, int) {
scale := min(
max(float64(outputWidth)/float64(width),
float64(outputHeight)/float64(height)),
float64(MaxInputDimension)/float64(max(width, height)))
// svgDensity returns the least density at which an SVG of width x height, return int(math.Round(float64(width) * scale)),
// its size at svgDefaultDensity, covers outputWidth x outputHeight, so that int(math.Round(float64(height) * scale))
// resize hardly changes it. Neither side is rendered above
// MaxInputDimension.
func svgDensity(width, height, outputWidth, outputHeight int) int {
scale := max(float64(outputWidth)/float64(width),
float64(outputHeight)/float64(height))
maxScale := float64(MaxInputDimension) / float64(max(width, height))
return min(int(math.Ceil(svgDefaultDensity*scale)),
int(svgDefaultDensity*maxScale))
} }
// renderSVG loads the SVG data rendered at density, in dots per inch. // renderSVG renders the SVG data, whose own size is width x height, at
// librsvg renders it and, given the SVG as bytes with no file name, loads // svgRenderSize for outputWidth x outputHeight. An SVG with one side more
// no file or URL the SVG refers to. // than MaxInputDimension times the other is ErrSVGTooThin. libvips'
func renderSVG(data []byte, density int) (*vips.ImageRef, error) { // thumbnail has librsvg render the SVG at that size directly, at any scale:
params := vips.NewImportParams() // govips takes a density only in whole dots per inch, which cannot render
params.Density.Set(density) // an SVG below 1/72 of its own size. Given the SVG as bytes with no file
// name, librsvg loads no file or URL the SVG refers to.
func renderSVG(
data []byte, width, height, outputWidth, outputHeight int,
) (*vips.ImageRef, error) {
if max(width, height) > MaxInputDimension*min(width, height) {
return nil, fmt.Errorf("%w: %dx%d", ErrSVGTooThin, width, height)
}
img, err := vips.LoadImageFromBuffer(data, params) renderWidth, renderHeight := svgRenderSize(width, height,
outputWidth, outputHeight)
img, err := vips.NewThumbnailWithSizeFromBuffer(data,
renderWidth, renderHeight, vips.InterestingNone, vips.SizeBoth)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to render SVG: %w", err) return nil, fmt.Errorf("failed to render SVG: %w", err)
} }
+93 -22
View File
@@ -177,46 +177,117 @@ func TestImageProcessor_SVGFetchesNothing(t *testing.T) {
} }
} }
// TestImageProcessor_RefusesSVGTooLarge checks that an SVG wider than // TestImageProcessor_RendersSVGLargerThanMaxInputDimension checks that an
// MaxInputDimension is ErrInputTooLarge, as a raster source is. // SVG whose own size is above MaxInputDimension is limited only by the
func TestImageProcessor_RefusesSVGTooLarge(t *testing.T) { // output checks: it is rendered at a raster size, orig of it is the SVG
// itself, and a raster format at its own size is ErrOutputTooLarge. The
// 1000000x1000000 one is rendered at a ten-thousandth of its own size.
func TestImageProcessor_RendersSVGLargerThanMaxInputDimension(t *testing.T) {
t.Parallel() t.Parallel()
_, _, err := processSVG(t, for _, svg := range []string{
`<svg xmlns="http://www.w3.org/2000/svg" width="10000" height="10"/>`, `<svg xmlns="http://www.w3.org/2000/svg" width="10000" height="10">` +
imagerequest.Transform{ `<rect width="10000" height="10"/></svg>`,
`<svg xmlns="http://www.w3.org/2000/svg" width="1000000" ` +
`height="1000000"><rect width="1000000" height="1000000"/></svg>`,
} {
result, _, err := processSVG(t, svg, imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 100}, Size: imagerequest.Size{Width: 100, Height: 100},
Format: imagerequest.FormatPNG, Format: imagerequest.FormatPNG,
}) })
if !errors.Is(err, ErrInputTooLarge) { if err != nil {
t.Errorf("SVG of 10000x10: error = %v, want %v", err, ErrInputTooLarge) t.Errorf("%s as PNG at 100x100: %v", svg, err)
} else {
t.Logf("%s as PNG at 100x100: %dx%d", svg, result.Width, result.Height)
if result.Width != 100 || result.Height != 100 {
t.Errorf("%s as PNG at 100x100 = %dx%d, want 100x100", svg,
result.Width, result.Height)
}
}
result, output, err := processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil || result.ContentType != imagerequest.FormatSVG.MIMEType() ||
string(output) != svg {
t.Errorf("orig of %s: error = %v, want the SVG itself", svg, err)
}
_, _, err = processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatPNG})
if !errors.Is(err, ErrOutputTooLarge) {
t.Errorf("%s as PNG at its own size: error = %v, want %v", svg,
err, ErrOutputTooLarge)
}
} }
} }
// TestSVGDensity checks the density at which an SVG is rendered to cover // TestImageProcessor_RefusesSVGTooThin checks that an SVG with one side more
// an output size, and that it is held to MaxInputDimension. // than MaxInputDimension times the other is ErrSVGTooThin at a raster size,
func TestSVGDensity(t *testing.T) { // that one at exactly that proportion is rendered, and that orig of either
// is the SVG itself.
func TestImageProcessor_RefusesSVGTooThin(t *testing.T) {
t.Parallel()
tooThin := `<svg xmlns="http://www.w3.org/2000/svg" width="81930" ` +
`height="10"><rect width="81930" height="10"/></svg>`
atTheLimit := `<svg xmlns="http://www.w3.org/2000/svg" width="81920" ` +
`height="10"><rect width="81920" height="10"/></svg>`
size := imagerequest.Size{Width: 100, Height: 100}
_, _, err := processSVG(t, tooThin,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
if !errors.Is(err, ErrSVGTooThin) {
t.Errorf("81930x10 SVG as PNG at 100x100: error = %v, want %v", err,
ErrSVGTooThin)
}
result, _, err := processSVG(t, atTheLimit,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
if err != nil {
t.Errorf("81920x10 SVG as PNG at 100x100: %v", err)
} else if result.Width != 100 || result.Height != 100 {
t.Errorf("81920x10 SVG as PNG at 100x100 = %dx%d, want 100x100",
result.Width, result.Height)
}
for _, svg := range []string{tooThin, atTheLimit} {
result, output, err := processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil || result.ContentType != imagerequest.FormatSVG.MIMEType() ||
string(output) != svg {
t.Errorf("orig of %s: error = %v, want the SVG itself", svg, err)
}
}
}
// TestSVGRenderSize checks the size at which an SVG is rendered to cover an
// output size, and that it is held to MaxInputDimension.
func TestSVGRenderSize(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { tests := []struct {
name string name string
width, height int width, height int
outputWidth, outputHeight int outputWidth, outputHeight int
want int wantWidth, wantHeight int
}{ }{
{"own size", 16, 12, 16, 12, svgDefaultDensity}, {"own size", 16, 12, 16, 12, 16, 12},
{"100 times larger", 16, 12, 1600, 1200, 7200}, {"100 times larger", 16, 12, 1600, 1200, 1600, 1200},
{"wider than the SVG's proportions", 16, 12, 160, 12, 720}, {"wider than the SVG's proportions", 16, 12, 160, 12, 160, 120},
{"10 times smaller", 100, 100, 10, 10, 8}, {"10 times smaller", 100, 100, 10, 10, 10, 10},
{"held to MaxInputDimension", 8192, 1, 8192, 8192, svgDefaultDensity}, {"held to MaxInputDimension", 8192, 1, 8192, 8192, 8192, 1},
{"held to MaxInputDimension when enlarged", 1024, 1, 8192, 8192, 576}, {"held to MaxInputDimension when enlarged", 1024, 1, 8192, 8192, 8192, 8},
{"far above MaxInputDimension", 1000000, 1000000, 100, 100, 100, 100},
} }
for _, tt := range tests { for _, tt := range tests {
got := svgDensity(tt.width, tt.height, tt.outputWidth, tt.outputHeight) gotWidth, gotHeight := svgRenderSize(tt.width, tt.height,
if got != tt.want { tt.outputWidth, tt.outputHeight)
t.Errorf("%s: svgDensity(%d, %d, %d, %d) = %d, want %d", tt.name, if gotWidth != tt.wantWidth || gotHeight != tt.wantHeight {
tt.width, tt.height, tt.outputWidth, tt.outputHeight, got, tt.want) t.Errorf("%s: svgRenderSize(%d, %d, %d, %d) = %dx%d, want %dx%d",
tt.name, tt.width, tt.height, tt.outputWidth, tt.outputHeight,
gotWidth, gotHeight, tt.wantWidth, tt.wantHeight)
} }
} }
} }