Compare commits

..
1 Commits
Author SHA1 Message Date
sneak 632fe0bd36 Support SVG sources (closes #68)
check / check (push) Canceled after 0s
The fetcher accepts image/svg+xml again and internal/magic detects SVG.
A raster format renders the SVG with librsvg at the density that makes
it cover the output size, so an enlarged SVG stays sharp; the rendered
size is held to MaxInputDimension. orig of an SVG at the orig size is
the SVG unchanged, cached as a variant; at any other size it is
ErrSVGNotOriginalSize, answered 400. The cache reads up to 1024 first
bytes of a file to find its format, as an SVG's root element can come
after a prolog. Tests that pinned SVG as refused now expect it to be
read.

Model: opus-5-5
2026-11-11 00:38:16 +00:00
6 changed files with 65 additions and 199 deletions

No files matched your search

+14 -22
View File
@@ -147,7 +147,7 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1 times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
use. Rendering an SVG source can need more, depending on what it draws. use.
- **Temporary files:** while pixa processes a large source, libvips writes its - **Temporary files:** while pixa processes a large source, libvips writes its
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
written, so a listing of `/tmp` does not show it, but it keeps its space, on written, so a listing of `/tmp` does not show it, but it keeps its space, on
@@ -328,8 +328,6 @@ path under `/v1/` answers 200, in maintenance mode too.
format its `Content-Type` names. format its `Content-Type` names.
- 415 when the upstream sends an image whose first bytes libvips reads as - 415 when the upstream sends an image whose first bytes libvips reads as
another format. another format.
- 415 for an SVG source with one side more than 8192 times the other, in any
format but `orig`.
- 502 when the upstream answers with an error status. - 502 when the upstream answers with an error status.
- 502 for 5 minutes after that for the same source URL, when pixa has - 502 for 5 minutes after that for the same source URL, when pixa has
neither the image nor its source image in its cache. neither the image nor its source image in its cache.
@@ -474,11 +472,9 @@ format, such as an SVG whose first bytes pass the check for AVIF, is refused
with 415 too. with 415 too.
For any format but `orig`, pixa renders an SVG source at the requested size, so For any format but `orig`, pixa renders an SVG source at the requested size, so
an enlarged SVG stays sharp. At the `orig` size it is rendered at its own size. an enlarged SVG stays sharp. At the `orig` size it is rendered at its own size,
The `orig` size of an SVG above 8192 on a side is refused with 400. An SVG with from the `width` and `height` of its root element. Rendering loads no file or
one side more than 8192 times the other is refused with 415 in those formats: URL that the SVG refers to.
rendered with no side above 8192 pixels, its shorter side would be under 1
pixel. Rendering loads no file or URL that the SVG refers to.
`orig` of an SVG source is the SVG itself, unchanged, and is cached like any `orig` of an SVG source is the SVG itself, unchanged, and is cached like any
other image. It is given only at the `orig` size, since pixa does not resize an other image. It is given only at the `orig` size, since pixa does not resize an
@@ -540,9 +536,7 @@ can set, so that no encode takes more than 40 seconds of the default
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
up to about 11 seconds. These limits do not bound rendering an SVG source, which up to about 11 seconds.
happens within the encode: its time and memory grow with what the SVG draws,
such as filter effects.
The login form (`POST /`) is limited to 5 attempts per minute per client The login form (`POST /`) is limited to 5 attempts per minute per client
address, counting an IPv6 client by its /64; an attempt over the limit is address, counting an IPv6 client by its /64; an attempt over the limit is
@@ -624,24 +618,22 @@ session.
### Image Metadata ### Image Metadata
pixa decodes and re-encodes every image it serves, except `orig` of an SVG pixa decodes and re-encodes every image it serves, and removes all metadata from
source, which is the SVG unchanged. It removes all metadata from every image it the output: EXIF (GPS position, camera make, model and serial number, capture
re-encodes: EXIF (GPS position, camera make, model and serial number, capture
time, embedded thumbnail), XMP, IPTC and the ICC colour profile. This cannot be time, embedded thumbnail), XMP, IPTC and the ICC colour profile. This cannot be
turned off. turned off. The one exception is `orig` of an SVG source, which is the SVG
unchanged, its metadata included.
- For a source in any format but SVG, the `orig` format means the source's own - The `orig` format means the source's own format, not the source's bytes: an
format, not the source's bytes: an `orig` image is re-encoded and stripped `orig` image is re-encoded and stripped like any other.
like any other. For an SVG source, `orig` is the SVG's own bytes, its metadata
included.
- An image with an EXIF orientation is turned upright first, so it displays the - An image with an EXIF orientation is turned upright first, so it displays the
same without the tag; a requested size applies to the upright image. same without the tag; a requested size applies to the upright image.
- An image with an ICC profile is converted to sRGB first, since clients show an - An image with an ICC profile is converted to sRGB first, since clients show an
image with no profile as sRGB. Colours outside sRGB, such as the most image with no profile as sRGB. Colours outside sRGB, such as the most
saturated ones in a Display P3 photo, are clipped. saturated ones in a Display P3 photo, are clipped.
- JPEG XL is an exception with libvips 8.16 and later: libvips then writes an - The one exception is JPEG XL with libvips 8.16 and later: libvips then writes
EXIF block of its own into the image, as govips cannot ask libvips to leave it an EXIF block of its own into the image, as govips cannot ask libvips to leave
out. It holds the image's size and otherwise fixed values, such as an it out. It holds the image's size and otherwise fixed values, such as an
orientation of 1 and a resolution of 72 dpi, and nothing from the source. orientation of 1 and a resolution of 72 dpi, and nothing from the source.
### Source Hosts ### Source Hosts
+1 -10
View File
@@ -26,20 +26,13 @@ resolved. The disk cache is now size-bounded with LRU eviction
# Next Step # Next Step
P1: security: bound the time and memory of rendering an SVG source. Both grow P2: security: per-origin rate limiting
with what the SVG draws, such as filter effects, and govips cannot stop or limit
a render (https://git.eeqj.de/sneak/pixa/issues/68).
# Completed Steps # Completed Steps
- 2026-11-11 SVG is a source format again (closes #68), by the owner's ruling. - 2026-11-11 SVG is a source format again (closes #68), by the owner's ruling.
- The fetcher accepts `image/svg+xml`, and `internal/magic` detects SVG. - The fetcher accepts `image/svg+xml`, and `internal/magic` detects SVG.
- A raster format renders the SVG at the output size, so it stays sharp. - A raster format renders the SVG at the output size, so it stays sharp.
libvips' thumbnail renders it, at any scale.
- An SVG's own size is not limited. The output checks limit the size it is
rendered at, and no side is rendered above 8192.
- An SVG with one side more than 8192 times the other is `ErrSVGTooThin`,
answered 415.
- `orig` of an SVG at the `orig` size is the SVG unchanged, cached as a - `orig` of an SVG at the `orig` size is the SVG unchanged, cached as a
variant. At any other size it is `ErrSVGNotOriginalSize`, answered 400. variant. At any other size it is `ErrSVGNotOriginalSize`, answered 400.
- The cache reads up to 1024 first bytes of a file to find its format. - The cache reads up to 1024 first bytes of a file to find its format.
@@ -1410,8 +1403,6 @@ a render (https://git.eeqj.de/sneak/pixa/issues/68).
# Future Steps # Future Steps
- P2: security
- per-origin rate limiting
- P2: HTTP response handling - P2: HTTP response handling
- Last-Modified headers - Last-Modified headers
- P2: configuration - P2: configuration
-4
View File
@@ -267,10 +267,6 @@ func (s *Handlers) respondImageError(
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat): errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
status = http.StatusUnsupportedMediaType status = http.StatusUnsupportedMediaType
message = "unsupported source format" message = "unsupported source format"
case errors.Is(err, imageprocessor.ErrSVGTooThin):
status = http.StatusUnsupportedMediaType
message = fmt.Sprintf("SVG source too thin to render: one side is "+
"more than %d times the other", imageprocessor.MaxInputDimension)
case errors.Is(err, imageprocessor.ErrOutputTooLarge): case errors.Is(err, imageprocessor.ErrOutputTooLarge):
status = http.StatusBadRequest status = http.StatusBadRequest
message = fmt.Sprintf("size too large for the source: the side "+ message = fmt.Sprintf("size too large for the source: the side "+
@@ -47,30 +47,3 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
} }
} }
} }
// TestSVGTooThinAnswers415 checks that respondImageError answers 415 with a
// clear error for an SVG source too thin to render, wrapped as the service
// wraps it.
func TestSVGTooThinAnswers415(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imagerequest.Request{
Source: imagerequest.Source{Host: "svg.example.com", Path: "/a.svg"},
}
err := fmt.Errorf("image processing failed: %w: 81930x10",
imageprocessor.ErrSVGTooThin)
rec := httptest.NewRecorder()
h.respondImageError(rec, req, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("status = %d, want %d", rec.Code,
http.StatusUnsupportedMediaType)
}
if !strings.Contains(rec.Body.String(), "SVG source too thin to render") {
t.Errorf("body = %q, want the SVG too thin error", rec.Body.String())
}
}
+28 -43
View File
@@ -124,11 +124,6 @@ var ErrUnsupportedInputFormat = errors.New("unsupported input format")
var ErrSVGNotOriginalSize = errors.New( var ErrSVGNotOriginalSize = errors.New(
"an SVG is returned as SVG only at the orig size") "an SVG is returned as SVG only at the orig size")
// ErrSVGTooThin is returned, before rendering, when an SVG source has one
// side more than MaxInputDimension times the other. Rendered with its longer
// side at most MaxInputDimension, its shorter side would be under 1 pixel.
var ErrSVGTooThin = errors.New("SVG source too thin to render")
// ErrTooManyImages is returned when MaxConcurrentProcessing images are being // ErrTooManyImages is returned when MaxConcurrentProcessing images are being
// processed and none finishes within ProcessingWaitTimeout. // processed and none finishes within ProcessingWaitTimeout.
var ErrTooManyImages = errors.New("too many images being processed at once") var ErrTooManyImages = errors.New("too many images being processed at once")
@@ -204,7 +199,7 @@ func (p *ImageProcessor) Process(
return nil, err return nil, err
} }
img, err := loadImage(data, sourceFormat) img, err := loadImage(data)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -234,8 +229,8 @@ func (p *ImageProcessor) Process(
// An SVG is rendered again at about the output size, to stay sharp // An SVG is rendered again at about the output size, to stay sharp
if sourceFormat == imagerequest.FormatSVG { if sourceFormat == imagerequest.FormatSVG {
rendered, err := renderSVG(data, origWidth, origHeight, rendered, err := renderSVG(data,
outputWidth, outputHeight) svgDensity(origWidth, origHeight, outputWidth, outputHeight))
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -317,11 +312,8 @@ func resolveOutputFormat(
// loadImage decodes data and turns the image upright: encode strips the EXIF // loadImage decodes data and turns the image upright: encode strips the EXIF
// orientation tag, and sizes must be worked out on the upright image. An // orientation tag, and sizes must be worked out on the upright image. An
// image with a side above MaxInputDimension is ErrInputTooLarge, which guards // image with a side above MaxInputDimension is ErrInputTooLarge, which guards
// against decompression bombs. An SVG is not checked: only its header has // against decompression bombs.
// been read, and the output checks limit the size it is rendered at. func loadImage(data []byte) (*vips.ImageRef, error) {
func loadImage(
data []byte, sourceFormat imagerequest.Format,
) (*vips.ImageRef, error) {
img, err := vips.NewImageFromBuffer(data) img, err := vips.NewImageFromBuffer(data)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to decode image: %w", err) return nil, fmt.Errorf("failed to decode image: %w", err)
@@ -334,8 +326,7 @@ func loadImage(
return nil, fmt.Errorf("failed to auto-rotate: %w", err) return nil, fmt.Errorf("failed to auto-rotate: %w", err)
} }
if sourceFormat != imagerequest.FormatSVG && if img.Width() > MaxInputDimension || img.Height() > MaxInputDimension {
(img.Width() > MaxInputDimension || img.Height() > MaxInputDimension) {
img.Close() img.Close()
return nil, ErrInputTooLarge return nil, ErrInputTooLarge
@@ -366,38 +357,32 @@ func svgAsItIs(
}, nil }, nil
} }
// svgRenderSize returns the least size in the proportions of width x height, // svgDefaultDensity is the density, in dots per inch, at which libvips
// an SVG's own size, that covers outputWidth x outputHeight, so that resize // renders an SVG unless told otherwise. At it, an SVG of width="64" is 64
// hardly changes it. Neither side is above MaxInputDimension. // pixels wide.
func svgRenderSize(width, height, outputWidth, outputHeight int) (int, int) { const svgDefaultDensity = 72
scale := min(
max(float64(outputWidth)/float64(width),
float64(outputHeight)/float64(height)),
float64(MaxInputDimension)/float64(max(width, height)))
return int(math.Round(float64(width) * scale)), // svgDensity returns the least density at which an SVG of width x height,
int(math.Round(float64(height) * scale)) // its size at svgDefaultDensity, covers outputWidth x outputHeight, so that
// resize hardly changes it. Neither side is rendered above
// MaxInputDimension.
func svgDensity(width, height, outputWidth, outputHeight int) int {
scale := max(float64(outputWidth)/float64(width),
float64(outputHeight)/float64(height))
maxScale := float64(MaxInputDimension) / float64(max(width, height))
return min(int(math.Ceil(svgDefaultDensity*scale)),
int(svgDefaultDensity*maxScale))
} }
// renderSVG renders the SVG data, whose own size is width x height, at // renderSVG loads the SVG data rendered at density, in dots per inch.
// svgRenderSize for outputWidth x outputHeight. An SVG with one side more // librsvg renders it and, given the SVG as bytes with no file name, loads
// than MaxInputDimension times the other is ErrSVGTooThin. libvips' // no file or URL the SVG refers to.
// thumbnail has librsvg render the SVG at that size directly, at any scale: func renderSVG(data []byte, density int) (*vips.ImageRef, error) {
// govips takes a density only in whole dots per inch, which cannot render params := vips.NewImportParams()
// an SVG below 1/72 of its own size. Given the SVG as bytes with no file params.Density.Set(density)
// name, librsvg loads no file or URL the SVG refers to.
func renderSVG(
data []byte, width, height, outputWidth, outputHeight int,
) (*vips.ImageRef, error) {
if max(width, height) > MaxInputDimension*min(width, height) {
return nil, fmt.Errorf("%w: %dx%d", ErrSVGTooThin, width, height)
}
renderWidth, renderHeight := svgRenderSize(width, height, img, err := vips.LoadImageFromBuffer(data, params)
outputWidth, outputHeight)
img, err := vips.NewThumbnailWithSizeFromBuffer(data,
renderWidth, renderHeight, vips.InterestingNone, vips.SizeBoth)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to render SVG: %w", err) return nil, fmt.Errorf("failed to render SVG: %w", err)
} }
+22 -93
View File
@@ -177,117 +177,46 @@ func TestImageProcessor_SVGFetchesNothing(t *testing.T) {
} }
} }
// TestImageProcessor_RendersSVGLargerThanMaxInputDimension checks that an // TestImageProcessor_RefusesSVGTooLarge checks that an SVG wider than
// SVG whose own size is above MaxInputDimension is limited only by the // MaxInputDimension is ErrInputTooLarge, as a raster source is.
// output checks: it is rendered at a raster size, orig of it is the SVG func TestImageProcessor_RefusesSVGTooLarge(t *testing.T) {
// itself, and a raster format at its own size is ErrOutputTooLarge. The
// 1000000x1000000 one is rendered at a ten-thousandth of its own size.
func TestImageProcessor_RendersSVGLargerThanMaxInputDimension(t *testing.T) {
t.Parallel() t.Parallel()
for _, svg := range []string{ _, _, err := processSVG(t,
`<svg xmlns="http://www.w3.org/2000/svg" width="10000" height="10">` + `<svg xmlns="http://www.w3.org/2000/svg" width="10000" height="10"/>`,
`<rect width="10000" height="10"/></svg>`, imagerequest.Transform{
`<svg xmlns="http://www.w3.org/2000/svg" width="1000000" ` +
`height="1000000"><rect width="1000000" height="1000000"/></svg>`,
} {
result, _, err := processSVG(t, svg, imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 100}, Size: imagerequest.Size{Width: 100, Height: 100},
Format: imagerequest.FormatPNG, Format: imagerequest.FormatPNG,
}) })
if err != nil { if !errors.Is(err, ErrInputTooLarge) {
t.Errorf("%s as PNG at 100x100: %v", svg, err) t.Errorf("SVG of 10000x10: error = %v, want %v", err, ErrInputTooLarge)
} else {
t.Logf("%s as PNG at 100x100: %dx%d", svg, result.Width, result.Height)
if result.Width != 100 || result.Height != 100 {
t.Errorf("%s as PNG at 100x100 = %dx%d, want 100x100", svg,
result.Width, result.Height)
}
}
result, output, err := processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil || result.ContentType != imagerequest.FormatSVG.MIMEType() ||
string(output) != svg {
t.Errorf("orig of %s: error = %v, want the SVG itself", svg, err)
}
_, _, err = processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatPNG})
if !errors.Is(err, ErrOutputTooLarge) {
t.Errorf("%s as PNG at its own size: error = %v, want %v", svg,
err, ErrOutputTooLarge)
}
} }
} }
// TestImageProcessor_RefusesSVGTooThin checks that an SVG with one side more // TestSVGDensity checks the density at which an SVG is rendered to cover
// than MaxInputDimension times the other is ErrSVGTooThin at a raster size, // an output size, and that it is held to MaxInputDimension.
// that one at exactly that proportion is rendered, and that orig of either func TestSVGDensity(t *testing.T) {
// is the SVG itself.
func TestImageProcessor_RefusesSVGTooThin(t *testing.T) {
t.Parallel()
tooThin := `<svg xmlns="http://www.w3.org/2000/svg" width="81930" ` +
`height="10"><rect width="81930" height="10"/></svg>`
atTheLimit := `<svg xmlns="http://www.w3.org/2000/svg" width="81920" ` +
`height="10"><rect width="81920" height="10"/></svg>`
size := imagerequest.Size{Width: 100, Height: 100}
_, _, err := processSVG(t, tooThin,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
if !errors.Is(err, ErrSVGTooThin) {
t.Errorf("81930x10 SVG as PNG at 100x100: error = %v, want %v", err,
ErrSVGTooThin)
}
result, _, err := processSVG(t, atTheLimit,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
if err != nil {
t.Errorf("81920x10 SVG as PNG at 100x100: %v", err)
} else if result.Width != 100 || result.Height != 100 {
t.Errorf("81920x10 SVG as PNG at 100x100 = %dx%d, want 100x100",
result.Width, result.Height)
}
for _, svg := range []string{tooThin, atTheLimit} {
result, output, err := processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil || result.ContentType != imagerequest.FormatSVG.MIMEType() ||
string(output) != svg {
t.Errorf("orig of %s: error = %v, want the SVG itself", svg, err)
}
}
}
// TestSVGRenderSize checks the size at which an SVG is rendered to cover an
// output size, and that it is held to MaxInputDimension.
func TestSVGRenderSize(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { tests := []struct {
name string name string
width, height int width, height int
outputWidth, outputHeight int outputWidth, outputHeight int
wantWidth, wantHeight int want int
}{ }{
{"own size", 16, 12, 16, 12, 16, 12}, {"own size", 16, 12, 16, 12, svgDefaultDensity},
{"100 times larger", 16, 12, 1600, 1200, 1600, 1200}, {"100 times larger", 16, 12, 1600, 1200, 7200},
{"wider than the SVG's proportions", 16, 12, 160, 12, 160, 120}, {"wider than the SVG's proportions", 16, 12, 160, 12, 720},
{"10 times smaller", 100, 100, 10, 10, 10, 10}, {"10 times smaller", 100, 100, 10, 10, 8},
{"held to MaxInputDimension", 8192, 1, 8192, 8192, 8192, 1}, {"held to MaxInputDimension", 8192, 1, 8192, 8192, svgDefaultDensity},
{"held to MaxInputDimension when enlarged", 1024, 1, 8192, 8192, 8192, 8}, {"held to MaxInputDimension when enlarged", 1024, 1, 8192, 8192, 576},
{"far above MaxInputDimension", 1000000, 1000000, 100, 100, 100, 100},
} }
for _, tt := range tests { for _, tt := range tests {
gotWidth, gotHeight := svgRenderSize(tt.width, tt.height, got := svgDensity(tt.width, tt.height, tt.outputWidth, tt.outputHeight)
tt.outputWidth, tt.outputHeight) if got != tt.want {
if gotWidth != tt.wantWidth || gotHeight != tt.wantHeight { t.Errorf("%s: svgDensity(%d, %d, %d, %d) = %d, want %d", tt.name,
t.Errorf("%s: svgRenderSize(%d, %d, %d, %d) = %dx%d, want %dx%d", tt.width, tt.height, tt.outputWidth, tt.outputHeight, got, tt.want)
tt.name, tt.width, tt.height, tt.outputWidth, tt.outputHeight,
gotWidth, gotHeight, tt.wantWidth, tt.wantHeight)
} }
} }
} }