Its last sentence said asIPv4 differs from addr only for an
IPv4-compatible address, which is untrue for any other IPv6 address,
where asIPv4 is the zero netip.Addr. It now says that matching asIPv4
as well as addr refuses more only for an IPv4-compatible address.
Model: opus-5-5
isBlockedIP again matches an IPv4-mapped address (::ffff:0:0/96) only
as the IPv4 address it holds, as on next: Go keeps a plain IPv4 address
in that form, so matching it as written let an IPv6 entry such as ::/0
refuse every IPv4 address. An IPv4-compatible address (::/96) is still
matched both as written and as the IPv4 address it holds.
ipv6AndIPv4Forms is gone. README.md says the mapped form is matched as
IPv4 only.
Model: opus-5-5
isBlockedIP refuses an address when a blocked_networks entry contains
it as an IPv6 address or contains the IPv4 address it holds, for the
mapped (::ffff:0:0/96) and the IPv4-compatible (::/96) form alike, so
an IPv6 entry refuses everything it refused on next. ipv6AndIPv4Forms
gives the two forms. Go keeps a plain IPv4 address in the mapped form,
so an entry covering ::ffff:0:0/96, such as ::/0, now refuses every
IPv4 address too.
Model: opus-5-5
64:ff9b:1::/48 joins the built-in blocked ranges next to 64:ff9b::/96.
isBlockedIP now reads an address in ::/96 as the IPv4 address in its
last four bytes before matching blocked_networks, as it already unmapped
the ::ffff: form. The reading isPrivateIP gained for ::/96 moved into
ipv4CompatibleToIPv4, which both call.
Model: opus-5-5
The check that keeps pixa from fetching internal addresses read an
IPv6 address as IPv4 only in the mapped form (::ffff:a.b.c.d), so the
IPv4-compatible form, such as ::127.0.0.1, was judged as IPv6 and
passed. An address in ::/96 is now judged by the IPv4 address it holds,
so one holding a private, loopback or link-local address is refused on
both image routes before any upstream request. The NAT64 range
64:ff9b::/96 stays blocked whole, as it was on next.
Model: opus-5-5
The fetcher took the host from the source URL and stripped a port with
net.SplitHostPort, which fails when there is no port, so a bracketed
IPv6 host such as [2001:db8::a] kept its brackets, its name lookup
failed, and the request answered 500. validateURL now takes the host
with url.URL.Hostname(), which drops the brackets and any port, so such
a host is fetched like one with a port, through /v1/image/ and through
an encrypted URL, and a private IPv6 address is still refused.
Model: opus-5-5
Once upstream_max_response_size bytes were read, the fetcher refused the next read without looking for the end of the body. Go's HTTP client returns the end of a body sent with a Content-Length together with its last bytes, but the end of a chunked body can come in a later read, so a chunked body of exactly the limit was refused after it was downloaded whole and the image routes answered 500. Once the limit is used up, the fetcher now reads one more byte: if the body has ended it is accepted, and if there is more it is refused with ErrResponseTooLarge.
Model: opus-5-5
The upstream fetch no longer accepts image/svg+xml, and both image
routes answer 415, not 500, for any upstream Content-Type outside the
source formats. Before decoding, the image processor names the input's
format from govips' DetermineImageType and refuses, with 415, input
whose first bytes libvips reads as anything but the six source formats,
so an SVG passing the first-bytes check for AVIF never reaches librsvg.
orig of such an input is refused instead of being saved as JPEG.
README.md drops SVG from the source formats.
SVG detection stays in internal/magic, unused on the request path, for
#168. An AVIF image sequence
(brand avis) answers 415.
Model: opus-5-5
A JPEG XL source is accepted, and orig of one is JPEG XL. The format
jxl works in plain and encrypted URLs and on the generator page,
served as image/jxl; auto chooses it first when Accept names
image/jxl.
govips sends libvips a JPEG XL distance, which overrides the quality,
so q becomes a distance, keeping 100 lossy. Metadata is removed from
the image before the JPEG XL save, as govips cannot have libvips strip
it, and the image is given 72 dpi so that the EXIF block libvips 8.16
adds holds nothing from the source. The sRGB conversion moved ahead of
the format switch, and a CMYK image with no ICC profile is converted
to sRGB, as libvips cannot save CMYK as JPEG XL.
Model: opus-5-5
TestImageProxyFlow in internal/server starts the database, handlers and
middleware from the constructors pixad uses, with a fresh state
directory, and replaces only the upstream origin with an httptest
server. For a resize with a format change and for orig it checks a 200
MISS with the right type and size, then a HIT after one upstream request,
and the files and rows the cache keeps. Two optional test seams make
that possible: httpfetcher.Config.DialContext and handlers.Params.Fetcher.
pixad sets neither and the config file and environment cannot, and tests
show production still uses the checked dialer and builds its own fetcher.
Model: opus-5-5
Every response carries X-Request-Id, the upstream fetch sends it, and
the "upstream fetched", "image converted" and "image served" lines log
it as request_id. pixa's own RequestID middleware keeps a request's own
ID only when it is at most 64 letters, digits, '-', '_' or '.', and
otherwise makes a random one with crypto/rand, so nothing a client
chooses freely and nothing about the host reaches upstream. /v1/e/ now
sets ETag, answers a matching If-None-Match with 304 and is routed for
HEAD, through notModified, which both image handlers call. No Vary is
added: go-chi/cors already sends Vary: Origin.
Model: opus-5-5
Each upstream host's semaphore now counts the fetches holding or waiting
for one of its slots, and is removed from hostSems when the last of them
gives its slot back or stops waiting, so a long-running pixad no longer
keeps one semaphore per host it ever fetched from.
VariantStorage.Delete removes the variant's .meta file too, a missing one
not being an error; DeleteWithMeta, which eviction called for that, is
gone.
Model: opus-5-5
Nothing bounded total in-flight work, so a burst of cache misses across
hosts could exhaust memory. Two settings now do: max_concurrent_processing
(default the CPUs Go uses) and upstream_connections (default 64, beside
the per-host limit). A request that finds either full waits up to 10
seconds, then gets 503; a slot is released on every path. No request
holds source bytes while it waits: a cached source is read only after
the processing slot is taken, and a fetched one only while it holds its
upstream connection. libvips runs one worker thread per image with its
operation cache off. Both waits count toward downstream_timeout, as the
README says.
Model: opus-5-5
Adds the blocked_networks config key: a list of CIDRs, parsed with net/netip, that is added to the built-in list of address ranges the fetcher refuses to contact and can never remove an entry from it. An invalid CIDR aborts startup naming the key and the value.
The built-in list gains CGNAT 100.64.0.0/10, IETF protocol assignments 192.0.0.0/24, benchmark 198.18.0.0/15 and NAT64 64:ff9b::/96. Resolved addresses are unmapped before matching, so IPv4-mapped IPv6 forms are caught too. Enforcement stays in the dial-time re-resolution, which is what closes the DNS rebinding window.
What a reader would trip over: 192.0.0.0/24 is now blocked but TEST-NET-1 (192.0.2.0/24), which the Fetch tests use as a public upstream, is a different range and stays dialable. The package-level dialer enforces the built-in ranges only; operator entries are applied by the fetcher.
Disclosure: one nolint:gochecknoglobals on the immutable built-in prefix list.
Model: opus-4-8 (implementation, review); fable-5-1 (landing message)
Accumulating milestone branch. One squashed commit per closed issue; `next` is kept green and mergeable to `main` at any time without notice.
Landed so far:
- `chore: update golangci-lint to v2.12.2 with canonical config` (#54) — canonical v2-schema `.golangci.yml`, pins bumped in `Dockerfile` and `script/bootstrap`, tree at `0 issues.`. Three behaviour deltas are recorded in that PR's body: `Cache.StoreVariant` takes a context, `MetadataStorage.Store` no longer leaks temp files on failure, and the `signing_key` too-short error text gained a `value too short:` prefix.
Sequencing for the milestone is tracked in #103.
Reviewed-on: #105
Co-authored-by: clawbot <clawbot@noreply.example.org>
Extracts the HTTP fetcher concern out of `internal/imgcache/` into its own `internal/httpfetcher/` package, per the plan in [issue #39](#39).
This is one of four planned extractions; only the fetcher is moved here so the diff stays reviewable. The remaining three (signature, magic, urlparser) will land in separate PRs.
## What moved
From `internal/imgcache/fetcher.go` and `internal/imgcache/mock_fetcher.go` into `internal/httpfetcher/`:
- `HTTPFetcher` type, its `New` constructor, and `Config` (formerly `FetcherConfig`) with `DefaultConfig`
- SSRF-safe dialer, `validateURL`, `isPrivateIP`, `isLocalhost`, `extractHost`
- Per-host connection semaphore (rate limiting) and `limitedReader`
- Content-type validation (`isAllowedContentType`, `detectContentTypeFromPath`)
- All related error values (`ErrSSRFBlocked`, `ErrUpstreamError`, `ErrUpstreamTimeout`, `ErrPayloadTooLarge`, `ErrDisallowedContentType`)
- All related constants (`DefaultFetchTimeout`, `DefaultMaxPayloadBytes`, `DefaultMaxConnectionsPerHost`, etc.)
- The `Fetcher` interface and `FetchResult` type (moved here to keep the import edge one-way: `imgcache` depends on `httpfetcher`, never the reverse)
- `MockFetcher` test helper
## Renames (no stuttering)
- `NewHTTPFetcher` → `httpfetcher.New`
- `FetcherConfig` → `httpfetcher.Config`
- `DefaultFetcherConfig` → `httpfetcher.DefaultConfig`
- `NewMockFetcher` → `httpfetcher.NewMock`
The `ServiceConfig.FetcherConfig` field name is retained — it describes what kind of config the field holds (not a stutter).
## Behavior
Pure refactor. No behavior changes. All existing tests pass; unit tests for the new package are included.
`docker build .` passes (fmt-check, lint, test, build).
refs #39
Co-authored-by: clawbot <clawbot@eeqj.de>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #43
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
sneak
clawbot
