The source URL is built by joining the decoded path and query as text, so some characters reach the upstream wrong #288

Closed
opened 2026-10-16 01:37:27 +02:00 by clawbot · 2 comments
Collaborator

Where: internal/imgcache (the source URL a request fetches, built from the decoded source path and query) and ValidateRequest, on next after #287. Found by the review of #287, with test upstreams that recorded each request.

What is wrong: the decoded source path and query are joined into the source URL as text, not encoded as a URL path and query. So:

  • A space in a source query, written %20 as README.md allows, is sent upstream as a raw space, and the request fails with 502.
  • A literal % in a source path, written %25 as README.md asks (100%25.png), makes a source URL Go cannot parse, and the route answers 401 "unauthorized", where the README says the URL is accepted.
  • A source path holding %20 and one holding %2520 reach the upstream as the same request (Go's client encodes the space again), but are signed and cached under different keys: another unsigned spelling of one upstream request on an allowlisted host, the problem #275 and #286 closed for other characters.

Severity: low

Definition of done:

  1. The source URL is built from the decoded host, path and query by URL encoding, so each decoded source path and query is exactly one upstream request: a space in the query is sent as %20, 100%25.png is fetched as 100%25.png, and %20 and %2520 fetch different upstream paths, each matching its own cache key.
  2. The generator page (internal/handlers/auth.go, for logged-in users) produces, for a source URL holding %20, %25 or %23 in its path or query, an image URL that fetches exactly that source URL, or tells the user it cannot.
  3. Tests first, through the router against a test upstream that records the request it receives; README.md stays true; make check green.

Package: internal/imgcache, and internal/handlers for the generator.

Model: opus-5-5

Where: `internal/imgcache` (the source URL a request fetches, built from the decoded source path and query) and `ValidateRequest`, on `next` after https://git.eeqj.de/sneak/pixa/pulls/287. Found by the review of https://git.eeqj.de/sneak/pixa/pulls/287, with test upstreams that recorded each request. What is wrong: the decoded source path and query are joined into the source URL as text, not encoded as a URL path and query. So: - A space in a source query, written `%20` as `README.md` allows, is sent upstream as a raw space, and the request fails with 502. - A literal `%` in a source path, written `%25` as `README.md` asks (`100%25.png`), makes a source URL Go cannot parse, and the route answers 401 "unauthorized", where the README says the URL is accepted. - A source path holding `%20` and one holding `%2520` reach the upstream as the same request (Go's client encodes the space again), but are signed and cached under different keys: another unsigned spelling of one upstream request on an allowlisted host, the problem https://git.eeqj.de/sneak/pixa/issues/275 and https://git.eeqj.de/sneak/pixa/issues/286 closed for other characters. Severity: low Definition of done: 1. The source URL is built from the decoded host, path and query by URL encoding, so each decoded source path and query is exactly one upstream request: a space in the query is sent as `%20`, `100%25.png` is fetched as `100%25.png`, and `%20` and `%2520` fetch different upstream paths, each matching its own cache key. 2. The generator page (`internal/handlers/auth.go`, for logged-in users) produces, for a source URL holding `%20`, `%25` or `%23` in its path or query, an image URL that fetches exactly that source URL, or tells the user it cannot. 3. Tests first, through the router against a test upstream that records the request it receives; `README.md` stays true; `make check` green. Package: `internal/imgcache`, and `internal/handlers` for the generator. Model: opus-5-5
clawbot self-assigned this 2026-10-16 01:37:35 +02:00
Author
Collaborator

Plan:

  • The source URL a request fetches is built with net/url from the source host, path and query instead of joined as text. The source path stays decoded, as pixa reads it now, and is fetched percent-encoded again: 100%25.png fetches /100%25.png, and my%20cat.png and my%2520cat.png fetch /my%20cat.png and /my%2520cat.png, each under its own signature and cache key.
  • The source query stays as the source URL writes it, which is what the README.md example shows (my%2520cat.jpg in the image URL fetches my%20cat.jpg). When pixa reads the image URL it percent-encodes the characters a browser would encode in a query: a space, ", <, >, control characters, characters outside ASCII, and a % that does not begin a percent-escape. So name=my%20cat and name=my%2520cat in the image URL are one source query, name=my%20cat, signed, cached and fetched in that form.
  • The generator page checks the URL its image URL would fetch against the source URL that was entered. If they differ, for example for a path holding %2F, it answers 400 and shows the URL it would fetch. Source URLs holding %20, %25 and %23 fetch exactly what was entered.
  • README.md will say how the source path and query are fetched, and that the generator refuses such a URL.
  • Tests come first, in internal/server, through the router, against a test upstream that records each request it receives. No existing test changes.

Judgement call: the query is not decoded the way the path is. Encoding every % in it as well would make the README example fetch my%2520cat.jpg, and a %26 inside a query value could no longer be sent.

Model: opus-5-5

Plan: - The source URL a request fetches is built with `net/url` from the source host, path and query instead of joined as text. The source path stays decoded, as pixa reads it now, and is fetched percent-encoded again: `100%25.png` fetches `/100%25.png`, and `my%20cat.png` and `my%2520cat.png` fetch `/my%20cat.png` and `/my%2520cat.png`, each under its own signature and cache key. - The source query stays as the source URL writes it, which is what the `README.md` example shows (`my%2520cat.jpg` in the image URL fetches `my%20cat.jpg`). When pixa reads the image URL it percent-encodes the characters a browser would encode in a query: a space, `"`, &lt;, &gt;, control characters, characters outside ASCII, and a `%` that does not begin a percent-escape. So `name=my%20cat` and `name=my%2520cat` in the image URL are one source query, `name=my%20cat`, signed, cached and fetched in that form. - The generator page checks the URL its image URL would fetch against the source URL that was entered. If they differ, for example for a path holding `%2F`, it answers 400 and shows the URL it would fetch. Source URLs holding `%20`, `%25` and `%23` fetch exactly what was entered. - `README.md` will say how the source path and query are fetched, and that the generator refuses such a URL. - Tests come first, in `internal/server`, through the router, against a test upstream that records each request it receives. No existing test changes. Judgement call: the query is not decoded the way the path is. Encoding every `%` in it as well would make the README example fetch `my%2520cat.jpg`, and a `%26` inside a query value could no longer be sent. Model: opus-5-5
Author
Collaborator

Built as planned in #289. When the generator refuses a source URL, the page says Source URL cannot be fetched as written: pixa would fetch followed by that URL. This also happens for characters the upstream would most likely serve the same way, such as ( in a path, which pixa fetches as %28.

Model: opus-5-5

Built as planned in https://git.eeqj.de/sneak/pixa/pulls/289. When the generator refuses a source URL, the page says `Source URL cannot be fetched as written: pixa would fetch` followed by that URL. This also happens for characters the upstream would most likely serve the same way, such as `(` in a path, which pixa fetches as `%28`. Model: opus-5-5
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/pixa#288