From 1379497897510e858a38a0ac72250bae578784c7 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Wed, 11 Nov 2026 13:07:16 +0100 Subject: [PATCH] Keep only open work in TODO.md (closes #348) (#349) --- TODO.md | 1389 +------------------------------------------------------ 1 file changed, 9 insertions(+), 1380 deletions(-) diff --git a/TODO.md b/TODO.md index 8416fe5..9c99a32 100644 --- a/TODO.md +++ b/TODO.md @@ -2,7 +2,7 @@ - branch per issue from `next` - do the work in Next Step -- move Next Step to the top of Completed Steps +- delete Next Step from `TODO.md` - `TODO.md` merges with git's union merge (`.gitattributes`), which never reports a conflict: read the merged entries after every merge or rebase - move the top item of Future Steps into Next Step @@ -16,1394 +16,23 @@ # Status -pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on -`next`, and `main` receives only the milestone PR that the owner merges. `next` -is at the canonical `golangci-lint` v2.14.0 config and is green. Recent work -extracted the internal/magic, internal/allowlist, internal/httpfetcher, and -internal/signature packages. The gosec findings from the 2026-07-06 survey are -resolved. The disk cache is now size-bounded with LRU eviction -(`cache_max_bytes`), closing the unbounded disk growth DoS vector. +pre-1.0. The `1.0.0` milestone is in progress. # Next Step P2: security: per-origin rate limiting -# Completed Steps - -- 2026-10-27 the reconciliation pass skips a directory it cannot list and a file - whose info it cannot read, and goes on (closes #343). It stopped before its - later checks at a directory it could not list, or at a source or variant file - whose info it could not read. - - It logs the warning "skipped unreadable entry in cache directory", with - the path and the error. - - A file renamed or removed after its directory was listed gets no warning. - - `reconcileFiles` reads each file's info once, before the temp-file check, - and hands it to `sweepStaleTempFile`, `reconcileVariantFile` or - `reconcileSourceFile`. - - A cancelled context still stops the walk with its error. -- 2026-10-26 the reconciliation pass drops a variant's accounting row only when - its file is missing (closes #340). Any other error reading the file's size - keeps the row. It dropped the row on any error and logged the file as missing. - - The kept row gets the warning "kept accounting row for unreadable variant - file", with the variant key and the error. - - The pass checks for `ErrNotFound`, which `FileStore.Size` returns for a - missing file. -- 2026-10-26 on both image routes, a request is answered 499 and logged at info - level when its client goes away before its image is ready, unless it is the - request making the image (closes #339). It was answered 500 and logged at - error level, as pixa's own failure. - - The line is "client went away before the image was ready". It carries the - `request_id`, the host and the path. - - The 499 has no body, since nobody reads it. - - `serveImage` answers it before calling `respondImageError`, which cannot - see the request's context. - - An error wrapping `context.Canceled` while the request's context is live - is still answered 500 and logged at error level. -- 2026-10-26 `README.md`'s Design section starts with a Concepts list (closes - #321). Each concept gets one sentence, under the name the code gives it. - - Storage calls the transformed images variants, in shorter sentences. - - Storage's paths end in the source key and the variant key. - - Storage lists the image cache's tables in the database. - - Architecture names `internal/providers`, and lists the packages that work - with the concepts. -- 2026-10-26 one table in `internal/config` lists every setting once (closes - #320). `Config.settings` gives each setting's key, its environment variable - and the function that loads it. - - Most rows load through `field`, whose arguments give the setting's type, - its default and its check. - - `db_url` and `cache_max_bytes` have their own load functions. The default - of `db_url` comes from `state_dir`, and `cache_max_bytes` records whether - it is set. - - Loading, the unknown-key check, the unknown-variable check and the names - in error messages all read the table. - - `allowlist_hosts` loads as `referer_blocklist` does, through - `parseHostList`. - - The metrics username and password are checked together once the table has - loaded. - - When a config holds more than one invalid value, the error names the one - that comes first in the table. - - `README.md` lists the settings one item each, where it had a table. A test - checks that this list and `configs/config.example.yml` name exactly the - keys of the table. - - The key constants moved to the tests, their only users. - - Removed: - - `isKnownConfigKey`, `envVarNames`, `strictLoader` and `validate` - - `validateAllowlistHostsValue` and `getStringSlice`, with the three - tests of `getStringSlice` -- 2026-10-26 a fetched image's format is detected from its bytes once, as an - `imagerequest.Format` (closes #319). The upstream `Content-Type` check in the - fetcher stays as the filter before downloading. - - `magic.DetectFormat` returns an `imagerequest.Format`, and - `magic.ValidateMagicBytes` returns the format it checked against the - `Content-Type`. - - `Format.MIMEType` gives each format's MIME type. The fetcher, the mock - fetcher, the cache's read of a variant's type and the handler's `Accept` - matching use it. - - `Cache.Source` returns a source's format, read from its file's first bytes - as `Cache.Variant` reads a variant's. A file in no source format is an - error, and the proxy fetches the source again. - - The proxy hands the format to `ImageProcessor.Process`. The processor - refuses input that libvips would load as another format, as before, with - `ErrUnsupportedInputFormat`. It refuses a format that is not one of the - six source formats the same way. - - `internal/magic` no longer detects SVG, which pixa does not read. SVG data - is `ErrUnknownFormat`. - - Removed: - - `magic.MIMEType`, `magic.ImageFormat` and `magic.MIMEToImageFormat` - - `detectFormat`, `formatFromString`, `FormatToMIME` and - `Result.InputFormat` of the processor - - the fetcher's MIME constants of the six formats - - `errNotAnOutputFormat` of the cache -- 2026-10-26 the cache keeps its source images in one `imgcache.FileStore` and - its variants in another (closes #318). A `FileStore` is a directory of files, - each named by a hex key, at `//`. - - `ContentStorage` and `VariantStorage` are gone. - - `FileStore` has these methods: - - `Write`, which writes a file under a temp name, syncs it and renames - it into place - - `Open`, which returns the file and its size - - `Size` - - `Delete` - - `FileStore` has no `Exists`. The reconciliation pass checks that a - variant's file is there with `Size`. The tests look for the file on disk. - - `Cache.Variant` reads the content type from the variant file's first - bytes, as `VariantStorage.LoadWithContentType` did. A cache hit still - makes no database call. - - `storage.go` is now `filestore.go`. `internal/imgcache/testdata/red.avif`, - which no test used, is gone. -- 2026-10-26 `internal/imgcache` holds the cache only (closes #317). The new - package `internal/imageproxy` fetches, transforms, caches and serves images. - It imports `imgcache`, and `imgcache` does not import it. - - `imgcache.Service` is `imageproxy.Proxy`, built by `imageproxy.New` from - an `imageproxy.Config`. - - `ImageResponse`, `CacheStatus`, `ErrNegativeCached` and - `ErrProcessingTimedOut` moved with it, under their names. - - `imgcache.CacheConfig` is `imgcache.Config`. - - The cache has one read and one write for each kind it keeps. Each takes - the place of an earlier method: - - `Variant`, of `GetVariant` - - `PutVariant`, of `StoreVariant` - - `Source`, of `LookupSource` followed by `GetSourceContent` - - `PutSource`, of `StoreSource` - - `Failure`, of `checkNegativeCache` - - `PutFailure`, of `StoreNegative` - - `Variant` and `Source` each open the file and hand its access time to the - background writer. A cache hit still makes no database call. - - `imgcache.go` and `module.go` are gone. - - `imageproxy.New` makes the counters of upstream fetches and transforms, - and registers them on the registry it is given. The cache keeps only the - hit and miss counters. - - A cached source whose row cannot be read is logged as "failed to load - cached source, fetching". It was logged as "source lookup failed". - - The proxy's tests moved to `internal/imageproxy`. They reach the cache - through its database, its state directory and its metrics registry. - - The tests of an empty or cut cached file are split in two: - - `internal/imgcache` checks that the reconciliation pass removes the - file and its row, with a warning naming the file. - - `internal/imageproxy` removes the file and its row as the pass does, - and checks that `Get` fetches or converts again. -- 2026-10-26 both image routes serve an image with one function, `serveImage`, - and answer its errors with one, `respondImageError` (closes #316). - `HandleImage` parses the path and verifies the signature, and `HandleImageEnc` - decrypts the token. Each then calls `serveImage`. - - `handleImageError`, the encrypted route's own mapping, is gone. - - `/v1/image/` answers 504 when the upstream has not sent its response - headers within `upstream_fetch_timeout`, as `/v1/e/` did. It answered 500. - - `respondImageError` logs the image service's errors as "failed to get - image". Each answer gets the same level on `/v1/image/` and on `/v1/e/`: - - 500: error, as pixa's own failure - - 400: warn - - 403: warn - - 415: warn - - 502: warn - - 503: warn - - 504: warn - - Before, `/v1/image/` logged all of these answers at error level. `/v1/e/` - logged only its 500, as "image request failed", without the host and the - path. - - `respondImageError` does not log the 415 for a body refused for its first - bytes, on either route. The image service logs it as "source refused", at - warn level. - - The "image served" line carries the host, the path and `fetched_bytes` on - both routes. The debug line "encrypted image request" is gone. - - The debug line "failed to decrypt URL" is now "invalid encrypted URL". A - token that decrypts to a payload that does not decode also gets it. - - A failed write is logged once, as "failed to write response" at warn - level, since most often the client has gone away. It carries the - `request_id`, the host, the path, `served_bytes` and the error, and is not - also logged as "image served". -- 2026-10-26 every key derived from the signing key is derived in one place - (closes #315): `seal.DeriveKeys` returns them as one `seal.Keys`, with their - salts declared beside it. `internal/providers` calls it once at startup. Each - key is derived as before, with `seal.DeriveKey` and the same salt. - - The session manager, the encrypted URL generator and the CSRF protection - each take their own key instead of the signing key. - - `session.NewManager`, `encurl.NewGenerator`, `newCSRFProtect` and - `handlers.New` return no error, as nothing in them can fail now. - - The signing key itself stays the HMAC key of signed URLs and the login - password. - - `README.md` says under `signing_key` what it protects. -- 2026-10-26 the image request holds only what names the cached object (closes - #314): its source and its transform, which the variant key covers. The signed - URL format, the signing algorithm and every route's answers are unchanged. - - The `/v1/image/` handler checks `allowlist_hosts` and verifies the - signature itself, with `Signer.Verify(req, sig, expires)`. - - Each image handler keeps its URL's expiry as a value of its own, for - `Cache-Control`. `encurl`'s `Generate` takes the expiry and its `Parse` - returns it. - - The fetcher takes the source and builds its URL with `Source.URL`, over - http only when the fetcher's own `AllowHTTP` is set. `validateURL` takes - that URL, so no URL string is built and parsed again. - - `chooseAutoFormat` returns the format it chooses, and the handler puts it - into the request it parsed. - - The image service serves images through `Get`, and has `WaitForProcessing` - for shutdown. - - Removed: - - `Signature`, `Expires`, `AllowHTTP` and `SourceURL` of the image - request - - `Service.ValidateRequest`, and the service's signer, allowlist and - `AllowHTTP` - - `extractHost` of the fetcher and `urlToFSPath` of the mock fetcher -- 2026-10-26 pixad's fx graph builds the image pipeline (closes #313). - `database.New` now opens the database, so the parts below can read it while - they are built; its stop hook still runs after the cache's. The cache's - `MaxBytes` is the one value for the disk cache limit, and 0 turns the disk - cache off. For an omitted `cache_max_bytes`, `internal/providers` works out - `config.DefaultCacheMaxBytes` before it builds the cache. `internal/app` and - the server tests build the parts below with `providers.Constructors()`; the - image proxy test replaces only the fetcher's dial function. - - Built by the providers in `internal/providers`, and given to - `handlers.New`: - - the cache - - the fetcher - - the processor - - the image service - - the session manager - - the encrypted URL generator - - Removed: - - `initImageService` and `handlers.Params.Fetcher` - - `CacheConfig.UseDefaultMaxBytes`, `CacheConfig.DisableDiskCache` and - `Cache.disabled` - - `httpfetcher.DefaultConfig()` and the fetcher's copies of setting - defaults - - `imageprocessor.DefaultMaxInputBytes`, and the processor's defaults - for a zero `MaxInputBytes` or `MaxConcurrentProcessing` -- 2026-10-26 the code nothing in production called is gone (closes #312). The - handlers tests sign their image URLs with `generateSignedURL`, a test helper - that writes them as the removed generator did. Removed: - - `Service.Warm` and the `ImageCache` interface - - `Service.GenerateSignedURL` and `Signer.GenerateSignedURL` - - `signature.ParseParams` - - `imagerequest.ParseImagePath` - - `PeekAndValidate`, `IsSupportedMIMEType`, `ImageFormatToMIME` and - `FormatOriginal` of `internal/magic` - - `SupportedInputFormats` and `SupportedOutputFormats` of the image - processor - - `IsEmpty` and `Count` of the allowlist - - `ssrfSafeDialer` - - `CacheStale`, `ErrCacheMiss`, `ErrNegativeCache` and - `ImageResponse.LastModified` - - `ContentStorage.Load` and `VariantStorage.Load` -- 2026-10-26 cache hits, misses, upstream fetches, the bytes those fetches read - and transforms are counted as Prometheus counters in memory (closes #311). - They are `pixa_cache_hits_total`, `pixa_cache_misses_total`, - `pixa_upstream_fetches_total`, `pixa_upstream_fetch_bytes_total` and - `pixa_transforms_total`, fields of the cache. `NewCache` makes them and - registers them on the registry it is given (`internal/imgcache/counters.go`). - pixad gives it Prometheus' default registry, which `/metrics` serves; each - test gives it a new one. The `Increment` methods add to their cache's - counters. `cache_stats`, `Cache.Stats`, `Service.Stats`, `CacheStats` and the - pending counts are gone. The background goroutine writes only access times - now: `StartPendingAccessTimeWrites` starts it and - `StopPendingAccessTimeWrites` stops it, in `pendingaccesstimes.go`. -- 2026-10-21 the schema file declares only the tables and columns pixa writes - and reads (closes #310): `output_content` and `request_cache` are gone, and so - are the columns nothing read: the host, path, query, `content_type` and - `fetched_at` of `source_content`, the `content_type` and `created_at` of - `variant_content`, and the `status_code`, `error_message` and `fetched_at` of - `negative_cache`. `StoreSource` no longer takes the fetch result, - `LookupSource` returns only the source key, `StoreVariant` no longer takes a - content type, `StoreNegative` no longer takes a status code or message, and - reconciliation adopts a variant file without reading its first bytes. -- 2026-10-21 a cache hit makes no database call (closes #309): `Service.Get` - opens the variant file with `Cache.GetVariant` and nothing else, with no - existence check first. The content type comes from the file's first bytes; a - file whose first bytes are not those of an output format, SVG included, is a - miss, and the variant is made again. A hit hands its access time to a channel - that holds up to 10,000 and drops the rest, and its hit count to the pending - counts, without waiting; every count is now a pending count. A miss that finds - a cached source hands that source's access time to a second such channel. The - goroutine `StartPendingCountWrites` starts writes, once a second, the access - times in one transaction and the counts in one `UPDATE`; - `StopPendingCountWrites` writes what is left once at shutdown and logs - separately the access times and the counts it could not write. - `checkNegativeCache` no longer deletes an expired row; `CleanExpired` in the - eviction goroutine does. The `.meta` files, `VariantMeta`, `LoadWithMeta`, - `variantContentTypeFromSidecar`, `metaCache`, `Lookup` and `golang-lru` are - gone. `make bench` (`script/bench`) runs `BenchmarkHandleImage_CacheHits` with - Go on the host after `script/bootstrap --cgo`, outside `make check`. -- 2026-10-21 a source is stored the way a variant is (closes #308): one - `source_content` row and one file per source URL, the file named by the - source's `SourceKey`, the SHA-256 of its host, path and query made as the - variant key is, under `cache/sources/`. A store writes the file, then the row, - and removes the file when the row cannot be written; an eviction deletes the - row, then the file, with no lock and no transaction. `source_metadata`, the - JSON files under `cache/metadata/`, `MetadataStorage` and the content lock are - gone. Two source URLs that serve the same bytes now store them twice. -- 2026-10-21 one image request type instead of five (closes #307): the new - package `internal/imagerequest`, which imports no other package of pixa, holds - `imagerequest.Request` with its two parts, `Source` (host, path and query) and - `Transform` (size, format, quality and fit), the `Format` and `Fit` types, the - checks, the parsing of the `/v1/image/` path and the variant key. - `imgcache.ParsedURL`, `imgcache.ImageRequest`, `signature.Request`, - `imageprocessor.Request`, the second `Size`, format and fit types and the - conversions between them are gone. The image processor takes a `Transform` - only, and `encurl` keeps a private struct with the token's short CBOR field - names, so tokens made before still decode. Every URL of both image routes - answers as before. -- 2026-10-21 the Gitea workflow checks out the repo and runs `script/cibuild`, - nothing else (closes #203): its `script/docker-smoke` step is removed, as the - canonical `REPO_POLICIES.md` says, since `make check` covers what CI needs to - check. CI no longer starts the image and waits for its healthcheck. - `script/docker-smoke` and `make docker-smoke` stay, to run by hand. -- 2026-10-20 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34 - seconds (closes #272): with one libvips thread, 8192x8192 random pixels with - an alpha channel took about 61 seconds to encode as WebP, 81 as JPEG XL and, - at quality 85, 96 as AVIF, so with the default `downstream_timeout` of 60 - seconds such an image was never delivered, yet its encode held a processing - slot to the end. WebP is now saved at effort 2 and JPEG XL at effort 4, which - take up to about 23 and 27 seconds at any quality. AVIF was already at the - lowest effort govips can set, so an AVIF output of more than 16,777,216 pixels - is refused with 400 before anything is decoded; 4096x4096, 8192x2048 or - 2048x8192 takes up to about 34 seconds at any quality. Resizing with - `fit=cover` already decodes the input, so the output's dimensions are worked - out before resizing. `README.md` gives the limit and the new times. -- 2026-10-20 a source whose first bytes do not match its `Content-Type` answers - 415 (closes #253): a body whose first bytes are not those of the format its - `Content-Type` names, such as a JPEG sent as `image/png`, text, or a body - under 12 bytes, answered 500 on both image routes and was logged as an error - of pixa's, and each request for it fetched the whole body again. Both routes - now answer 415, and the image service logs it at warning level as a refused - source. It also gets a negative cache entry, as an upstream error status does, - so the source URL is not fetched again for 5 minutes, and meanwhile an image - that would need that fetch answers 502. -- 2026-10-20 `AGENTS.md`: a bug fix may change an existing test without asking - when that test pins exactly the behavior the filed bug calls wrong, and the - change moves it to the fixed behavior and nothing else (closes #280). -- 2026-10-18 re-vendored from `sneak/prompts` commit `0653579` (closes #221): - each `Dockerfile` stage installs its system packages and downloads the Go - modules above `ARG CACHEBUST`, so check builds keep them, and the lint step - keeps golangci-lint's cache. -- 2026-10-19 a `trusted_proxies` entry in the IPv4-mapped form trusted nothing; - it is now read as the IPv4 range it covers by the same code that reads - `blocked_networks`, and such an entry shorter than `/96` aborts startup - (closes #303). -- 2026-10-18 the next reconciliation pass waits at least ten times as long as - the last one took, so passes on a large cache no longer run back to back - (closes #250). -- 2026-10-18 a `blocked_networks` entry in the IPv4-mapped form refused nothing; - `::ffff:10.0.0.0/104` is now read as `10.0.0.0/8`, and such an entry shorter - than `/96` aborts startup (closes #301). -- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal - addresses refuses the local-use NAT64 range (`64:ff9b:1::/48`, RFC 8215) as it - refuses `64:ff9b::/96`, and refuses an IPv6 address in the IPv4-compatible - form (`::/96`) when `blocked_networks` contains either the address as written - or the IPv4 address it holds (closes #299). A source host such as - `[64:ff9b:1::a00:1]` was fetched, which on a network that runs NAT64 on that - range reaches the IPv4 address it holds, here `10.0.0.1`; and with - `203.0.113.0/24` in `blocked_networks`, `[::ffff:203.0.113.5]` was refused but - `[::203.0.113.5]` was fetched. Both now answer 403. The mapped form - (`::ffff:a.b.c.d`) is still matched only as the IPv4 address it holds: Go - keeps a plain IPv4 address in that form, and an IPv6 entry such as `::/0` must - not refuse every IPv4 address. -- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal - addresses judges an IPv6 address in the IPv4-compatible form (`::/96`), such - as `::127.0.0.1` or `::10.0.0.1`, by the IPv4 address in its last four bytes - (closes #297). It read only the mapped form (`::ffff:a.b.c.d`) as IPv4, so a - source host such as `[::127.0.0.1]` was checked as a public IPv6 address and - fetched, which on a network that routes it reaches the internal IPv4 host; it - is now refused with 403. The NAT64 form (`64:ff9b::/96`) was already refused - as a whole, as one of the built-in blocked ranges, and still is. -- 2026-10-16 a source host that is an IPv6 address in brackets with no port, - such as `[2001:db8::a]`, is fetched like one with a port, through `/v1/image/` - and through an encrypted URL (closes #295). The fetcher's check of the source - URL kept the brackets when no port followed them, could not look the host up, - and the request answered 500. It now checks the address without brackets, so a - private one is refused with 403, as with a port. -- 2026-10-16 a `state_dir` whose name holds `%`, `?` or `#` keeps its database - at `state.sqlite3` in that directory (closes #290): the `db_url` derived from - it put the name into a `file:` URL as it was, so SQLite read `%41` as `A`, a - `?` or `#` ended the path early, and the journal mode parameter was lost. The - derived `db_url` is now built with Go's `net/url`, which writes those - characters as `%25`, `%3F` and `%23`. For a `state_dir` of `/` or one starting - with `//` it starts `file:////`, so SQLite no longer reads the first name in - the path as a host. -- 2026-10-16 a size with one side `0` keeps the side computed from the source's - proportions at 1 or more (closes #283): it was rounded down, so for a thin - source it came out at 0, and libvips then made the size square. An 8192x1 - source at `100x0` came out at 100x100, and at `1300x0` with `fit=cover` made - libvips ask for about 39 GB at once; an 8192x2 source at `100x0` answered 500. - `fit=contain` and `fit=inside` work both sides out again from the smaller of - the two scales and `fit=outside` from the larger, and each keeps both sides at - 1 or more too, so a 1x8192 source at `0x100` with `fit=contain`, and a 49x8192 - source at `0x100` or a 49x49 source at `1x1` with `fit=outside`, no longer - answer 500. An 8192x1 source at `1300x0` now comes out at 1300x1, and the - process running it alone peaked at about 34 MiB resident. With `fit=outside` - the image is scaled to cover the size, so that source at `100x0` comes out at - 8192x1. `README.md` now says this under Signature Specification, and says how - a `0` side is worked out where it said `0` meant the original. -- 2026-10-16 the generator page lower-cases the source host and writes an empty - source path as `/` before it makes the encrypted URL, as `/v1/image/` does - (closes #293). A host written in upper and in lower case, or a source URL with - no path and with the path `/`, made two cache keys and two upstream requests - for one image. -- 2026-10-16 the source URL a request fetches is built with `net/url` from the - source host, path and query, not joined as text (closes #288). The source - path, decoded, is fetched percent-encoded again, so `100%25.png` fetches - `/100%25.png` where it answered 401, and `my%20cat.png` and `my%2520cat.png` - fetch different paths where both fetched `/my%20cat.png` under two cache keys. - The source query stays as the source URL writes it; the characters a browser - percent-encodes in a query, such as `'` or a space, which was sent raw and - answered 502, are percent-encoded when the image URL is read, so the query is - signed and cached as it is sent. The generator page answers 400, showing the - URL pixa would fetch, for a source URL its image URL would fetch written - another way, such as with `%2F` in the path, whatever else the path holds, or - with a user name and password. It also answers 400 for a source path holding a - `..` segment, as `/v1/image/` refuses one. -- 2026-10-15 a thin source requested with `fit=cover` no longer makes libvips - build a huge image (closes #282): libvips' thumbnail scaled the whole source - to cover the requested size and copied that into memory before it cut out the - centre, so an 8192x64 source at 8192x8192 made it allocate about 24 GiB, and - an 8192x1 source answered 500. The image processor now first cuts the centre - of the source, widened to whole source pixels, and thumbnail scales and cuts - only that, by the same amount on both sides as before. What it copies into - memory is then larger than the output by less than one scaled source pixel on - each side. With the runtime image's libvips and JPEG output at 8192x8192, - resident memory peaked at about 242 MiB for the 8192x64 source and 431 MiB for - the 8192x1 one. The output can sit a fraction of an output pixel from where it - was. -- 2026-10-15 a `/v1/image/` URL whose source path or source query holds a `#`, - written as `%23`, is refused with 400 (closes #286): the source URL built from - it ended at the `#` when fetched, so each text after it fetched the same - upstream image, without a signature for an allowlisted host, and cached it - under keys of its own. `README.md` says neither may hold a `#`. -- 2026-10-15 the host of a `/v1/image/` URL must be a host name or an IPv4 - address (letters, digits, hyphens, underscores and dots) or an IPv6 address in - brackets, either one optionally followed by `:` and a port, and any other host - is refused with 400 (closes #275): only `@` was refused, so a `?` written as - `%3F` gave the host `cdn.example.com?x=1`, which Go's URL parser reads as - `cdn.example.com` with a query. Such a URL passed the allowlist, and each text - after the `?` fetched the host's root path and cached the answer under keys of - its own. A `#`, `%`, `\` or space written percent-encoded is refused the same - way, and so is an IPv6 address without brackets, which needed a signature - before. Underscores are kept because `allowlist_hosts` accepts host names - holding them. -- 2026-10-15 the README says how a source URL with a query is written into an - image URL, and the image route accepts it (closes #255): the query follows the - source path after `%3F`. pixa now reads the image URL's path decoded once, so - the `?` before the query, any `?` in it and every `%` in the source path or - query must be percent-encoded, and any other character may be. The route used - to parse chi's wildcard, which holds the path as the client wrote it whenever - that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as - `%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from - being read as `?`, and the request failed with 401. `GenerateSignedURL` now - writes Go's own encoding of the path. The README has an example with a query - and its signature. The path traversal check no longer reads the source query, - and keeps a `%` that does not start a percent-escape as it is instead of - refusing the URL, so a source path or query holding `%25` is accepted, while a - double-encoded `..` next to such a `%` is still refused. -- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees, - rounded down, and at least 1 (closes #242): it defaulted to one image per CPU, - and a container with no CPU limit sees every CPU of the host, so on a 48-CPU - host pixa processed 48 images at once, each of which can take about 4.2 GiB. - `config.DefaultMaxConcurrentProcessing` gives the default to the config and to - `imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it. - "Running under upaas" in `README.md` gives the worst-case memory, measured, - and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and - `PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the - file libvips writes to `/tmp` for each large source, up to 1 GiB, measured, - which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where - `/tmp` is a tmpfs. -- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit - `e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`, - `.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`, - `.prettierignore`, `.prettierrc`, `REPO_POLICIES.md` and the standard scripts - are its copies. pixa's own entries follow the standard part: `/bin`, `/data` - and the local config files in `.dockerignore`; Go's `*.log`, `*.out` and - `*.test`, `vendor/`, the binaries, `/data/`, `*.sqlite3` and the local config - files in `.gitignore`. The workflow keeps its `script/docker-smoke` step while - https://git.eeqj.de/sneak/pixa/issues/203 is open. The scripts pass a new - `CACHEBUST` build argument in place of `--no-cache`, and every `Dockerfile` - stage declares it first, so every `RUN` step still runs on every build while - Go's build cache, mounted on the steps that compile, is kept; both `go test` - runs keep `-count=1`. The lint and test builds write no image. The lint phase - runs golangci-lint v2.14.0, the test phase runs on the Debian Go image, and - the build stage marks `/src` safe for git. `script/bootstrap`, `script/fmt` - and `script/fmt-check` use the node on the `PATH` only when it has the pinned - major version. Of golangci-lint v2.14.0's two findings, an unused `nolint` - directive is removed, and gosec's new path check is marked as not applying to - the fixed config file locations. The agent guidance moves unchanged from - `CLAUDE.md` to a root `AGENTS.md`, where `REPO_POLICIES.md` keeps it. -- 2026-10-15 the request metrics record any method that is not a standard one as - `other` (closes #274): their `method` label was each request's method as sent, - and Go's HTTP server accepts any method name, so a client sending made-up - methods added series to the Prometheus registry that were never removed. The - label is now the request's method only for `GET`, `HEAD`, `POST`, `PUT`, - `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. The metrics library reads - the method from the request it is handed, so the metrics middleware hands it a - copy of the request carrying the label; the handlers still see the method the - client sent. -- 2026-10-15 logging out ends every login session (closes #256): a login session - was a cookie with nothing kept on the server, so `/logout` removed it from one - browser and a copy of it kept working for its 30 days. Every login is made - with the one signing key, so logging out now ends them all: the session - manager keeps a random value, made with `crypto/rand` when the new `logout` - table has none and replaced by each logout, saves it there so that it outlasts - a restart, puts the value at login in each session, and refuses a session - carrying another value. The check for a logout compares no time, so a login - after a logout works whatever the host's clock does; a session's 30 days are - still checked against the clock. Losing the state directory ends every login - session, and no logout can make an ended one work again. Logging out is now - `POST /logout` from a form on the generator page, behind the CSRF check the - other forms have, so a link or image elsewhere on the same site cannot end the - sessions. A logout without a valid login session ends none and only removes - the cookie, as anyone can take a CSRF token from the login page. A logout that - cannot be saved answers 500. `session.NewManager`, which the tests use, keeps - the value in memory only; pixad uses `session.NewManagerWithDatabase`. -- 2026-10-15 a cached file left empty or partial is no longer served, or fetched - again on every request (closes #252). Source and transformed image files, and - each transformed image's `.meta` file, are synced to disk before they are - renamed into place, and their directory after; the `.meta` file is written - before the image, and a failed directory sync removes the renamed file. The - reconciliation pass, at startup and every five minutes, removes a cached - source or transformed image whose file is empty, or not the size the database - records for it, with its rows and a warning naming the file, so the next - request fetches the source or converts the image again; requests compare no - sizes. Storing a source over a file of another size replaces that file. A - source that fails to decode is not removed for that alone: a damaged upstream - image would then be fetched on every request. -- 2026-10-15 a transcode stops once its request's `downstream_timeout` has - ended, unless it has begun encoding (closes #243): the image processor looked - at its context only while it waited for a processing slot, so an image was - decoded and encoded, holding the slot, however long after its request's - deadline that took. `Process` now returns the context's error, and gives the - slot back, when the context has ended by the time it has the slot or before it - encodes. A request that waited for a shared transcode stopped at the first - request's deadline is answered 503 `server busy, try again later`, not 500, - though its own `downstream_timeout` may not have ended. govips gives pixa no - way to stop libvips once an encode has begun, so that encode keeps its slot - until it finishes; the limit of 8192 on each side of the source and of the - output bounds how long that takes. With one libvips thread, 8192x8192 random - pixels take up to about 1 second to encode as JPEG, 15 as PNG, 16 as GIF, 63 - as WebP, 110 as AVIF and 96 as JPEG XL, the last three with an alpha channel, - plus up to about 11 seconds to decode the source; `README.md` gives these - times and no longer says the work stops when `downstream_timeout` ends. The - settings of the WebP, AVIF and JPEG XL encoders, which can hold a slot past - the default `downstream_timeout` of 60 seconds, are unchanged. -- 2026-10-15 "Running under upaas" in `README.md` tells the operator to set - `PIXA_CACHE_MAX_BYTES` (closes #251): it listed the variable as optional, but - under upaas the volume is a directory on the host's filesystem. The default, - 75% of the sum of that filesystem's free space and what the cache holds, and - at least 500 MiB, let one pixa take three quarters or more of the space the - host and every other app had left, and all of it when that was 500 MiB or - less. The section now says so. The default is unchanged. -- 2026-10-15 the request metrics are labelled by route, not by path (closes - #254): their `handler` label was each request's path, so every new path, a 404 - or an encrypted URL, added series to the Prometheus registry that were never - removed, and `/metrics` listed every encrypted URL's token. The label is now - the pattern of the chi route the request matches, such as `/v1/image/*`, or - `unmatched` for a request that matches none: one the router answers 404 or - 405, or a CORS preflight. The metrics library takes the label before the - request is routed, so the metrics middleware asks the router which route - matches. -- 2026-10-15 expired negative cache rows are removed on a schedule (closes - #244): the row of a failed fetch, which makes pixa answer 502 for that source - URL for 5 minutes, was deleted only when the same URL was requested again - after that, so a client asking for a new missing path each time added rows - that stayed for good. The eviction goroutine now calls `CleanExpired` at - startup and on each periodic tick, every 5 minutes in pixad, so a row stays at - most about 10 minutes. `CleanExpired` deletes at most 1000 rows per statement, - so a request waits behind 1000 deletions at most, however many rows have - expired. The goroutine runs with the disk cache disabled - (`cache_max_bytes: 0`) too, where it does nothing else, as the negative cache - is used either way. -- 2026-10-15 a cached variant is served while its source URL is in the negative - cache (closes #245): `Get` checked the negative cache before the variant - cache, so once a fetch of the source failed, as one does when another variant - misses after the source was evicted and the upstream is down, every variant of - that source answered 502 for 5 minutes, also one on disk or one that could be - made from the cached source. The negative cache is now checked only just - before a fetch from the upstream, so a cached variant is served and a variant - not in the cache is made from the cached source when there is one. `README.md` - says so under `/v1/image/`. A refusal from the negative cache now counts as a - miss, as any other request that misses the variant cache does. -- 2026-10-15 the host of a `/v1/image/` URL is lower-cased, and one with a port - needs a signature (closes #246): with `cdn.example.com` allowlisted, - `CDN.Example.com` and `cdn.example.com:8443` passed the allowlist too, and - each spelling was fetched and cached on its own and had connections of its own - under `upstream_connections_per_host`. The image URL parser now lower-cases - the host and refuses one holding `@`, which a URL reads as a user name and - password, with 400. A host with a port, or with a `:` and no port, needs a - signature even when it is allowlisted. That is checked in the image service, - not in `allowlist.IsAllowed`, which also matches `referer_blocklist`, where a - `Referer` is blocked whatever its port. A URL is signed over its host as it - writes it, in lower case, with any IPv6 brackets, `:` and port kept. -- 2026-10-15 a `:` in the source path or query no longer moves where the path - ends and the query begins (closes #247): the signed data and the cache key - joined host, path and query with `:` and no escaping, so a URL signed for - `https://h/a:b` verified for `https://h/a?b:`; the two also shared a cache - key, and whichever was fetched first was served for both. In both, the path - and the query now have each `%` written as `%25` and each `:` as `%3A`; a path - and query holding neither are signed and keyed as before. Signature - Specification in `README.md` says so. -- 2026-10-15 the example config allowlists no host (closes #249): it allowlisted - `github.com` and `user-images.githubusercontent.com`, where anyone can put an - image, so a deployment set up as Getting Started says served those images to - anyone without a signature, at any size and in any format. - `configs/config.example.yml` now leaves `allowlist_hosts` out, with one - commented entry under `example.com`. Getting Started and "Running under upaas" - in `README.md` say that every image on an allowlisted host is served without a - signature, so an allowlisted host must be one the operator controls. -- 2026-10-15 a chunked upstream body of exactly `upstream_max_response_size` - bytes is accepted (closes #248): once the limit was used up, the fetcher's - next read refused the body without looking for its end. Go's HTTP client - returns the end of a body sent with a `Content-Length` together with its last - bytes, but the end of a chunked body can come in a later read, so such a body - was refused after it had been downloaded whole, and the image routes - answered 500. Once the limit is used up, the fetcher now reads one more byte: - if the body has ended it is accepted, and if there is more it is refused with - `ErrResponseTooLarge`, as before. -- 2026-10-13 `fit=outside` no longer makes an output above 8192 on a side - (closes #257): it scales the image to cover the requested size, and the side - scaled past that size was held to no maximum, so `1x8192?fit=outside` of a - 3000x600 source was resized to 40960x8192 and encoded. The image processor now - refuses such a size with `ErrOutputTooLarge` before resizing, and both image - routes answer it 400, as for a size with one side `0`. The other fit modes - make the requested size or one within it, so none can go above 8192. -- 2026-10-13 a size with one side `0` no longer makes an output above 8192 on a - side (closes #241): that side is computed from the source's proportions, and - nothing held it to the 8192 a requested side is held to, so `0x8192` of a - 3000x600 source was resized to 40960x8192 and encoded. The image processor now - refuses such a size with `ErrOutputTooLarge` once it knows the source's - dimensions, before resizing, and both image routes answer it 400 with the JSON - error body. The refusal is not cached as a failed fetch, so the same source is - still served at other sizes. `imgcache.MaxDimension` is now the image - processor's `MaxOutputDimension`, so the two cannot differ. -- 2026-10-13 the unused cache lifetime is removed (closes #69): - `CacheConfig.CacheTTL`, which pixad set to `DefaultCacheTTL` (24 hours), was - never read, so cached images never expired. The field and the constant are - removed. Cached source and transformed images stay until eviction removes them - to keep the cache within `cache_max_bytes`. -- 2026-10-13 SVG sources are refused, and `orig` is never saved as JPEG in place - of the source's own format (closes #68): the upstream fetch no longer accepts - `image/svg+xml`, which libvips would render. Both image routes answer 415 for - an upstream `Content-Type` that is not one of the source formats, SVG - included, where they answered 500. Before decoding, the image processor - refuses with `ErrUnsupportedInputFormat`, also answered 415, any input that - govips' `DetermineImageType`, which picks the libvips loader from the first - bytes, does not name as a source format. That catches an SVG whose first bytes - pass the first-bytes check for another type, as an XML comment can make them - do for AVIF; an SVG whose first bytes do not pass that check answers 500, as - before, and so does a damaged image in a source format, whose first bytes - `DetermineImageType` names. `orig` is always the source's own format, where a - source the processor could not name was saved as JPEG. `README.md` no longer - lists SVG. The SVG detection in `internal/magic` stays: the request path does - not call it, and it goes with the unused code of - https://git.eeqj.de/sneak/pixa/issues/168. -- 2026-10-12 per-client limit on the image routes (closes #236): the new setting - `image_requests_per_minute` (`PIXA_IMAGE_REQUESTS_PER_MINUTE`), default 600, - is how many `GET` and `HEAD` requests one client may send to `/v1/image/` and - `/v1/e/` together per minute, cache hits included; the next is answered 429 - with `Retry-After`. It uses the login limit's `RateLimit` middleware, so - clients are told apart the same way, and runs after the CORS middleware, so a - preflight is not counted. `0` turns the limit off; a negative or non-integer - value aborts startup. -- 2026-10-08 a request past its deadline no longer waits for the database to - count it (closes #224). On a busy host, `TestService_Get_ReturnsByItsDeadline` - failed because its request, past its deadline, still waited for its miss count - to be written, a write with no deadline, which in pixad waits for the one - database connection every request shares. Each count write now keeps the - request's deadline but not its cancellation. A count not written by then is - kept in memory, where `Stats` sees it, and one goroutine of the cache writes - those counts, one write at a time, and once more at shutdown before the - database closes. The test phase of the `Dockerfile` also runs at most 4 tests - of a package at once (`go test -parallel 4`), where it ran as many as the host - has CPUs: on a busy host they then wait so long to be scheduled that a test - that times a request can fail. `-p`, how many packages are tested at once, is - unchanged, as capping it also slows compiling. -- 2026-10-08 every output format is saved with settings pixa sets on purpose - (closes #232): each format has its own govips export, as JPEG XL does, in - place of govips' generic `Export`, which sent libvips a zero for some settings - it was not given. PNG gets libvips' default compression, 6, where it had none, - and WebP libvips' default effort, 4, where it had 0. GIF was already at - libvips' default effort, 7, and JPEG output is unchanged. AVIF is saved at - effort 1, the lowest govips can set, where it had libvips' default, 4. With - one libvips thread, an 8192x8192 image of random pixels, the worst case, takes - about 51 seconds as AVIF at effort 1 and 43 as WebP at effort 4, against the - default `downstream_timeout` of 60 seconds. On an image of milder noise, which - AVIF at effort 1 saves in about 12 seconds, effort 4 takes minutes. AVIF is - also saved with 8 bits per sample from a 16-bit source, which libvips would - save with 12: a 16-bit 8192x8192 image of milder noise takes about 54 seconds - at effort 1 with 12 bits, nearly all of the default `downstream_timeout`, and - about 12 with 8. -- 2026-10-08 JPEG XL is the default output (closes #222): a `/v1/image/` URL - whose last segment is a size with no format, such as `800x600` or `orig`, is - served as JPEG XL and signed as `jxl`, so it has the signature of the same URL - ending in `.jxl`. An encrypted URL whose token holds no format is served as - JPEG XL (`encurl.DefaultFormat`). The generator page selects JPEG XL until - another format is chosen, and a form with an empty format, or none, makes a - JPEG XL URL whose name ends in `.jxl`. The image processor no longer takes an - empty format as `orig`: both routes give every request a format, and it - refuses a request with none. `auto` still ends with JPEG. -- 2026-10-08 JPEG XL as an input and output format (part of #222): a source - whose bytes start with either JPEG XL signature, the bare codestream's `FF 0A` - or the container's, is detected as `image/jxl`, which the upstream fetch - accepts; `orig` of such a source is JPEG XL. The format `jxl` works in plain - and encrypted URLs and on the generator page, served as `image/jxl` and cached - like the other formats. `auto` chooses JPEG XL first when `Accept` names - `image/jxl`. govips sends libvips a JPEG XL distance with every save, so - libvips ignores the quality: pixa turns `q` into a distance with libvips' own - formula, keeping 100 lossy, and removes the metadata from the image before - saving, as govips cannot have libvips strip it from JPEG XL. libvips 8.16 and - later still write an EXIF block of their own into JPEG XL, from the image's - size, orientation and resolution, and fixed values; the image is upright and - is given 72 dpi before the save, so the block holds nothing from the source. - An image with an ICC profile is converted to sRGB before the JPEG XL save too, - and a CMYK image with none is converted to sRGB, as libvips cannot save CMYK - as JPEG XL. libvips' default effort, 7, is kept. JPEG XL is not yet the - default output. -- 2026-10-08 pixad runs on the Alpine release it is built on (closes #229): the - runtime stage of the `Dockerfile` uses `alpine:3.22`, the release of the - `golang:1.25.4-alpine3.22` image that the test phase and the build stage use, - so all three have libvips 8.16, where the runtime image had 8.15. -- 2026-10-08 requests no longer wait behind eviction queries that read a whole - table (closes #227): the new `cache_usage` table holds the total cache usage, - kept up to date by triggers on `source_content` and `variant_content` in the - statement that adds, removes or resizes a row, and `UsageBytes` reads it. The - reconciliation pass reads the content tables 1000 rows per query, sums them - and corrects the total when it differs, unless a row changed while it summed. - Source rows get `last_accessed_at` when added, so choosing source images to - evict reads that column's index instead of sorting the whole table. -- 2026-10-08 libvips' JPEG XL support is installed and required (part of #222): - `script/bootstrap --cgo` installs `vips-jxl` when its package manager is apk, - as Alpine's `vips` package lacks the support, and the runtime stage of the - `Dockerfile` installs it too. `imgcache.NewService` fails, naming the fix, - when `imageprocessor.CheckJPEGXLSupport` finds that libvips cannot load and - save JPEG XL, so pixad does not start without it. A test saves an image as - JPEG XL with govips and loads it back. JPEG XL is not yet a format pixa - serves. -- 2026-10-08 SQLite writes no longer fail with "database is locked" under load - (closes #223): `internal/database` opens the database with one connection, so - pixa's own reads and writes run on it one at a time instead of competing for - SQLite's lock, where a write that kept losing could wait past the five-second - busy timeout and be lost. The busy timeout stays, for another program writing - to the same file. No code in pixa keeps rows or a transaction open while it - runs another query, which with one connection would wait forever. -- 2026-10-05 the format `auto` (closes #88): a format in the `/v1/image/` path, - an encrypted URL's token and the generator page's format choice, chosen for - each request from `Accept` once the signature or token is checked: AVIF when - the header names `image/avif`, else WebP when it names `image/webp`, else JPEG - when the first of `image/jpeg`, `image/*` and `*/*` that it names allows it, - or when it names nothing; `q=0` refuses a format. AVIF and WebP must be named, - as clients that cannot show them send the wildcards too. A header that allows - none of the three answers 406, one that does not parse 400. The signature and - the token cover `auto` itself; the cache key and `ETag` use the format chosen. - Answers from the point the format is chosen carry `Vary: Accept`, next to the - CORS `Vary: Origin`; fixed-format answers do not. -- 2026-10-05 the markdown is formatted with prettier (closes #100): `script/fmt` - and `script/fmt-check` run prettier 3.8.1, pinned in `package.json` and - `yarn.lock`, on `**/*.md` after `gofmt`, with four-space tabs and - `proseWrap: always` as `.prettierrc` says; `.prettierignore` keeps it off - `REPO_POLICIES.md`, the copy from `sneak/prompts`, and `vendor/`. Plain - `script/bootstrap` installs Node and Yarn as the one in `sneak/prompts` does - and then prettier; `script/bootstrap --cgo` does not, as the `Dockerfile` - stages that run it format nothing. The HTML templates stay unformatted: - prettier cannot parse a Go template action inside a tag. The markdown was - reflowed in a commit of its own. -- 2026-10-05 lint and tests run as the `lint` and `test` phases of the - `Dockerfile`, built with `--no-cache` (closes #202): `script/check`, - `script/cibuild`, `script/docker`, `script/lint`, `script/test`, - `script/setup` and `script/install-precommit` are now the copies from - `sneak/prompts` `main`, unchanged. The `lint` phase runs golangci-lint from - the image `REPO_POLICIES.md` names, with `libvips-dev` from `apt-get`; the - `test` phase runs the tests with a 90-second timeout; the build stage depends - on both. `Dockerfile.lint` and the `CHECK_EPOCH` build argument are gone, the - formatting check runs on the host, and `make docker-versioned` and - `make docker-test` call the scripts. `script/bootstrap`, `script/fmt`, - `script/fmt-check`, `script/precommit` and `script/projectname` stay pixa's - own. `script/bootstrap` installs git, make and Go, refreshing apt's package - lists before its first apt install; with `--cgo`, which only the `test` phase - and the build stage pass, it also installs the C compiler and the libvips and - libheif libraries. The stage that compiles still takes the version from - `git describe` when no `VERSION` is given, per - https://git.eeqj.de/sneak/pixa/issues/166, so the copied scripts' comment that - `.dockerignore` leaves out `.git` does not hold for pixa. -- 2026-10-04 load test (closes #81): `script/loadtest [duration [clients]]` - (`make loadtest`, defaults `10s` and `4`), a benchmark that `script/check` - does not run, measures three scenarios, each against a new pixad container and - a new upstream host, `cmd/loadtest-origin`: `hit` (one cached image), `miss` - (a new source image every request) and `herd` (each new source image asked for - by all clients at once). For each it prints vegeta's report (requests per - second, latency percentiles, status codes), pixad's peak resident memory and - the requests that reached the origin. `README.md` says how to run it and read - it, and keeps 1-5k r/s as a target not yet measured. First measurement, with - the defaults on a shared 48-CPU machine with other work running: a baseline - for later changes, not a test of the target. `hit` 1413 r/s, p50 0.7 ms, p95 - 8.7 ms, p99 44 ms, peak 53 MiB (4 clients that each wait for their answer, so - not pixad's limit); `miss` 70 r/s, p50 52 ms, p95 91 ms, p99 122 ms, peak 100 - MiB, one fetch per request; `herd` 74 r/s, p50 52 ms, p95 69 ms, p99 111 ms, - peak 60 MiB, 188 fetches for 749 requests. -- 2026-10-04 the CI checkout fetches the tags (closes #208): the checkout step - in `.gitea/workflows/check.yml` sets `fetch-depth: 0`, as `REPO_POLICIES.md` - asks of a repo that takes its version from the tags, so a CI build of a tagged - commit stamps the tag from `git describe` instead of a bare commit. -- 2026-10-04 `config.yml` stays out of git and the Docker build context (closes - #212): `.gitignore` now ignores `config.yml`, the config file Getting Started - creates with the signing key, and `.dockerignore` leaves it out in every - directory and in any letter case, as it already did `config.yaml` and - `config.dev.yml`. -- 2026-10-04 local config files stay out of the Docker build context (closes - #211): `.dockerignore` now leaves out `config.yaml` and `config.dev.yml` in - every directory and in any letter case, the local config files `.gitignore` - keeps out of git because they can hold the signing key. - `configs/config.example.yml` is still sent. `config.yml`, which Getting - Started creates, is in neither file: - https://git.eeqj.de/sneak/pixa/issues/212. -- 2026-10-04 `cmd/pixad/main.go` is one call into `internal/` (closes #206): - what it did (the command line and its `--config` flag, setting - `PIXA_CONFIG_PATH`, ignoring `SIGPIPE`, starting the fx app) is now `Run` in - `internal/app`, unchanged, and `main` calls it with `Version`, which the build - still sets through `-X main.Version`. That code had no tests to move. -- 2026-10-04 `.gitignore` ignores `.claude/` (closes #204): the entry and its - comment are copied from the canonical `.gitignore` in `sneak/prompts`, - unanchored so it matches at every depth. `.dockerignore` already has - `.claude`. -- 2026-10-04 `.dockerignore` keeps secrets out at every depth (closes #205): the - file is now the standard one from `sneak/prompts`, whose patterns match in - every directory and, for environment files and private keys, in any letter - case, so a nested `.env` or `server.key` no longer reaches the build context. - pixa still sends `.git` without `.git/config` in place of the standard file's - `.git` line, and still leaves out `.gitignore`, `/bin` and `/data`. -- 2026-10-04 `REPO_POLICIES.md` matches the canonical copy again (closes #196): - it is replaced, unchanged, by `prompts/REPO_POLICIES.md` from `sneak/prompts` - `main`. The rules it adds that pixa's tree breaks are filed: - https://git.eeqj.de/sneak/pixa/issues/202 (lint and tests as `Dockerfile` - phases built with `--no-cache`), https://git.eeqj.de/sneak/pixa/issues/203 - (the workflow's `script/docker-smoke` step), - https://git.eeqj.de/sneak/pixa/issues/204 (`.claude/` in `.gitignore`), - https://git.eeqj.de/sneak/pixa/issues/205 (`.dockerignore` patterns at every - depth), https://git.eeqj.de/sneak/pixa/issues/206 (a thin `cmd/pixad/main.go`) - and https://git.eeqj.de/sneak/pixa/issues/208 (`fetch-depth: 0` on the CI - checkout, so the build sees the tags). Its rule that no build stage runs - `git describe` is not followed: pixa takes the version from the `.git` in the - build context, per https://git.eeqj.de/sneak/pixa/issues/166, as the copy on - `sneak/prompts` `next` already says. -- 2026-10-04 an integration test of the image proxy flow (closes #80): - `TestImageProxyFlow` in `internal/server` starts the database, handlers and - middleware from the constructors `pixad` uses, with a fresh state directory, - and replaces only the upstream origin with a local test server. For a resize - with a change to JPEG and for `orig`, the first request goes through the - router, the real fetcher, libvips, the disk cache and SQLite and answers 200 - with the right content type and size and `X-Pixa-Cache: MISS`; the second - answers `HIT` with the same image and the upstream has had one request; the - source and the converted image are then in `cache/sources` and - `cache/variants`, with their rows in `source_content`, `source_metadata` and - `variant_content`. Two optional fields make this possible, which `pixad` does - not set and the config file and environment cannot: - `httpfetcher.Config.DialContext` connects in place of the dialer that refuses - internal addresses, the URL and redirect checks still running, and - `handlers.Params.Fetcher` replaces the fetcher the handlers build. -- 2026-10-04 a URL made on the generator page with a `ttl` is tested to expire - (closes #199): a new test in `internal/handlers` makes a URL on the generator - page with a `ttl` of one second, checks that `/v1/e/` serves it at once, waits - two seconds and checks that it then answers 410. The test waits for real, as - pixa reads the clock directly when it makes and checks a URL; it waits two - seconds because the time a URL expires is kept in whole seconds. Test only. -- 2026-10-04 referer blocklist (closes #90): `referer_blocklist` - (`PIXA_REFERER_BLOCKLIST`) lists hosts, written and matched as for - `allowlist_hosts` with the same matcher; an entry of either list that is - neither a host name (letters, digits, hyphens, underscores and dots, with at - most one leading dot) nor an IP address, such as one with a port or a `*.` - wildcard, aborts startup naming the setting and the entry. Both image routes - refuse a request whose `Referer` names a listed host with 403 and a JSON error - before the signature, the cache and the upstream fetch, so it fetches nothing - and is refused whether or not the image is cached. A request with no - `Referer`, or one that does not parse as a URL with a host, is served, so the - list is easily got around; `README.md` and `configs/config.example.yml` say - so. It does not apply to the login and generator pages. -- 2026-10-04 fewer files in the repository root (closes #97): - `config.example.yml` moved unchanged to `configs/config.example.yml`, and - `README.md`, the comments in `internal/config/config.go` and the startup error - for the placeholder signing key name the new path; `scripts/manual-test.sh` - and its directory are deleted, as the handler tests in `internal/handlers` - cover every check it made except two: fetching a real image from the internet, - and a URL made on the generator page with a `ttl` answering 410 once the `ttl` - has passed (https://git.eeqj.de/sneak/pixa/issues/199); `CONVENTIONS.md` is - deleted, as `REPO_POLICIES.md` links the canonical Go HTTP server conventions. -- 2026-10-04 SQLite writes no longer fail with "database is locked" (closes - #198): pixa adds `_pragma=busy_timeout(5000)` to every `db_url`, so a write - that finds another in progress on another connection waits up to five seconds - for it, and the default `db_url` turns on WAL mode with - `_pragma=journal_mode(WAL)`. The old default's `_journal_mode=WAL` is not a - parameter the driver reads, so the database was never in WAL mode. -- 2026-10-04 `TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup` only - passes through a periodic pass (closes #189): it slept for three eviction - intervals before writing its file, and a startup pass still running then could - adopt the file itself. It now holds the test database's only connection until - the startup pass waits for it after walking the empty variant directory, - writes the file and lets the connection go, as - `TestEvictionRunsOnPeriodicSchedule` does, so only a periodic reconciliation - pass can adopt the file. Test only. -- 2026-10-04 logging in, logging out, the URL generator and `/v1/e/` have - handler tests (closes #77): new tests in `internal/handlers`, with no network, - check that `GET /` without a login session shows the login form; a wrong key - shows it again with an error and sets no session cookie; the right key answers - 303 to `/` with a session cookie marked `Secure`, `HttpOnly` and - `SameSite=Strict`, with which `GET /` shows the generator page; `GET /logout` - answers 303 to `/` with an empty session cookie sent with `Max-Age=0`; - `POST /generate` without a login session answers 303 to `/`; `/v1/e/` serves - the image for a valid token, answers 410 for an expired one and 400 for one - with a character changed, cut short or made with another signing key; and a - URL made on the generator page is served by `/v1/e/`. No code changes. -- 2026-10-04 `TODO.md` merges with git's union merge (closes #190): a root - `.gitattributes`, copied from `sneak/prompts`, marks it `merge=union`, so two - branches that each add an entry at the top of Completed Steps merge without a - conflict and keep both entries. Git now never reports a conflict in `TODO.md`: - a real one keeps both versions of the lines, and two entries that share an - identical line can end up one inside the other, which a rebase can do to an - entry already on `next`. The Workflow above says to read the merged entries - after every merge or rebase. -- 2026-10-04 the default `cache_max_bytes` no longer shrinks as the cache fills - (closes #184): for an omitted key, the cache works out the limit when it - opens, after the database is open, as 75% of the sum of the free space on the - filesystem containing `/cache/` and what the cache already holds by - its own size accounting, at least 500 MiB, so a cache filled to its limit - keeps that limit across a restart. The computation and its tests moved from - `internal/config` to `internal/imgcache`; the config only records whether the - key was set. -- 2026-10-04 `TestEvictionRunsOnPeriodicSchedule` no longer races the evictor - (closes #183): it wrote each variant file and then inserted its accounting row - by hand, and a reconciliation pass between the two adopted the file first, so - the insert failed. It now writes the files only, while holding the test - database's only connection so the evictor's startup pass waits after walking - the empty variant directory; a periodic reconciliation pass then adopts the - files and the eviction pass after it evicts them. No other test in - `internal/imgcache` inserts a row by hand after starting the evictor. Test - only. -- 2026-10-04 a config file pixa cannot read aborts startup (closes #176): of the - places pixa looks for its config file on its own, only one where the file does - not exist is passed over; any other error, such as a directory on the path - that pixa may not enter, aborts startup naming the file, as a file that does - not parse already did. -- 2026-10-04 `.golangci.yml` re-vendored from the canonical copy (closes #57): - the deprecated `gomodguard` is switched off, so lint runs print no deprecation - warning; its successor `gomodguard_v2` runs with the shared module block list, - and `depguard` keeps `net/http/httptest` out of files that are not tests. The - tree needed no code changes. -- 2026-10-04 the Content-Security-Policy allows no inline script or style - (closes #125): `script-src` and `style-src` are `'self'` only. The generator - page's two inline `onclick` handlers moved into - `internal/static/generator.js`, attached with `addEventListener`; the bundled - Tailwind script, which built styles in the browser, is replaced by a small - hand-written `internal/static/style.css` with only the rules the login and - generator pages use, the templates carrying a few plain class names in place - of Tailwind's. No build step. The pages keep their layout, not every pixel of - it. -- 2026-10-04 deployment guide and example Caddy config (closes #89): - "Deployment" in `README.md` says what the reverse proxy in front of pixa must - do (terminate TLS; pass `Host`, `Origin` and `Referer` on unchanged; set - `X-Forwarded-For`, with `trusted_proxies` to match; wait at least - `downstream_timeout`; optionally refuse `/metrics`) and what pixa does itself, - that the state directory needs a persistent volume and what `cache_max_bytes` - counts, the health check for a load balancer, what a stop does and its exit - codes, and what running outside Docker needs; `configs/Caddyfile` is the - example, checked with `caddy validate`. -- 2026-10-04 the metrics basic auth, CORS preflight, request logging and metrics - recording have tests (closes #79): `MetricsAuth` on its own answers 401 with a - challenge without credentials or with a wrong username or password and lets - the configured ones through; a preflight request gets `*` for any origin when - `access_control_allow_origin` is `*` and no `Access-Control-Allow-Origin` from - another origin than the configured one; a `POST /` carrying the signing key - leaves no trace of it in the request log line, and the login handler's own log - lines leave out the submitted key; the metrics middleware on its own records a - request it served, and the router records nothing while no metrics username is - set. Not tested: that the router puts the basic auth in front of `/metrics` - and records requests when a metrics username is set. Only one test per package - can set up `/metrics`, and in `internal/server` that is - `TestMaintenanceModeKeepsOtherRoutes`, which needs the owner's approval to - change; #180 holds it. Tests only; the basic auth library already compares the - password in constant time. -- 2026-10-04 the image route's signature check and error answers are tested - (closes #76): new tests in `internal/handlers`, with no network, check the - status and JSON error body for a missing, wrong, unpadded, upper-case or - expired signature on a host not on the allowlist, or a valid one sent for its - parent domain, a sibling host, a subdomain or the host with another domain - appended (401), an unparseable path (400), `localhost` as the upstream host - (403) and an upstream error (502); that an allowlisted host is served without - a signature, another host only with a valid one; and the answers of - `/robots.txt` and the health check. No code changes. -- 2026-10-04 request IDs returned and passed on, and `/v1/e/` revalidates - (closes #84): pixa's own `RequestID` middleware, in place of chi's, gives each - request an ID, its own `X-Request-ID` when that is at most 64 letters, digits, - `-`, `_` or `.` and a random one otherwise, stores it where chi's did and - sends it back as `X-Request-ID` on every response; the upstream fetch sends - that ID, and the "upstream fetched", "image converted" and "image served" log - lines carry it as `request_id`, a fetch shared by several requests carrying - the first request's; `/v1/e/` sets `ETag`, answers a matching `If-None-Match` - with 304 and is routed for `HEAD`, the `ETag` and 304 code being - `notModified`, which `/v1/image/` calls too; its token checks moved unchanged - into `parseImageEncRequest` to keep `HandleImageEnc` within the line limit; no - `Vary` is added, as no response depends on a request header except the image - routes' CORS headers, for which `go-chi/cors` already sends `Vary: Origin`; - `Vary: Accept` is left to #88. -- 2026-10-04 routes, encrypted URLs and config file documented (closes #75): - "Routes" in `README.md` lists every route with its method, purpose, what it - needs and the status codes it answers with, and says `q` and `fit` are part of - what is cached; "Encrypted URLs" covers logging in, making one on the - generator page, how long it lasts and the 410 once it has expired; - "Configuration" gives the order in which pixa looks for its config file; - `config.example.yml` lists `db_url` and `env` and gives every key's default; - `scripts/manual-test.sh` is left to #97. -- 2026-10-04 shutdown stops cache eviction in progress (closes #102): - `StartEviction` runs the eviction goroutine with its own context, which - `StopEviction` cancels, so a pass in progress stops at its next database call, - file, row or eviction candidate instead of running to completion, and no pass - starts after it, so a stop logs at most one warning; `StopEviction` takes a - context and, when that context ends before the goroutine exits, stops waiting - and returns its error; the handlers' stop hook passes fx's stop context, so an - eviction still running when fx's stop deadline ends fails the stop and makes - the exit code 1. -- 2026-10-04 dead code in `internal/imgcache` is gone (closes #73): `Purge`, - which only returned an error and which nothing called, is no longer part of - the `ImageCache` interface or `Service`; the `SignatureValidator`, `Allowlist` - and `Storage` interfaces, which nothing implemented or used, are deleted. - Nothing else changes. -- 2026-10-04 upstream host semaphores and variant `.meta` files no longer - outlive their use (closes #87): the fetcher counts the fetches holding or - waiting for a slot of each upstream host's semaphore and removes the host's - semaphore once none is left, so fetches from many hosts no longer leave one - semaphore each until restart; `VariantStorage.Delete` removes the variant's - `.meta` file along with it, a missing `.meta` file not being an error, and - `DeleteWithMeta`, which eviction called for that, is gone. -- 2026-10-04 `README.md` matches the code (closes #74): "Storage" names the - cache directories pixa uses (`cache/sources`, `cache/metadata`, - `cache/variants`) and how files are named in each, and the comments in - `001_schema.sql` name the same paths; the routes and the signature section - list the same output formats, `jpg` and `original` included; the TLS sentence - names `allow_http` as its exception; "Metrics" says only generic HTTP and Go - runtime metrics exist, measured and served only when the metrics username and - password are set. -- 2026-10-03 shutdown sets the exit code and waits for image processing (closes - #86): fx alone handles SIGINT and SIGTERM, and the server's own signal handler - is gone; fx's `Run` in `cmd/pixad` exits with the shutdown's code: 0 for a - signal, 1 when the HTTP server cannot listen or the app fails to start or to - stop; the server's stop hook, which fx waits for, stops the HTTP server, waits - for the images still being processed, both within 5 seconds, then flushes - Sentry; images still being processed after that are logged with their count - and make the exit code 1; a Sentry DSN that cannot be used fails startup, so - the stop hooks of what had already started run, instead of exiting the process - from a goroutine; the eviction loop is left to #102. -- 2026-10-03 every `script/cibuild` and `script/docker` run executes the checks - (closes #101): the `Dockerfile` declares `CHECK_EPOCH` above `make fmt-check` - and `make lint` in the lint stage and above `make test` in the build stage, - and each of those steps names it in its command; both scripts pass a new value - on every run, so Docker runs the checks instead of reusing cached results, - while the `script/bootstrap` steps stay cached; a plain `docker build .` still - works, leaves it empty, and reuses the check steps only for an identical build - context; the `script/cibuild` comment and `README.md` no longer say that any - successful build implies a green repo. -- 2026-09-29 share concurrent misses (closes #65): requests that miss the same - variant at once (the same cache key, so quality and fit included) share one - upstream fetch or cached source read and one transcode through - `golang.org/x/sync/singleflight`; the first request's processing ignores its - cancellation but keeps its deadline, and the others wait for its image or - error holding no upstream connection or processing slot, and stop waiting when - their own context ends; the request doing the processing waits for it even - then, until it ends, which can be after its deadline; a request whose context - has already ended starts nothing; each request counts one miss, and the - processing counts its fetch and transcode once; a panic while processing is - reported to Sentry when `sentry_dsn` is set and becomes an error for every - waiting request instead of stopping pixad; documented in `README.md`. -- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS - middleware, with the `access_control_allow_origin` origin, moved from the - router root onto a `/v1` subrouter holding `/v1/image/` and `/v1/e/`, where it - still answers a preflight `OPTIONS` request; the login and URL generator - pages, `/metrics` and the other routes send no `Access-Control-Allow-Origin`; - documented in `README.md` and `config.example.yml`. -- 2026-10-02 a plain `docker build .` stamps the tag or short commit, not `dev` - (closes #166): `.dockerignore` lets `.git` into the build context, without - `.git/config`; with no `VERSION` build argument the `Dockerfile` takes the - version from `git describe --tags --always`, and fails the build if the - context carries `.git` and no version comes out; `ARG VERSION` has no default; - pixad logs its version, with its name and architecture, as its first log line - at startup. -- 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes #159): - `deploy/docker-entrypoint.sh` creates the directory if it is missing, gives - the directory and everything in it to `pixad` when the directory or one of its - top-level entries belongs to another user or group, sets its mode to `750`, - then runs the server as `pixad`; data left by an earlier run under another uid - is taken over this way; "Running under upaas" in `README.md` no longer tells - the operator to create or chown the host directory. -- 2026-09-29 variant content types kept in memory (closes #70): - `Cache.metaCache` holds the content types of up to 10,000 variants in an LRU - (`github.com/hashicorp/golang-lru/v2`), filled by `StoreVariant` and by - `GetVariant` after it reads a `.meta` file, where a type `StoreVariant` added - meanwhile is kept over the one read, and never with the - `application/octet-stream` served for a variant without one; for a variant it - holds, `GetVariant` skips the `.meta` read, still opening the variant file and - taking the size from it; eviction removes the entry before deleting the files, - and `GetVariant` removes it when the file will not open; the cap is a - constant, not a setting; the unused `variantMeta` type is gone; `README.md` - describes it. -- 2026-09-29 maintenance mode refuses image requests (closes #71): while - `maintenance_mode` is on, `/v1/image/` and `/v1/e/` answer 503 with a - `Retry-After` header and the JSON error body, from one middleware in - `internal/server/routes.go`; the health check stays 200 and reports - `maintenance_mode`, as the image's Docker `HEALTHCHECK` requests it and upaas - marks a deploy failed when its container is unhealthy; the login and URL - generator pages and `/metrics` keep working; documented in `README.md` and - `config.example.yml`. -- 2026-09-29 bound concurrent image processing and upstream fetches (closes - #64): `max_concurrent_processing` (default the number of CPUs pixa can use) - limits the images decoded and encoded at once, and `upstream_connections` - (default 64) the connections to all upstream hosts together, on top of - `upstream_connections_per_host`; a fetch holds its connection until its image - has been processed, and a request whose source is cached reads it only once it - has a processing slot; a request that finds either limit reached waits up to - 10 seconds for a free one, then gets 503 `server busy, try again later`; - libvips runs one worker thread per image with its operation cache off; - documented in `README.md` and `config.example.yml`. -- 2026-09-29 Dockerfiles install through `script/bootstrap` (closes #95): the - `Dockerfile` lint and build stages and `Dockerfile.lint` copy `script/`, - `go.mod` and `go.sum`, then run `script/bootstrap` in place of their own - `apk add` lines, so the build dependencies are listed in one place; - `script/bootstrap` now also installs a C compiler when `gcc` is missing; the - build uses `-trimpath` and `-s -w` and keeps `CGO_ENABLED=1` for govips; - `ARG VERSION` sits just above the build, so a new version reruns neither - `script/bootstrap` nor the tests. -- 2026-09-29 migrations at the path `REPO_POLICIES.md` sets (closes #96): the - migration files moved, contents unchanged, from `internal/database/schema/` to - `internal/db/migrations/` as `000_migration.sql` and `001_schema.sql`; the - `internal/db/migrations` package embeds them and `internal/database` reads - them through its `FS()`; the `internal/database` package itself stays; the - version still comes from the filename prefix, so a database that has recorded - versions 0 and 1 runs neither again. -- 2026-09-29 `trusted_proxies` advice and signature padding in `README.md` - (closes #150): the login-limit paragraph, the `trusted_proxies` entry and - `config.example.yml` say to set `trusted_proxies` to the address pixa sees for - requests that come through the proxy, which the request log shows as - `remoteIP` while it is not trusted; for a proxy on the Docker host that - connects over `127.0.0.1` that is the Docker network's gateway, not the - proxy's own address; the signature section says `sig` is base64url with the - `=` padding kept, and gives the example's `sig` for a stated signing key. -- 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the - `pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of - the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad` is - not owned on the host by a person's login account; the first-run step of - "Running under upaas" in `README.md` names the uid and gid. -- 2026-09-29 `max-age` never outlives an expiring URL (closes #63): both image - routes build `Cache-Control` from the request's `Expires`, which an encrypted - URL's expiry now fills too; `max-age` is one year, or the whole seconds left - until the `exp` of a `/v1/image/` URL or the expiry of an encrypted URL when - that is sooner, never negative; an allowlisted host's URL that has an `exp` - follows it too; `immutable` stays, as freshness now ends at the expiry; - documented in `README.md`. -- 2026-09-28 add the four settings `README.md` documented but pixa did not have, - which aborted startup as unknown keys (closes #61): - `access_control_allow_origin` (default `*`, the CORS origin), - `upstream_fetch_timeout` (default `30s`), `upstream_max_response_size` - (default 50 MiB) and `downstream_timeout` (default `60s`, both the server's - write timeout and the per-request timeout); each has a `PIXA_` variable; - durations are positive Go duration strings, the size a whole number of bytes - up to 1 GiB, the origin `*` or one `http` or `https` origin as `README.md` - describes it; an invalid value aborts startup naming the key and the value; - documented in `config.example.yml` and `README.md`. -- 2026-09-28 cache stats report real numbers (closes #56): `Cache.Stats` counts - the cached source images and processed variants (`source_content` plus - `variant_content`) and takes their size from `Cache.UsageBytes`, instead of - reading `request_cache` and `output_content`, which nothing writes; those two - tables are left in the schema; a disabled disk cache reports no items and no - size. A hit is counted even when the request context has ended. A miss is - counted after it is served or fails, also when the request context has ended - by then, with the bytes it read from upstream, so `upstream_fetch_count` and - `upstream_fetch_bytes` move, including for an upstream body that fails partway - or a fetched source that then fails the magic byte check; `transform_count` - counts each image the image processor transcodes. -- 2026-09-28 strip metadata from processed images (closes #82): every output is - exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC - profile; the image is first turned upright with `AutoRotate` (before sizes are - worked out) and, when it has an ICC profile, converted to sRGB; the `orig` - format is re-encoded and stripped like any other, as pixa never serves the - source bytes; there is no setting to keep metadata; documented in `README.md`. -- 2026-09-28 rate limit the login form (closes #66): `POST /` is limited to 5 - attempts per minute per client address, and an attempt over the limit is - refused with 429 and a `Retry-After` header; the address is the one - `internal/clientip` resolves through `trusted_proxies`, an IPv6 client is - counted by its /64, and an IPv4-mapped address as the IPv4 address it carries; - the limit is a `RateLimit` middleware in `internal/middleware` on - `github.com/go-chi/httprate`, which the image routes can reuse; the library - keeps counts for the current and the previous minute only; documented in - `README.md`. -- 2026-09-28 refuse an unparseable `exp` on `/v1/image/` and log swallowed cache - errors (closes #72): an `exp` in the URL that is not a whole number, an empty - `exp=` included, is a 400 naming `exp` and the value, instead of being ignored - and answered with 401 as if the URL had no `exp`; only an `exp` missing from - the URL is unchanged; `README.md` says so where it documents `exp`. A failed - variant `.meta` write, source metadata JSON write, `Stats` count query, stats - counter update, negative cache write or expired negative cache delete is now - logged at `warn` with the path or key and the error, and stays non-fatal. -- 2026-09-28 refuse an empty `fit` on `/v1/image/` (closes #139): a `fit` in the - URL with an empty value (`fit=`) is a 400 naming `fit`, instead of being - served as `cover` and verified against a signature made for `cover`; only a - `fit` missing from the URL is still `cover`; any other value still goes - through the existing fit-mode check; `README.md` says so where it documents - `fit`. -- 2026-09-28 refuse an invalid `q` on `/v1/image/` (closes #134): a `q` that is - not a whole number from 1 to 100, an empty `q` included, is a 400 naming `q` - and the value, instead of being served at the default 85; the route reads `q` - with the generator's quality check (`parseFormInt` with `minQuality` and - `maxQuality`); only a `q` missing from the URL is still 85; a query string - that cannot be decoded, such as `q=80%`, is a 400 showing it; any query - parameter given more than once (`q`, `fit`, `sig`, `exp` alike) is a 400 - naming it, so none is read from its first value only; `README.md` states the - range and both query-string rules. -- 2026-09-28 unknown `PIXA_` environment variables abort startup (closes #133): - a variable whose name starts with `PIXA_` but is neither a setting's variable - nor `PIXA_CONFIG_PATH` aborts startup naming it, as an unknown config key - does, and `PIXA_PORT` is named with a pointer to `PORT`; the check runs after - the config file loads, so the variables the file's `env:` section sets are - checked too; documented in `README.md`. -- 2026-09-28 start on a fresh upaas volume (closes #129): the image starts as - root only to give `/var/lib/pixa` to `pixad` when `pixad` does not own it - (`deploy/docker-entrypoint.sh`), then runs the server as `pixad` through - `su-exec`, so a root-owned host directory bind-mounted there no longer stops - the container at startup; `README.md` gains a "Running under upaas" section. -- 2026-09-28 run all linting in Docker via `Dockerfile.lint` + `script/lint` - (closes #104): `make lint` calls `script/lint`, the only way the linter is - run; inside a container (both Dockerfiles set `container=docker`) it runs - `golangci-lint`, anywhere else it builds the hash-pinned `Dockerfile.lint`, - whose last step runs `script/lint` again; the `Dockerfile` lint stage runs - `make lint`; no host or nix-shell `golangci-lint` path remains - (`script/bootstrap` installs no linter); a per-run `CACHEBUST` build-arg keeps - the lint step from being served from cache, and a tmpfs mount on that step - keeps Go's and golangci-lint's caches out of its layer, so a run leaves no - large build cache behind; `golangci-lint config verify` stays out, as it - fetches its schema over an unpinned live HTTPS call -- 2026-09-28 every setting as an environment variable (closes #128, also covers - #99): each config key can be set by `PIXA_` plus the key in upper case (`.` - written as `_`), and the port by `PORT`; a variable present in the - environment, even empty, wins over the config file, which wins over the - default; the typed getters read the variable first, so every existing check - applies to it and a bad value aborts startup naming the variable; lists are - comma-separated, and an empty variable (or `""` in the file) is an empty list; - the Docker image no longer bakes in `config.docker.yml` or passes `--config`, - and its `HEALTHCHECK` probes `PORT` (default `8080`); the config file is - looked for under `/etc/pixa` and `~/.config/pixa` instead of the daemon name - `pixad`; documented in `README.md` and `config.example.yml`. -- 2026-09-28 quality and fit in the URL signature (closes #60): the signed data - is now `host:path:query:width:height:format:expiration:quality:fit`, using - `85` and `cover` when the URL has no `q` or `fit`, so one signed URL can no - longer be replayed across other quality and fit values to create unauthorized - cache entries and transcodes; the known-answer vectors in - `internal/signature/golden_test.go` and the README signature specification - describe the new format. -- 2026-09-28 Docker image healthcheck (closes #111): a `HEALTHCHECK` in the - runtime stage probing `/.well-known/healthcheck.json` with busybox `wget`; - `script/docker-smoke` (`make docker-smoke`) builds the image, starts it with a - throwaway `PIXA_SIGNING_KEY`, and passes only once Docker reports it healthy - within 30 seconds, removing the container on exit; the Gitea workflow runs it - after `script/cibuild`. -- 2026-09-21 trusted-proxy client IP resolution (closes #94): a - `trusted_proxies` config key taking a list of CIDRs, parsed by the same - `net/netip` list parser as `blocked_networks` (an invalid entry aborts startup - naming the key and value; an omitted key defaults to the RFC 1918 private - ranges, an explicitly empty list trusts no one, and an explicit list replaces - the default); a new `internal/clientip` package resolves the client address by - honoring `X-Forwarded-For` only when the direct peer is a trusted proxy, - walking the chain right-to-left to the rightmost non-proxy entry, so a client - connecting directly cannot spoof its address; the resolved address is stored - in the request context by a new middleware and used by the request-logging - middleware and the login-attempt logs in place of the raw peer address; - documented in `README.md` and `config.example.yml`. -- 2026-09-21 blocked networks configuration extending SSRF protection: a - `blocked_networks` config key taking a list of CIDRs (parsed with `net/netip`, - an invalid entry aborts startup naming the key and value), added to the - built-in blocklist rather than replacing it; the built-in ranges extended to - CGNAT `100.64.0.0/10`, IETF protocol assignments `192.0.0.0/24`, benchmark - `198.18.0.0/15`, and NAT64 `64:ff9b::/96` (IPv4-mapped forms covered); - enforcement stays in the dial-time re-resolution so the DNS-rebinding window - remains closed; documented in `README.md` and `config.example.yml`. -- 2026-09-21 validate dimensions and fit mode on the encrypted-URL route and the - token generator (closes #62): `imgcache.ValidateDimension` alone holds the - `MaxDimension` bound and is used by the path parser, by the new - `ValidateImageRequest` (which also applies `ValidateFitMode`) and by the - generator; both the `/v1/image/` and `/v1/e/` routes call - `ValidateImageRequest`, so an over-limit size or an unknown fit mode is a 400 - rather than an out-of-memory or a 500 from the processor; the URL generator - answers 400 naming the field for a `width` or `height` that is not a number or - fails the shared check, a `quality` that is not a number from 1 to 100, a - `ttl` that is not a number from 0 to the largest number of seconds the expiry - calculation can hold, or an unknown `fit`; an empty `quality` is 85 and an - empty `ttl` never expires; the form's width and height inputs stop at 8192 -- 2026-09-21 http.Server hardening (closes #92): added `HTTPReadHeaderTimeout` - (10s, bounds the slowloris header dribble) and `HTTPIdleTimeout` (120s, bounds - keep-alive reuse) alongside the existing timeouts and wired them onto the - server; added a `LimitBody` middleware capping the two form POST bodies - (`POST /`, `POST /generate`) at `MaxFormBytes` (1 MiB) and returning 413, - applied ahead of the CSRF middleware so an oversized body is refused as 413 - rather than being read as a missing CSRF token (403); left `WriteTimeout` at - 60s unchanged -- 2026-08-07 update golangci-lint to v2.12.2 with the canonical `.golangci.yml` - (v2 schema, `default: all` minus six disabled linters, `lll` 88, tests - included): bumped the pinned `golangci/golangci-lint:v2.12.2-alpine` image in - `Dockerfile` and the release-archive sha256 pins in `script/bootstrap`; fixed - the findings the stricter config surfaced (notably `paralleltest`, `wsl_v5`, - `goconst`, `lll`, `noinlineerr`, `err113`, `errcheck`, `testpackage` — - white-box test files renamed to `*_internal_test.go`), including #55's code - absorbed after it merged, iterating the pinned linter to `0 issues.`; no - single finding total is substantiable, since golangci-lint's `uniq-by-line` - reveals new findings on a line as others there are fixed — the documented - re-measurements were 81 after the #53 merge and 149 after the #55 merge; three - behavior changes, so not a pure no-op: `Cache.StoreVariant` now takes a - `context.Context` (`noctx`), so a cancelled request skips its best-effort - accounting row; `MetadataStorage.Store`'s cleanup defer was dead on `main` and - leaked `.tmp-*.json` on failure, now fixed with explicit removals; and the - `signing_key` validation error text gained `value too short: `; the eviction - loop's uncancellable context is deferred to #102 under a - `//nolint:contextcheck`; three `//nolint:tagliatelle` directives keep the - snake_case JSON wire/disk formats unchanged; `make check` green -- 2026-08-07 implement cache size management and eviction (closes #51): new - `cache_max_bytes` config key validated by the startup framework (explicit - values used exactly with no floor, `0` disables the disk cache entirely, - omitted defaults to max(75% of free space on the filesystem containing - `/cache/`, 500 MiB), logged at startup); processed variants are now - tracked in the database (a new `variant_content` table and an LRU timestamp on - `source_content`) so total usage is two SUMs, never a directory scan on the - hot path; a background goroutine evicts globally least-recently-used entries - (variants and source blobs merged) to the limit, woken by a periodic ticker - and by write-pressure notifications from stores; a source blob and ALL of its - `source_metadata` references are deleted in one transaction before the file is - unlinked, so multi-referenced blobs are never removed while referenced and - rows never point at deleted files; a startup and periodic reconciliation pass - adopts untracked variant files, drops rows for missing files, removes - unreachable source blobs, and sweeps stale temp files -- 2026-08-07 validate configuration on startup, fail fast on bad config (closes - #52): a config value that is set but unparseable or invalid aborts startup - naming the key and value (defaults apply only to omitted keys), unknown config - keys abort startup, a malformed config file aborts instead of being skipped, - and `state_dir` is verified creatable and writable before the listener binds -- 2026-08-07 manual test pass of the auth and encrypted URL flows against a - locally built and running `pixad` (built from `main` at `6573b9d`, port 18099, - local throwaway config); all six checks passed, plus all nine tests in - `scripts/manual-test.sh` (closes #49): - - [x] visit `/` and see the login form: HTTP 200, `Pixa - Login` page with - `name="key"` password form - - [x] wrong key shows an error: POST `/` with `key=wrong-key` returned HTTP - 200 login page containing "Invalid signing key" - - [x] correct signing key shows the generator form: POST `/` returned HTTP - 303 to `/` with - `Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict`; GET - `/` with that cookie rendered `Pixa - URL Generator` with the - `/generate` form and logout link - - [x] a generated encrypted URL serves the image: POST `/generate` - (ttl=3600) produced a `/v1/e//img.jpeg` URL that returned HTTP - 200, `Content-Type: image/jpeg`, an 800x600 baseline JPEG of 61706 - bytes - - [x] an expired URL (short TTL) returns 410: a ttl=1 URL fetched after 3 s - returned HTTP 410 Gone with - `{"error":"URL has expired","status":410,...}` - - [x] logout redirects back to login: GET `/logout` returned HTTP 303 to `/` - with `Set-Cookie: pixa_session=; Max-Age=0`; subsequent GET `/` - rendered the login form again -- 2026-08-07 fix the two remaining gosec findings (G124 in internal/session): - session cookies now always carry Secure/HttpOnly/SameSite=Strict on both the - set and clear paths; `make check` green (closes #47) -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile - shims, README Entrypoints section -- 2026-04-07 extract magic byte detection into internal/magic (#42) -- 2026-03-25 extract allowlist package from internal/imgcache (#41) -- 2026-03-25 move schema_migrations table creation into 000.sql (#36) -- 2026-03-20 enforce and document exact-match-only signature verification (#40) -- 2026-03-20 bound imageprocessor.Process input read to prevent unbounded memory - use (#37); consolidate appname into an internal/globals constant (#34) -- 2026-03-18 parse version prefix from migration filenames (#33) -- 2026-03-15 QA audit fixes for 1.0/MVP readiness (#25) -- 2026-03-02 split Dockerfile with pre-built golangci-lint stage for faster CI - (#23) -- 2026-02-25 repo policy compliance: CI workflow, hash-pinned images, - golangci-lint and gosec fixes of that date (#14); arm64 Docker build fix (#16) -- 2026-01-08 WebP and AVIF encoding support via govips (both former P0 image - processing items, now done) - # Future Steps - P2: HTTP response handling - Last-Modified headers -- P2: configuration - - YAML config file support - P2: operational - - optional Sentry error reporting - - comprehensive request logging - - Prometheus performance metrics + - Prometheus metrics still missing: + - cache hits and misses, split into source images and variants + - cache size on disk and its limit + - evictions and the bytes they free + - upstream fetch duration and errors + - transcode duration + - images in processing - measure the 1k to 5k req/s target with `script/loadtest` on a machine not shared with other work