sneak/pixa

Bound concurrent image processing and upstream fetches (closes #64) #148

Open

clawbot wants to merge 6 commits from issue-64-concurrency-limits into next

Author	SHA1	Message	Date
clawbot	f0b67596c7	Say downstream_timeout covers the connection and processing waits (closes #64 ) check / check (push) Successful in 3m22s Details The waits of up to 10 seconds for an upstream connection and for a processing slot run inside downstream_timeout, and a request whose downstream_timeout ends during one gets 500, not 503. The docs now say so and advise keeping downstream_timeout longer than upstream_fetch_timeout plus 20 seconds. Model: opus-5-5	2026-09-29 08:03:29 +00:00
clawbot	add0e2ba6a	Keep newFromSmartConfig within the length limit (closes #64 ) check / check (push) Successful in 3m41s Details Rebasing onto the four settings from #142 put newFromSmartConfig at 82 lines, over the linter's 80-line limit. validateKnownKeys now returns early for a nil config (no config file) itself, as lookupValue already does, so the caller drops its own nil check. Behavior is unchanged. Model: opus-5-5	2026-09-29 07:32:56 +00:00
clawbot	d800aa62a7	Read a cached source only once a processing slot is taken (closes #64 ) A request whose source was in the disk cache read the whole file into memory, then waited for a processing slot, so a burst of new sizes for one large cached image held one copy per waiting request, with no ceiling. The service now opens the cached file and hands it to the image processor, which reads it only after taking its slot. The file's size, now returned by GetSourceContent, still sends an empty or oversized cached source to upstream instead. A cached file that fails while being read now fails the request instead of being fetched again. Model: opus-5-5	2026-09-29 07:32:56 +00:00
clawbot	9e1f5d4cea	Test that a cached source is read only with a processing slot (closes #64 ) Failing test: with the only processing slot held, a request for a new width of a cached image waits for the slot while the cached file is rewritten; the answer must come from the rewritten file, so the request read none of the source before it had a slot. Also a test that a fetch whose request context ends while it waits for a connection shared by all hosts gives its host's slot back; that one passes already. Model: opus-5-5	2026-09-29 07:32:45 +00:00
clawbot	fffd97d3d8	Bound concurrent image processing and upstream fetches (closes #64 ) max_concurrent_processing (default: the number of CPUs Go uses) bounds the images processed at once, and upstream_connections (default 64) the fetches from all upstream hosts together, beside the per-host limit. A request that finds either full waits up to 10 seconds, then gets 503 "server busy, try again later". The processor holds its slot from before it reads the input until it returns, and takes a free slot even after the request context has ended; a fetch holds its connection until the response body is closed, after its image is processed. libvips now starts with one worker thread per image and no operation cache. Both settings have PIXA_ variables and are in README.md and config.example.yml. Model: opus-5-5	2026-09-29 07:32:45 +00:00
clawbot	6cb280919f	Test the processing and upstream connection limits (closes #64 ) Failing tests for two limits that do not exist yet. Config: max_concurrent_processing and upstream_connections, their defaults, and valid and invalid values from the file and the environment. Image processor: never more images at once than its limit, waiting and then failing with ErrTooManyImages when no slot frees, and freeing its slot on every error. Fetcher: connections to all hosts counted together, apart from the per-host limit, and freed on errors. Both image routes answer 503 when either wait gives up. TestEnvironmentSetsEveryKey sets the two new variables, as it compares the whole config. The tests do not compile until the limits exist. Model: opus-5-5	2026-09-29 07:32:32 +00:00