s3.part_size was loaded and defaulted to 5MiB, but the S3 client never received it: PutObjectWithProgress built its uploader with a fixed 10MiB part. The S3 client's Config now has a PartSize field, and s3.part_size is passed through both for storage_url with s3:// and for the s3.* fields. Validate rejects a part size below 5MiB or above 5GiB. The default is now set before the file is parsed, as for chunk_size, so an explicit part_size: 0 fails at load.
What the diff does not show:
S3 takes at most 10,000 parts per upload, and the uploader cannot seek the reader it is given to learn its size and enlarge the parts itself. PutObjectWithProgress therefore raises the part size for one upload to the smallest that fits; at the 5MiB default that applies above about 48.8GiB.
5MB in the config file is 5,000,000 bytes, below S3's minimum, so the documented part_size: 5MB would now fail at load. The docs and test/config.yaml now say 5MiB.
The storage test counts the part requests an in-process S3 server receives for an 18MiB upload at a 6MiB part size: three, where the SDK default sends four.
The two existing Validate tests build a Config by hand, skipping the defaults Load sets, so they now set the part size.
Disclosures:
Judgement call: the 5GiB maximum is enforced along with the 5MiB minimum the issue names.
Unverified end to end: the raised part size is tested as a calculation; an upload over 48.8GiB is too large for a test.
Model: opus-5-5
Fixes https://git.eeqj.de/sneak/vaultik/issues/232.
`s3.part_size` was loaded and defaulted to 5MiB, but the S3 client never received it: `PutObjectWithProgress` built its uploader with a fixed 10MiB part. The S3 client's `Config` now has a `PartSize` field, and `s3.part_size` is passed through both for `storage_url` with `s3://` and for the `s3.*` fields. `Validate` rejects a part size below 5MiB or above 5GiB. The default is now set before the file is parsed, as for `chunk_size`, so an explicit `part_size: 0` fails at load.
What the diff does not show:
- S3 takes at most 10,000 parts per upload, and the uploader cannot seek the reader it is given to learn its size and enlarge the parts itself. `PutObjectWithProgress` therefore raises the part size for one upload to the smallest that fits; at the 5MiB default that applies above about 48.8GiB.
- `5MB` in the config file is 5,000,000 bytes, below S3's minimum, so the documented `part_size: 5MB` would now fail at load. The docs and `test/config.yaml` now say `5MiB`.
- The storage test counts the part requests an in-process S3 server receives for an 18MiB upload at a 6MiB part size: three, where the SDK default sends four.
- The two existing `Validate` tests build a `Config` by hand, skipping the defaults `Load` sets, so they now set the part size.
Disclosures:
- Judgement call: the 5GiB maximum is enforced along with the 5MiB minimum the issue names.
- Unverified end to end: the raised part size is tested as a calculation; an upload over 48.8GiB is too large for a test.
Model: opus-5-5
internal/s3/client.go:133: with the 5MiB default (internal/config/config.go:36), the largest blob S3 storage can take drops from about 97.7GiB to about 48.8GiB. The uploader is given a reader it cannot seek, so it cannot enlarge parts to fit and stops at S3's 10,000-part limit. A blob_size_limit between the two that worked before still loads, then fails the backup after about 48.8GiB of the blob has been sent. Acceptable: no blob within blob_size_limit can need more than 10,000 parts, for example PutObjectWithProgress raising the part size for one upload to fit the exact size it is already given, or config load rejecting, for S3 storage, a blob_size_limit larger than s3.part_size times 10,000; with a test.
internal/config/config.go:301: an explicit part_size: 0 is replaced by the default after the file is parsed, so a part size S3 refuses loads silently as 5MiB, against item 2 of the definition of done in #232. chunk_size, blob_size_limit and compression_level get their defaults before parsing, so an explicit 0 for them fails at load. Acceptable: s3.part_size gets its default the same way, with a test that loads a file setting part_size: 0 and gets the error.
test/config.yaml:22 still labels part_size: 5242880 as # 5MB, the value this change documents as below the minimum. Acceptable: # 5MiB.
Model: opus-5-5
1. `internal/s3/client.go:133`: with the 5MiB default (`internal/config/config.go:36`), the largest blob S3 storage can take drops from about 97.7GiB to about 48.8GiB. The uploader is given a reader it cannot seek, so it cannot enlarge parts to fit and stops at S3's 10,000-part limit. A `blob_size_limit` between the two that worked before still loads, then fails the backup after about 48.8GiB of the blob has been sent. Acceptable: no blob within `blob_size_limit` can need more than 10,000 parts, for example `PutObjectWithProgress` raising the part size for one upload to fit the exact size it is already given, or config load rejecting, for S3 storage, a `blob_size_limit` larger than `s3.part_size` times 10,000; with a test.
2. `internal/config/config.go:301`: an explicit `part_size: 0` is replaced by the default after the file is parsed, so a part size S3 refuses loads silently as 5MiB, against item 2 of the definition of done in https://git.eeqj.de/sneak/vaultik/issues/232. `chunk_size`, `blob_size_limit` and `compression_level` get their defaults before parsing, so an explicit 0 for them fails at load. Acceptable: `s3.part_size` gets its default the same way, with a test that loads a file setting `part_size: 0` and gets the error.
3. `test/config.yaml:22` still labels `part_size: 5242880` as `# 5MB`, the value this change documents as below the minimum. Acceptable: `# 5MiB`.
Model: opus-5-5
s3.part_size was loaded and defaulted but never reached the S3 client,
whose uploader used a fixed 10MiB part. The client now takes the part
size from the config, for storage_url and for the s3.* fields, and
config load rejects a value below 5MiB or above 5GiB, an explicit 0
included. A blob too large for S3's limit of 10,000 parts at that size
is uploaded in larger parts, since the uploader cannot learn the size of
the reader it is given. The docs gave the default as 5MB, which the
config file reads as 5,000,000 bytes, below the minimum; they now say
5MiB.
Judgement call: the 5GiB maximum is enforced along with the 5MiB
minimum the issue names.
Model: opus-5-5
PutObjectWithProgress now raises the part size for one upload to the smallest that fits the size it is given into 10,000 parts (uploadPartSize in internal/s3/client.go, tested in internal/s3/client_internal_test.go); the trap disclosure is dropped from the PR body and commit.
s3.part_size now gets its default before the file is parsed, as chunk_size does; TestLoadS3PartSize loads a file setting part_size: 0 and gets the error, and one without it and gets 5MiB.
test/config.yaml now labels the value # 5MiB.
Model: opus-5-5
Rework delta:
1. `PutObjectWithProgress` now raises the part size for one upload to the smallest that fits the size it is given into 10,000 parts (`uploadPartSize` in `internal/s3/client.go`, tested in `internal/s3/client_internal_test.go`); the trap disclosure is dropped from the PR body and commit.
2. `s3.part_size` now gets its default before the file is parsed, as `chunk_size` does; `TestLoadS3PartSize` loads a file setting `part_size: 0` and gets the error, and one without it and gets 5MiB.
3. `test/config.yaml` now labels the value `# 5MiB`.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #232.
s3.part_sizewas loaded and defaulted to 5MiB, but the S3 client never received it:PutObjectWithProgressbuilt its uploader with a fixed 10MiB part. The S3 client'sConfignow has aPartSizefield, ands3.part_sizeis passed through both forstorage_urlwiths3://and for thes3.*fields.Validaterejects a part size below 5MiB or above 5GiB. The default is now set before the file is parsed, as forchunk_size, so an explicitpart_size: 0fails at load.What the diff does not show:
PutObjectWithProgresstherefore raises the part size for one upload to the smallest that fits; at the 5MiB default that applies above about 48.8GiB.5MBin the config file is 5,000,000 bytes, below S3's minimum, so the documentedpart_size: 5MBwould now fail at load. The docs andtest/config.yamlnow say5MiB.Validatetests build aConfigby hand, skipping the defaultsLoadsets, so they now set the part size.Disclosures:
Model: opus-5-5
internal/s3/client.go:133: with the 5MiB default (internal/config/config.go:36), the largest blob S3 storage can take drops from about 97.7GiB to about 48.8GiB. The uploader is given a reader it cannot seek, so it cannot enlarge parts to fit and stops at S3's 10,000-part limit. Ablob_size_limitbetween the two that worked before still loads, then fails the backup after about 48.8GiB of the blob has been sent. Acceptable: no blob withinblob_size_limitcan need more than 10,000 parts, for examplePutObjectWithProgressraising the part size for one upload to fit the exact size it is already given, or config load rejecting, for S3 storage, ablob_size_limitlarger thans3.part_sizetimes 10,000; with a test.internal/config/config.go:301: an explicitpart_size: 0is replaced by the default after the file is parsed, so a part size S3 refuses loads silently as 5MiB, against item 2 of the definition of done in #232.chunk_size,blob_size_limitandcompression_levelget their defaults before parsing, so an explicit 0 for them fails at load. Acceptable:s3.part_sizegets its default the same way, with a test that loads a file settingpart_size: 0and gets the error.test/config.yaml:22still labelspart_size: 5242880as# 5MB, the value this change documents as below the minimum. Acceptable:# 5MiB.Model: opus-5-5
98b981203eto28299a3f4fRework delta:
PutObjectWithProgressnow raises the part size for one upload to the smallest that fits the size it is given into 10,000 parts (uploadPartSizeininternal/s3/client.go, tested ininternal/s3/client_internal_test.go); the trap disclosure is dropped from the PR body and commit.s3.part_sizenow gets its default before the file is parsed, aschunk_sizedoes;TestLoadS3PartSizeloads a file settingpart_size: 0and gets the error, and one without it and gets 5MiB.test/config.yamlnow labels the value# 5MiB.Model: opus-5-5
Review passed.
Model: opus-5-5