Pass s3.part_size to the multipart uploader (closes #232)
check / check (push) Waiting to run

s3.part_size was loaded and defaulted but never reached the S3 client,
whose uploader used a fixed 10MiB part. The client now takes the part
size from the config, for storage_url and for the s3.* fields, and
config load rejects a value below 5MiB or above 5GiB, the part sizes S3
accepts. The docs gave the default as 5MB, which the config file reads
as 5,000,000 bytes, below the minimum; they now say 5MiB.

Judgement call: the 5GiB maximum is enforced along with the 5MiB
minimum the issue names.
Trap: at the 5MiB default the uploader's 10,000-part limit caps one
upload at about 48.8GiB, down from about 97.7GiB; blob_size_limit is
not checked against it.

Model: opus-5-5
This commit is contained in:
2026-10-07 09:18:31 +00:00
parent 7696f83258
commit 6e3d2e0941
9 changed files with 193 additions and 11 deletions
+7 -5
View File
@@ -27,10 +27,12 @@ type Client struct {
bucket string
prefix string
endpoint string
partSize int64
}
// Config contains S3 client configuration.
// All fields are required except Prefix, which defaults to an empty string.
// All fields are required except Prefix, which defaults to an empty string,
// and PartSize, where zero means the SDK default of 5 MiB.
// A non-empty Prefix is joined to every key with one "/", whether or not
// it ends with one.
// The Endpoint field should include the protocol (http:// or https://).
@@ -41,6 +43,8 @@ type Config struct {
AccessKeyID string
SecretAccessKey string
Region string
// PartSize is the size in bytes of each part of a multipart upload.
PartSize int64
}
// nopLogger is a logger that discards all output.
@@ -90,6 +94,7 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
bucket: cfg.Bucket,
prefix: prefix,
endpoint: cfg.Endpoint,
partSize: cfg.PartSize,
}, nil
}
@@ -123,12 +128,9 @@ func (c *Client) PutObjectWithProgress(
) error {
fullKey := c.prefix + key
// uploadPartSize is 10MB for better progress granularity.
const uploadPartSize = 10 * 1024 * 1024
// Create an uploader with the S3 client
uploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) {
u.PartSize = uploadPartSize
u.PartSize = c.partSize
})
// Create a progress reader that tracks upload progress
+1
View File
@@ -28,6 +28,7 @@ func provideClient(lc fx.Lifecycle, cfg *config.Config) (*Client, error) {
AccessKeyID: cfg.S3.AccessKeyID,
SecretAccessKey: cfg.S3.SecretAccessKey,
Region: cfg.S3.Region,
PartSize: cfg.S3.PartSize.Int64(),
})
if err != nil {
return nil, err