With an s3://bucket/prefix URL, objects are stored as prefixblobs/... instead of prefix/blobs/... #222

Closed
opened 2026-10-06 01:49:44 +02:00 by clawbot · 1 comment
Collaborator

The URL parser strips only the leading slash from the path (internal/storage/url.go:106). The S3 client then concatenates prefix + key at every call site (internal/s3/client.go:91, :114, :147 and the rest). Measured on next at 0700901: s3://test-bucket/myprefix?endpoint=... stored an object as myprefixblobs/aa/bb/....

The README gives the URL shape s3://bucket/prefix?... (:411) and documents the layout as BUCKET/PREFIX/blobs/... and BUCKET/PREFIX/metadata/... (:435-443). As things stand, .../prefix and .../prefix/ are two different destinations, so a recovery host that guesses the trailing slash wrong sees no snapshots.

Acceptable: the README layout. A non-empty prefix is joined to every key with exactly one /, whether or not the URL ended with one. Nothing is installed anywhere before 1.0, so no existing store needs migrating.

Definition of done

  1. s3://b/p and s3://b/p/ both store an object as p/blobs/.... An empty prefix stores blobs/....
  2. One test pins the stored key for each of the three URL shapes, and covers listing as well as writing.
  3. make check passes.

Model: fable-5-1 (audit); opus-5-5 (issue)

The URL parser strips only the leading slash from the path (`internal/storage/url.go:106`). The S3 client then concatenates `prefix + key` at every call site (`internal/s3/client.go:91`, `:114`, `:147` and the rest). Measured on `next` at `0700901`: `s3://test-bucket/myprefix?endpoint=...` stored an object as `myprefixblobs/aa/bb/...`. The README gives the URL shape `s3://bucket/prefix?...` (`:411`) and documents the layout as `BUCKET/PREFIX/blobs/...` and `BUCKET/PREFIX/metadata/...` (`:435-443`). As things stand, `.../prefix` and `.../prefix/` are two different destinations, so a recovery host that guesses the trailing slash wrong sees no snapshots. Acceptable: the README layout. A non-empty prefix is joined to every key with exactly one `/`, whether or not the URL ended with one. Nothing is installed anywhere before 1.0, so no existing store needs migrating. ## Definition of done 1. `s3://b/p` and `s3://b/p/` both store an object as `p/blobs/...`. An empty prefix stores `blobs/...`. 2. One test pins the stored key for each of the three URL shapes, and covers listing as well as writing. 3. `make check` passes. Model: fable-5-1 (audit); opus-5-5 (issue)
clawbot self-assigned this 2026-10-06 01:49:44 +02:00
Author
Collaborator

Fixed in #248. The S3 client now joins a non-empty prefix to every key with one /, so s3://b/p and s3://b/p/ both store p/blobs/... and an empty prefix stores blobs/.... The s3.prefix config setting goes through the same client and gets the same join.

Model: opus-5-5

Fixed in https://git.eeqj.de/sneak/vaultik/pulls/248. The S3 client now joins a non-empty prefix to every key with one `/`, so `s3://b/p` and `s3://b/p/` both store `p/blobs/...` and an empty prefix stores `blobs/...`. The `s3.prefix` config setting goes through the same client and gets the same join. Model: opus-5-5
Sign in to join this conversation.