An S3 listing the server marks truncated without a continuation token is taken as complete, so prune can delete blobs still in use #297

Open
opened 2026-10-13 00:47:08 +02:00 by clawbot · 1 comment
Collaborator

Found by the defect hunt in #288 (part 5).

What is wrong: Client.ListObjects and ListObjectsStream (internal/s3/client.go) use the SDK paginator, which ends silently when a page has IsTruncated=true but no NextContinuationToken; a server that answers V2 listings V1-style does that. prune then counts only the manifests on the first page and treats the blobs of the remaining snapshots as unreferenced (collectReferencedBlobs, internal/vaultik/prune.go).

How it was shown: a throwaway test in internal/storage put gofakes3 behind a handler that pages by 2 and removes NextContinuationToken, then stored 5 keys: List returned 2 of 5 keys and no error, and the same for ListStream.

Definition of done

  • A page marked truncated without a continuation token is an error from List and ListStream, so prune, snapshot list and remote nuke take their existing path for a destination that cannot be listed, and prune deletes nothing.
  • A test covers the cut-short listing for both methods.

Model: opus-5-5

Found by the defect hunt in https://git.eeqj.de/sneak/vaultik/issues/288 (part 5). **What is wrong:** `Client.ListObjects` and `ListObjectsStream` (`internal/s3/client.go`) use the SDK paginator, which ends silently when a page has `IsTruncated=true` but no `NextContinuationToken`; a server that answers V2 listings V1-style does that. `prune` then counts only the manifests on the first page and treats the blobs of the remaining snapshots as unreferenced (`collectReferencedBlobs`, `internal/vaultik/prune.go`). **How it was shown:** a throwaway test in `internal/storage` put gofakes3 behind a handler that pages by 2 and removes `NextContinuationToken`, then stored 5 keys: `List` returned 2 of 5 keys and no error, and the same for `ListStream`. **Definition of done** - A page marked truncated without a continuation token is an error from `List` and `ListStream`, so `prune`, `snapshot list` and `remote nuke` take their existing path for a destination that cannot be listed, and `prune` deletes nothing. - A test covers the cut-short listing for both methods. Model: opus-5-5
clawbot added the critical label 2026-10-13 00:47:08 +02:00
clawbot self-assigned this 2026-10-13 00:47:08 +02:00
Author
Collaborator

Implemented in #333: ListObjects and ListObjectsStream now return ErrNoContinuationToken for a page marked truncated with no continuation token, so List and ListStream fail on it and their callers take their existing path for a destination that cannot be listed. A new test in internal/storage/s3_test.go covers both methods.

Model: opus-5-5

Implemented in https://git.eeqj.de/sneak/vaultik/pulls/333: `ListObjects` and `ListObjectsStream` now return `ErrNoContinuationToken` for a page marked truncated with no continuation token, so `List` and `ListStream` fail on it and their callers take their existing path for a destination that cannot be listed. A new test in `internal/storage/s3_test.go` covers both methods. Model: opus-5-5
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/vaultik#297