Write rclone uploads under a temporary name, re-upload short blobs (closes #266)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The rclone backend wrote each object straight to its key, so killing an upload to a local or sftp remote left a truncated object there. The next backup found the key, skipped the upload and recorded a snapshot that could not be restored. On a remote where rclone says a file can be seen while it is still being written, an object is now written under a name ending in `.partial` and moved onto its key with the remote's server-side move, as rclone's own copy does; listings skip such names. A backup also uploads a blob again when the stored object's size differs from the blob's. Judgement call: the temporary name is used only where rclone sets its PartialUploads feature; other remotes already show an object only once complete. Model: opus-5-5
This commit is contained in:
@@ -14,6 +14,9 @@ import (
|
||||
// errStreamInterrupted stands in for an upload cut off mid-stream.
|
||||
var errStreamInterrupted = errors.New("connection reset mid-upload")
|
||||
|
||||
// testBlobKey is a key laid out as a blob's key is.
|
||||
const testBlobKey = "blobs/aa/bb/aabbccddeeff"
|
||||
|
||||
// failingReader yields its data once, then fails.
|
||||
type failingReader struct {
|
||||
data []byte
|
||||
@@ -43,7 +46,7 @@ func TestFileStorer_InterruptedWriteLeavesNoTrustedObject(t *testing.T) {
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
key := "blobs/aa/bb/aabbccddeeff"
|
||||
key := testBlobKey
|
||||
|
||||
err = f.PutWithProgress(ctx, key, &failingReader{data: []byte("partial")}, 4096, nil)
|
||||
if err == nil {
|
||||
@@ -79,7 +82,7 @@ func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
realKey := "blobs/aa/bb/aabbccddeeff"
|
||||
realKey := testBlobKey
|
||||
|
||||
err = f.Put(ctx, realKey, strings.NewReader("blob-bytes"))
|
||||
if err != nil {
|
||||
|
||||
Reference in New Issue
Block a user