Write rclone uploads under a temporary name and move them into place (closes #266)
check / check (push) Waiting to run

The rclone backend wrote each object straight to its key, so killing an
upload to a local or sftp remote left a truncated object there that the
next backup trusted.

On every remote with a server-side move, an object is now written under
a name ending in `.partial` and moved onto its key with rclone's
operations.Move, which removes an object already at the key first;
drive, dropbox and others will not move onto one. Listings skip
`.partial` names. Rclone's own copy also requires the PartialUploads
flag; this does not, because hdfs shows a file while it is written
without setting it. Remotes without a move are written in place.

Model: opus-5-5
This commit is contained in:
2026-10-07 19:59:56 +00:00
parent b5389a62b5
commit 8cc13cf532
7 changed files with 404 additions and 36 deletions
+5 -2
View File
@@ -14,6 +14,9 @@ import (
// errStreamInterrupted stands in for an upload cut off mid-stream.
var errStreamInterrupted = errors.New("connection reset mid-upload")
// testBlobKey is a key laid out as a blob's key is.
const testBlobKey = "blobs/aa/bb/aabbccddeeff"
// failingReader yields its data once, then fails.
type failingReader struct {
data []byte
@@ -43,7 +46,7 @@ func TestFileStorer_InterruptedWriteLeavesNoTrustedObject(t *testing.T) {
}
ctx := context.Background()
key := "blobs/aa/bb/aabbccddeeff"
key := testBlobKey
err = f.PutWithProgress(ctx, key, &failingReader{data: []byte("partial")}, 4096, nil)
if err == nil {
@@ -79,7 +82,7 @@ func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
}
ctx := context.Background()
realKey := "blobs/aa/bb/aabbccddeeff"
realKey := testBlobKey
err = f.Put(ctx, realKey, strings.NewReader("blob-bytes"))
if err != nil {