Write rclone uploads under a temporary name, re-upload short blobs (closes #266)
check / check (push) Waiting to run

The rclone backend wrote each object straight to its key, so killing an
upload to a local or sftp remote left a truncated object there. The next
backup found the key, skipped the upload and recorded a snapshot that
could not be restored.

On a remote where rclone says a file can be seen while it is still being
written, an object is now written under a name ending in `.partial` and
moved onto its key with the remote's server-side move, as rclone's own
copy does; listings skip such names. A backup also uploads a blob again
when the stored object's size differs from the blob's.

Judgement call: the temporary name is used only where rclone sets its
PartialUploads feature; other remotes already show an object only once
complete.

Model: opus-5-5
This commit is contained in:
2026-10-07 16:22:07 +00:00
parent d87202fb70
commit 42f6ae67a5
9 changed files with 292 additions and 41 deletions
+3 -3
View File
@@ -50,9 +50,9 @@ const storageDirPerm = 0o755
// temp file carrying this suffix and only renames it onto the real key once
// the whole object is on disk, so an interrupted write can never leave a
// truncated object at the key a later run would Stat and trust as a complete
// blob. List and ListStream skip these files, so a leftover from an
// interrupted write is never listed or trusted as a blob; it is otherwise
// harmless and is overwritten when the same key is written again.
// blob. The rclone backend's upload does the same on remotes that need it.
// List and ListStream skip these files, so a leftover from an interrupted
// write is never listed or trusted as a blob; it is otherwise harmless.
const tempSuffix = ".partial"
// Put stores data at the specified key.