blocker: panic when uploading tcp connection is lost #476

Closed
opened 2026-11-19 09:56:09 +01:00 by sneak · 3 comments
Owner
》 Blob upload 4ad73bc51895...: 42 MB / 10 GB (0.4%) at 2 Mbit/sec, blob upload elapsed: 2m19s, blob upload ETA: 10:25:31 (est remain 9h38m43s).
00:48:21 ERROR Failed to upload blob hash=4ad73bc51895b38b0299c885518ee59cc41f82516191cd31593c43beb2d4eebd error=uploading object: Update ReadFrom failed: connection lost caller=scanner.go:1195
panic: close of closed channel

goroutine 67 [running]:
sneak.berlin/go/vaultik/internal/snapshot.(*ProgressReporter).Stop(0x74049e71e730)
        /Users/user/dev/vaultik/internal/snapshot/progress.go:115 +0x70
sneak.berlin/go/vaultik/internal/snapshot.(*Scanner).Scan(0x74049de12280, {0x10a09db48, 0x74049e8123c0}, {0x74049de94680, 0x38}, {0x74049e6c42c0?, 0x0?})
        /Users/user/dev/vaultik/internal/snapshot/scanner.go:230 +0x788
sneak.berlin/go/vaultik/internal/vaultik.(*Vaultik).scanAllDirectories(0x74049e030000, 0x74049de12280, {0x74049e5b0480, 0x3, 0x3?}, {0x74049e6c42c0, 0x3a})
        /Users/user/dev/vaultik/internal/vaultik/snapshot.go:247 +0x438
sneak.berlin/go/vaultik/internal/vaultik.(*Vaultik).createNamedSnapshot(0x74049e030000, 0x74049e71c340, {0x74049eb15710, 0xf}, {0x74049e8dac80, 0x1b}, 0x1, 0x1)
        /Users/user/dev/vaultik/internal/vaultik/snapshot.go:184 +0x524
sneak.berlin/go/vaultik/internal/vaultik.(*Vaultik).CreateSnapshot(0x74049e030000, 0x74049e71c340)
        /Users/user/dev/vaultik/internal/vaultik/snapshot.go:80 +0x298
sneak.berlin/go/vaultik/internal/cli.newSnapshotCreateCommand.func1.1.1.1()
        /Users/user/dev/vaultik/internal/cli/snapshot.go:77 +0x24
created by sneak.berlin/go/vaultik/internal/cli.newSnapshotCreateCommand.func1.1.1 in goroutine 66
        /Users/user/dev/vaultik/internal/cli/snapshot.go:75 +0x6c
user@heraklion:~$

there needs to be graceful handling of blob upload failures, and retries. retry by default 5 times with 3 second delays.

``` 》 Blob upload 4ad73bc51895...: 42 MB / 10 GB (0.4%) at 2 Mbit/sec, blob upload elapsed: 2m19s, blob upload ETA: 10:25:31 (est remain 9h38m43s). 00:48:21 ERROR Failed to upload blob hash=4ad73bc51895b38b0299c885518ee59cc41f82516191cd31593c43beb2d4eebd error=uploading object: Update ReadFrom failed: connection lost caller=scanner.go:1195 panic: close of closed channel goroutine 67 [running]: sneak.berlin/go/vaultik/internal/snapshot.(*ProgressReporter).Stop(0x74049e71e730) /Users/user/dev/vaultik/internal/snapshot/progress.go:115 +0x70 sneak.berlin/go/vaultik/internal/snapshot.(*Scanner).Scan(0x74049de12280, {0x10a09db48, 0x74049e8123c0}, {0x74049de94680, 0x38}, {0x74049e6c42c0?, 0x0?}) /Users/user/dev/vaultik/internal/snapshot/scanner.go:230 +0x788 sneak.berlin/go/vaultik/internal/vaultik.(*Vaultik).scanAllDirectories(0x74049e030000, 0x74049de12280, {0x74049e5b0480, 0x3, 0x3?}, {0x74049e6c42c0, 0x3a}) /Users/user/dev/vaultik/internal/vaultik/snapshot.go:247 +0x438 sneak.berlin/go/vaultik/internal/vaultik.(*Vaultik).createNamedSnapshot(0x74049e030000, 0x74049e71c340, {0x74049eb15710, 0xf}, {0x74049e8dac80, 0x1b}, 0x1, 0x1) /Users/user/dev/vaultik/internal/vaultik/snapshot.go:184 +0x524 sneak.berlin/go/vaultik/internal/vaultik.(*Vaultik).CreateSnapshot(0x74049e030000, 0x74049e71c340) /Users/user/dev/vaultik/internal/vaultik/snapshot.go:80 +0x298 sneak.berlin/go/vaultik/internal/cli.newSnapshotCreateCommand.func1.1.1.1() /Users/user/dev/vaultik/internal/cli/snapshot.go:77 +0x24 created by sneak.berlin/go/vaultik/internal/cli.newSnapshotCreateCommand.func1.1.1 in goroutine 66 /Users/user/dev/vaultik/internal/cli/snapshot.go:75 +0x6c user@heraklion:~$ ``` there needs to be graceful handling of blob upload failures, and retries. retry by default 5 times with 3 second delays.
clawbot added the critical label 2026-11-19 10:09:10 +01:00
Collaborator

PR: #479 (branch issue-476-retry-blob-upload, base next).

A failed blob upload now gets up to five retries, 3 seconds apart, each announced with a warning. If the last try fails too, the backup stops with the upload's error. The panic in the report was already fixed on next by #253; a new test covers a failing upload with the progress reporter on.

  • Judgement call: "retry 5 times" is read as five retries after the first try. The count and the delay are constants, not config keys.
  • Deviation: next had been deleted from the remote after the merge into main. I recreated it at main's tip (4605eae), which has the same tree as the old next.

Model: opus-5-5

PR: https://git.eeqj.de/sneak/vaultik/pulls/479 (branch `issue-476-retry-blob-upload`, base `next`). A failed blob upload now gets up to five retries, 3 seconds apart, each announced with a warning. If the last try fails too, the backup stops with the upload's error. The panic in the report was already fixed on `next` by https://git.eeqj.de/sneak/vaultik/issues/253; a new test covers a failing upload with the progress reporter on. - Judgement call: "retry 5 times" is read as five retries after the first try. The count and the delay are constants, not config keys. - Deviation: `next` had been deleted from the remote after the merge into `main`. I recreated it at `main`'s tip (`4605eae`), which has the same tree as the old `next`. Model: opus-5-5
Author
Owner

this might be old i was testing on a very old version, verify before working it

this might be old i was testing on a very old version, verify before working it
sneak closed this issue 2026-11-19 10:31:59 +01:00
Collaborator

Verified: you were right, the panic itself is already fixed on next (#253). PR 479 adds only the upload retries this issue asks for, plus a test of a failing upload with progress shown, so the old panic stays fixed.

Model: opus-5-5

Verified: you were right, the panic itself is already fixed on `next` (https://git.eeqj.de/sneak/vaultik/issues/253). PR 479 adds only the upload retries this issue asks for, plus a test of a failing upload with progress shown, so the old panic stays fixed. Model: opus-5-5
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/vaultik#476