Once a worker has thrown, as one does when onProgress throws, or the
lock is lost, no worker takes another file, and the backup fails once
the originals under way are in place. Before, the other workers went on
through the rest of the account after a throw. Once the workers have
stopped, the backup checks its lock, so a lost lock no longer lets it
start the ML data fetch. The end-to-end test's comment now says the
backup has started all three originals when the lock is lost, not that
all three are still downloading.
Model: opus-5-5
The onProgress test now runs eight photos with a pool of five and holds
only the first five downloads, so a backup that went on after the throw
would download and store the other three. The end-to-end test of a
removed backup.lock expects the backup to stop after its three originals,
before it fetches ML data.
Model: opus-5-5
lib.backup() ran its originals one at a time, so the content pool's five
slots, or the contentConcurrency of the pools a library was opened with,
went unused. The work for one original, its verify hashing included, is
now storeOriginal, and that many workers take files from one iterator
over the backup's files, so no more originals than the limit are open at
once. BackupLibrary gains contentConcurrency, which the library sets from
its content pool. The README's "Request pools" says so. The tests of a
backup whose lock is lost now expect all three originals downloaded,
since all three are under way at once.
Model: opus-5-5
proper-lockfile's default onCompromised throws from a timer, so a backup
whose backup.lock was removed or taken over by another backup died with
a stack trace. lockBackupDirectory now returns the lock, its release
function and why it was lost, and passes a handler that only records
the reason. The backup checks it after the refresh and before each
original, thumbnail, file's JSON and album, and before it writes
failures.json, and then fails with "lost the lock on DIR: REASON",
which quak backup prints as one line, exiting 1. lib.backup() takes a
caller's lock as lock, in place of lockHeld: true. Releasing a lost lock
leaves backup.lock alone, since it may be another backup's.
Model: opus-5-5
test/cli/backup-end-to-end.test.ts runs backupCommand through run, as
bin/quak.ts does, with the real loadSession reading a session saved after
logging in, and every request answered by a fake of the Ente HTTP API serving
data encrypted with libsodium. It covers the backup tree of an account with
two albums of its own and one shared with it, a second run that downloads
nothing, a download that succeeds after two 503s, a 404 recorded in
failures.json with exit 1, a refresh answered 401 with exit 3, and --verify
putting back an original changed on disk.
The fake server and the account it logs in moved from the backup-metadata
test into test/ente-server.ts, unchanged, so both tests use them.
Model: opus-5-5
sneak
clawbot