Compare commits
7
Commits
0f7bcb3553
...
next
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
8583863ec0 | ||
|
clawbot
|
960a7ce8a6 | ||
|
clawbot
|
20410f8f71 | ||
|
clawbot
|
d016dbfaa7 | ||
|
clawbot
|
5a940d4e4e | ||
|
clawbot
|
3412e42af5 | ||
|
clawbot
|
22ee2db479 |
@@ -19,6 +19,10 @@ encrypting and uploading them back to the server.
|
||||
|
||||
## Getting Started
|
||||
|
||||
The commands below need git, node 22 or later, and yarn. `make install`, which
|
||||
builds the single binary that "Running the backup from cron" uses, also needs
|
||||
[nix](https://nixos.org/download).
|
||||
|
||||
```bash
|
||||
git clone https://git.eeqj.de/sneak/quak.git
|
||||
cd quak
|
||||
@@ -86,9 +90,12 @@ enumeration/download calls) is exported too and documented under Design below.
|
||||
## Running the backup from cron
|
||||
|
||||
`quak backup` never prompts, so cron can run it. `make install` builds quak as a
|
||||
single binary and copies it to `~/bin/quak`. Log in once with it, as the user
|
||||
the cron job will run as; the session is saved in that user's data directory
|
||||
(see "Session handling"):
|
||||
single binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
|
||||
it needs [nix](https://nixos.org/download), even where bun is already installed:
|
||||
`make install` runs `make build-bin`, which bundles the built `dist/` with bun
|
||||
run through `nix-shell -p bun`. Log in once with `~/bin/quak`, as the user the
|
||||
cron job will run as; the session is saved in that user's data directory (see
|
||||
"Session handling"):
|
||||
|
||||
```bash
|
||||
~/bin/quak login
|
||||
@@ -127,21 +134,36 @@ matters.
|
||||
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `0` | The backup is complete: every original is at its save path, and no file failed. |
|
||||
| `1` | The run finished with files in `failures.json` or an album it could not decrypt, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||
| Code | Meaning |
|
||||
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `0` | The backup is complete: every original is at its save path, and no file failed. |
|
||||
| `1` | The run finished with files in `failures.json` or albums it could not decrypt or write (see "Backup layout"), or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||
|
||||
`<message>` is the error's message, followed by the message of each error that
|
||||
caused it. Under Node, as `yarn quak` runs quak, `fetch` reports every failure
|
||||
below HTTP as just `fetch failed` and gives the reason as its cause, so a run
|
||||
that could not reach the server prints, for example,
|
||||
`quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443`. When the server has
|
||||
several addresses and none of them could be reached, the line gives the reason
|
||||
for each, separated by `, `. A file whose download failed that way is in
|
||||
`failures.json` with that error, as a transient failure. So is each file with no
|
||||
ML data cached when the ML data fetch failed that way, its error starting
|
||||
`ML data: `. The `~/bin/quak` that `make install` builds runs under bun, and
|
||||
prints bun's own wording for such a failure, with no reason after it, such as
|
||||
`quak: Unable to connect. Is the computer able to access the url?` for a refused
|
||||
connection. It records such a failure in `failures.json` as transient too.
|
||||
|
||||
After a `1`, the next run fetches each missing original again, fetches the ML
|
||||
data that is not cached, and rebuilds the album links. An album quak could not
|
||||
decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
|
||||
the exit code at `1` until the server sends a copy that decrypts, or deletes it
|
||||
(see "Backup layout"). An original that a `--verify` run could not read, or put
|
||||
back with bytes that still do not match, stays at its save path. Only a later
|
||||
`--verify` run checks it again: a run without `--verify` leaves it as it is and
|
||||
takes the file out of `failures.json`, so that run can exit 0.
|
||||
(see "Backup layout"). An original that a `--verify` run could not read, or
|
||||
could not replace with a download that matches, stays at its save path as it
|
||||
was. Only a later `--verify` run checks it again: a run without `--verify`
|
||||
leaves it as it is and takes the file out of `failures.json`, so that run can
|
||||
exit 0.
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -347,7 +369,8 @@ quak/
|
||||
exif.ts EXIF read from an image's bytes with exifreader
|
||||
mldata-fetch.ts fetch + decrypt per-file ML data
|
||||
filename.ts safe file names from server metadata
|
||||
errors.ts error types shared across layers
|
||||
errors.ts error types shared across layers, and how an error
|
||||
is described
|
||||
retry.ts retry classifier + exponential backoff with jitter
|
||||
thumbnails.ts detect + regenerate missing thumbnails
|
||||
client.ts high-level Client class assembled from the above
|
||||
@@ -474,6 +497,17 @@ request is repeated only when repeating it could produce a different answer:
|
||||
declining to retry a transient one costs a single file that the next run picks
|
||||
up.
|
||||
|
||||
The `~/bin/quak` that `make install` builds runs under bun, whose `fetch`
|
||||
rejects with a plain `Error` that has no `cause` and carries a code of bun's
|
||||
own. quak retries a refused connection, a name that does not resolve, a reset
|
||||
connection and a deadline under bun as it does under Node. Bun reports a refused
|
||||
connection, and a TLS handshake that failed, as `ConnectionRefused`. For a name
|
||||
that does not resolve it gives `ConnectionRefused` on some requests and
|
||||
`FailedToOpenSocket` on others. A connection reset or closed before the response
|
||||
ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`.
|
||||
A certificate bun rejects, reported with a code such as
|
||||
`DEPTH_ZERO_SELF_SIGNED_CERT`, is not retried.
|
||||
|
||||
Backoff is exponential with full jitter: the delay before retry _n_ is
|
||||
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
|
||||
is the ceiling and the wait is drawn below it, so a client that lost many
|
||||
@@ -529,9 +563,13 @@ routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the
|
||||
same reason, despite looking like connect-time failures: on Linux an ICMP
|
||||
unreachable arriving mid-flight, or a local interface going down after the
|
||||
request was written, delivers them on an already-established socket. They stay
|
||||
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
|
||||
one whole object at one key in one request, so replaying it has no partial state
|
||||
to damage.
|
||||
retryable for the idempotent calls. So do bun's `ConnectionRefused` and
|
||||
`FailedToOpenSocket`, on which a `POST` or `PUT` is not replayed either. When a
|
||||
connection bun kept open from an earlier request closes before any answer, bun
|
||||
sends the request again on a new connection, and if that one fails it gives one
|
||||
of these codes although the server may already have the request. `putFile` is
|
||||
exempt: a presigned PUT stores one whole object at one key in one request, so
|
||||
replaying it has no partial state to damage.
|
||||
|
||||
A download is retried as a whole — request, stream consumption, and decryption —
|
||||
because a socket reset after the response headers have arrived surfaces in the
|
||||
@@ -655,8 +693,8 @@ reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||
|
||||
`backup --verify` also hashes the originals already in the backup and replaces
|
||||
any that do not match the content hash Ente records; one it cannot replace goes
|
||||
into `failures.json` (see "Backup layout").
|
||||
any that do not match the content hash Ente records; one it cannot replace stays
|
||||
as it was and goes into `failures.json` (see "Backup layout").
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
@@ -779,9 +817,9 @@ A running backup loses its lock when someone removes `backup.lock`, or when
|
||||
another backup takes it over because the running one left it untouched for 10
|
||||
seconds, as a stopped or stalled process does. The backup finds this out the
|
||||
next time it goes to touch the lock, which it does every 5 seconds, and then
|
||||
stops before it starts another file. The original it is downloading at that
|
||||
moment is still stored. It leaves `failures.json` and `backup.lock` as they are,
|
||||
since another backup may be using them. `quak backup` prints
|
||||
stops before it starts another file. The originals it is downloading at that
|
||||
moment are still stored. It leaves `failures.json` and `backup.lock` as they
|
||||
are, since another backup may be using them. `quak backup` prints
|
||||
`quak: lost the lock on <dir>: <reason>` as one line and exits 1, and
|
||||
`lib.backup()` rejects with `lost the lock on <dir>: <reason>`.
|
||||
|
||||
@@ -790,8 +828,23 @@ after the file's title, both with unsafe characters replaced. When two
|
||||
collections would get the same name, or two symlinks in one collection the same
|
||||
name (ignoring case in both), each of them gets its ID added: two albums named
|
||||
`Trip` become `Trip (10)/` and `Trip (11)/`, and two files titled `IMG_0001.JPG`
|
||||
become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. IDs never change, so a
|
||||
name stays the same from run to run until such a clash appears or goes away.
|
||||
become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. A collection whose
|
||||
directory would have the name of another collection's JSON, again ignoring case,
|
||||
gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album
|
||||
named `x.json` gets the directory `x.json (12)/`. A name too long for a file
|
||||
name, which most file systems limit to 255 bytes, gets its ID added too. A name
|
||||
with its ID added is cut short between two characters in the part before the ID,
|
||||
just far enough that it fits. When a symlink's extension alone leaves no room
|
||||
for the ID, the extension is cut short and the part before the ID keeps only the
|
||||
bytes that cut leaves over. A collection's directory name also leaves room for
|
||||
the `.json` its JSON adds. IDs never change, so a name stays the same from run
|
||||
to run until such a clash appears or goes away.
|
||||
|
||||
A collection whose directory or JSON cannot be written, such as one whose
|
||||
directory name is taken by a file you put in `collections/`, does not stop the
|
||||
run. A run without `--json` prints `FAILED album <name>: <reason>` for it,
|
||||
`--json` and the `BackupResult` list its ID in `failedAlbums`, and the exit code
|
||||
is 1. The other collections and `failures.json` are still written.
|
||||
|
||||
A live photo, which Ente stores as one ZIP of its image and its video, is stored
|
||||
as those two files, which a photo viewer can open: each is
|
||||
@@ -826,33 +879,41 @@ original already at its save path the way a download is checked (see "On-disk
|
||||
cache layout" below): its bytes, read in chunks, or a live photo's image and
|
||||
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
|
||||
hash its metadata records is left as it is. One that does not is logged on one
|
||||
line naming the file, deleted (a live photo's image and video both), and put
|
||||
back in the same run like a missing one: downloaded, or copied from the cache if
|
||||
the cache holds it. What is put back is hashed too, because a copy from the
|
||||
cache is not checked as a download is. If it still does not match, it stays at
|
||||
its save path and the file goes into `failures.json`, as it does when the
|
||||
download fails. A file whose metadata records no hash is left as it is and
|
||||
counted as unchecked. A stored original that cannot be read is left as it is and
|
||||
counts as failed. The summary and `--json` add the counts `verified`,
|
||||
`mismatched` and `unchecked`, all of originals that were already stored; one
|
||||
first downloaded in this run is in none of them. A mismatch that was put back
|
||||
with matching bytes does not make the exit code non-zero. Without `--verify`
|
||||
nothing is hashed, the summary is unchanged, and the three counts are 0 in
|
||||
`--json`.
|
||||
line naming the file, and replaced once a matching copy is downloaded, in the
|
||||
same run. The copy is downloaded even when the cache holds the original, because
|
||||
a copy from the cache is not checked as a download is. It is written to a
|
||||
temporary file beside the original, a live photo's image and video each to its
|
||||
own, hashed the same way, and renamed over the original only when it matches; a
|
||||
live photo's image and video are both hashed before either is renamed. Until
|
||||
then the original stays as it was: when the download fails, or what it brings
|
||||
does not match either, the temporary files are deleted, the original keeps its
|
||||
bytes and its album links, and the file goes into `failures.json`. A run that
|
||||
stops between renaming a live photo's new image and its new video leaves the new
|
||||
image beside the old video, which a later `--verify` run replaces if the video
|
||||
was the part that did not match. A file whose metadata records no hash is left
|
||||
as it is and counted as unchecked. A stored original that cannot be read is left
|
||||
as it is and counts as failed. The summary and `--json` add the counts
|
||||
`verified`, `mismatched` and `unchecked`, all of originals that were already
|
||||
stored; one first downloaded in this run is in none of them. A mismatch that was
|
||||
replaced with matching bytes does not make the exit code non-zero. Without
|
||||
`--verify` nothing is hashed, the summary is unchanged, and the three counts are
|
||||
0 in `--json`.
|
||||
|
||||
Each original is written to a temporary file in the same directory, synced to
|
||||
disk, and renamed into place, so an original is either complete or absent, even
|
||||
after a power cut. A downloaded original's temporary file is named
|
||||
`.quak-<pid>-<random>.tmp`, one copied from the cache
|
||||
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is
|
||||
killed can leave one of these temporary files behind; the next backup deletes
|
||||
those whose process is no longer running. The content cache uses the same
|
||||
scheme, and opening a library deletes the temporary files in the cache whose
|
||||
process is no longer running, so a download another process has in progress in
|
||||
the same cache is left alone. The rename replaces whatever was at the
|
||||
destination rather than writing through it: a symlink there is replaced, not
|
||||
followed, and the new file has the temporary file's permissions, not those of
|
||||
the file it replaced.
|
||||
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A copy that
|
||||
`--verify` downloads to replace an original is written to such a
|
||||
`.quak-<pid>-<random>.tmp` too, a live photo's image and video to
|
||||
`.quak-<pid>-<random>.<ext>`, each with its own extension. A run that is killed
|
||||
can leave one of these temporary files behind; the next backup deletes those
|
||||
whose process is no longer running. The content cache uses the same scheme, and
|
||||
opening a library deletes the temporary files in the cache whose process is no
|
||||
longer running, so a download another process has in progress in the same cache
|
||||
is left alone. The rename replaces whatever was at the destination rather than
|
||||
writing through it: a symlink there is replaced, not followed, and the new file
|
||||
has the temporary file's permissions, not those of the file it replaced.
|
||||
|
||||
## TODO
|
||||
|
||||
@@ -1085,9 +1146,10 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
|
||||
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
|
||||
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
|
||||
tree it writes. Whether the backup took the lock or its caller did, once the
|
||||
lock is lost the backup rejects with `lost the lock on <dir>: <reason>` before
|
||||
it starts another file.
|
||||
tree it writes, and for the albums `BackupResult` lists in
|
||||
`undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
|
||||
its caller did, once the lock is lost the backup rejects with
|
||||
`lost the lock on <dir>: <reason>` before it starts another file.
|
||||
|
||||
### Request pools
|
||||
|
||||
@@ -1096,6 +1158,11 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
Each pool orders on-demand work ahead of background/precache work and dedups
|
||||
in-flight fetches by key, and an idle pool never lends its slots to a busy one.
|
||||
|
||||
`lib.backup()`, and so `quak backup`, works on up to the content pool's limit of
|
||||
originals at once, 5 unless the library was opened with other `pools`. The limit
|
||||
covers everything it does for one original, hashing it for `verify` included, so
|
||||
however large the account is, no more originals than that are open at once.
|
||||
|
||||
### On-disk cache layout
|
||||
|
||||
Under `cacheDirectory`:
|
||||
|
||||
@@ -17,14 +17,79 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
None: no implementation work is open. The cache design,
|
||||
https://git.eeqj.de/sneak/quak/issues/36, waits on sneak's review.
|
||||
Two bugs are open: https://git.eeqj.de/sneak/quak/issues/198 and
|
||||
https://git.eeqj.de/sneak/quak/issues/205. The check of quak against a live Ente
|
||||
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
||||
has, so it waits on him.
|
||||
|
||||
Tagging and releases are decided by sneak alone, and happen only when he
|
||||
declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-16: With `--verify`, an original whose hash does not match is no
|
||||
longer removed before its replacement is downloaded (issue 199).
|
||||
`replaceOriginal` in `src/library/content.ts` downloads the replacement to a
|
||||
temporary file beside it, a live photo's image and video each to its own,
|
||||
hashes it and renames it over the original only when it matches. When the
|
||||
download fails or does not match either, the original keeps its bytes and
|
||||
links, and the file goes into `failures.json`. The replacement is always
|
||||
downloaded, through the new `fetchOriginal` of the content cache, never copied
|
||||
from the cache.
|
||||
|
||||
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
|
||||
connection and a name that does not resolve, and the backup records either as
|
||||
transient (issue 206). Bun's `fetch` reports them with codes of its own,
|
||||
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
|
||||
as transport failures. A `POST` or `PUT` is not replayed on them, because bun
|
||||
also gives them when it resends a request on a new connection after a
|
||||
connection it kept open closed, and the server may already have the request.
|
||||
Bun already gave `ECONNRESET` for a reset and a `TimeoutError` for a deadline.
|
||||
`classify` in `src/backup.ts` now records any `TimeoutError` as transient, so
|
||||
a stalled download, whose error says only that it stalled, is no longer
|
||||
`unknown`. The shapes were found by running the binary against local
|
||||
listeners.
|
||||
|
||||
- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
|
||||
ends the run before the other albums and `failures.json` are written (issue
|
||||
197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
|
||||
`failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
|
||||
whose folder would have the name of another album's JSON, as `x.json` beside
|
||||
`x` would, gets its ID added. A folder or link name over 255 bytes is cut
|
||||
short between two characters, by `shortenToBytes` in `src/filename.ts`, and
|
||||
gets the ID.
|
||||
|
||||
- 2026-10-15: A network failure is printed and recorded with its reason, not as
|
||||
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
|
||||
error's message followed by the message of each error on its `cause` chain,
|
||||
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
|
||||
reason when a host has several and none could be reached. `run` prints that,
|
||||
and the backup logs it and records it in `failures.json`, where `classify` now
|
||||
calls the failure transient when the chain holds an errno that
|
||||
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
|
||||
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
|
||||
which both use. Tests refuse every connection of a backup, then only one
|
||||
file's download, and fail the ML data fetch.
|
||||
|
||||
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
|
||||
Getting Started and in "Running the backup from cron": `make build-bin` runs
|
||||
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
|
||||
|
||||
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
|
||||
pool's limit of originals at once, 5 by default, instead of one at a time
|
||||
(issue 182). That many workers take files from one shared list, each doing all
|
||||
the work for one original, its `--verify` hashing included, so no more
|
||||
originals than that are open at once. `BackupLibrary` says the limit as
|
||||
`contentConcurrency`. A test holds each download at the fake Ente server to
|
||||
count those in flight. Once a worker has thrown, as one does when `onProgress`
|
||||
throws, or the lock is lost, no worker takes another file; the backup fails
|
||||
once the originals under way are in place, and a lost lock stops it before it
|
||||
fetches ML data.
|
||||
|
||||
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
|
||||
review, though that issue closed on 2026-10-03 (issue 194). It now names the
|
||||
issues open on the tracker.
|
||||
|
||||
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
|
||||
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
|
||||
that records why the lock was lost instead of throwing from its timer, and
|
||||
|
||||
+224
-167
@@ -6,9 +6,9 @@
|
||||
// file. It waits for a completed refresh of the library (a failed one fails the
|
||||
// backup before any file is touched), then, for every file in scope, puts its
|
||||
// original at its save path under `downloadDirectory`, as `Photo.download()`
|
||||
// does, waits for an ML data fetch, and rebuilds the derived views (per-file
|
||||
// sidecars, per-collection symlink trees, per-collection JSON) from the model.
|
||||
// The on-disk layout:
|
||||
// does, up to the content pool's limit of them at once, waits for an ML data
|
||||
// fetch, and rebuilds the derived views (per-file sidecars, per-collection
|
||||
// symlink trees, per-collection JSON) from the model. The on-disk layout:
|
||||
//
|
||||
// <downloadDirectory>/
|
||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||
@@ -41,8 +41,9 @@
|
||||
//
|
||||
// With `verify`, each original already at its save path is hashed as the
|
||||
// download check hashes it, and one that does not match the content hash its
|
||||
// metadata records is removed and fetched again in the same run. What is put
|
||||
// back is hashed too, and recorded as failed if it still does not match.
|
||||
// metadata records is replaced once a matching copy is downloaded, in the same
|
||||
// run. Until then it stays as it is: a download that fails, or that does not
|
||||
// match either, leaves it in place and is recorded as failed.
|
||||
//
|
||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
||||
@@ -50,7 +51,9 @@
|
||||
// running attempt count, and the last-tried time, and the run continues.
|
||||
// A file the library could not decrypt is recorded there too, unless a copy of
|
||||
// it in another album decrypted; an album it could not decrypt is logged by ID
|
||||
// and listed in `result.undecryptableAlbums`, as its name is not known.
|
||||
// and listed in `result.undecryptableAlbums`, as its name is not known. An
|
||||
// album whose folder or JSON cannot be written is logged by name and
|
||||
// listed in `result.failedAlbums`, and the run goes on to the next album.
|
||||
// `result.failed` — and thus the CLI's exit code — stays non-zero while any
|
||||
// failure remains unresolved and clears once every one succeeds. Each run
|
||||
// reconciles the ledger against the files it attempted, so an entry for a file
|
||||
@@ -59,7 +62,6 @@
|
||||
// code.
|
||||
|
||||
import {
|
||||
createReadStream,
|
||||
lstatSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
@@ -75,25 +77,28 @@ import { readFile } from "node:fs/promises";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import lockfile from "proper-lockfile";
|
||||
|
||||
import {
|
||||
chunkHashFinal,
|
||||
chunkHashInit,
|
||||
chunkHashUpdate,
|
||||
init,
|
||||
} from "./crypto/index.js";
|
||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||
import { sanitizeFileName, withExtension } from "./filename.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import {
|
||||
MAX_NAME_BYTES,
|
||||
sanitizeFileName,
|
||||
shortenToBytes,
|
||||
withExtension,
|
||||
} from "./filename.js";
|
||||
import {
|
||||
copyAtomic,
|
||||
placeOriginal,
|
||||
replaceOriginal,
|
||||
savePath,
|
||||
storedAtSavePath,
|
||||
storedHash,
|
||||
} from "./library/content.js";
|
||||
import { representative } from "./library/records.js";
|
||||
import type { Undecryptable } from "./library/store.js";
|
||||
import { extractImageMetadata } from "./metadata-backup.js";
|
||||
import type { MLData } from "./mldata-fetch.js";
|
||||
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
||||
import { hasTransportCode } from "./retry.js";
|
||||
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
@@ -143,9 +148,9 @@ export interface BackupResult {
|
||||
// Originals already at their save path and left untouched.
|
||||
skipped: number;
|
||||
// With `verify`, the originals already at their save path whose hash
|
||||
// matched, those whose hash did not (each removed and fetched again), and
|
||||
// those whose metadata records no hash (left as they are). All three are
|
||||
// zero without `verify`.
|
||||
// matched, those whose hash did not (each replaced once a matching copy is
|
||||
// downloaded), and those whose metadata records no hash (left as they
|
||||
// are). All three are zero without `verify`.
|
||||
verified: number;
|
||||
mismatched: number;
|
||||
unchecked: number;
|
||||
@@ -158,6 +163,9 @@ export interface BackupResult {
|
||||
// The IDs of the albums the library could not decrypt, whose files it
|
||||
// could not list; the CLI exits non-zero while there are any.
|
||||
undecryptableAlbums: number[];
|
||||
// The IDs of the albums whose folder or JSON this run could not write;
|
||||
// the CLI exits non-zero while there are any.
|
||||
failedAlbums: number[];
|
||||
}
|
||||
|
||||
// The slice of the library that backup drives. `Library` implements it; a test
|
||||
@@ -180,7 +188,19 @@ export interface BackupLibrary {
|
||||
fileID: number,
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }>;
|
||||
// Fetch an original's bytes to `destination` through the content pool,
|
||||
// a live photo as its image and its video beside it, even when the cache
|
||||
// or the library's save path already holds it, and return where they
|
||||
// landed. With `verify`, an original whose hash does not match is replaced
|
||||
// with what this fetches.
|
||||
fetchOriginal(
|
||||
fileID: number,
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }>;
|
||||
thumbnail(fileID: number): Promise<{ path: string }>;
|
||||
// How many originals the backup works on at once: the limit of the
|
||||
// library's content pool.
|
||||
contentConcurrency: number;
|
||||
// Wait for an ML data fetch to complete, joining one already running or
|
||||
// starting one. Rejects with the reason when it fails; resolves at once
|
||||
// when the library cannot fetch ML data.
|
||||
@@ -216,11 +236,16 @@ const isPresent = (path: string): boolean => {
|
||||
|
||||
// Best-effort classification for the ledger. Retryable server/network problems
|
||||
// are transient; refusals and local filesystem/decrypt errors are permanent;
|
||||
// anything else is unknown. Both the error code and message are inspected.
|
||||
// anything else is unknown. The error code and the error as `describeError`
|
||||
// gives it are inspected, and so is each errno on the error's cause chain,
|
||||
// where Node's `fetch` puts it. An error named `TimeoutError` is transient
|
||||
// whatever its message says, as `isRetryable` retries it: each of quak's
|
||||
// deadlines aborts with one, and the download's says only that it stalled.
|
||||
const classify = (err: unknown): FailureClass => {
|
||||
if (hasTransportCode(err)) return "transient";
|
||||
const e = err as NodeJS.ErrnoException;
|
||||
const text =
|
||||
`${e?.code ?? ""} ${err instanceof Error ? err.message : String(err)}`.toLowerCase();
|
||||
if (e?.name === "TimeoutError") return "transient";
|
||||
const text = `${e?.code ?? ""} ${describeError(err)}`.toLowerCase();
|
||||
if (
|
||||
/timeout|timed out|econnreset|econnrefused|econnaborted|network|socket|eai_again|throttl|temporarily|429|500|502|503|504/.test(
|
||||
text,
|
||||
@@ -238,9 +263,6 @@ const classify = (err: unknown): FailureClass => {
|
||||
return "unknown";
|
||||
};
|
||||
|
||||
const errorMessage = (err: unknown): string =>
|
||||
err instanceof Error ? err.message : String(err);
|
||||
|
||||
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
|
||||
// non-symlink entry. Throws on failure (a directory in the way, no permission)
|
||||
// so the caller records it and moves on rather than aborting the run.
|
||||
@@ -260,31 +282,47 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
|
||||
// The on-disk names for the entries of one directory, in entry order. Each
|
||||
// name is used as is unless another entry would get the same name, ignoring
|
||||
// case (two names that differ only in case are one entry on a case-insensitive
|
||||
// file system); then every entry sharing it gets ` (<id>)`, before the
|
||||
// extension when `beforeExtension` is set. A name with an ID added can match
|
||||
// another entry's own name (`IMG (6).JPG`), so this repeats until no name is
|
||||
// shared. IDs are stable, so the names are too.
|
||||
// file system), or it is too long for a file name; then it gets ` (<id>)`,
|
||||
// right before the extension when `beforeExtension` is set. To fit, the part
|
||||
// before the ID is cut short as far as it must be. When the extension alone
|
||||
// leaves no room for the ID, the extension is cut short and the part before
|
||||
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each
|
||||
// entry also has the JSON file `<name>.json` beside it, as an album does: a
|
||||
// name then leaves room for `.json`, and one that is another entry's JSON name
|
||||
// gets its ID added too.
|
||||
// Two names with different IDs added never match, but one can match another
|
||||
// entry's own name (`IMG (6).JPG`), so this repeats until no name is shared.
|
||||
// IDs are stable, so the names are too.
|
||||
const uniqueNames = (
|
||||
entries: { id: number; name: string }[],
|
||||
beforeExtension: boolean,
|
||||
opts: { beforeExtension?: boolean; withJSON?: boolean },
|
||||
): string[] => {
|
||||
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
|
||||
const withID = (id: number, name: string): string => {
|
||||
const ext = beforeExtension ? extname(name) : "";
|
||||
const stem = name.slice(0, name.length - ext.length);
|
||||
return `${stem} (${id})${ext}`;
|
||||
const tag = ` (${id})`;
|
||||
const fullExt = opts.beforeExtension ? extname(name) : "";
|
||||
const stem = name.slice(0, name.length - fullExt.length);
|
||||
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag));
|
||||
const room = maxBytes - Buffer.byteLength(tag + ext);
|
||||
return shortenToBytes(stem, room) + tag + ext;
|
||||
};
|
||||
const names = entries.map((e) => e.name);
|
||||
const suffixed = new Set<number>();
|
||||
for (;;) {
|
||||
const counts = new Map<string, number>();
|
||||
for (const name of names) {
|
||||
const count = (name: string): void => {
|
||||
const key = name.toLowerCase();
|
||||
counts.set(key, (counts.get(key) ?? 0) + 1);
|
||||
};
|
||||
for (const name of names) {
|
||||
count(name);
|
||||
if (opts.withJSON) count(`${name}.json`);
|
||||
}
|
||||
let changed = false;
|
||||
for (const [i, { id, name }] of entries.entries()) {
|
||||
if (suffixed.has(i)) continue;
|
||||
if (counts.get(name.toLowerCase()) === 1) continue;
|
||||
const fits = Buffer.byteLength(name) <= maxBytes;
|
||||
if (fits && counts.get(name.toLowerCase()) === 1) continue;
|
||||
names[i] = withID(id, name);
|
||||
suffixed.add(i);
|
||||
changed = true;
|
||||
@@ -411,26 +449,6 @@ const saveLedger = (path: string, ledger: Map<number, FailureEntry>): void => {
|
||||
);
|
||||
};
|
||||
|
||||
// The content hash of the original stored at `stored`, computed as the download
|
||||
// check computes it: over each file's bytes, read in chunks, and for a live
|
||||
// photo `<imageHash>:<videoHash>`.
|
||||
const storedHash = async (stored: {
|
||||
path: string;
|
||||
videoPath?: string;
|
||||
}): Promise<string> => {
|
||||
await init();
|
||||
const hashFile = async (path: string): Promise<string> => {
|
||||
const state = chunkHashInit();
|
||||
for await (const chunk of createReadStream(path)) {
|
||||
chunkHashUpdate(state, chunk as Buffer);
|
||||
}
|
||||
return chunkHashFinal(state);
|
||||
};
|
||||
const hash = await hashFile(stored.path);
|
||||
if (stored.videoPath === undefined) return hash;
|
||||
return `${hash}:${await hashFile(stored.videoPath)}`;
|
||||
};
|
||||
|
||||
// A file's EXIF, XMP and dimensions as its JSON holds them: what
|
||||
// `extractImageMetadata` found in its original, or why the original could not
|
||||
// be read.
|
||||
@@ -467,7 +485,7 @@ const imageMetadataFor = async (
|
||||
const bytes = await readFile(originalPath);
|
||||
return { imageMetadata: extractImageMetadata(bytes) ?? {} };
|
||||
} catch (err) {
|
||||
return { imageMetadataError: errorMessage(err) };
|
||||
return { imageMetadataError: describeError(err) };
|
||||
}
|
||||
};
|
||||
|
||||
@@ -607,7 +625,7 @@ const runLockedBackup = async (
|
||||
// and thumbnail both fail this run must not double its attempt count
|
||||
// or appear twice in errors.
|
||||
if (failedThisRun.has(fileID)) return;
|
||||
const error = errorMessage(err);
|
||||
const error = describeError(err);
|
||||
errors.push({ fileID, title, collection, error });
|
||||
const prior = ledger.get(fileID);
|
||||
ledger.set(fileID, {
|
||||
@@ -652,77 +670,30 @@ const runLockedBackup = async (
|
||||
// through the content cache/pools, as `Photo.download()` does, and fetch
|
||||
// the optional thumbnails; a present file is left as is. With `verify`, a
|
||||
// present original is hashed first, and one that does not match the hash
|
||||
// its metadata records is removed and fetched like a missing one. One
|
||||
// that cannot be read is recorded as failed and left where it is.
|
||||
if (includeOriginals) {
|
||||
for (const [fileID, file] of distinct) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
let stored = storedAtSavePath(downloadDirectory, file);
|
||||
let mismatch = false;
|
||||
if (stored !== undefined && verify) {
|
||||
try {
|
||||
if (file.metadata.hash === undefined) {
|
||||
unchecked++;
|
||||
} else if (
|
||||
(await storedHash(stored)) === file.metadata.hash
|
||||
) {
|
||||
verified++;
|
||||
} else {
|
||||
log(
|
||||
`MISMATCH original ${file.metadata.title} (${fileID}): its bytes do not match its content hash`,
|
||||
);
|
||||
mismatched++;
|
||||
mismatch = true;
|
||||
rmSync(stored.path);
|
||||
if (stored.videoPath !== undefined) {
|
||||
rmSync(stored.videoPath);
|
||||
}
|
||||
stored = undefined;
|
||||
}
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
err,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (stored !== undefined) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
// its metadata records is downloaded again and replaced by
|
||||
// `replaceOriginal`, which leaves it as it was when that download fails
|
||||
// or does not match either. One that cannot be read is recorded as failed
|
||||
// and left where it is. `storeOriginal` does this for one original, and
|
||||
// records its failure.
|
||||
const storeOriginal = async (file: EnteFile): Promise<void> => {
|
||||
const stored = storedAtSavePath(downloadDirectory, file);
|
||||
let mismatch = false;
|
||||
if (stored !== undefined && verify) {
|
||||
try {
|
||||
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
||||
// A fetched original is written straight to its save path (a
|
||||
// live photo beside it); only one that was already cached
|
||||
// elsewhere is copied.
|
||||
const placed = await placeOriginal(
|
||||
downloadDirectory,
|
||||
file,
|
||||
(dest) => lib.original(fileID, dest),
|
||||
);
|
||||
storedThisRun.add(fileID);
|
||||
// A copy from the cache is not checked as a download is and can
|
||||
// hold the same bad bytes, so what is put back after a
|
||||
// mismatch is hashed too. A bad copy stays where it is and the
|
||||
// file fails.
|
||||
if (
|
||||
mismatch &&
|
||||
(await storedHash(placed)) !== file.metadata.hash
|
||||
) {
|
||||
throw new Error(
|
||||
"the original put back does not match its content hash either",
|
||||
if (file.metadata.hash === undefined) {
|
||||
unchecked++;
|
||||
} else if ((await storedHash(stored)) === file.metadata.hash) {
|
||||
verified++;
|
||||
} else {
|
||||
log(
|
||||
`MISMATCH original ${file.metadata.title} (${file.id}): its bytes do not match its content hash`,
|
||||
);
|
||||
mismatched++;
|
||||
mismatch = true;
|
||||
}
|
||||
downloaded++;
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED original ${file.metadata.title}: ${errorMessage(err)}`,
|
||||
`FAILED verifying original ${file.metadata.title}: ${describeError(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
@@ -730,8 +701,72 @@ const runLockedBackup = async (
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
err,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (stored !== undefined && !mismatch) {
|
||||
skipped++;
|
||||
return;
|
||||
}
|
||||
try {
|
||||
log(`Fetching original ${file.metadata.title} (${file.id})...`);
|
||||
if (mismatch) {
|
||||
await replaceOriginal(downloadDirectory, file, (dest) =>
|
||||
lib.fetchOriginal(file.id, dest),
|
||||
);
|
||||
} else {
|
||||
// A fetched original is written straight to its save path (a
|
||||
// live photo beside it); only one that was already cached
|
||||
// elsewhere is copied.
|
||||
await placeOriginal(downloadDirectory, file, (dest) =>
|
||||
lib.original(file.id, dest),
|
||||
);
|
||||
}
|
||||
storedThisRun.add(file.id);
|
||||
downloaded++;
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED original ${file.metadata.title}: ${describeError(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
err,
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Up to the content pool's limit of originals are worked on at once,
|
||||
// their `verify` hashing included, so however large the account is, no
|
||||
// more than that many are open. Each worker takes the next file from the
|
||||
// iterator they all share until none is left. Once a worker has thrown,
|
||||
// as one does when `onProgress` throws, or the lock is lost, no worker
|
||||
// takes another file. The backup fails only once every worker has
|
||||
// stopped, so none writes into the backup after it has ended. It checks
|
||||
// the lock then too, so it does not go on to the thumbnails or the ML
|
||||
// data fetch without it.
|
||||
if (includeOriginals) {
|
||||
const pending = distinct.values();
|
||||
let workerThrew = false;
|
||||
const worker = async (): Promise<void> => {
|
||||
for (const file of pending) {
|
||||
if (workerThrew || lock.lost) return;
|
||||
try {
|
||||
await storeOriginal(file);
|
||||
} catch (err) {
|
||||
workerThrew = true;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
};
|
||||
const workers = await Promise.allSettled(
|
||||
Array.from({ length: lib.contentConcurrency }, worker),
|
||||
);
|
||||
for (const w of workers) {
|
||||
if (w.status === "rejected") throw w.reason;
|
||||
}
|
||||
if (lock.lost) throw lock.lost;
|
||||
}
|
||||
|
||||
if (includeThumbnails) {
|
||||
@@ -760,12 +795,14 @@ const runLockedBackup = async (
|
||||
// instead and is recorded as failed. The next run fetches its ML data
|
||||
// again because none is cached.
|
||||
if (includeOriginals) {
|
||||
let mlDataFailure: unknown;
|
||||
let mlDataError: string | undefined;
|
||||
try {
|
||||
log("Fetching ML data...");
|
||||
await lib.fetchMLData();
|
||||
} catch (err) {
|
||||
mlDataError = errorMessage(err);
|
||||
mlDataFailure = err;
|
||||
mlDataError = describeError(err);
|
||||
log(`FAILED ML data: ${mlDataError}`);
|
||||
}
|
||||
for (const file of distinct.values()) {
|
||||
@@ -789,7 +826,7 @@ const runLockedBackup = async (
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
new Error(`ML data: ${mlDataError}`),
|
||||
new Error("ML data", { cause: mlDataFailure }),
|
||||
);
|
||||
} else {
|
||||
writeSidecar(path, file, { mlData }, image);
|
||||
@@ -808,7 +845,7 @@ const runLockedBackup = async (
|
||||
id: c.id,
|
||||
name: sanitizeFileName(c.name, `collection-${c.id}`),
|
||||
})),
|
||||
false,
|
||||
{ withJSON: true },
|
||||
);
|
||||
const albumDirNames = new Map(
|
||||
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
||||
@@ -821,61 +858,72 @@ const runLockedBackup = async (
|
||||
downloadDirectory,
|
||||
);
|
||||
} catch (err) {
|
||||
log(`FAILED removing old album directories: ${errorMessage(err)}`);
|
||||
log(`FAILED removing old album directories: ${describeError(err)}`);
|
||||
}
|
||||
|
||||
// An album whose folder or JSON cannot be written, such as one whose
|
||||
// folder name is taken by a file the user put there, is logged and listed
|
||||
// in the result, and the run goes on to the next album.
|
||||
const failedAlbums: number[] = [];
|
||||
for (const c of collections) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const colDirName = albumDirNames.get(c.id)!;
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
// Every album links the one original, saved from the file's entry in
|
||||
// `distinct`.
|
||||
const files = filesByCollection.get(c.id) ?? [];
|
||||
const links = files.flatMap((f) =>
|
||||
linksFor(
|
||||
f,
|
||||
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
|
||||
),
|
||||
);
|
||||
const linkNames = uniqueNames(links, true);
|
||||
try {
|
||||
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
||||
} catch (err) {
|
||||
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
|
||||
}
|
||||
const colDirName = albumDirNames.get(c.id)!;
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
const metaFiles = files.map((f) => ({
|
||||
id: f.id,
|
||||
metadata: f.metadata,
|
||||
}));
|
||||
for (const [i, link] of links.entries()) {
|
||||
if (!includeOriginals || link.target === undefined) continue;
|
||||
const linkName = linkNames[i]!;
|
||||
// Every album links the one original, saved from the file's entry
|
||||
// in `distinct`.
|
||||
const files = filesByCollection.get(c.id) ?? [];
|
||||
const links = files.flatMap((f) =>
|
||||
linksFor(
|
||||
f,
|
||||
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
|
||||
),
|
||||
);
|
||||
const linkNames = uniqueNames(links, { beforeExtension: true });
|
||||
try {
|
||||
rebuildSymlink(
|
||||
join(colDir, linkName),
|
||||
relative(colDir, link.target),
|
||||
);
|
||||
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
link.file.id,
|
||||
link.file.metadata.title,
|
||||
c.name,
|
||||
err,
|
||||
`FAILED removing old links in ${c.name}: ${describeError(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
writeAlbumJSON(
|
||||
join(collectionsDir, `${colDirName}.json`),
|
||||
c,
|
||||
metaFiles,
|
||||
);
|
||||
const metaFiles = files.map((f) => ({
|
||||
id: f.id,
|
||||
metadata: f.metadata,
|
||||
}));
|
||||
for (const [i, link] of links.entries()) {
|
||||
if (!includeOriginals || link.target === undefined) continue;
|
||||
const linkName = linkNames[i]!;
|
||||
try {
|
||||
rebuildSymlink(
|
||||
join(colDir, linkName),
|
||||
relative(colDir, link.target),
|
||||
);
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
link.file.id,
|
||||
link.file.metadata.title,
|
||||
c.name,
|
||||
err,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
writeAlbumJSON(
|
||||
join(collectionsDir, `${colDirName}.json`),
|
||||
c,
|
||||
metaFiles,
|
||||
);
|
||||
} catch (err) {
|
||||
log(`FAILED album ${c.name}: ${describeError(err)}`);
|
||||
failedAlbums.push(c.id);
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile the ledger against what this run actually attempted: an entry
|
||||
@@ -901,6 +949,7 @@ const runLockedBackup = async (
|
||||
failed: ledger.size,
|
||||
errors,
|
||||
undecryptableAlbums,
|
||||
failedAlbums,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -960,6 +1009,14 @@ export const runBackup = async (
|
||||
"open the library with one)",
|
||||
);
|
||||
}
|
||||
// As `BoundedPool` does. A limit below 1, or one that is not a number,
|
||||
// would back up no original and end with no error.
|
||||
const { contentConcurrency } = lib;
|
||||
if (!Number.isInteger(contentConcurrency) || contentConcurrency < 1) {
|
||||
throw new RangeError(
|
||||
`contentConcurrency must be a positive integer, got ${contentConcurrency}`,
|
||||
);
|
||||
}
|
||||
if (opts.lock) {
|
||||
return runLockedBackup(lib, opts, downloadDirectory, opts.lock);
|
||||
}
|
||||
|
||||
+3
-1
@@ -470,7 +470,9 @@ export const backupCommand = async (
|
||||
}
|
||||
}
|
||||
|
||||
return result.failed > 0 || result.undecryptableAlbums.length > 0
|
||||
return result.failed > 0 ||
|
||||
result.undecryptableAlbums.length > 0 ||
|
||||
result.failedAlbums.length > 0
|
||||
? 1
|
||||
: 0;
|
||||
} finally {
|
||||
|
||||
+7
-6
@@ -2,14 +2,17 @@
|
||||
|
||||
import type { Writable } from "node:stream";
|
||||
import { ApiError } from "./api/client.js";
|
||||
import { describeError } from "./errors.js";
|
||||
|
||||
// Run a command and exit with its code once stdout/stderr have drained.
|
||||
// Exiting before the drain can truncate piped output, and the library can keep
|
||||
// the event loop alive after a command returns, so a plain return could hang.
|
||||
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
||||
// the stack trace, and exits 1. A 401 from the server means it no longer
|
||||
// accepts the saved session: that prints one line saying to log in again and
|
||||
// exits 3, as a missing or corrupt session file does.
|
||||
// the stack trace, and exits 1. MESSAGE is the error as `describeError` gives
|
||||
// it, so it carries the reason Node's `fetch` puts on `cause`. A 401 from the
|
||||
// server means it no longer accepts the saved session: that prints one line
|
||||
// saying to log in again and exits 3, as a missing or corrupt session file
|
||||
// does.
|
||||
export const run = async (
|
||||
command: Promise<number>,
|
||||
stdout: Writable,
|
||||
@@ -26,9 +29,7 @@ export const run = async (
|
||||
);
|
||||
code = 3;
|
||||
} else {
|
||||
stderr.write(
|
||||
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
|
||||
);
|
||||
stderr.write(`quak: ${describeError(err)}\n`);
|
||||
code = 1;
|
||||
}
|
||||
}
|
||||
|
||||
+15
-9
@@ -135,15 +135,17 @@ const streamDecrypt = async (
|
||||
// cannot tell a partial chunk from a corrupt one, so this
|
||||
// is reported as the truncation it almost always is, with
|
||||
// the authentication failure kept as the error's cause.
|
||||
// Only the pull is guarded: a sink failure on a chunk
|
||||
// that did authenticate is a disk error, not a
|
||||
// The message leaves that failure to the cause, which
|
||||
// `describeError` appends to what `quak backup` prints and
|
||||
// records. Only the pull is guarded: a sink failure on a
|
||||
// chunk that did authenticate is a disk error, not a
|
||||
// truncation.
|
||||
let pulled;
|
||||
try {
|
||||
pulled = pullStreamChunk(state, buffer);
|
||||
} catch (err) {
|
||||
throw new TruncatedStreamError(
|
||||
`download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
|
||||
`download: stream truncated: response body ended with ${buffer.length} trailing bytes (transfer stopped mid-chunk, or the data is corrupt)`,
|
||||
{ cause: err },
|
||||
);
|
||||
}
|
||||
@@ -197,11 +199,13 @@ const isRunning = (pid: number): boolean => {
|
||||
};
|
||||
|
||||
// Delete the temp files a killed process left in `dir`: the writer's
|
||||
// `.quak-<pid>-<random>.tmp` and the backup copy's
|
||||
// `.quak-backup-<name>-<pid>-<random>.tmp`. Only files whose process is no
|
||||
// longer running are removed, so another process writing into the same
|
||||
// directory keeps its own. A reused process ID can only keep a leftover a while
|
||||
// longer, never remove a live one.
|
||||
// `.quak-<pid>-<random>.tmp`, the backup copy's
|
||||
// `.quak-backup-<name>-<pid>-<random>.tmp`, and the image and video of a live
|
||||
// photo downloaded to replace a stored one, `.quak-<pid>-<random>.<ext>` (see
|
||||
// `replaceOriginal`). Only files whose process is no longer running are
|
||||
// removed, so another process writing into the same directory keeps its own. A
|
||||
// reused process ID can only keep a leftover a while longer, never remove a
|
||||
// live one.
|
||||
export const removeLeftoverTempFiles = (dir: string): void => {
|
||||
let names: string[];
|
||||
try {
|
||||
@@ -210,7 +214,9 @@ export const removeLeftoverTempFiles = (dir: string): void => {
|
||||
return;
|
||||
}
|
||||
for (const name of names) {
|
||||
const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.tmp$/.exec(name);
|
||||
const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.[0-9A-Za-z]+$/.exec(
|
||||
name,
|
||||
);
|
||||
if (match && !isRunning(Number(match[1]))) {
|
||||
rmSync(join(dir, name), { force: true });
|
||||
}
|
||||
|
||||
+71
-3
@@ -1,10 +1,13 @@
|
||||
// Error types that more than one layer of quak needs to recognise.
|
||||
// Error types that more than one layer of quak needs to recognise, and how an
|
||||
// error is described in what quak prints and in `failures.json`.
|
||||
//
|
||||
// They live here rather than beside the code that throws them so that the
|
||||
// The types live here rather than beside the code that throws them so that the
|
||||
// retry classifier can identify them without importing the HTTP client or the
|
||||
// download layer — both of which import the classifier. `ApiError` is
|
||||
// re-exported from `src/api/client.ts`, which is where callers have always
|
||||
// imported it from and where it still belongs conceptually.
|
||||
// imported it from and where it still belongs conceptually. The walk along an
|
||||
// error's `cause` chain is here too, because the retry classifier reads the
|
||||
// chain for errnos and `describeError` for messages.
|
||||
|
||||
export class ApiError extends Error {
|
||||
readonly status: number;
|
||||
@@ -43,3 +46,68 @@ export class TruncatedStreamError extends Error {
|
||||
this.name = "TruncatedStreamError";
|
||||
}
|
||||
}
|
||||
|
||||
// `cause` is an arbitrary user-settable property and nothing prevents it from
|
||||
// forming a cycle, so the walk is bounded. Hanging the process would be worse
|
||||
// than misclassifying an error or describing only part of it.
|
||||
const MAX_CAUSE_DEPTH = 8;
|
||||
|
||||
// The error and each error on its `cause` chain, top first, at most
|
||||
// `MAX_CAUSE_DEPTH` of them. `complete` is false when the walk stopped at the
|
||||
// depth limit with more of the chain still below it.
|
||||
export const causeChain = (
|
||||
err: unknown,
|
||||
): { chain: object[]; complete: boolean } => {
|
||||
const chain: object[] = [];
|
||||
let current: unknown = err;
|
||||
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
|
||||
if (current === null || typeof current !== "object") {
|
||||
return { chain, complete: true };
|
||||
}
|
||||
chain.push(current);
|
||||
const { cause } = current as { cause?: unknown };
|
||||
if (cause === current) return { chain, complete: true };
|
||||
current = cause;
|
||||
}
|
||||
return { chain, complete: current === null || typeof current !== "object" };
|
||||
};
|
||||
|
||||
// An error as quak prints it and records it in `failures.json`: its message,
|
||||
// then the message of each error on its `cause` chain, joined with ": ". Node's
|
||||
// `fetch` rejects with just "fetch failed" and puts the reason on `cause`, so
|
||||
// this gives "fetch failed: connect ECONNREFUSED 1.2.3.4:443". When the host
|
||||
// has several addresses and every one fails, the cause is an `AggregateError`
|
||||
// with an empty message and one error per address in `errors`, so an
|
||||
// `AggregateError` is given as the messages of its errors, joined with ", ".
|
||||
// A cause that is not an object, such as a thrown string, is given as
|
||||
// `String(cause)`. Each message is trimmed of the white space around it, as
|
||||
// OpenSSL ends its own with a newline. A message the text already holds is
|
||||
// left out.
|
||||
export const describeError = (err: unknown): string => {
|
||||
if (!(err instanceof Error)) return String(err);
|
||||
const { chain } = causeChain(err);
|
||||
const messages = chain
|
||||
.slice(1)
|
||||
.map((cause) =>
|
||||
cause instanceof AggregateError
|
||||
? cause.errors
|
||||
.map((e: unknown) =>
|
||||
(e instanceof Error ? e.message : String(e)).trim(),
|
||||
)
|
||||
.join(", ")
|
||||
: (cause as { message?: unknown }).message,
|
||||
);
|
||||
// `causeChain` stops at a cause that is not an object, so that one is read
|
||||
// from the last error on the chain.
|
||||
const last = chain[chain.length - 1] as { cause?: unknown };
|
||||
if (last.cause !== undefined && typeof last.cause !== "object") {
|
||||
messages.push(String(last.cause));
|
||||
}
|
||||
let text = err.message.trim();
|
||||
for (const message of messages) {
|
||||
if (typeof message !== "string") continue;
|
||||
const trimmed = message.trim();
|
||||
if (!text.includes(trimmed)) text += `: ${trimmed}`;
|
||||
}
|
||||
return text;
|
||||
};
|
||||
|
||||
+19
-1
@@ -4,7 +4,8 @@
|
||||
// hands us, and quak does not trust the server. Any name taken from them and
|
||||
// used on disk goes through here, so it can only ever name one file inside the
|
||||
// directory the caller chose: never a path, never `..`, never hidden, never a
|
||||
// Windows device name.
|
||||
// Windows device name. Such a name can still be too long for a file name, and
|
||||
// `shortenToBytes` cuts one short.
|
||||
//
|
||||
// A path the user typed (`--out`, `outPath`) is not passed through here: the
|
||||
// caller is trusted, the server is not.
|
||||
@@ -39,3 +40,20 @@ export const safeExtension = (title: string): string => {
|
||||
// `name` with its extension, if it has one, replaced by `ext` (".mov").
|
||||
export const withExtension = (name: string, ext: string): string =>
|
||||
name.slice(0, name.length - extname(name).length) + ext;
|
||||
|
||||
// The longest file name, in bytes, that ext4 and most other file systems allow.
|
||||
export const MAX_NAME_BYTES = 255;
|
||||
|
||||
// `name` cut short to at most `maxBytes` bytes of UTF-8, the encoding a file
|
||||
// name is written to disk in, between two characters, never inside one. A name
|
||||
// that fits comes back unchanged.
|
||||
export const shortenToBytes = (name: string, maxBytes: number): string => {
|
||||
let bytes = 0;
|
||||
let end = 0;
|
||||
for (const character of name) {
|
||||
bytes += Buffer.byteLength(character);
|
||||
if (bytes > maxBytes) break;
|
||||
end += character.length;
|
||||
}
|
||||
return name.slice(0, end);
|
||||
};
|
||||
|
||||
+116
-2
@@ -13,6 +13,8 @@
|
||||
// originals, the thumbnail pool for thumbnails. The pool limits concurrency,
|
||||
// orders on-demand work ahead of background, and dedups by key so a fileID
|
||||
// requested twice while the first is still in flight downloads once.
|
||||
// `fetchOriginal` passes no key, so it is not shared with a fetch of the same
|
||||
// file already in flight.
|
||||
//
|
||||
// Integrity. The reused streaming decrypt is the enforced guarantee: every
|
||||
// chunk is authenticated and the writer renames the file into place only once
|
||||
@@ -27,9 +29,10 @@
|
||||
// It also names each original's save path under the download directory
|
||||
// (`savePath`), where `Photo.download()` and `lib.backup()` put it. A copy in
|
||||
// the cache does not count as saved there, but is copied there rather than
|
||||
// fetched again.
|
||||
// fetched again. An original stored there whose bytes do not match its content
|
||||
// hash is replaced only by a download that does (`replaceOriginal`).
|
||||
|
||||
import { existsSync, readFileSync, statSync } from "node:fs";
|
||||
import { createReadStream, existsSync, readFileSync, statSync } from "node:fs";
|
||||
import {
|
||||
chmod,
|
||||
copyFile,
|
||||
@@ -44,6 +47,12 @@ import {
|
||||
import { basename, dirname, extname, join } from "node:path";
|
||||
|
||||
import type { ApiClient } from "../api/client.js";
|
||||
import {
|
||||
chunkHashFinal,
|
||||
chunkHashInit,
|
||||
chunkHashUpdate,
|
||||
init,
|
||||
} from "../crypto/index.js";
|
||||
import {
|
||||
downloadFile,
|
||||
downloadThumbnail,
|
||||
@@ -349,6 +358,26 @@ export const storedAtSavePath = (
|
||||
return storedOriginal(dirname(path), basename(path, extname(path)), file);
|
||||
};
|
||||
|
||||
// The content hash of the original stored at `stored`, computed as the download
|
||||
// check computes it: over each file's bytes, read in chunks, and for a live
|
||||
// photo `<imageHash>:<videoHash>`.
|
||||
export const storedHash = async (stored: {
|
||||
path: string;
|
||||
videoPath?: string;
|
||||
}): Promise<string> => {
|
||||
await init();
|
||||
const hashFile = async (path: string): Promise<string> => {
|
||||
const state = chunkHashInit();
|
||||
for await (const chunk of createReadStream(path)) {
|
||||
chunkHashUpdate(state, chunk as Buffer);
|
||||
}
|
||||
return chunkHashFinal(state);
|
||||
};
|
||||
const hash = await hashFile(stored.path);
|
||||
if (stored.videoPath === undefined) return hash;
|
||||
return `${hash}:${await hashFile(stored.videoPath)}`;
|
||||
};
|
||||
|
||||
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
||||
// `dest` appears only once it is whole ("present means complete"). As in the
|
||||
// download writer, the temp file is fsynced before the rename and the directory
|
||||
@@ -405,6 +434,62 @@ export const placeOriginal = async (
|
||||
return { path, videoPath };
|
||||
};
|
||||
|
||||
// Replace the original of `file` stored at its save path under `root` with a
|
||||
// new copy, keeping the stored one until the copy is complete and its hash,
|
||||
// computed as `storedHash` computes it, is the content hash the file's metadata
|
||||
// records. `get` is given a temporary file beside the save path and writes the
|
||||
// copy there, a live photo as its image and its video beside that file, each
|
||||
// with its own extension. Only a copy that matches is renamed over the stored
|
||||
// one, and a live photo's image and video are both hashed before either is
|
||||
// renamed. The image is renamed first: should the run stop, or the video's
|
||||
// rename fail, before the video is renamed, the new image is left beside the
|
||||
// old video, and a later backup with `verify` replaces the video if the video
|
||||
// was the part that did not match. When `get` fails or the copy does not match,
|
||||
// the temporary files are removed and the stored original stays as it was.
|
||||
export const replaceOriginal = async (
|
||||
root: string,
|
||||
file: EnteFile,
|
||||
get: (dest: string) => Promise<{ path: string; videoPath?: string }>,
|
||||
): Promise<{ path: string; videoPath?: string }> => {
|
||||
const dest = savePath(root, file);
|
||||
const dir = dirname(dest);
|
||||
// Named as the download writer names its temporary files; a live photo's
|
||||
// image and video get this name with their own extensions. The next
|
||||
// backup removes any that a killed run leaves (`removeLeftoverTempFiles`).
|
||||
const tmp = join(
|
||||
dir,
|
||||
`.quak-${process.pid}-${Math.random().toString(36).slice(2)}.tmp`,
|
||||
);
|
||||
let got: { path: string; videoPath?: string } | undefined;
|
||||
try {
|
||||
got = await get(tmp);
|
||||
if ((await storedHash(got)) !== file.metadata.hash) {
|
||||
throw new Error(
|
||||
"the downloaded original does not match its content hash either",
|
||||
);
|
||||
}
|
||||
if (got.videoPath === undefined) {
|
||||
await rename(got.path, dest);
|
||||
await fsyncPath(dir);
|
||||
return { path: dest };
|
||||
}
|
||||
const path = withExtension(dest, extname(got.path));
|
||||
const videoPath = withExtension(dest, extname(got.videoPath));
|
||||
await rename(got.path, path);
|
||||
await rename(got.videoPath, videoPath);
|
||||
const name = basename(dest, extname(dest));
|
||||
await writeLivePhotoJSON(dir, name, { path, videoPath });
|
||||
return { path, videoPath };
|
||||
} finally {
|
||||
// Whatever was not renamed into place. A failure to remove it must
|
||||
// not replace the error that explains what went wrong.
|
||||
for (const path of [tmp, got?.path, got?.videoPath]) {
|
||||
if (path === undefined) continue;
|
||||
await rm(path, { force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
private readonly pools: RequestPools;
|
||||
private readonly source: ContentSource;
|
||||
@@ -543,6 +628,35 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
};
|
||||
}
|
||||
|
||||
// Fetch an original through the content pool to `destination`, a live
|
||||
// photo as its image and its video beside it, and return where it landed.
|
||||
// Unlike `backupOriginal`, it fetches even when the cache or the save path
|
||||
// already holds the original, and it records nothing: what it writes is
|
||||
// the caller's. It is not shared with a fetch of the same file already
|
||||
// under way, which writes somewhere else.
|
||||
async fetchOriginal(
|
||||
fileID: number,
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }> {
|
||||
const file = this.getFile(fileID);
|
||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||
return this.pools.content.run(
|
||||
async () => {
|
||||
const fetched = await this.fetchInto(
|
||||
file,
|
||||
destination,
|
||||
"original",
|
||||
undefined,
|
||||
);
|
||||
for (const path of [fetched.path, fetched.videoPath]) {
|
||||
if (path !== undefined) await chmod(path, FILE_MODE);
|
||||
}
|
||||
return fetched;
|
||||
},
|
||||
{ priority: "on-demand" },
|
||||
);
|
||||
}
|
||||
|
||||
async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
|
||||
return this.ensureThumbnails(args);
|
||||
}
|
||||
|
||||
@@ -606,7 +606,10 @@ export class Library {
|
||||
undecryptable: () => this.store.listUndecryptable(),
|
||||
original: (fileID, destination) =>
|
||||
cache!.backupOriginal(fileID, destination),
|
||||
fetchOriginal: (fileID, destination) =>
|
||||
cache!.fetchOriginal(fileID, destination),
|
||||
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
||||
contentConcurrency: this.pools.content.concurrency,
|
||||
fetchMLData: () => this.fetchMLDataNow(),
|
||||
mlData: (fileID) => this.mldata.forFile({ fileID }),
|
||||
},
|
||||
|
||||
+38
-22
@@ -10,7 +10,7 @@
|
||||
// declining to retry a transient one costs a single file that the next run
|
||||
// picks up anyway. When the evidence is ambiguous, fail fast.
|
||||
|
||||
import { ApiError, TruncatedStreamError } from "./errors.js";
|
||||
import { ApiError, causeChain, TruncatedStreamError } from "./errors.js";
|
||||
|
||||
export interface RetryOptions {
|
||||
// Total calls, not retries: `attempts: 1` disables retrying.
|
||||
@@ -58,6 +58,14 @@ export const resolveRetryOptions = (
|
||||
});
|
||||
|
||||
// Transport failures: the request did not complete, for reasons below HTTP.
|
||||
//
|
||||
// The errnos are Node's. The binary `make install` builds runs under bun,
|
||||
// whose `fetch` rejects with a plain `Error` that has no `cause` and a `code`
|
||||
// of bun's own. Bun gives `ConnectionRefused` for a refused connection and for
|
||||
// a TLS handshake that failed, and `ConnectionRefused` or `FailedToOpenSocket`
|
||||
// for a name that does not resolve. For a connection reset or closed before
|
||||
// the response ended, it gives Node's `ECONNRESET`. These were found by
|
||||
// running that binary, built with bun 1.3.13, against listeners on 127.0.0.1.
|
||||
const TRANSPORT_CODES = new Set([
|
||||
"ECONNRESET",
|
||||
"ECONNABORTED",
|
||||
@@ -70,6 +78,9 @@ const TRANSPORT_CODES = new Set([
|
||||
"ENETUNREACH",
|
||||
"ENETRESET",
|
||||
"ENETDOWN",
|
||||
// Bun's.
|
||||
"ConnectionRefused",
|
||||
"FailedToOpenSocket",
|
||||
]);
|
||||
|
||||
// The subset of the above that can only be reported before a TCP connection
|
||||
@@ -87,13 +98,14 @@ const TRANSPORT_CODES = new Set([
|
||||
// ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
|
||||
// they remain retryable for idempotent calls; only replay eligibility is
|
||||
// narrowed. See `isSafeToReplay`.
|
||||
//
|
||||
// Bun's `ConnectionRefused` and `FailedToOpenSocket` are absent too, so a
|
||||
// `POST` or `PUT` is not replayed on them. When a connection bun kept open from
|
||||
// an earlier request closes before any answer, bun sends the request again on
|
||||
// a new connection, and if that one fails it gives one of these codes although
|
||||
// the server may already have the request.
|
||||
const CONNECT_CODES = new Set(["ENOTFOUND", "EAI_AGAIN", "ECONNREFUSED"]);
|
||||
|
||||
// `cause` is an arbitrary user-settable property and nothing prevents it from
|
||||
// forming a cycle, so the walk is bounded. Hanging the process would be a
|
||||
// worse outcome than any misclassification.
|
||||
const MAX_CAUSE_DEPTH = 8;
|
||||
|
||||
// Collect every `code` in an error's cause chain. undici does not put the
|
||||
// errno on the error it throws — it hangs the underlying socket error off
|
||||
// `cause`, sometimes more than one level down — so a classifier that only read
|
||||
@@ -101,20 +113,21 @@ const MAX_CAUSE_DEPTH = 8;
|
||||
// connection permanent. `complete` is false when the walk stopped at the
|
||||
// depth limit with more of the chain still below it.
|
||||
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
|
||||
const { chain, complete } = causeChain(err);
|
||||
const codes: string[] = [];
|
||||
let current: unknown = err;
|
||||
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
|
||||
if (current === null || typeof current !== "object") {
|
||||
return { codes, complete: true };
|
||||
}
|
||||
const { code, cause } = current as { code?: unknown; cause?: unknown };
|
||||
for (const link of chain) {
|
||||
const { code } = link as { code?: unknown };
|
||||
if (typeof code === "string") codes.push(code);
|
||||
if (cause === current) return { codes, complete: true };
|
||||
current = cause;
|
||||
}
|
||||
return { codes, complete: current === null || typeof current !== "object" };
|
||||
return { codes, complete };
|
||||
};
|
||||
|
||||
// Does the error's cause chain hold the code of a transport failure, one of
|
||||
// `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in
|
||||
// `src/backup.ts` calls it transient.
|
||||
export const hasTransportCode = (err: unknown): boolean =>
|
||||
causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
|
||||
|
||||
const isAbort = (err: unknown): boolean => {
|
||||
if (err === null || typeof err !== "object") return false;
|
||||
const { name } = err as { name?: unknown };
|
||||
@@ -156,9 +169,10 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
// deliberately literal. Demanding a recognised `cause` instead would
|
||||
// classify real network failures as permanent and fail backups that should
|
||||
// have succeeded; the cost of the imprecision is bounded by the attempt
|
||||
// count.
|
||||
// count. Bun's fetch rejects with a plain `Error` instead, which is
|
||||
// recognised by its code (see `TRANSPORT_CODES`).
|
||||
if (err instanceof TypeError) return true;
|
||||
return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
|
||||
return hasTransportCode(err);
|
||||
};
|
||||
|
||||
// Could the first attempt already have taken effect on the server?
|
||||
@@ -174,11 +188,13 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
//
|
||||
// Everything else is ambiguous. A 5xx proves the server did process the
|
||||
// request. A reset or a broken pipe can arrive after it was fully sent and
|
||||
// acted on. A routing errno can be delivered on an established socket. A
|
||||
// deadline says nothing at all about the server's state. So every errno in the
|
||||
// cause chain must be a connect errno: one other errno anywhere in the chain
|
||||
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt
|
||||
// too: the links below the limit were never read.
|
||||
// acted on. A routing errno can be delivered on an established socket. Bun's
|
||||
// `ConnectionRefused` and `FailedToOpenSocket` can come from a new connection
|
||||
// bun opened to resend a request that already went out on a connection it kept
|
||||
// open. A deadline says nothing at all about the server's state. So every code in the
|
||||
// cause chain must be one of `CONNECT_CODES`: one other code anywhere in the
|
||||
// chain is doubt, and doubt is not replayed. A chain longer than the walk is
|
||||
// doubt too: the links below the limit were never read.
|
||||
export const isSafeToReplay = (err: unknown): boolean => {
|
||||
const { codes, complete } = causeCodes(err);
|
||||
return (
|
||||
|
||||
@@ -0,0 +1,608 @@
|
||||
/**
|
||||
* How many originals `lib.backup()` works on at once.
|
||||
*
|
||||
* The backup puts up to the content pool's limit of originals at their save
|
||||
* paths at the same time: 5 for a library opened with the default
|
||||
* `RequestPools`, as `quak backup` opens it, or the `contentConcurrency` of the
|
||||
* pools a library was opened with. Everything the backup does for one original
|
||||
* counts against that limit, hashing it for `verify` included, so however large
|
||||
* the account is, the backup has no more than that many originals open at once.
|
||||
*
|
||||
* As in `backup-end-to-end.test.ts`, the real `Client` and `Library` run
|
||||
* against `buildMetaFetch` from `test/ente-server.ts`, a fake of the Ente HTTP
|
||||
* API serving data this file encrypts with libsodium. Here a test can also hold
|
||||
* each original's download until it lets it go, to see how many the backup has
|
||||
* in flight at once.
|
||||
*
|
||||
* The account, user 42, has one album of its own, Holiday, of eight photos,
|
||||
* all taken at noon on 1 March 2024, each with an original of its own.
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
lstatSync,
|
||||
mkdtempSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import type { BackupOptions } from "../../src/backup.js";
|
||||
import { Client, type ClientSnapshot } from "../../src/client.js";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import { RequestPools } from "../../src/library/pools.js";
|
||||
import {
|
||||
buildAccount,
|
||||
buildMetaFetch,
|
||||
encryptSecretbox,
|
||||
encryptStreamBlob,
|
||||
} from "../ente-server.js";
|
||||
import { blake2b } from "../live-photo.js";
|
||||
|
||||
const EMAIL = "backup@example.com";
|
||||
const PASSWORD = "backup-password";
|
||||
// The fake server logs every account in as user 42.
|
||||
const USER_ID = 42;
|
||||
|
||||
// Noon on 1 March 2024 in the time zone of the machine running the test, in
|
||||
// microseconds, as Ente records times.
|
||||
const TAKEN = new Date(2024, 2, 1, 12).getTime() * 1000;
|
||||
|
||||
// The eight photos' IDs. Photo `id` is titled `photo-<id>.jpg`, and its
|
||||
// original is a few bytes naming it.
|
||||
const PHOTO_IDS = [101, 102, 103, 104, 105, 106, 107, 108];
|
||||
const originalOf = (id: number): Uint8Array =>
|
||||
new TextEncoder().encode(`the photo ${id}`);
|
||||
|
||||
// Where a backup to `dir` saves photo `id`'s original.
|
||||
const savedAt = (dir: string, id: number): string =>
|
||||
join(dir, "2024/2024-03/2024-03-01", `2024-03-01.${id}.jpg`);
|
||||
|
||||
// The fake server, the session it logs the account in with, and the
|
||||
// directory every backup below is made in.
|
||||
let server: typeof globalThis.fetch;
|
||||
let session: ClientSnapshot;
|
||||
let root: string;
|
||||
|
||||
// The fake server, holding the account encrypted as Ente's clients upload it.
|
||||
const buildServer = async () => {
|
||||
const account = await buildAccount(PASSWORD);
|
||||
|
||||
// The album as `/collections/v2` lists it: its key encrypted with the
|
||||
// account's master key, its name with its own key.
|
||||
const albumKey = sodium.crypto_secretbox_keygen();
|
||||
const encryptedAlbumKey = encryptSecretbox(albumKey, account.masterKey);
|
||||
const encryptedName = encryptSecretbox(
|
||||
new TextEncoder().encode("Holiday"),
|
||||
albumKey,
|
||||
);
|
||||
const album = {
|
||||
id: 10,
|
||||
owner: { id: USER_ID },
|
||||
encryptedKey: toBase64(encryptedAlbumKey.ciphertext),
|
||||
keyDecryptionNonce: toBase64(encryptedAlbumKey.nonce),
|
||||
encryptedName: toBase64(encryptedName.ciphertext),
|
||||
nameDecryptionNonce: toBase64(encryptedName.nonce),
|
||||
type: "album",
|
||||
updationTime: TAKEN,
|
||||
};
|
||||
|
||||
// Each photo as the album's `/collections/v2/diff` lists it: its key
|
||||
// encrypted with the album's key, and its metadata, holding the hash of
|
||||
// its original, encrypted with its own key. Its original, encrypted with
|
||||
// its own key too, is what the files CDN serves for it.
|
||||
const ciphertexts: Record<number, Uint8Array> = {};
|
||||
const photos = PHOTO_IDS.map((id) => {
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const original = encryptStreamBlob(originalOf(id), key);
|
||||
ciphertexts[id] = original.ciphertext;
|
||||
const encryptedKey = encryptSecretbox(key, albumKey);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title: `photo-${id}.jpg`,
|
||||
fileType: 0,
|
||||
hash: blake2b(originalOf(id)),
|
||||
creationTime: TAKEN,
|
||||
modificationTime: TAKEN,
|
||||
}),
|
||||
),
|
||||
key,
|
||||
);
|
||||
return {
|
||||
id,
|
||||
collectionID: 10,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: toBase64(encryptedKey.ciphertext),
|
||||
keyDecryptionNonce: toBase64(encryptedKey.nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
file: { decryptionHeader: toBase64(original.header) },
|
||||
thumbnail: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
updationTime: TAKEN,
|
||||
};
|
||||
});
|
||||
|
||||
return buildMetaFetch({
|
||||
verifier: account.verifier,
|
||||
srpAttributes: account.srpAttributes,
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections: [album],
|
||||
filesByCollection: { 10: photos },
|
||||
encryptedMLData: {},
|
||||
fileCiphertexts: ciphertexts,
|
||||
});
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
root = mkdtempSync(join(tmpdir(), "quak-backup-concurrency-"));
|
||||
server = await buildServer();
|
||||
const client = await Client.login({
|
||||
email: EMAIL,
|
||||
password: PASSWORD,
|
||||
apiOptions: { fetch: server },
|
||||
});
|
||||
session = client.toJSON();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// The ID of the photo whose original a request downloads, or undefined for any
|
||||
// other request. quak downloads an original from
|
||||
// `https://files.ente.io/?fileID=<id>`.
|
||||
const downloadOf = (input: RequestInfo | URL): number | undefined => {
|
||||
const url = new URL(String(input));
|
||||
return url.hostname === "files.ente.io"
|
||||
? Number(url.searchParams.get("fileID"))
|
||||
: undefined;
|
||||
};
|
||||
|
||||
// The fake server, except that it answers 404 to every download of photo
|
||||
// `failing`'s original.
|
||||
const failingFor =
|
||||
(failing: number): typeof globalThis.fetch =>
|
||||
async (input, init) =>
|
||||
downloadOf(input) === failing
|
||||
? new Response("not found", { status: 404 })
|
||||
: server(input, init);
|
||||
|
||||
// The fake server, holding each download of an original until the test lets
|
||||
// it go. `held` lists the downloads waiting, the first to arrive first;
|
||||
// `most()` is the largest number that ever waited at once. A held download of
|
||||
// photo `failing`'s original is answered 404 once it is let go.
|
||||
const holdingServer = (failing?: number) => {
|
||||
const held: { id: number; release: () => void }[] = [];
|
||||
let most = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
const id = downloadOf(input);
|
||||
if (id !== undefined) {
|
||||
await new Promise<void>((release) => {
|
||||
held.push({ id, release });
|
||||
most = Math.max(most, held.length);
|
||||
});
|
||||
if (id === failing) {
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
return { fetch, held, most: () => most };
|
||||
};
|
||||
|
||||
// Let the held downloads go one at a time, the first to arrive first, each
|
||||
// once `limit` of them wait, or all the downloads still to come when there
|
||||
// are fewer. Each of the eight photos is downloaded once. With a backup that
|
||||
// downloads fewer than `limit` at once, this fails waiting for them. Each wait
|
||||
// allows five seconds, for a loaded machine's disk to sync the files written.
|
||||
const releaseOneAtATime = async (
|
||||
held: { release: () => void }[],
|
||||
limit: number,
|
||||
): Promise<void> => {
|
||||
for (let left = PHOTO_IDS.length; left > 0; left--) {
|
||||
await vi.waitFor(
|
||||
() => expect(held).toHaveLength(Math.min(limit, left)),
|
||||
{ interval: 5, timeout: 5000 },
|
||||
);
|
||||
held.shift()!.release();
|
||||
}
|
||||
};
|
||||
|
||||
// A library of the account opened as `quak backup` opens one, with no
|
||||
// precache and no background refresh, sending its requests through `fetch`.
|
||||
// It backs up to `dir` and keeps its cache beside it. Its request pools are
|
||||
// `pools`, or by default the library's own.
|
||||
const openLibrary = (
|
||||
dir: string,
|
||||
fetch: typeof globalThis.fetch,
|
||||
pools?: RequestPools,
|
||||
): Promise<Library> =>
|
||||
Library.open({
|
||||
client: Client.fromJSON(session, { fetch }),
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: `${dir}-cache`,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
pools,
|
||||
});
|
||||
|
||||
// Every entry under `dir`, relative to it, sorted, a symlink as
|
||||
// `name -> target`.
|
||||
const listTree = (dir: string): string[] =>
|
||||
readdirSync(dir, { recursive: true })
|
||||
.sort()
|
||||
.map((name) => {
|
||||
const path = join(dir, name);
|
||||
return lstatSync(path).isSymbolicLink()
|
||||
? `${name} -> ${readlinkSync(path)}`
|
||||
: name;
|
||||
});
|
||||
|
||||
const expectStored = (dir: string, ids: number[]): void => {
|
||||
for (const id of ids) {
|
||||
expect(readFileSync(savedAt(dir, id))).toEqual(
|
||||
Buffer.from(originalOf(id)),
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
describe("lib.backup()", () => {
|
||||
// A library opened with the default pools, as `quak backup` opens one,
|
||||
// downloads up to five originals at once. As soon as one finishes, the
|
||||
// backup starts on the next photo, so five stay in flight until fewer
|
||||
// than five are left.
|
||||
it("downloads five originals at once, and never more, with the default pools", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const holding = holdingServer();
|
||||
const lib = await openLibrary(dir, holding.fetch);
|
||||
try {
|
||||
const backup = lib.backup();
|
||||
await releaseOneAtATime(holding.held, 5);
|
||||
const result = await backup;
|
||||
|
||||
expect(holding.most()).toBe(5);
|
||||
expect(result).toMatchObject({
|
||||
totalFiles: 8,
|
||||
downloaded: 8,
|
||||
failed: 0,
|
||||
});
|
||||
expectStored(dir, PHOTO_IDS);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// The limit is the content pool's, so a library opened with pools of
|
||||
// another `contentConcurrency` backs up that many at once.
|
||||
it("downloads as many originals at once as the content pool of the pools it was opened with allows", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const holding = holdingServer();
|
||||
const lib = await openLibrary(
|
||||
dir,
|
||||
holding.fetch,
|
||||
new RequestPools({ contentConcurrency: 2 }),
|
||||
);
|
||||
try {
|
||||
const backup = lib.backup();
|
||||
await releaseOneAtATime(holding.held, 2);
|
||||
const result = await backup;
|
||||
|
||||
expect(holding.most()).toBe(2);
|
||||
expect(result).toMatchObject({ downloaded: 8, failed: 0 });
|
||||
expectStored(dir, PHOTO_IDS);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// With `verify`, hashing an original already stored is part of the work
|
||||
// on it, and counts against the limit too: a large backup never has more
|
||||
// than five originals open for hashing. Here every stored original has
|
||||
// changed on disk, so each is hashed, found not to match, and downloaded
|
||||
// again. While five downloads wait, the backup has hashed only those five
|
||||
// photos' originals; it opens the next one once a download finishes.
|
||||
it("with verify, hashes no more originals ahead of their downloads than the limit", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const first = await openLibrary(dir, server);
|
||||
try {
|
||||
await first.backup();
|
||||
} finally {
|
||||
await first.close();
|
||||
}
|
||||
for (const id of PHOTO_IDS) {
|
||||
writeFileSync(savedAt(dir, id), "not the photo any more");
|
||||
}
|
||||
|
||||
const holding = holdingServer();
|
||||
const lib = await openLibrary(dir, holding.fetch);
|
||||
const lines: string[] = [];
|
||||
const mismatches = () =>
|
||||
lines.filter((line) => line.startsWith("MISMATCH"));
|
||||
try {
|
||||
const backup = lib.backup({
|
||||
verify: true,
|
||||
onProgress: (line) => lines.push(line),
|
||||
});
|
||||
await vi.waitFor(() => expect(holding.held).toHaveLength(5), {
|
||||
interval: 5,
|
||||
timeout: 5000,
|
||||
});
|
||||
expect(mismatches()).toHaveLength(5);
|
||||
|
||||
await releaseOneAtATime(holding.held, 5);
|
||||
const result = await backup;
|
||||
|
||||
expect(holding.most()).toBe(5);
|
||||
expect(mismatches()).toHaveLength(8);
|
||||
expect(result).toMatchObject({
|
||||
downloaded: 8,
|
||||
verified: 0,
|
||||
mismatched: 8,
|
||||
failed: 0,
|
||||
});
|
||||
expectStored(dir, PHOTO_IDS);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// The server answers 404 for photo 103's original. A 404 is not retried,
|
||||
// so that download fails at once, while other downloads are still in
|
||||
// flight. The backup records the failure and goes on: the downloads
|
||||
// beside it finish, the photos after it are downloaded, and only photo
|
||||
// 103 is missing from the backup and in failures.json.
|
||||
it("goes on with the other originals when one fails", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const holding = holdingServer(103);
|
||||
const lib = await openLibrary(dir, holding.fetch);
|
||||
try {
|
||||
const backup = lib.backup();
|
||||
await releaseOneAtATime(holding.held, 5);
|
||||
const result = await backup;
|
||||
|
||||
expect(result).toMatchObject({
|
||||
totalFiles: 8,
|
||||
downloaded: 7,
|
||||
failed: 1,
|
||||
});
|
||||
expect(result.errors.map((e) => e.fileID)).toEqual([103]);
|
||||
expectStored(
|
||||
dir,
|
||||
PHOTO_IDS.filter((id) => id !== 103),
|
||||
);
|
||||
expect(existsSync(savedAt(dir, 103))).toBe(false);
|
||||
const failures = JSON.parse(
|
||||
readFileSync(join(dir, "failures.json"), "utf-8"),
|
||||
);
|
||||
expect(Object.keys(failures.files)).toEqual(["103"]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A throw from `onProgress` fails the backup: `lib.backup()` rejects with
|
||||
// it. From then on the backup starts no further original. It finishes the
|
||||
// ones it is working on, and rejects only once they are in place, so
|
||||
// nothing is written into the backup after it has ended and removed its
|
||||
// lock. Here the first five downloads wait. Photo 103's is let go first
|
||||
// and answered 404, and the line reporting that failure throws while the
|
||||
// other four wait. Only those five downloads are held: a backup that went
|
||||
// on would store the other three photos.
|
||||
it("when onProgress throws, starts no further original, and rejects once the originals it was working on are in place", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const downloads: number[] = [];
|
||||
const held: { id: number; release: () => void }[] = [];
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
const id = downloadOf(input);
|
||||
if (id !== undefined) {
|
||||
downloads.push(id);
|
||||
if (downloads.length <= 5) {
|
||||
await new Promise<void>((release) =>
|
||||
held.push({ id, release }),
|
||||
);
|
||||
}
|
||||
if (id === 103) {
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const lib = await openLibrary(dir, fetch);
|
||||
const lines: string[] = [];
|
||||
try {
|
||||
const backup = lib.backup({
|
||||
onProgress: (line) => {
|
||||
lines.push(line);
|
||||
if (line.startsWith("FAILED")) {
|
||||
throw new Error("the progress output is closed");
|
||||
}
|
||||
},
|
||||
});
|
||||
// The photos whose originals are in place, and whether the lock
|
||||
// is there, at the moment `lib.backup()` rejects.
|
||||
let storedWhenRejected: number[] = [];
|
||||
let lockedWhenRejected = true;
|
||||
const rejection = backup.catch((err: unknown) => {
|
||||
storedWhenRejected = PHOTO_IDS.filter((id) =>
|
||||
existsSync(savedAt(dir, id)),
|
||||
);
|
||||
lockedWhenRejected = existsSync(join(dir, "backup.lock"));
|
||||
return err;
|
||||
});
|
||||
|
||||
await vi.waitFor(() => expect(held).toHaveLength(5), {
|
||||
interval: 5,
|
||||
timeout: 5000,
|
||||
});
|
||||
const at103 = held.findIndex((h) => h.id === 103);
|
||||
held.splice(at103, 1)[0]!.release();
|
||||
await vi.waitFor(
|
||||
() =>
|
||||
expect(
|
||||
lines.some((line) => line.startsWith("FAILED")),
|
||||
).toBe(true),
|
||||
{ interval: 5, timeout: 5000 },
|
||||
);
|
||||
for (const download of held) download.release();
|
||||
|
||||
expect(await rejection).toEqual(
|
||||
new Error("the progress output is closed"),
|
||||
);
|
||||
expect(downloads.sort()).toEqual([101, 102, 103, 104, 105]);
|
||||
expect(storedWhenRejected).toEqual([101, 102, 104, 105]);
|
||||
expect(lockedWhenRejected).toBe(false);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// Once the backup's lock is lost, another backup may be writing to the
|
||||
// directory, so the backup starts no further original. It finishes the
|
||||
// ones it is working on, and `lib.backup()` rejects once they are all in
|
||||
// place. Here `backup.lock` is removed while the first five downloads
|
||||
// wait, and proper-lockfile's timer finds it gone. As in
|
||||
// `backup-end-to-end.test.ts`, only `setTimeout` and `clearTimeout` are
|
||||
// faked, so the test can fire that timer without waiting 5 seconds. Only
|
||||
// the first five downloads are held: a backup that went on would store
|
||||
// the other three photos.
|
||||
it("when its lock is lost, starts no further original, and rejects once the originals it was working on are in place", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const lockPath = join(dir, "backup.lock");
|
||||
const downloads: number[] = [];
|
||||
const held: (() => void)[] = [];
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
const id = downloadOf(input);
|
||||
if (id !== undefined) {
|
||||
downloads.push(id);
|
||||
if (downloads.length <= 5) {
|
||||
await new Promise<void>((release) => held.push(release));
|
||||
}
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const lib = await openLibrary(dir, fetch);
|
||||
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
|
||||
try {
|
||||
const backup = lib.backup();
|
||||
// The photos whose originals are in place at the moment
|
||||
// `lib.backup()` rejects.
|
||||
let storedWhenRejected: number[] = [];
|
||||
const rejection = backup.catch((err: unknown) => {
|
||||
storedWhenRejected = PHOTO_IDS.filter((id) =>
|
||||
existsSync(savedAt(dir, id)),
|
||||
);
|
||||
return err;
|
||||
});
|
||||
|
||||
await vi.waitFor(() => expect(held).toHaveLength(5), {
|
||||
interval: 5,
|
||||
timeout: 5000,
|
||||
});
|
||||
rmSync(lockPath, { recursive: true });
|
||||
// Fire proper-lockfile's timer, and wait for the look at the lock
|
||||
// on disk that it starts to finish: Node lists a file system call
|
||||
// in progress as an `FSReqCallback`.
|
||||
vi.advanceTimersByTime(5_000);
|
||||
while (process.getActiveResourcesInfo().includes("FSReqCallback")) {
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
}
|
||||
for (const release of held) release();
|
||||
|
||||
expect(await rejection).toEqual(
|
||||
new Error(
|
||||
`lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'`,
|
||||
),
|
||||
);
|
||||
expect(downloads.sort()).toEqual([101, 102, 103, 104, 105]);
|
||||
expect(storedWhenRejected).toEqual([101, 102, 103, 104, 105]);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// Working on several originals at once changes only how long a backup
|
||||
// takes. The same two runs, with a content pool of one, which backs up
|
||||
// one original at a time, and with the default pools, end the same way:
|
||||
// the same result, the same progress lines (in another order), the same
|
||||
// failures.json, and the same tree. The first run stores every photo but
|
||||
// 103, whose download is answered 404. Then two stored originals change
|
||||
// on disk, and a run with `verify` puts them back and tries 103 again.
|
||||
it("ends as a backup of one original at a time does", async () => {
|
||||
const twoRuns = async (pools: RequestPools) => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const lines: string[] = [];
|
||||
const backup = async (opts: BackupOptions) => {
|
||||
const lib = await openLibrary(dir, failingFor(103), pools);
|
||||
try {
|
||||
return await lib.backup({
|
||||
...opts,
|
||||
onProgress: (line) => lines.push(line),
|
||||
});
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
const first = await backup({});
|
||||
writeFileSync(savedAt(dir, 101), "not the photo any more");
|
||||
writeFileSync(savedAt(dir, 105), "not the photo any more");
|
||||
const second = await backup({ verify: true });
|
||||
|
||||
// Each entry of failures.json without the time it was last
|
||||
// tried, which differs from run to run.
|
||||
const { files } = JSON.parse(
|
||||
readFileSync(join(dir, "failures.json"), "utf-8"),
|
||||
) as { files: Record<string, { lastTriedAt?: number }> };
|
||||
for (const entry of Object.values(files)) delete entry.lastTriedAt;
|
||||
|
||||
return {
|
||||
first,
|
||||
second,
|
||||
lines: lines.sort(),
|
||||
failures: files,
|
||||
tree: listTree(dir),
|
||||
};
|
||||
};
|
||||
|
||||
const oneAtATime = await twoRuns(
|
||||
new RequestPools({ contentConcurrency: 1 }),
|
||||
);
|
||||
const fiveAtATime = await twoRuns(new RequestPools());
|
||||
|
||||
expect(oneAtATime.first).toMatchObject({
|
||||
totalFiles: 8,
|
||||
downloaded: 7,
|
||||
skipped: 0,
|
||||
failed: 1,
|
||||
});
|
||||
expect(oneAtATime.second).toMatchObject({
|
||||
totalFiles: 8,
|
||||
downloaded: 2,
|
||||
skipped: 5,
|
||||
verified: 5,
|
||||
mismatched: 2,
|
||||
unchecked: 0,
|
||||
failed: 1,
|
||||
});
|
||||
expect(oneAtATime.failures).toEqual({
|
||||
"103": expect.objectContaining({ fileID: 103, attempts: 2 }),
|
||||
});
|
||||
expect(fiveAtATime).toEqual(oneAtATime);
|
||||
});
|
||||
});
|
||||
@@ -274,15 +274,17 @@ afterAll(() => {
|
||||
|
||||
// Run `quak backup <dir>` as `bin/quak.ts` runs it, sending its requests
|
||||
// through `fetch`, and return its exit code and everything it printed on
|
||||
// stderr. Its requests are retried as those of `quak backup` are, except that
|
||||
// a retry does not wait. Its cache is beside `dir`, so each run of one
|
||||
// directory finds the cache the run before it left, as cron's runs find the
|
||||
// user's.
|
||||
// stderr and on stdout. Its requests are retried as those of `quak backup`
|
||||
// are, except that a retry does not wait. A download is given up after
|
||||
// `downloadTimeoutMs` with no bytes received, 60 seconds unless given. Its
|
||||
// cache is beside `dir`, so each run of one directory finds the cache the run
|
||||
// before it left, as cron's runs find the user's.
|
||||
const backup = async (
|
||||
dir: string,
|
||||
opts: { verify?: boolean } = {},
|
||||
opts: { verify?: boolean; json?: boolean } = {},
|
||||
fetch = server,
|
||||
): Promise<{ code: number; stderr: string }> => {
|
||||
downloadTimeoutMs?: number,
|
||||
): Promise<{ code: number; stderr: string; stdout: string }> => {
|
||||
let printed = "";
|
||||
const stderr = new Writable({
|
||||
write(chunk: Buffer, _encoding, done) {
|
||||
@@ -290,8 +292,10 @@ const backup = async (
|
||||
done();
|
||||
},
|
||||
});
|
||||
let output = "";
|
||||
const stdout = new Writable({
|
||||
write(_chunk, _encoding, done) {
|
||||
write(chunk: Buffer, _encoding, done) {
|
||||
output += chunk.toString();
|
||||
done();
|
||||
},
|
||||
});
|
||||
@@ -308,6 +312,7 @@ const backup = async (
|
||||
loadSession(path, {
|
||||
fetch,
|
||||
retry: { ...UNATTENDED_RETRY_OPTIONS, sleep: async () => {} },
|
||||
downloadTimeoutMs,
|
||||
}),
|
||||
login: prompted,
|
||||
prompt: prompted,
|
||||
@@ -316,7 +321,7 @@ const backup = async (
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(backupCommand(ctx, dir, opts), stdout, stderr, resolve);
|
||||
});
|
||||
return { code, stderr: printed };
|
||||
return { code, stderr: printed, stdout: output };
|
||||
};
|
||||
|
||||
// The ID of the file whose original a request downloads, or undefined for any
|
||||
@@ -329,6 +334,41 @@ const downloadOf = (input: RequestInfo | URL): number | undefined => {
|
||||
: undefined;
|
||||
};
|
||||
|
||||
// The error Node gives for a connection to `address` that was refused.
|
||||
const refusedAt = (address: string): Error =>
|
||||
Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
|
||||
code: "ECONNREFUSED",
|
||||
});
|
||||
|
||||
// What Node's `fetch` rejects with when the server refuses the connection, as
|
||||
// for every failure below HTTP: a TypeError that says only "fetch failed",
|
||||
// with the reason, and its errno, as its cause. When the server has several
|
||||
// addresses and each one refuses, the cause is an AggregateError with an empty
|
||||
// message, one error per address and the errno of the first. `reason` is the
|
||||
// error as quak prints it and records it in `failures.json`.
|
||||
const REFUSED = [
|
||||
{
|
||||
addresses: "its one address",
|
||||
error: (): TypeError =>
|
||||
new TypeError("fetch failed", { cause: refusedAt("1.2.3.4:443") }),
|
||||
reason: "fetch failed: connect ECONNREFUSED 1.2.3.4:443",
|
||||
},
|
||||
{
|
||||
addresses: "each of its two addresses",
|
||||
error: (): TypeError =>
|
||||
new TypeError("fetch failed", {
|
||||
cause: Object.assign(
|
||||
new AggregateError([
|
||||
refusedAt("2001:db8::1:443"),
|
||||
refusedAt("1.2.3.4:443"),
|
||||
]),
|
||||
{ code: "ECONNREFUSED" },
|
||||
),
|
||||
}),
|
||||
reason: "fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443",
|
||||
},
|
||||
];
|
||||
|
||||
// The fake server, pushing onto `downloads` the ID of each file whose original
|
||||
// is downloaded.
|
||||
const recordingDownloads =
|
||||
@@ -592,6 +632,40 @@ describe("quak backup", () => {
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
// The user has put a file in collections/ with the name of the Trips
|
||||
// folder, so quak cannot make that folder. It says so on one line, writes
|
||||
// the two albums after it, and exits 1, and --json lists the album's ID
|
||||
// in `failedAlbums`. No file failed, so there is no failures.json.
|
||||
it("exits 1 when an album's folder cannot be made, writes the other albums, and lists it in --json", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
mkdirSync(join(dir, "collections"));
|
||||
writeFileSync(join(dir, "collections", "Trips"), "mine");
|
||||
|
||||
const first = await backup(dir);
|
||||
const { code, stdout } = await backup(dir, { json: true });
|
||||
|
||||
expect(first.code).toBe(1);
|
||||
expect(first.stderr).toMatch(/^FAILED album Trips: EEXIST: /m);
|
||||
expect(code).toBe(1);
|
||||
expect(JSON.parse(stdout)).toMatchObject({
|
||||
failed: 0,
|
||||
undecryptableAlbums: [],
|
||||
failedAlbums: [10],
|
||||
});
|
||||
const collections = join(dir, "collections");
|
||||
expect(readJSON(join(collections, "Family.json")).id).toBe(20);
|
||||
expect(readdirSync(join(collections, "Family")).sort()).toEqual([
|
||||
"IMG_0002.heic",
|
||||
"IMG_0002.mov",
|
||||
"beach.jpg",
|
||||
]);
|
||||
expect(readJSON(join(collections, "Friend's Wedding.json")).id).toBe(
|
||||
30,
|
||||
);
|
||||
expect(existsSync(join(collections, "Trips.json"))).toBe(false);
|
||||
expect(existsSync(join(dir, "failures.json"))).toBe(false);
|
||||
});
|
||||
|
||||
// The server no longer accepts the saved session's token, as after
|
||||
// `quak logout` elsewhere, and answers every request with 401. The run
|
||||
// stops at the refresh it starts with, says to log in again, and exits 3.
|
||||
@@ -619,6 +693,190 @@ describe("quak backup", () => {
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
});
|
||||
|
||||
// The server refuses every connection, at its one address or at each of
|
||||
// several. quak tries each request again, here without waiting, and then
|
||||
// the run stops at the refresh it starts with. The line it prints gives
|
||||
// the reason Node's `fetch` put on the error's cause, so a cron log tells
|
||||
// a refused connection from a name that did not resolve, and the exit
|
||||
// code is 1.
|
||||
it.each(REFUSED)(
|
||||
"prints why the refresh could not reach the server at $addresses, and exits 1",
|
||||
async ({ error, reason }) => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const fetch: typeof globalThis.fetch = async () => {
|
||||
throw error();
|
||||
};
|
||||
|
||||
const { code, stderr } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toBe(
|
||||
"Starting backup...\n" +
|
||||
"Refreshing library...\n" +
|
||||
`quak: ${reason}\n`,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
// The files CDN refuses every connection for toast.mp4, at its one
|
||||
// address or at each of several. Once its retries are spent, the file
|
||||
// goes into `failures.json` with the reason, as a transient failure,
|
||||
// since a later run may well reach the CDN, and the lines the backup
|
||||
// prints for it give the same reason. The rest of the account is backed
|
||||
// up, and the exit code is 1.
|
||||
it.each(REFUSED)(
|
||||
"records why a download could not reach the server at $addresses in failures.json, as transient, and exits 1",
|
||||
async ({ error, reason }) => {
|
||||
let attempts = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (downloadOf(input) === 200) {
|
||||
attempts++;
|
||||
throw error();
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code, stderr } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(Object.keys(failures.files)).toEqual(["200"]);
|
||||
expect(failures.files["200"]).toMatchObject({
|
||||
fileID: 200,
|
||||
title: "toast.mp4",
|
||||
classification: "transient",
|
||||
attempts: 1,
|
||||
error: reason,
|
||||
});
|
||||
expect(stderr).toContain(`FAILED original toast.mp4: ${reason}\n`);
|
||||
expect(stderr).toContain(
|
||||
`[Friend's Wedding] toast.mp4 (id 200): ${reason}\n`,
|
||||
);
|
||||
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
|
||||
Buffer.from(JPEG_WITH_EXIF),
|
||||
);
|
||||
expect(existsSync(join(dir, DAY, "2024-03-01.200.mp4"))).toBe(
|
||||
false,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
// The download of toast.mp4 fails as bun's `fetch` reports a failure, in
|
||||
// the binary `make install` builds: a plain `Error` with bun's own code
|
||||
// and message, and no cause. These are what bun gave for a refused
|
||||
// connection, a name that did not resolve and a reset connection. The
|
||||
// file is tried as often as under Node, and goes into `failures.json`
|
||||
// with that message, as a transient failure.
|
||||
it.each([
|
||||
[
|
||||
"ConnectionRefused",
|
||||
"Unable to connect. Is the computer able to access the url?",
|
||||
],
|
||||
["FailedToOpenSocket", "Was there a typo in the url or port?"],
|
||||
[
|
||||
"ECONNRESET",
|
||||
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
|
||||
],
|
||||
])(
|
||||
"records a download that failed with bun's %s in failures.json, as transient",
|
||||
async (bunCode, message) => {
|
||||
let attempts = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (downloadOf(input) === 200) {
|
||||
attempts++;
|
||||
throw Object.assign(new Error(message), {
|
||||
code: bunCode,
|
||||
errno: 0,
|
||||
});
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(failures.files["200"]).toMatchObject({
|
||||
classification: "transient",
|
||||
error: message,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
// The download of toast.mp4 stalls: the server sends the headers and then
|
||||
// never a byte of the body. Each attempt is given up after 50 ms, quak's
|
||||
// own deadline for this run, with a `TimeoutError` whose message says
|
||||
// only that the download stalled. The file goes into `failures.json` as a
|
||||
// transient failure, as a request that ran out of time is retried.
|
||||
it("records a stalled download in failures.json, as transient", async () => {
|
||||
let attempts = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (downloadOf(input) === 200) {
|
||||
attempts++;
|
||||
return new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
pull: () => new Promise<void>(() => {}),
|
||||
}),
|
||||
);
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code } = await backup(dir, {}, fetch, 50);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(failures.files["200"]).toMatchObject({
|
||||
classification: "transient",
|
||||
error: "download stalled: no bytes received for 50 ms",
|
||||
});
|
||||
});
|
||||
|
||||
// The name of Ente's API server does not resolve when quak fetches the ML
|
||||
// data, after every original has been backed up. No file has ML data
|
||||
// cached yet, so each goes into `failures.json` with the reason, as a
|
||||
// transient failure, as a download that could not reach the server does,
|
||||
// and the exit code is 1.
|
||||
it("records why the ML data fetch could not reach the server in failures.json, as transient, and exits 1", async () => {
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (new URL(String(input)).pathname === "/files/data/fetch") {
|
||||
throw new TypeError("fetch failed", {
|
||||
cause: Object.assign(
|
||||
new Error("getaddrinfo ENOTFOUND api.ente.io"),
|
||||
{ code: "ENOTFOUND" },
|
||||
),
|
||||
});
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code, stderr } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
const error =
|
||||
"ML data: fetch failed: getaddrinfo ENOTFOUND api.ente.io";
|
||||
expect(stderr).toContain(`FAILED ${error}\n`);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(
|
||||
Object.values(failures.files).map((f) => [
|
||||
f.fileID,
|
||||
f.classification,
|
||||
f.error,
|
||||
]),
|
||||
).toEqual([
|
||||
[100, "transient", error],
|
||||
[200, "transient", error],
|
||||
[300, "transient", error],
|
||||
]);
|
||||
});
|
||||
|
||||
// Something changed beach.jpg on disk after it was backed up. A run with
|
||||
// --verify hashes every stored original, finds that one no longer matches
|
||||
// the hash Ente recorded, says so, and downloads it again; it put back the
|
||||
@@ -704,10 +962,11 @@ describe("a backup whose lock is lost", () => {
|
||||
}
|
||||
};
|
||||
|
||||
// beach.jpg, whose download was under way, is stored; no other file is
|
||||
// started. `quak backup` prints why as one line, with no stack trace, and
|
||||
// exits 1.
|
||||
it("stops quak backup before its next file when backup.lock is removed, and exits 1", async () => {
|
||||
// The backup downloads up to five originals at once, so it has started
|
||||
// all three when the lock is lost, and beach.jpg is stored. The backup
|
||||
// then stops before it fetches ML data. `quak backup` prints why as one
|
||||
// line, with no stack trace, and exits 1.
|
||||
it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => {
|
||||
const result = backup(dir, {}, fetch);
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
@@ -720,9 +979,11 @@ describe("a backup whose lock is lost", () => {
|
||||
"Starting backup...\n" +
|
||||
"Refreshing library...\n" +
|
||||
"Fetching original beach.jpg (100)...\n" +
|
||||
"Fetching original IMG_0002.HEIC (300)...\n" +
|
||||
"Fetching original toast.mp4 (200)...\n" +
|
||||
`quak: lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'\n`,
|
||||
);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
|
||||
Buffer.from(JPEG_WITH_EXIF),
|
||||
);
|
||||
@@ -798,7 +1059,7 @@ describe("a backup whose lock is lost", () => {
|
||||
await expect(result).rejects.toThrow(
|
||||
`lost the lock on ${dir}: Unable to update lock within the stale threshold`,
|
||||
);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
expect(existsSync(lockPath)).toBe(true);
|
||||
} finally {
|
||||
await lib.close();
|
||||
@@ -823,7 +1084,7 @@ describe("a backup whose lock is lost", () => {
|
||||
const lost = `lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'`;
|
||||
await expect(result).rejects.toThrow(lost);
|
||||
expect(lock.lost?.message).toBe(lost);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
await lock.release();
|
||||
|
||||
+354
-21
@@ -191,9 +191,12 @@ class HostileClient extends MockClient {
|
||||
}
|
||||
|
||||
// A content source that writes byte buffers of the expected length and can be
|
||||
// told to fail one fileID's original, to exercise per-file resilience.
|
||||
// told to fail one fileID's original, as a server error or as a 404, or to
|
||||
// write other bytes for it, to exercise per-file resilience.
|
||||
interface StubSource extends ContentSource {
|
||||
failID?: number;
|
||||
notFoundID?: number;
|
||||
otherBytesID?: number;
|
||||
failThumbID?: number;
|
||||
originalCalls: number;
|
||||
}
|
||||
@@ -204,8 +207,10 @@ const stubSource = (): StubSource => {
|
||||
original: async ({ file: f, destination }) => {
|
||||
s.originalCalls++;
|
||||
if (s.failID === f.id) throw new Error("HTTP 500 from server");
|
||||
if (s.notFoundID === f.id) throw new Error("HTTP 404 from server");
|
||||
const size = SIZE_BY_ID[f.id] ?? 10;
|
||||
writeFileSync(destination, Buffer.alloc(size));
|
||||
const fill = s.otherBytesID === f.id ? 2 : 0;
|
||||
writeFileSync(destination, Buffer.alloc(size, fill));
|
||||
return { bytesWritten: size };
|
||||
},
|
||||
thumbnail: async ({ file: f, destination }) => {
|
||||
@@ -696,17 +701,26 @@ describe("lib.backup", () => {
|
||||
// A child that has already exited: its process ID is not running.
|
||||
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
const leftover = `.quak-backup-2026-03-01.100.jpg-${exitedPID}-abc123.tmp`;
|
||||
// A live photo's image and video downloaded to replace a stored one,
|
||||
// before they were renamed into place.
|
||||
const replacement = [
|
||||
`.quak-${exitedPID}-abc789.heic`,
|
||||
`.quak-${exitedPID}-abc789.mov`,
|
||||
];
|
||||
// This test's own process stands in for a backup running at the same
|
||||
// time.
|
||||
const inProgress = `.quak-backup-2026-03-01.101.jpg-${process.pid}-def456.tmp`;
|
||||
writeFileSync(join(day, leftover), "partial");
|
||||
writeFileSync(join(day, inProgress), "partial");
|
||||
for (const name of [leftover, ...replacement, inProgress]) {
|
||||
writeFileSync(join(day, name), "partial");
|
||||
}
|
||||
const lib = await openLibrary(stubSource());
|
||||
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const names = readdirSync(day);
|
||||
expect(names).not.toContain(leftover);
|
||||
expect(names).not.toContain(replacement[0]);
|
||||
expect(names).not.toContain(replacement[1]);
|
||||
expect(names).toContain(inProgress);
|
||||
lib.close();
|
||||
});
|
||||
@@ -909,6 +923,27 @@ describe("the backup lock", () => {
|
||||
expect(existsSync(lockPath(outDir))).toBe(false);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// `runBackup` takes a library object from its caller. With a
|
||||
// `contentConcurrency` below 1, or one that is not a number, it would
|
||||
// back up no original and report no error. It rejects any that is not a
|
||||
// positive integer, as `BoundedPool` does, before it takes the lock,
|
||||
// which would create the download directory, and before it asks the
|
||||
// library for anything: the object here has nothing else.
|
||||
it("rejects a library whose contentConcurrency is not a positive integer, before taking the lock", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
for (const contentConcurrency of [undefined, 0, -1, 1.5, NaN]) {
|
||||
const lib = { contentConcurrency } as unknown as BackupLibrary;
|
||||
|
||||
await expect(
|
||||
runBackup(lib, { downloadDirectory: outDir }),
|
||||
).rejects.toThrow(
|
||||
`contentConcurrency must be a positive integer, got ${contentConcurrency}`,
|
||||
);
|
||||
|
||||
expect(existsSync(outDir)).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
|
||||
@@ -1273,7 +1308,8 @@ describe("image metadata in each file's JSON", () => {
|
||||
});
|
||||
|
||||
// With `verify`, each original already stored is hashed, and one that does not
|
||||
// match the content hash its metadata records is downloaded again.
|
||||
// match the content hash its metadata records is downloaded again, and replaced
|
||||
// only by a download that matches.
|
||||
describe("backup with verify", () => {
|
||||
// MockClient's files, each recording the hash of what `stubSource` writes
|
||||
// for it, except diagram.png (200), which records none.
|
||||
@@ -1316,6 +1352,12 @@ describe("backup with verify", () => {
|
||||
const good = Buffer.alloc(SIZE_BY_ID[100]!);
|
||||
const corrupt = Buffer.alloc(SIZE_BY_ID[100]!, 1);
|
||||
|
||||
// The temporary files left in the date folder of the backup in `outDir`.
|
||||
const tempFiles = (outDir: string): string[] =>
|
||||
readdirSync(join(outDir, DAY)).filter((name) =>
|
||||
name.startsWith(".quak-"),
|
||||
);
|
||||
|
||||
it("leaves an original that matches its hash, and one with no hash, as they are", async () => {
|
||||
const { lib, source, outDir } = await backedUp();
|
||||
const calls = source.originalCalls;
|
||||
@@ -1360,13 +1402,16 @@ describe("backup with verify", () => {
|
||||
expect(log.filter((msg) => msg.startsWith("MISMATCH"))).toEqual([
|
||||
"MISMATCH original beach.jpg (100): its bytes do not match its content hash",
|
||||
]);
|
||||
expect(tempFiles(outDir)).toEqual([]);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("records a failed download in failures.json, with the original removed", async () => {
|
||||
// A copy whose hash does not match is still mostly the photo, so the
|
||||
// backup keeps it until it holds a better one.
|
||||
it("keeps an original that does not match its hash when its download answers 404", async () => {
|
||||
const { lib, source, outDir } = await backedUp();
|
||||
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||
source.failID = 100;
|
||||
source.notFoundID = 100;
|
||||
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
@@ -1379,18 +1424,18 @@ describe("backup with verify", () => {
|
||||
failed: 1,
|
||||
});
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
|
||||
expect(existsSync(saved(outDir, "100.jpg"))).toBe(false);
|
||||
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
|
||||
expect(
|
||||
readlinkSync(join(outDir, "collections", "Vacation", "beach.jpg")),
|
||||
).toBe(linkTo("100.jpg"));
|
||||
expect(tempFiles(outDir)).toEqual([]);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("records in failures.json an original the cache puts back with the same bad bytes", async () => {
|
||||
const lib = await openLibrary(stubSource(), new HashedClient());
|
||||
const outDir = join(root, "backup");
|
||||
// The cache holds a bad copy, and a backup copies an original the
|
||||
// cache holds to its save path.
|
||||
const cached = await lib.photos.byID({ fileID: 100 })!.original();
|
||||
writeFileSync(cached.path, corrupt);
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
it("keeps an original that does not match its hash when its download does not match either", async () => {
|
||||
const { lib, source, outDir } = await backedUp();
|
||||
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||
source.otherBytesID = 100;
|
||||
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
@@ -1403,10 +1448,39 @@ describe("backup with verify", () => {
|
||||
failed: 1,
|
||||
});
|
||||
expect(result.errors.map((e) => e.error)).toEqual([
|
||||
"the original put back does not match its content hash either",
|
||||
"the downloaded original does not match its content hash either",
|
||||
]);
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
|
||||
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
|
||||
expect(tempFiles(outDir)).toEqual([]);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// The cache can hold the same bad bytes, as a copy from it is not checked
|
||||
// as a download is, so the replacement is always downloaded.
|
||||
it("downloads the replacement even when the cache holds the original", async () => {
|
||||
const source = stubSource();
|
||||
const lib = await openLibrary(source, new HashedClient());
|
||||
const outDir = join(root, "backup");
|
||||
// The cache holds a bad copy, and a backup copies an original the
|
||||
// cache holds to its save path.
|
||||
const cached = await lib.photos.byID({ fileID: 100 })!.original();
|
||||
writeFileSync(cached.path, corrupt);
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
const calls = source.originalCalls;
|
||||
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
verify: true,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
downloaded: 1,
|
||||
mismatched: 1,
|
||||
failed: 0,
|
||||
});
|
||||
expect(source.originalCalls).toBe(calls + 1);
|
||||
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(good);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
@@ -1477,20 +1551,26 @@ describe("backup album folders", () => {
|
||||
files: EnteFile[];
|
||||
}
|
||||
|
||||
const libraryOf = (albums: Album[]): BackupLibrary => ({
|
||||
// The original of the file `notFound`, if given, always fails to download.
|
||||
const libraryOf = (albums: Album[], notFound?: number): BackupLibrary => ({
|
||||
refresh: async () => {},
|
||||
listCollections: () => albums.map((a) => a.collection),
|
||||
listFiles: (id) =>
|
||||
albums.find((a) => a.collection.id === id)?.files ?? [],
|
||||
undecryptable: () => [],
|
||||
original: async (fileID) => {
|
||||
if (fileID === notFound) throw new Error("HTTP 404");
|
||||
const path = join(root, `source-${fileID}`);
|
||||
writeFileSync(path, `original ${fileID}`);
|
||||
return { path };
|
||||
},
|
||||
fetchOriginal: async () => {
|
||||
throw new Error("no verify in this stand-in");
|
||||
},
|
||||
thumbnail: async () => {
|
||||
throw new Error("no thumbnails in this stand-in");
|
||||
},
|
||||
contentConcurrency: 5,
|
||||
fetchMLData: async () => {},
|
||||
mlData: async () => undefined,
|
||||
whoami: () => ({ email: "backup@example.com", userID: USER_ID }),
|
||||
@@ -1578,6 +1658,252 @@ describe("backup album folders", () => {
|
||||
expect(albumID(outDir, "Trip (11) (12).json")).toBe(12);
|
||||
});
|
||||
|
||||
// The next three tests end with the album "ok" (14), whose one file
|
||||
// always fails to download, so each run has a failure to record in
|
||||
// failures.json after the album folders are written.
|
||||
const ok: Album = {
|
||||
collection: collection(14, "ok"),
|
||||
files: [file(5, 14, "e.jpg")],
|
||||
};
|
||||
|
||||
// The folder of an album named "x.json" would be the JSON file of the
|
||||
// album "x", and once ended every run there. Two runs, as on two nights,
|
||||
// write the same tree, and the failed file's attempt count goes up.
|
||||
it("adds the ID to an album whose folder would have another album's JSON name", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf(
|
||||
[
|
||||
{
|
||||
collection: collection(10, "x"),
|
||||
files: [file(1, 10, "a.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(11, "x.json"),
|
||||
files: [file(2, 11, "b.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(12, "Trip"),
|
||||
files: [file(3, 12, "c.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(13, "TRIP.json"),
|
||||
files: [file(4, 13, "d.jpg")],
|
||||
},
|
||||
ok,
|
||||
],
|
||||
5,
|
||||
);
|
||||
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(result.failedAlbums).toEqual([]);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"TRIP.json (13)/",
|
||||
`TRIP.json (13)/d.jpg -> ${linkTo("4.jpg")}`,
|
||||
"TRIP.json (13).json",
|
||||
"Trip/",
|
||||
`Trip/c.jpg -> ${linkTo("3.jpg")}`,
|
||||
"Trip.json",
|
||||
"ok/",
|
||||
"ok.json",
|
||||
"x/",
|
||||
`x/a.jpg -> ${linkTo("1.jpg")}`,
|
||||
"x.json",
|
||||
"x.json (11)/",
|
||||
`x.json (11)/b.jpg -> ${linkTo("2.jpg")}`,
|
||||
"x.json (11).json",
|
||||
]);
|
||||
expect(albumID(outDir, "x.json")).toBe(10);
|
||||
expect(albumID(outDir, "x.json (11).json")).toBe(11);
|
||||
expect(albumID(outDir, "TRIP.json (13).json")).toBe(13);
|
||||
expect(albumID(outDir, "ok.json")).toBe(14);
|
||||
expect(result.failed).toBe(1);
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 2 });
|
||||
});
|
||||
|
||||
// A name over 255 bytes is refused as a file name, and an album folder's
|
||||
// name leaves room for `.json` in its JSON's. "é" takes two bytes, and a
|
||||
// name is cut between two characters.
|
||||
it("cuts an album name and a title too long for a file name short and adds the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf(
|
||||
[
|
||||
{
|
||||
collection: collection(10, "L".repeat(300)),
|
||||
files: [file(1, 10, `${"é".repeat(150)}.jpg`)],
|
||||
},
|
||||
ok,
|
||||
],
|
||||
5,
|
||||
);
|
||||
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const folder = `${"L".repeat(245)} (10)`;
|
||||
const link = `${"é".repeat(123)} (1).jpg`;
|
||||
expect(Buffer.byteLength(`${folder}.json`)).toBe(255);
|
||||
expect(Buffer.byteLength(link)).toBe(254);
|
||||
expect(result.failedAlbums).toEqual([]);
|
||||
expect(tree(outDir)).toEqual([
|
||||
`${folder}/`,
|
||||
`${folder}/${link} -> ${linkTo("1.jpg")}`,
|
||||
`${folder}.json`,
|
||||
"ok/",
|
||||
"ok.json",
|
||||
]);
|
||||
expect(albumID(outDir, `${folder}.json`)).toBe(10);
|
||||
expect(albumID(outDir, "ok.json")).toBe(14);
|
||||
expect(result.failed).toBe(1);
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 2 });
|
||||
});
|
||||
|
||||
// The user has put a file in collections/ with the name of the album
|
||||
// Trip's folder, so the folder cannot be made.
|
||||
it("goes on to the next album when an album's folder cannot be made, and lists it as failed", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf(
|
||||
[
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [file(1, 10, "a.jpg")],
|
||||
},
|
||||
ok,
|
||||
],
|
||||
5,
|
||||
);
|
||||
mkdirSync(join(outDir, "collections"), { recursive: true });
|
||||
writeFileSync(join(outDir, "collections", "Trip"), "mine");
|
||||
const logged: string[] = [];
|
||||
|
||||
const result = await runBackup(lib, {
|
||||
downloadDirectory: outDir,
|
||||
onProgress: (message) => logged.push(message),
|
||||
});
|
||||
|
||||
expect(result.failedAlbums).toEqual([10]);
|
||||
expect(logged).toContainEqual(
|
||||
expect.stringMatching(/^FAILED album Trip: EEXIST: /),
|
||||
);
|
||||
expect(tree(outDir)).toEqual(["Trip", "ok/", "ok.json"]);
|
||||
expect(albumID(outDir, "ok.json")).toBe(14);
|
||||
expect(existsSync(saved(outDir, "1.jpg"))).toBe(true);
|
||||
expect(result.failed).toBe(1);
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
|
||||
});
|
||||
|
||||
// The extensions of these titles leave no room for the ID, so each is cut
|
||||
// short and the name before the ID keeps only the bytes that cut leaves
|
||||
// over. File 1's extension is 303 bytes and is cut to 251, which leaves
|
||||
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three
|
||||
// bytes, which leaves one, for the "a". Neither extension is only letters
|
||||
// and digits, so both originals are saved with `.bin`, a name that fits.
|
||||
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(1, 10, `a.${"L".repeat(300)}-x`),
|
||||
file(2, 10, `a.${"€".repeat(90)}`),
|
||||
],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const link = ` (1).${"L".repeat(250)}`;
|
||||
const link2 = `a (2).${"€".repeat(83)}`;
|
||||
expect(Buffer.byteLength(link)).toBe(255);
|
||||
expect(Buffer.byteLength(link2)).toBe(255);
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip/",
|
||||
`Trip/${link} -> ${linkTo("1.bin")}`,
|
||||
`Trip/${link2} -> ${linkTo("2.bin")}`,
|
||||
"Trip.json",
|
||||
]);
|
||||
});
|
||||
|
||||
// Files 1 and 3 share a title, so each gets its ID before the
|
||||
// extension. File 2's title is too long, and its extension leaves no
|
||||
// room for the ID. Were the ID put at the end of file 2's title cut
|
||||
// short, its link would have file 1's name, and the link made second
|
||||
// would replace the first.
|
||||
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const shared = `x.${"Q".repeat(245)} (2)`;
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(1, 10, shared),
|
||||
file(2, 10, `x (1).${"Q".repeat(250)}-`),
|
||||
file(3, 10, shared),
|
||||
],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip/",
|
||||
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
|
||||
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
|
||||
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
|
||||
"Trip.json",
|
||||
]);
|
||||
});
|
||||
|
||||
// A title of 255 bytes is the longest file name allowed, and an album
|
||||
// name of 250 bytes the longest whose JSON name fits; both are kept as
|
||||
// they are. A name one byte longer gets the ID and is cut short. "é"
|
||||
// takes two bytes, and in both names the room left ends inside one, so
|
||||
// the cut falls before it, one byte short of the room.
|
||||
it("keeps a name at the byte limit and adds the ID to one a byte over it", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const album = "A".repeat(250);
|
||||
const title = `${"t".repeat(251)}.jpg`;
|
||||
const longAlbum = `${"é".repeat(125)}B`;
|
||||
const longTitle = `${"é".repeat(126)}.jpg`;
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, album),
|
||||
files: [file(1, 10, title), file(2, 10, longTitle)],
|
||||
},
|
||||
{
|
||||
collection: collection(11, longAlbum),
|
||||
files: [file(3, 11, "c.jpg")],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const folder = `${"é".repeat(122)} (11)`;
|
||||
const link = `${"é".repeat(123)} (2).jpg`;
|
||||
expect(Buffer.byteLength(`${album}.json`)).toBe(255);
|
||||
expect(Buffer.byteLength(title)).toBe(255);
|
||||
expect(Buffer.byteLength(longAlbum)).toBe(251);
|
||||
expect(Buffer.byteLength(longTitle)).toBe(256);
|
||||
expect(Buffer.byteLength(`${folder}.json`)).toBe(254);
|
||||
expect(Buffer.byteLength(link)).toBe(254);
|
||||
expect(result.failed).toBe(0);
|
||||
expect(result.failedAlbums).toEqual([]);
|
||||
expect(tree(outDir)).toEqual([
|
||||
`${album}/`,
|
||||
`${album}/${title} -> ${linkTo("1.jpg")}`,
|
||||
`${album}/${link} -> ${linkTo("2.jpg")}`,
|
||||
`${album}.json`,
|
||||
`${folder}/`,
|
||||
`${folder}/c.jpg -> ${linkTo("3.jpg")}`,
|
||||
`${folder}.json`,
|
||||
]);
|
||||
expect(albumID(outDir, `${album}.json`)).toBe(10);
|
||||
expect(albumID(outDir, `${folder}.json`)).toBe(11);
|
||||
});
|
||||
|
||||
it("changes nothing on a second run over an unchanged account", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
@@ -1974,11 +2300,12 @@ describe("backup of live photos", () => {
|
||||
Buffer.from(IMAGE),
|
||||
);
|
||||
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
|
||||
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
|
||||
expect(tree(outDir)).toEqual(linked);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("removes both files of a live photo that does not match its hash when downloading it again fails", async () => {
|
||||
it("keeps both files of a live photo that does not match its hash when downloading it again fails", async () => {
|
||||
const { file: live, body } = await asLivePhoto(
|
||||
file(500, 10, "IMG_0500.HEIC"),
|
||||
);
|
||||
@@ -1996,8 +2323,14 @@ describe("backup of live photos", () => {
|
||||
|
||||
expect(result).toMatchObject({ mismatched: 1, failed: 1 });
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
|
||||
expect(existsSync(saved(outDir, "500.heic"))).toBe(false);
|
||||
expect(existsSync(saved(outDir, "500.mov"))).toBe(false);
|
||||
expect(readFileSync(saved(outDir, "500.heic"))).toEqual(
|
||||
Buffer.from(IMAGE),
|
||||
);
|
||||
expect(readFileSync(saved(outDir, "500.mov"), "utf-8")).toBe(
|
||||
"another few seconds of video",
|
||||
);
|
||||
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
|
||||
expect(tree(outDir)).toEqual(linked);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
|
||||
@@ -48,6 +48,78 @@ describe("run", () => {
|
||||
);
|
||||
});
|
||||
|
||||
// Node's `fetch` rejects every failure below HTTP with a TypeError that
|
||||
// says only "fetch failed", and hangs the reason, with its errno, off
|
||||
// `cause`. The printed line carries the reason, so a cron log tells a
|
||||
// refused connection from a name that did not resolve.
|
||||
it("prints the reason a fetch failed, from the error's cause", async () => {
|
||||
const refused = new TypeError("fetch failed", {
|
||||
cause: Object.assign(
|
||||
new Error("connect ECONNREFUSED 1.2.3.4:443"),
|
||||
{ code: "ECONNREFUSED" },
|
||||
),
|
||||
});
|
||||
const result = await runToExit(Promise.reject(refused));
|
||||
expect(result).toEqual({
|
||||
code: 1,
|
||||
stdout: "",
|
||||
stderr: "quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443\n",
|
||||
});
|
||||
});
|
||||
|
||||
// When the host has several addresses and every one refuses, Node's
|
||||
// `fetch` gives as the cause an AggregateError with an empty message and
|
||||
// one error per address. The printed line gives each address's reason.
|
||||
it("prints the reason for each address a fetch could not reach", async () => {
|
||||
const refusedAt = (address: string): Error =>
|
||||
Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
|
||||
code: "ECONNREFUSED",
|
||||
});
|
||||
const refused = new TypeError("fetch failed", {
|
||||
cause: Object.assign(
|
||||
new AggregateError([
|
||||
refusedAt("2001:db8::1:443"),
|
||||
refusedAt("1.2.3.4:443"),
|
||||
]),
|
||||
{ code: "ECONNREFUSED" },
|
||||
),
|
||||
});
|
||||
const result = await runToExit(Promise.reject(refused));
|
||||
expect(result).toEqual({
|
||||
code: 1,
|
||||
stdout: "",
|
||||
stderr: "quak: fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443\n",
|
||||
});
|
||||
});
|
||||
|
||||
// An error that already quotes its cause's message is printed as it is.
|
||||
it("does not print a cause's message twice", async () => {
|
||||
const reset = new Error("read ECONNRESET");
|
||||
const wrapped = new Error(`download: file 7: ${reset.message}`, {
|
||||
cause: reset,
|
||||
});
|
||||
const result = await runToExit(Promise.reject(wrapped));
|
||||
expect(result.stderr).toBe("quak: download: file 7: read ECONNRESET\n");
|
||||
});
|
||||
|
||||
// OpenSSL ends its message for a failed TLS connection, such as one to a
|
||||
// server that answers in plain HTTP, with a newline. The printed line is
|
||||
// still one line.
|
||||
it("prints a cause's message without the white space around it", async () => {
|
||||
const wrongVersion = new TypeError("fetch failed", {
|
||||
cause: Object.assign(
|
||||
new Error(
|
||||
"C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
|
||||
),
|
||||
{ code: "ERR_SSL_WRONG_VERSION_NUMBER" },
|
||||
),
|
||||
});
|
||||
const result = await runToExit(Promise.reject(wrongVersion));
|
||||
expect(result.stderr).toBe(
|
||||
"quak: fetch failed: C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints a thrown value that is not an Error", async () => {
|
||||
const result = await runToExit(Promise.reject("offline"));
|
||||
expect(result).toEqual({
|
||||
@@ -57,6 +129,14 @@ describe("run", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// A cause that is not an Error, such as a thrown string, is printed as a
|
||||
// thrown value that is not an Error is.
|
||||
it("prints a cause that is not an Error", async () => {
|
||||
const wrapped = new Error("ML data", { cause: "offline" });
|
||||
const result = await runToExit(Promise.reject(wrapped));
|
||||
expect(result.stderr).toBe("quak: ML data: offline\n");
|
||||
});
|
||||
|
||||
it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
|
||||
const result = await runToExit(
|
||||
Promise.reject(new ApiError("unauthorized", 401)),
|
||||
|
||||
@@ -74,7 +74,11 @@ import {
|
||||
} from "vitest";
|
||||
import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js";
|
||||
import { ApiClient } from "../../src/api/client.js";
|
||||
import { ApiError, TruncatedStreamError } from "../../src/errors.js";
|
||||
import {
|
||||
ApiError,
|
||||
describeError,
|
||||
TruncatedStreamError,
|
||||
} from "../../src/errors.js";
|
||||
import type { RetryOptions } from "../../src/retry.js";
|
||||
import {
|
||||
downloadFile,
|
||||
@@ -834,6 +838,11 @@ describe.each(entryPoints)(
|
||||
expect(((err as Error).cause as Error).message).toMatch(
|
||||
/authentication failed/i,
|
||||
);
|
||||
// As quak prints it and records it in `failures.json`, the error
|
||||
// states the authentication failure once, from its cause.
|
||||
expect(describeError(err)).toMatch(
|
||||
/^download: stream truncated: response body ended with \d+ trailing bytes \(transfer stopped mid-chunk, or the data is corrupt\): secretstream chunk authentication failed$/,
|
||||
);
|
||||
|
||||
// And, as with every other failure, the destination is untouched
|
||||
// and no staged temp file survives.
|
||||
|
||||
@@ -14,7 +14,11 @@
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { safeExtension, sanitizeFileName } from "../../src/filename.js";
|
||||
import {
|
||||
safeExtension,
|
||||
sanitizeFileName,
|
||||
shortenToBytes,
|
||||
} from "../../src/filename.js";
|
||||
|
||||
const FALLBACK = "file-42";
|
||||
|
||||
@@ -91,3 +95,26 @@ describe("safeExtension", () => {
|
||||
expect(safeExtension("x.jpg\0")).toBe(".bin");
|
||||
});
|
||||
});
|
||||
|
||||
describe("shortenToBytes", () => {
|
||||
it("returns a name that fits unchanged", () => {
|
||||
expect(shortenToBytes("beach", 5)).toBe("beach");
|
||||
expect(shortenToBytes("café", 5)).toBe("café");
|
||||
expect(shortenToBytes("", 0)).toBe("");
|
||||
});
|
||||
|
||||
it("cuts a name to the number of bytes given", () => {
|
||||
expect(shortenToBytes("L".repeat(300), 255)).toBe("L".repeat(255));
|
||||
});
|
||||
|
||||
// In UTF-8 "é" takes two bytes and an emoji four, which a JavaScript
|
||||
// string holds as two halves.
|
||||
it("cuts between two characters, never inside one", () => {
|
||||
expect(shortenToBytes("ééé", 5)).toBe("éé");
|
||||
expect(shortenToBytes("a\u{1F600}b", 4)).toBe("a");
|
||||
expect(shortenToBytes("a\u{1F600}b", 5)).toBe("a\u{1F600}");
|
||||
expect(Buffer.byteLength(shortenToBytes("é".repeat(200), 255))).toBe(
|
||||
254,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -77,6 +77,14 @@ const recordingSleep = (): {
|
||||
const errnoError = (code: string, message = code): Error =>
|
||||
Object.assign(new Error(message), { code });
|
||||
|
||||
/**
|
||||
* An error shaped like what bun's `fetch` rejects with for a failure below
|
||||
* HTTP: a plain `Error` with bun's own message, a `code`, an `errno` of 0, and
|
||||
* no `cause`.
|
||||
*/
|
||||
const bunError = (code: string, message: string): Error =>
|
||||
Object.assign(new Error(message), { code, errno: 0 });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Classification
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -244,6 +252,82 @@ describe("isRetryable: transport failures", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("isRetryable and isSafeToReplay: bun's fetch errors", () => {
|
||||
// The binary `make install` builds runs quak under bun, whose `fetch`
|
||||
// does not reject as Node's does. Each shape here was found by running
|
||||
// that binary, built with bun 1.3.13, against listeners on 127.0.0.1 and
|
||||
// a name under `.invalid`.
|
||||
|
||||
it("retries a refused connection, and does not replay it", () => {
|
||||
// Bun gives the same error when the TLS handshake fails, and when a
|
||||
// new connection it opened to resend a request is refused. That
|
||||
// request may already have reached the server on a connection bun
|
||||
// kept open from an earlier request.
|
||||
const refused = bunError(
|
||||
"ConnectionRefused",
|
||||
"Unable to connect. Is the computer able to access the url?",
|
||||
);
|
||||
expect(isRetryable(refused)).toBe(true);
|
||||
expect(isSafeToReplay(refused)).toBe(false);
|
||||
});
|
||||
|
||||
it("retries a name that does not resolve, with either code bun gives, and does not replay it", () => {
|
||||
// For the same name bun gave `ConnectionRefused` on some requests and
|
||||
// `FailedToOpenSocket` on others. Either can also come from a new
|
||||
// connection bun opened to resend a request, as above.
|
||||
for (const unresolved of [
|
||||
bunError(
|
||||
"ConnectionRefused",
|
||||
"Unable to connect. Is the computer able to access the url?",
|
||||
),
|
||||
bunError(
|
||||
"FailedToOpenSocket",
|
||||
"Was there a typo in the url or port?",
|
||||
),
|
||||
]) {
|
||||
expect(isRetryable(unresolved)).toBe(true);
|
||||
expect(isSafeToReplay(unresolved)).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("retries a connection closed before the response ended, and does not replay it", () => {
|
||||
// Bun gives Node's errno, `ECONNRESET`, for a reset and for a close,
|
||||
// before the response or part-way through the body. The request may
|
||||
// already have reached the server.
|
||||
const reset = bunError(
|
||||
"ECONNRESET",
|
||||
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
|
||||
);
|
||||
expect(isRetryable(reset)).toBe(true);
|
||||
expect(isSafeToReplay(reset)).toBe(false);
|
||||
});
|
||||
|
||||
it("retries a request that ran out of time, and does not replay it", () => {
|
||||
// What bun's fetch rejects with when `AbortSignal.timeout()`, which
|
||||
// bounds each JSON request, fires.
|
||||
const timedOut = new DOMException(
|
||||
"The operation timed out.",
|
||||
"TimeoutError",
|
||||
);
|
||||
expect(isRetryable(timedOut)).toBe(true);
|
||||
expect(isSafeToReplay(timedOut)).toBe(false);
|
||||
});
|
||||
|
||||
it("does not retry a certificate bun rejects", () => {
|
||||
// A certificate that fails verification fails the same way however
|
||||
// often it is asked. Under Node it is retried only because every
|
||||
// `TypeError` is.
|
||||
expect(
|
||||
isRetryable(
|
||||
bunError(
|
||||
"DEPTH_ZERO_SELF_SIGNED_CERT",
|
||||
"self signed certificate",
|
||||
),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("isRetryable: stream truncation versus corruption", () => {
|
||||
it("retries a truncated stream", () => {
|
||||
// Truncation is a transfer that stopped early. The bytes that did
|
||||
|
||||
Reference in New Issue
Block a user