Compare commits
7
Commits
5176b18779
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
22ee2db479 | ||
|
clawbot
|
e8e8e00887 | ||
|
clawbot
|
bb3b5cd2da | ||
|
clawbot
|
288825b11a | ||
|
clawbot
|
54133cfa5f | ||
|
clawbot
|
ab28d86aaa | ||
|
clawbot
|
2402ec97f7 |
@@ -128,18 +128,20 @@ matters.
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `0` | The backup is complete: every original is at its save path, and no file failed. |
|
||||
| `1` | The run finished with files in `failures.json`, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||
| `1` | The run finished with files in `failures.json` or an album it could not decrypt, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||
|
||||
After a `1`, the next run fetches each missing original again, fetches the ML
|
||||
data that is not cached, and rebuilds the album links. An original that a
|
||||
`--verify` run could not read, or put back with bytes that still do not match,
|
||||
stays at its save path. Only a later `--verify` run checks it again: a run
|
||||
without `--verify` leaves it as it is and takes the file out of `failures.json`,
|
||||
so that run can exit 0.
|
||||
data that is not cached, and rebuilds the album links. An album quak could not
|
||||
decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
|
||||
the exit code at `1` until the server sends a copy that decrypts, or deletes it
|
||||
(see "Backup layout"). An original that a `--verify` run could not read, or put
|
||||
back with bytes that still do not match, stays at its save path. Only a later
|
||||
`--verify` run checks it again: a run without `--verify` leaves it as it is and
|
||||
takes the file out of `failures.json`, so that run can exit 0.
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -623,8 +625,17 @@ library. The read commands — `collections`, `files`, `get`, `get-thumb`,
|
||||
answer, so they report current account state rather than whatever the cache last
|
||||
held. If that round-trip fails, the command prints the error on one line and
|
||||
exits 1, or 3 when the server no longer accepts the saved session (see "Session
|
||||
handling"). `--cache-dir` overrides where the cache lives; without it each
|
||||
account gets its own directory under the per-user cache path.
|
||||
handling"). An album or file quak cannot decrypt does not fail it; the command
|
||||
answers from what did decrypt (see "Default reads vs. fresh reads").
|
||||
`--cache-dir` overrides where the cache lives; without it each account gets its
|
||||
own directory under the per-user cache path.
|
||||
|
||||
An ID given to `files --collection`, `get`, `get-thumb` or
|
||||
`helper fix-missing-thumbnails --file` is written in decimal digits, no larger
|
||||
than 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused
|
||||
before any request with one line, `Invalid file ID: <arg>` or
|
||||
`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses
|
||||
the whole command, so nothing is fixed.
|
||||
|
||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
||||
@@ -734,12 +745,25 @@ succeeds, or once it is no longer in the library or in the backup's scope. The
|
||||
library's `lib.backup({ includeThumbnails: true })` also writes
|
||||
`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not.
|
||||
|
||||
A file in the backup's scope that the library could not decrypt (see "Default
|
||||
reads vs. fresh reads") goes into `failures.json` with an error that starts
|
||||
`could not be decrypted:` and gives the reason. If a copy of it in another album
|
||||
decrypted, it does not: its original is stored from that copy, and the album
|
||||
where it does not decrypt neither links nor lists it. An album the library could
|
||||
not decrypt has no name quak can read and no files it can list. A run without
|
||||
`--json` prints one line for it,
|
||||
`FAILED album <id>: could not be decrypted: <reason>`, and `--json` and the
|
||||
`BackupResult` list its ID in `undecryptableAlbums`. Such a file in
|
||||
`failures.json`, or such an album, keeps the exit code at 1, run after run,
|
||||
until the server sends a copy that decrypts or deletes it; the next run then
|
||||
drops it.
|
||||
|
||||
`backup.lock` keeps two backups of the same directory from running at once, such
|
||||
as a cron run that starts while the previous one is still going. A backup
|
||||
creates `<dir>` if it is missing and takes the lock before its refresh, and
|
||||
removes the lock when it ends, whether it succeeds or fails; `quak backup` takes
|
||||
it before it opens its library, so a refused run sends no request. The lock is a
|
||||
directory that
|
||||
removes the lock when it ends, whether it succeeds or fails, unless it lost the
|
||||
lock (see below); `quak backup` takes it before it opens its library, so a
|
||||
refused run sends no request. The lock is a directory that
|
||||
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates
|
||||
and keeps touching while the backup runs. A second backup of the directory, from
|
||||
another process or the same one, fails at once: `quak backup` prints
|
||||
@@ -751,6 +775,16 @@ over, so nobody has to remove it. The lock is outside the date folders and
|
||||
`collections/`, so it is never taken for an original, and the removal of old
|
||||
album directories never touches it.
|
||||
|
||||
A running backup loses its lock when someone removes `backup.lock`, or when
|
||||
another backup takes it over because the running one left it untouched for 10
|
||||
seconds, as a stopped or stalled process does. The backup finds this out the
|
||||
next time it goes to touch the lock, which it does every 5 seconds, and then
|
||||
stops before it starts another file. The original it is downloading at that
|
||||
moment is still stored. It leaves `failures.json` and `backup.lock` as they are,
|
||||
since another backup may be using them. `quak backup` prints
|
||||
`quak: lost the lock on <dir>: <reason>` as one line and exits 1, and
|
||||
`lib.backup()` rejects with `lost the lock on <dir>: <reason>`.
|
||||
|
||||
A collection's directory and JSON are named after the collection, and a symlink
|
||||
after the file's title, both with unsafe characters replaced. When two
|
||||
collections would get the same name, or two symlinks in one collection the same
|
||||
@@ -770,9 +804,12 @@ both, each named after the title with that file's extension (`IMG_0001.heic` and
|
||||
|
||||
Each run removes the symlinks into the date folders that no longer belong in
|
||||
their collection's directory, and the directories (and JSON) of collections that
|
||||
were deleted or renamed. Nothing else in `collections/` is touched: a file or a
|
||||
symlink you put there stays, and a directory that still holds one after its
|
||||
symlinks are removed stays too, with its JSON.
|
||||
were deleted or renamed. An album quak can no longer decrypt loses its directory
|
||||
and JSON the same way, and a file whose copy in an album quak can no longer
|
||||
decrypt loses its symlink there and its entry in that album's JSON; the
|
||||
originals stay at their save paths. Nothing else in `collections/` is touched: a
|
||||
file or a symlink you put there stays, and a directory that still holds one
|
||||
after its symlinks are removed stays too, with its JSON.
|
||||
|
||||
Each file is downloaded exactly once regardless of how many collections it
|
||||
appears in, and written once: straight to its save path, with no copy left in
|
||||
@@ -781,7 +818,8 @@ subsequent runs, existing originals are skipped. If a download fails, the error
|
||||
is logged and the backup continues with the next file. The exit code is non-zero
|
||||
if any files failed. `quak backup` opens its library with the thumbnail and
|
||||
originals precache off, so the only file content it fetches is the originals the
|
||||
backup stores.
|
||||
backup stores. It also opens it with no background timer, so it asks the server
|
||||
for changes only as it starts, not again while it downloads.
|
||||
|
||||
With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each
|
||||
original already at its save path the way a download is checked (see "On-disk
|
||||
@@ -843,7 +881,8 @@ The library's primary surface is the cache-backed `Library`; the lower-level
|
||||
`Client` sits underneath it and is covered by the Design sections above. The
|
||||
test suite is the canonical, executable documentation — `test/library/` and
|
||||
`test/client/usage.test.ts` walk most operations, `test/cli/backup.test.ts`
|
||||
walks `lib.backup()`, and `yarn test` verifies them.
|
||||
walks `lib.backup()`, `test/cli/backup-end-to-end.test.ts` runs `quak backup`
|
||||
against a fake of the Ente HTTP API, and `yarn test` verifies them.
|
||||
|
||||
### Opening a library
|
||||
|
||||
@@ -861,7 +900,7 @@ background, so an unreachable server does not block opening.
|
||||
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
||||
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
|
||||
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached |
|
||||
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
||||
| `refreshIntervalSeconds` | `3` | background refresh cadence; `Infinity` for no background timer |
|
||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||
@@ -906,6 +945,16 @@ rejects the caller (default reads stay silent and keep serving the last good
|
||||
copy). The CLI's read commands use fresh reads (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/75).
|
||||
|
||||
An album or file in the server's answer that quak cannot decrypt, or whose
|
||||
metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
|
||||
album with all its files, and drops any older copy the cache held, so reads
|
||||
answer from what did decrypt. The library keeps a list of what it left out, with
|
||||
the reason, in `metadata.json`: the next refresh asks the server only for what
|
||||
changed, so it would not see the record again. A record stays on that list until
|
||||
a later refresh gets a copy that decrypts or learns that the server deleted it.
|
||||
`lib.backup()` reports each such album, and each such file in its scope that has
|
||||
no copy that decrypted (see "Backup layout").
|
||||
|
||||
### Read surface
|
||||
|
||||
- `lib.albums.list()` → `Album[]`, newest-updated first.
|
||||
@@ -1021,21 +1070,24 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
directory, and fails at once with an error whose `code` is `ELOCKED` while
|
||||
another backup of it runs. A caller can instead take the lock itself, before
|
||||
it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)`
|
||||
takes it, failing the same way, and returns the function that releases it, and
|
||||
the caller passes `lockHeld: true` to the backup. It waits for a refresh as
|
||||
`fresh()` does, puts every in-scope original not already at its save path
|
||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
||||
rebuilds the on-disk backup tree with a durable failure ledger. Each file's
|
||||
JSON holds its ML data and, for an image (for a live photo, its image), its
|
||||
original's EXIF and XMP, with its dimensions for a JPEG only; a video's JSON
|
||||
holds none of these three. A fetched original is written straight to its save
|
||||
path and not into the cache, which then counts it as present; one the cache
|
||||
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
|
||||
back to the library's), `includeOriginals` (default `true`),
|
||||
`includeThumbnails` (default `false`), `onlyAlbumNames`, `verify` (default
|
||||
`false`), `onProgress`, and `lockHeld` (default `false`). See Backup layout
|
||||
above for the tree it writes.
|
||||
takes it, failing the same way, and returns a `BackupLock`, whose `release()`
|
||||
removes the lock unless it was lost and whose `lost` holds why it was lost.
|
||||
The caller passes it to the backup as `lock`, and releases it once the backup
|
||||
ends. It waits for a refresh as `fresh()` does, puts every in-scope original
|
||||
not already at its save path there as `photo.download()` does (and, with
|
||||
`includeThumbnails`, fetches thumbnails) through the content cache, waits for
|
||||
an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
|
||||
ledger. Each file's JSON holds its ML data and, for an image (for a live
|
||||
photo, its image), its original's EXIF and XMP, with its dimensions for a JPEG
|
||||
only; a video's JSON holds none of these three. A fetched original is written
|
||||
straight to its save path and not into the cache, which then counts it as
|
||||
present; one the cache already held is copied from there. `BackupOptions`:
|
||||
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
|
||||
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
|
||||
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
|
||||
tree it writes. Whether the backup took the lock or its caller did, once the
|
||||
lock is lost the backup rejects with `lost the lock on <dir>: <reason>` before
|
||||
it starts another file.
|
||||
|
||||
### Request pools
|
||||
|
||||
@@ -1095,7 +1147,7 @@ from a very old client, is stored unchecked.
|
||||
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
||||
- `src/exif.ts`: `ExifTags`, `PhotoExif`
|
||||
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
||||
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
||||
- `src/backup.ts`: `BackupLock`, `BackupOptions`, `BackupResult`, `BackupError`
|
||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions`
|
||||
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
|
||||
|
||||
@@ -17,14 +17,72 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
None: no implementation work is open. The cache design,
|
||||
https://git.eeqj.de/sneak/quak/issues/36, waits on sneak's review.
|
||||
`quak backup` downloading one original at a time,
|
||||
https://git.eeqj.de/sneak/quak/issues/182, is fixed in
|
||||
https://git.eeqj.de/sneak/quak/pulls/192, which waits on review. No other
|
||||
implementation work is open. The check of quak against a live Ente account,
|
||||
https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak has, so
|
||||
it waits on him.
|
||||
|
||||
Tagging and releases are decided by sneak alone, and happen only when he
|
||||
declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
|
||||
review, though that issue closed on 2026-10-03 (issue 194). It now names the
|
||||
issues open on the tracker.
|
||||
|
||||
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
|
||||
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
|
||||
that records why the lock was lost instead of throwing from its timer, and
|
||||
returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
|
||||
backup then stops before it starts another file with
|
||||
`lost the lock on DIR: REASON`, which `quak backup` prints as one line before
|
||||
it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
|
||||
`test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
|
||||
in its place, while a download is held open.
|
||||
|
||||
- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
|
||||
JSON, no longer fails the refresh, so one bad record no longer stops
|
||||
`quak backup` from storing anything (issue 181). The refresh leaves it out, an
|
||||
album with its files, and the library keeps it with the reason in
|
||||
`metadata.json` until the server sends a copy that decrypts or deletes it.
|
||||
`quak backup` stores every other file, records each such file in
|
||||
`failures.json` with an error starting `could not be decrypted:`, names each
|
||||
such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
|
||||
and exits 1 while any remain. A file whose copy in another album decrypts is
|
||||
not recorded: its original is stored from that copy, and the album where it
|
||||
does not decrypt neither links nor lists it. The read commands answer from
|
||||
what decrypted.
|
||||
|
||||
- 2026-10-14: `quak backup` no longer asks the server for changes every 3
|
||||
seconds while it runs (issue 183). It opens its library with
|
||||
`refreshIntervalSeconds: Infinity`, which the library now takes as no
|
||||
background timer, so the backup refreshes only as it starts and no timed
|
||||
refresh starts another ML data pass during the run. A test holds a backup's
|
||||
download open while two hours pass and counts its `/collections/v2` requests.
|
||||
|
||||
- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
|
||||
does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
|
||||
session saved after logging in, and every request answered by a fake of the
|
||||
Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
|
||||
of an account with two albums of its own and one shared with it, a second run
|
||||
that downloads nothing, a download that succeeds after two 503s, a 404 that
|
||||
lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
|
||||
`--verify` putting back an original changed on disk. The fake server and the
|
||||
account it logs in moved from the backup-metadata test into
|
||||
`test/ente-server.ts`, unchanged.
|
||||
|
||||
- 2026-10-14: `files --collection`, `get`, `get-thumb` and
|
||||
`helper fix-missing-thumbnails --file` take an ID only when it is written in
|
||||
decimal digits and is a safe integer (issue 185). Before, `Number()` read
|
||||
`0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
|
||||
ID it could not read without a word. Anything else now prints
|
||||
`Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
|
||||
any request; one bad `--file` ID fails the whole command. `parseID` in
|
||||
`src/cli-commands.ts` does the check for all four.
|
||||
|
||||
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
|
||||
log in once as the job's user, a crontab with a backup every night and
|
||||
`--verify` on Sundays appending to a log file, and the `HOME` and
|
||||
|
||||
+122
-39
@@ -1,8 +1,9 @@
|
||||
// The backup command, rebuilt on the library API (issue #51).
|
||||
//
|
||||
// `lib.backup()` takes the lock in `downloadDirectory` (unless its caller
|
||||
// already holds it), and fails at once when another backup of it holds the
|
||||
// lock. It waits for a completed refresh of the library (a failed one fails the
|
||||
// passes one in), and fails at once when another backup of it holds the
|
||||
// lock; when the lock is lost while it runs, it fails before it starts another
|
||||
// file. It waits for a completed refresh of the library (a failed one fails the
|
||||
// backup before any file is touched), then, for every file in scope, puts its
|
||||
// original at its save path under `downloadDirectory`, as `Photo.download()`
|
||||
// does, waits for an ML data fetch, and rebuilds the derived views (per-file
|
||||
@@ -47,6 +48,9 @@
|
||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
||||
// failed is caught, recorded in `failures.json` with a classification, a
|
||||
// running attempt count, and the last-tried time, and the run continues.
|
||||
// A file the library could not decrypt is recorded there too, unless a copy of
|
||||
// it in another album decrypted; an album it could not decrypt is logged by ID
|
||||
// and listed in `result.undecryptableAlbums`, as its name is not known.
|
||||
// `result.failed` — and thus the CLI's exit code — stays non-zero while any
|
||||
// failure remains unresolved and clears once every one succeeds. Each run
|
||||
// reconciles the ledger against the files it attempted, so an entry for a file
|
||||
@@ -86,12 +90,21 @@ import {
|
||||
storedAtSavePath,
|
||||
} from "./library/content.js";
|
||||
import { representative } from "./library/records.js";
|
||||
import type { Undecryptable } from "./library/store.js";
|
||||
import { extractImageMetadata } from "./metadata-backup.js";
|
||||
import type { MLData } from "./mldata-fetch.js";
|
||||
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
||||
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
// The lock in a backup directory, as `lockBackupDirectory` returns it.
|
||||
export interface BackupLock {
|
||||
// Removes `backup.lock`, unless the lock was lost.
|
||||
release(): Promise<void>;
|
||||
// Why the lock was lost, once it is.
|
||||
lost?: Error;
|
||||
}
|
||||
|
||||
export interface BackupOptions {
|
||||
// Where the backup tree lives. `lib.backup()` defaults it to the library's
|
||||
// download directory; `runBackup` with none throws before any network
|
||||
@@ -108,10 +121,11 @@ export interface BackupOptions {
|
||||
// bytes do not match the content hash its metadata records. Default false.
|
||||
verify?: boolean;
|
||||
onProgress?: ProgressCallback;
|
||||
// The caller already holds the lock in `downloadDirectory`, taken with
|
||||
// `lockBackupDirectory`, and releases it itself, so the backup does not
|
||||
// take it. `quak backup` takes it before it opens its library.
|
||||
lockHeld?: boolean;
|
||||
// The lock in `downloadDirectory`, when the caller took it with
|
||||
// `lockBackupDirectory` and releases it itself. The backup then does not
|
||||
// take the lock, and fails once this one is lost. `quak backup` takes it
|
||||
// before it opens its library.
|
||||
lock?: BackupLock;
|
||||
}
|
||||
|
||||
export interface BackupError {
|
||||
@@ -141,6 +155,9 @@ export interface BackupResult {
|
||||
failed: number;
|
||||
// This run's per-file errors, in encounter order.
|
||||
errors: BackupError[];
|
||||
// The IDs of the albums the library could not decrypt, whose files it
|
||||
// could not list; the CLI exits non-zero while there are any.
|
||||
undecryptableAlbums: number[];
|
||||
}
|
||||
|
||||
// The slice of the library that backup drives. `Library` implements it; a test
|
||||
@@ -151,6 +168,9 @@ export interface BackupLibrary {
|
||||
refresh(): Promise<void>;
|
||||
listCollections(): Collection[];
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
// The albums, and files in albums, that the library could not decrypt,
|
||||
// which `listCollections` and `listFiles` leave out.
|
||||
undecryptable(): Undecryptable[];
|
||||
// Get an original's bytes onto disk through the content cache/pools,
|
||||
// returning where they landed: `destination` when they were fetched now,
|
||||
// otherwise wherever they already were (the cache, or the library's save
|
||||
@@ -500,11 +520,14 @@ const writeAlbumJSON = (
|
||||
writeFileSync(path, JSON.stringify(album, null, 2));
|
||||
};
|
||||
|
||||
// The backup itself, which `runBackup` below runs while the lock is held.
|
||||
// The backup itself, which `runBackup` below runs while `lock` is held. Once
|
||||
// the lock is lost, another backup may be writing to `downloadDirectory`, so
|
||||
// this one fails before it starts another file.
|
||||
const runLockedBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
downloadDirectory: string,
|
||||
lock: BackupLock,
|
||||
): Promise<BackupResult> => {
|
||||
const includeOriginals = opts.includeOriginals ?? true;
|
||||
const includeThumbnails = opts.includeThumbnails ?? false;
|
||||
@@ -514,6 +537,7 @@ const runLockedBackup = async (
|
||||
|
||||
log("Refreshing library...");
|
||||
await lib.refresh();
|
||||
if (lock.lost) throw lock.lost;
|
||||
|
||||
const collectionsDir = join(downloadDirectory, "collections");
|
||||
const thumbnailsDir = join(downloadDirectory, "thumbnails");
|
||||
@@ -574,33 +598,56 @@ const runLockedBackup = async (
|
||||
let unchecked = 0;
|
||||
|
||||
const recordFailure = (
|
||||
file: EnteFile,
|
||||
fileID: number,
|
||||
title: string,
|
||||
collection: string,
|
||||
err: unknown,
|
||||
): void => {
|
||||
// Count at most one attempt per file per run: a file whose original
|
||||
// and thumbnail both fail this run must not double its attempt count
|
||||
// or appear twice in errors.
|
||||
if (failedThisRun.has(file.id)) return;
|
||||
if (failedThisRun.has(fileID)) return;
|
||||
const error = errorMessage(err);
|
||||
errors.push({
|
||||
fileID: file.id,
|
||||
title: file.metadata.title,
|
||||
collection,
|
||||
error,
|
||||
});
|
||||
const prior = ledger.get(file.id);
|
||||
ledger.set(file.id, {
|
||||
fileID: file.id,
|
||||
title: file.metadata.title,
|
||||
errors.push({ fileID, title, collection, error });
|
||||
const prior = ledger.get(fileID);
|
||||
ledger.set(fileID, {
|
||||
fileID,
|
||||
title,
|
||||
classification: classify(err),
|
||||
attempts: (prior?.attempts ?? 0) + 1,
|
||||
lastTriedAt: now,
|
||||
error,
|
||||
});
|
||||
failedThisRun.add(file.id);
|
||||
failedThisRun.add(fileID);
|
||||
};
|
||||
|
||||
// What the library could not decrypt. An album is logged by its ID, as
|
||||
// its name is not known, so a run limited to some albums logs it too. A
|
||||
// file in an album in scope is backed up from a copy of it in another
|
||||
// album that decrypted, even an album out of scope. With no such copy it
|
||||
// is recorded as failed, with an empty title as its metadata is not known.
|
||||
const scope = new Set(collections.map((c) => c.id));
|
||||
const undecryptableAlbums: number[] = [];
|
||||
for (const record of lib.undecryptable()) {
|
||||
const error = `could not be decrypted: ${record.error}`;
|
||||
if (record.fileID === undefined) {
|
||||
log(`FAILED album ${record.collectionID}: ${error}`);
|
||||
undecryptableAlbums.push(record.collectionID);
|
||||
} else if (scope.has(record.collectionID)) {
|
||||
const copies = memberships.get(record.fileID);
|
||||
if (copies) {
|
||||
distinct.set(record.fileID, representative(copies));
|
||||
} else {
|
||||
recordFailure(
|
||||
record.fileID,
|
||||
"",
|
||||
collectionName.get(record.collectionID) ?? "",
|
||||
new Error(error),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 1: get the bytes. Put each pending original at its save path
|
||||
// through the content cache/pools, as `Photo.download()` does, and fetch
|
||||
// the optional thumbnails; a present file is left as is. With `verify`, a
|
||||
@@ -609,6 +656,7 @@ const runLockedBackup = async (
|
||||
// that cannot be read is recorded as failed and left where it is.
|
||||
if (includeOriginals) {
|
||||
for (const [fileID, file] of distinct) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
let stored = storedAtSavePath(downloadDirectory, file);
|
||||
let mismatch = false;
|
||||
if (stored !== undefined && verify) {
|
||||
@@ -636,7 +684,8 @@ const runLockedBackup = async (
|
||||
`FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
file,
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
err,
|
||||
);
|
||||
@@ -676,7 +725,8 @@ const runLockedBackup = async (
|
||||
`FAILED original ${file.metadata.title}: ${errorMessage(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
file,
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
err,
|
||||
);
|
||||
@@ -686,6 +736,7 @@ const runLockedBackup = async (
|
||||
|
||||
if (includeThumbnails) {
|
||||
for (const [fileID, file] of distinct) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const dest = join(thumbnailsDir, `${fileID}.jpg`);
|
||||
if (isPresent(dest)) continue;
|
||||
try {
|
||||
@@ -693,7 +744,8 @@ const runLockedBackup = async (
|
||||
await copyAtomic(path, dest);
|
||||
} catch (err) {
|
||||
recordFailure(
|
||||
file,
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
err,
|
||||
);
|
||||
@@ -717,6 +769,7 @@ const runLockedBackup = async (
|
||||
log(`FAILED ML data: ${mlDataError}`);
|
||||
}
|
||||
for (const file of distinct.values()) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const stored = storedAtSavePath(downloadDirectory, file);
|
||||
if (stored === undefined) continue;
|
||||
const path = withExtension(
|
||||
@@ -733,7 +786,8 @@ const runLockedBackup = async (
|
||||
if (mlData === undefined && mlDataError !== undefined) {
|
||||
writeSidecar(path, file, { mlDataError }, image);
|
||||
recordFailure(
|
||||
file,
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
new Error(`ML data: ${mlDataError}`),
|
||||
);
|
||||
@@ -759,6 +813,7 @@ const runLockedBackup = async (
|
||||
const albumDirNames = new Map(
|
||||
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
||||
);
|
||||
if (lock.lost) throw lock.lost;
|
||||
try {
|
||||
removeStaleAlbumDirs(
|
||||
collectionsDir,
|
||||
@@ -770,6 +825,7 @@ const runLockedBackup = async (
|
||||
}
|
||||
|
||||
for (const c of collections) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const colDirName = albumDirNames.get(c.id)!;
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
@@ -806,7 +862,12 @@ const runLockedBackup = async (
|
||||
log(
|
||||
`FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`,
|
||||
);
|
||||
recordFailure(link.file, c.name, err);
|
||||
recordFailure(
|
||||
link.file.id,
|
||||
link.file.metadata.title,
|
||||
c.name,
|
||||
err,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -822,7 +883,9 @@ const runLockedBackup = async (
|
||||
// its failure resolved; a file gone from the library (deleted) or outside
|
||||
// this run's scope is not something this run can resolve, so keeping its
|
||||
// stale entry would keep the exit code non-zero forever — a single
|
||||
// since-deleted photo would fail every future scheduled backup.
|
||||
// since-deleted photo would fail every future scheduled backup. A backup
|
||||
// whose lock was lost leaves the ledger as it is.
|
||||
if (lock.lost) throw lock.lost;
|
||||
for (const fileID of [...ledger.keys()]) {
|
||||
if (!failedThisRun.has(fileID)) ledger.delete(fileID);
|
||||
}
|
||||
@@ -837,23 +900,42 @@ const runLockedBackup = async (
|
||||
unchecked,
|
||||
failed: ledger.size,
|
||||
errors,
|
||||
undecryptableAlbums,
|
||||
};
|
||||
};
|
||||
|
||||
// Only one backup of a directory runs at a time, in this process or another.
|
||||
// Creates `downloadDirectory` if it is missing, takes the lock in it and
|
||||
// returns the function that releases it; while another backup holds the lock,
|
||||
// fails at once with an error whose `code` is `ELOCKED`. The lock is the
|
||||
// directory `backup.lock`, whose modification time proper-lockfile keeps
|
||||
// current while it is held. One it has not touched for 10 seconds was left by
|
||||
// a run that could not remove it, and is taken over.
|
||||
// returns it; while another backup holds the lock, fails at once with an error
|
||||
// whose `code` is `ELOCKED`. The lock is the directory `backup.lock`, whose
|
||||
// modification time proper-lockfile keeps current while it is held. One it has
|
||||
// not touched for 10 seconds was left by a run that could not remove it, and
|
||||
// is taken over.
|
||||
//
|
||||
// proper-lockfile finds the lock lost when `backup.lock` is gone or is another
|
||||
// backup's, and calls `onCompromised` from a timer, where throwing would end
|
||||
// the process. The `onCompromised` given here sets the lock's `lost` instead,
|
||||
// and the backup holding the lock stops before it starts another file (see
|
||||
// `runLockedBackup`). Releasing a lost lock leaves `backup.lock` alone, since
|
||||
// it may be another backup's.
|
||||
export const lockBackupDirectory = async (
|
||||
downloadDirectory: string,
|
||||
): Promise<() => Promise<void>> => {
|
||||
): Promise<BackupLock> => {
|
||||
mkdirSync(downloadDirectory, { recursive: true });
|
||||
let release: () => Promise<void>;
|
||||
const lock: BackupLock = {
|
||||
release: async () => {
|
||||
if (lock.lost === undefined) await release();
|
||||
},
|
||||
};
|
||||
try {
|
||||
return await lockfile.lock(downloadDirectory, {
|
||||
release = await lockfile.lock(downloadDirectory, {
|
||||
lockfilePath: join(downloadDirectory, "backup.lock"),
|
||||
onCompromised: (err) => {
|
||||
lock.lost = new Error(
|
||||
`lost the lock on ${downloadDirectory}: ${err.message}`,
|
||||
);
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
@@ -862,10 +944,11 @@ export const lockBackupDirectory = async (
|
||||
{ code: "ELOCKED" },
|
||||
);
|
||||
}
|
||||
return lock;
|
||||
};
|
||||
|
||||
// Runs the backup holding the lock, which it takes and releases itself unless
|
||||
// the caller already holds it (`opts.lockHeld`).
|
||||
// the caller passes the lock it took (`opts.lock`).
|
||||
export const runBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
@@ -877,13 +960,13 @@ export const runBackup = async (
|
||||
"open the library with one)",
|
||||
);
|
||||
}
|
||||
if (opts.lockHeld) {
|
||||
return runLockedBackup(lib, opts, downloadDirectory);
|
||||
if (opts.lock) {
|
||||
return runLockedBackup(lib, opts, downloadDirectory, opts.lock);
|
||||
}
|
||||
const release = await lockBackupDirectory(downloadDirectory);
|
||||
const lock = await lockBackupDirectory(downloadDirectory);
|
||||
try {
|
||||
return await runLockedBackup(lib, opts, downloadDirectory);
|
||||
return await runLockedBackup(lib, opts, downloadDirectory, lock);
|
||||
} finally {
|
||||
await release();
|
||||
await lock.release();
|
||||
}
|
||||
};
|
||||
|
||||
+39
-25
@@ -20,7 +20,7 @@ import {
|
||||
type ClientSnapshot,
|
||||
type LoginOptions,
|
||||
} from "./client.js";
|
||||
import { lockBackupDirectory } from "./backup.js";
|
||||
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
||||
import { init } from "./crypto/index.js";
|
||||
import {
|
||||
defaultCacheDirectory,
|
||||
@@ -97,6 +97,20 @@ const requireSession = (ctx: CliContext): Client | undefined => {
|
||||
return client;
|
||||
};
|
||||
|
||||
// `text` as an ID, or undefined after telling the user it is not one. An ID is
|
||||
// written in decimal digits and is a safe integer; `Number()` alone would read
|
||||
// `0x10` as 16, `1e3` as 1000 and "" as 0.
|
||||
const parseID = (
|
||||
ctx: CliContext,
|
||||
kind: "file" | "collection",
|
||||
text: string,
|
||||
): number | undefined => {
|
||||
const id = Number(text);
|
||||
if (/^[0-9]+$/.test(text) && Number.isSafeInteger(id)) return id;
|
||||
ctx.stderr.write(`Invalid ${kind} ID: ${text}\n`);
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// A library client that omits `fetchMLData`, so the point commands below do not
|
||||
// kick the library's background ML backfill: they read metadata, or fetch one
|
||||
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
|
||||
@@ -247,11 +261,8 @@ export const filesCommand = async (
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
const collectionID = Number(opts.collection);
|
||||
if (!Number.isFinite(collectionID)) {
|
||||
ctx.stderr.write("Invalid collection ID\n");
|
||||
return 1;
|
||||
}
|
||||
const collectionID = parseID(ctx, "collection", opts.collection);
|
||||
if (collectionID === undefined) return 1;
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
@@ -289,11 +300,8 @@ export const getCommand = async (
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
return 1;
|
||||
}
|
||||
const fileID = parseID(ctx, "file", fileIDStr);
|
||||
if (fileID === undefined) return 1;
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
@@ -347,11 +355,8 @@ export const getThumbCommand = async (
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
return 1;
|
||||
}
|
||||
const fileID = parseID(ctx, "file", fileIDStr);
|
||||
if (fileID === undefined) return 1;
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
@@ -411,9 +416,9 @@ export const backupCommand = async (
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
// The lock is taken before the library opens and starts its refresh, so a
|
||||
// run refused while another backup of `dir` runs sends no request.
|
||||
let release: () => Promise<void>;
|
||||
let lock: BackupLock;
|
||||
try {
|
||||
release = await lockBackupDirectory(dir);
|
||||
lock = await lockBackupDirectory(dir);
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
ctx.stderr.write(`quak: ${(err as Error).message}\n`);
|
||||
@@ -422,17 +427,20 @@ export const backupCommand = async (
|
||||
try {
|
||||
// The precache is off: the backup fetches what it needs, and must not
|
||||
// also fill the cache with every thumbnail and the recent originals.
|
||||
// No refresh starts on a timer: the backup lists its files once, from
|
||||
// the refresh it waits for, and can run for hours.
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
lockHeld: true,
|
||||
lock,
|
||||
verify: opts.verify,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
@@ -462,12 +470,14 @@ export const backupCommand = async (
|
||||
}
|
||||
}
|
||||
|
||||
return result.failed > 0 ? 1 : 0;
|
||||
return result.failed > 0 || result.undecryptableAlbums.length > 0
|
||||
? 1
|
||||
: 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
} finally {
|
||||
await release();
|
||||
await lock.release();
|
||||
}
|
||||
};
|
||||
|
||||
@@ -516,15 +526,19 @@ export const fixMissingThumbnailsCommand = async (
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
// One bad --file ID fails the whole command, before anything is fetched.
|
||||
let fileIDs: number[] = [];
|
||||
for (const text of opts.file ?? []) {
|
||||
const fileID = parseID(ctx, "file", text);
|
||||
if (fileID === undefined) return 1;
|
||||
fileIDs.push(fileID);
|
||||
}
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Refresh first so files added since the cache was written are found;
|
||||
// a failed refresh throws.
|
||||
await lib.fresh();
|
||||
let fileIDs: number[];
|
||||
if (opts.file && opts.file.length > 0) {
|
||||
fileIDs = opts.file.map(Number).filter(Number.isFinite);
|
||||
} else {
|
||||
if (fileIDs.length === 0) {
|
||||
ctx.stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
|
||||
+41
-14
@@ -44,20 +44,28 @@ export interface ClientSnapshot {
|
||||
|
||||
// The result of a resumable enumeration. Live decrypted records and deleted
|
||||
// ids are kept apart on purpose: a tombstone carries no key or metadata to
|
||||
// decrypt, so it is a bare id rather than a hollowed-out record. `cursor` is
|
||||
// the max `updationTime` seen, to pass back into the next call.
|
||||
// decrypt, so it is a bare id rather than a hollowed-out record. A live row
|
||||
// that cannot be decrypted, or whose metadata cannot be parsed, is in
|
||||
// `undecryptable` with the reason instead, so one bad row does not fail the
|
||||
// call; absent means there was none. `cursor` is the max `updationTime` seen,
|
||||
// those rows included, to pass back into the next call.
|
||||
export interface CollectionsPage {
|
||||
collections: Collection[];
|
||||
deleted: number[];
|
||||
undecryptable?: { id: number; error: string }[];
|
||||
cursor: number;
|
||||
}
|
||||
|
||||
export interface FilesPage {
|
||||
files: EnteFile[];
|
||||
deleted: number[];
|
||||
undecryptable?: { id: number; error: string }[];
|
||||
cursor: number;
|
||||
}
|
||||
|
||||
const errorMessage = (err: unknown): string =>
|
||||
err instanceof Error ? err.message : String(err);
|
||||
|
||||
export class Client {
|
||||
private readonly api: ApiClient;
|
||||
private readonly email: string;
|
||||
@@ -237,11 +245,12 @@ export class Client {
|
||||
}
|
||||
|
||||
// Enumerate collections changed since `sinceTime`. Live collections are
|
||||
// decrypted; tombstoned ones (isDeleted) are surfaced as bare ids. The
|
||||
// returned cursor is the max `updationTime` seen — including tombstones, so
|
||||
// the next sync resumes past them — falling back to `sinceTime` when the
|
||||
// response is empty. `/collections/v2` returns the whole changed set in one
|
||||
// response, so there is no pagination here.
|
||||
// decrypted; tombstoned ones (isDeleted) are surfaced as bare ids, and
|
||||
// ones that do not decrypt as ids with the reason. The returned cursor is
|
||||
// the max `updationTime` seen — including tombstones, so the next sync
|
||||
// resumes past them — falling back to `sinceTime` when the response is
|
||||
// empty. `/collections/v2` returns the whole changed set in one response,
|
||||
// so there is no pagination here.
|
||||
async collectionsSince(args: {
|
||||
sinceTime: number;
|
||||
}): Promise<CollectionsPage> {
|
||||
@@ -254,11 +263,13 @@ export class Client {
|
||||
|
||||
const collections: Collection[] = [];
|
||||
const deleted: number[] = [];
|
||||
const undecryptable: { id: number; error: string }[] = [];
|
||||
let cursor = args.sinceTime;
|
||||
for (const raw of raws) {
|
||||
if (raw.isDeleted) {
|
||||
deleted.push(raw.id);
|
||||
} else {
|
||||
try {
|
||||
collections.push(
|
||||
decryptCollection(
|
||||
raw,
|
||||
@@ -270,15 +281,22 @@ export class Client {
|
||||
this.userID,
|
||||
),
|
||||
);
|
||||
} catch (err) {
|
||||
undecryptable.push({
|
||||
id: raw.id,
|
||||
error: errorMessage(err),
|
||||
});
|
||||
}
|
||||
}
|
||||
if (raw.updationTime > cursor) cursor = raw.updationTime;
|
||||
}
|
||||
return { collections, deleted, cursor };
|
||||
return { collections, deleted, undecryptable, cursor };
|
||||
}
|
||||
|
||||
// Enumerate a collection's files changed since `sinceTime`, paginating the
|
||||
// diff from that cursor. Live rows are decrypted; tombstoned ones are
|
||||
// surfaced as bare ids. Returns the final cursor to resume from.
|
||||
// surfaced as bare ids, and ones that do not decrypt or parse as ids with
|
||||
// the reason. Returns the final cursor to resume from.
|
||||
async filesSince(args: {
|
||||
collectionID: number;
|
||||
collectionKey: Uint8Array;
|
||||
@@ -288,6 +306,7 @@ export class Client {
|
||||
const { collectionID, collectionKey } = args;
|
||||
const files: EnteFile[] = [];
|
||||
const deleted: number[] = [];
|
||||
const undecryptable: { id: number; error: string }[] = [];
|
||||
let cursor = args.sinceTime;
|
||||
for (;;) {
|
||||
const { diff, hasMore } = await this.api.getJSON<{
|
||||
@@ -300,7 +319,14 @@ export class Client {
|
||||
if (raw.isDeleted) {
|
||||
deleted.push(raw.id);
|
||||
} else {
|
||||
try {
|
||||
files.push(decryptFile(raw, collectionKey));
|
||||
} catch (err) {
|
||||
undecryptable.push({
|
||||
id: raw.id,
|
||||
error: errorMessage(err),
|
||||
});
|
||||
}
|
||||
}
|
||||
if (raw.updationTime > pageMax) pageMax = raw.updationTime;
|
||||
}
|
||||
@@ -321,18 +347,19 @@ export class Client {
|
||||
}
|
||||
cursor = pageMax;
|
||||
}
|
||||
return { files, deleted, cursor };
|
||||
return { files, deleted, undecryptable, cursor };
|
||||
}
|
||||
|
||||
// Whole-account listing: every live collection, deletions hidden. A thin
|
||||
// wrapper over `collectionsSince` from the beginning of time.
|
||||
// Whole-account listing: every live collection that decrypts, deletions
|
||||
// hidden. A thin wrapper over `collectionsSince` from the beginning of
|
||||
// time.
|
||||
async listCollections(): Promise<Collection[]> {
|
||||
const { collections } = await this.collectionsSince({ sinceTime: 0 });
|
||||
return collections;
|
||||
}
|
||||
|
||||
// Every live file in a collection, deletions hidden. A thin wrapper over
|
||||
// `filesSince` from the beginning of time.
|
||||
// Every live file in a collection that decrypts, deletions hidden. A thin
|
||||
// wrapper over `filesSince` from the beginning of time.
|
||||
async listFiles(
|
||||
collectionID: number,
|
||||
collectionKey: Uint8Array,
|
||||
|
||||
@@ -69,6 +69,7 @@ export {
|
||||
type EnsureEvent,
|
||||
lockBackupDirectory,
|
||||
runBackup,
|
||||
type BackupLock,
|
||||
type BackupOptions,
|
||||
type BackupResult,
|
||||
type BackupError,
|
||||
|
||||
+36
-5
@@ -96,6 +96,7 @@ import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
|
||||
export {
|
||||
lockBackupDirectory,
|
||||
runBackup,
|
||||
type BackupLock,
|
||||
type BackupOptions,
|
||||
type BackupResult,
|
||||
type BackupError,
|
||||
@@ -179,6 +180,9 @@ export interface LibraryOptions {
|
||||
// directory at open. The content cache treats an original already stored
|
||||
// at its save path as present.
|
||||
downloadDirectory?: string;
|
||||
// How long after one refresh ends the background timer starts the next.
|
||||
// `Infinity` starts none: the library then refreshes only as it opens and
|
||||
// when `fresh()` or `backup()` asks.
|
||||
refreshIntervalSeconds?: number;
|
||||
onProgress?: RefreshProgressCallback;
|
||||
// The bounded request pools (issue #45), shared by the ML-data fetch (the
|
||||
@@ -567,9 +571,10 @@ export class Library {
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless
|
||||
// `opts.lockHeld` says the caller holds it, failing at once while another
|
||||
// backup of it runs (see `runBackup`). Waits for a
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
// caller passes the lock it took as `opts.lock`, failing at once while
|
||||
// another backup of it runs, and before it starts another file once the
|
||||
// lock is lost (see `lockBackupDirectory`). Waits for a
|
||||
// completed refresh, as `fresh()` does, joining one already running, and
|
||||
// rejects before touching any file but the lock when it fails. Then puts
|
||||
// pending originals at their save paths as `Photo.download()` does (and
|
||||
@@ -598,6 +603,7 @@ export class Library {
|
||||
refresh: () => this.refreshNow(),
|
||||
listCollections: () => this.store.listCollections(),
|
||||
listFiles: (id) => this.store.listFiles(id),
|
||||
undecryptable: () => this.store.listUndecryptable(),
|
||||
original: (fileID, destination) =>
|
||||
cache!.backupOriginal(fileID, destination),
|
||||
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
||||
@@ -627,7 +633,8 @@ export class Library {
|
||||
}
|
||||
|
||||
private scheduleNext(): void {
|
||||
if (this.closed) return;
|
||||
// `setTimeout` would turn an infinite delay into 1 ms.
|
||||
if (this.closed || this.intervalMs === Infinity) return;
|
||||
this.timer = setTimeout(() => {
|
||||
void this.runRefresh().then(() => this.scheduleNext());
|
||||
}, this.intervalMs);
|
||||
@@ -699,7 +706,10 @@ export class Library {
|
||||
|
||||
// Fetch every change since the stored cursor, then commit. All network
|
||||
// reads happen before any store mutation, so a fetch that throws leaves the
|
||||
// store untouched and the previous snapshot intact.
|
||||
// store untouched and the previous snapshot intact. A collection or file
|
||||
// that does not decrypt is left out instead, as if deleted, with the reason
|
||||
// kept in the store: the cursor moves past it, so no later refresh sees it
|
||||
// again until the server's copy changes.
|
||||
private async refreshOnce(): Promise<void> {
|
||||
const page = await this.client.collectionsSince({
|
||||
sinceTime: this.store.collectionsSinceTime,
|
||||
@@ -736,9 +746,17 @@ export class Library {
|
||||
this.store.deleteCollection(id);
|
||||
changed = true;
|
||||
}
|
||||
if (this.store.deleteUndecryptable(id)) changed = true;
|
||||
}
|
||||
// A collection left out takes its files with it.
|
||||
for (const { id, error } of page.undecryptable ?? []) {
|
||||
this.store.deleteCollection(id);
|
||||
this.store.putUndecryptable({ collectionID: id, error });
|
||||
changed = true;
|
||||
}
|
||||
for (const collection of page.collections) {
|
||||
this.store.putCollection(collection);
|
||||
this.store.deleteUndecryptable(collection.id);
|
||||
changed = true;
|
||||
}
|
||||
|
||||
@@ -748,9 +766,22 @@ export class Library {
|
||||
this.store.deleteFile(collectionID, id);
|
||||
changed = true;
|
||||
}
|
||||
if (this.store.deleteUndecryptable(collectionID, id)) {
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
for (const { id, error } of filePage.undecryptable ?? []) {
|
||||
this.store.deleteFile(collectionID, id);
|
||||
this.store.putUndecryptable({
|
||||
collectionID,
|
||||
fileID: id,
|
||||
error,
|
||||
});
|
||||
changed = true;
|
||||
}
|
||||
for (const f of filePage.files) {
|
||||
this.store.putFile(f);
|
||||
this.store.deleteUndecryptable(collectionID, f.id);
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
|
||||
+50
-5
@@ -2,10 +2,11 @@
|
||||
//
|
||||
// The store keeps one `metadata.json` file holding the account's server
|
||||
// state: the user id, a schema version, the cursor for the incremental
|
||||
// collections listing, and the decrypted collection and file records. The
|
||||
// whole file is read into RAM on load and rewritten as a whole on save; there
|
||||
// is no partial update and no lock file. A separate refresh unit populates the
|
||||
// store from the server — this module only stores what it is given.
|
||||
// collections listing, the decrypted collection and file records, and the
|
||||
// collections and files the refresh could not decrypt. The whole file is read
|
||||
// into RAM on load and rewritten as a whole on save; there is no partial
|
||||
// update and no lock file. A separate refresh unit populates the store from
|
||||
// the server — this module only stores what it is given.
|
||||
//
|
||||
// The file is a cache, so it is never trusted to exist or to be intact: a
|
||||
// missing or unreadable file loads as an empty store rather than an error, and
|
||||
@@ -33,12 +34,22 @@ const FILE_MODE = 0o600;
|
||||
type StoredCollection = Omit<Collection, "key"> & { key: string };
|
||||
type StoredFile = Omit<EnteFile, "key"> & { key: string };
|
||||
|
||||
// A collection, or with `fileID` a file in it, that the last refresh to fetch
|
||||
// it could not decrypt or parse, and why. It is kept until a later refresh
|
||||
// fetches it again, because a refresh fetches only what changed.
|
||||
export interface Undecryptable {
|
||||
collectionID: number;
|
||||
fileID?: number;
|
||||
error: string;
|
||||
}
|
||||
|
||||
interface StoredMetadata {
|
||||
schemaVersion: number;
|
||||
userID: number;
|
||||
collectionsSinceTime: Microseconds;
|
||||
collections: StoredCollection[];
|
||||
files: StoredFile[];
|
||||
undecryptable: Undecryptable[];
|
||||
}
|
||||
|
||||
const encodeKey = (key: Uint8Array): string =>
|
||||
@@ -67,6 +78,7 @@ export class MetadataStore {
|
||||
|
||||
private readonly collections = new Map<number, Collection>();
|
||||
private readonly files = new Map<string, EnteFile>();
|
||||
private undecryptable: Undecryptable[] = [];
|
||||
|
||||
private constructor(path: string) {
|
||||
this.path = path;
|
||||
@@ -107,6 +119,12 @@ export class MetadataStore {
|
||||
};
|
||||
store.files.set(fileKey(file.collectionID, file.id), file);
|
||||
}
|
||||
for (const record of parsed.undecryptable) {
|
||||
if (typeof record.collectionID !== "number") {
|
||||
throw new Error("damaged record of what did not decrypt");
|
||||
}
|
||||
store.undecryptable.push(record);
|
||||
}
|
||||
} catch {
|
||||
// Any corruption discards the partial result: a half-read cache is
|
||||
// worse than an empty one, since the refresh unit will rebuild it.
|
||||
@@ -134,6 +152,7 @@ export class MetadataStore {
|
||||
...f,
|
||||
key: encodeKey(f.key),
|
||||
})),
|
||||
undecryptable: this.undecryptable,
|
||||
};
|
||||
|
||||
const dir = dirname(this.path);
|
||||
@@ -165,7 +184,8 @@ export class MetadataStore {
|
||||
this.collections.set(collection.id, collection);
|
||||
}
|
||||
|
||||
// Removing a collection also drops its file memberships: a file record is
|
||||
// Removing a collection also drops its file memberships, and any record
|
||||
// that it or one of its files could not be decrypted: a file record is
|
||||
// only meaningful as part of a collection the cache still knows about.
|
||||
deleteCollection(id: number): void {
|
||||
this.collections.delete(id);
|
||||
@@ -174,6 +194,9 @@ export class MetadataStore {
|
||||
this.files.delete(key);
|
||||
}
|
||||
}
|
||||
this.undecryptable = this.undecryptable.filter(
|
||||
(record) => record.collectionID !== id,
|
||||
);
|
||||
}
|
||||
|
||||
getFile(collectionID: number, fileID: number): EnteFile | undefined {
|
||||
@@ -205,4 +228,26 @@ export class MetadataStore {
|
||||
deleteFile(collectionID: number, fileID: number): void {
|
||||
this.files.delete(fileKey(collectionID, fileID));
|
||||
}
|
||||
|
||||
listUndecryptable(): Undecryptable[] {
|
||||
return [...this.undecryptable];
|
||||
}
|
||||
|
||||
// Replaces any earlier record of the same collection or file.
|
||||
putUndecryptable(record: Undecryptable): void {
|
||||
this.deleteUndecryptable(record.collectionID, record.fileID);
|
||||
this.undecryptable.push(record);
|
||||
}
|
||||
|
||||
// Forgets the record of the collection, or with `fileID` of that file in
|
||||
// it. Returns whether there was one.
|
||||
deleteUndecryptable(collectionID: number, fileID?: number): boolean {
|
||||
const before = this.undecryptable.length;
|
||||
this.undecryptable = this.undecryptable.filter(
|
||||
(record) =>
|
||||
record.collectionID !== collectionID ||
|
||||
record.fileID !== fileID,
|
||||
);
|
||||
return this.undecryptable.length < before;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,832 @@
|
||||
/**
|
||||
* `quak backup`, end to end.
|
||||
*
|
||||
* Each test runs the command as cron runs it: `backupCommand` reads the
|
||||
* session file that logging in saved, opens the library on the real `Client`
|
||||
* and backs the whole account up into a directory, and `run` turns the outcome
|
||||
* into the exit code, as `bin/quak.ts` does. Only the server is a stand-in:
|
||||
* `buildMetaFetch` from `test/ente-server.ts` answers the Ente HTTP API from
|
||||
* data this file encrypts with libsodium, as Ente's clients encrypt it. So
|
||||
* quak's own code fetches each original, decrypts it, checks it against the
|
||||
* hash its uploader recorded and unpacks a live photo, and quak's own retries
|
||||
* repeat a failed request. Two tests run `lib.backup()` instead, as a program
|
||||
* using quak as a library does.
|
||||
*
|
||||
* The account, user 42, has three albums:
|
||||
*
|
||||
* Trips its own: beach.jpg
|
||||
* Family its own: beach.jpg and the live photo IMG_0002.HEIC
|
||||
* Friend's Wedding shared with it by user 99: toast.mp4
|
||||
*
|
||||
* beach.jpg is one file in two albums. It holds EXIF (a Canon EOS R5), and
|
||||
* Ente has ML data for it. All three files were taken at noon on 1 March 2024.
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
lstatSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { Writable } from "node:stream";
|
||||
import { gzipSync } from "node:zlib";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import {
|
||||
afterAll,
|
||||
afterEach,
|
||||
beforeAll,
|
||||
beforeEach,
|
||||
describe,
|
||||
expect,
|
||||
it,
|
||||
vi,
|
||||
} from "vitest";
|
||||
|
||||
import {
|
||||
backupCommand,
|
||||
saveSession,
|
||||
type CliContext,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { Library, lockBackupDirectory } from "../../src/library/index.js";
|
||||
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
|
||||
import {
|
||||
buildAccount,
|
||||
buildMetaFetch,
|
||||
encryptSecretbox,
|
||||
encryptStreamBlob,
|
||||
} from "../ente-server.js";
|
||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||
import { JPEG_WITH_EXIF } from "../exif-jpeg.js";
|
||||
import { blake2b, livePhotoHash, livePhotoZip, VIDEO } from "../live-photo.js";
|
||||
|
||||
const EMAIL = "backup@example.com";
|
||||
const PASSWORD = "backup-password";
|
||||
// The fake server logs every account in as user 42.
|
||||
const USER_ID = 42;
|
||||
const FRIEND_ID = 99;
|
||||
|
||||
// Noon on 1 March 2024 in the time zone of the machine running the test, the
|
||||
// one quak dates save paths in, in microseconds, as Ente records times.
|
||||
const TAKEN = new Date(2024, 2, 1, 12).getTime() * 1000;
|
||||
// The date folder every original is saved in.
|
||||
const DAY = "2024/2024-03/2024-03-01";
|
||||
|
||||
const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
|
||||
|
||||
// Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
|
||||
// embedding.
|
||||
const ML_DATA = {
|
||||
face: { faces: [{ faceID: "100_1", score: 0.98 }] },
|
||||
clip: { embedding: [0.5, 0.6, 0.7] },
|
||||
};
|
||||
|
||||
// Each file's ID, its owner, its original as uploaded, and the metadata its
|
||||
// uploader recorded. `fileType` is Ente's number for the kind: 0 an image, 1 a
|
||||
// video, 2 a live photo, whose original is a ZIP of its image and its video.
|
||||
// `hash` is the content hash a download must match.
|
||||
const BEACH = {
|
||||
id: 100,
|
||||
ownerID: USER_ID,
|
||||
original: JPEG_WITH_EXIF,
|
||||
metadata: {
|
||||
title: "beach.jpg",
|
||||
fileType: 0,
|
||||
hash: blake2b(JPEG_WITH_EXIF),
|
||||
},
|
||||
};
|
||||
const TOAST_VIDEO = {
|
||||
id: 200,
|
||||
ownerID: FRIEND_ID,
|
||||
original: TOAST,
|
||||
metadata: { title: "toast.mp4", fileType: 1, hash: blake2b(TOAST) },
|
||||
};
|
||||
const LIVE_PHOTO = {
|
||||
id: 300,
|
||||
ownerID: USER_ID,
|
||||
original: livePhotoZip({
|
||||
"image.heic": HEIC_WITH_EXIF,
|
||||
"video.mov": VIDEO,
|
||||
}),
|
||||
metadata: {
|
||||
title: "IMG_0002.HEIC",
|
||||
fileType: 2,
|
||||
hash: livePhotoHash(HEIC_WITH_EXIF, VIDEO),
|
||||
},
|
||||
};
|
||||
|
||||
// The fake Ente server, as the `fetch` quak sends its requests through.
|
||||
let server: typeof globalThis.fetch;
|
||||
let root: string;
|
||||
let sessionDir: string;
|
||||
|
||||
// The fake server, holding the account encrypted as Ente's clients upload it.
|
||||
const buildServer = async () => {
|
||||
const account = await buildAccount(PASSWORD);
|
||||
|
||||
// An album as `/collections/v2` lists it, its name encrypted with its own
|
||||
// key. An album of the account's own carries that key encrypted with the
|
||||
// master key; an album shared with it carries the key sealed to its public
|
||||
// key, and no nonce.
|
||||
const album = (id: number, name: string, ownerID: number) => {
|
||||
const key = sodium.crypto_secretbox_keygen();
|
||||
const encryptedName = encryptSecretbox(
|
||||
new TextEncoder().encode(name),
|
||||
key,
|
||||
);
|
||||
let keyFields: { encryptedKey: string; keyDecryptionNonce?: string };
|
||||
if (ownerID === USER_ID) {
|
||||
const encryptedKey = encryptSecretbox(key, account.masterKey);
|
||||
keyFields = {
|
||||
encryptedKey: toBase64(encryptedKey.ciphertext),
|
||||
keyDecryptionNonce: toBase64(encryptedKey.nonce),
|
||||
};
|
||||
} else {
|
||||
const sealedKey = sodium.crypto_box_seal(key, account.publicKey);
|
||||
keyFields = { encryptedKey: toBase64(sealedKey) };
|
||||
}
|
||||
const row = {
|
||||
id,
|
||||
owner: { id: ownerID },
|
||||
...keyFields,
|
||||
encryptedName: toBase64(encryptedName.ciphertext),
|
||||
nameDecryptionNonce: toBase64(encryptedName.nonce),
|
||||
type: "album",
|
||||
updationTime: TAKEN,
|
||||
};
|
||||
return { key, row };
|
||||
};
|
||||
const trips = album(10, "Trips", USER_ID);
|
||||
const family = album(20, "Family", USER_ID);
|
||||
const wedding = album(30, "Friend's Wedding", FRIEND_ID);
|
||||
|
||||
// Each file's original is encrypted with the file's own key, as one
|
||||
// secretstream, which the server serves whole.
|
||||
const encrypted = (file: typeof BEACH) => {
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
return { file, key, content: encryptStreamBlob(file.original, key) };
|
||||
};
|
||||
const beach = encrypted(BEACH);
|
||||
const toast = encrypted(TOAST_VIDEO);
|
||||
const livePhoto = encrypted(LIVE_PHOTO);
|
||||
|
||||
// A file as an album's `/collections/v2/diff` lists it: its key encrypted
|
||||
// with the album's key, its metadata encrypted with its own key, and the
|
||||
// header that starts the decryption of its original.
|
||||
const inAlbum = (
|
||||
{ file, key, content }: ReturnType<typeof encrypted>,
|
||||
collectionID: number,
|
||||
albumKey: Uint8Array,
|
||||
) => {
|
||||
const encryptedKey = encryptSecretbox(key, albumKey);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
...file.metadata,
|
||||
creationTime: TAKEN,
|
||||
modificationTime: TAKEN,
|
||||
}),
|
||||
),
|
||||
key,
|
||||
);
|
||||
return {
|
||||
id: file.id,
|
||||
collectionID,
|
||||
ownerID: file.ownerID,
|
||||
encryptedKey: toBase64(encryptedKey.ciphertext),
|
||||
keyDecryptionNonce: toBase64(encryptedKey.nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
file: { decryptionHeader: toBase64(content.header) },
|
||||
thumbnail: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
updationTime: TAKEN,
|
||||
};
|
||||
};
|
||||
|
||||
// ML data is gzipped JSON, encrypted with the file's key.
|
||||
const mlData = encryptStreamBlob(
|
||||
gzipSync(JSON.stringify(ML_DATA)),
|
||||
beach.key,
|
||||
);
|
||||
|
||||
return buildMetaFetch({
|
||||
verifier: account.verifier,
|
||||
srpAttributes: account.srpAttributes,
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections: [trips.row, family.row, wedding.row],
|
||||
filesByCollection: {
|
||||
10: [inAlbum(beach, 10, trips.key)],
|
||||
20: [
|
||||
inAlbum(beach, 20, family.key),
|
||||
inAlbum(livePhoto, 20, family.key),
|
||||
],
|
||||
30: [inAlbum(toast, 30, wedding.key)],
|
||||
},
|
||||
encryptedMLData: {
|
||||
100: {
|
||||
encryptedData: toBase64(mlData.ciphertext),
|
||||
decryptionHeader: toBase64(mlData.header),
|
||||
},
|
||||
},
|
||||
fileCiphertexts: {
|
||||
100: beach.content.ciphertext,
|
||||
200: toast.content.ciphertext,
|
||||
300: livePhoto.content.ciphertext,
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
root = mkdtempSync(join(tmpdir(), "quak-backup-end-to-end-"));
|
||||
server = await buildServer();
|
||||
|
||||
// Log in once, as `quak login` does: the session file it saves is what
|
||||
// every backup below reads.
|
||||
sessionDir = join(root, "session");
|
||||
const client = await Client.login({
|
||||
email: EMAIL,
|
||||
password: PASSWORD,
|
||||
apiOptions: { fetch: server },
|
||||
});
|
||||
saveSession(sessionDir, client.toJSON());
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Run `quak backup <dir>` as `bin/quak.ts` runs it, sending its requests
|
||||
// through `fetch`, and return its exit code and everything it printed on
|
||||
// stderr. Its requests are retried as those of `quak backup` are, except that
|
||||
// a retry does not wait. Its cache is beside `dir`, so each run of one
|
||||
// directory finds the cache the run before it left, as cron's runs find the
|
||||
// user's.
|
||||
const backup = async (
|
||||
dir: string,
|
||||
opts: { verify?: boolean } = {},
|
||||
fetch = server,
|
||||
): Promise<{ code: number; stderr: string }> => {
|
||||
let printed = "";
|
||||
const stderr = new Writable({
|
||||
write(chunk: Buffer, _encoding, done) {
|
||||
printed += chunk.toString();
|
||||
done();
|
||||
},
|
||||
});
|
||||
const stdout = new Writable({
|
||||
write(_chunk, _encoding, done) {
|
||||
done();
|
||||
},
|
||||
});
|
||||
// A backup never prompts: any of these ends the run with exit 1.
|
||||
const prompted = async (): Promise<never> => {
|
||||
throw new Error("quak backup prompted");
|
||||
};
|
||||
const ctx: CliContext = {
|
||||
stdout,
|
||||
stderr,
|
||||
sessionDir,
|
||||
cacheDir: `${dir}-cache`,
|
||||
loadSession: (path) =>
|
||||
loadSession(path, {
|
||||
fetch,
|
||||
retry: { ...UNATTENDED_RETRY_OPTIONS, sleep: async () => {} },
|
||||
}),
|
||||
login: prompted,
|
||||
prompt: prompted,
|
||||
promptSecret: prompted,
|
||||
};
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(backupCommand(ctx, dir, opts), stdout, stderr, resolve);
|
||||
});
|
||||
return { code, stderr: printed };
|
||||
};
|
||||
|
||||
// The ID of the file whose original a request downloads, or undefined for any
|
||||
// other request. quak downloads an original from
|
||||
// `https://files.ente.io/?fileID=<id>`.
|
||||
const downloadOf = (input: RequestInfo | URL): number | undefined => {
|
||||
const url = new URL(String(input));
|
||||
return url.hostname === "files.ente.io"
|
||||
? Number(url.searchParams.get("fileID"))
|
||||
: undefined;
|
||||
};
|
||||
|
||||
// The fake server, pushing onto `downloads` the ID of each file whose original
|
||||
// is downloaded.
|
||||
const recordingDownloads =
|
||||
(downloads: number[]): typeof globalThis.fetch =>
|
||||
async (input, init) => {
|
||||
const fileID = downloadOf(input);
|
||||
if (fileID !== undefined) downloads.push(fileID);
|
||||
return server(input, init);
|
||||
};
|
||||
|
||||
// Every entry under `dir`, relative to it, sorted, a symlink as
|
||||
// `name -> target`.
|
||||
const listTree = (dir: string): string[] =>
|
||||
readdirSync(dir, { recursive: true })
|
||||
.sort()
|
||||
.map((name) => {
|
||||
const path = join(dir, name);
|
||||
return lstatSync(path).isSymbolicLink()
|
||||
? `${name} -> ${readlinkSync(path)}`
|
||||
: name;
|
||||
});
|
||||
|
||||
const readJSON = (path: string) => JSON.parse(readFileSync(path, "utf-8"));
|
||||
|
||||
describe("the first backup of the account", () => {
|
||||
let dir: string;
|
||||
let code: number;
|
||||
const downloads: number[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = mkdtempSync(join(root, "backup-"));
|
||||
({ code } = await backup(dir, {}, recordingDownloads(downloads)));
|
||||
});
|
||||
|
||||
it("exits 0, having downloaded each file once", () => {
|
||||
expect(code).toBe(0);
|
||||
// beach.jpg is in two albums but is downloaded, and stored, once.
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
});
|
||||
|
||||
// The tree "Backup layout" in the README describes. Each original is at
|
||||
// its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
|
||||
// the extension of its title; the live photo is its image and its video,
|
||||
// each with the extension it had inside the ZIP, and a `.livephoto.json`
|
||||
// naming the two. Beside each original is its JSON. Each album is a
|
||||
// folder of links named after the files' titles, with a JSON file beside
|
||||
// it. No file failed, so there is no `failures.json`, and the run removed
|
||||
// its lock, `backup.lock`, as it ended.
|
||||
it("lays the backup out as the README describes", () => {
|
||||
expect(listTree(dir)).toEqual([
|
||||
"2024",
|
||||
"2024/2024-03",
|
||||
"2024/2024-03/2024-03-01",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.100.jpg",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.100.json",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.200.json",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.200.mp4",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.heic",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.json",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.livephoto.json",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.mov",
|
||||
"account.json",
|
||||
"collections",
|
||||
"collections/Family",
|
||||
"collections/Family.json",
|
||||
"collections/Family/IMG_0002.heic -> ../../2024/2024-03/2024-03-01/2024-03-01.300.heic",
|
||||
"collections/Family/IMG_0002.mov -> ../../2024/2024-03/2024-03-01/2024-03-01.300.mov",
|
||||
"collections/Family/beach.jpg -> ../../2024/2024-03/2024-03-01/2024-03-01.100.jpg",
|
||||
"collections/Friend's Wedding",
|
||||
"collections/Friend's Wedding.json",
|
||||
"collections/Friend's Wedding/toast.mp4 -> ../../2024/2024-03/2024-03-01/2024-03-01.200.mp4",
|
||||
"collections/Trips",
|
||||
"collections/Trips.json",
|
||||
"collections/Trips/beach.jpg -> ../../2024/2024-03/2024-03-01/2024-03-01.100.jpg",
|
||||
]);
|
||||
});
|
||||
|
||||
it("stores each original decrypted, a live photo as its image and its video", () => {
|
||||
const day = join(dir, DAY);
|
||||
expect(readFileSync(join(day, "2024-03-01.100.jpg"))).toEqual(
|
||||
Buffer.from(JPEG_WITH_EXIF),
|
||||
);
|
||||
expect(readFileSync(join(day, "2024-03-01.200.mp4"))).toEqual(
|
||||
Buffer.from(TOAST),
|
||||
);
|
||||
expect(readFileSync(join(day, "2024-03-01.300.heic"))).toEqual(
|
||||
Buffer.from(HEIC_WITH_EXIF),
|
||||
);
|
||||
expect(readFileSync(join(day, "2024-03-01.300.mov"))).toEqual(
|
||||
Buffer.from(VIDEO),
|
||||
);
|
||||
expect(readJSON(join(day, "2024-03-01.300.livephoto.json"))).toEqual({
|
||||
image: "2024-03-01.300.heic",
|
||||
video: "2024-03-01.300.mov",
|
||||
});
|
||||
});
|
||||
|
||||
// A file's JSON holds what Ente records for it, decrypted, and what quak
|
||||
// read from its original: its ML data when Ente has any, and, for an
|
||||
// image, its EXIF. A video gets no image metadata.
|
||||
it("writes each file's JSON with its ML data and image metadata", () => {
|
||||
const day = join(dir, DAY);
|
||||
|
||||
const beach = readJSON(join(day, "2024-03-01.100.json"));
|
||||
expect(beach).toMatchObject({
|
||||
id: 100,
|
||||
ownerID: USER_ID,
|
||||
metadata: {
|
||||
title: "beach.jpg",
|
||||
fileType: "image",
|
||||
creationTime: TAKEN,
|
||||
hash: BEACH.metadata.hash,
|
||||
},
|
||||
updationTime: TAKEN,
|
||||
});
|
||||
expect(beach.mlData).toEqual(ML_DATA);
|
||||
expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
|
||||
expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
|
||||
|
||||
const toast = readJSON(join(day, "2024-03-01.200.json"));
|
||||
expect(toast).toMatchObject({
|
||||
id: 200,
|
||||
ownerID: FRIEND_ID,
|
||||
metadata: { title: "toast.mp4", fileType: "video" },
|
||||
});
|
||||
expect(toast).not.toHaveProperty("mlData");
|
||||
expect(toast).not.toHaveProperty("imageMetadata");
|
||||
expect(toast).not.toHaveProperty("imageMetadataError");
|
||||
|
||||
// A live photo's image metadata is read from its image.
|
||||
const livePhoto = readJSON(join(day, "2024-03-01.300.json"));
|
||||
expect(livePhoto).toMatchObject({
|
||||
id: 300,
|
||||
ownerID: USER_ID,
|
||||
metadata: {
|
||||
title: "IMG_0002.HEIC",
|
||||
fileType: "livePhoto",
|
||||
hash: LIVE_PHOTO.metadata.hash,
|
||||
},
|
||||
});
|
||||
expect(livePhoto.imageMetadata.exif.Make.description).toBe("Canon");
|
||||
});
|
||||
|
||||
it("writes each album's JSON and the account's", () => {
|
||||
const albumJSON = (name: string) =>
|
||||
readJSON(join(dir, "collections", `${name}.json`));
|
||||
const fileIDs = (album: { files: { id: number }[] }) =>
|
||||
album.files.map((f) => f.id);
|
||||
|
||||
const trips = albumJSON("Trips");
|
||||
expect(trips).toMatchObject({
|
||||
id: 10,
|
||||
name: "Trips",
|
||||
type: "album",
|
||||
ownerID: USER_ID,
|
||||
isShared: false,
|
||||
updationTime: TAKEN,
|
||||
});
|
||||
expect(fileIDs(trips)).toEqual([100]);
|
||||
|
||||
const family = albumJSON("Family");
|
||||
expect(family).toMatchObject({ id: 20, isShared: false });
|
||||
expect(fileIDs(family)).toEqual([100, 300]);
|
||||
|
||||
const wedding = albumJSON("Friend's Wedding");
|
||||
expect(wedding).toMatchObject({
|
||||
id: 30,
|
||||
ownerID: FRIEND_ID,
|
||||
isShared: true,
|
||||
});
|
||||
expect(fileIDs(wedding)).toEqual([200]);
|
||||
|
||||
expect(readJSON(join(dir, "account.json"))).toEqual({
|
||||
email: EMAIL,
|
||||
userID: USER_ID,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("quak backup", () => {
|
||||
// An original already at its save path is complete, so a later run, such
|
||||
// as the next night's from cron, only rewrites the JSON files and the
|
||||
// album links.
|
||||
it("downloads nothing on a second run, and exits 0", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
expect((await backup(dir)).code).toBe(0);
|
||||
const firstTree = listTree(dir);
|
||||
|
||||
const downloads: number[] = [];
|
||||
const { code } = await backup(dir, {}, recordingDownloads(downloads));
|
||||
expect(code).toBe(0);
|
||||
expect(downloads).toEqual([]);
|
||||
expect(listTree(dir)).toEqual(firstTree);
|
||||
});
|
||||
|
||||
// The files CDN is briefly unavailable for toast.mp4: its first two
|
||||
// downloads answer 503. A 5xx is worth repeating, so quak tries again,
|
||||
// here without waiting, and the third download succeeds.
|
||||
it("stores a file whose download fails twice with 503 and then succeeds, and exits 0", async () => {
|
||||
let attempts = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (downloadOf(input) === 200 && ++attempts <= 2) {
|
||||
return new Response("unavailable", { status: 503 });
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(attempts).toBe(3);
|
||||
expect(readFileSync(join(dir, DAY, "2024-03-01.200.mp4"))).toEqual(
|
||||
Buffer.from(TOAST),
|
||||
);
|
||||
expect(existsSync(join(dir, "failures.json"))).toBe(false);
|
||||
});
|
||||
|
||||
// The server answers 404 for toast.mp4 on every try. A 404 is an answer,
|
||||
// not a hiccup, so quak does not repeat it: the file goes into
|
||||
// `failures.json` with the kind of failure, how many times it has been
|
||||
// tried and why, the backup goes on with the other files, and the exit
|
||||
// code is 1, so the next run tries again.
|
||||
it("records a file whose download always answers 404 in failures.json, stores the rest, and exits 1", async () => {
|
||||
let attempts = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (downloadOf(input) === 200) {
|
||||
attempts++;
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code, stderr } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(attempts).toBe(1);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(Object.keys(failures.files)).toEqual(["200"]);
|
||||
expect(failures.files["200"]).toMatchObject({
|
||||
fileID: 200,
|
||||
title: "toast.mp4",
|
||||
classification: "permanent",
|
||||
attempts: 1,
|
||||
});
|
||||
expect(failures.files["200"].error).toContain("404");
|
||||
expect(stderr).toContain("[Friend's Wedding] toast.mp4 (id 200): ");
|
||||
|
||||
// The rest of the account is backed up; the video is not, and its
|
||||
// album folder has no link to it.
|
||||
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
|
||||
Buffer.from(JPEG_WITH_EXIF),
|
||||
);
|
||||
expect(readFileSync(join(dir, DAY, "2024-03-01.300.heic"))).toEqual(
|
||||
Buffer.from(HEIC_WITH_EXIF),
|
||||
);
|
||||
expect(existsSync(join(dir, DAY, "2024-03-01.200.mp4"))).toBe(false);
|
||||
expect(
|
||||
readdirSync(join(dir, "collections", "Friend's Wedding")),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
// The server no longer accepts the saved session's token, as after
|
||||
// `quak logout` elsewhere, and answers every request with 401. The run
|
||||
// stops at the refresh it starts with, says to log in again, and exits 3.
|
||||
// It took its lock first, so every request was sent while the lock was
|
||||
// there, and it removed the lock as it ended: the directory is left as it
|
||||
// was, empty.
|
||||
it("exits 3 when the refresh is answered with 401, writing nothing but the lock", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const lockAtEachRequest: boolean[] = [];
|
||||
const fetch: typeof globalThis.fetch = async () => {
|
||||
lockAtEachRequest.push(existsSync(join(dir, "backup.lock")));
|
||||
return new Response(null, { status: 401 });
|
||||
};
|
||||
|
||||
const { code, stderr } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(3);
|
||||
expect(stderr).toBe(
|
||||
"Starting backup...\n" +
|
||||
"Refreshing library...\n" +
|
||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
||||
);
|
||||
expect(lockAtEachRequest).not.toHaveLength(0);
|
||||
expect(lockAtEachRequest).not.toContain(false);
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
});
|
||||
|
||||
// Something changed beach.jpg on disk after it was backed up. A run with
|
||||
// --verify hashes every stored original, finds that one no longer matches
|
||||
// the hash Ente recorded, says so, and downloads it again; it put back the
|
||||
// right bytes, so the exit code is 0.
|
||||
it("--verify puts back an original that changed on disk, and exits 0", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
expect((await backup(dir)).code).toBe(0);
|
||||
const beach = join(dir, DAY, "2024-03-01.100.jpg");
|
||||
writeFileSync(beach, "not the photo any more");
|
||||
|
||||
const downloads: number[] = [];
|
||||
const { code, stderr } = await backup(
|
||||
dir,
|
||||
{ verify: true },
|
||||
recordingDownloads(downloads),
|
||||
);
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(downloads).toEqual([100]);
|
||||
expect(readFileSync(beach)).toEqual(Buffer.from(JPEG_WITH_EXIF));
|
||||
expect(stderr).toContain(
|
||||
"MISMATCH original beach.jpg (100): its bytes do not match its content hash",
|
||||
);
|
||||
expect(stderr).toContain(" Verified: 2\n Mismatched: 1\n");
|
||||
});
|
||||
});
|
||||
|
||||
// `backup.lock` is a directory that proper-lockfile touches every 5 seconds,
|
||||
// from a timer, while the backup runs. Each time, it first looks at the lock
|
||||
// on disk. The lock is lost when it is gone, or when another backup has made
|
||||
// its own in its place, as one does with a lock nobody touched for 10
|
||||
// seconds. Another backup may then be writing to the directory, so the backup
|
||||
// stops before it starts another file.
|
||||
//
|
||||
// The fake server holds one file's download open until the test lets it go,
|
||||
// so the lock can be lost while the backup runs: beach.jpg's, the first file
|
||||
// backed up, unless the test names another. Only `setTimeout` and
|
||||
// `clearTimeout` are faked, so a test can fire proper-lockfile's timer without
|
||||
// waiting 5 seconds.
|
||||
describe("a backup whose lock is lost", () => {
|
||||
let dir: string;
|
||||
let lockPath: string;
|
||||
let downloads: number[];
|
||||
let heldFile: number;
|
||||
let heldRequested: Promise<void>;
|
||||
let releaseHeld: () => void;
|
||||
let fetch: typeof globalThis.fetch;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(root, "backup-"));
|
||||
lockPath = join(dir, "backup.lock");
|
||||
downloads = [];
|
||||
heldFile = BEACH.id;
|
||||
let requested!: () => void;
|
||||
heldRequested = new Promise((resolve) => {
|
||||
requested = resolve;
|
||||
});
|
||||
const released = new Promise<void>((resolve) => {
|
||||
releaseHeld = resolve;
|
||||
});
|
||||
const recording = recordingDownloads(downloads);
|
||||
fetch = async (input, init) => {
|
||||
if (downloadOf(input) === heldFile) {
|
||||
requested();
|
||||
await released;
|
||||
}
|
||||
return recording(input, init);
|
||||
};
|
||||
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
// Fire proper-lockfile's timer, and wait for the look at the lock on disk
|
||||
// that it starts to finish: Node lists a file system call in progress as
|
||||
// an `FSReqCallback`.
|
||||
const fireLockTimer = async (): Promise<void> => {
|
||||
vi.advanceTimersByTime(5_000);
|
||||
while (process.getActiveResourcesInfo().includes("FSReqCallback")) {
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
}
|
||||
};
|
||||
|
||||
// beach.jpg, whose download was under way, is stored; no other file is
|
||||
// started. `quak backup` prints why as one line, with no stack trace, and
|
||||
// exits 1.
|
||||
it("stops quak backup before its next file when backup.lock is removed, and exits 1", async () => {
|
||||
const result = backup(dir, {}, fetch);
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
const { code, stderr } = await result;
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toBe(
|
||||
"Starting backup...\n" +
|
||||
"Refreshing library...\n" +
|
||||
"Fetching original beach.jpg (100)...\n" +
|
||||
`quak: lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'\n`,
|
||||
);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
|
||||
Buffer.from(JPEG_WITH_EXIF),
|
||||
);
|
||||
});
|
||||
|
||||
// Here the lock is lost while toast.mp4, the last original, downloads.
|
||||
// All three originals are stored, beside what the backup wrote before it
|
||||
// started them, but no file's JSON and no album is written.
|
||||
it("stops quak backup before any file's JSON when the lock is lost during the last original", async () => {
|
||||
heldFile = TOAST_VIDEO.id;
|
||||
const result = backup(dir, {}, fetch);
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
const { code, stderr } = await result;
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain(`quak: lost the lock on ${dir}: `);
|
||||
expect(listTree(dir)).toEqual([
|
||||
"2024",
|
||||
"2024/2024-03",
|
||||
"2024/2024-03/2024-03-01",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.100.jpg",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.200.mp4",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.heic",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.livephoto.json",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.mov",
|
||||
"account.json",
|
||||
"collections",
|
||||
]);
|
||||
});
|
||||
|
||||
// Nothing touches the lock but quak, which finds it as it left it each
|
||||
// time it looks, so the backup runs to the end and removes its lock.
|
||||
it("lets quak backup run to the end while its lock stays", async () => {
|
||||
const result = backup(dir, {}, fetch);
|
||||
await heldRequested;
|
||||
await fireLockTimer();
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
const { code } = await result;
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
expect(existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
// The library a program using quak opens to back `dir` up.
|
||||
const openLibrary = () =>
|
||||
Library.open({
|
||||
client: loadSession(join(sessionDir, "session.json"), { fetch })!,
|
||||
cacheDirectory: `${dir}-cache`,
|
||||
downloadDirectory: dir,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
// A program using quak as a library sees the same: `lib.backup()` rejects
|
||||
// with the reason. Here another backup has made its own lock in place of
|
||||
// quak's, and that lock is left where it is.
|
||||
it("rejects lib.backup() when another backup has taken the lock over", async () => {
|
||||
const lib = await openLibrary();
|
||||
try {
|
||||
const result = lib.backup();
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
mkdirSync(lockPath);
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
|
||||
await expect(result).rejects.toThrow(
|
||||
`lost the lock on ${dir}: Unable to update lock within the stale threshold`,
|
||||
);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
expect(existsSync(lockPath)).toBe(true);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A program can take the lock before it opens its library, as `quak
|
||||
// backup` does: `lockBackupDirectory` returns the lock, which the program
|
||||
// passes to `lib.backup()` and releases once the backup ends. The backup
|
||||
// checks it as it checks a lock it took, and the lock's `lost` says why it
|
||||
// was lost.
|
||||
it("rejects lib.backup() given its caller's lock once that lock is lost", async () => {
|
||||
const lock = await lockBackupDirectory(dir);
|
||||
const lib = await openLibrary();
|
||||
try {
|
||||
const result = lib.backup({ lock });
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
|
||||
const lost = `lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'`;
|
||||
await expect(result).rejects.toThrow(lost);
|
||||
expect(lock.lost?.message).toBe(lost);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,554 @@
|
||||
/**
|
||||
* `quak backup` over an account holding an album or a file quak cannot decrypt
|
||||
* (issue 181).
|
||||
*
|
||||
* The server sends every album and file encrypted, and a refresh decrypts each
|
||||
* one it fetches. A record whose key no key of the account opens, or whose
|
||||
* metadata is not JSON, must not fail the refresh: a refresh that fails commits
|
||||
* nothing, so `quak backup` would store nothing, on every run, until the
|
||||
* server's copy changed. Instead the refresh leaves that record out, and the
|
||||
* library keeps a list of what it left out in its cache. It has to: the next
|
||||
* refresh asks the server only for what changed since the last one, so it will
|
||||
* not see the record again.
|
||||
*
|
||||
* The backup stores every other file. It records each file it could not
|
||||
* decrypt in `failures.json`, prints one line for each album it could not
|
||||
* decrypt, and exits 1 while any remain. Once the server sends a copy that
|
||||
* decrypts, or deletes the record, the next run drops it and can exit 0. The
|
||||
* read commands answer from what did decrypt.
|
||||
*
|
||||
* These tests run the commands with a real saved session over a stand-in Ente
|
||||
* server, which serves albums, files and originals encrypted with libsodium as
|
||||
* the real server does. A record is broken the way a corrupt one arrives: its
|
||||
* key replaced by bytes no key opens, or its metadata text that is not JSON.
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
mkdtempSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
backupCommand,
|
||||
collectionsCommand,
|
||||
filesCommand,
|
||||
saveSession,
|
||||
type CliContext,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import { Client, type ClientSnapshot } from "../../src/client.js";
|
||||
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
|
||||
// stores times, so its original is saved in the folder `DAY` whatever the time
|
||||
// zone of the machine running the tests.
|
||||
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
|
||||
const DAY = join("2026", "2026-03", "2026-03-01");
|
||||
|
||||
type Row = { id: number; updationTime: number } & Record<string, unknown>;
|
||||
|
||||
// `plaintext` sealed under `key` with a fresh nonce, both base64, as Ente
|
||||
// sends a key or an album name.
|
||||
const secretbox = (
|
||||
plaintext: Uint8Array,
|
||||
key: Uint8Array,
|
||||
): { ciphertext: string; nonce: string } => {
|
||||
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
|
||||
return {
|
||||
ciphertext: toBase64(
|
||||
sodium.crypto_secretbox_easy(plaintext, nonce, key),
|
||||
),
|
||||
nonce: toBase64(nonce),
|
||||
};
|
||||
};
|
||||
|
||||
// What a corrupt row carries in place of its sealed key: random bytes of the
|
||||
// same length, which no key opens.
|
||||
const corruptKey = (): string =>
|
||||
toBase64(
|
||||
sodium.randombytes_buf(
|
||||
sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES,
|
||||
),
|
||||
);
|
||||
|
||||
const json = (body: unknown): Response =>
|
||||
new Response(JSON.stringify(body), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
|
||||
// A stand-in Ente server for one account. It holds each album, and each file in
|
||||
// an album, as the row `/collections/v2` and `/collections/v2/diff` serve, and
|
||||
// answers a request with the rows whose `updationTime` is past the request's
|
||||
// `sinceTime`, as the real server does. A change to a file also moves its
|
||||
// album's `updationTime`, which is what tells a client to list the album's
|
||||
// files again. Originals are served by file ID, as the files CDN serves them.
|
||||
class FakeEnte {
|
||||
private readonly masterKey = sodium.crypto_secretbox_keygen();
|
||||
private readonly keyPair = sodium.crypto_box_keypair();
|
||||
private readonly albums = new Map<number, Row>();
|
||||
private readonly albumKeys = new Map<number, Uint8Array>();
|
||||
// Each album's file rows, by album ID and then file ID.
|
||||
private readonly files = new Map<number, Map<number, Row>>();
|
||||
// Each file's key and its original as served, by file ID. A file in two
|
||||
// albums is one original under one key, sealed under each album's key.
|
||||
private readonly originals = new Map<
|
||||
number,
|
||||
{ key: Uint8Array; header: string; body: Uint8Array }
|
||||
>();
|
||||
|
||||
// The session `quak login` would have saved for the account.
|
||||
snapshot(): ClientSnapshot {
|
||||
return {
|
||||
email: "user@example.com",
|
||||
userID: USER_ID,
|
||||
token: "token",
|
||||
masterKey: toBase64(this.masterKey),
|
||||
secretKey: toBase64(this.keyPair.privateKey),
|
||||
publicKey: toBase64(this.keyPair.publicKey),
|
||||
};
|
||||
}
|
||||
|
||||
// Album `id`, named `name`, as of time `t`. With `corrupt`, its key cannot
|
||||
// be decrypted. An album keeps its key from one call to the next, so its
|
||||
// files' rows still decrypt once it is put back without `corrupt`.
|
||||
putAlbum(id: number, name: string, t: number, corrupt = false): void {
|
||||
const key = this.albumKeys.get(id) ?? sodium.crypto_secretbox_keygen();
|
||||
this.albumKeys.set(id, key);
|
||||
const sealedKey = secretbox(key, this.masterKey);
|
||||
const sealedName = secretbox(new TextEncoder().encode(name), key);
|
||||
this.albums.set(id, {
|
||||
id,
|
||||
owner: { id: USER_ID },
|
||||
encryptedKey: corrupt ? corruptKey() : sealedKey.ciphertext,
|
||||
keyDecryptionNonce: sealedKey.nonce,
|
||||
encryptedName: sealedName.ciphertext,
|
||||
nameDecryptionNonce: sealedName.nonce,
|
||||
type: "album",
|
||||
updationTime: t,
|
||||
});
|
||||
if (!this.files.has(id)) this.files.set(id, new Map());
|
||||
}
|
||||
|
||||
// File `id`, titled `photo-<id>.jpg`, in album `albumID` as of time `t`.
|
||||
// With `corruptKey`, its key cannot be decrypted. With `metadata`, its
|
||||
// metadata decrypts to that text instead of the JSON a client uploads.
|
||||
putFile(
|
||||
albumID: number,
|
||||
id: number,
|
||||
t: number,
|
||||
opts: { corruptKey?: boolean; metadata?: string } = {},
|
||||
): void {
|
||||
const original = this.originalOf(id);
|
||||
const sealedKey = secretbox(original.key, this.albumKeys.get(albumID)!);
|
||||
const metadata =
|
||||
opts.metadata ??
|
||||
JSON.stringify({
|
||||
title: `photo-${id}.jpg`,
|
||||
fileType: 0,
|
||||
creationTime: TAKEN,
|
||||
modificationTime: TAKEN,
|
||||
});
|
||||
const sealedMetadata = encryptBlob(
|
||||
new TextEncoder().encode(metadata),
|
||||
original.key,
|
||||
);
|
||||
this.files.get(albumID)!.set(id, {
|
||||
id,
|
||||
collectionID: albumID,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: opts.corruptKey ? corruptKey() : sealedKey.ciphertext,
|
||||
keyDecryptionNonce: sealedKey.nonce,
|
||||
metadata: {
|
||||
encryptedData: toBase64(sealedMetadata.ciphertext),
|
||||
decryptionHeader: toBase64(sealedMetadata.header),
|
||||
},
|
||||
file: { decryptionHeader: original.header },
|
||||
thumbnail: { decryptionHeader: original.header },
|
||||
updationTime: t,
|
||||
});
|
||||
this.moveAlbum(albumID, t);
|
||||
}
|
||||
|
||||
// A deleted record stays on the server as a tombstone holding only its
|
||||
// ID, its time and `isDeleted`.
|
||||
deleteFile(albumID: number, id: number, t: number): void {
|
||||
this.files
|
||||
.get(albumID)!
|
||||
.set(id, { id, updationTime: t, isDeleted: true });
|
||||
this.moveAlbum(albumID, t);
|
||||
}
|
||||
|
||||
deleteAlbum(id: number, t: number): void {
|
||||
this.albums.set(id, { id, updationTime: t, isDeleted: true });
|
||||
}
|
||||
|
||||
// The `fetch` the saved session's client is given.
|
||||
readonly fetch = (async (input: string | URL | Request) => {
|
||||
const url = new URL(input instanceof Request ? input.url : input);
|
||||
const since = Number(url.searchParams.get("sinceTime"));
|
||||
const changed = (rows: Iterable<Row>): Row[] =>
|
||||
[...rows].filter((row) => row.updationTime > since);
|
||||
switch (url.pathname) {
|
||||
case "/collections/v2":
|
||||
return json({ collections: changed(this.albums.values()) });
|
||||
case "/collections/v2/diff": {
|
||||
const albumID = Number(url.searchParams.get("collectionID"));
|
||||
const rows = this.files.get(albumID)?.values() ?? [];
|
||||
return json({ diff: changed(rows), hasMore: false });
|
||||
}
|
||||
case "/files/data/fetch":
|
||||
// No file has ML data.
|
||||
return json({ data: [] });
|
||||
default: {
|
||||
const fileID = Number(url.searchParams.get("fileID"));
|
||||
const original = this.originals.get(fileID);
|
||||
return original
|
||||
? new Response(original.body)
|
||||
: new Response("not found", { status: 404 });
|
||||
}
|
||||
}
|
||||
}) as typeof globalThis.fetch;
|
||||
|
||||
private originalOf(id: number): {
|
||||
key: Uint8Array;
|
||||
header: string;
|
||||
body: Uint8Array;
|
||||
} {
|
||||
const known = this.originals.get(id);
|
||||
if (known) return known;
|
||||
const key = sodium.crypto_secretbox_keygen();
|
||||
const { header, ciphertext } = encryptBlob(
|
||||
new TextEncoder().encode(`original of ${id}`),
|
||||
key,
|
||||
);
|
||||
const original = { key, header: toBase64(header), body: ciphertext };
|
||||
this.originals.set(id, original);
|
||||
return original;
|
||||
}
|
||||
|
||||
private moveAlbum(albumID: number, t: number): void {
|
||||
this.albums.set(albumID, {
|
||||
...this.albums.get(albumID)!,
|
||||
updationTime: t,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Album 1, "Trip", holds file 10, which decrypts, file 11, whose key does not,
|
||||
// and file 12, whose metadata is not JSON. Album 2, "Family", holds file 20,
|
||||
// but the album's own key does not decrypt. All of it as of time 1.
|
||||
const brokenAccount = (): FakeEnte => {
|
||||
const ente = new FakeEnte();
|
||||
ente.putAlbum(1, "Trip", 1);
|
||||
ente.putFile(1, 10, 1);
|
||||
ente.putFile(1, 11, 1, { corruptKey: true });
|
||||
ente.putFile(1, 12, 1, { metadata: "not json" });
|
||||
ente.putAlbum(2, "Family", 1, true);
|
||||
ente.putFile(2, 20, 1);
|
||||
return ente;
|
||||
};
|
||||
|
||||
// Album 1, "Trip", holds file 10, and album 2, "Family", holds file 20. Both
|
||||
// files decrypt, but album 2's own key does not, so the album is the only
|
||||
// record left out. All of it as of time 1.
|
||||
const brokenAlbumAccount = (): FakeEnte => {
|
||||
const ente = new FakeEnte();
|
||||
ente.putAlbum(1, "Trip", 1);
|
||||
ente.putFile(1, 10, 1);
|
||||
ente.putAlbum(2, "Family", 1, true);
|
||||
ente.putFile(2, 20, 1);
|
||||
return ente;
|
||||
};
|
||||
|
||||
// Collects everything written to it.
|
||||
class Output {
|
||||
text = "";
|
||||
write(text: string): void {
|
||||
this.text += text;
|
||||
}
|
||||
}
|
||||
|
||||
let root: string;
|
||||
let stdout: Output;
|
||||
let stderr: Output;
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "quak-undecryptable-"));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// A context for one command, with fresh output, whose saved session talks to
|
||||
// `ente`. Every run shares one cache directory, as runs of `quak` do.
|
||||
const context = (ente: FakeEnte): CliContext => {
|
||||
stdout = new Output();
|
||||
stderr = new Output();
|
||||
saveSession(join(root, "session"), ente.snapshot());
|
||||
return {
|
||||
stdout,
|
||||
stderr,
|
||||
sessionDir: join(root, "session"),
|
||||
cacheDir: join(root, "cache"),
|
||||
loadSession: (path) =>
|
||||
loadSession(path, { fetch: ente.fetch, retry: { attempts: 1 } }),
|
||||
login: async () => {
|
||||
throw new Error("login not expected");
|
||||
},
|
||||
prompt: async () => {
|
||||
throw new Error("prompt not expected");
|
||||
},
|
||||
promptSecret: async () => {
|
||||
throw new Error("prompt not expected");
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
const backupDir = (): string => join(root, "backup");
|
||||
|
||||
// Runs `quak backup` into the backup directory and returns its exit code.
|
||||
const backup = (ente: FakeEnte, opts: { json?: boolean } = {}) =>
|
||||
backupCommand(context(ente), backupDir(), opts);
|
||||
|
||||
// The backup's `failures.json` entries by file ID, or {} when it has none.
|
||||
const failures = (): Record<string, { error: string; attempts: number }> => {
|
||||
const path = join(backupDir(), "failures.json");
|
||||
return existsSync(path)
|
||||
? JSON.parse(readFileSync(path, "utf-8")).files
|
||||
: {};
|
||||
};
|
||||
|
||||
// Whether file `id`'s original is at its save path in the backup.
|
||||
const stored = (id: number): boolean =>
|
||||
existsSync(join(backupDir(), DAY, `2026-03-01.${id}.jpg`));
|
||||
|
||||
// The IDs of the files that album `name`'s JSON lists.
|
||||
const albumFiles = (name: string): number[] =>
|
||||
JSON.parse(
|
||||
readFileSync(join(backupDir(), "collections", `${name}.json`), "utf-8"),
|
||||
).files.map((f: { id: number }) => f.id);
|
||||
|
||||
describe("quak backup with records it cannot decrypt", () => {
|
||||
it("backs up every other file, records each file it cannot decrypt in failures.json, names each such album, and exits 1", async () => {
|
||||
expect(await backup(brokenAccount())).toBe(1);
|
||||
|
||||
expect(stored(10)).toBe(true);
|
||||
// Without its album's key, file 20 cannot even be listed.
|
||||
expect(stored(20)).toBe(false);
|
||||
expect(failures()).toEqual({
|
||||
"11": expect.objectContaining({
|
||||
fileID: 11,
|
||||
error: expect.stringMatching(
|
||||
/^could not be decrypted: .*wrong secret key/,
|
||||
),
|
||||
}),
|
||||
"12": expect.objectContaining({
|
||||
fileID: 12,
|
||||
error: expect.stringMatching(/^could not be decrypted: .*JSON/),
|
||||
}),
|
||||
});
|
||||
expect(stderr.text).toMatch(
|
||||
/^FAILED album 2: could not be decrypted: .*wrong secret key/m,
|
||||
);
|
||||
expect(stderr.text).toContain(" Failed: 2\n");
|
||||
});
|
||||
|
||||
it("names each album it cannot decrypt in the --json result", async () => {
|
||||
expect(await backup(brokenAccount(), { json: true })).toBe(1);
|
||||
|
||||
expect(JSON.parse(stdout.text)).toMatchObject({
|
||||
downloaded: 1,
|
||||
failed: 2,
|
||||
undecryptableAlbums: [2],
|
||||
});
|
||||
});
|
||||
|
||||
it("still reports them on the next run, though the server does not send them again", async () => {
|
||||
const ente = brokenAccount();
|
||||
await backup(ente);
|
||||
|
||||
// Nothing changed on the server, so this run's refresh gets no rows at
|
||||
// all. What it reports, it remembered from the first run.
|
||||
expect(await backup(ente)).toBe(1);
|
||||
|
||||
expect(Object.keys(failures())).toEqual(["11", "12"]);
|
||||
expect(failures()["11"]!.attempts).toBe(2);
|
||||
expect(stderr.text).toMatch(/^FAILED album 2: could not be decrypted/m);
|
||||
});
|
||||
|
||||
it("drops each once the server sends a copy that decrypts, and then exits 0", async () => {
|
||||
const ente = brokenAccount();
|
||||
await backup(ente);
|
||||
|
||||
// At time 2 files 11 and 12 and album 2 change on the server, and now
|
||||
// each decrypts. Album 2 brings file 20 with it.
|
||||
ente.putFile(1, 11, 2);
|
||||
ente.putFile(1, 12, 2);
|
||||
ente.putAlbum(2, "Family", 2);
|
||||
expect(await backup(ente)).toBe(0);
|
||||
|
||||
expect([11, 12, 20].map(stored)).toEqual([true, true, true]);
|
||||
expect(existsSync(join(backupDir(), "failures.json"))).toBe(false);
|
||||
expect(stderr.text).not.toContain("FAILED");
|
||||
});
|
||||
|
||||
it("drops each once the server deletes it, and then exits 0", async () => {
|
||||
const ente = brokenAccount();
|
||||
await backup(ente);
|
||||
|
||||
ente.deleteFile(1, 11, 2);
|
||||
ente.deleteFile(1, 12, 2);
|
||||
ente.deleteAlbum(2, 2);
|
||||
expect(await backup(ente)).toBe(0);
|
||||
|
||||
expect(existsSync(join(backupDir(), "failures.json"))).toBe(false);
|
||||
expect(stderr.text).not.toContain("FAILED");
|
||||
});
|
||||
|
||||
it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album decrypts", async () => {
|
||||
const ente = brokenAlbumAccount();
|
||||
|
||||
// No file is in `failures.json`: the album alone keeps the exit code
|
||||
// at 1.
|
||||
expect(await backup(ente)).toBe(1);
|
||||
expect(failures()).toEqual({});
|
||||
|
||||
ente.putAlbum(2, "Family", 2);
|
||||
expect(await backup(ente)).toBe(0);
|
||||
expect(stored(20)).toBe(true);
|
||||
});
|
||||
|
||||
it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album is deleted", async () => {
|
||||
const ente = brokenAlbumAccount();
|
||||
|
||||
expect(await backup(ente)).toBe(1);
|
||||
expect(failures()).toEqual({});
|
||||
|
||||
ente.deleteAlbum(2, 2);
|
||||
expect(await backup(ente)).toBe(0);
|
||||
});
|
||||
|
||||
it("removes what an earlier run linked for an album or a file that no longer decrypts, and keeps the originals", async () => {
|
||||
// At time 1 everything decrypts: the first run links files 10 and 11
|
||||
// in Trip and file 20 in Family.
|
||||
const ente = new FakeEnte();
|
||||
ente.putAlbum(1, "Trip", 1);
|
||||
ente.putFile(1, 10, 1);
|
||||
ente.putFile(1, 11, 1);
|
||||
ente.putAlbum(2, "Family", 1);
|
||||
ente.putFile(2, 20, 1);
|
||||
expect(await backup(ente)).toBe(0);
|
||||
const collections = join(backupDir(), "collections");
|
||||
expect(readdirSync(join(collections, "Trip")).sort()).toEqual([
|
||||
"photo-10.jpg",
|
||||
"photo-11.jpg",
|
||||
]);
|
||||
expect(readdirSync(join(collections, "Family"))).toEqual([
|
||||
"photo-20.jpg",
|
||||
]);
|
||||
|
||||
// At time 2 the server's copy of file 11 in Trip, and album 2 itself,
|
||||
// no longer decrypt. A run builds the album folders, their JSON and
|
||||
// their links from what the library holds, so it removes them as if
|
||||
// the album and the file had been deleted: Family's folder and JSON
|
||||
// go, and Trip neither links nor lists file 11. The originals stay at
|
||||
// their save paths, and the run exits 1.
|
||||
ente.putFile(1, 11, 2, { corruptKey: true });
|
||||
ente.putAlbum(2, "Family", 2, true);
|
||||
expect(await backup(ente)).toBe(1);
|
||||
|
||||
expect(readdirSync(collections).sort()).toEqual(["Trip", "Trip.json"]);
|
||||
expect(readdirSync(join(collections, "Trip"))).toEqual([
|
||||
"photo-10.jpg",
|
||||
]);
|
||||
expect(albumFiles("Trip")).toEqual([10]);
|
||||
expect([10, 11, 20].map(stored)).toEqual([true, true, true]);
|
||||
expect(Object.keys(failures())).toEqual(["11"]);
|
||||
});
|
||||
|
||||
it("backs up a file that does not decrypt in one album but does in another, and does not report it", async () => {
|
||||
// File 30 is in albums 1 and 3, each holding its key sealed under the
|
||||
// album's own key. In album 1 that is corrupt; in album 3 it is not,
|
||||
// and one copy that decrypts is all the backup needs.
|
||||
const ente = new FakeEnte();
|
||||
ente.putAlbum(1, "Trip", 1);
|
||||
ente.putFile(1, 30, 1, { corruptKey: true });
|
||||
ente.putAlbum(3, "Favorites", 1);
|
||||
ente.putFile(3, 30, 1);
|
||||
|
||||
expect(await backup(ente)).toBe(0);
|
||||
|
||||
expect(stored(30)).toBe(true);
|
||||
expect(failures()).toEqual({});
|
||||
// Album 1 has no copy of file 30 that decrypts, so it neither links
|
||||
// the original nor lists the file; album 3 lists it.
|
||||
expect(readdirSync(join(backupDir(), "collections", "Trip"))).toEqual(
|
||||
[],
|
||||
);
|
||||
expect(albumFiles("Trip")).toEqual([]);
|
||||
expect(albumFiles("Favorites")).toEqual([30]);
|
||||
});
|
||||
|
||||
it("backs up that file in a run limited to the album where it does not decrypt", async () => {
|
||||
// The same account, backed up by `lib.backup()` limited to album 1,
|
||||
// "Trip", as `quak backup` cannot be. File 30 is in Trip, so the run
|
||||
// must store it, from its copy in album 3 that the run otherwise
|
||||
// skips.
|
||||
const ente = new FakeEnte();
|
||||
ente.putAlbum(1, "Trip", 1);
|
||||
ente.putFile(1, 30, 1, { corruptKey: true });
|
||||
ente.putAlbum(3, "Favorites", 1);
|
||||
ente.putFile(3, 30, 1);
|
||||
const lib = await Library.open({
|
||||
client: Client.fromJSON(ente.snapshot(), {
|
||||
fetch: ente.fetch,
|
||||
retry: { attempts: 1 },
|
||||
}),
|
||||
downloadDirectory: backupDir(),
|
||||
cacheDirectory: join(root, "cache"),
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: backupDir(),
|
||||
onlyAlbumNames: ["Trip"],
|
||||
});
|
||||
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
|
||||
expect(stored(30)).toBe(true);
|
||||
expect(failures()).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe("the read commands with records quak cannot decrypt", () => {
|
||||
it("quak collections and quak files answer from what decrypts", async () => {
|
||||
const ente = brokenAccount();
|
||||
|
||||
expect(await collectionsCommand(context(ente), {})).toBe(0);
|
||||
expect(stdout.text).toBe("1\talbum\tTrip\n");
|
||||
|
||||
expect(await filesCommand(context(ente), { collection: "1" })).toBe(0);
|
||||
expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
|
||||
});
|
||||
});
|
||||
@@ -1482,6 +1482,7 @@ describe("backup album folders", () => {
|
||||
listCollections: () => albums.map((a) => a.collection),
|
||||
listFiles: (id) =>
|
||||
albums.find((a) => a.collection.id === id)?.files ?? [],
|
||||
undecryptable: () => [],
|
||||
original: async (fileID) => {
|
||||
const path = join(root, `source-${fileID}`);
|
||||
writeFileSync(path, `original ${fileID}`);
|
||||
|
||||
+269
-12
@@ -25,6 +25,7 @@ import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { PassThrough } from "node:stream";
|
||||
import * as jpegJs from "jpeg-js";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import {
|
||||
describe,
|
||||
it,
|
||||
@@ -55,7 +56,7 @@ import { loadSession } from "../../src/cli-session.js";
|
||||
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
|
||||
import { savePath, type ContentSource } from "../../src/library/content.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { defaultCacheDirectory } from "../../src/library/index.js";
|
||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||
import {
|
||||
@@ -511,11 +512,6 @@ describe("files", () => {
|
||||
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
|
||||
expect(stderr.text).toBe("Collection 9 not found\n");
|
||||
});
|
||||
|
||||
it("exits 1 for a collection ID that is not a number", async () => {
|
||||
expect(await filesCommand(context(), { collection: "abc" })).toBe(1);
|
||||
expect(stderr.text).toBe("Invalid collection ID\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("get and get-thumb", () => {
|
||||
@@ -546,12 +542,6 @@ describe("get and get-thumb", () => {
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
|
||||
it("both exit 1 for a file ID that is not a number", async () => {
|
||||
expect(await getCommand(context(), "abc", {})).toBe(1);
|
||||
expect(await getThumbCommand(context(), "abc", {})).toBe(1);
|
||||
expect(stderr.text).toBe("Invalid file ID\nInvalid file ID\n");
|
||||
});
|
||||
});
|
||||
|
||||
// A live photo, which Ente stores as one ZIP, is written as its image and its
|
||||
@@ -914,6 +904,146 @@ describe("backup", () => {
|
||||
// Opening the library would have made its cache directory.
|
||||
expect(existsSync(join(root, "cache"))).toBe(false);
|
||||
});
|
||||
|
||||
// A backup runs for as long as its downloads take, hours on a large
|
||||
// account. It lists its files from the refresh it waits for as it starts,
|
||||
// so it must not ask the server for changes again on a timer while it
|
||||
// runs: each refresh is a request, and each also starts a pass that asks
|
||||
// again for the ML data of every file Ente holds none for.
|
||||
//
|
||||
// A saved session is read back by `loadSession`, its `fetch` a fake of the
|
||||
// Ente HTTP API serving one album with one photo, encrypted as the server
|
||||
// serves them. The fake holds the photo's download open while the clock
|
||||
// moves on by two hours, then serves it. Only `setTimeout` and
|
||||
// `clearTimeout` are faked: the library's refresh timer runs on them.
|
||||
it("asks the server for changes only as it starts, however long it runs", async () => {
|
||||
const masterKey = new Uint8Array(32).fill(1);
|
||||
const albumKey = new Uint8Array(32).fill(2);
|
||||
const photoKey = new Uint8Array(32).fill(3);
|
||||
saveSession(join(root, "session"), {
|
||||
email: "cli@example.com",
|
||||
userID: USER_ID,
|
||||
token: "token",
|
||||
masterKey: toBase64(masterKey),
|
||||
secretKey: toBase64(new Uint8Array(32)),
|
||||
publicKey: toBase64(new Uint8Array(32)),
|
||||
});
|
||||
// A secretbox as the server sends one: ciphertext and nonce, base64.
|
||||
const box = (plaintext: Uint8Array, key: Uint8Array) => {
|
||||
const nonce = sodium.randombytes_buf(
|
||||
sodium.crypto_secretbox_NONCEBYTES,
|
||||
);
|
||||
return {
|
||||
ciphertext: toBase64(
|
||||
sodium.crypto_secretbox_easy(plaintext, nonce, key),
|
||||
),
|
||||
nonce: toBase64(nonce),
|
||||
};
|
||||
};
|
||||
const albumKeyBox = box(albumKey, masterKey);
|
||||
const albumName = box(new TextEncoder().encode("Vacation"), albumKey);
|
||||
const photoKeyBox = box(photoKey, albumKey);
|
||||
const metadata = encryptBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title: "beach.jpg",
|
||||
fileType: 0,
|
||||
creationTime: 1_700_000_000_000_000,
|
||||
modificationTime: 1_700_000_000_000_000,
|
||||
}),
|
||||
),
|
||||
photoKey,
|
||||
);
|
||||
const content = encryptBlob(
|
||||
new TextEncoder().encode("the photo's bytes"),
|
||||
photoKey,
|
||||
);
|
||||
const album = {
|
||||
id: 1,
|
||||
owner: { id: USER_ID },
|
||||
encryptedKey: albumKeyBox.ciphertext,
|
||||
keyDecryptionNonce: albumKeyBox.nonce,
|
||||
encryptedName: albumName.ciphertext,
|
||||
nameDecryptionNonce: albumName.nonce,
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
};
|
||||
const photo = {
|
||||
id: 100,
|
||||
collectionID: 1,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: photoKeyBox.ciphertext,
|
||||
keyDecryptionNonce: photoKeyBox.nonce,
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
file: { decryptionHeader: toBase64(content.header) },
|
||||
// The backup fetches no thumbnail.
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: 1,
|
||||
};
|
||||
|
||||
// Each refresh starts with `collectionsSince`, which sends one
|
||||
// `/collections/v2` request.
|
||||
let refreshes = 0;
|
||||
let downloadRequested!: () => void;
|
||||
const requested = new Promise<void>((resolve) => {
|
||||
downloadRequested = resolve;
|
||||
});
|
||||
let releaseDownload!: () => void;
|
||||
const released = new Promise<void>((resolve) => {
|
||||
releaseDownload = resolve;
|
||||
});
|
||||
const json = (body: unknown): Response =>
|
||||
new Response(JSON.stringify(body), {
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
const fetch = async (url: string | URL): Promise<Response> => {
|
||||
const { hostname, pathname } = new URL(String(url));
|
||||
if (hostname === "files.ente.io") {
|
||||
downloadRequested();
|
||||
await released;
|
||||
return new Response(content.ciphertext);
|
||||
}
|
||||
if (pathname === "/collections/v2") {
|
||||
refreshes++;
|
||||
return json({ collections: [album] });
|
||||
}
|
||||
if (pathname === "/collections/v2/diff") {
|
||||
return json({ diff: [photo], hasMore: false });
|
||||
}
|
||||
// Ente holds no ML data for the photo.
|
||||
if (pathname === "/files/data/fetch") return json({ data: [] });
|
||||
return new Response("not found", { status: 404 });
|
||||
};
|
||||
const ctx = {
|
||||
...context(),
|
||||
// The download's idle deadline is set past the two hours, or quak
|
||||
// would give up the held download as stalled.
|
||||
loadSession: (path: string) =>
|
||||
loadSession(path, {
|
||||
fetch: fetch as typeof globalThis.fetch,
|
||||
downloadTimeoutMs: 3 * 3_600_000,
|
||||
}),
|
||||
};
|
||||
|
||||
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
|
||||
try {
|
||||
const backup = backupCommand(ctx, join(root, "backup"), {});
|
||||
await requested;
|
||||
vi.advanceTimersByTime(2 * 3_600_000);
|
||||
releaseDownload();
|
||||
expect(await backup).toBe(0);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
|
||||
// Two refreshes, both before the download: the library opens onto the
|
||||
// empty cache with one, and the backup waits for the other before it
|
||||
// lists its files.
|
||||
expect(refreshes).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("helper list-missing-thumbnails", () => {
|
||||
@@ -1035,3 +1165,130 @@ describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
||||
expect(stdout.text).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
// `files --collection`, `get`, `get-thumb` and `helper fix-missing-thumbnails
|
||||
// --file` take an ID only when it is written in decimal digits and is a safe
|
||||
// integer. `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty
|
||||
// string as 0, so the command would act on a file the user never named.
|
||||
// Anything else is refused with one line naming it, and exit 1, before any
|
||||
// request.
|
||||
//
|
||||
// These run on a real client, read back from a saved session, over a fake Ente
|
||||
// server holding an empty account, which records each request it receives. With
|
||||
// a good ID the command goes on to ask the server, and finds nothing there.
|
||||
describe("an ID given on the command line", () => {
|
||||
const NOT_IDS = [
|
||||
"0x10",
|
||||
"1e3",
|
||||
"1.5",
|
||||
"-1",
|
||||
"",
|
||||
"abc",
|
||||
// One past Number.MAX_SAFE_INTEGER: from here on a number cannot
|
||||
// hold every integer, so two IDs could read as one.
|
||||
"9007199254740992",
|
||||
];
|
||||
|
||||
const requests: Request[] = [];
|
||||
|
||||
const emptyAccount = (): CliContext => {
|
||||
const ctx = context();
|
||||
const key = toBase64(new Uint8Array(32));
|
||||
saveSession(ctx.sessionDir, {
|
||||
email: "cli@example.com",
|
||||
userID: USER_ID,
|
||||
token: "token",
|
||||
masterKey: key,
|
||||
secretKey: key,
|
||||
publicKey: key,
|
||||
});
|
||||
return {
|
||||
...ctx,
|
||||
loadSession: (path) =>
|
||||
loadSession(path, {
|
||||
fetch: async (url, init) => {
|
||||
requests.push(new Request(url, init));
|
||||
return new Response(
|
||||
JSON.stringify({ collections: [] }),
|
||||
{ headers: { "content-type": "application/json" } },
|
||||
);
|
||||
},
|
||||
}),
|
||||
};
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
requests.length = 0;
|
||||
});
|
||||
|
||||
it("files looks up a decimal collection ID", async () => {
|
||||
const ctx = emptyAccount();
|
||||
expect(await filesCommand(ctx, { collection: "1" })).toBe(1);
|
||||
expect(stderr.text).toBe("Collection 1 not found\n");
|
||||
expect(requests).not.toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(NOT_IDS)(
|
||||
"files refuses --collection %j before any request",
|
||||
async (arg) => {
|
||||
const ctx = emptyAccount();
|
||||
expect(await filesCommand(ctx, { collection: arg })).toBe(1);
|
||||
expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(requests).toHaveLength(0);
|
||||
},
|
||||
);
|
||||
|
||||
it("get looks up a decimal file ID", async () => {
|
||||
expect(await getCommand(emptyAccount(), "100", {})).toBe(1);
|
||||
expect(stderr.text).toBe("File 100 not found\n");
|
||||
expect(requests).not.toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(NOT_IDS)("get refuses %j before any request", async (arg) => {
|
||||
expect(await getCommand(emptyAccount(), arg, {})).toBe(1);
|
||||
expect(stderr.text).toBe(`Invalid file ID: ${arg}\n`);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(requests).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("get-thumb looks up a decimal file ID", async () => {
|
||||
expect(await getThumbCommand(emptyAccount(), "100", {})).toBe(1);
|
||||
expect(stderr.text).toBe("File 100 not found\n");
|
||||
expect(requests).not.toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(NOT_IDS)("get-thumb refuses %j before any request", async (arg) => {
|
||||
expect(await getThumbCommand(emptyAccount(), arg, {})).toBe(1);
|
||||
expect(stderr.text).toBe(`Invalid file ID: ${arg}\n`);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(requests).toHaveLength(0);
|
||||
});
|
||||
|
||||
// The fixer reports each file it was given; on this account neither is
|
||||
// found, which is a failure, so it exits 1.
|
||||
it("fix-missing-thumbnails looks up each decimal --file ID", async () => {
|
||||
const ctx = emptyAccount();
|
||||
const opts = { file: ["100", "200"], json: true };
|
||||
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(1);
|
||||
expect(JSON.parse(stdout.text)).toMatchObject([
|
||||
{ fileID: 100, reason: "file not found in any collection" },
|
||||
{ fileID: 200, reason: "file not found in any collection" },
|
||||
]);
|
||||
expect(requests).not.toHaveLength(0);
|
||||
});
|
||||
|
||||
// One bad ID fails the whole command, so the good one before it is not
|
||||
// fixed either.
|
||||
it.each(NOT_IDS)(
|
||||
"fix-missing-thumbnails refuses --file 100 %j before any request",
|
||||
async (arg) => {
|
||||
const ctx = emptyAccount();
|
||||
const opts = { file: ["100", arg] };
|
||||
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(1);
|
||||
expect(stderr.text).toBe(`Invalid file ID: ${arg}\n`);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(requests).toHaveLength(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -37,15 +37,8 @@ import {
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { SRP, SrpServer } from "fast-srp-hap";
|
||||
import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
init,
|
||||
toBase64,
|
||||
deriveKEK,
|
||||
deriveLoginSubkey,
|
||||
encryptBlob,
|
||||
} from "../../src/crypto/index.js";
|
||||
import { init, toBase64, encryptBlob } from "../../src/crypto/index.js";
|
||||
import * as jpegJs from "jpeg-js";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
@@ -54,7 +47,13 @@ import {
|
||||
type MetadataBackupOptions,
|
||||
} from "../../src/metadata-backup.js";
|
||||
import { backupMetadataCommand } from "../../src/cli-commands.js";
|
||||
import type { KeyAttributes } from "../../src/auth/types.js";
|
||||
import {
|
||||
buildAccount,
|
||||
buildMetaFetch,
|
||||
encryptSecretbox,
|
||||
encryptStreamBlob,
|
||||
type MetaMockState,
|
||||
} from "../ente-server.js";
|
||||
|
||||
// One file per ML data request, so the two files of the mock account are
|
||||
// fetched in two requests and one of them can fail on its own.
|
||||
@@ -65,80 +64,13 @@ vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
|
||||
|
||||
const TEST_EMAIL = "metabackup@example.com";
|
||||
const TEST_PASSWORD = "metapass";
|
||||
const TEST_OPS = 2;
|
||||
const TEST_MEM = 64 * 1024 * 1024;
|
||||
|
||||
interface MetaMockState {
|
||||
verifier: Buffer;
|
||||
srpAttributes: Record<string, unknown>;
|
||||
keyAttributes: KeyAttributes;
|
||||
encryptedToken: string;
|
||||
collections: Record<string, unknown>[];
|
||||
filesByCollection: Record<number, Record<string, unknown>[]>;
|
||||
// For ML data and EXIF tests
|
||||
encryptedMLData: Record<
|
||||
number,
|
||||
{ encryptedData: string; decryptionHeader: string }
|
||||
>;
|
||||
fileCiphertexts: Record<number, Uint8Array>;
|
||||
}
|
||||
|
||||
let mock: MetaMockState;
|
||||
let testDir: string;
|
||||
|
||||
const encryptSecretbox = (plaintext: Uint8Array, key: Uint8Array) => {
|
||||
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
|
||||
const ciphertext = sodium.crypto_secretbox_easy(plaintext, nonce, key);
|
||||
return { ciphertext, nonce };
|
||||
};
|
||||
|
||||
const encryptStreamBlob = (plaintext: Uint8Array, key: Uint8Array) => {
|
||||
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
|
||||
const ciphertext = sodium.crypto_secretstream_xchacha20poly1305_push(
|
||||
push.state,
|
||||
plaintext,
|
||||
null,
|
||||
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
|
||||
);
|
||||
return { ciphertext, header: push.header };
|
||||
};
|
||||
|
||||
const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
|
||||
const kek = await deriveKEK(TEST_PASSWORD, kekSalt, TEST_OPS, TEST_MEM);
|
||||
const loginSubKeyBytes = deriveLoginSubkey(kek);
|
||||
const srpUserID = "meta-srp";
|
||||
const srpSalt = sodium.randombytes_buf(16);
|
||||
const verifier = SRP.computeVerifier(
|
||||
SRP.params["4096"],
|
||||
Buffer.from(srpSalt),
|
||||
Buffer.from(srpUserID),
|
||||
Buffer.from(loginSubKeyBytes),
|
||||
);
|
||||
|
||||
const masterKey = sodium.randombytes_buf(32);
|
||||
const { ciphertext: encMK, nonce: mkNonce } = encryptSecretbox(
|
||||
masterKey,
|
||||
kek,
|
||||
);
|
||||
const kp = sodium.crypto_box_keypair();
|
||||
const { ciphertext: encSK, nonce: skNonce } = encryptSecretbox(
|
||||
kp.privateKey,
|
||||
masterKey,
|
||||
);
|
||||
const tokenBytes = sodium.randombytes_buf(32);
|
||||
const encToken = sodium.crypto_box_seal(tokenBytes, kp.publicKey);
|
||||
|
||||
const keyAttributes: KeyAttributes = {
|
||||
kekSalt: toBase64(kekSalt),
|
||||
encryptedKey: toBase64(encMK),
|
||||
keyDecryptionNonce: toBase64(mkNonce),
|
||||
publicKey: toBase64(kp.publicKey),
|
||||
encryptedSecretKey: toBase64(encSK),
|
||||
secretKeyDecryptionNonce: toBase64(skNonce),
|
||||
memLimit: TEST_MEM,
|
||||
opsLimit: TEST_OPS,
|
||||
};
|
||||
const account = await buildAccount(TEST_PASSWORD);
|
||||
const masterKey = account.masterKey;
|
||||
|
||||
// Collection 1: "Vacation" with collection-level pubMagicMetadata
|
||||
const ck1 = sodium.crypto_secretbox_keygen();
|
||||
@@ -332,17 +264,10 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
rawFile1.file.decryptionHeader = toBase64(filePush1.header);
|
||||
|
||||
return {
|
||||
verifier,
|
||||
srpAttributes: {
|
||||
srpUserID,
|
||||
srpSalt: toBase64(srpSalt),
|
||||
memLimit: TEST_MEM,
|
||||
opsLimit: TEST_OPS,
|
||||
kekSalt: toBase64(kekSalt),
|
||||
isEmailMFAEnabled: false,
|
||||
},
|
||||
keyAttributes,
|
||||
encryptedToken: toBase64(encToken),
|
||||
verifier: account.verifier,
|
||||
srpAttributes: account.srpAttributes,
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections: [rawColl1, rawColl2],
|
||||
filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
|
||||
encryptedMLData: {
|
||||
@@ -355,90 +280,6 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
};
|
||||
};
|
||||
|
||||
// `failMLDataFor`: answer 500 to every ML data request that asks for this file.
|
||||
const buildMetaFetch = (m: MetaMockState, failMLDataFor?: number) => {
|
||||
let srpServer: SrpServer;
|
||||
return (async (
|
||||
input: RequestInfo | URL,
|
||||
init?: RequestInit,
|
||||
): Promise<Response> => {
|
||||
const url =
|
||||
typeof input === "string"
|
||||
? input
|
||||
: input instanceof URL
|
||||
? input.href
|
||||
: input.url;
|
||||
const path = new URL(url).pathname;
|
||||
const json = (body: unknown) =>
|
||||
new Response(JSON.stringify(body), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
|
||||
if (path === "/users/srp/attributes")
|
||||
return json({ attributes: m.srpAttributes });
|
||||
if (path === "/users/srp/create-session") {
|
||||
const body = JSON.parse(init?.body as string);
|
||||
const serverKey = await SRP.genKey();
|
||||
srpServer = new SrpServer(
|
||||
SRP.params["4096"],
|
||||
m.verifier,
|
||||
serverKey,
|
||||
);
|
||||
const B = srpServer.computeB();
|
||||
srpServer.setA(Buffer.from(body.srpA, "base64"));
|
||||
return json({ sessionID: "s1", srpB: B.toString("base64") });
|
||||
}
|
||||
if (path === "/users/srp/verify-session") {
|
||||
const body = JSON.parse(init?.body as string);
|
||||
srpServer.checkM1(Buffer.from(body.srpM1, "base64"));
|
||||
return json({
|
||||
srpM2: srpServer.computeM2().toString("base64"),
|
||||
id: 42,
|
||||
keyAttributes: m.keyAttributes,
|
||||
encryptedToken: m.encryptedToken,
|
||||
});
|
||||
}
|
||||
if (path === "/collections/v2")
|
||||
return json({ collections: m.collections });
|
||||
if (path === "/collections/v2/diff") {
|
||||
const collID = Number(
|
||||
new URL(url).searchParams.get("collectionID"),
|
||||
);
|
||||
return json({
|
||||
diff: m.filesByCollection[collID] ?? [],
|
||||
hasMore: false,
|
||||
});
|
||||
}
|
||||
if (path === "/files/data/fetch") {
|
||||
const body = JSON.parse(init?.body as string);
|
||||
if ((body.fileIDs as number[]).includes(failMLDataFor!))
|
||||
return new Response("server error", { status: 500 });
|
||||
const data = (body.fileIDs as number[])
|
||||
.filter((id: number) => m.encryptedMLData[id])
|
||||
.map((id: number) => ({
|
||||
fileID: id,
|
||||
...m.encryptedMLData[id],
|
||||
updatedAt: 1700000000000000,
|
||||
}));
|
||||
return json({ data });
|
||||
}
|
||||
if (
|
||||
url.includes("files.ente.io") ||
|
||||
path.startsWith("/files/download/")
|
||||
) {
|
||||
const parsed = new URL(url);
|
||||
const fileID = Number(
|
||||
parsed.searchParams.get("fileID") ?? path.split("/").pop(),
|
||||
);
|
||||
const ct = m.fileCiphertexts[fileID];
|
||||
if (ct) return new Response(ct, { status: 200 });
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
return new Response("not found", { status: 404 });
|
||||
}) as typeof globalThis.fetch;
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
|
||||
@@ -10,7 +10,9 @@
|
||||
* The return shape keeps live records and tombstones apart — `collections` /
|
||||
* `files` are decrypted live records, `deleted` is a plain list of the ids the
|
||||
* server tombstoned. A tombstone carries no decryptable key or metadata, so it
|
||||
* is a bare id rather than a hollowed-out `Collection` / `EnteFile`.
|
||||
* is a bare id rather than a hollowed-out `Collection` / `EnteFile`. A live row
|
||||
* that does not decrypt, or whose metadata is not JSON, is listed under
|
||||
* `undecryptable` with the reason, so one bad row does not fail the call.
|
||||
*
|
||||
* All tests inject a fake `fetch` and drive a real `Client` (built with
|
||||
* `Client.fromJSON`) so the decryption path runs for real. Live rows are built
|
||||
@@ -91,27 +93,38 @@ const ownedCollectionRow = (
|
||||
};
|
||||
};
|
||||
|
||||
/** A live file row inside a collection, keyed under that collection's key. */
|
||||
/**
|
||||
* A live file row inside a collection, keyed under that collection's key. Its
|
||||
* metadata decrypts to `metadata` when given, otherwise to the JSON a client
|
||||
* uploads for `title`.
|
||||
*/
|
||||
const fileRow = (
|
||||
collectionKey: Uint8Array,
|
||||
opts: { id: number; title: string; updationTime: number },
|
||||
opts: {
|
||||
id: number;
|
||||
title: string;
|
||||
updationTime: number;
|
||||
metadata?: string;
|
||||
},
|
||||
): Record<string, unknown> => {
|
||||
const fileKey = sodium.crypto_secretbox_keygen();
|
||||
const { ciphertext: encFileKey, nonce: fileKeyNonce } = secretboxEncrypt(
|
||||
fileKey,
|
||||
collectionKey,
|
||||
);
|
||||
const metadata = {
|
||||
const metadata =
|
||||
opts.metadata ??
|
||||
JSON.stringify({
|
||||
title: opts.title,
|
||||
fileType: 0,
|
||||
creationTime: opts.updationTime,
|
||||
modificationTime: opts.updationTime,
|
||||
};
|
||||
});
|
||||
const push =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_init_push(fileKey);
|
||||
const encMeta = sodium.crypto_secretstream_xchacha20poly1305_push(
|
||||
push.state,
|
||||
new TextEncoder().encode(JSON.stringify(metadata)),
|
||||
new TextEncoder().encode(metadata),
|
||||
null,
|
||||
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
|
||||
);
|
||||
@@ -131,6 +144,21 @@ const fileRow = (
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* `row` with its sealed key replaced by random bytes of the same length, as a
|
||||
* corrupt row arrives: no key opens it, so the row cannot be decrypted.
|
||||
*/
|
||||
const withCorruptKey = (
|
||||
row: Record<string, unknown>,
|
||||
): Record<string, unknown> => ({
|
||||
...row,
|
||||
encryptedKey: toBase64(
|
||||
sodium.randombytes_buf(
|
||||
sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES,
|
||||
),
|
||||
),
|
||||
});
|
||||
|
||||
/** A tombstone row. Never decrypted, so only these fields are ever read. */
|
||||
const tombstoneRow = (
|
||||
id: number,
|
||||
@@ -278,6 +306,58 @@ describe("Client.filesSince", () => {
|
||||
// It gave up after the first page rather than looping.
|
||||
expect(sinceTimes).toEqual(["50"]);
|
||||
});
|
||||
|
||||
it("leaves out each file it cannot decrypt or parse, with the reason, and decrypts the rest", async () => {
|
||||
const keys = buildKeys();
|
||||
const collectionKey = sodium.crypto_secretbox_keygen();
|
||||
|
||||
// File 1002's key is sealed under no key quak holds, and file 1003's
|
||||
// metadata decrypts to text that is not JSON. Neither may cost the
|
||||
// caller file 1001: each is left out of `files` and reported under
|
||||
// `undecryptable` with the reason instead of failing the call.
|
||||
const { fetch } = recordingFetch(
|
||||
jsonResponse({
|
||||
diff: [
|
||||
fileRow(collectionKey, {
|
||||
id: 1001,
|
||||
title: "fine.jpg",
|
||||
updationTime: 100,
|
||||
}),
|
||||
withCorruptKey(
|
||||
fileRow(collectionKey, {
|
||||
id: 1002,
|
||||
title: "bad-key.jpg",
|
||||
updationTime: 200,
|
||||
}),
|
||||
),
|
||||
fileRow(collectionKey, {
|
||||
id: 1003,
|
||||
title: "bad-metadata.jpg",
|
||||
updationTime: 300,
|
||||
metadata: "not json",
|
||||
}),
|
||||
],
|
||||
hasMore: false,
|
||||
}),
|
||||
);
|
||||
const client = Client.fromJSON(snapshotFor(keys), { fetch });
|
||||
|
||||
const { files, undecryptable, cursor } = await client.filesSince({
|
||||
collectionID: 1,
|
||||
collectionKey,
|
||||
sinceTime: 0,
|
||||
});
|
||||
|
||||
expect(files.map((f) => f.id)).toEqual([1001]);
|
||||
expect(undecryptable).toEqual([
|
||||
{ id: 1002, error: expect.stringContaining("wrong secret key") },
|
||||
{ id: 1003, error: expect.stringContaining("JSON") },
|
||||
]);
|
||||
// The cursor moves past the two left out, as past any other row, so
|
||||
// the server does not send them again until its copy changes. The
|
||||
// caller has to keep what it was told here.
|
||||
expect(cursor).toBe(300);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Client.collectionsSince", () => {
|
||||
@@ -331,6 +411,42 @@ describe("Client.collectionsSince", () => {
|
||||
expect(result.cursor).toBe(150);
|
||||
expect(sinceTimes).toEqual(["150"]);
|
||||
});
|
||||
|
||||
it("leaves out an album it cannot decrypt, with the reason, and decrypts the rest", async () => {
|
||||
const keys = buildKeys();
|
||||
|
||||
// Album 2's key is sealed under no key quak holds. Without its key
|
||||
// neither its name nor its files can be read, so it is left out and
|
||||
// reported under `undecryptable`; album 1 is unaffected.
|
||||
const { fetch } = recordingFetch(
|
||||
jsonResponse({
|
||||
collections: [
|
||||
ownedCollectionRow(keys.masterKey, {
|
||||
id: 1,
|
||||
name: "Vacation",
|
||||
updationTime: 100,
|
||||
}),
|
||||
withCorruptKey(
|
||||
ownedCollectionRow(keys.masterKey, {
|
||||
id: 2,
|
||||
name: "Broken",
|
||||
updationTime: 200,
|
||||
}),
|
||||
),
|
||||
],
|
||||
}),
|
||||
);
|
||||
const client = Client.fromJSON(snapshotFor(keys), { fetch });
|
||||
|
||||
const { collections, undecryptable, cursor } =
|
||||
await client.collectionsSince({ sinceTime: 0 });
|
||||
|
||||
expect(collections.map((c) => c.name)).toEqual(["Vacation"]);
|
||||
expect(undecryptable).toEqual([
|
||||
{ id: 2, error: expect.stringContaining("wrong secret key") },
|
||||
]);
|
||||
expect(cursor).toBe(200);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Client list wrappers still hide deletions", () => {
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
/**
|
||||
* A fake of the Ente HTTP API, for tests that run quak's real `Client` against
|
||||
* a server.
|
||||
*
|
||||
* `buildMetaFetch` is a `fetch` that answers the requests quak makes, from a
|
||||
* `MetaMockState` the test builds: the SRP login, `/collections/v2`, each
|
||||
* collection's `/collections/v2/diff`, the ML data of `/files/data/fetch`, and
|
||||
* each file's content from the files CDN, a 404 for a file it holds none for.
|
||||
* The test encrypts everything in that state with libsodium, as Ente's clients
|
||||
* do, so quak decrypts it with its own code. Nothing hits the network.
|
||||
*
|
||||
* `buildAccount` builds what the server holds to log an account in, and the
|
||||
* keys a test encrypts that account's collections with. Call `init()` from
|
||||
* `src/crypto` and await `sodium.ready` before using either.
|
||||
*/
|
||||
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { SRP, SrpServer } from "fast-srp-hap";
|
||||
import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
|
||||
import type { KeyAttributes } from "../src/auth/types.js";
|
||||
|
||||
const TEST_OPS = 2;
|
||||
const TEST_MEM = 64 * 1024 * 1024;
|
||||
|
||||
export interface MetaMockState {
|
||||
verifier: Buffer;
|
||||
srpAttributes: Record<string, unknown>;
|
||||
keyAttributes: KeyAttributes;
|
||||
encryptedToken: string;
|
||||
collections: Record<string, unknown>[];
|
||||
filesByCollection: Record<number, Record<string, unknown>[]>;
|
||||
// For ML data and EXIF tests
|
||||
encryptedMLData: Record<
|
||||
number,
|
||||
{ encryptedData: string; decryptionHeader: string }
|
||||
>;
|
||||
fileCiphertexts: Record<number, Uint8Array>;
|
||||
}
|
||||
|
||||
export const encryptSecretbox = (plaintext: Uint8Array, key: Uint8Array) => {
|
||||
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
|
||||
const ciphertext = sodium.crypto_secretbox_easy(plaintext, nonce, key);
|
||||
return { ciphertext, nonce };
|
||||
};
|
||||
|
||||
export const encryptStreamBlob = (plaintext: Uint8Array, key: Uint8Array) => {
|
||||
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
|
||||
const ciphertext = sodium.crypto_secretstream_xchacha20poly1305_push(
|
||||
push.state,
|
||||
plaintext,
|
||||
null,
|
||||
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
|
||||
);
|
||||
return { ciphertext, header: push.header };
|
||||
};
|
||||
|
||||
// An account whose password is `password`. The first four fields are those of
|
||||
// `MetaMockState`; with the master key the test encrypts the keys of the
|
||||
// account's own collections, and to the public key it seals the keys of the
|
||||
// collections other accounts share with it.
|
||||
export const buildAccount = async (password: string) => {
|
||||
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
|
||||
const kek = await deriveKEK(password, kekSalt, TEST_OPS, TEST_MEM);
|
||||
const loginSubKeyBytes = deriveLoginSubkey(kek);
|
||||
const srpUserID = "meta-srp";
|
||||
const srpSalt = sodium.randombytes_buf(16);
|
||||
const verifier = SRP.computeVerifier(
|
||||
SRP.params["4096"],
|
||||
Buffer.from(srpSalt),
|
||||
Buffer.from(srpUserID),
|
||||
Buffer.from(loginSubKeyBytes),
|
||||
);
|
||||
|
||||
const masterKey = sodium.randombytes_buf(32);
|
||||
const { ciphertext: encMK, nonce: mkNonce } = encryptSecretbox(
|
||||
masterKey,
|
||||
kek,
|
||||
);
|
||||
const kp = sodium.crypto_box_keypair();
|
||||
const { ciphertext: encSK, nonce: skNonce } = encryptSecretbox(
|
||||
kp.privateKey,
|
||||
masterKey,
|
||||
);
|
||||
const tokenBytes = sodium.randombytes_buf(32);
|
||||
const encToken = sodium.crypto_box_seal(tokenBytes, kp.publicKey);
|
||||
|
||||
const keyAttributes: KeyAttributes = {
|
||||
kekSalt: toBase64(kekSalt),
|
||||
encryptedKey: toBase64(encMK),
|
||||
keyDecryptionNonce: toBase64(mkNonce),
|
||||
publicKey: toBase64(kp.publicKey),
|
||||
encryptedSecretKey: toBase64(encSK),
|
||||
secretKeyDecryptionNonce: toBase64(skNonce),
|
||||
memLimit: TEST_MEM,
|
||||
opsLimit: TEST_OPS,
|
||||
};
|
||||
|
||||
return {
|
||||
verifier,
|
||||
srpAttributes: {
|
||||
srpUserID,
|
||||
srpSalt: toBase64(srpSalt),
|
||||
memLimit: TEST_MEM,
|
||||
opsLimit: TEST_OPS,
|
||||
kekSalt: toBase64(kekSalt),
|
||||
isEmailMFAEnabled: false,
|
||||
},
|
||||
keyAttributes,
|
||||
encryptedToken: toBase64(encToken),
|
||||
masterKey,
|
||||
publicKey: kp.publicKey,
|
||||
};
|
||||
};
|
||||
|
||||
// `failMLDataFor`: answer 500 to every ML data request that asks for this file.
|
||||
export const buildMetaFetch = (m: MetaMockState, failMLDataFor?: number) => {
|
||||
let srpServer: SrpServer;
|
||||
return (async (
|
||||
input: RequestInfo | URL,
|
||||
init?: RequestInit,
|
||||
): Promise<Response> => {
|
||||
const url =
|
||||
typeof input === "string"
|
||||
? input
|
||||
: input instanceof URL
|
||||
? input.href
|
||||
: input.url;
|
||||
const path = new URL(url).pathname;
|
||||
const json = (body: unknown) =>
|
||||
new Response(JSON.stringify(body), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
|
||||
if (path === "/users/srp/attributes")
|
||||
return json({ attributes: m.srpAttributes });
|
||||
if (path === "/users/srp/create-session") {
|
||||
const body = JSON.parse(init?.body as string);
|
||||
const serverKey = await SRP.genKey();
|
||||
srpServer = new SrpServer(
|
||||
SRP.params["4096"],
|
||||
m.verifier,
|
||||
serverKey,
|
||||
);
|
||||
const B = srpServer.computeB();
|
||||
srpServer.setA(Buffer.from(body.srpA, "base64"));
|
||||
return json({ sessionID: "s1", srpB: B.toString("base64") });
|
||||
}
|
||||
if (path === "/users/srp/verify-session") {
|
||||
const body = JSON.parse(init?.body as string);
|
||||
srpServer.checkM1(Buffer.from(body.srpM1, "base64"));
|
||||
return json({
|
||||
srpM2: srpServer.computeM2().toString("base64"),
|
||||
id: 42,
|
||||
keyAttributes: m.keyAttributes,
|
||||
encryptedToken: m.encryptedToken,
|
||||
});
|
||||
}
|
||||
if (path === "/collections/v2")
|
||||
return json({ collections: m.collections });
|
||||
if (path === "/collections/v2/diff") {
|
||||
const collID = Number(
|
||||
new URL(url).searchParams.get("collectionID"),
|
||||
);
|
||||
return json({
|
||||
diff: m.filesByCollection[collID] ?? [],
|
||||
hasMore: false,
|
||||
});
|
||||
}
|
||||
if (path === "/files/data/fetch") {
|
||||
const body = JSON.parse(init?.body as string);
|
||||
if ((body.fileIDs as number[]).includes(failMLDataFor!))
|
||||
return new Response("server error", { status: 500 });
|
||||
const data = (body.fileIDs as number[])
|
||||
.filter((id: number) => m.encryptedMLData[id])
|
||||
.map((id: number) => ({
|
||||
fileID: id,
|
||||
...m.encryptedMLData[id],
|
||||
updatedAt: 1700000000000000,
|
||||
}));
|
||||
return json({ data });
|
||||
}
|
||||
if (
|
||||
url.includes("files.ente.io") ||
|
||||
path.startsWith("/files/download/")
|
||||
) {
|
||||
const parsed = new URL(url);
|
||||
const fileID = Number(
|
||||
parsed.searchParams.get("fileID") ?? path.split("/").pop(),
|
||||
);
|
||||
const ct = m.fileCiphertexts[fileID];
|
||||
if (ct) return new Response(ct, { status: 200 });
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
return new Response("not found", { status: 404 });
|
||||
}) as typeof globalThis.fetch;
|
||||
};
|
||||
@@ -172,6 +172,29 @@ describe("MetadataStore", () => {
|
||||
expect(store.listCollections()).toEqual([]);
|
||||
});
|
||||
|
||||
it("loads an empty store when the list of what the refresh could not decrypt is damaged", async () => {
|
||||
// A backup reads that list on every run, so a damaged one that loaded
|
||||
// would fail every backup until the cache was deleted. A string is
|
||||
// damaged too, though JavaScript can loop over its characters.
|
||||
mkdirSync(join(dir, "cache"), { recursive: true });
|
||||
for (const undecryptable of [{}, "not a list", [1]]) {
|
||||
writeFileSync(
|
||||
path,
|
||||
JSON.stringify({
|
||||
schemaVersion: METADATA_SCHEMA_VERSION,
|
||||
userID: 42,
|
||||
collectionsSinceTime: 0,
|
||||
collections: [],
|
||||
files: [],
|
||||
undecryptable,
|
||||
}),
|
||||
);
|
||||
const store = await MetadataStore.load(path);
|
||||
expect(store.userID).toBe(0);
|
||||
expect(store.listUndecryptable()).toEqual([]);
|
||||
}
|
||||
});
|
||||
|
||||
it("creates the directory 0700 and the file 0600", async () => {
|
||||
const store = await MetadataStore.load(path);
|
||||
store.putCollection(sampleCollection());
|
||||
|
||||
Reference in New Issue
Block a user