sneak/quak
1
0
Fork 0
Code Issues 8 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak/quak:issue-107-live-photos
Branches Tags
sneak/quak:main sneak/quak:issue-107-live-photos sneak/quak:next2 sneak/quak:next
..
compare: sneak/quak:a92e4929f8a78426a7f892903f3a083d9264a9d1
Branches Tags
sneak/quak:issue-107-live-photos sneak/quak:next2 sneak/quak:next sneak/quak:main
1 Commits
issue-107-live-photos .. a92e4929f8
Author SHA1 Message Date
clawbot a92e4929f8 Make lint and test phases of the Dockerfile (closes #96)
check / check (push) Successful in 5m22s
Details 
Follows the template: Dockerfile.lint is gone; the Dockerfile has a lint
phase (eslint, prettier --check .) and a test phase (vitest), and its last
stage compiles and depends on both. script/lint and script/test build one
phase each with --no-cache; script/docker and script/cibuild pass
--no-cache, so CHECK_EPOCH and LINT_EPOCH are removed. script/cibuild is
the single image build, so CI runs lint and the tests once each. The tests
that checked the old layout are deleted, REPO_POLICIES.md is re-copied and
the README describes the new layout.

Model: opus-5-5
 2026-09-23 03:38:46 +00:00
35 changed files with 399 additions and 3454 deletions
Expand all files Collapse all files
Dockerfile
-6
View File
@@ -31,12 +31,6 @@ RUN script/bootstrap
COPY . .
# Unlike the template, the suite runs as the image's non-root `node` user:
# root ignores directory permissions, so the tests of a destination that is
# not writable would otherwise fail. vitest writes into /app.
RUN chown -R node:node /app
USER node
RUN timeout 90 yarn run vitest run --reporter=dot || \
{ echo "--- Rerunning with verbose for details ---"; \
timeout 90 yarn run vitest run --reporter=verbose; exit 1; }
README.md
+41 -151
View File
@@ -214,23 +214,11 @@ quak/
auth/ login flow (SRP + email OTP + TOTP), key unwrap
model/ decrypted Collection, File, Metadata types + decrypt fns
download/ streaming file/thumbnail download + decryption
library/ the cache-backed Library: metadata store, read
surface, records, content cache, precache, ML data
and search, request pools
backup.ts resilient full-account backup with dedup
metadata-backup.ts
backup-metadata: all decrypted metadata as JSON
mldata-fetch.ts fetch + decrypt per-file ML data
filename.ts safe file names from server metadata
errors.ts error types shared across layers
retry.ts retry classifier + exponential backoff with jitter
thumbnails.ts detect + regenerate missing thumbnails
client.ts high-level Client class assembled from the above
cli-commands.ts the CLI's commands as functions returning exit codes
cli-output.ts how the CLI prints a file's title and time
cli-read.ts fresh reads for the CLI's read commands
cli-run.ts run a command, print its error, exit with its code
cli-session.ts read the saved session file back into a Client
index.ts public library exports
bin/
quak.ts CLI entrypoint (commander.js)
@@ -308,7 +296,6 @@ Endpoints used:
encrypted token plus key attributes.
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
- `POST /users/two-factor/verify`: TOTP second factor.
- `POST /users/logout`: end the calling token's session (`quak logout`).
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
microsecond timestamp; pass 0 for a full enumeration.
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
@@ -423,9 +410,7 @@ whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating; it checks every
field and each key's length first, and throws an error naming the bad field.
`client.logout()` clears the token and zeroes the key buffers in place; every
later call on that client throws. It does not contact the server, so the token
stays valid there and in any saved snapshot; `await client.logoutOnServer()`
first ends the session on the server (`POST /users/logout`).
later call on that client throws.
The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
@@ -435,21 +420,13 @@ you would treat the password itself. A missing file is reported as "not logged
in"; a file that exists but is corrupt is reported as such, naming the bad
field. Both exit with status 1.
`quak logout` ends the session on the server, so the token in `session.json`
stops working even in a copy of the file, and then deletes the file. If the
server call fails (or the file is corrupt), the file is still deleted, the
command says the server session could not be ended, and it exits with status 1.
It does not delete the cache: it prints the account's cache directory and says
it still holds decrypted data (file keys in `metadata.json`, cached originals
and thumbnails), for the user to delete if they want it gone.
### CLI surface
```
quak [--cache-dir <path>] <command> global: local metadata/content cache location
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
quak whoami print logged-in account as JSON
quak logout end the session, delete it
quak logout delete saved session
quak collections [--json] list all collections
quak files --collection <id> [--json] list files in a collection
quak get <fileID> [--out path] [--collection] download and decrypt a file
@@ -461,40 +438,22 @@ quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbn
```
Every command runs on the same cache-backed library. The read commands —
`collections`, `files`, `get`, `get-thumb`, `backup-metadata`,
`helper list-missing-thumbnails` and `helper fix-missing-thumbnails` — force a
fresh server round-trip before they answer, so they report current account state
rather than whatever the cache last held. If that round-trip fails, the command
prints the error on one line and exits 1. `--cache-dir` overrides where the
cache lives; without it each account gets its own directory under the per-user
cache path.
`collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip
before they answer, so they report current account state rather than whatever
the cache last held. `--cache-dir` overrides where the cache lives; without it
each account gets its own directory under the per-user cache path.
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
accepted for backward compatibility but ignored. For a live photo, `get` writes
its image and its video, each named after the title with its own extension, as
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
`--out PATH`, the image is written to `PATH` and the video beside it, with
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
refused. `backup-metadata --exif` (alias `--all`) additionally downloads each
file to extract full EXIF/IPTC/XMP metadata. The listing and backup commands
support `--json` for machine-readable output.
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
still fails after its retries, the error is logged, each of its files is written
with the reason in an `mlDataError` field instead of `mlData`, and the dump goes
on. The exit code is non-zero if any ML data request failed.
accepted for backward compatibility but ignored. `backup-metadata --exif` (alias
`--all`) additionally downloads each file to extract full EXIF/IPTC/XMP
metadata. The listing and backup commands support `--json` for machine-readable
output.
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
distinct from a `failed` repair, and does not affect the exit code; a genuine
failure still exits non-zero. The server accepts a new thumbnail only from the
file's owner and only when it is no larger than the thumbnail size it records
for the file. So a file another account owns, in an album shared with you, is
skipped by both thumbnail helpers without being fetched, and the fixer skips a
file whose recorded thumbnail size is 0 or unknown. Otherwise the fixer lowers
the quality and size of the thumbnail until it fits, and skips the file if even
the smallest does not.
failure still exits non-zero.
### Backup layout
@@ -503,65 +462,26 @@ the smallest does not.
```
<dir>/
originals/
<fileID>.<ext> actual file content (one per unique file,
two for a live photo: see below)
<fileID>.<ext> actual file content (one per unique file)
<fileID>.json all decrypted metadata for that file
<fileID>.livephoto.json which of a live photo's two files is which
collections/
<name>/
<title> -> ../../originals/<fileID>.<ext> (symlink)
<name>.json collection metadata + file list
failures.json files that failed and have not yet succeeded
```
`failures.json` records each failed file with the kind of failure, how many
times it has been tried and when it was last tried. A file leaves it once it
succeeds, or once it is no longer in the library or in the backup's scope. The
library's `lib.backup({ includeThumbnails: true })` also writes
`thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
A collection's directory and JSON are named after the collection, and a symlink
after the file's title, both with unsafe characters replaced. When two
collections would get the same name, or two symlinks in one collection the same
name (ignoring case in both), each of them gets its ID added: two albums named
`Trip` become `Trip (10)/` and `Trip (11)/`, and two files titled `IMG_0001.JPG`
become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. IDs never change, so a
name stays the same from run to run until such a clash appears or goes away.
A live photo, which Ente stores as one ZIP of its image and its video, is stored
as those two files, which a photo viewer can open: each is
`originals/<fileID>.<ext>` with the extension it has inside the ZIP (for example
`12345.heic` and `12345.mov`), and `<fileID>.livephoto.json` names the two. The
live photo counts as stored only when both files are present and not empty. Its
album folder links both, each named after the title with that file's extension
(`IMG_0001.heic` and `IMG_0001.mov`). A live photo that an earlier version of
quak stored as the ZIP, under the image's name, is replaced by its two files on
the next run, and the ZIP and its link are removed.
Each run removes the symlinks into `originals/` that no longer belong in their
collection's directory, and the directories (and JSON) of collections that were
deleted or renamed. Nothing else in `collections/` is touched: a file or a
symlink you put there stays, and a directory that still holds one after its
symlinks are removed stays too, with its JSON.
Each file is downloaded exactly once regardless of how many collections it
appears in, and written once: straight into `originals/`, with no copy left in
the cache. An original the cache already held is copied from there instead. On
subsequent runs, existing originals are skipped. If a download fails, the error
is logged and the backup continues with the next file. The exit code is non-zero
if any files failed. `quak backup` opens its library with the thumbnail and
originals precache off, so it fetches only what the backup stores.
appears in. On subsequent runs, existing originals are skipped. If a download
fails, the error is logged and the backup continues with the next file. The exit
code is non-zero if any files failed.
Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even
after a power cut. A downloaded original's temporary file is named
`.quak-<pid>-<random>.tmp`, one copied from the cache
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
one of these temporary files behind; the next backup deletes those whose process
is no longer running. The content cache uses the same scheme, and opening a
library deletes the temporary files in the cache whose process is no longer
running, so a download another process has in progress in the same cache is left
alone. The rename replaces whatever was at the destination rather than writing
Each original is copied to a temporary file named
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
to disk, and renamed into place, so an original is either complete or absent,
even after a power cut. A run that is killed can leave one of these temporary
files behind; the next backup deletes those whose process is no longer running.
Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
names. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.
@@ -571,12 +491,7 @@ temporary file's permissions, not those of the file it replaced.
errors
- [x] Update the API reference section below to match the current implementation
- [x] `make docker` green
- [x] Store live photos in a form a photo viewer can open
(https://git.eeqj.de/sneak/quak/issues/107): unpacked into the image and
the video
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
- [ ] Tag `v1.0.0`
Future (desktop client, separate repo):
@@ -596,11 +511,10 @@ test suite is the canonical, executable documentation — `test/library/` and
### Opening a library
`Library.open(options)` loads the on-disk cache, starts the background refresh
loop, and resolves to a `Library`. On an empty cache it awaits the first
refresh, so it opens onto the account's data whenever the server is reachable;
if that refresh fails, it opens with no data and records the error in
`lib.status()`. On an existing cache it returns immediately and refreshes in the
background, so an unreachable server does not block opening.
loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
so it never opens onto empty data; on an existing cache it returns immediately
and refreshes in the background, so an unreachable server does not block
opening.
`LibraryOptions`:
@@ -665,9 +579,7 @@ An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
(newest first). A `Photo` exposes its record fields, `photo.record()` →
`PhotoRecord`, and two content methods:
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
full-resolution file. For a live photo, `path` and `bytes` are its image's and
`videoPath` is its video.
- `await photo.original(opts?)` → `{ path, bytes }` — the full-resolution file.
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
Both serve from the on-disk content cache when the bytes are present and
@@ -687,7 +599,7 @@ The GUI-facing records hold no key material and no binary, so they survive
- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds),
`fileType`, optional `caption` / `width` / `height` / `latitude` /
`longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath`
once the bytes are cached (for a live photo, `originalPath` is its image).
once the bytes are cached.
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
`fileIDs` (newest first).
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
@@ -713,15 +625,12 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere.
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
in-scope original not already in the backup (and, with `includeThumbnails`,
thumbnails) through the content cache, and rebuilds the on-disk backup tree
with a durable failure ledger. A fetched original is written straight into the
backup's `originals/` and not into the cache, which then counts it as present;
one the cache already held is copied from there. `BackupOptions`:
`downloadDirectory` (falls back to the one `open()` was given),
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
writes.
in-scope original (and, with `includeThumbnails`, thumbnails) through the
content cache, and rebuilds the on-disk backup tree with a durable failure
ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
was given), `includeOriginals` (default `true`), `includeThumbnails` (default
`false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
tree it writes.
### Request pools
@@ -738,8 +647,6 @@ Under `cacheDirectory`:
<cacheDirectory>/
metadata.json decrypted account state + refresh cursor
originals/<fileID>.<ext> cached full-resolution files
originals/<fileID>.livephoto.json
which of a live photo's two files is which
thumbnails/<fileID>.jpg cached thumbnails
mldata/
<fileID>.json one decrypted ML payload per file
@@ -747,26 +654,11 @@ Under `cacheDirectory`:
fetched.json per-file fetch bookkeeping
```
When `metadata.json` belongs to a different account than the client's,
`Library.open` deletes it and `mldata/` and starts from an empty cache. Cached
originals and thumbnails are kept; they are reached only through the files the
current account's records name.
A live photo's original is cached as in the backup: its image and its video,
each `originals/<fileID>.<ext>` with its own extension, and
`originals/<fileID>.livephoto.json` naming them; the two are evicted together. A
live photo that an earlier version cached as its ZIP is not served, and is
replaced by its two files the next time it is read.
A stored file appears only via an atomic temp-then-rename, so its presence means
it is complete. Every downloaded original (by `quak get`, the cache, or
`backup`) whose metadata records a content hash (`FileMetadata.hash`) is hashed
as it is written: unkeyed BLAKE2b with a 64-byte output, standard base64. A live
photo arrives as a ZIP and is unpacked as it is written; its image and its video
are hashed separately and joined as `<imageHash>:<videoHash>`, and neither is
stored unless both are complete and match. A mismatch stores nothing and fails
the download with an error naming the file ID. An original with no recorded
hash, from a very old client, is stored unchecked.
it is complete. The design also calls for a content-hash comparison against
`FileMetadata.hash` on each fetched original; that check is deferred (issue
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
cannot yet be confirmed against the repo's fixtures.
### Key types by source file
@@ -827,10 +719,8 @@ documents:
markdown. Use `make fmt` to format. Use `yarn` not `npm`.
- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure.
`make test` must finish in under 60 seconds (the hard cap) and should finish
in under 20. The 90-second `timeout` in the `test` phase of the `Dockerfile`
is a backstop that catches a hung test, not the time limit. Use `mkdtempSync`
for temporary directories, never manual timestamp paths.
`make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary
directories, never manual timestamp paths.
- **Code style:** `const` for everything, `let` if reassignment is needed, never
`var`. Avoid unnecessary comments. No hand-rolled crypto. The
TODO.md
+1 -121
View File
@@ -14,136 +14,16 @@ pre-1.0
# Next Step
None: every issue still open is done on `next` or `next2` and waits for it to
reach `main`.
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
Tag v1.0.0.
# Completed Steps
- 2026-09-23: Live photos are stored as their image and their video (issue 107).
A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
`<fileID>.<ext>` for the image and for the video, each with its extension from
the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
checked against the recorded hash and renamed into place only when both are
complete. The backup and the content cache count the live photo as stored only
with both files, the album folder links both, `quak get` writes both, and the
content result gives the video as `videoPath`. A ZIP an earlier version stored
is replaced on the next backup run, and in the cache when the photo is next
read.
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
`package.json` is marked `"private": true` and the `files` field is gone.
`engines.node` is `>=22`, the major version `script/bootstrap` and the
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
stay internal to the CLI.
- 2026-09-23: Brought the README and this file in line with the tree (issue
111). The layout lists `src/library/` and the other source files, the backup
layout names `failures.json` and the optional `thumbnails/`, "Opening a
library" says what happens when the first refresh fails, the Testing section
gives the 60-second hard cap and 20-second target for `make test` and names
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
and "Tag v1.0.0" is no longer listed as the next step.
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
`loginCommand` takes its login function and its prompts from `CliContext`, and
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
failed login, and the saved session's modes, and show that `--exif` and
`--all` each turn on EXIF extraction and that it is off without them.
- 2026-09-23: `quak backup` writes each original once and no longer fills the
cache (issue 106). An original fetched for a backup is written by the download
writer straight into the backup's `originals/`, and the content cache records
it there instead of keeping its own copy; one the cache already held is still
copied. `quak backup` opens its library with the thumbnail and originals
precache off.
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
awaits `lib.fresh()` before reading, so a file added since the cache was
written is included, and a failed refresh prints one line and exits 1 instead
of answering from a stale or empty cache. The README lists them among the
commands that refresh first.
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
`fresh()` does, and rejects before touching any file when it fails, leaving
`failures.json` as it was, so `quak backup` prints the error as one line and
exits 1 instead of backing up the previous run's file list, or nothing, and
exiting 0.
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
the CLI exits 1 once output has drained. The wrapper that does this moved from
`bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
`program.parseAsync()`.
- 2026-09-23: Opening a library no longer deletes another process's download in
progress (issue 105). The download writer's temp files are named
`.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
backup into the download module, deletes a `.quak-*.tmp` file only when the
process ID in its name is no longer running. The content cache calls it at
`open()` for `originals/` and `thumbnails/`, the backup as before.
- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
`script/lint` and `script/test` each build one with `--no-cache`, and the last
stage compiles and depends on both, so `script/cibuild` is one build.
`Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
are gone; `REPO_POLICIES.md` is re-copied.
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
always refuses (issue 109). Both thumbnail helpers skip a file another account
owns without fetching it. The fixer skips a file whose recorded thumbnail size
is 0 or unknown before downloading it, and otherwise tries smaller encodings
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
no larger than that size, skipping the file if none fits.
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
download a live photo whose ZIP names an unknown compression method, one whose
ZIP has no image entry and one with no video entry, and check that nothing is
stored and the error names the file ID; the unreadable one is not retried.
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
`session.json` even when that call fails, says so and exits 1. It prints the
account's cache directory and says it still holds decrypted data. The default
cache path is now `defaultCacheDirectory()` in the library, shared with
`Library.open`.
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
in one collection with the same title, and two collections with the same name,
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
none replaces another's symlink or JSON. Each run removes symlinks into
`originals/` for files no longer in the collection, and the folders of deleted
or renamed collections, leaving anything else in `collections/` alone. The
README backup layout states the naming rule.
- 2026-09-23: Checked downloaded originals against their recorded content hash
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
stores nothing on a mismatch, failing with an error naming the file ID. A live
photo ZIP is unpacked as it streams with `fflate` and its image and video
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
clients' `imageHash` and `videoHash` fields for live photos. A file with no
recorded hash is stored unchecked.
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
When `metadata.json` in the cache directory was written for a different,
non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
and starts empty, so the first refresh enumerates from 0. This only happens
with `--cache-dir` or an explicit `cacheDirectory`; the default path already
includes the user ID. A test opens one account's cache as another account.
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
(issue 101). Each request of up to 200 files is tried on its own; a failed one
is logged, its files are written with the reason in `mlDataError`, and the
command exits 1 once the dump is complete. `fetchMLData`, which only this
command used, is gone; the command calls `fetchMLDataBatch` per batch.
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
only place it is written: `src/index.ts` imports it for `VERSION` and
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
bin/quak.ts
+19 -10
View File
@@ -1,7 +1,6 @@
#!/usr/bin/env node
import { stdout, stderr } from "node:process";
import { input, password } from "@inquirer/prompts";
import { Command } from "commander";
import envPaths from "env-paths";
import { init } from "../src/crypto/index.js";
@@ -19,9 +18,7 @@ import {
listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../src/cli-commands.js";
import { run as runCommand } from "../src/cli-run.js";
import { loadSession } from "../src/cli-session.js";
import { Client } from "../src/client.js";
import { VERSION } from "../src/index.js";
const paths = envPaths("quak", { suffix: "" });
@@ -44,13 +41,25 @@ const context = (): CliContext => ({
sessionDir: paths.data,
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
loadSession,
login: (opts) => Client.login(opts),
prompt: (message) => input({ message }),
promptSecret: (message) => password({ message, mask: true }),
});
const run = (command: Promise<number>): Promise<void> =>
runCommand(command, stdout, stderr, (code) => process.exit(code));
// Run a command and exit with its code once stdout/stderr have drained.
// Exiting before the drain can truncate piped output, and the library can keep
// the event loop alive after a command returns, so a plain return could hang.
const run = async (command: Promise<number>): Promise<void> => {
process.exitCode = await command;
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) {
process.exit();
return;
}
let remaining = pending.length;
for (const s of pending) {
s.once("drain", () => {
if (--remaining === 0) process.exit();
});
}
};
program
.command("login")
@@ -64,7 +73,7 @@ program
program
.command("logout")
.description("End the session on the server and delete the saved session")
.description("Delete the saved session")
.action(() => run(logoutCommand(context())));
program
@@ -160,4 +169,4 @@ helper
);
await init();
await program.parseAsync();
program.parse();
package.json
+5 -12
View File
@@ -9,23 +9,17 @@
"type": "git",
"url": "https://git.eeqj.de/sneak/quak.git"
},
"private": true,
"type": "module",
"engines": {
"node": ">=22"
},
"main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts",
"exports": {
".": {
"types": "./dist/src/index.d.ts",
"import": "./dist/src/index.js"
},
"./package.json": "./package.json"
},
"bin": {
"quak": "./dist/bin/quak.js"
},
"files": [
"dist/",
"README.md",
"LICENSE"
],
"scripts": {
"build": "script/build",
"quak": "node ./dist/bin/quak.js",
@@ -48,7 +42,6 @@
"env-paths": "4.0.0",
"exif-reader": "2.0.3",
"fast-srp-hap": "2.0.4",
"fflate": "0.8.3",
"jpeg-js": "0.4.4",
"libsodium-wrappers-sumo": "0.8.4"
}
src/backup.ts
+65 -215
View File
@@ -1,11 +1,9 @@
// The backup command, rebuilt on the library API (issue #51).
//
// `lib.backup()` waits for a completed refresh of the library (a failed one
// fails the backup before any file is touched), then, for every file in scope,
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
// derived views (per-file sidecars, per-collection symlink trees,
// per-collection JSON) from the model. The on-disk layout is the historical
// one:
// `lib.backup()` refreshes the library, then, for every file in scope, gets its
// original bytes onto disk under `downloadDirectory` and rebuilds the derived
// views (per-file sidecars, per-collection symlink trees, per-collection JSON)
// from the model. The on-disk layout is the historical one, unchanged:
//
// <downloadDirectory>/
// originals/<fileID>.<ext> the decrypted bytes
@@ -14,18 +12,12 @@
// collections/<name>.json per-collection metadata
// failures.json durable ledger of unresolved failures
//
// A live photo's original is its image and its video, `<fileID>.<ext>` each
// with its own extension, and `originals/<fileID>.livephoto.json` naming them;
// its album folders link both.
//
// Crash-safety rests on two properties. Bytes are present-means-complete: an
// original appears under `originals/` only via the content layer's atomic
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
// interrupted run resumes by listing the directory. The derived views hold no
// unique state, so they are rebuilt every run; that repairs stale sidecars and
// missing or broken symlinks left by an earlier crash. A rebuild also removes
// the symlinks into originals/ that no longer belong to an album, and the
// directories of albums that no longer exist.
// missing or broken symlinks left by an earlier crash.
//
// Resilience (issue #8): no per-file condition aborts the run. A failed
// download or a failed symlink is caught, recorded in `failures.json` with a
@@ -42,22 +34,16 @@ import {
readdirSync,
readFileSync,
readlinkSync,
rmdirSync,
rmSync,
statSync,
symlinkSync,
writeFileSync,
} from "node:fs";
import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, extname, join, relative } from "node:path";
import { basename, dirname, join, relative } from "node:path";
import { fsyncPath, removeLeftoverTempFiles } from "./download/index.js";
import { sanitizeFileName, withExtension } from "./filename.js";
import {
originalName,
storedOriginal,
writeLivePhoto,
} from "./library/content.js";
import { fsyncPath } from "./download/index.js";
import { safeExtension, sanitizeFileName } from "./filename.js";
import type { Collection, EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void;
@@ -106,14 +92,8 @@ export interface BackupLibrary {
listCollections(): Collection[];
listFiles(collectionID: number): EnteFile[];
// Get an original's bytes onto disk through the content cache/pools,
// returning where they landed: `destination` when they were fetched now,
// otherwise wherever they already were (the cache, or a prior backup). A
// live photo lands as its image and its video, fetched now beside
// `destination`.
original(
fileID: number,
destination: string,
): Promise<{ path: string; videoPath?: string }>;
// returning where they landed (the cache, or a prior backup).
original(fileID: number): Promise<{ path: string }>;
thumbnail(fileID: number): Promise<{ path: string }>;
}
@@ -130,6 +110,12 @@ interface FailureEntry {
const LEDGER_VERSION = 1;
// The originals/ filename for a file: `<id><ext>`, the extension taken from the
// title (or `.bin`). Matches the content cache's own naming so a present check
// lines up with what a fetch would write.
const originalName = (file: EnteFile): string =>
`${file.id}${safeExtension(file.metadata.title)}`;
// A regular file with content is treated as complete. A zero-byte file is not:
// it is the shape an aborted write leaves and must be re-fetched.
const isPresent = (path: string): boolean => {
@@ -195,29 +181,34 @@ const copyAtomic = async (src: string, dest: string): Promise<void> => {
}
};
// Put an original the library returned at `dest` in originals/, where a fresh
// fetch already wrote it. A live photo's image and video go beside `dest`: when
// they came from the cache they are copied, after removing whatever was at
// `dest` (an earlier version's ZIP of the two). Then the JSON file naming them
// is written, which is what makes the live photo count as stored.
const placeOriginal = async (
file: EnteFile,
dest: string,
got: { path: string; videoPath?: string },
): Promise<void> => {
if (got.videoPath === undefined) {
await copyAtomic(got.path, dest);
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files `copyAtomic` leaves behind when a backup is killed
// before its rename. Only files whose process is no longer running are
// removed, so a backup running at the same time keeps its own. A reused
// process ID can only keep a leftover a while longer, never remove a live one.
const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
const originalsDir = dirname(dest);
const path = join(originalsDir, basename(got.path));
const videoPath = join(originalsDir, basename(got.videoPath));
if (got.path !== path) {
await rm(dest, { force: true });
await copyAtomic(got.path, path);
await copyAtomic(got.videoPath, videoPath);
for (const name of names) {
const match = /^\.quak-backup-.*-(\d+)-[0-9a-z]*\.tmp$/.exec(name);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
}
await writeLivePhoto(originalsDir, file.id, { path, videoPath });
};
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
@@ -236,111 +227,6 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
symlinkSync(target, linkPath);
};
// The on-disk names for the entries of one directory, in entry order. Each
// name is used as is unless another entry would get the same name, ignoring
// case (two names that differ only in case are one entry on a case-insensitive
// file system); then every entry sharing it gets ` (<id>)`, before the
// extension when `beforeExtension` is set. A name with an ID added can match
// another entry's own name (`IMG (6).JPG`), so this repeats until no name is
// shared. IDs are stable, so the names are too.
const uniqueNames = (
entries: { id: number; name: string }[],
beforeExtension: boolean,
): string[] => {
const withID = (id: number, name: string): string => {
const ext = beforeExtension ? extname(name) : "";
const stem = name.slice(0, name.length - ext.length);
return `${stem} (${id})${ext}`;
};
const names = entries.map((e) => e.name);
const suffixed = new Set<number>();
for (;;) {
const counts = new Map<string, number>();
for (const name of names) {
const key = name.toLowerCase();
counts.set(key, (counts.get(key) ?? 0) + 1);
}
let changed = false;
for (const [i, { id, name }] of entries.entries()) {
if (suffixed.has(i)) continue;
if (counts.get(name.toLowerCase()) === 1) continue;
names[i] = withID(id, name);
suffixed.add(i);
changed = true;
}
if (!changed) return names;
}
};
// The links a file gets in its album's folder: one named after its title, to
// its original if that is stored. A stored live photo gets two, to its image
// and its video, each named after the title with that file's extension.
const linksFor = (
file: EnteFile,
stored: { path: string; videoPath?: string } | undefined,
): { id: number; name: string; file: EnteFile; target?: string }[] => {
const name = sanitizeFileName(file.metadata.title, `file-${file.id}`);
if (stored?.videoPath === undefined) {
return [{ id: file.id, name, file, target: stored?.path }];
}
return [stored.path, stored.videoPath].map((target) => ({
id: file.id,
name: withExtension(name, extname(target)),
file,
target,
}));
};
// Remove the symlinks in the album directory `dir` that point into
// `originalsDir` and are not named in `keep`. Nothing else in the directory
// is touched: anything else there was put there by the user.
const removeStaleLinks = (
dir: string,
keep: Set<string>,
originalsDir: string,
): void => {
const target = relative(dir, originalsDir);
for (const name of readdirSync(dir)) {
if (keep.has(name)) continue;
const path = join(dir, name);
if (
lstatSync(path).isSymbolicLink() &&
dirname(readlinkSync(path)) === target
) {
rmSync(path);
}
}
};
// Remove the directories under `collectionsDir` that an earlier run wrote for
// an album that is gone or renamed: a directory not named in `current` with a
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
// symlinks into originals/ are removed; if that leaves it empty, it and its
// JSON are deleted, otherwise both stay for what the user put there.
const removeStaleAlbumDirs = (
collectionsDir: string,
current: Set<string>,
originalsDir: string,
): void => {
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
if (!entry.isDirectory() || current.has(entry.name)) continue;
const jsonPath = join(collectionsDir, `${entry.name}.json`);
try {
const album = JSON.parse(readFileSync(jsonPath, "utf-8")) as {
id?: unknown;
};
if (typeof album.id !== "number") continue;
} catch {
continue;
}
const dir = join(collectionsDir, entry.name);
removeStaleLinks(dir, new Set(), originalsDir);
if (readdirSync(dir).length > 0) continue;
rmdirSync(dir);
rmSync(jsonPath);
}
};
const loadLedger = (path: string): Map<number, FailureEntry> => {
const ledger = new Map<number, FailureEntry>();
try {
@@ -417,10 +303,9 @@ export const runBackup = async (
// Collections in scope, and the distinct files across them (a file shared
// by two albums is one original).
const allCollections = lib.listCollections();
const collections = allCollections.filter((c) =>
only ? only.has(c.name) : true,
);
const collections = lib
.listCollections()
.filter((c) => (only ? only.has(c.name) : true));
const collectionName = new Map<number, string>();
for (const c of collections) collectionName.set(c.id, c.name);
@@ -470,21 +355,15 @@ export const runBackup = async (
// tree; a present file is left as is.
if (includeOriginals) {
for (const [fileID, file] of distinct) {
if (storedOriginal(originalsDir, file) !== undefined) {
const dest = join(originalsDir, originalName(file));
if (isPresent(dest)) {
skipped++;
continue;
}
const dest = join(originalsDir, originalName(file));
try {
log(`Fetching original ${file.metadata.title} (${fileID})...`);
// A fetched original is written straight to `dest` (a live
// photo beside it); only one that was already cached elsewhere
// is copied.
await placeOriginal(
file,
dest,
await lib.original(fileID, dest),
);
const { path } = await lib.original(fileID);
await copyAtomic(path, dest);
downloaded++;
} catch (err) {
log(
@@ -520,67 +399,38 @@ export const runBackup = async (
// every present original (this repairs stale ones).
if (includeOriginals) {
for (const [fileID, file] of distinct) {
if (storedOriginal(originalsDir, file) !== undefined) {
const orig = join(originalsDir, originalName(file));
if (isPresent(orig)) {
writeSidecar(join(originalsDir, `${fileID}.json`), file);
}
}
}
// Then the per-collection symlink trees and JSON. Directory names are
// chosen across every album, not just those in scope, so a scoped run
// names an album the same as a full one and never takes the directory of
// an album it skipped. Stale entries are removed before anything is
// rebuilt, so on a case-insensitive file system removing an old name can
// never remove the new one.
const dirNames = uniqueNames(
allCollections.map((c) => ({
id: c.id,
name: sanitizeFileName(c.name, `collection-${c.id}`),
})),
false,
);
const albumDirNames = new Map(
allCollections.map((c, i) => [c.id, dirNames[i]!]),
);
try {
removeStaleAlbumDirs(collectionsDir, new Set(dirNames), originalsDir);
} catch (err) {
log(`FAILED removing old album directories: ${errorMessage(err)}`);
}
// Then the per-collection symlink trees and JSON.
for (const c of collections) {
const colDirName = albumDirNames.get(c.id)!;
const colDirName = sanitizeFileName(c.name, `collection-${c.id}`);
const colDir = join(collectionsDir, colDirName);
mkdirSync(colDir, { recursive: true });
const files = filesByCollection.get(c.id) ?? [];
const links = files.flatMap((f) =>
linksFor(f, storedOriginal(originalsDir, f)),
);
const linkNames = uniqueNames(links, true);
try {
removeStaleLinks(colDir, new Set(linkNames), originalsDir);
} catch (err) {
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
}
const metaFiles = files.map((f) => ({
id: f.id,
metadata: f.metadata,
}));
for (const [i, link] of links.entries()) {
if (!includeOriginals || link.target === undefined) continue;
const linkName = linkNames[i]!;
const metaFiles: { id: number; metadata: EnteFile["metadata"] }[] = [];
for (const file of files) {
metaFiles.push({ id: file.id, metadata: file.metadata });
if (!includeOriginals) continue;
const orig = join(originalsDir, originalName(file));
if (!isPresent(orig)) continue;
const linkName = sanitizeFileName(
file.metadata.title,
`file-${file.id}`,
);
const linkPath = join(colDir, linkName);
try {
rebuildSymlink(
join(colDir, linkName),
relative(colDir, link.target),
);
rebuildSymlink(linkPath, relative(colDir, orig));
} catch (err) {
log(
`FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`,
);
recordFailure(link.file, c.name, err);
recordFailure(file, c.name, err);
}
}
src/cli-commands.ts
+27 -100
View File
@@ -2,30 +2,22 @@
//
// Each command takes its options and a `CliContext` and resolves to the exit
// code; a thrown error is left to the caller. Nothing here calls
// `process.exit`: `bin/quak.ts` wires these to the command line, and `run` in
// `cli-run.ts` prints a thrown error as one line and exits once output has
// drained. Output must stay byte-identical (see `cli-output.ts`).
// `process.exit`: `bin/quak.ts` wires these to the command line and exits with
// the returned code once output has drained. Output must stay byte-identical
// (see `cli-output.ts`).
import { input, password as passwordPrompt } from "@inquirer/prompts";
import {
copyFileSync,
existsSync,
mkdirSync,
statSync,
unlinkSync,
writeFileSync,
} from "node:fs";
import { extname, join } from "node:path";
import {
type Client,
type ClientSnapshot,
type LoginOptions,
} from "./client.js";
import { join } from "node:path";
import { Client, type ClientSnapshot } from "./client.js";
import { init } from "./crypto/index.js";
import {
defaultCacheDirectory,
Library,
type LibraryClient,
} from "./library/index.js";
import { Library, type LibraryClient } from "./library/index.js";
import {
fileListRow,
fileListLine,
@@ -33,7 +25,6 @@ import {
thumbnailName,
} from "./cli-output.js";
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
import { withExtension } from "./filename.js";
import { runMetadataBackup } from "./metadata-backup.js";
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
@@ -48,12 +39,6 @@ export interface CliContext {
// Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
loadSession: (path: string) => Client | null;
// Used by `login` only. The CLI passes `Client.login` and terminal
// prompts; tests pass fakes.
login: (opts: LoginOptions) => Promise<Client>;
prompt: (message: string) => Promise<string>;
// Like `prompt`, but the answer is masked as it is typed.
promptSecret: (message: string) => Promise<string>;
}
const sessionPath = (ctx: CliContext): string =>
@@ -119,19 +104,24 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
precacheOriginals: false,
});
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
export const loginCommand = async (ctx: CliContext): Promise<number> => {
await init();
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
ctx.stderr.write("Authenticating...\n");
try {
const client = await ctx.login({
const client = await Client.login({
email,
password,
totp: async () => ctx.prompt("TOTP code: "),
emailOTP: async () => ctx.prompt("Email verification code: "),
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(ctx.sessionDir, client.toJSON());
@@ -156,43 +146,14 @@ export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
return 0;
};
// Ends the session on the server, then deletes the session file even when that
// failed, and exits 1 if it did. The cache is left in place; the user is told
// where it is.
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
const path = sessionPath(ctx);
if (!existsSync(path)) {
ctx.stderr.write("No session found.\n");
return 0;
}
await init();
let cacheDir = ctx.cacheDir;
let failure: string | undefined;
try {
const client = ctx.loadSession(path);
if (client) {
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
await client.logoutOnServer();
client.logout();
}
} catch (err) {
failure = err instanceof Error ? err.message : String(err);
}
unlinkSync(path);
if (failure === undefined) {
ctx.stderr.write("Session ended on the server.\n");
if (existsSync(sessionPath(ctx))) {
unlinkSync(sessionPath(ctx));
ctx.stderr.write("Session deleted.\n");
} else {
ctx.stderr.write(
`Could not end the session on the server: ${failure}\n`,
);
ctx.stderr.write("No session found.\n");
}
ctx.stderr.write("Session deleted.\n");
if (cacheDir !== undefined) {
ctx.stderr.write(
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.\n`,
);
}
return failure === undefined ? 0 : 1;
return 0;
};
export const collectionsCommand = async (
@@ -307,29 +268,8 @@ export const getCommand = async (
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
if (result.videoPath === undefined) {
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
}
// A live photo is written as its image and its video, each named after
// the title with its own extension, as Ente's clients name them. With
// --out, the image goes there and the video beside it.
const imageOut =
opts.out ?? withExtension(outPath, extname(result.path));
const videoOut = withExtension(outPath, extname(result.videoPath));
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
ctx.stderr.write(
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
);
return 1;
}
copyFileSync(result.path, imageOut);
copyFileSync(result.videoPath, videoOut);
ctx.stderr.write(
`${result.bytes} bytes -> ${imageOut}\n` +
`${statSync(videoOut).size} bytes -> ${videoOut}\n`,
);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
} finally {
await lib.close();
@@ -383,14 +323,11 @@ export const backupMetadataCommand = async (
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Refresh first so the dump holds current account state, not what the
// cache last held; a failed refresh throws.
await lib.fresh();
const { failedMLBatches } = await runMetadataBackup(lib, client, dir, {
await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
});
return failedMLBatches > 0 ? 1 : 0;
return 0;
} finally {
await lib.close();
}
@@ -406,14 +343,10 @@ export const backupCommand = async (
if (!client) return 1;
ctx.stderr.write("Starting backup...\n");
// The precache is off: the backup fetches what it needs, and must not
// also fill the cache with every thumbnail and the recent originals.
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: ctx.cacheDir,
precacheThumbnails: false,
precacheOriginals: false,
});
try {
const result = await lib.backup({
@@ -456,9 +389,6 @@ export const listMissingThumbnailsCommand = async (
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Refresh first so files added since the cache was written are
// checked; a failed refresh throws.
await lib.fresh();
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
});
@@ -494,9 +424,6 @@ export const fixMissingThumbnailsCommand = async (
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Refresh first so files added since the cache was written are found;
// a failed refresh throws.
await lib.fresh();
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
@@ -535,7 +462,7 @@ export const fixMissingThumbnailsCommand = async (
ctx.stderr.write(` Skipped: ${skipped}\n`);
ctx.stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) {
ctx.stderr.write("\nSkipped:\n");
ctx.stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) {
ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
src/cli-run.ts
-36
View File
@@ -1,36 +0,0 @@
// Runs one CLI command for `bin/quak.ts` and exits with its code.
import type { Writable } from "node:stream";
// Run a command and exit with its code once stdout/stderr have drained.
// Exiting before the drain can truncate piped output, and the library can keep
// the event loop alive after a command returns, so a plain return could hang.
// An error the command throws is printed as one `quak: MESSAGE` line, without
// the stack trace, and exits 1.
export const run = async (
command: Promise<number>,
stdout: Writable,
stderr: Writable,
exit: (code: number) => void,
): Promise<void> => {
let code: number;
try {
code = await command;
} catch (err) {
stderr.write(
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
);
code = 1;
}
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) {
exit(code);
return;
}
let remaining = pending.length;
for (const s of pending) {
s.once("drain", () => {
if (--remaining === 0) exit(code);
});
}
};
src/client.ts
-8
View File
@@ -217,14 +217,6 @@ export class Client {
};
}
// Ends this client's session on the server (`POST /users/logout`), so the
// token stops working everywhere, including in any saved copy of it. This
// client is left as it was; call `logout()` to clear it.
async logoutOnServer(): Promise<void> {
this.assertLoggedIn();
await this.api.postJSON("/users/logout", {});
}
// Zeroes the key buffers in place, so any copy of the reference held
// elsewhere is wiped too. Every method checks `assertLoggedIn` before
// touching the keys, so nothing decrypts with the zeroed keys.
src/crypto/hash.ts
-22
View File
@@ -1,22 +0,0 @@
import sodium, { type StateAddress } from "libsodium-wrappers-sumo";
import { toBase64 } from "./encoding.js";
// The content hash an uploading client records in a file's metadata: unkeyed
// BLAKE2b with a 64-byte output over the original's bytes, fed in chunks, as
// standard base64 with padding. Named after the upstream client's functions.
// The output length is read at call time for the same reason as
// `streamTagFinal` in stream.ts: libsodium sets its constants only once ready.
export const chunkHashInit = (): StateAddress =>
sodium.crypto_generichash_init(null, sodium.crypto_generichash_BYTES_MAX);
export const chunkHashUpdate = (state: StateAddress, chunk: Uint8Array): void =>
sodium.crypto_generichash_update(state, chunk);
export const chunkHashFinal = (state: StateAddress): string =>
toBase64(
sodium.crypto_generichash_final(
state,
sodium.crypto_generichash_BYTES_MAX,
),
);
src/crypto/index.ts
-1
View File
@@ -7,7 +7,6 @@ export {
} from "./encoding.js";
export { deriveKEK, deriveLoginSubkey } from "./kdf.js";
export { decryptBox, decryptSealed } from "./box.js";
export { chunkHashFinal, chunkHashInit, chunkHashUpdate } from "./hash.js";
export {
decryptBlob,
encryptBlob,
src/download/index.ts
+30 -267
View File
@@ -1,13 +1,8 @@
import { randomBytes } from "node:crypto";
import { readdirSync, rmSync } from "node:fs";
import { randomUUID } from "node:crypto";
import { open, rename, rm } from "node:fs/promises";
import type { FileHandle } from "node:fs/promises";
import { dirname, join } from "node:path";
import { Unzip, UnzipInflate } from "fflate";
import {
chunkHashFinal,
chunkHashInit,
chunkHashUpdate,
fromBase64,
initStreamPull,
pullStreamChunk,
@@ -16,18 +11,14 @@ import {
streamTagFinal,
} from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js";
import { safeExtension, sanitizeFileName, withExtension } from "../filename.js";
import { sanitizeFileName } from "../filename.js";
import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js";
export interface DownloadResult {
// Where the file was written. A live photo is written as two files, its
// image here and its video at `videoPath` (see `decryptLivePhoto`).
path: string;
// The decrypted length; for a live photo, that of the ZIP it arrives as.
bytesWritten: number;
videoPath?: string;
}
// Fired as decrypted plaintext accumulates, with the running total of
@@ -49,9 +40,9 @@ const ENC_CHUNK_SIZE = STREAM_CHUNK_SIZE + STREAM_CHUNK_OVERHEAD;
// new: a body cut short still decrypts and authenticates up to its last whole
// chunk, so the absence of TAG_FINAL is the sole evidence it was cut short, and
// this throws rather than let a caller keep a short file. The sink has already
// seen those chunks by then; the callers (`decryptToTemp`, `decryptLivePhoto`)
// stage them in temp files that are renamed into place only on a clean return,
// so a throw leaves nothing on disk.
// seen those chunks by then; the caller (`decryptToTemp`) stages them in a temp
// file that is renamed into place only on a clean return, so a throw leaves
// nothing on disk.
const streamDecrypt = async (
stream: ReadableStream<Uint8Array>,
header: Uint8Array,
@@ -185,45 +176,6 @@ export const fsyncPath = async (path: string): Promise<void> => {
}
};
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files a killed process left in `dir`: the writer's
// `.quak-<pid>-<random>.tmp` and the backup copy's
// `.quak-backup-<name>-<pid>-<random>.tmp`. Only files whose process is no
// longer running are removed, so another process writing into the same
// directory keeps its own. A reused process ID can only keep a leftover a while
// longer, never remove a live one.
export const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
for (const name of names) {
const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.tmp$/.exec(name);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
}
};
// A new temp file name in `dir`. The random suffix keeps concurrent downloads
// of the same destination from stepping on each other's temporary file; the
// process ID lets `removeLeftoverTempFiles` tell a leftover from a write in
// progress.
const tempPathIn = (dir: string): string =>
join(dir, `.quak-${process.pid}-${randomBytes(16).toString("hex")}.tmp`);
// Stage a write to `destination` atomically and durably, then rename it into
// place. `fill` writes the contents into the open temp file handle — either the
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
@@ -248,7 +200,9 @@ const stageAtomic = async (
fill: (handle: FileHandle) => Promise<void>,
): Promise<void> => {
const dir = dirname(destination);
const tmpPath = tempPathIn(dir);
// The random suffix keeps concurrent downloads of the same destination
// from stepping on each other's temporary file.
const tmpPath = join(dir, `.quak-${randomUUID()}.tmp`);
try {
const handle = await open(tmpPath, "w");
try {
@@ -283,201 +237,42 @@ export const writeAtomic = async (
): Promise<void> =>
stageAtomic(destination, (handle) => handle.writeFile(plaintext));
// Refuse an original whose bytes do not hash to what its uploader recorded.
// The error is not retried.
const checkHash = (file: EnteFile, actual: string): void => {
if (actual !== file.metadata.hash) {
throw new Error(
`download: file ${file.id}: content hash ${actual} does not match the hash its uploader recorded, ${file.metadata.hash}`,
);
}
};
// Decrypt `stream` straight to `destination`, one plaintext chunk at a time,
// under the atomic writer's temp-then-rename discipline. Memory stays bounded
// by the chunk size: each decrypted chunk is written to the temp file and
// dropped. The rename happens only after the stream authenticates as terminated
// on TAG_FINAL; a truncated stream throws and leaves the destination untouched.
// Returns the plaintext length written.
//
// `original` is the file whose original this is (none for a thumbnail, which
// has no recorded hash). When its metadata has a hash, the decrypted bytes are
// hashed as they stream and must match it, or nothing is stored.
const decryptToTemp = async (
destination: string,
stream: ReadableStream<Uint8Array>,
header: Uint8Array,
key: Uint8Array,
onProgress?: ProgressCallback,
original?: EnteFile,
): Promise<number> => {
const hash =
original?.metadata.hash === undefined ? undefined : chunkHashInit();
let bytesWritten = 0;
await stageAtomic(destination, async (handle) => {
bytesWritten = await streamDecrypt(
stream,
header,
key,
async (plaintext) => {
if (hash !== undefined) chunkHashUpdate(hash, plaintext);
await handle.write(plaintext);
},
onProgress,
);
if (original !== undefined && hash !== undefined) {
checkHash(original, chunkHashFinal(hash));
}
});
return bytesWritten;
};
// One of the two parts of a live photo being unpacked: the ZIP entry whose
// name starts with `kind`, written to its own temp file.
interface LivePhotoPart {
kind: "image" | "video";
tmpPath: string;
handle: FileHandle;
hash: ReturnType<typeof chunkHashInit>;
// Decompressed bytes not yet written.
pending: Uint8Array[];
// The entry's extension, set once all of the entry has been read.
ext?: string;
}
const openPart = async (
kind: "image" | "video",
dir: string,
): Promise<LivePhotoPart> => {
const tmpPath = tempPathIn(dir);
const handle = await open(tmpPath, "w");
return { kind, tmpPath, handle, hash: chunkHashInit(), pending: [] };
};
// A live photo arrives as a ZIP of its image and its video. Ente's clients
// name the entries `image.<ext>` and `video.<ext>`, and like the upstream
// client's decoder this takes the first entries whose names start with `image`
// and `video`. It is written unpacked: each part is named `destination` with
// the extension replaced by its own entry's, and the two must differ ignoring
// case. When the file records a hash, `<imageHash>:<videoHash>` must match it,
// each over that part's own bytes. Only then is whatever was at `destination`
// removed and the image, then the video, renamed into place; on any failure
// neither is stored.
//
// The ZIP is chosen by its uploader and may expand enormously, so each part is
// written as it decompresses and never held. fflate's `Unzip` inflates each
// push in one piece before `push` returns, and deflate expands at most about
// 1000-fold, so the ZIP is pushed in 4 KiB slices, keeping each decompressed
// piece near 4 MiB, one plaintext chunk, and each piece is written before the
// next slice is pushed. Every entry is started, even one that is not kept,
// because fflate keeps an unstarted entry's data in memory.
const decryptLivePhoto = async (
destination: string,
stream: ReadableStream<Uint8Array>,
header: Uint8Array,
key: Uint8Array,
onProgress: ProgressCallback | undefined,
file: EnteFile,
): Promise<DownloadResult> => {
const sliceSize = 4096;
const dir = dirname(destination);
const fail = (message: string, cause?: unknown): Error =>
new Error(`download: file ${file.id}: ${message}`, { cause });
const parts: LivePhotoPart[] = [];
try {
const image = await openPart("image", dir);
parts.push(image);
const video = await openPart("video", dir);
parts.push(video);
const claimed = new Set<LivePhotoPart>();
const unzip = new Unzip((entry) => {
const part = parts.find(
(p) => !claimed.has(p) && entry.name.startsWith(p.kind),
await stageAtomic(destination, async (handle) => {
bytesWritten = await streamDecrypt(
stream,
header,
key,
async (plaintext) => {
await handle.write(plaintext);
},
onProgress,
);
if (part !== undefined) claimed.add(part);
entry.ondata = (err, data, final) => {
if (err) throw err;
if (part === undefined) return;
chunkHashUpdate(part.hash, data);
part.pending.push(data);
if (final) part.ext = safeExtension(entry.name);
};
entry.start();
});
unzip.register(UnzipInflate);
const push = async (
data: Uint8Array,
final: boolean,
): Promise<void> => {
// fflate reports a bad ZIP by throwing, sometimes a TypeError,
// which the retry would take for a network failure; a bad ZIP is
// never retried.
try {
unzip.push(data, final);
} catch (err) {
throw fail("live photo is not a readable ZIP", err);
}
for (const part of parts) {
for (const piece of part.pending)
await part.handle.write(piece);
part.pending = [];
}
};
const bytesWritten = await streamDecrypt(
stream,
header,
key,
async (plaintext) => {
for (let i = 0; i < plaintext.length; i += sliceSize) {
await push(plaintext.subarray(i, i + sliceSize), false);
}
},
onProgress,
);
await push(new Uint8Array(0), true);
if (image.ext === undefined || video.ext === undefined) {
throw fail(
"live photo ZIP does not hold both an image and a video",
);
}
if (file.metadata.hash !== undefined) {
checkHash(
file,
`${chunkHashFinal(image.hash)}:${chunkHashFinal(video.hash)}`,
);
}
if (image.ext.toLowerCase() === video.ext.toLowerCase()) {
throw fail(
`live photo's image and video have the same extension, ${video.ext}`,
);
}
const path = withExtension(destination, image.ext);
const videoPath = withExtension(destination, video.ext);
for (const part of parts) {
await part.handle.sync();
await part.handle.close();
}
await rm(destination, { force: true });
await rename(image.tmpPath, path);
try {
await rename(video.tmpPath, videoPath);
} catch (err) {
await rm(path, { force: true }).catch(() => undefined);
throw err;
}
await fsyncPath(dir);
return { path, bytesWritten, videoPath };
} catch (err) {
// Best-effort cleanup, as in `stageAtomic`.
for (const part of parts) {
await part.handle.close().catch(() => undefined);
await rm(part.tmpPath, { force: true }).catch(() => undefined);
}
// Cancel the body so its connection is closed now rather than held
// until the stream is garbage collected. A backup run carries on past
// a failed file, so without this every failure would hold a socket.
// This covers every failure, including a temp file that cannot be
// opened and a header that is rejected before the body is read.
await stream.cancel(err).catch(() => undefined);
throw err;
}
return bytesWritten;
};
// Fetch a stream and decrypt it to `destination`, retrying the whole sequence.
@@ -507,45 +302,12 @@ const fetchAndDecrypt = async (
key: Uint8Array,
destination: string,
onProgress?: ProgressCallback,
original?: EnteFile,
): Promise<DownloadResult> =>
): Promise<number> =>
withRetry(async () => {
const stream = await openStream();
try {
if (original?.metadata.fileType === "livePhoto") {
return await decryptLivePhoto(
destination,
stream,
header,
key,
onProgress,
original,
);
}
const bytesWritten = await decryptToTemp(
destination,
stream,
header,
key,
onProgress,
original,
);
return { path: destination, bytesWritten };
} catch (err) {
// Cancel the body so its connection is closed now rather than held
// until the stream is garbage collected. A backup run carries on
// past a failed file, so without this every failure would hold a
// socket. This covers every failure, including a temp file that
// cannot be opened and a header that is rejected before the body
// is read.
await stream.cancel(err).catch(() => undefined);
throw err;
}
return decryptToTemp(destination, stream, header, key, onProgress);
}, api.getRetryOptions());
// Write `file`'s original to `outPath`. A live photo is written as its image
// and its video beside `outPath` instead, and whatever was at `outPath` is
// removed (see `decryptLivePhoto`).
export const downloadFile = async (
api: ApiClient,
file: EnteFile,
@@ -557,15 +319,15 @@ export const downloadFile = async (
const resolvedPath =
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
const header = fromBase64(file.file.decryptionHeader);
return fetchAndDecrypt(
const bytesWritten = await fetchAndDecrypt(
api,
() => api.getFileStream(file.id, { retry: false }),
header,
file.key,
resolvedPath,
onProgress,
file,
);
return { path: resolvedPath, bytesWritten };
};
export const downloadThumbnail = async (
@@ -578,7 +340,7 @@ export const downloadThumbnail = async (
outPath ??
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
const header = fromBase64(file.thumbnail.decryptionHeader);
return fetchAndDecrypt(
const bytesWritten = await fetchAndDecrypt(
api,
() => api.getThumbnailStream(file.id, { retry: false }),
header,
@@ -586,4 +348,5 @@ export const downloadThumbnail = async (
resolvedPath,
onProgress,
);
return { path: resolvedPath, bytesWritten };
};
src/filename.ts
-4
View File
@@ -35,7 +35,3 @@ export const safeExtension = (title: string): string => {
const ext = extname(title);
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
};
// `name` with its extension, if it has one, replaced by `ext` (".mov").
export const withExtension = (name: string, ext: string): string =>
name.slice(0, name.length - extname(name).length) + ext;
src/library/content.ts
+71 -247
View File
@@ -1,13 +1,11 @@
// The on-disk content and thumbnail cache keyed by fileID (issue #46).
//
// Layout under `cacheDirectory`: `originals/<fileID>.<ext>` and
// `thumbnails/<fileID>.<ext>`, flat directories at 0700 with files at 0600. A
// live photo's original is two files, its image and its video, with
// `originals/<fileID>.livephoto.json` naming them. Content appears only by the
// streaming atomic writer's rename (the download layer, #40), so a file that
// exists is whole — "present means complete". The directory listing taken at
// `open()` is the record of what is cached, and the orphan temp files a crashed
// write may have left are reaped there.
// `thumbnails/<fileID>.<ext>`, flat directories at 0700 with files at 0600.
// Content appears only by the streaming atomic writer's rename (the download
// layer, #40), so a file that exists is whole — "present means complete". The
// directory listing taken at `open()` is the record of what is cached, and the
// orphan temp files a crashed write may have left are reaped there.
//
// A fetch goes through the shared request pools (#45): the content pool for
// originals, the thumbnail pool for thumbnails. The pool limits concurrency,
@@ -17,14 +15,14 @@
// Integrity. The reused streaming decrypt is the enforced guarantee: every
// chunk is authenticated and the writer renames the file into place only once
// the stream ends on TAG_FINAL, so a truncated or corrupt fetch throws and
// nothing is stored. For an original whose metadata records a content hash
// (`FileMetadata.hash`), the writer also hashes the decrypted bytes and stores
// nothing if they differ, failing the fetch with an error naming the file. An
// original with no recorded hash is stored unchecked, as the upstream client
// does; thumbnails have none. On top of that this module refuses to record a
// stored file that came out empty.
// nothing is stored. On top of that this module refuses to record a stored file
// that came out empty. The design also asks for a content-hash comparison
// against `FileMetadata.hash` (with a `fileSize` fallback); that is deferred —
// see the PR — because the exact hash construction cannot be confirmed against
// the repo's fixtures and `FileBlob.size` is the encrypted object size, not the
// decrypted length this layer has.
import { existsSync, readFileSync, statSync } from "node:fs";
import { existsSync, statSync } from "node:fs";
import {
chmod,
mkdir,
@@ -34,15 +32,13 @@ import {
statfs,
utimes,
} from "node:fs/promises";
import { basename, dirname, extname, join } from "node:path";
import { dirname, extname, join } from "node:path";
import type { ApiClient } from "../api/client.js";
import {
downloadFile,
downloadThumbnail,
type ProgressCallback,
removeLeftoverTempFiles,
writeAtomic,
} from "../download/index.js";
import { safeExtension } from "../filename.js";
import type { EnteFile } from "../model/types.js";
@@ -50,6 +46,8 @@ import type { Priority, RequestPools } from "./pools.js";
const DIR_MODE = 0o700;
const FILE_MODE = 0o600;
const TEMP_PREFIX = ".quak-";
const TEMP_SUFFIX = ".tmp";
const GIB = 1024 * 1024 * 1024;
// Owner ruling (#36): bound the originals cache at 100 GiB, but back off when
// the volume has under 50 GiB free so the cache never crowds the disk.
@@ -80,8 +78,6 @@ const poolPriorityOf = (priority: ThumbnailPriority): Priority =>
export interface ContentResult {
path: string;
bytes: number;
// A live photo's video. `path` and `bytes` are then its image's.
videoPath?: string;
}
// Progress for a single `original`/`thumbnail` call. A present file emits one
@@ -132,14 +128,11 @@ export interface ThumbnailsAPI {
// stand-in so the cache logic runs with no crypto and no network. Pool routing,
// dedup, present-checks and integrity live in the cache, not here.
export interface ContentSource {
// Writes the original at `destination`. A live photo is written beside it
// as its image and its video instead, and their paths are returned, as
// `downloadFile` does.
original(args: {
file: EnteFile;
destination: string;
onProgress?: ProgressCallback;
}): Promise<{ bytesWritten: number; path?: string; videoPath?: string }>;
}): Promise<{ bytesWritten: number }>;
thumbnail(args: {
file: EnteFile;
destination: string;
@@ -217,9 +210,7 @@ class AbortDrop extends Error {
}
}
// The originals/ name for a file: `<fileID><ext>`, the extension taken from the
// title (or `.bin`). A backup names its originals the same way.
export const originalName = (file: EnteFile): string =>
const originalName = (file: EnteFile): string =>
`${file.id}${safeExtension(file.metadata.title)}`;
// The fileID a cache filename encodes, or undefined when the name is not one
@@ -241,73 +232,6 @@ const fileSize = (path: string): number | undefined => {
}
};
// Whether `path` is a regular file with content. A zero-byte file is the shape
// an aborted write leaves, so it does not count.
const hasContent = (path: string | undefined): boolean =>
path !== undefined && (fileSize(path) ?? 0) > 0;
// A live photo's image and video are named with the extensions from inside its
// ZIP, so their names alone do not say which is which. Wherever the cache or a
// backup stores one, this JSON file beside them names both.
const livePhotoFileName = (fileID: number): string =>
`${fileID}.livephoto.json`;
// The image and video that the live photo's JSON file in `dir` names, or
// undefined when there is none. Only names of the form the cache writes are
// taken, so the file cannot point outside `dir`.
const readLivePhoto = (
dir: string,
fileID: number,
): { path: string; videoPath: string } | undefined => {
const valid = (name: unknown): name is string =>
typeof name === "string" && name === `${fileID}${safeExtension(name)}`;
try {
const { image, video } = JSON.parse(
readFileSync(join(dir, livePhotoFileName(fileID)), "utf-8"),
);
if (valid(image) && valid(video)) {
return { path: join(dir, image), videoPath: join(dir, video) };
}
} catch {
// No such file, or not one the cache wrote.
}
return undefined;
};
// Write the JSON file naming a live photo's image and video, both in `dir`.
export const writeLivePhoto = (
dir: string,
fileID: number,
stored: { path: string; videoPath: string },
): Promise<void> =>
writeAtomic(
join(dir, livePhotoFileName(fileID)),
new TextEncoder().encode(
JSON.stringify({
image: basename(stored.path),
video: basename(stored.videoPath),
}),
),
);
// The original of `file` as the cache or a backup stored it in `dir`, when all
// of it is there: `<fileID><ext>`, or a live photo's image and video.
export const storedOriginal = (
dir: string,
file: EnteFile,
): { path: string; videoPath?: string } | undefined => {
if (file.metadata.fileType !== "livePhoto") {
const path = join(dir, originalName(file));
return hasContent(path) ? { path } : undefined;
}
const stored = readLivePhoto(dir, file.id);
return stored !== undefined &&
hasContent(stored.path) &&
hasContent(stored.videoPath)
? stored
: undefined;
};
export class ContentCache implements PhotoContent, ThumbnailsAPI {
private readonly pools: RequestPools;
private readonly source: ContentSource;
@@ -315,17 +239,10 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
private readonly getFile: (fileID: number) => EnteFile | undefined;
private readonly originalsDir: string;
private readonly thumbnailsDir: string;
// fileID -> absolute path of the cached bytes, and for a live photo's
// original its video's, seeded from the directory listing at open() and
// extended as fetches store new files.
private readonly originals = new Map<
number,
{ path: string; videoPath?: string }
>();
private readonly thumbnails = new Map<
number,
{ path: string; videoPath?: string }
>();
// fileID -> absolute path of the cached bytes, seeded from the directory
// listing at open() and extended as fetches store new files.
private readonly originals = new Map<number, string>();
private readonly thumbnails = new Map<number, string>();
private readonly maxOriginalsBytes: number;
private readonly freeBelowBytes: number;
private readonly isPinned: (fileID: number) => boolean;
@@ -385,9 +302,9 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
pathsFor(fileID: number): CachedPaths {
const out: CachedPaths = {};
const original = this.originals.get(fileID);
if (original !== undefined) out.originalPath = original.path;
if (original !== undefined) out.originalPath = original;
const thumbnail = this.thumbnails.get(fileID);
if (thumbnail !== undefined) out.thumbnailPath = thumbnail.path;
if (thumbnail !== undefined) out.thumbnailPath = thumbnail;
return out;
}
@@ -405,27 +322,6 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
}
// Get an original for a backup. One not present anywhere is written
// straight to `destination` and recorded there, so no second copy lands
// in the cache; one already present is returned where it is.
async backupOriginal(
fileID: number,
destination: string,
): Promise<ContentResult> {
const result = await this.acquire(
fileID,
"original",
"on-demand",
undefined,
{ destination },
);
return {
path: result.path,
bytes: result.bytes,
videoPath: result.videoPath,
};
}
async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
return this.ensureThumbnails(args);
}
@@ -522,72 +418,51 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
? { status: "skipped", bytes: result.bytes }
: { status: "done", bytes: result.bytes },
);
return {
path: result.path,
bytes: result.bytes,
videoPath: result.videoPath,
};
return { path: result.path, bytes: result.bytes };
}
// The core: return the cached path if present, else fetch through the pool,
// store, and return it. `cached` distinguishes a present hit (no network,
// no download event) from a fresh fetch. A fetched original is stored at
// `opts.destination` when given, instead of in `originalsDir`. A live
// photo's original is present only with its video, and is returned with
// it.
// no download event) from a fresh fetch.
private async acquire(
fileID: number,
kind: Kind,
priority: Priority,
signal: AbortSignal | undefined,
opts?: { onByte?: ProgressCallback; destination?: string },
): Promise<{
path: string;
bytes: number;
videoPath?: string;
cached: boolean;
}> {
opts?: { onByte?: ProgressCallback },
): Promise<{ path: string; bytes: number; cached: boolean }> {
const file = this.getFile(fileID);
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
const isLivePhoto =
kind === "original" && file.metadata.fileType === "livePhoto";
const known = kind === "original" ? this.originals : this.thumbnails;
const cached = known.get(fileID);
if (cached !== undefined) {
const size = fileSize(cached.path);
if (
size !== undefined &&
size > 0 &&
(!isLivePhoto || hasContent(cached.videoPath))
) {
const size = fileSize(cached);
if (size !== undefined && size > 0) {
// Returning an original's path is a use: bump its mtime so LRU
// order reflects it and survives a restart with no ledger.
if (
kind === "original" &&
dirname(cached.path) === this.originalsDir
dirname(cached) === this.originalsDir
)
await this.touch(cached.path);
return { ...cached, bytes: size, cached: true };
await this.touch(cached);
return { path: cached, bytes: size, cached: true };
}
// A recorded file that has since gone, or a live photo an earlier
// version stored as one ZIP, re-fetches below.
// A recorded file that has since gone re-fetches below.
known.delete(fileID);
}
// An original a backup already stored counts as present.
if (kind === "original" && this.downloadDirectory !== undefined) {
const stored = storedOriginal(
join(this.downloadDirectory, "originals"),
file,
const backupPath = join(
this.downloadDirectory,
"originals",
originalName(file),
);
if (stored !== undefined) {
this.originals.set(fileID, stored);
return {
...stored,
bytes: fileSize(stored.path) ?? 0,
cached: true,
};
const size = fileSize(backupPath);
if (size !== undefined && size > 0) {
this.originals.set(fileID, backupPath);
return { path: backupPath, bytes: size, cached: true };
}
}
@@ -595,7 +470,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
kind === "original" ? this.originalsDir : this.thumbnailsDir;
const dest =
kind === "original"
? (opts?.destination ?? join(dir, originalName(file)))
? join(dir, originalName(file))
: join(dir, `${fileID}${THUMBNAIL_EXT}`);
const pool =
kind === "original" ? this.pools.content : this.pools.thumbnails;
@@ -616,39 +491,21 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
? this.beginOriginalWrite(fileID)
: null;
try {
const stored = await this.download(
file,
dest,
kind,
opts?.onByte,
);
for (const path of [stored.path, stored.videoPath]) {
if (path === undefined) continue;
await chmod(path, FILE_MODE);
if ((await stat(path)).size === 0) {
throw new Error(
`content cache: ${kind} ${fileID} stored empty`,
);
}
await this.download(file, dest, kind, opts?.onByte);
await chmod(dest, FILE_MODE);
const size = (await stat(dest)).size;
if (size === 0) {
throw new Error(
`content cache: ${kind} ${fileID} stored empty`,
);
}
// A backup records its own live photos.
if (
stored.videoPath !== undefined &&
opts?.destination === undefined
) {
await writeLivePhoto(dir, fileID, {
path: stored.path,
videoPath: stored.videoPath,
});
}
known.set(fileID, stored);
known.set(fileID, dest);
// A fresh original may have crossed the limit; make room by
// evicting least-recently-used originals. An over-budget
// fetch keeps the file it returns, and no overlapping
// sibling is evicted. Thumbnails are never bounded.
if (write) await this.enforceOriginalsLimit(write);
const size = (await stat(stored.path)).size;
return { ...stored, bytes: size, cached: false };
return { path: dest, bytes: size, cached: false };
} finally {
if (write) this.inFlightOriginals.delete(write);
}
@@ -657,24 +514,18 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
);
}
// Fetch into `destination`, returning where the bytes landed: there, or
// for a live photo, its image and video beside it.
private async download(
file: EnteFile,
destination: string,
kind: Kind,
onProgress: ProgressCallback | undefined,
): Promise<{ path: string; videoPath?: string }> {
): Promise<number> {
const args = { file, destination, onProgress };
if (kind === "thumbnail") {
await this.source.thumbnail(args);
return { path: destination };
}
const result = await this.source.original(args);
return {
path: result.path ?? destination,
videoPath: result.videoPath,
};
const result =
kind === "original"
? await this.source.original(args)
: await this.source.thumbnail(args);
return result.bytesWritten;
}
// Best-effort bump of a file's mtime to now; a failed touch must never fail
@@ -685,14 +536,13 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
}
// Every stored original that lives under `originalsDir` (a backup-directory
// hit recorded in the map is excluded), with its size and mtime; a live
// photo's size includes its video. Entries whose file has vanished are
// dropped from the map. Backups and thumbnails are never counted.
// hit recorded in the map is excluded), with its size and mtime. Entries
// whose file has vanished are dropped from the map. Backups and thumbnails
// are never counted.
private async measureOriginals(): Promise<{
entries: {
fileID: number;
path: string;
videoPath?: string;
size: number;
mtimeMs: number;
}[];
@@ -701,28 +551,21 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
const entries: {
fileID: number;
path: string;
videoPath?: string;
size: number;
mtimeMs: number;
}[] = [];
let used = 0;
for (const [fileID, { path, videoPath }] of this.originals) {
for (const [fileID, path] of this.originals) {
if (dirname(path) !== this.originalsDir) continue;
try {
const s = await stat(path);
const size =
s.size +
(videoPath === undefined
? 0
: (await stat(videoPath)).size);
entries.push({
fileID,
path,
videoPath,
size,
size: s.size,
mtimeMs: s.mtimeMs,
});
used += size;
used += s.size;
} catch {
this.originals.delete(fileID);
}
@@ -786,18 +629,6 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
for (const e of evictable) {
if (remaining <= limit) break;
await rm(e.path, { force: true });
// A live photo goes whole: its video and the JSON file
// naming the two go with its image.
if (e.videoPath !== undefined) {
await rm(e.videoPath, { force: true });
await rm(
join(
this.originalsDir,
livePhotoFileName(e.fileID),
),
{ force: true },
);
}
this.originals.delete(e.fileID);
remaining -= e.size;
}
@@ -822,30 +653,23 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
await chmod(dir, DIR_MODE);
}
private async scan(
dir: string,
into: Map<number, { path: string; videoPath?: string }>,
): Promise<void> {
// Another process sharing this cache may still be writing its temp
// files, so only those whose process has exited are removed.
removeLeftoverTempFiles(dir);
private async scan(dir: string, into: Map<number, string>): Promise<void> {
let entries: string[];
try {
entries = await readdir(dir);
} catch {
return;
}
const names = new Set(entries);
for (const name of entries) {
if (name.startsWith(TEMP_PREFIX) && name.endsWith(TEMP_SUFFIX)) {
await rm(join(dir, name), { force: true }).catch(
() => undefined,
);
continue;
}
const id = fileIDFromName(name);
const path = join(dir, name);
if (id === undefined || !existsSync(path)) continue;
// A live photo's image and video are one entry, as the JSON file
// beside them names them.
const livePhoto = names.has(livePhotoFileName(id))
? readLivePhoto(dir, id)
: undefined;
into.set(id, livePhoto ?? { path });
if (id !== undefined && existsSync(path)) into.set(id, path);
}
}
}
src/library/index.ts
+12 -31
View File
@@ -26,7 +26,6 @@
// store marked unsaved until a later save actually lands, so a stuck disk is
// never masked by a subsequent empty refresh.
import { rm } from "node:fs/promises";
import { join } from "node:path";
import envPaths from "env-paths";
@@ -98,11 +97,6 @@ export {
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
// The account's cache directory when `cacheDirectory` is not given: the
// env-paths cache directory plus the user id, so each account has its own.
export const defaultCacheDirectory = (userID: number): string =>
join(envPaths("quak", { suffix: "" }).cache, String(userID));
// Project a metadata store into by-id records, filling each record's cache
// paths from the content cache when one is given. Shared by the live read
// projection and the precache's initial seeding at open().
@@ -349,21 +343,11 @@ export class Library {
static async open(opts: LibraryOptions): Promise<Library> {
const { userID } = opts.client.whoami();
const cacheDirectory =
opts.cacheDirectory ?? defaultCacheDirectory(userID);
const metadataPath = join(cacheDirectory, "metadata.json");
let store = await MetadataStore.load(metadataPath);
// A cache directory given explicitly can hold another account's cache.
// Its records and cursor are not this account's, so delete it and the
// ML data beside it and start empty. A user ID of 0 means the cache
// was never refreshed and so holds nothing to discard.
if (store.userID !== 0 && store.userID !== userID) {
await rm(metadataPath, { force: true });
await rm(join(cacheDirectory, "mldata"), {
recursive: true,
force: true,
});
store = await MetadataStore.load(metadataPath);
}
opts.cacheDirectory ??
join(envPaths("quak", { suffix: "" }).cache, String(userID));
const store = await MetadataStore.load(
join(cacheDirectory, "metadata.json"),
);
const intervalMs =
(opts.refreshIntervalSeconds ?? DEFAULT_REFRESH_INTERVAL_SECONDS) *
1000;
@@ -538,13 +522,11 @@ export class Library {
}
// Back up every in-scope file to `downloadDirectory` in the historical
// on-disk layout, with a durable failure ledger (issue #51). Waits for a
// completed refresh first, as `fresh()` does, joining one already running,
// and rejects before touching any file when it fails. Then fetches pending
// originals (and optional thumbnails) through the content cache and pools,
// and rebuilds the derived symlink/JSON views from the model. Throws before
// any network work when no download directory is available or no content
// cache backs the originals it must fetch.
// on-disk layout, with a durable failure ledger (issue #51). Refreshes
// first, fetches pending originals (and optional thumbnails) through the
// content cache and pools, then rebuilds the derived symlink/JSON views
// from the model. Throws before any network work when no download directory
// is available or no content cache backs the originals it must fetch.
backup(opts?: BackupOptions): Promise<BackupResult> {
const downloadDirectory =
opts?.downloadDirectory ?? this.downloadDirectory;
@@ -568,11 +550,10 @@ export class Library {
const cache = this.cache;
return runBackup(
{
refresh: () => this.refreshNow(),
refresh: () => this.runRefresh(),
listCollections: () => this.store.listCollections(),
listFiles: (id) => this.store.listFiles(id),
original: (fileID, destination) =>
cache!.backupOriginal(fileID, destination),
original: (fileID) => cache!.original(fileID),
thumbnail: (fileID) => cache!.thumbnail(fileID),
},
{ ...opts, downloadDirectory },
src/library/read.ts
+1 -2
View File
@@ -80,8 +80,7 @@ export class Photo {
}
// Fetch and cache the full-resolution original, returning its on-disk path
// and byte length; for a live photo, its image's, and its video's path as
// `videoPath`. Served from the cache (or the backup download directory)
// and byte length. Served from the cache (or the backup download directory)
// when already present, otherwise fetched through the content pool.
async original(opts?: ContentOptions): Promise<ContentResult> {
return this.contentOrThrow().original(this.rec.fileID, opts);
src/library/records.ts
-1
View File
@@ -42,7 +42,6 @@ export interface PhotoRecord {
isArchived: boolean;
isHidden: boolean;
// Local cache paths, set once a later phase caches the bytes; unset here.
// A live photo's `originalPath` is its image.
thumbnailPath?: string;
originalPath?: string;
}
src/metadata-backup.ts
+10 -38
View File
@@ -5,11 +5,7 @@ import exifReader from "exif-reader";
import type { Client } from "./client.js";
import type { Library, Photo } from "./library/index.js";
import { sanitizeFileName } from "./filename.js";
import {
fetchMLDataBatch,
MLDATA_BATCH_SIZE,
type MLData,
} from "./mldata-fetch.js";
import { fetchMLData } from "./mldata-fetch.js";
import type { EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void;
@@ -128,8 +124,7 @@ export const extractImageMetadata = (
// Read a file's original bytes through the library's content cache and extract
// its embedded image metadata. The bytes come from `photo.original()` — the
// same on-disk cache the rest of the library fills — rather than a fresh
// per-call download to a throwaway temp file. For a live photo, its `path` is
// the image.
// per-call download to a throwaway temp file.
const extractExif = async (
photo: Photo,
): Promise<Record<string, unknown> | undefined> => {
@@ -141,15 +136,14 @@ const extractExif = async (
// Dump every decrypted metadata layer the account holds into a directory tree
// of plain JSON: account, per-collection, and per-file records including the
// private and public magic metadata and (by default) the ML data. Collections
// and files are enumerated from the library's cache, which the caller refreshes
// first. Returns how many ML data requests failed; their files are still
// written, with `mlDataError` in place of `mlData`.
// and files are enumerated from the library's cache rather than a fresh server
// scan; the ML fetch and EXIF extraction are unchanged.
export const runMetadataBackup = async (
lib: Library,
client: Client,
outDir: string,
opts?: MetadataBackupOptions,
): Promise<{ failedMLBatches: number }> => {
): Promise<void> => {
const log = opts?.onProgress ?? (() => {});
const wantExif = opts?.exif ?? false;
@@ -214,31 +208,12 @@ export const runMetadataBackup = async (
}
}
// One failed request (retries exhausted) must not end the dump: its files
// get the reason in `mlDataError` and the other batches go on.
log("Fetching ML data (face detections, CLIP embeddings)...");
const mlDataMap = new Map<number, MLData>();
const mlDataErrors = new Map<number, string>();
let failedMLBatches = 0;
const fileIDs = [...fileKeys.keys()];
for (let i = 0; i < fileIDs.length; i += MLDATA_BATCH_SIZE) {
const batch = fileIDs.slice(i, i + MLDATA_BATCH_SIZE);
try {
const result = await fetchMLDataBatch(
client.getApiClient(),
batch,
fileKeys,
);
for (const [id, payload] of result) mlDataMap.set(id, payload);
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
failedMLBatches++;
log(
`ML data request for ${batch.length} file(s) failed: ${reason}`,
);
for (const id of batch) mlDataErrors.set(id, reason);
}
}
const mlDataMap = await fetchMLData(
client.getApiClient(),
[...fileKeys.keys()],
fileKeys,
);
log(`Got ML data for ${mlDataMap.size} file(s)`);
const writtenFileIDs = new Set<number>();
@@ -258,8 +233,6 @@ export const runMetadataBackup = async (
const ml = mlDataMap.get(file.id);
if (ml) fileMeta.mlData = ml;
const mlError = mlDataErrors.get(file.id);
if (mlError) fileMeta.mlDataError = mlError;
if (wantExif && !writtenFileIDs.has(file.id)) {
log(`[${file.metadata.title}] Extracting EXIF...`);
@@ -280,5 +253,4 @@ export const runMetadataBackup = async (
}
log("Metadata backup complete.");
return { failedMLBatches };
};
src/mldata-fetch.ts
+25 -2
View File
@@ -5,8 +5,9 @@
// comes back encrypted under the file's own key and gzipped; decrypting and
// gunzipping yields the JSON payload
// `{ face: { faces: [...] }, clip: { embedding } }`. Ente caps a request at 200
// ids, so callers that want many at once split them into batches of
// `MLDATA_BATCH_SIZE` and call `fetchMLDataBatch` once per batch.
// ids, so `fetchMLData` batches for callers that want many at once while
// `fetchMLDataBatch` is the single-request unit the library submits to its
// request pool.
import { gunzipSync } from "node:zlib";
@@ -68,3 +69,25 @@ export const fetchMLDataBatch = async (
}
return result;
};
// Fetch ML data for arbitrarily many ids, batching at `MLDATA_BATCH_SIZE`. Used
// by the one-shot metadata backup; the library fetches through its request pool
// with `fetchMLDataBatch` instead.
export const fetchMLData = async (
api: ApiClient,
fileIDs: number[],
fileKeys: Map<number, Uint8Array>,
): Promise<Map<number, MLData>> => {
const result = new Map<number, MLData>();
for (let i = 0; i < fileIDs.length; i += MLDATA_BATCH_SIZE) {
const batch = fileIDs.slice(i, i + MLDATA_BATCH_SIZE);
for (const [id, payload] of await fetchMLDataBatch(
api,
batch,
fileKeys,
)) {
result.set(id, payload);
}
}
return result;
};
src/model/decrypt.ts
+1 -23
View File
@@ -34,28 +34,6 @@ const FILE_TYPE_MAP: Record<number, FileType> = {
const parseFileType = (n: number): FileType => FILE_TYPE_MAP[n] ?? "unknown";
// The hash the uploading client recorded for the original's bytes, read the
// way the upstream client's `metadataHash` reads it: `hash` if present,
// otherwise, for a live photo from an older client that wrote the two parts
// separately, `<imageHash>:<videoHash>`. A field that is not a non-empty
// string counts as absent, and a file with no hash at all is normal.
const expectedHash = (json: Record<string, unknown>): string | undefined => {
const text = (v: unknown): string | undefined =>
typeof v === "string" && v !== "" ? v : undefined;
const hash = text(json.hash);
if (hash !== undefined) return hash;
const imageHash = text(json.imageHash);
const videoHash = text(json.videoHash);
if (
json.fileType === 2 &&
imageHash !== undefined &&
videoHash !== undefined
) {
return `${imageHash}:${videoHash}`;
}
return undefined;
};
export const decryptCollection = (
raw: RawCollection,
keys: KeyMaterial,
@@ -137,7 +115,7 @@ export const decryptFile = (
modificationTime: metadataJSON.modificationTime ?? 0,
latitude: metadataJSON.latitude,
longitude: metadataJSON.longitude,
hash: expectedHash(metadataJSON),
hash: metadataJSON.hash,
};
const magicMetadata = decryptMagicMetadata(raw.magicMetadata, key);
src/model/types.ts
-3
View File
@@ -29,9 +29,6 @@ export interface FileMetadata {
modificationTime: Microseconds;
latitude?: number;
longitude?: number;
// The content hash the uploader recorded (see `expectedHash` in
// decrypt.ts); `downloadFile` refuses an original that does not match it.
// Absent for files from very old clients.
hash?: string;
}
src/thumbnails.ts
+30 -88
View File
@@ -7,21 +7,8 @@ import { ApiError } from "./api/client.js";
import { encryptBlob, toBase64 } from "./crypto/index.js";
import type { EnteFile } from "./model/types.js";
// The server refuses a thumbnail larger than the one it already records for the
// file (`thumbnail.size`, the encrypted size), so these encodings are tried
// from largest to smallest and the first that fits is uploaded.
const THUMB_ENCODINGS = [
{ maxDimension: 720, quality: 50 },
{ maxDimension: 720, quality: 30 },
{ maxDimension: 480, quality: 30 },
{ maxDimension: 320, quality: 20 },
{ maxDimension: 160, quality: 20 },
];
// The server accepts a new thumbnail only from the file's owner, so files other
// people own in albums shared with this account are never checked or repaired.
const NOT_OWNED_REASON =
"owned by another account (only the owner can replace its thumbnail)";
const THUMB_MAX_DIMENSION = 720;
const THUMB_JPEG_QUALITY = 50;
export interface MissingThumbnailInfo {
fileID: number;
@@ -32,12 +19,11 @@ export interface MissingThumbnailInfo {
// Three outcomes, not two. "fixed": a thumbnail was generated and uploaded.
// "failed": something went wrong (download, encode, upload) and the file still
// has no thumbnail. "skipped": the server would refuse any thumbnail for the
// file or this helper cannot regenerate it — a file another account owns, a
// recorded thumbnail size nothing fits within, a video, or an image that is
// not a baseline JPEG. Skipped is a deliberate, expected outcome, not an error
// (issue #17): the repair path is JPEG-only because `jpeg-js` is, and a PNG or
// HEIC is left for a format-aware tool rather than reported as a failure.
// has no thumbnail. "skipped": the file is a format this helper cannot
// regenerate — a video, or an image that is not a baseline JPEG. Skipped is a
// deliberate, expected outcome, not an error (issue #17): the repair path is
// JPEG-only because `jpeg-js` is, and a PNG or HEIC is left for a format-aware
// tool rather than reported as a failure.
export type ThumbnailFixStatus = "fixed" | "skipped" | "failed";
export interface ThumbnailFixResult {
@@ -59,7 +45,6 @@ export type ProgressCallback = (message: string) => void;
// exists, so it is logged and the file is left unreported. That distinction is
// what stops `fix-missing-thumbnails` from regenerating and uploading over
// thumbnails that were fine all along while the CDN was briefly returning 500s.
// Files another account owns are logged as skipped and not checked.
export const listMissingThumbnails = async (
lib: Library,
client: Client,
@@ -67,7 +52,6 @@ export const listMissingThumbnails = async (
): Promise<MissingThumbnailInfo[]> => {
const log = onProgress ?? (() => {});
const api = client.getApiClient();
const { userID } = client.whoami();
const missing: MissingThumbnailInfo[] = [];
const seen = new Set<number>();
@@ -76,13 +60,6 @@ export const listMissingThumbnails = async (
for (const photo of album.photos.list()) {
if (seen.has(photo.fileID)) continue;
seen.add(photo.fileID);
const file = lib.getFile(album.collectionID, photo.fileID);
if (file && file.ownerID !== userID) {
log(
`[${album.name}] Skipping ${photo.title}: ${NOT_OWNED_REASON}`,
);
continue;
}
try {
const stream = await api.getThumbnailStream(photo.fileID);
const reader = stream.getReader();
@@ -158,13 +135,17 @@ const resizeRGBA = (
return dst;
};
const generateThumbnail = (
decoded: { data: Uint8Array; width: number; height: number },
maxDimension: number,
quality: number,
): Uint8Array => {
const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: true,
});
const { width: srcW, height: srcH } = decoded;
const scale = Math.min(maxDimension / srcW, maxDimension / srcH, 1);
const scale = Math.min(
THUMB_MAX_DIMENSION / srcW,
THUMB_MAX_DIMENSION / srcH,
1,
);
const dstW = Math.round(srcW * scale);
const dstH = Math.round(srcH * scale);
@@ -177,7 +158,7 @@ const generateThumbnail = (
const encoded = jpeg.encode(
{ data: pixels, width: dstW, height: dstH },
quality,
THUMB_JPEG_QUALITY,
);
return new Uint8Array(encoded.data);
};
@@ -190,35 +171,14 @@ const generateThumbnail = (
const isJpeg = (bytes: Uint8Array): boolean =>
bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xd8;
// The reason a file cannot have a JPEG thumbnail regenerated for it, known from
// its record alone before any bytes are fetched, or undefined when it might. A
// still image still has to be checked against its actual bytes once
// downloaded.
const reasonToSkip = (file: EnteFile, userID: number): string | undefined => {
if (file.ownerID !== userID) {
return NOT_OWNED_REASON;
}
// The reason a file cannot have a JPEG thumbnail regenerated for it from its
// metadata alone, before any bytes are fetched, or undefined when it might. A
// non-image (video, live photo) is unsupported outright; a still image still
// has to be checked against its actual bytes once downloaded.
const unsupportedByType = (file: EnteFile): string | undefined => {
if (file.metadata.fileType !== "image") {
return `unsupported file type: ${file.metadata.fileType} (only JPEG images can be regenerated)`;
}
if (!file.thumbnail.size) {
return `recorded thumbnail size is ${file.thumbnail.size ?? "unknown"} (the server refuses a thumbnail larger than the one it records)`;
}
return undefined;
};
// Encrypt the largest encoding of the decoded image whose ciphertext is no
// larger than `maxSize`, or return undefined when even the smallest is larger.
const encryptThumbnailWithin = (
decoded: { data: Uint8Array; width: number; height: number },
key: Uint8Array,
maxSize: number,
): { header: Uint8Array; ciphertext: Uint8Array } | undefined => {
for (const { maxDimension, quality } of THUMB_ENCODINGS) {
const thumbJpeg = generateThumbnail(decoded, maxDimension, quality);
const encrypted = encryptBlob(thumbJpeg, key);
if (encrypted.ciphertext.length <= maxSize) return encrypted;
}
return undefined;
};
@@ -237,7 +197,6 @@ export const fixMissingThumbnails = async (
const log = onProgress ?? (() => {});
const results: ThumbnailFixResult[] = [];
const api = client.getApiClient();
const { userID } = client.whoami();
// Resolve each requested fileID to its file record and owning album by
// enumerating the library, each file taken from the first album that holds
@@ -278,19 +237,18 @@ export const fixMissingThumbnails = async (
const { file, collectionName } = entry;
const title = file.metadata.title;
const skipReason = reasonToSkip(file, userID);
if (skipReason) {
log(`[${collectionName}] Skipping ${title}: ${skipReason}`);
const typeReason = unsupportedByType(file);
if (typeReason) {
log(`[${collectionName}] Skipping ${title}: ${typeReason}`);
results.push({
fileID,
title,
collection: collectionName,
status: "skipped",
reason: skipReason,
reason: typeReason,
});
continue;
}
const maxSize = file.thumbnail.size!;
try {
const photo = lib.photos.byID({ fileID });
@@ -319,28 +277,12 @@ export const fixMissingThumbnails = async (
}
log(`[${collectionName}] Generating thumbnail for ${title}...`);
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: true,
});
const fitting = encryptThumbnailWithin(decoded, file.key, maxSize);
if (!fitting) {
const reason = `no thumbnail encoding fits the recorded thumbnail size of ${maxSize} bytes`;
log(`[${collectionName}] Skipping ${title}: ${reason}`);
results.push({
fileID,
title,
collection: collectionName,
status: "skipped",
reason,
});
continue;
}
const { header, ciphertext } = fitting;
const thumbJpeg = generateThumbnail(fileBytes);
log(
`[${collectionName}] Uploading thumbnail (${ciphertext.length} bytes)...`,
`[${collectionName}] Encrypting and uploading thumbnail (${thumbJpeg.length} bytes)...`,
);
const { header, ciphertext } = encryptBlob(thumbJpeg, file.key);
const md5 = createHash("md5").update(ciphertext).digest("base64");
const { objectKey, url } = await api.getUploadURL(
ciphertext.length,
test/cli/backup.test.ts
+1 -627
View File
@@ -40,7 +40,6 @@ import {
readFileSync,
readlinkSync,
rmSync,
symlinkSync,
writeFileSync,
} from "node:fs";
import { spawnSync } from "node:child_process";
@@ -48,18 +47,10 @@ import { join } from "node:path";
import { tmpdir } from "node:os";
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { runBackup, type BackupLibrary } from "../../src/backup.js";
import { Library } from "../../src/library/index.js";
import type { ContentSource } from "../../src/library/content.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import {
asLivePhoto,
cdnSource,
IMAGE,
livePhotoZip,
VIDEO,
} from "../live-photo.js";
// `open` and `rename` are wrapped to record, in order, every fsync and rename,
// so a test can pin the sequence "fsync the temp file, rename, fsync the
@@ -569,39 +560,11 @@ describe("lib.backup", () => {
lib.close();
});
it("fetches each original once and writes it only into the backup", async () => {
// A backup of a 500 GB account must write 500 GB, not a copy in the
// cache as well: an original fetched for the backup goes straight
// into its originals/, and the cache records it there.
const source = stubSource();
const lib = await openLibrary(source);
const outDir = join(root, "backup");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result.downloaded).toBe(3);
expect(source.originalCalls).toBe(3);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
const stored = readdirSync(join(outDir, "originals")).filter(
(name) => !name.endsWith(".json"),
);
expect(stored.sort()).toEqual(["100.jpg", "101.jpg", "200.png"]);
// The cache counts the backup's copy as present: reading the
// original afterwards fetches nothing and answers with that copy.
const read = await lib.photos.byID({ fileID: 100 })!.original();
expect(read.path).toBe(join(outDir, "originals", "100.jpg"));
expect(source.originalCalls).toBe(3);
await lib.close();
});
it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
it("fsyncs a copied original before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource());
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const dest = join(originals, "100.jpg");
// Only an original already in the cache is copied into the backup;
// one fetched for the backup is written there by the download writer.
await lib.photos.byID({ fileID: 100 })!.original();
fsEvents.length = 0;
await lib.backup({ downloadDirectory: outDir });
@@ -659,592 +622,3 @@ describe("lib.backup", () => {
lib.close();
});
});
// Every refresh fails, as with an expired session or no network.
class FailingClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
throw new Error("HTTP 401 from server");
}
}
// Holds its refresh open until `release()` is called, then reports a third
// album, so a backup can be started while that refresh is still running.
class HeldClient extends MockClient {
release!: () => void;
private held = new Promise<void>((resolve) => {
this.release = resolve;
});
override async collectionsSince(): Promise<CollectionsPage> {
await this.held;
return {
collections: [collection(3, "Later")],
deleted: [],
cursor: 2,
};
}
override async filesSince(args: {
collectionID: number;
}): Promise<FilesPage> {
if (args.collectionID !== 3) return super.filesSince(args);
return { files: [file(300, 3, "late.jpg")], deleted: [], cursor: 2 };
}
}
// Fill the library cache on disk, so the next open starts its refresh in the
// background instead of waiting for it.
const fillCache = async (): Promise<void> => {
const lib = await openLibrary(stubSource());
await lib.close();
};
describe("the refresh before a backup", () => {
it("waits for a refresh already running and backs up what it found", async () => {
await fillCache();
const client = new HeldClient();
const lib = await openLibrary(stubSource(), client);
const outDir = join(root, "backup");
const backup = lib.backup({ downloadDirectory: outDir });
client.release();
const result = await backup;
expect(result.totalFiles).toBe(4);
expect(existsSync(join(outDir, "originals", "300.jpg"))).toBe(true);
await lib.close();
});
it("fails before any download when the refresh fails, leaving failures.json as it was", async () => {
await fillCache();
const outDir = join(root, "backup");
seedLedger(outDir, 100, "beach.jpg");
const ledgerPath = join(outDir, "failures.json");
const ledgerBefore = readFileSync(ledgerPath, "utf-8");
const source = stubSource();
const lib = await openLibrary(source, new FailingClient());
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
"HTTP 401 from server",
);
expect(source.originalCalls).toBe(0);
expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore);
expect(existsSync(join(outDir, "originals"))).toBe(false);
await lib.close();
});
it("fails when the refresh fails on an empty cache, instead of backing up nothing", async () => {
const source = stubSource();
const lib = await openLibrary(source, new FailingClient());
const outDir = join(root, "backup");
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
"HTTP 401 from server",
);
expect(source.originalCalls).toBe(0);
expect(existsSync(outDir)).toBe(false);
await lib.close();
});
});
// Every entry under collections/, one level of directories deep, with each
// symlink's target.
const tree = (outDir: string): string[] => {
const lines: string[] = [];
const list = (dir: string, prefix: string): void => {
for (const name of readdirSync(dir).sort()) {
const path = join(dir, name);
const st = lstatSync(path);
if (st.isSymbolicLink()) {
lines.push(`${prefix}${name} -> ${readlinkSync(path)}`);
} else if (st.isDirectory() && prefix === "") {
lines.push(`${name}/`);
list(path, `${name}/`);
} else {
lines.push(`${prefix}${name}`);
}
}
};
list(join(outDir, "collections"), "");
return lines;
};
// The album folders under collections/, driven through `runBackup` with a
// stand-in library whose albums a test changes between runs.
describe("backup album folders", () => {
interface Album {
collection: Collection;
files: EnteFile[];
}
const libraryOf = (albums: Album[]): BackupLibrary => ({
refresh: async () => {},
listCollections: () => albums.map((a) => a.collection),
listFiles: (id) =>
albums.find((a) => a.collection.id === id)?.files ?? [],
original: async (fileID) => {
const path = join(root, `source-${fileID}`);
writeFileSync(path, `original ${fileID}`);
return { path };
},
thumbnail: async () => {
throw new Error("no thumbnails in this stand-in");
},
});
const albumID = (outDir: string, jsonName: string): number =>
JSON.parse(readFileSync(join(outDir, "collections", jsonName), "utf-8"))
.id;
it("gives every file and every album its own name when names repeat", async () => {
const outDir = join(root, "backup");
const lib = libraryOf([
{
collection: collection(10, "Trip"),
files: [
file(1, 10, "IMG_0001.JPG"),
file(2, 10, "IMG_0001.JPG"),
file(4, 10, "img_0001.jpg"),
file(3, 10, "other.jpg"),
],
},
{
collection: collection(11, "Trip"),
files: [file(3, 11, "other.jpg")],
},
]);
const result = await runBackup(lib, { downloadDirectory: outDir });
expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([
"Trip (10)/",
"Trip (10)/IMG_0001 (1).JPG -> ../../originals/1.JPG",
"Trip (10)/IMG_0001 (2).JPG -> ../../originals/2.JPG",
"Trip (10)/img_0001 (4).jpg -> ../../originals/4.jpg",
"Trip (10)/other.jpg -> ../../originals/3.jpg",
"Trip (10).json",
"Trip (11)/",
"Trip (11)/other.jpg -> ../../originals/3.jpg",
"Trip (11).json",
]);
expect(albumID(outDir, "Trip (10).json")).toBe(10);
expect(albumID(outDir, "Trip (11).json")).toBe(11);
});
it("keeps names unique when a name with an ID added is another entry's own name", async () => {
const outDir = join(root, "backup");
const lib = libraryOf([
{
collection: collection(10, "Trip"),
files: [
file(5, 10, "IMG (6).JPG"),
file(6, 10, "IMG.JPG"),
file(7, 10, "IMG.JPG"),
],
},
{
collection: collection(11, "Trip"),
files: [file(8, 11, "a.jpg")],
},
{
collection: collection(12, "Trip (11)"),
files: [file(9, 12, "b.jpg")],
},
]);
const result = await runBackup(lib, { downloadDirectory: outDir });
expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([
"Trip (10)/",
"Trip (10)/IMG (6) (5).JPG -> ../../originals/5.JPG",
"Trip (10)/IMG (6).JPG -> ../../originals/6.JPG",
"Trip (10)/IMG (7).JPG -> ../../originals/7.JPG",
"Trip (10).json",
"Trip (11)/",
"Trip (11)/a.jpg -> ../../originals/8.jpg",
"Trip (11) (12)/",
"Trip (11) (12)/b.jpg -> ../../originals/9.jpg",
"Trip (11) (12).json",
"Trip (11).json",
]);
expect(albumID(outDir, "Trip (10).json")).toBe(10);
expect(albumID(outDir, "Trip (11).json")).toBe(11);
expect(albumID(outDir, "Trip (11) (12).json")).toBe(12);
});
it("changes nothing on a second run over an unchanged account", async () => {
const outDir = join(root, "backup");
const lib = libraryOf([
{
collection: collection(10, "Trip"),
files: [
file(1, 10, "IMG_0001.JPG"),
file(2, 10, "IMG_0001.JPG"),
],
},
{
collection: collection(11, "Trip"),
files: [file(3, 11, "other.jpg")],
},
]);
await runBackup(lib, { downloadDirectory: outDir });
const before = tree(outDir);
const second = await runBackup(lib, { downloadDirectory: outDir });
expect(second.downloaded).toBe(0);
expect(second.failed).toBe(0);
expect(tree(outDir)).toEqual(before);
});
it("leaves the albums an onlyAlbumNames run skips as they were", async () => {
const outDir = join(root, "backup");
// "trip" is skipped by the scoped run but its name clashes with the
// in-scope "Trip", so "Trip" must keep its ID suffix.
const lib = libraryOf([
{
collection: collection(10, "Trip"),
files: [file(1, 10, "a.jpg")],
},
{
collection: collection(11, "trip"),
files: [file(2, 11, "b.jpg")],
},
{
collection: collection(12, "Work"),
files: [file(3, 12, "c.jpg")],
},
]);
const json = (name: string): string =>
readFileSync(join(outDir, "collections", name), "utf-8");
await runBackup(lib, { downloadDirectory: outDir });
const before = tree(outDir);
const skippedJSON = [json("trip (11).json"), json("Work.json")];
const scoped = await runBackup(lib, {
downloadDirectory: outDir,
onlyAlbumNames: ["Trip"],
});
expect(scoped.failed).toBe(0);
expect(before).toEqual([
"Trip (10)/",
"Trip (10)/a.jpg -> ../../originals/1.jpg",
"Trip (10).json",
"Work/",
"Work/c.jpg -> ../../originals/3.jpg",
"Work.json",
"trip (11)/",
"trip (11)/b.jpg -> ../../originals/2.jpg",
"trip (11).json",
]);
expect(tree(outDir)).toEqual(before);
expect([json("trip (11).json"), json("Work.json")]).toEqual(
skippedJSON,
);
});
it("removes links and album folders that are gone, and nothing the user added", async () => {
const outDir = join(root, "backup");
const albums: Album[] = [
{
collection: collection(10, "Trip"),
files: [
file(1, 10, "IMG_0001.JPG"),
file(2, 10, "IMG_0001.JPG"),
file(3, 10, "other.jpg"),
],
},
{
collection: collection(12, "Work"),
files: [file(5, 12, "a.jpg")],
},
{
collection: collection(13, "Old"),
files: [file(5, 13, "a.jpg")],
},
];
const lib = libraryOf(albums);
await runBackup(lib, { downloadDirectory: outDir });
// What the user put in the tree: a note and a symlink of their own in
// an album, a note in an album about to be renamed, and a folder quak
// did not create.
const collectionsDir = join(outDir, "collections");
writeFileSync(join(collectionsDir, "Trip", "notes.txt"), "mine");
symlinkSync("../elsewhere", join(collectionsDir, "Trip", "mine"));
writeFileSync(join(collectionsDir, "Work", "keep.txt"), "mine");
mkdirSync(join(collectionsDir, "Mine"));
writeFileSync(join(collectionsDir, "Mine", "keep.txt"), "mine");
// File 2 leaves Trip, Work is renamed Office, Old is deleted.
albums[0]!.files.splice(1, 1);
albums[1]!.collection = collection(12, "Office");
albums.splice(2, 1);
const result = await runBackup(lib, { downloadDirectory: outDir });
expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([
"Mine/",
"Mine/keep.txt",
"Office/",
"Office/a.jpg -> ../../originals/5.jpg",
"Office.json",
"Trip/",
"Trip/IMG_0001.JPG -> ../../originals/1.JPG",
"Trip/mine -> ../elsewhere",
"Trip/notes.txt",
"Trip/other.jpg -> ../../originals/3.jpg",
"Trip.json",
"Work/",
"Work/keep.txt",
"Work.json",
]);
});
// One album backed up, then a folder the user made beside it holding a
// symlink into originals/, with `json` (if given) as its sibling JSON.
const backupWithUserFolder = async (
json: string | undefined,
): Promise<{ outDir: string; failed: number }> => {
const outDir = join(root, "backup");
const lib = libraryOf([
{
collection: collection(10, "Trip"),
files: [file(1, 10, "a.jpg")],
},
]);
await runBackup(lib, { downloadDirectory: outDir });
const collectionsDir = join(outDir, "collections");
mkdirSync(join(collectionsDir, "Mine"));
symlinkSync(
"../../originals/1.jpg",
join(collectionsDir, "Mine", "a.jpg"),
);
if (json !== undefined) {
writeFileSync(join(collectionsDir, "Mine.json"), json);
}
const result = await runBackup(lib, { downloadDirectory: outDir });
return { outDir, failed: result.failed };
};
it("leaves a user folder with no JSON beside it as it was", async () => {
const { outDir, failed } = await backupWithUserFolder(undefined);
expect(failed).toBe(0);
expect(tree(outDir)).toEqual([
"Mine/",
"Mine/a.jpg -> ../../originals/1.jpg",
"Trip/",
"Trip/a.jpg -> ../../originals/1.jpg",
"Trip.json",
]);
});
it("leaves a user folder whose JSON has no album ID as it was", async () => {
const json = '{"name":"Mine"}';
const { outDir, failed } = await backupWithUserFolder(json);
expect(failed).toBe(0);
expect(tree(outDir)).toEqual([
"Mine/",
"Mine/a.jpg -> ../../originals/1.jpg",
"Mine.json",
"Trip/",
"Trip/a.jpg -> ../../originals/1.jpg",
"Trip.json",
]);
expect(
readFileSync(join(outDir, "collections", "Mine.json"), "utf-8"),
).toBe(json);
});
});
// A live photo, which Ente stores as one ZIP, is backed up as its image and
// its video, which a photo viewer can open, beside a JSON file naming them,
// and its album folder links both. These tests download a live photo ZIP
// through the real download layer (test/live-photo.ts).
describe("backup of live photos", () => {
// An account of one album, Trip (10), holding `files`.
class TripClient extends MockClient {
constructor(private readonly files: EnteFile[]) {
super();
}
override async collectionsSince(): Promise<CollectionsPage> {
return {
collections: [collection(10, "Trip")],
deleted: [],
cursor: 1,
};
}
override async filesSince(): Promise<FilesPage> {
return { files: this.files, deleted: [], cursor: 1 };
}
}
const open = (files: EnteFile[], bodies: Map<number, Uint8Array>) =>
openLibrary(cdnSource(bodies), new TripClient(files));
// What an earlier version stored for live photo 500: the ZIP under the
// image's name, and its link.
const earlierZIP = (outDir: string): void => {
mkdirSync(join(outDir, "originals"), { recursive: true });
mkdirSync(join(outDir, "collections", "Trip"), { recursive: true });
writeFileSync(join(outDir, "originals", "500.HEIC"), livePhotoZip());
symlinkSync(
"../../originals/500.HEIC",
join(outDir, "collections", "Trip", "IMG_0500.HEIC"),
);
};
const stored = ["500.heic", "500.json", "500.livephoto.json", "500.mov"];
const linked = [
"Trip/",
"Trip/IMG_0500.heic -> ../../originals/500.heic",
"Trip/IMG_0500.mov -> ../../originals/500.mov",
"Trip.json",
];
it("stores a live photo as its image and its video and links both", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(originals).sort()).toEqual(stored);
expect(readFileSync(join(originals, "500.heic"))).toEqual(
Buffer.from(IMAGE),
);
expect(readFileSync(join(originals, "500.mov"))).toEqual(
Buffer.from(VIDEO),
);
expect(tree(outDir)).toEqual(linked);
// Both parts are there, so the next run fetches nothing.
const second = await lib.backup({ downloadDirectory: outDir });
expect(second).toMatchObject({ downloaded: 0, skipped: 1, failed: 0 });
await lib.close();
});
it("gives both links of each live photo their own names when titles clash", async () => {
const a = await asLivePhoto(file(500, 10, "IMG_0001.HEIC"));
const b = await asLivePhoto(file(501, 10, "IMG_0001.HEIC"));
const lib = await open(
[a.file, b.file],
new Map([
[500, a.body],
[501, b.body],
]),
);
const outDir = join(root, "backup");
await lib.backup({ downloadDirectory: outDir });
expect(tree(outDir)).toEqual([
"Trip/",
"Trip/IMG_0001 (500).heic -> ../../originals/500.heic",
"Trip/IMG_0001 (500).mov -> ../../originals/500.mov",
"Trip/IMG_0001 (501).heic -> ../../originals/501.heic",
"Trip/IMG_0001 (501).mov -> ../../originals/501.mov",
"Trip.json",
]);
await lib.close();
});
it("replaces the ZIP an earlier version stored, and its link", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const outDir = join(root, "backup");
earlierZIP(outDir);
const lib = await open([live], new Map([[500, body]]));
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(tree(outDir)).toEqual(linked);
await lib.close();
});
it("stores nothing for a live photo that fails its hash, and keeps what was there", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
livePhotoZip(),
"not:the recorded hash",
);
const outDir = join(root, "backup");
earlierZIP(outDir);
const lib = await open([live], new Map([[500, body]]));
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 0, failed: 1 });
expect(result.errors.map((e) => e.fileID)).toEqual([500]);
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
expect(readdirSync(join(outDir, "originals"))).toEqual(["500.HEIC"]);
expect(tree(outDir)).toEqual([
"Trip/",
"Trip/IMG_0500.HEIC -> ../../originals/500.HEIC",
"Trip.json",
]);
await lib.close();
});
it("copies both parts of a live photo the cache already holds", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const lib = await open([live], new Map([[500, body]]));
const cached = await lib.photos.byID({ fileID: 500 })!.original();
const outDir = join(root, "backup");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(readFileSync(join(outDir, "originals", "500.mov"))).toEqual(
Buffer.from(VIDEO),
);
expect(tree(outDir)).toEqual(linked);
// The cache keeps its own copy.
expect(existsSync(cached.videoPath!)).toBe(true);
await lib.close();
});
it("serves a live photo the backup stored to a library reading the backup", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup");
await lib.backup({ downloadDirectory: outDir });
await lib.close();
// Another cache over the same backup, whose server has nothing.
const reader = await Library.open({
client: new TripClient([live]),
cacheDirectory: join(root, "other-cache"),
downloadDirectory: outDir,
contentSource: cdnSource(new Map()),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const read = await reader.photos.byID({ fileID: 500 })!.original();
expect(read).toEqual({
path: join(outDir, "originals", "500.heic"),
videoPath: join(outDir, "originals", "500.mov"),
bytes: IMAGE.length,
});
await reader.close();
});
});
test/cli/commands.test.ts
+24 -483
View File
@@ -12,9 +12,7 @@
import {
existsSync,
mkdirSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
statSync,
@@ -22,22 +20,11 @@ import {
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { PassThrough } from "node:stream";
import * as jpegJs from "jpeg-js";
import {
describe,
it,
expect,
vi,
beforeAll,
beforeEach,
afterEach,
} from "vitest";
import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
import {
type CliContext,
saveSession,
loginCommand,
whoamiCommand,
logoutCommand,
collectionsCommand,
@@ -45,25 +32,13 @@ import {
getCommand,
getThumbCommand,
backupCommand,
backupMetadataCommand,
listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../../src/cli-commands.js";
import { run } from "../../src/cli-run.js";
import { loadSession } from "../../src/cli-session.js";
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
import type { Client, ClientSnapshot } from "../../src/client.js";
import type { ContentSource } from "../../src/library/content.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { init, toBase64 } from "../../src/crypto/index.js";
import { defaultCacheDirectory } from "../../src/library/index.js";
import {
asLivePhoto,
cdnSource,
IMAGE,
livePhotoHash,
livePhotoZip,
VIDEO,
} from "../live-photo.js";
import { init } from "../../src/crypto/index.js";
const USER_ID = 42;
@@ -106,26 +81,8 @@ const FILES: Record<number, EnteFile[]> = {
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
// original fail; `emptyThumbID` makes the server report that file's
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
// the collection's updationTime as the server does, and `refreshError` makes
// listing collections fail with that message.
const fakeClient = (
opts: {
failID?: number;
emptyThumbID?: number;
withNewFile?: boolean;
refreshError?: string;
} = {},
) => {
const collections = opts.withNewFile
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
: COLLECTIONS;
const files = opts.withNewFile
? {
...FILES,
1: [...FILES[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
}
: FILES;
// thumbnail as empty.
const fakeClient = (opts: { failID?: number; emptyThumbID?: number } = {}) => {
const source: ContentSource = {
original: async ({ file: f, destination }) => {
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
@@ -139,19 +96,18 @@ const fakeClient = (
};
const fake = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
collectionsSince: async () => {
if (opts.refreshError) throw new Error(opts.refreshError);
return { collections, deleted: [], cursor: 1 };
},
collectionsSince: async () => ({
collections: COLLECTIONS,
deleted: [],
cursor: 1,
}),
filesSince: async (args: { collectionID: number }) => ({
files: files[args.collectionID] ?? [],
files: FILES[args.collectionID] ?? [],
deleted: [],
cursor: 1,
}),
contentSource: () => source,
getApiClient: () => ({
// The ML data request of `backup-metadata`: no file has any.
postJSON: async () => ({ data: [] }),
getThumbnailStream: async (fileID: number) =>
new ReadableStream<Uint8Array>({
start(controller) {
@@ -185,15 +141,6 @@ const context = (client: Client | null = fakeClient()): CliContext => ({
sessionDir: join(root, "session"),
cacheDir: join(root, "cache"),
loadSession: () => client,
login: async () => {
throw new Error("login not expected");
},
prompt: async () => {
throw new Error("prompt not expected");
},
promptSecret: async () => {
throw new Error("prompt not expected");
},
});
beforeAll(async () => {
@@ -229,6 +176,19 @@ describe("session file", () => {
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
});
it("is removed by logout", async () => {
const ctx = context();
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe("Session deleted.\n");
});
it("logout without a session says so and exits 0", async () => {
expect(await logoutCommand(context())).toBe(0);
expect(stderr.text).toBe("No session found.\n");
});
it("a missing session exits 1 with 'Not logged in'", async () => {
const ctx = { ...context(), loadSession };
expect(await whoamiCommand(ctx)).toBe(1);
@@ -251,184 +211,6 @@ describe("session file", () => {
});
});
// The login function is a fake that hands back a client whose snapshot is
// `snapshot`; each prompt is recorded and answered with "123456".
describe("login", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "token",
masterKey: "a",
secretKey: "b",
publicKey: "c",
};
const loggedIn = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
toJSON: () => snapshot,
} as unknown as Client;
let prompts: string[];
const loginContext = (
login: (opts: LoginOptions) => Promise<Client>,
): CliContext => ({
...context(),
login,
prompt: async (message) => {
prompts.push(message);
return "123456";
},
promptSecret: async (message) => {
prompts.push(message);
return "123456";
},
});
beforeEach(() => {
prompts = [];
vi.stubEnv("QUAK_EMAIL", "cli@example.com");
vi.stubEnv("QUAK_PASSWORD", "hunter2");
});
afterEach(() => {
vi.unstubAllEnvs();
});
it("takes the email and password from the environment without a prompt", async () => {
const calls: LoginOptions[] = [];
const ctx = loginContext(async (opts) => {
calls.push(opts);
return loggedIn;
});
expect(await loginCommand(ctx)).toBe(0);
expect(prompts).toEqual([]);
expect(calls).toHaveLength(1);
expect(calls[0]!.email).toBe("cli@example.com");
expect(calls[0]!.password).toBe("hunter2");
const path = join(ctx.sessionDir, "session.json");
expect(stderr.text).toBe(
"Authenticating...\n" +
`Logged in as cli@example.com (user ${USER_ID})\n` +
`Session saved to ${path}\n`,
);
});
it("saves the session with mode 0600 in a directory with mode 0700", async () => {
const ctx = loginContext(async () => loggedIn);
expect(await loginCommand(ctx)).toBe(0);
expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
const path = join(ctx.sessionDir, "session.json");
expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
});
it("asks for the TOTP code when the account needs one", async () => {
let code: string | undefined;
const ctx = loginContext(async (opts) => {
code = await opts.totp!();
return loggedIn;
});
expect(await loginCommand(ctx)).toBe(0);
expect(prompts).toEqual(["TOTP code: "]);
expect(code).toBe("123456");
});
it("a failed login exits 1, says why and writes no session", async () => {
const ctx = loginContext(async () => {
throw new Error("HTTP 401 from server");
});
expect(await loginCommand(ctx)).toBe(1);
expect(stderr.text).toBe(
"Authenticating...\nLogin failed: HTTP 401 from server\n",
);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
});
});
// These use a real client read from the session file, over a fake API that
// records each request and answers with `status`.
describe("logout", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "saved-token",
masterKey: toBase64(new Uint8Array(32)),
secretKey: toBase64(new Uint8Array(32)),
publicKey: toBase64(new Uint8Array(32)),
};
const requests: Request[] = [];
const logoutContext = (status: number): CliContext => ({
...context(),
loadSession: (path) =>
loadSession(path, {
fetch: async (url, init) => {
requests.push(new Request(url, init));
return new Response(JSON.stringify({}), {
status,
headers: { "content-type": "application/json" },
});
},
}),
});
beforeEach(() => {
requests.length = 0;
});
it("ends the session on the server, then deletes the file", async () => {
const ctx = logoutContext(200);
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(requests).toHaveLength(1);
expect(requests[0]!.method).toBe("POST");
expect(new URL(requests[0]!.url).pathname).toBe("/users/logout");
expect(requests[0]!.headers.get("X-Auth-Token")).toBe("saved-token");
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe(
"Session ended on the server.\n" +
"Session deleted.\n" +
`Cache directory ${ctx.cacheDir} still holds decrypted data; delete it to remove that data.\n`,
);
});
it("still deletes the file when the server call fails, and says so", async () => {
const ctx = logoutContext(500);
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(1);
expect(requests).toHaveLength(1);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe(
"Could not end the session on the server: HTTP 500\n" +
"Session deleted.\n" +
`Cache directory ${ctx.cacheDir} still holds decrypted data; delete it to remove that data.\n`,
);
});
it("names the account's default cache directory without --cache-dir", async () => {
const ctx = { ...logoutContext(200), cacheDir: undefined };
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(stderr.text).toContain(
`Cache directory ${defaultCacheDirectory(USER_ID)} still holds decrypted data`,
);
});
it("without a session says so, calls nothing and exits 0", async () => {
expect(await logoutCommand(logoutContext(200))).toBe(0);
expect(requests).toHaveLength(0);
expect(stderr.text).toBe("No session found.\n");
});
});
describe("whoami", () => {
it("prints the account as one line of JSON", async () => {
expect(await whoamiCommand(context())).toBe(0);
@@ -541,120 +323,6 @@ describe("get and get-thumb", () => {
});
});
// A live photo, which Ente stores as one ZIP, is written as its image and its
// video, which a photo viewer can open. The account here is Vacation holding
// one live photo, 300, downloaded through the real download layer
// (test/live-photo.ts).
describe("a live photo", () => {
const livePhotoClient = async (image = IMAGE): Promise<Client> => {
const { file: live, body } = await asLivePhoto(
file(300, 1, "IMG_0300.HEIC"),
livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
livePhotoHash(image, VIDEO),
);
const fake = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
collectionsSince: async () => ({
collections: [collection(1, "Vacation")],
deleted: [],
cursor: 1,
}),
filesSince: async () => ({ files: [live], deleted: [], cursor: 1 }),
contentSource: () => cdnSource(new Map([[300, body]])),
// The ML data request of `backup-metadata`: no file has any.
getApiClient: () => ({ postJSON: async () => ({ data: [] }) }),
};
return fake as unknown as Client;
};
it("get writes its image and video, named after the title with their own extensions", async () => {
const ctx = context(await livePhotoClient());
const dir = join(root, "cwd");
mkdirSync(dir);
const previous = process.cwd();
process.chdir(dir);
try {
expect(await getCommand(ctx, "300", {})).toBe(0);
} finally {
process.chdir(previous);
}
expect(readdirSync(dir).sort()).toEqual([
"IMG_0300.heic",
"IMG_0300.mov",
]);
expect(readFileSync(join(dir, "IMG_0300.heic"))).toEqual(
Buffer.from(IMAGE),
);
expect(readFileSync(join(dir, "IMG_0300.mov"))).toEqual(
Buffer.from(VIDEO),
);
expect(stderr.text).toBe(
`${IMAGE.length} bytes -> IMG_0300.heic\n` +
`${VIDEO.length} bytes -> IMG_0300.mov\n`,
);
});
it("get --out writes the image there and the video beside it", async () => {
const out = join(root, "photo.jpg");
const video = join(root, "photo.mov");
expect(
await getCommand(context(await livePhotoClient()), "300", { out }),
).toBe(0);
expect(readFileSync(out)).toEqual(Buffer.from(IMAGE));
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
expect(stderr.text).toBe(
`${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
);
});
it("get exits 1 and writes nothing when --out has the video's extension", async () => {
const out = join(root, "photo.MOV");
expect(
await getCommand(context(await livePhotoClient()), "300", { out }),
).toBe(1);
expect(existsSync(out)).toBe(false);
expect(existsSync(join(root, "photo.mov"))).toBe(false);
expect(stderr.text).toBe(
`File 300 is a live photo, and its video would also be written to ${out}\n`,
);
});
it("backup-metadata --exif reads its image", async () => {
// A 4x4 JPEG, whose size can only come from reading the image; the
// ZIP and the video are not JPEGs.
const jpeg = jpegJs.encode(
{ data: new Uint8Array(4 * 4 * 4), width: 4, height: 4 },
50,
).data;
const dir = join(root, "dump");
expect(
await backupMetadataCommand(
context(await livePhotoClient(new Uint8Array(jpeg))),
dir,
{ exif: true },
),
).toBe(0);
const record = JSON.parse(
readFileSync(
join(dir, "collections", "1-Vacation", "300.json"),
"utf-8",
),
);
expect(record.imageMetadata).toMatchObject({
format: "jpeg",
width: 4,
height: 4,
});
});
});
describe("backup", () => {
it("exits 0 and prints a summary when every file is saved", async () => {
const dir = join(root, "backup");
@@ -669,16 +337,6 @@ describe("backup", () => {
expect(stdout.text).toBe("");
});
// The backup opens its library with the precache off: it fetches the
// originals it needs into the backup, and must not also fetch every
// thumbnail in the account, or keep originals, in the per-user cache.
it("leaves nothing in the cache's originals and thumbnails", async () => {
const dir = join(root, "backup");
expect(await backupCommand(context(), dir, {})).toBe(0);
expect(readdirSync(join(root, "cache", "thumbnails"))).toEqual([]);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
});
it("exits 1 and lists the file when one download fails", async () => {
const ctx = context(fakeClient({ failID: 101 }));
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
@@ -705,34 +363,6 @@ describe("backup", () => {
expect(result.errors[0].fileID).toBe(101);
expect(stderr.text).toBe("Starting backup...\n");
});
it("exits 1 with the error on one line when the refresh fails", async () => {
const client = {
...fakeClient(),
collectionsSince: async () => {
throw new Error("HTTP 401 from server");
},
} as unknown as Client;
const dir = join(root, "backup");
// Through `run`, as `bin/quak.ts` does, which prints a thrown error.
const runStderr = new PassThrough();
let runText = "";
runStderr.on("data", (chunk: Buffer) => {
runText += chunk.toString();
});
const code = await new Promise<number>((resolve) => {
void run(
backupCommand(context(client), dir, {}),
new PassThrough(),
runStderr,
resolve,
);
});
expect(code).toBe(1);
expect(runText).toBe("quak: HTTP 401 from server\n");
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
expect(existsSync(join(dir, "originals"))).toBe(false);
});
});
describe("helper list-missing-thumbnails", () => {
@@ -765,92 +395,3 @@ describe("helper list-missing-thumbnails", () => {
expect(stderr.text).toBe("");
});
});
describe("backup-metadata --exif", () => {
// Runs the command and returns what it printed to stderr.
const backupMetadata = async (opts: { exif?: boolean; all?: boolean }) => {
expect(
await backupMetadataCommand(context(), join(root, "dump"), opts),
).toBe(0);
return stderr.text;
};
it("--exif extracts EXIF", async () => {
expect(await backupMetadata({ exif: true })).toContain(
"[beach.jpg] Extracting EXIF...\n",
);
});
it("--all extracts EXIF", async () => {
expect(await backupMetadata({ all: true })).toContain(
"[beach.jpg] Extracting EXIF...\n",
);
});
it("without either flag extracts no EXIF", async () => {
expect(await backupMetadata({})).not.toContain("Extracting EXIF");
});
});
// Each test first runs `collections` so the cache holds the account as it was,
// then changes the server under it.
describe("backup-metadata and the thumbnail helpers refresh first", () => {
beforeEach(async () => {
expect(await collectionsCommand(context(), {})).toBe(0);
stdout.text = "";
stderr.text = "";
});
it("backup-metadata writes a file added since the cache was written", async () => {
const ctx = context(fakeClient({ withNewFile: true }));
const dir = join(root, "dump");
expect(await backupMetadataCommand(ctx, dir, {})).toBe(0);
expect(
existsSync(join(dir, "collections", "1-Vacation", "102.json")),
).toBe(true);
});
it("list-missing-thumbnails checks a file added since the cache was written", async () => {
const ctx = context(
fakeClient({ withNewFile: true, emptyThumbID: 102 }),
);
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(0);
expect(stdout.text).toBe(
"102\tnew.jpg\tVacation\tempty thumbnail (0 bytes)\n",
);
});
it("fix-missing-thumbnails finds a file added since the cache was written", async () => {
const ctx = context(fakeClient({ withNewFile: true }));
expect(
await fixMissingThumbnailsCommand(ctx, {
file: ["102"],
json: true,
}),
).toBe(0);
// Found, then skipped because the server records no thumbnail size
// for it; a file missing from the cache would fail as not found.
expect(JSON.parse(stdout.text)).toMatchObject([
{ fileID: 102, title: "new.jpg", status: "skipped" },
]);
});
// `run` in `cli-run.ts` prints a thrown error as one line and exits 1.
it("all three throw when the refresh fails", async () => {
const ctx = context(
fakeClient({ refreshError: "HTTP 503 from server" }),
);
const dir = join(root, "dump");
await expect(backupMetadataCommand(ctx, dir, {})).rejects.toThrow(
"HTTP 503 from server",
);
expect(existsSync(dir)).toBe(false);
await expect(listMissingThumbnailsCommand(ctx, {})).rejects.toThrow(
"HTTP 503 from server",
);
await expect(
fixMissingThumbnailsCommand(ctx, { file: ["100"] }),
).rejects.toThrow("HTTP 503 from server");
expect(stdout.text).toBe("");
});
});
test/cli/metadata-backup.test.ts
+2 -73
View File
@@ -38,7 +38,7 @@ import { join } from "node:path";
import { tmpdir } from "node:os";
import sodium from "libsodium-wrappers-sumo";
import { SRP, SrpServer } from "fast-srp-hap";
import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
import { beforeAll, afterAll, describe, expect, it } from "vitest";
import {
init,
toBase64,
@@ -53,16 +53,8 @@ import {
runMetadataBackup,
type MetadataBackupOptions,
} from "../../src/metadata-backup.js";
import { backupMetadataCommand } from "../../src/cli-commands.js";
import type { KeyAttributes } from "../../src/auth/types.js";
// One file per ML data request, so the two files of the mock account are
// fetched in two requests and one of them can fail on its own.
vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()),
MLDATA_BATCH_SIZE: 1,
}));
const TEST_EMAIL = "metabackup@example.com";
const TEST_PASSWORD = "metapass";
const TEST_OPS = 2;
@@ -355,8 +347,7 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
};
};
// `failMLDataFor`: answer 500 to every ML data request that asks for this file.
const buildMetaFetch = (m: MetaMockState, failMLDataFor?: number) => {
const buildMetaFetch = (m: MetaMockState) => {
let srpServer: SrpServer;
return (async (
input: RequestInfo | URL,
@@ -412,8 +403,6 @@ const buildMetaFetch = (m: MetaMockState, failMLDataFor?: number) => {
}
if (path === "/files/data/fetch") {
const body = JSON.parse(init?.body as string);
if ((body.fileIDs as number[]).includes(failMLDataFor!))
return new Response("server error", { status: 500 });
const data = (body.fileIDs as number[])
.filter((id: number) => m.encryptedMLData[id])
.map((id: number) => ({
@@ -647,63 +636,3 @@ describe("quak backup-metadata", () => {
expect(failedMeta.imageMetadataError).toEqual(expect.any(String));
});
});
describe("quak backup-metadata when an ML data request fails", () => {
// Run the CLI command against the mock and return its exit code, stderr
// and output directory.
const runCommand = async (failMLDataFor?: number) => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: {
fetch: buildMetaFetch(mock, failMLDataFor),
retry: { sleep: async () => {} },
},
});
const outDir = mkdtempSync(join(testDir, "ml-fail-"));
let stderr = "";
const code = await backupMetadataCommand(
{
stdout: { write: () => true },
stderr: { write: (text: string) => (stderr += text) },
sessionDir: testDir,
cacheDir: mkdtempSync(join(testDir, "cache-")),
loadSession: () => client,
},
outDir,
{},
);
return { code, stderr, outDir };
};
it("writes every file, marks the failed batch's files, and exits 1", async () => {
const { code, stderr, outDir } = await runCommand(200);
expect(code).toBe(1);
expect(stderr).toContain("ML data request for 1 file(s) failed");
const ok = JSON.parse(
readFileSync(
join(outDir, "collections", "10-Vacation", "100.json"),
"utf-8",
),
);
expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
expect(ok.mlDataError).toBeUndefined();
const failed = JSON.parse(
readFileSync(
join(outDir, "collections", "20-__Work", "200.json"),
"utf-8",
),
);
expect(failed.metadata.title).toBe("diagram.png");
expect(failed.mlData).toBeUndefined();
expect(failed.mlDataError).toContain("500");
});
it("exits 0 when every ML data request succeeds", async () => {
const { code } = await runCommand();
expect(code).toBe(0);
});
});
test/cli/run.test.ts
-58
View File
@@ -1,58 +0,0 @@
/**
* Tests for `run` in `src/cli-run.ts`, which every CLI command goes through.
*/
import { PassThrough } from "node:stream";
import { describe, it, expect } from "vitest";
import { run } from "../../src/cli-run.js";
// A stream whose written text is kept in `text`; writes finish at once, so
// nothing is left waiting to drain.
const collector = (): { stream: PassThrough; text: () => string } => {
const stream = new PassThrough();
const chunks: string[] = [];
stream.on("data", (chunk: Buffer) => chunks.push(chunk.toString()));
return { stream, text: () => chunks.join("") };
};
const runToExit = async (
command: Promise<number>,
): Promise<{ code: number; stdout: string; stderr: string }> => {
const stdout = collector();
const stderr = collector();
const code = await new Promise<number>((resolve) => {
void run(command, stdout.stream, stderr.stream, resolve);
});
return { code, stdout: stdout.text(), stderr: stderr.text() };
};
describe("run", () => {
it("exits with the code the command returns", async () => {
const result = await runToExit(Promise.resolve(3));
expect(result).toEqual({ code: 3, stdout: "", stderr: "" });
});
it("prints a thrown error as one line without a stack trace and exits 1", async () => {
const failing = async (): Promise<number> => {
throw new Error(
"ENOTDIR: not a directory, mkdir '/dev/null/x/originals'",
);
};
const result = await runToExit(failing());
expect(result.code).toBe(1);
expect(result.stdout).toBe("");
expect(result.stderr).toBe(
"quak: ENOTDIR: not a directory, mkdir '/dev/null/x/originals'\n",
);
});
it("prints a thrown value that is not an Error", async () => {
const result = await runToExit(Promise.reject("offline"));
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: offline\n",
});
});
});
test/crypto/hash.test.ts
-33
View File
@@ -1,33 +0,0 @@
import { beforeAll, describe, expect, it } from "vitest";
import {
chunkHashFinal,
chunkHashInit,
chunkHashUpdate,
init,
} from "../../src/crypto/index.js";
beforeAll(async () => {
await init();
});
describe("content hash", () => {
// RFC 7693 Appendix A: BLAKE2b-512 of "abc".
const abc = Buffer.from(
"ba80a53f981c4d0d6a2797b69f12f6e94c212f14685ac4b74b12bb6fdbffa2d1" +
"7d87c5392aab792dc252d5de4533cc9518d38aa8dbf1925ab92386edd4009923",
"hex",
).toString("base64");
it("is unkeyed BLAKE2b-512 in standard base64", () => {
const state = chunkHashInit();
chunkHashUpdate(state, new TextEncoder().encode("abc"));
expect(chunkHashFinal(state)).toBe(abc);
});
it("gives the same hash when the input arrives in chunks", () => {
const state = chunkHashInit();
chunkHashUpdate(state, new TextEncoder().encode("a"));
chunkHashUpdate(state, new TextEncoder().encode("bc"));
expect(chunkHashFinal(state)).toBe(abc);
});
});
test/crypto/kdf.test.ts
+4 -4
View File
@@ -29,10 +29,10 @@ describe("crypto.deriveKEK (Argon2id)", () => {
});
/**
* Cheap parameters used so the test suite stays under the 90-second
* `timeout` in the `test` phase of the `Dockerfile`. The real production
* parameters Ente uses are larger (memLimit up to 1 GiB, opsLimit 3-16).
* The algorithm is the same regardless of parameters.
* Cheap parameters used so the test suite stays under the 30-second
* budget. The real production parameters Ente uses are larger
* (memLimit up to 1 GiB, opsLimit 3-16). The algorithm is the same
* regardless of parameters.
*/
const TEST_OPS = 2;
const TEST_MEM = 64 * 1024 * 1024; // 64 MiB
test/download/download.test.ts
+28 -311
View File
@@ -55,14 +55,12 @@ import {
readFileSync,
rmSync,
mkdtempSync,
statSync,
writeFileSync,
} from "node:fs";
import { dirname, join } from "node:path";
import { tmpdir } from "node:os";
import { createHash } from "node:crypto";
import sodium from "libsodium-wrappers-sumo";
import { zipSync } from "fflate";
import {
beforeAll,
beforeEach,
@@ -82,7 +80,6 @@ import {
writeAtomic,
} from "../../src/download/index.js";
import type { EnteFile, FileMetadata } from "../../src/model/types.js";
import { IMAGE, livePhotoHash, livePhotoZip, VIDEO } from "../live-photo.js";
// ---------------------------------------------------------------------------
// Test helpers
@@ -108,8 +105,6 @@ import { IMAGE, livePhotoHash, livePhotoZip, VIDEO } from "../live-photo.js";
const renameHook = vi.hoisted(() => ({
calls: [] as { from: string; to: string; sourceExisted: boolean }[],
failWith: null as Error | null,
// When set, only a rename to this path fails.
failTo: null as string | null,
}));
/**
@@ -185,10 +180,7 @@ vi.mock("node:fs/promises", async (importOriginal) => {
sourceExisted: sourceExists(from),
});
durabilityHook.events.push(`rename:${to}`);
if (
renameHook.failWith !== null &&
(renameHook.failTo === null || renameHook.failTo === to)
) {
if (renameHook.failWith !== null) {
throw renameHook.failWith;
}
await actual.rename(from, to);
@@ -196,34 +188,9 @@ vi.mock("node:fs/promises", async (importOriginal) => {
};
});
/**
* `chunkHashUpdate` is wrapped to record the length of every piece hashed, so
* a test can show that a live photo entry reaches the hash in pieces far
* smaller than the entry, rather than decompressed whole first.
*/
const hashHook = vi.hoisted(() => ({
lengths: [] as number[],
}));
vi.mock("../../src/crypto/index.js", async (importOriginal) => {
const actual =
await importOriginal<typeof import("../../src/crypto/index.js")>();
return {
...actual,
chunkHashUpdate: (
...args: Parameters<typeof actual.chunkHashUpdate>
): void => {
hashHook.lengths.push(args[1].length);
actual.chunkHashUpdate(...args);
},
};
});
beforeEach(() => {
hashHook.lengths.length = 0;
renameHook.calls.length = 0;
renameHook.failWith = null;
renameHook.failTo = null;
durabilityHook.events.length = 0;
writeHook.writes.length = 0;
});
@@ -256,8 +223,7 @@ afterAll(() => {
* `sodium.randombytes_buf` goes through the wasm wrapper a byte at a time and
* costs roughly 20 seconds for the 4 MiB chunk below — about two hundred
* times what it costs to encrypt the same buffer, and on its own enough to
* push `make test` past the 90-second `timeout` in the `test` phase of the
* `Dockerfile`. This loop fills
* push `make test` past the 30-second cap in `script/test`. This loop fills
* 4 MiB in a few milliseconds.
*/
const patternBytes = (length: number, seed: number): Uint8Array => {
@@ -1043,28 +1009,32 @@ describe.each(entryPoints)(
expect(readdirSync(dir)).toEqual([]);
});
// Root ignores directory permissions, so this fails when run as root.
// The `test` phase of the `Dockerfile` runs as the `node` user.
it("fails without creating anything when the destination directory is not writable", async () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 34),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "never.bin");
chmodSync(dir, 0o500);
try {
await expect(
download(api, file, outPath),
).rejects.toMatchObject({ code: "EACCES" });
} finally {
chmodSync(dir, 0o700);
}
// Root ignores directory permissions, so this cannot fail as root
// (the Docker test image runs as root).
it.skipIf(process.getuid?.() === 0)(
"fails without creating anything when the destination directory is not writable",
async () => {
const key =
sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 34),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "never.bin");
chmodSync(dir, 0o500);
try {
await expect(
download(api, file, outPath),
).rejects.toMatchObject({ code: "EACCES" });
} finally {
chmodSync(dir, 0o700);
}
expect(readdirSync(dir)).toEqual([]);
});
expect(readdirSync(dir)).toEqual([]);
},
);
},
);
@@ -1568,13 +1538,7 @@ describe("writeAtomic", () => {
// directory so that new entry is on disk too. Do the directory fsync
// before the rename, or skip it, and a crash can lose the rename.
expect(durabilityHook.events).toHaveLength(3);
// The temp name carries this process's ID, so a library opening the
// same cache can tell a write in progress from a leftover.
const tempSync = durabilityHook.events[0]!;
expect(tempSync.startsWith(`sync:w:${dir}/`)).toBe(true);
expect(tempSync.slice(`sync:w:${dir}/`.length)).toMatch(
new RegExp(`^\\.quak-${process.pid}-[0-9a-f]{32}\\.tmp$`),
);
expect(durabilityHook.events[0]).toMatch(/^sync:w:.*\.tmp$/);
expect(durabilityHook.events[1]).toBe(`rename:${dest}`);
expect(durabilityHook.events[2]).toBe(`sync:r:${dir}`);
});
@@ -1649,250 +1613,3 @@ describe.each(entryPoints)("$name progress", ({ name, download }) => {
expectSameBytes(readFileSync(outPath), plaintext);
});
});
// Node's own BLAKE2b-512 is the reference, so the tests below do not depend on
// the code under test to compute what they expect.
const blake2b = (bytes: Uint8Array): string =>
createHash("blake2b512").update(bytes).digest("base64");
// Serve `plaintext` encrypted as file 999 with the given metadata, to be
// written to `f.bin` in a fresh directory. Four responses are scripted so a
// retried failure would show in `requests`.
const setup = (plaintext: Uint8Array, metadata: Partial<FileMetadata>) => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const file = buildMockEnteFile(key, header, header);
file.metadata = { ...file.metadata, ...metadata };
const body = { kind: "body", bytes: ciphertext } as const;
const { fetch, requests } = scriptedCdnFetch(body, body, body, body);
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 4 } });
const dir = mkdtempSync(join(testDir, "hash-"));
const outPath = join(dir, "f.bin");
return {
run: () => downloadFile(api, file, outPath),
api,
file,
dir,
outPath,
requests,
};
};
describe("downloadFile content hash", () => {
it("stores a file whose hash matches", async () => {
const plaintext = patternBytes(700, 80);
const t = setup(plaintext, { hash: blake2b(plaintext) });
await t.run();
expectSameBytes(readFileSync(t.outPath), plaintext);
});
it("rejects a mismatch, stores nothing, names the file and does not retry", async () => {
const t = setup(patternBytes(700, 80), {
hash: blake2b(patternBytes(700, 79)),
});
await expect(t.run()).rejects.toThrow(
/file 999: content hash .* does not match/,
);
expect(readdirSync(t.dir)).toEqual([]);
expect(t.requests()).toBe(1);
});
it("stores a file with no recorded hash unchecked", async () => {
const plaintext = patternBytes(700, 80);
const t = setup(plaintext, { hash: undefined });
await t.run();
expectSameBytes(readFileSync(t.outPath), plaintext);
});
it("stores a live photo whose image and video hashes match", async () => {
const t = setup(livePhotoZip(), {
fileType: "livePhoto",
hash: livePhotoHash(),
});
await t.run();
expect(readFileSync(join(t.dir, "f.heic"))).toEqual(Buffer.from(IMAGE));
expect(readFileSync(join(t.dir, "f.mov"))).toEqual(Buffer.from(VIDEO));
});
it("writes a large live photo entry as it decompresses, never whole", async () => {
// 64 MiB of zeros deflates to a few kilobytes, the shape of a ZIP
// that would exhaust memory if expanded whole.
const image = new Uint8Array(64 * 1024 * 1024);
const video = patternBytes(900, 83);
const zip = zipSync({ "image.heic": image, "video.mov": video });
const t = setup(zip, {
fileType: "livePhoto",
hash: livePhotoHash(image, video),
});
await t.run();
expect(statSync(join(t.dir, "f.heic")).size).toBe(image.length);
expectSameBytes(readFileSync(join(t.dir, "f.mov")), video);
const hashed = hashHook.lengths.reduce((a, b) => a + b, 0);
expect(hashed).toBe(image.length + video.length);
expect(Math.max(...hashHook.lengths)).toBeLessThanOrEqual(
2 * STREAM_CHUNK_SIZE,
);
const written = writeHook.writes.filter((w) => w.path.endsWith(".tmp"));
expect(Math.max(...written.map((w) => w.length))).toBeLessThanOrEqual(
2 * STREAM_CHUNK_SIZE,
);
});
it("rejects a live photo whose hash does not match, keeping what was there", async () => {
// The whole ZIP's hash is not the recorded one: each part is hashed.
const zip = livePhotoZip();
const t = setup(zip, { fileType: "livePhoto", hash: blake2b(zip) });
writeFileSync(t.outPath, "an earlier download");
await expect(t.run()).rejects.toThrow(
/file 999: content hash .* does not match/,
);
expect(readdirSync(t.dir)).toEqual(["f.bin"]);
expect(readFileSync(t.outPath, "utf-8")).toBe("an earlier download");
});
it("rejects a live photo that is not a readable ZIP and does not retry", async () => {
// Bytes 8-9 of a ZIP entry's local header name its compression
// method; 99 is one no reader knows, so the entry cannot be read.
const zip = livePhotoZip();
zip[8] = 99;
zip[9] = 0;
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash() });
await expect(t.run()).rejects.toThrow(
/file 999: live photo is not a readable ZIP/,
);
expect(readdirSync(t.dir)).toEqual([]);
expect(t.requests()).toBe(1);
});
it("rejects a live photo ZIP with no image entry", async () => {
const zip = livePhotoZip({ "video.mov": VIDEO });
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash() });
await expect(t.run()).rejects.toThrow(
/file 999: live photo ZIP does not hold both an image and a video/,
);
expect(readdirSync(t.dir)).toEqual([]);
});
it("rejects a live photo ZIP with no video entry", async () => {
const zip = livePhotoZip({ "image.heic": IMAGE });
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash() });
await expect(t.run()).rejects.toThrow(
/file 999: live photo ZIP does not hold both an image and a video/,
);
expect(readdirSync(t.dir)).toEqual([]);
});
});
// ---------------------------------------------------------------------------
// Live photos
//
// A live photo arrives as a ZIP of its image and its video. It is written as
// those two files, which a photo viewer can open, each named after the
// destination with its own extension from the ZIP, the way Ente's clients name
// them when they save one.
// ---------------------------------------------------------------------------
describe("downloadFile live photos", () => {
const livePhoto = { fileType: "livePhoto", hash: livePhotoHash() } as const;
it("names the image and the video after the title when no outPath is given", async () => {
const t = setup(livePhotoZip(), {
...livePhoto,
title: "IMG_1234.HEIC",
});
const result = await inDirectory(t.dir, () =>
downloadFile(t.api, t.file),
);
// `bytesWritten` is the length of the decrypted ZIP.
expect(result).toEqual({
path: "IMG_1234.heic",
videoPath: "IMG_1234.mov",
bytesWritten: livePhotoZip().length,
});
expect(readdirSync(t.dir).sort()).toEqual([
"IMG_1234.heic",
"IMG_1234.mov",
]);
});
it("gives each part its own extension from the ZIP, letters and digits only", async () => {
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
const t = setup(zip, livePhoto);
const result = await downloadFile(t.api, t.file, join(t.dir, "f.HEIC"));
expect(result.path).toBe(join(t.dir, "f.JPG"));
expect(result.videoPath).toBe(join(t.dir, "f.bin"));
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
});
it("replaces what was at the destination, such as an earlier ZIP of the two", async () => {
const t = setup(livePhotoZip(), livePhoto);
writeFileSync(t.outPath, livePhotoZip());
await t.run();
expect(readdirSync(t.dir).sort()).toEqual(["f.heic", "f.mov"]);
});
it("renames the image and then the video into place, each from its own temp file", async () => {
const t = setup(livePhotoZip(), livePhoto);
await t.run();
expect(
renameHook.calls.map((c) => [
dirname(c.from),
c.to,
c.sourceExisted,
]),
).toEqual([
[t.dir, join(t.dir, "f.heic"), true],
[t.dir, join(t.dir, "f.mov"), true],
]);
});
it("stores neither part when the video cannot be renamed into place", async () => {
const t = setup(livePhotoZip(), livePhoto);
renameHook.failWith = new Error("simulated rename failure");
renameHook.failTo = join(t.dir, "f.mov");
await expect(t.run()).rejects.toThrow("simulated rename failure");
expect(readdirSync(t.dir)).toEqual([]);
});
it("refuses an image and a video with the same extension, storing nothing", async () => {
// On a file system that ignores case, the two would be one file.
const zip = livePhotoZip({ "image.mov": IMAGE, "video.MOV": VIDEO });
const t = setup(zip, livePhoto);
writeFileSync(t.outPath, "an earlier download");
await expect(t.run()).rejects.toThrow(
/file 999: live photo's image and video have the same extension/,
);
expect(readdirSync(t.dir)).toEqual(["f.bin"]);
expect(t.requests()).toBe(1);
});
});
test/library/content.test.ts
+2 -151
View File
@@ -31,11 +31,8 @@ import {
existsSync,
writeFileSync,
mkdirSync,
readdirSync,
readFileSync,
statSync,
} from "node:fs";
import { spawnSync } from "node:child_process";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -46,13 +43,6 @@ import {
} from "../../src/library/content.js";
import { RequestPools } from "../../src/library/pools.js";
import type { EnteFile } from "../../src/model/types.js";
import {
asLivePhoto,
cdnSource,
IMAGE,
livePhotoZip,
VIDEO,
} from "../live-photo.js";
const file = (id: number, title = `file-${id}.jpg`): EnteFile => ({
id,
@@ -171,28 +161,15 @@ describe("ContentCache.open", () => {
expect(statSync(thumbnails).mode & 0o777).toBe(0o700);
});
it("removes temp files of an exited process, keeping those of a running one and complete content", async () => {
it("reaps orphan temp files but keeps complete content", async () => {
const originals = join(cacheDir, "originals");
const thumbnails = join(cacheDir, "thumbnails");
mkdirSync(originals, { recursive: true });
mkdirSync(thumbnails, { recursive: true });
// A child that has already exited: its process ID is not running.
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const orphan = join(originals, `.quak-${exitedPID}-abc123.tmp`);
const orphanThumb = join(thumbnails, `.quak-${exitedPID}-abc456.tmp`);
// This test's own process stands in for another process still
// downloading into the same cache.
const inProgress = join(originals, `.quak-${process.pid}-def123.tmp`);
const inProgressThumb = join(
thumbnails,
`.quak-${process.pid}-def456.tmp`,
);
const orphan = join(originals, ".quak-abc123.tmp");
const complete = join(originals, "1.jpg");
const thumb = join(thumbnails, "2.jpg");
writeFileSync(orphan, "half-written");
writeFileSync(orphanThumb, "half-written");
writeFileSync(inProgress, "half-written");
writeFileSync(inProgressThumb, "half-written");
writeFileSync(complete, "whole");
writeFileSync(thumb, "whole-thumb");
@@ -200,12 +177,8 @@ describe("ContentCache.open", () => {
await cache.open();
expect(existsSync(orphan)).toBe(false);
expect(existsSync(orphanThumb)).toBe(false);
expect(existsSync(inProgress)).toBe(true);
expect(existsSync(inProgressThumb)).toBe(true);
expect(existsSync(complete)).toBe(true);
expect(existsSync(thumb)).toBe(true);
expect(cache.pathsFor(1)).toEqual({ originalPath: complete });
});
it("records already-cached files so their paths appear in pathsFor", async () => {
@@ -467,125 +440,3 @@ describe("ContentCache.ensureThumbnails", () => {
expect(results[1]?.error).toMatch(/unknown file/i);
});
});
// A live photo's original is two files, its image and its video, which a
// photo viewer can open, and a JSON file naming them: the two are named with
// the extensions from inside the ZIP, so the names alone do not say which is
// which. These tests download a live photo ZIP through the real download
// layer (test/live-photo.ts).
describe("ContentCache live photos", () => {
const originals = (): string => join(cacheDir, "originals");
// A cache over the stand-in server, which holds `bodies` by file ID.
const cacheOf = (
files: EnteFile[],
bodies: Map<number, Uint8Array>,
): ContentCache => buildCache({ files, source: cdnSource(bodies) }).cache;
it("stores a live photo as its image and its video and a JSON file naming them", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open();
const result = await cache.original(5);
expect(result).toEqual({
path: join(originals(), "5.heic"),
videoPath: join(originals(), "5.mov"),
bytes: IMAGE.length,
});
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
expect(statSync(result.videoPath!).mode & 0o777).toBe(0o600);
expect(
JSON.parse(
readFileSync(join(originals(), "5.livephoto.json"), "utf-8"),
),
).toEqual({ image: "5.heic", video: "5.mov" });
expect(cache.pathsFor(5)).toEqual({ originalPath: result.path });
});
it("serves a stored live photo from disk after the cache is opened again", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
const first = cacheOf([live], new Map([[5, body]]));
await first.open();
const stored = await first.original(5);
// This server has nothing, so a fetch would fail.
const second = cacheOf([live], new Map());
await second.open();
const events: string[] = [];
const served = await second.original(5, {
onProgress: (e) => events.push(e.status),
});
expect(served).toEqual(stored);
expect(events).toEqual(["skipped"]);
});
it("replaces a live photo an earlier version stored as a ZIP under the image's name", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
mkdirSync(originals(), { recursive: true });
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open();
const result = await cache.original(5);
expect(result.videoPath).toBe(join(originals(), "5.mov"));
expect(readdirSync(originals()).sort()).toEqual([
"5.heic",
"5.livephoto.json",
"5.mov",
]);
});
it("evicts a live photo's image, video and JSON file together", async () => {
const a = await asLivePhoto(file(5, "a.HEIC"));
const b = await asLivePhoto(file(6, "b.HEIC"));
const size = IMAGE.length + VIDEO.length;
const cache = new ContentCache({
pools: new RequestPools(),
source: cdnSource(
new Map([
[5, a.body],
[6, b.body],
]),
),
cacheDirectory: cacheDir,
getFile: (id) => [a.file, b.file].find((f) => f.id === id),
// Room for one live photo, on a disk with plenty free.
cacheOriginalsMaxBytes: size,
freeBelowBytes: 0,
statfs: async () => ({ bsize: 1, bavail: 1e12 }),
});
await cache.open();
await cache.original(5);
await cache.original(6);
expect(readdirSync(originals()).sort()).toEqual([
"6.heic",
"6.livephoto.json",
"6.mov",
]);
expect(cache.originalsStatus().usedBytes).toBe(size);
});
it("stores nothing when a live photo does not match its recorded hash", async () => {
const { file: live, body } = await asLivePhoto(
file(5, "IMG_5.HEIC"),
livePhotoZip(),
"not:the recorded hash",
);
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open();
await expect(cache.original(5)).rejects.toThrow(
/file 5: content hash .* does not match/,
);
expect(readdirSync(originals())).toEqual([]);
expect(cache.pathsFor(5)).toEqual({});
});
});
test/library/mldata.test.ts
-61
View File
@@ -13,8 +13,6 @@
* 2. `Library` wiring: after each refresh the library fetches ML data through
* the metadata pool for every known file not yet cached, is incremental on
* later refreshes, and refetches a file whose `updationTime` advanced.
* Opening a cache directory written by another account starts empty,
* its ML data included (issue #104).
*
* Embedding values are chosen to be exactly representable as float32 so the
* round-trip through `clip.f32` compares equal.
@@ -500,63 +498,4 @@ describe("Library ML-data fetch on refresh", () => {
await lib.close();
}
});
it("starts empty when the cache directory holds another account's cache", async () => {
// Account A fills the cache directory: metadata and ML data.
const clientA = new MLMockClient();
clientA.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
clientA.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
clientA.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
const libA = await Library.open({
client: clientA,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
await vi.waitFor(
() => expect(libA.status().lastMLFetchAt).toBeGreaterThan(0),
{ timeout: 2000, interval: 5 },
);
} finally {
await libA.close();
}
// Account B opens the same directory.
const clientB = new MLMockClient();
clientB.userID = USER_ID + 1;
const sinceTimes: number[] = [];
const realCollectionsSince = clientB.collectionsSince.bind(clientB);
clientB.collectionsSince = async (args) => {
sinceTimes.push(args.sinceTime);
return realCollectionsSince(args);
};
const libB = await Library.open({
client: clientB,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
expect(sinceTimes[0]).toBe(0);
expect(libB.status().userID).toBe(USER_ID + 1);
expect(libB.listCollections()).toEqual([]);
expect(libB.getFile(1, 1001)).toBeUndefined();
expect(await libB.mldata.forFile({ fileID: 1001 })).toBeUndefined();
expect(
libB.mldata.searchByEmbedding({ embedding: [0.5, 0.25, 0.75] }),
).toEqual([]);
expect(
existsSync(join(cacheDirectory, "mldata", "1001.json")),
).toBe(false);
} finally {
await libB.close();
}
});
});
test/live-photo.ts
-77
View File
@@ -1,77 +0,0 @@
/**
* Live photo fixtures for the download, content cache, backup and CLI tests.
*
* Ente stores a live photo as one ZIP holding its image and its video, which
* Ente's clients name `image.<ext>` and `video.<ext>`. The ZIP is built here
* with fflate from small fixed bytes and encrypted the way the server serves a
* file under 4 MiB, as one secretstream chunk. `cdnSource` serves it to the
* real download layer, so a test checks what quak stores for a real one.
*/
import { createHash } from "node:crypto";
import { zipSync } from "fflate";
import { ApiClient } from "../src/api/client.js";
import { encryptBlob, init, toBase64 } from "../src/crypto/index.js";
import {
makeDownloadContentSource,
type ContentSource,
} from "../src/library/content.js";
import type { EnteFile } from "../src/model/types.js";
export const IMAGE = new TextEncoder().encode("the still image");
export const VIDEO = new TextEncoder().encode("the few seconds of video");
// A live photo's ZIP; by default the entries an iPhone's live photo gets.
export const livePhotoZip = (
entries: Record<string, Uint8Array> = {
"image.heic": IMAGE,
"video.mov": VIDEO,
},
): Uint8Array => zipSync(entries);
const blake2b = (bytes: Uint8Array): string =>
createHash("blake2b512").update(bytes).digest("base64");
// The hash Ente's clients record for a live photo: the unkeyed BLAKE2b-512 of
// the image and of the video, each in standard base64, joined by a colon.
export const livePhotoHash = (image = IMAGE, video = VIDEO): string =>
`${blake2b(image)}:${blake2b(video)}`;
// `file` as a live photo whose original is `zip` and whose recorded hash is
// `hash`, and `body`, what the server serves for it: `zip` encrypted under the
// file's key and header.
export const asLivePhoto = async (
file: EnteFile,
zip = livePhotoZip(),
hash = livePhotoHash(),
): Promise<{ file: EnteFile; body: Uint8Array }> => {
await init();
const key = new Uint8Array(32).fill(file.id & 0xff);
const { header, ciphertext } = encryptBlob(zip, key);
return {
file: {
...file,
key,
metadata: { ...file.metadata, fileType: "livePhoto", hash },
file: { decryptionHeader: toBase64(header) },
},
body: ciphertext,
};
};
// A content source that downloads through the real download layer from a
// stand-in server, which serves `bodies` by file ID and a 404 for any other.
export const cdnSource = (bodies: Map<number, Uint8Array>): ContentSource =>
makeDownloadContentSource(
new ApiClient({
fetch: (async (url: string | URL) => {
const fileID = new URL(String(url)).searchParams.get("fileID");
const body = bodies.get(Number(fileID));
return body === undefined
? new Response("not found", { status: 404 })
: new Response(body);
}) as typeof globalThis.fetch,
retry: { attempts: 1 },
}),
);
test/model/decrypt.test.ts
-40
View File
@@ -351,46 +351,6 @@ describe("model.decryptFile", () => {
}
});
it("reads the recorded content hash, joining an older live photo's two parts", () => {
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const hashOf = (metadata: Record<string, unknown>) =>
decryptFile(
buildRawFile(collectionKey, {
metadata: { title: "x", ...metadata },
}),
collectionKey,
).metadata.hash;
expect(hashOf({ fileType: 0, hash: "H" })).toBe("H");
expect(
hashOf({ fileType: 2, hash: "H", imageHash: "I", videoHash: "V" }),
).toBe("H");
expect(hashOf({ fileType: 2, imageHash: "I", videoHash: "V" })).toBe(
"I:V",
);
expect(hashOf({ fileType: 2, imageHash: "I" })).toBeUndefined();
expect(
hashOf({ fileType: 0, imageHash: "I", videoHash: "V" }),
).toBeUndefined();
expect(hashOf({ fileType: 0 })).toBeUndefined();
expect(hashOf({ fileType: 0, hash: 42 })).toBeUndefined();
expect(
hashOf({ fileType: 2, imageHash: "I", videoHash: 7 }),
).toBeUndefined();
// An empty string counts as absent, not as a hash to match.
expect(hashOf({ fileType: 0, hash: "" })).toBeUndefined();
expect(
hashOf({ fileType: 2, hash: "", imageHash: "I", videoHash: "V" }),
).toBe("I:V");
expect(
hashOf({ fileType: 2, imageHash: "", videoHash: "V" }),
).toBeUndefined();
expect(
hashOf({ fileType: 2, imageHash: "I", videoHash: "" }),
).toBeUndefined();
});
it("maps fileType numbers to FileType strings", () => {
// Ente uses: 0=image, 1=video, 2=livePhoto
const masterKey = sodium.crypto_secretbox_keygen();
test/thumbnails/thumbnails.test.ts
-143
View File
@@ -215,9 +215,6 @@ const buildThumbMock = async (opts?: {
thumbnail: {
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
},
// The encrypted size of the thumbnail the server records; large
// enough here that the default encoding fits.
info: { thumbSize: 1_000_000 },
updationTime: TEST_TIME,
};
};
@@ -449,32 +446,6 @@ const openLib = (client: Client): Promise<Library> =>
precacheOriginals: false,
});
/** The mock's raw record for one file, for a test to change before login. */
const rawFile = (m: ThumbMockState, fileID: number): Record<string, unknown> =>
m.filesByCollection[1]!.find((f) => f.id === fileID)!;
/** Replace the original the mock serves for one file. */
const replaceOriginal = (
m: ThumbMockState,
fileID: number,
body: Uint8Array,
): void => {
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(
m.fileKeys[fileID]!,
);
m.fileCiphertexts[fileID] =
sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
body,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
rawFile(m, fileID).file = { decryptionHeader: toBase64(push.header) };
};
const isOriginalDownload = (url: string): boolean =>
url.includes("files.ente.io") || url.includes("/files/download/");
const login = (fetch: typeof globalThis.fetch, retry?: RetryOptions) =>
Client.login({
email: TEST_EMAIL,
@@ -610,33 +581,6 @@ describe("listMissingThumbnails", () => {
// Should still be 2, not 4 (each file checked only once)
expect(missing.length).toBe(2);
});
it("skips a file another account owns without fetching its thumbnail", async () => {
const otherMock = await buildThumbMock();
rawFile(otherMock, 102).ownerID = 7;
const logs: string[] = [];
const counted = countingFetch(
buildThumbFetch(otherMock),
(url) => url.includes("thumbnails.ente.io") && url.includes("102"),
);
const client = await login(counted.fetch);
const lib = await openLib(client);
const missing = await listMissingThumbnails(lib, client, (msg) =>
logs.push(msg),
);
lib.close();
expect(missing.map((m) => m.fileID)).toEqual([101]);
expect(counted.matched()).toBe(0);
expect(
logs.some(
(l) =>
l.includes("Skipping file-102.jpg") &&
l.includes("another account"),
),
).toBe(true);
});
});
describe("fixMissingThumbnails", () => {
@@ -744,93 +688,6 @@ describe("fixMissingThumbnails", () => {
expect(fixMock.uploadedThumbnails.length).toBe(1);
expect(fixMock.uploadedThumbnails[0]!.fileID).toBe(101);
});
it("skips a file another account owns without downloading it", async () => {
// The server accepts a thumbnail only from the file's owner.
const fixMock = await buildThumbMock();
rawFile(fixMock, 101).ownerID = 7;
const counted = countingFetch(
buildThumbFetch(fixMock),
isOriginalDownload,
);
const client = await login(counted.fetch);
const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]);
lib.close();
expect(results[0]!.status).toBe("skipped");
expect(results[0]!.reason).toContain("another account");
expect(counted.matched()).toBe(0);
expect(fixMock.uploadedThumbnails.length).toBe(0);
});
it("skips a file whose recorded thumbnail size is 0 without downloading it", async () => {
// The server refuses a thumbnail larger than the one it records, and
// no thumbnail is 0 bytes.
const fixMock = await buildThumbMock();
rawFile(fixMock, 101).info = { thumbSize: 0 };
const counted = countingFetch(
buildThumbFetch(fixMock),
isOriginalDownload,
);
const client = await login(counted.fetch);
const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]);
lib.close();
expect(results[0]!.status).toBe("skipped");
expect(results[0]!.reason).toContain("recorded thumbnail size is 0");
expect(counted.matched()).toBe(0);
expect(fixMock.uploadedThumbnails.length).toBe(0);
});
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
// A noisy 400x300 JPEG, which the default encoding (quality 50, not
// resized because it is under 720 px) cannot compress below the size
// recorded here: one byte less than that encoding's ciphertext.
const fixMock = await buildThumbMock();
const w = 400;
const h = 300;
const noisy = new Uint8Array(
jpegJs.encode(
{
data: sodium.randombytes_buf(w * h * 4),
width: w,
height: h,
},
90,
).data,
);
replaceOriginal(fixMock, 101, noisy);
const decoded = jpegJs.decode(noisy, {
useTArray: true,
formatAsRGBA: true,
});
const defaultSize =
jpegJs.encode(decoded, 50).data.length +
sodium.crypto_secretstream_xchacha20poly1305_ABYTES;
const recordedSize = defaultSize - 1;
rawFile(fixMock, 101).info = { thumbSize: recordedSize };
const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]);
lib.close();
expect(results[0]!.status).toBe("fixed");
const upload = fixMock.uploadedThumbnails[0]!;
expect(upload.ciphertext.length).toBeLessThanOrEqual(recordedSize);
const decrypted = decryptBlob(
upload.ciphertext,
fromBase64(upload.decryptionHeader),
fixMock.fileKeys[101]!,
);
expect(decrypted[0]).toBe(0xff);
expect(decrypted[1]).toBe(0xd8);
});
});
describe("Client.getApiClient", () => {
yarn.lock
-5
View File
@@ -1069,11 +1069,6 @@ fastq@^1.6.0:
dependencies:
reusify "^1.0.4"
fflate@0.8.3:
version "0.8.3"
resolved "https://registry.yarnpkg.com/fflate/-/fflate-0.8.3.tgz#bc27d8eb30343d4d512abb03480202ce65d825fc"
integrity sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==
file-entry-cache@^8.0.0:
version "8.0.0"
resolved "https://registry.yarnpkg.com/file-entry-cache/-/file-entry-cache-8.0.0.tgz#7787bddcf1131bffb92636c69457bbc0edd6d81f"