Compare commits
2
Commits
1bd6d72876
...
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2402ec97f7 | ||
|
|
ddf58af3cc |
@@ -10,9 +10,12 @@ account into a deduplicated local directory tree, skipping files that already
|
|||||||
exist on disk and continuing past individual download failures instead of
|
exist on disk and continuing past individual download failures instead of
|
||||||
crashing. For each file it persists the basic metadata fields quak keeps (title,
|
crashing. For each file it persists the basic metadata fields quak keeps (title,
|
||||||
file type, creation and modification time, latitude, longitude, content hash),
|
file type, creation and modification time, latitude, longitude, content hash),
|
||||||
and the private and public magic metadata in full. A helper subcommand can
|
its update time, the private and public magic metadata in full, Ente's ML data
|
||||||
detect and regenerate missing thumbnails, encrypting and uploading them back to
|
for it when there is any, and, for an image (for a live photo, its image), its
|
||||||
the server.
|
original's EXIF and XMP, with its dimensions for a JPEG only; for a video it
|
||||||
|
keeps none of these three. It runs unattended from cron (see "Running the backup
|
||||||
|
from cron"). A helper subcommand can detect and regenerate missing thumbnails,
|
||||||
|
encrypting and uploading them back to the server.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
@@ -80,6 +83,64 @@ await lib.close();
|
|||||||
The lower-level `Client` (login, session serialization, and the raw
|
The lower-level `Client` (login, session serialization, and the raw
|
||||||
enumeration/download calls) is exported too and documented under Design below.
|
enumeration/download calls) is exported too and documented under Design below.
|
||||||
|
|
||||||
|
## Running the backup from cron
|
||||||
|
|
||||||
|
`quak backup` never prompts, so cron can run it. `make install` builds quak as a
|
||||||
|
single binary and copies it to `~/bin/quak`. Log in once with it, as the user
|
||||||
|
the cron job will run as; the session is saved in that user's data directory
|
||||||
|
(see "Session handling"):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
~/bin/quak login
|
||||||
|
```
|
||||||
|
|
||||||
|
Then add the backup to that user's crontab with `crontab -e`. These two lines
|
||||||
|
back up the account to `~/photos-backup` at 03:30 every night, with `--verify`
|
||||||
|
on Sundays, and append all output to `~/quak-backup.log`:
|
||||||
|
|
||||||
|
```
|
||||||
|
30 3 * * 1-6 $HOME/bin/quak backup $HOME/photos-backup >> $HOME/quak-backup.log 2>&1
|
||||||
|
30 3 * * 0 $HOME/bin/quak backup --verify $HOME/photos-backup >> $HOME/quak-backup.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
A run with `--verify` does all a plain run does, and also checks each original
|
||||||
|
already in the backup against the content hash Ente records for it, and replaces
|
||||||
|
any that do not match (see "Backup layout"). Sunday's run is the `--verify` one,
|
||||||
|
not a second job that night, because a backup that starts while another backup
|
||||||
|
of the same directory is running exits 2 without backing anything up.
|
||||||
|
|
||||||
|
Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab
|
||||||
|
names quak by its full path. To find the saved session, quak needs the same
|
||||||
|
`HOME` as when you logged in, which cron sets from the password file, and on
|
||||||
|
Linux the same `XDG_DATA_HOME`: quak looks for the session in
|
||||||
|
`$XDG_DATA_HOME/quak`, or in `~/.local/share/quak` when that is not set. Cron
|
||||||
|
does not set `XDG_DATA_HOME`, so if your login shell does, set it at the top of
|
||||||
|
the crontab too. Cron does not expand variables in such a line, so give the full
|
||||||
|
path:
|
||||||
|
|
||||||
|
```
|
||||||
|
XDG_DATA_HOME=/home/you/.data
|
||||||
|
```
|
||||||
|
|
||||||
|
On macOS the session is in `~/Library/Application Support/quak`, and only `HOME`
|
||||||
|
matters.
|
||||||
|
|
||||||
|
### Exit codes
|
||||||
|
|
||||||
|
| Code | Meaning |
|
||||||
|
| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
| `0` | The backup is complete: every original is at its save path, and no file failed. |
|
||||||
|
| `1` | The run finished with files in `failures.json`, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||||
|
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||||
|
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||||
|
|
||||||
|
After a `1`, the next run fetches each missing original again, fetches the ML
|
||||||
|
data that is not cached, and rebuilds the album links. An original that a
|
||||||
|
`--verify` run could not read, or put back with bytes that still do not match,
|
||||||
|
stays at its save path. Only a later `--verify` run checks it again: a run
|
||||||
|
without `--verify` leaves it as it is and takes the file out of `failures.json`,
|
||||||
|
so that run can exit 0.
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
`examples/download-albums.ts` downloads every album's photos and their metadata
|
`examples/download-albums.ts` downloads every album's photos and their metadata
|
||||||
@@ -549,7 +610,7 @@ quak collections [--json] list all collections
|
|||||||
quak files --collection <id> [--json] list files in a collection
|
quak files --collection <id> [--json] list files in a collection
|
||||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||||
quak backup <dir> [--json] full incremental backup
|
quak backup <dir> [--json] [--verify] full incremental backup
|
||||||
quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON
|
quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON
|
||||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||||
quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails
|
quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails
|
||||||
@@ -582,6 +643,10 @@ reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
|||||||
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
|
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
|
||||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||||
|
|
||||||
|
`backup --verify` also hashes the originals already in the backup and replaces
|
||||||
|
any that do not match the content hash Ente records; one it cannot replace goes
|
||||||
|
into `failures.json` (see "Backup layout").
|
||||||
|
|
||||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||||
fails, the error is logged, each of its files is written with the reason in an
|
fails, the error is logged, each of its files is written with the reason in an
|
||||||
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
||||||
@@ -611,8 +676,11 @@ the smallest does not.
|
|||||||
below)
|
below)
|
||||||
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
||||||
keeps, its update time, its private and
|
keeps, its update time, its private and
|
||||||
public magic metadata, its ML data, and
|
public magic metadata, its ML data, and,
|
||||||
its original's EXIF, XMP and dimensions
|
for an image (for a live photo, its
|
||||||
|
image), its original's EXIF and XMP, with
|
||||||
|
dimensions for a JPEG only; none of these
|
||||||
|
three for a video
|
||||||
YYYY-MM-DD.<fileID>.livephoto.json
|
YYYY-MM-DD.<fileID>.livephoto.json
|
||||||
which of a live photo's two files is which
|
which of a live photo's two files is which
|
||||||
collections/
|
collections/
|
||||||
@@ -715,6 +783,25 @@ if any files failed. `quak backup` opens its library with the thumbnail and
|
|||||||
originals precache off, so the only file content it fetches is the originals the
|
originals precache off, so the only file content it fetches is the originals the
|
||||||
backup stores.
|
backup stores.
|
||||||
|
|
||||||
|
With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each
|
||||||
|
original already at its save path the way a download is checked (see "On-disk
|
||||||
|
cache layout" below): its bytes, read in chunks, or a live photo's image and
|
||||||
|
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
|
||||||
|
hash its metadata records is left as it is. One that does not is logged on one
|
||||||
|
line naming the file, deleted (a live photo's image and video both), and put
|
||||||
|
back in the same run like a missing one: downloaded, or copied from the cache if
|
||||||
|
the cache holds it. What is put back is hashed too, because a copy from the
|
||||||
|
cache is not checked as a download is. If it still does not match, it stays at
|
||||||
|
its save path and the file goes into `failures.json`, as it does when the
|
||||||
|
download fails. A file whose metadata records no hash is left as it is and
|
||||||
|
counted as unchecked. A stored original that cannot be read is left as it is and
|
||||||
|
counts as failed. The summary and `--json` add the counts `verified`,
|
||||||
|
`mismatched` and `unchecked`, all of originals that were already stored; one
|
||||||
|
first downloaded in this run is in none of them. A mismatch that was put back
|
||||||
|
with matching bytes does not make the exit code non-zero. Without `--verify`
|
||||||
|
nothing is hashed, the summary is unchanged, and the three counts are 0 in
|
||||||
|
`--json`.
|
||||||
|
|
||||||
Each original is written to a temporary file in the same directory, synced to
|
Each original is written to a temporary file in the same directory, synced to
|
||||||
disk, and renamed into place, so an original is either complete or absent, even
|
disk, and renamed into place, so an original is either complete or absent, even
|
||||||
after a power cut. A downloaded original's temporary file is named
|
after a power cut. A downloaded original's temporary file is named
|
||||||
@@ -939,14 +1026,16 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
|||||||
`fresh()` does, puts every in-scope original not already at its save path
|
`fresh()` does, puts every in-scope original not already at its save path
|
||||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
thumbnails) through the content cache, waits for an ML data fetch, and
|
||||||
rebuilds the on-disk backup tree, each file's JSON with its ML data and its
|
rebuilds the on-disk backup tree with a durable failure ledger. Each file's
|
||||||
original's EXIF, XMP and dimensions, with a durable failure ledger. A fetched
|
JSON holds its ML data and, for an image (for a live photo, its image), its
|
||||||
original is written straight to its save path and not into the cache, which
|
original's EXIF and XMP, with its dimensions for a JPEG only; a video's JSON
|
||||||
then counts it as present; one the cache already held is copied from there.
|
holds none of these three. A fetched original is written straight to its save
|
||||||
`BackupOptions`: `downloadDirectory` (falls back to the library's),
|
path and not into the cache, which then counts it as present; one the cache
|
||||||
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
|
||||||
`onlyAlbumNames`, `onProgress`, and `lockHeld` (default `false`). See Backup
|
back to the library's), `includeOriginals` (default `true`),
|
||||||
layout above for the tree it writes.
|
`includeThumbnails` (default `false`), `onlyAlbumNames`, `verify` (default
|
||||||
|
`false`), `onProgress`, and `lockHeld` (default `false`). See Backup layout
|
||||||
|
above for the tree it writes.
|
||||||
|
|
||||||
### Request pools
|
### Request pools
|
||||||
|
|
||||||
|
|||||||
@@ -25,6 +25,25 @@ declares one.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
|
||||||
|
log in once as the job's user, a crontab with a backup every night and
|
||||||
|
`--verify` on Sundays appending to a log file, and the `HOME` and
|
||||||
|
`XDG_DATA_HOME` the job needs to find the saved session. A table gives each
|
||||||
|
exit code: 0, the backup is complete; 1, files are in `failures.json` or
|
||||||
|
another error stopped the run; 2, another backup of the directory is running;
|
||||||
|
3, there is no usable session. The introduction lists everything the backup
|
||||||
|
keeps for each file.
|
||||||
|
|
||||||
|
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
|
||||||
|
each original already at its save path as the download check does, streamed, a
|
||||||
|
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
|
||||||
|
the content hash its metadata records is logged, removed (both files of a live
|
||||||
|
photo) and put back in the same run, downloaded or copied from the cache, and
|
||||||
|
what is put back is hashed too. One that still does not match, or whose
|
||||||
|
download fails, goes into `failures.json`. One with no recorded hash is left
|
||||||
|
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
|
||||||
|
`unchecked`. Without `--verify` nothing is hashed.
|
||||||
|
|
||||||
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
||||||
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
||||||
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
||||||
|
|||||||
+5
-1
@@ -130,9 +130,13 @@ program
|
|||||||
)
|
)
|
||||||
.argument("<dir>", "Output directory")
|
.argument("<dir>", "Output directory")
|
||||||
.option("--json", "Print result as JSON instead of human-readable summary")
|
.option("--json", "Print result as JSON instead of human-readable summary")
|
||||||
|
.option(
|
||||||
|
"--verify",
|
||||||
|
"Re-hash stored originals and download again any that do not match",
|
||||||
|
)
|
||||||
// A backup usually runs from cron with nobody watching, so every request
|
// A backup usually runs from cron with nobody watching, so every request
|
||||||
// it makes retries for longer than the other commands' requests do.
|
// it makes retries for longer than the other commands' requests do.
|
||||||
.action((dir: string, opts: { json?: boolean }) =>
|
.action((dir: string, opts: { json?: boolean; verify?: boolean }) =>
|
||||||
run(
|
run(
|
||||||
backupCommand(
|
backupCommand(
|
||||||
{
|
{
|
||||||
|
|||||||
+104
-4
@@ -38,6 +38,11 @@
|
|||||||
// why they could not be read), so that a run does not read every stored
|
// why they could not be read), so that a run does not read every stored
|
||||||
// original again; a sidecar without them gets them read from the original.
|
// original again; a sidecar without them gets them read from the original.
|
||||||
//
|
//
|
||||||
|
// With `verify`, each original already at its save path is hashed as the
|
||||||
|
// download check hashes it, and one that does not match the content hash its
|
||||||
|
// metadata records is removed and fetched again in the same run. What is put
|
||||||
|
// back is hashed too, and recorded as failed if it still does not match.
|
||||||
|
//
|
||||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
// download, a failed symlink, or ML data missing because the ML data fetch
|
||||||
// failed is caught, recorded in `failures.json` with a classification, a
|
// failed is caught, recorded in `failures.json` with a classification, a
|
||||||
@@ -50,6 +55,7 @@
|
|||||||
// code.
|
// code.
|
||||||
|
|
||||||
import {
|
import {
|
||||||
|
createReadStream,
|
||||||
lstatSync,
|
lstatSync,
|
||||||
mkdirSync,
|
mkdirSync,
|
||||||
readdirSync,
|
readdirSync,
|
||||||
@@ -65,6 +71,12 @@ import { readFile } from "node:fs/promises";
|
|||||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||||
import lockfile from "proper-lockfile";
|
import lockfile from "proper-lockfile";
|
||||||
|
|
||||||
|
import {
|
||||||
|
chunkHashFinal,
|
||||||
|
chunkHashInit,
|
||||||
|
chunkHashUpdate,
|
||||||
|
init,
|
||||||
|
} from "./crypto/index.js";
|
||||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||||
import { sanitizeFileName, withExtension } from "./filename.js";
|
import { sanitizeFileName, withExtension } from "./filename.js";
|
||||||
import {
|
import {
|
||||||
@@ -92,6 +104,9 @@ export interface BackupOptions {
|
|||||||
includeThumbnails?: boolean;
|
includeThumbnails?: boolean;
|
||||||
// Restrict the backup to albums with these names; others are left untouched.
|
// Restrict the backup to albums with these names; others are left untouched.
|
||||||
onlyAlbumNames?: string[];
|
onlyAlbumNames?: string[];
|
||||||
|
// Hash each original already at its save path, and fetch again any whose
|
||||||
|
// bytes do not match the content hash its metadata records. Default false.
|
||||||
|
verify?: boolean;
|
||||||
onProgress?: ProgressCallback;
|
onProgress?: ProgressCallback;
|
||||||
// The caller already holds the lock in `downloadDirectory`, taken with
|
// The caller already holds the lock in `downloadDirectory`, taken with
|
||||||
// `lockBackupDirectory`, and releases it itself, so the backup does not
|
// `lockBackupDirectory`, and releases it itself, so the backup does not
|
||||||
@@ -113,6 +128,13 @@ export interface BackupResult {
|
|||||||
downloaded: number;
|
downloaded: number;
|
||||||
// Originals already at their save path and left untouched.
|
// Originals already at their save path and left untouched.
|
||||||
skipped: number;
|
skipped: number;
|
||||||
|
// With `verify`, the originals already at their save path whose hash
|
||||||
|
// matched, those whose hash did not (each removed and fetched again), and
|
||||||
|
// those whose metadata records no hash (left as they are). All three are
|
||||||
|
// zero without `verify`.
|
||||||
|
verified: number;
|
||||||
|
mismatched: number;
|
||||||
|
unchecked: number;
|
||||||
// Files with an unresolved failure after this run (the ledger size); the
|
// Files with an unresolved failure after this run (the ledger size); the
|
||||||
// CLI exits non-zero while this is above zero. A file can be both
|
// CLI exits non-zero while this is above zero. A file can be both
|
||||||
// downloaded and failed if its bytes landed but its symlink did not.
|
// downloaded and failed if its bytes landed but its symlink did not.
|
||||||
@@ -369,6 +391,26 @@ const saveLedger = (path: string, ledger: Map<number, FailureEntry>): void => {
|
|||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// The content hash of the original stored at `stored`, computed as the download
|
||||||
|
// check computes it: over each file's bytes, read in chunks, and for a live
|
||||||
|
// photo `<imageHash>:<videoHash>`.
|
||||||
|
const storedHash = async (stored: {
|
||||||
|
path: string;
|
||||||
|
videoPath?: string;
|
||||||
|
}): Promise<string> => {
|
||||||
|
await init();
|
||||||
|
const hashFile = async (path: string): Promise<string> => {
|
||||||
|
const state = chunkHashInit();
|
||||||
|
for await (const chunk of createReadStream(path)) {
|
||||||
|
chunkHashUpdate(state, chunk as Buffer);
|
||||||
|
}
|
||||||
|
return chunkHashFinal(state);
|
||||||
|
};
|
||||||
|
const hash = await hashFile(stored.path);
|
||||||
|
if (stored.videoPath === undefined) return hash;
|
||||||
|
return `${hash}:${await hashFile(stored.videoPath)}`;
|
||||||
|
};
|
||||||
|
|
||||||
// A file's EXIF, XMP and dimensions as its JSON holds them: what
|
// A file's EXIF, XMP and dimensions as its JSON holds them: what
|
||||||
// `extractImageMetadata` found in its original, or why the original could not
|
// `extractImageMetadata` found in its original, or why the original could not
|
||||||
// be read.
|
// be read.
|
||||||
@@ -468,6 +510,7 @@ const runLockedBackup = async (
|
|||||||
const includeThumbnails = opts.includeThumbnails ?? false;
|
const includeThumbnails = opts.includeThumbnails ?? false;
|
||||||
const log = opts.onProgress ?? (() => {});
|
const log = opts.onProgress ?? (() => {});
|
||||||
const only = opts.onlyAlbumNames ? new Set(opts.onlyAlbumNames) : undefined;
|
const only = opts.onlyAlbumNames ? new Set(opts.onlyAlbumNames) : undefined;
|
||||||
|
const verify = opts.verify ?? false;
|
||||||
|
|
||||||
log("Refreshing library...");
|
log("Refreshing library...");
|
||||||
await lib.refresh();
|
await lib.refresh();
|
||||||
@@ -526,6 +569,9 @@ const runLockedBackup = async (
|
|||||||
const storedThisRun = new Set<number>();
|
const storedThisRun = new Set<number>();
|
||||||
let downloaded = 0;
|
let downloaded = 0;
|
||||||
let skipped = 0;
|
let skipped = 0;
|
||||||
|
let verified = 0;
|
||||||
|
let mismatched = 0;
|
||||||
|
let unchecked = 0;
|
||||||
|
|
||||||
const recordFailure = (
|
const recordFailure = (
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
@@ -557,10 +603,47 @@ const runLockedBackup = async (
|
|||||||
|
|
||||||
// Phase 1: get the bytes. Put each pending original at its save path
|
// Phase 1: get the bytes. Put each pending original at its save path
|
||||||
// through the content cache/pools, as `Photo.download()` does, and fetch
|
// through the content cache/pools, as `Photo.download()` does, and fetch
|
||||||
// the optional thumbnails; a present file is left as is.
|
// the optional thumbnails; a present file is left as is. With `verify`, a
|
||||||
|
// present original is hashed first, and one that does not match the hash
|
||||||
|
// its metadata records is removed and fetched like a missing one. One
|
||||||
|
// that cannot be read is recorded as failed and left where it is.
|
||||||
if (includeOriginals) {
|
if (includeOriginals) {
|
||||||
for (const [fileID, file] of distinct) {
|
for (const [fileID, file] of distinct) {
|
||||||
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
|
let stored = storedAtSavePath(downloadDirectory, file);
|
||||||
|
let mismatch = false;
|
||||||
|
if (stored !== undefined && verify) {
|
||||||
|
try {
|
||||||
|
if (file.metadata.hash === undefined) {
|
||||||
|
unchecked++;
|
||||||
|
} else if (
|
||||||
|
(await storedHash(stored)) === file.metadata.hash
|
||||||
|
) {
|
||||||
|
verified++;
|
||||||
|
} else {
|
||||||
|
log(
|
||||||
|
`MISMATCH original ${file.metadata.title} (${fileID}): its bytes do not match its content hash`,
|
||||||
|
);
|
||||||
|
mismatched++;
|
||||||
|
mismatch = true;
|
||||||
|
rmSync(stored.path);
|
||||||
|
if (stored.videoPath !== undefined) {
|
||||||
|
rmSync(stored.videoPath);
|
||||||
|
}
|
||||||
|
stored = undefined;
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
log(
|
||||||
|
`FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
|
||||||
|
);
|
||||||
|
recordFailure(
|
||||||
|
file,
|
||||||
|
collectionName.get(file.collectionID) ?? "",
|
||||||
|
err,
|
||||||
|
);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (stored !== undefined) {
|
||||||
skipped++;
|
skipped++;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -569,10 +652,24 @@ const runLockedBackup = async (
|
|||||||
// A fetched original is written straight to its save path (a
|
// A fetched original is written straight to its save path (a
|
||||||
// live photo beside it); only one that was already cached
|
// live photo beside it); only one that was already cached
|
||||||
// elsewhere is copied.
|
// elsewhere is copied.
|
||||||
await placeOriginal(downloadDirectory, file, (dest) =>
|
const placed = await placeOriginal(
|
||||||
lib.original(fileID, dest),
|
downloadDirectory,
|
||||||
|
file,
|
||||||
|
(dest) => lib.original(fileID, dest),
|
||||||
);
|
);
|
||||||
storedThisRun.add(fileID);
|
storedThisRun.add(fileID);
|
||||||
|
// A copy from the cache is not checked as a download is and can
|
||||||
|
// hold the same bad bytes, so what is put back after a
|
||||||
|
// mismatch is hashed too. A bad copy stays where it is and the
|
||||||
|
// file fails.
|
||||||
|
if (
|
||||||
|
mismatch &&
|
||||||
|
(await storedHash(placed)) !== file.metadata.hash
|
||||||
|
) {
|
||||||
|
throw new Error(
|
||||||
|
"the original put back does not match its content hash either",
|
||||||
|
);
|
||||||
|
}
|
||||||
downloaded++;
|
downloaded++;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(
|
log(
|
||||||
@@ -735,6 +832,9 @@ const runLockedBackup = async (
|
|||||||
totalFiles: distinct.size,
|
totalFiles: distinct.size,
|
||||||
downloaded,
|
downloaded,
|
||||||
skipped,
|
skipped,
|
||||||
|
verified,
|
||||||
|
mismatched,
|
||||||
|
unchecked,
|
||||||
failed: ledger.size,
|
failed: ledger.size,
|
||||||
errors,
|
errors,
|
||||||
};
|
};
|
||||||
|
|||||||
+7
-1
@@ -402,7 +402,7 @@ export const backupMetadataCommand = async (
|
|||||||
export const backupCommand = async (
|
export const backupCommand = async (
|
||||||
ctx: CliContext,
|
ctx: CliContext,
|
||||||
dir: string,
|
dir: string,
|
||||||
opts: { json?: boolean },
|
opts: { json?: boolean; verify?: boolean },
|
||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
@@ -433,6 +433,7 @@ export const backupCommand = async (
|
|||||||
const result = await lib.backup({
|
const result = await lib.backup({
|
||||||
downloadDirectory: dir,
|
downloadDirectory: dir,
|
||||||
lockHeld: true,
|
lockHeld: true,
|
||||||
|
verify: opts.verify,
|
||||||
onProgress: (msg) => {
|
onProgress: (msg) => {
|
||||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||||
},
|
},
|
||||||
@@ -445,6 +446,11 @@ export const backupCommand = async (
|
|||||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||||
|
if (opts.verify) {
|
||||||
|
ctx.stderr.write(` Verified: ${result.verified}\n`);
|
||||||
|
ctx.stderr.write(` Mismatched: ${result.mismatched}\n`);
|
||||||
|
ctx.stderr.write(` Unchecked: ${result.unchecked}\n`);
|
||||||
|
}
|
||||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||||
if (result.errors.length > 0) {
|
if (result.errors.length > 0) {
|
||||||
ctx.stderr.write("\nFailed files:\n");
|
ctx.stderr.write("\nFailed files:\n");
|
||||||
|
|||||||
@@ -62,6 +62,7 @@ import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
|||||||
import { JPEG_WITH_EXIF } from "../exif-jpeg.js";
|
import { JPEG_WITH_EXIF } from "../exif-jpeg.js";
|
||||||
import {
|
import {
|
||||||
asLivePhoto,
|
asLivePhoto,
|
||||||
|
blake2b,
|
||||||
cdnSource,
|
cdnSource,
|
||||||
IMAGE,
|
IMAGE,
|
||||||
livePhotoHash,
|
livePhotoHash,
|
||||||
@@ -1271,6 +1272,181 @@ describe("image metadata in each file's JSON", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// With `verify`, each original already stored is hashed, and one that does not
|
||||||
|
// match the content hash its metadata records is downloaded again.
|
||||||
|
describe("backup with verify", () => {
|
||||||
|
// MockClient's files, each recording the hash of what `stubSource` writes
|
||||||
|
// for it, except diagram.png (200), which records none.
|
||||||
|
class HashedClient extends MockClient {
|
||||||
|
override async filesSince(args: {
|
||||||
|
collectionID: number;
|
||||||
|
}): Promise<FilesPage> {
|
||||||
|
const page = await super.filesSince(args);
|
||||||
|
const files = page.files.map((f) =>
|
||||||
|
f.id === 200
|
||||||
|
? f
|
||||||
|
: {
|
||||||
|
...f,
|
||||||
|
metadata: {
|
||||||
|
...f.metadata,
|
||||||
|
hash: blake2b(Buffer.alloc(SIZE_BY_ID[f.id]!)),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
return { ...page, files };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A backup of the account in `root/backup`, the library that made it, and
|
||||||
|
// the source it fetched from.
|
||||||
|
const backedUp = async (): Promise<{
|
||||||
|
lib: Library;
|
||||||
|
source: StubSource;
|
||||||
|
outDir: string;
|
||||||
|
}> => {
|
||||||
|
const source = stubSource();
|
||||||
|
const lib = await openLibrary(source, new HashedClient());
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
return { lib, source, outDir };
|
||||||
|
};
|
||||||
|
|
||||||
|
// What `stubSource` writes for beach.jpg (100), and other bytes of the
|
||||||
|
// same length.
|
||||||
|
const good = Buffer.alloc(SIZE_BY_ID[100]!);
|
||||||
|
const corrupt = Buffer.alloc(SIZE_BY_ID[100]!, 1);
|
||||||
|
|
||||||
|
it("leaves an original that matches its hash, and one with no hash, as they are", async () => {
|
||||||
|
const { lib, source, outDir } = await backedUp();
|
||||||
|
const calls = source.originalCalls;
|
||||||
|
|
||||||
|
const result = await lib.backup({
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
verify: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toMatchObject({
|
||||||
|
downloaded: 0,
|
||||||
|
skipped: 3,
|
||||||
|
verified: 2,
|
||||||
|
mismatched: 0,
|
||||||
|
unchecked: 1,
|
||||||
|
failed: 0,
|
||||||
|
});
|
||||||
|
expect(source.originalCalls).toBe(calls);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("downloads again an original that does not match its hash", async () => {
|
||||||
|
const { lib, outDir } = await backedUp();
|
||||||
|
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||||
|
const log: string[] = [];
|
||||||
|
|
||||||
|
const result = await lib.backup({
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
verify: true,
|
||||||
|
onProgress: (msg) => log.push(msg),
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toMatchObject({
|
||||||
|
downloaded: 1,
|
||||||
|
skipped: 2,
|
||||||
|
verified: 1,
|
||||||
|
mismatched: 1,
|
||||||
|
unchecked: 1,
|
||||||
|
failed: 0,
|
||||||
|
});
|
||||||
|
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(good);
|
||||||
|
expect(log.filter((msg) => msg.startsWith("MISMATCH"))).toEqual([
|
||||||
|
"MISMATCH original beach.jpg (100): its bytes do not match its content hash",
|
||||||
|
]);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("records a failed download in failures.json, with the original removed", async () => {
|
||||||
|
const { lib, source, outDir } = await backedUp();
|
||||||
|
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||||
|
source.failID = 100;
|
||||||
|
|
||||||
|
const result = await lib.backup({
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
verify: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toMatchObject({
|
||||||
|
downloaded: 0,
|
||||||
|
mismatched: 1,
|
||||||
|
failed: 1,
|
||||||
|
});
|
||||||
|
expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
|
||||||
|
expect(existsSync(saved(outDir, "100.jpg"))).toBe(false);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("records in failures.json an original the cache puts back with the same bad bytes", async () => {
|
||||||
|
const lib = await openLibrary(stubSource(), new HashedClient());
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
// The cache holds a bad copy, and a backup copies an original the
|
||||||
|
// cache holds to its save path.
|
||||||
|
const cached = await lib.photos.byID({ fileID: 100 })!.original();
|
||||||
|
writeFileSync(cached.path, corrupt);
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
const result = await lib.backup({
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
verify: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toMatchObject({
|
||||||
|
downloaded: 0,
|
||||||
|
mismatched: 1,
|
||||||
|
failed: 1,
|
||||||
|
});
|
||||||
|
expect(result.errors.map((e) => e.error)).toEqual([
|
||||||
|
"the original put back does not match its content hash either",
|
||||||
|
]);
|
||||||
|
expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
|
||||||
|
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("hashes nothing without verify", async () => {
|
||||||
|
const { lib, outDir } = await backedUp();
|
||||||
|
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||||
|
|
||||||
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(result).toMatchObject({
|
||||||
|
downloaded: 0,
|
||||||
|
skipped: 3,
|
||||||
|
verified: 0,
|
||||||
|
mismatched: 0,
|
||||||
|
unchecked: 0,
|
||||||
|
failed: 0,
|
||||||
|
});
|
||||||
|
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Root ignores file permissions, so this fails when run as root. The
|
||||||
|
// `test` phase of the `Dockerfile` runs as the `node` user.
|
||||||
|
it("records an original it cannot read as failed, and leaves it", async () => {
|
||||||
|
const { lib, outDir } = await backedUp();
|
||||||
|
const original = saved(outDir, "100.jpg");
|
||||||
|
chmodSync(original, 0o000);
|
||||||
|
|
||||||
|
const result = await lib
|
||||||
|
.backup({ downloadDirectory: outDir, verify: true })
|
||||||
|
.finally(() => chmodSync(original, 0o600));
|
||||||
|
|
||||||
|
expect(result).toMatchObject({ downloaded: 0, verified: 1, failed: 1 });
|
||||||
|
expect(result.errors.map((e) => e.fileID)).toEqual([100]);
|
||||||
|
expect(result.errors[0]!.error).toMatch(/EACCES/);
|
||||||
|
expect(readFileSync(original)).toEqual(good);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
// Every entry under collections/, one level of directories deep, with each
|
// Every entry under collections/, one level of directories deep, with each
|
||||||
// symlink's target.
|
// symlink's target.
|
||||||
const tree = (outDir: string): string[] => {
|
const tree = (outDir: string): string[] => {
|
||||||
@@ -1766,6 +1942,64 @@ describe("backup of live photos", () => {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
|
it("verifies a live photo's image and video together, and downloads it again when one does not match", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(
|
||||||
|
file(500, 10, "IMG_0500.HEIC"),
|
||||||
|
);
|
||||||
|
const lib = await open([live], new Map([[500, body]]));
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
const good = await lib.backup({
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
verify: true,
|
||||||
|
});
|
||||||
|
expect(good).toMatchObject({ skipped: 1, verified: 1, mismatched: 0 });
|
||||||
|
|
||||||
|
const video = saved(outDir, "500.mov");
|
||||||
|
writeFileSync(video, "another few seconds of video");
|
||||||
|
const bad = await lib.backup({
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
verify: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(bad).toMatchObject({
|
||||||
|
downloaded: 1,
|
||||||
|
verified: 0,
|
||||||
|
mismatched: 1,
|
||||||
|
failed: 0,
|
||||||
|
});
|
||||||
|
expect(readFileSync(saved(outDir, "500.heic"))).toEqual(
|
||||||
|
Buffer.from(IMAGE),
|
||||||
|
);
|
||||||
|
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
|
||||||
|
expect(tree(outDir)).toEqual(linked);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("removes both files of a live photo that does not match its hash when downloading it again fails", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(
|
||||||
|
file(500, 10, "IMG_0500.HEIC"),
|
||||||
|
);
|
||||||
|
const bodies = new Map([[500, body]]);
|
||||||
|
const lib = await open([live], bodies);
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
writeFileSync(saved(outDir, "500.mov"), "another few seconds of video");
|
||||||
|
bodies.delete(500);
|
||||||
|
|
||||||
|
const result = await lib.backup({
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
verify: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toMatchObject({ mismatched: 1, failed: 1 });
|
||||||
|
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
|
||||||
|
expect(existsSync(saved(outDir, "500.heic"))).toBe(false);
|
||||||
|
expect(existsSync(saved(outDir, "500.mov"))).toBe(false);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
it("serves a live photo the backup stored to a library reading the backup", async () => {
|
it("serves a live photo the backup stored to a library reading the backup", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(
|
const { file: live, body } = await asLivePhoto(
|
||||||
file(500, 10, "IMG_0500.HEIC"),
|
file(500, 10, "IMG_0500.HEIC"),
|
||||||
|
|||||||
@@ -53,13 +53,14 @@ import {
|
|||||||
import { run } from "../../src/cli-run.js";
|
import { run } from "../../src/cli-run.js";
|
||||||
import { loadSession } from "../../src/cli-session.js";
|
import { loadSession } from "../../src/cli-session.js";
|
||||||
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
|
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
|
||||||
import type { ContentSource } from "../../src/library/content.js";
|
import { savePath, type ContentSource } from "../../src/library/content.js";
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||||
import { defaultCacheDirectory } from "../../src/library/index.js";
|
import { defaultCacheDirectory } from "../../src/library/index.js";
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||||
import {
|
import {
|
||||||
asLivePhoto,
|
asLivePhoto,
|
||||||
|
blake2b,
|
||||||
cdnSource,
|
cdnSource,
|
||||||
IMAGE,
|
IMAGE,
|
||||||
livePhotoHash,
|
livePhotoHash,
|
||||||
@@ -744,6 +745,61 @@ describe("backup", () => {
|
|||||||
expect(stderr.text).toBe("Starting backup...\n");
|
expect(stderr.text).toBe("Starting backup...\n");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("--verify downloads again an original that does not match its hash, prints the counts, and exits 0", async () => {
|
||||||
|
// Each file records the hash of the original the fake writes for it.
|
||||||
|
const client = {
|
||||||
|
...fakeClient(),
|
||||||
|
filesSince: async (args: { collectionID: number }) => ({
|
||||||
|
files: (FILES[args.collectionID] ?? []).map((f) => ({
|
||||||
|
...f,
|
||||||
|
metadata: {
|
||||||
|
...f.metadata,
|
||||||
|
hash: blake2b(Buffer.alloc(7, f.id & 0xff)),
|
||||||
|
},
|
||||||
|
})),
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
}),
|
||||||
|
} as unknown as Client;
|
||||||
|
const ctx = context(client);
|
||||||
|
const dir = join(root, "backup");
|
||||||
|
expect(await backupCommand(ctx, dir, {})).toBe(0);
|
||||||
|
writeFileSync(savePath(dir, FILES[1]![0]!), "corrupt");
|
||||||
|
|
||||||
|
expect(await backupCommand(ctx, dir, { verify: true })).toBe(0);
|
||||||
|
|
||||||
|
expect(stderr.text).toContain(
|
||||||
|
"MISMATCH original beach.jpg (100): its bytes do not match its content hash\n",
|
||||||
|
);
|
||||||
|
expect(stderr.text).toContain(
|
||||||
|
" Downloaded: 1\n" +
|
||||||
|
" Skipped: 2\n" +
|
||||||
|
" Verified: 2\n" +
|
||||||
|
" Mismatched: 1\n" +
|
||||||
|
" Unchecked: 0\n" +
|
||||||
|
" Failed: 0\n",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("--verify --json adds the verified, mismatched and unchecked counts", async () => {
|
||||||
|
const dir = join(root, "backup");
|
||||||
|
expect(await backupCommand(context(), dir, {})).toBe(0);
|
||||||
|
|
||||||
|
const code = await backupCommand(context(), dir, {
|
||||||
|
verify: true,
|
||||||
|
json: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(code).toBe(0);
|
||||||
|
expect(JSON.parse(stdout.text)).toMatchObject({
|
||||||
|
skipped: 3,
|
||||||
|
verified: 0,
|
||||||
|
mismatched: 0,
|
||||||
|
unchecked: 3,
|
||||||
|
failed: 0,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
it("exits 1 and lists each file when the ML data fetch fails", async () => {
|
it("exits 1 and lists each file when the ML data fetch fails", async () => {
|
||||||
const client = {
|
const client = {
|
||||||
...fakeClient(),
|
...fakeClient(),
|
||||||
|
|||||||
+2
-1
@@ -30,7 +30,8 @@ export const livePhotoZip = (
|
|||||||
},
|
},
|
||||||
): Uint8Array => zipSync(entries);
|
): Uint8Array => zipSync(entries);
|
||||||
|
|
||||||
const blake2b = (bytes: Uint8Array): string =>
|
// The content hash Ente's clients record for an original's bytes.
|
||||||
|
export const blake2b = (bytes: Uint8Array): string =>
|
||||||
createHash("blake2b512").update(bytes).digest("base64");
|
createHash("blake2b512").update(bytes).digest("base64");
|
||||||
|
|
||||||
// The hash Ente's clients record for a live photo: the unkeyed BLAKE2b-512 of
|
// The hash Ente's clients record for a live photo: the unkeyed BLAKE2b-512 of
|
||||||
|
|||||||
Reference in New Issue
Block a user