Compare commits
1
Commits
next
...
c4c2cf3af4
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c4c2cf3af4 |
@@ -600,7 +600,8 @@ the smallest does not.
|
|||||||
below)
|
below)
|
||||||
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
||||||
keeps, its update time, its private and
|
keeps, its update time, its private and
|
||||||
public magic metadata, and its ML data
|
public magic metadata, its ML data, and
|
||||||
|
its original's EXIF, XMP and dimensions
|
||||||
YYYY-MM-DD.<fileID>.livephoto.json
|
YYYY-MM-DD.<fileID>.livephoto.json
|
||||||
which of a live photo's two files is which
|
which of a live photo's two files is which
|
||||||
collections/
|
collections/
|
||||||
@@ -637,6 +638,16 @@ not in the cache gets the reason in `mlDataError` instead and counts as failed,
|
|||||||
and the next run fetches it again. The JSON files are rewritten on every run, so
|
and the next run fetches it again. The JSON files are rewritten on every run, so
|
||||||
ML data that arrived since the last run appears.
|
ML data that arrived since the last run appears.
|
||||||
|
|
||||||
|
A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
|
||||||
|
from its original, or from a live photo's image: `format`, `width` and `height`
|
||||||
|
for a JPEG, `exif` (or `exifRaw` and `exifError`), and `xmp`. An original with
|
||||||
|
none of these gets `{}`. A video gets no `imageMetadata`: reading a whole video
|
||||||
|
to look for tags is not worth it. When the original cannot be read, the reason
|
||||||
|
is in `imageMetadataError` instead; neither the file nor the run fails. The
|
||||||
|
original is read when the run stores it, or when the JSON beside it has neither
|
||||||
|
field, as one an earlier version wrote. Otherwise the field is taken from that
|
||||||
|
JSON when it is rewritten, so a run does not read every stored original again.
|
||||||
|
|
||||||
`failures.json` records each failed file with the kind of failure, how many
|
`failures.json` records each failed file with the kind of failure, how many
|
||||||
times it has been tried and when it was last tried. A file leaves it once it
|
times it has been tried and when it was last tried. A file leaves it once it
|
||||||
succeeds, or once it is no longer in the library or in the backup's scope. The
|
succeeds, or once it is no longer in the library or in the backup's scope. The
|
||||||
@@ -894,13 +905,14 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
|||||||
`fresh()` does, puts every in-scope original not already at its save path
|
`fresh()` does, puts every in-scope original not already at its save path
|
||||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
thumbnails) through the content cache, waits for an ML data fetch, and
|
||||||
rebuilds the on-disk backup tree, each file's JSON with its ML data, with a
|
rebuilds the on-disk backup tree, each file's JSON with its ML data and its
|
||||||
durable failure ledger. A fetched original is written straight to its save
|
original's EXIF, XMP and dimensions, with a durable failure ledger. A fetched
|
||||||
path and not into the cache, which then counts it as present; one the cache
|
original is written straight to its save path and not into the cache, which
|
||||||
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
|
then counts it as present; one the cache already held is copied from there.
|
||||||
back to the library's), `includeOriginals` (default `true`),
|
`BackupOptions`: `downloadDirectory` (falls back to the library's),
|
||||||
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See
|
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
||||||
Backup layout above for the tree it writes.
|
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
|
||||||
|
writes.
|
||||||
|
|
||||||
### Request pools
|
### Request pools
|
||||||
|
|
||||||
|
|||||||
@@ -25,6 +25,15 @@ declares one.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
|
||||||
|
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
|
||||||
|
(issue 167): for a live photo from its image, for a video nothing, and `{}`
|
||||||
|
for an original with none of them. A failed read puts the reason in
|
||||||
|
`imageMetadataError` and fails neither the file nor the run. An original is
|
||||||
|
read when the run stores it or when its JSON has neither field; otherwise the
|
||||||
|
field is taken from that JSON. The hand-built JPEGs moved to
|
||||||
|
`test/exif-jpeg.ts`, beside the HEIC.
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` writes the account and album records
|
- 2026-10-06: `quak backup` writes the account and album records
|
||||||
`backup-metadata` writes (issue 166): `account.json` with the account's
|
`backup-metadata` writes (issue 166): `account.json` with the account's
|
||||||
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
|
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
|
||||||
|
|||||||
+72
-12
@@ -11,7 +11,9 @@
|
|||||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||||
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
||||||
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
|
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
|
||||||
// the file's ML data
|
// the file's ML data and its
|
||||||
|
// original's EXIF, XMP and
|
||||||
|
// dimensions
|
||||||
// collections/<name>/<title> symlink to the original
|
// collections/<name>/<title> symlink to the original
|
||||||
// collections/<name>.json per-collection metadata
|
// collections/<name>.json per-collection metadata
|
||||||
// account.json the account's email and user ID
|
// account.json the account's email and user ID
|
||||||
@@ -28,7 +30,10 @@
|
|||||||
// no unique state, so they are rebuilt every run; that repairs stale sidecars
|
// no unique state, so they are rebuilt every run; that repairs stale sidecars
|
||||||
// and missing or broken symlinks left by an earlier crash. A rebuild also
|
// and missing or broken symlinks left by an earlier crash. A rebuild also
|
||||||
// removes the symlinks to originals that no longer belong to an album, and the
|
// removes the symlinks to originals that no longer belong to an album, and the
|
||||||
// directories of albums that no longer exist.
|
// directories of albums that no longer exist. The one thing a sidecar takes
|
||||||
|
// from the sidecar it replaces is its original's EXIF, XMP and dimensions (or
|
||||||
|
// why they could not be read), so that a run does not read every stored
|
||||||
|
// original again; a sidecar without them gets them read from the original.
|
||||||
//
|
//
|
||||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
// download, a failed symlink, or ML data missing because the ML data fetch
|
||||||
@@ -53,6 +58,7 @@ import {
|
|||||||
symlinkSync,
|
symlinkSync,
|
||||||
writeFileSync,
|
writeFileSync,
|
||||||
} from "node:fs";
|
} from "node:fs";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||||
|
|
||||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||||
@@ -64,6 +70,7 @@ import {
|
|||||||
storedAtSavePath,
|
storedAtSavePath,
|
||||||
} from "./library/content.js";
|
} from "./library/content.js";
|
||||||
import { representative } from "./library/records.js";
|
import { representative } from "./library/records.js";
|
||||||
|
import { extractImageMetadata } from "./metadata-backup.js";
|
||||||
import type { MLData } from "./mldata-fetch.js";
|
import type { MLData } from "./mldata-fetch.js";
|
||||||
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
||||||
|
|
||||||
@@ -354,12 +361,53 @@ const saveLedger = (path: string, ledger: Map<number, FailureEntry>): void => {
|
|||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
// The file's JSON: its basic fields, its magic metadata, and its ML data, or
|
// A file's EXIF, XMP and dimensions as its JSON holds them: what
|
||||||
// the reason the ML data is missing.
|
// `extractImageMetadata` found in its original, or why the original could not
|
||||||
|
// be read.
|
||||||
|
interface ImageMetadata {
|
||||||
|
imageMetadata?: Record<string, unknown>;
|
||||||
|
imageMetadataError?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The image metadata for the file whose original is at `originalPath` (for a
|
||||||
|
// live photo, its image) and whose JSON is at `jsonPath`. A video gets none,
|
||||||
|
// as `photo.exif()` reads none. An original stored before this run is not read
|
||||||
|
// again when its JSON already holds image metadata: that is kept. A failed
|
||||||
|
// read gives the reason, and fails neither the file nor the run. An original
|
||||||
|
// with no EXIF, XMP or JPEG dimensions gets `{}`, so it is not read again.
|
||||||
|
const imageMetadataFor = async (
|
||||||
|
file: EnteFile,
|
||||||
|
originalPath: string,
|
||||||
|
jsonPath: string,
|
||||||
|
storedThisRun: boolean,
|
||||||
|
): Promise<ImageMetadata> => {
|
||||||
|
if (file.metadata.fileType === "video") return {};
|
||||||
|
if (!storedThisRun) {
|
||||||
|
try {
|
||||||
|
const { imageMetadata, imageMetadataError } = JSON.parse(
|
||||||
|
readFileSync(jsonPath, "utf-8"),
|
||||||
|
) as ImageMetadata;
|
||||||
|
if (imageMetadata !== undefined || imageMetadataError !== undefined)
|
||||||
|
return { imageMetadata, imageMetadataError };
|
||||||
|
} catch {
|
||||||
|
// No JSON yet, or one that cannot be parsed: read the original.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const bytes = await readFile(originalPath);
|
||||||
|
return { imageMetadata: extractImageMetadata(bytes) ?? {} };
|
||||||
|
} catch (err) {
|
||||||
|
return { imageMetadataError: errorMessage(err) };
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// The file's JSON: its basic fields, its magic metadata, its ML data or the
|
||||||
|
// reason the ML data is missing, and its image metadata.
|
||||||
const writeSidecar = (
|
const writeSidecar = (
|
||||||
path: string,
|
path: string,
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
ml: { mlData?: MLData; mlDataError?: string },
|
ml: { mlData?: MLData; mlDataError?: string },
|
||||||
|
image: ImageMetadata,
|
||||||
): void => {
|
): void => {
|
||||||
const meta: Record<string, unknown> = {
|
const meta: Record<string, unknown> = {
|
||||||
id: file.id,
|
id: file.id,
|
||||||
@@ -372,6 +420,10 @@ const writeSidecar = (
|
|||||||
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
||||||
if (ml.mlData) meta.mlData = ml.mlData;
|
if (ml.mlData) meta.mlData = ml.mlData;
|
||||||
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
|
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
|
||||||
|
if (image.imageMetadata) meta.imageMetadata = image.imageMetadata;
|
||||||
|
if (image.imageMetadataError) {
|
||||||
|
meta.imageMetadataError = image.imageMetadataError;
|
||||||
|
}
|
||||||
writeFileSync(path, JSON.stringify(meta, null, 2));
|
writeFileSync(path, JSON.stringify(meta, null, 2));
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -468,6 +520,7 @@ export const runBackup = async (
|
|||||||
|
|
||||||
const errors: BackupError[] = [];
|
const errors: BackupError[] = [];
|
||||||
const failedThisRun = new Set<number>();
|
const failedThisRun = new Set<number>();
|
||||||
|
const storedThisRun = new Set<number>();
|
||||||
let downloaded = 0;
|
let downloaded = 0;
|
||||||
let skipped = 0;
|
let skipped = 0;
|
||||||
|
|
||||||
@@ -516,6 +569,7 @@ export const runBackup = async (
|
|||||||
await placeOriginal(downloadDirectory, file, (dest) =>
|
await placeOriginal(downloadDirectory, file, (dest) =>
|
||||||
lib.original(fileID, dest),
|
lib.original(fileID, dest),
|
||||||
);
|
);
|
||||||
|
storedThisRun.add(fileID);
|
||||||
downloaded++;
|
downloaded++;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(
|
log(
|
||||||
@@ -549,9 +603,10 @@ export const runBackup = async (
|
|||||||
|
|
||||||
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
||||||
// every present original (this repairs stale ones), each with the file's
|
// every present original (this repairs stale ones), each with the file's
|
||||||
// ML data once an ML data fetch has completed. When the fetch fails, a
|
// ML data once an ML data fetch has completed, and its image metadata.
|
||||||
// file with no cached ML data gets the reason instead and is recorded as
|
// When the fetch fails, a file with no cached ML data gets the reason
|
||||||
// failed. The next run fetches its ML data again because none is cached.
|
// instead and is recorded as failed. The next run fetches its ML data
|
||||||
|
// again because none is cached.
|
||||||
if (includeOriginals) {
|
if (includeOriginals) {
|
||||||
let mlDataError: string | undefined;
|
let mlDataError: string | undefined;
|
||||||
try {
|
try {
|
||||||
@@ -562,23 +617,28 @@ export const runBackup = async (
|
|||||||
log(`FAILED ML data: ${mlDataError}`);
|
log(`FAILED ML data: ${mlDataError}`);
|
||||||
}
|
}
|
||||||
for (const file of distinct.values()) {
|
for (const file of distinct.values()) {
|
||||||
if (storedAtSavePath(downloadDirectory, file) === undefined) {
|
const stored = storedAtSavePath(downloadDirectory, file);
|
||||||
continue;
|
if (stored === undefined) continue;
|
||||||
}
|
|
||||||
const path = withExtension(
|
const path = withExtension(
|
||||||
savePath(downloadDirectory, file),
|
savePath(downloadDirectory, file),
|
||||||
".json",
|
".json",
|
||||||
);
|
);
|
||||||
|
const image = await imageMetadataFor(
|
||||||
|
file,
|
||||||
|
stored.path,
|
||||||
|
path,
|
||||||
|
storedThisRun.has(file.id),
|
||||||
|
);
|
||||||
const mlData = await lib.mlData(file.id);
|
const mlData = await lib.mlData(file.id);
|
||||||
if (mlData === undefined && mlDataError !== undefined) {
|
if (mlData === undefined && mlDataError !== undefined) {
|
||||||
writeSidecar(path, file, { mlDataError });
|
writeSidecar(path, file, { mlDataError }, image);
|
||||||
recordFailure(
|
recordFailure(
|
||||||
file,
|
file,
|
||||||
collectionName.get(file.collectionID) ?? "",
|
collectionName.get(file.collectionID) ?? "",
|
||||||
new Error(`ML data: ${mlDataError}`),
|
new Error(`ML data: ${mlDataError}`),
|
||||||
);
|
);
|
||||||
} else {
|
} else {
|
||||||
writeSidecar(path, file, { mlData });
|
writeSidecar(path, file, { mlData }, image);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -33,6 +33,7 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import {
|
import {
|
||||||
|
chmodSync,
|
||||||
existsSync,
|
existsSync,
|
||||||
lstatSync,
|
lstatSync,
|
||||||
mkdirSync,
|
mkdirSync,
|
||||||
@@ -53,12 +54,16 @@ import { runBackup, type BackupLibrary } from "../../src/backup.js";
|
|||||||
import { Library } from "../../src/library/index.js";
|
import { Library } from "../../src/library/index.js";
|
||||||
import type { ContentSource } from "../../src/library/content.js";
|
import type { ContentSource } from "../../src/library/content.js";
|
||||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||||
|
import { extractImageMetadata } from "../../src/metadata-backup.js";
|
||||||
import type { MLData } from "../../src/mldata-fetch.js";
|
import type { MLData } from "../../src/mldata-fetch.js";
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
|
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||||
|
import { JPEG_WITH_EXIF } from "../exif-jpeg.js";
|
||||||
import {
|
import {
|
||||||
asLivePhoto,
|
asLivePhoto,
|
||||||
cdnSource,
|
cdnSource,
|
||||||
IMAGE,
|
IMAGE,
|
||||||
|
livePhotoHash,
|
||||||
livePhotoZip,
|
livePhotoZip,
|
||||||
VIDEO,
|
VIDEO,
|
||||||
} from "../live-photo.js";
|
} from "../live-photo.js";
|
||||||
@@ -1018,6 +1023,179 @@ describe("account and album records", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// What a file's JSON holds as `imageMetadata` for an original of `bytes`.
|
||||||
|
const imageMetadataOf = (bytes: Uint8Array): unknown =>
|
||||||
|
JSON.parse(JSON.stringify(extractImageMetadata(bytes)));
|
||||||
|
|
||||||
|
describe("image metadata in each file's JSON", () => {
|
||||||
|
// Serves `files` in the Vacation album and nothing in Work.
|
||||||
|
class FilesClient extends MockClient {
|
||||||
|
constructor(private readonly files: EnteFile[]) {
|
||||||
|
super();
|
||||||
|
}
|
||||||
|
override async filesSince(args: {
|
||||||
|
collectionID: number;
|
||||||
|
}): Promise<FilesPage> {
|
||||||
|
return {
|
||||||
|
files: args.collectionID === 1 ? this.files : [],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Writes `originals.get(fileID)` as each file's original, looked up at
|
||||||
|
// each fetch, so a test can change it between runs.
|
||||||
|
const bytesSource = (
|
||||||
|
originals: Map<number, Uint8Array>,
|
||||||
|
): ContentSource => ({
|
||||||
|
original: async ({ file: f, destination }) => {
|
||||||
|
const bytes = originals.get(f.id)!;
|
||||||
|
writeFileSync(destination, bytes);
|
||||||
|
return { bytesWritten: bytes.length };
|
||||||
|
},
|
||||||
|
thumbnail: async () => {
|
||||||
|
throw new Error("no thumbnails in this source");
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
// A JPEG, a HEIC, a video, and a file holding no image metadata. The
|
||||||
|
// video's bytes are the JPEG's, so reading it would find EXIF.
|
||||||
|
const clip = file(602, 1, "clip.mov");
|
||||||
|
const files = [
|
||||||
|
file(600, 1, "photo.jpg"),
|
||||||
|
file(601, 1, "photo.heic"),
|
||||||
|
{ ...clip, metadata: { ...clip.metadata, fileType: "video" as const } },
|
||||||
|
file(603, 1, "notes.png"),
|
||||||
|
];
|
||||||
|
const originals = (): Map<number, Uint8Array> =>
|
||||||
|
new Map([
|
||||||
|
[600, JPEG_WITH_EXIF],
|
||||||
|
[601, HEIC_WITH_EXIF],
|
||||||
|
[602, JPEG_WITH_EXIF],
|
||||||
|
[603, new TextEncoder().encode("not an image")],
|
||||||
|
]);
|
||||||
|
const open = (bytes = originals()): Promise<Library> =>
|
||||||
|
openLibrary(bytesSource(bytes), new FilesClient(files));
|
||||||
|
|
||||||
|
// Rewrite the JSON of `fileID` without its image metadata, as an earlier
|
||||||
|
// version of quak wrote it.
|
||||||
|
const dropImageMetadata = (outDir: string, fileID: number): void => {
|
||||||
|
const json = fileJSON(outDir, fileID);
|
||||||
|
delete json.imageMetadata;
|
||||||
|
writeFileSync(saved(outDir, `${fileID}.json`), JSON.stringify(json));
|
||||||
|
};
|
||||||
|
|
||||||
|
it("writes each new original's image metadata, and none for a video", async () => {
|
||||||
|
const lib = await open();
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
|
||||||
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(result).toMatchObject({ downloaded: 4, failed: 0 });
|
||||||
|
expect(fileJSON(outDir, 600).imageMetadata).toEqual(
|
||||||
|
imageMetadataOf(JPEG_WITH_EXIF),
|
||||||
|
);
|
||||||
|
expect(fileJSON(outDir, 601).imageMetadata).toEqual(
|
||||||
|
imageMetadataOf(HEIC_WITH_EXIF),
|
||||||
|
);
|
||||||
|
expect(fileJSON(outDir, 602)).not.toHaveProperty("imageMetadata");
|
||||||
|
expect(fileJSON(outDir, 602)).not.toHaveProperty("imageMetadataError");
|
||||||
|
// Empty, so that the next run does not read it again.
|
||||||
|
expect(fileJSON(outDir, 603).imageMetadata).toEqual({});
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reads a live photo's image", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(
|
||||||
|
file(500, 1, "IMG_0500.HEIC"),
|
||||||
|
livePhotoZip({ "image.heic": HEIC_WITH_EXIF, "video.mov": VIDEO }),
|
||||||
|
livePhotoHash(HEIC_WITH_EXIF, VIDEO),
|
||||||
|
);
|
||||||
|
const lib = await openLibrary(
|
||||||
|
cdnSource(new Map([[500, body]])),
|
||||||
|
new FilesClient([live]),
|
||||||
|
);
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(fileJSON(outDir, 500).imageMetadata).toEqual(
|
||||||
|
imageMetadataOf(HEIC_WITH_EXIF),
|
||||||
|
);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps the image metadata of an original stored by an earlier run without reading it again", async () => {
|
||||||
|
const lib = await open();
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
// A read of the original now would give the HEIC's.
|
||||||
|
writeFileSync(saved(outDir, "600.jpg"), HEIC_WITH_EXIF);
|
||||||
|
|
||||||
|
const second = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(second).toMatchObject({ downloaded: 0, failed: 0 });
|
||||||
|
expect(fileJSON(outDir, 600).imageMetadata).toEqual(
|
||||||
|
imageMetadataOf(JPEG_WITH_EXIF),
|
||||||
|
);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reads an original stored by an earlier version, whose JSON has no image metadata", async () => {
|
||||||
|
const lib = await open();
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
dropImageMetadata(outDir, 600);
|
||||||
|
|
||||||
|
const second = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(second).toMatchObject({ downloaded: 0, failed: 0 });
|
||||||
|
expect(fileJSON(outDir, 600).imageMetadata).toEqual(
|
||||||
|
imageMetadataOf(JPEG_WITH_EXIF),
|
||||||
|
);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reads an original the run stores again, though its JSON has image metadata", async () => {
|
||||||
|
const bytes = originals();
|
||||||
|
const lib = await open(bytes);
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
rmSync(saved(outDir, "600.jpg"));
|
||||||
|
bytes.set(600, HEIC_WITH_EXIF);
|
||||||
|
|
||||||
|
const second = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(second).toMatchObject({ downloaded: 1, failed: 0 });
|
||||||
|
expect(fileJSON(outDir, 600).imageMetadata).toEqual(
|
||||||
|
imageMetadataOf(HEIC_WITH_EXIF),
|
||||||
|
);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Root ignores file permissions, so this fails when run as root. The
|
||||||
|
// `test` phase of the `Dockerfile` runs as the `node` user.
|
||||||
|
it("gives the reason an original could not be read, failing neither the file nor the run", async () => {
|
||||||
|
const lib = await open();
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
dropImageMetadata(outDir, 600);
|
||||||
|
const original = saved(outDir, "600.jpg");
|
||||||
|
chmodSync(original, 0o000);
|
||||||
|
|
||||||
|
const second = await lib
|
||||||
|
.backup({ downloadDirectory: outDir })
|
||||||
|
.finally(() => chmodSync(original, 0o600));
|
||||||
|
|
||||||
|
expect(second).toMatchObject({ failed: 0, errors: [] });
|
||||||
|
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
|
||||||
|
expect(fileJSON(outDir, 600)).not.toHaveProperty("imageMetadata");
|
||||||
|
expect(fileJSON(outDir, 600).imageMetadataError).toMatch(/EACCES/);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
// Every entry under collections/, one level of directories deep, with each
|
// Every entry under collections/, one level of directories deep, with each
|
||||||
// symlink's target.
|
// symlink's target.
|
||||||
const tree = (outDir: string): string[] => {
|
const tree = (outDir: string): string[] => {
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
/**
|
||||||
|
* Hand-built JPEGs for the tests of `exif()` and of the image metadata
|
||||||
|
* `quak backup` records: one whose EXIF holds the same values as `exif.heic`
|
||||||
|
* (see `exif-heic.ts`), and one whose EXIF cannot be parsed.
|
||||||
|
*/
|
||||||
|
|
||||||
|
// Big-endian bytes for the hand-built JPEG below.
|
||||||
|
const u16 = (n: number): number[] => [n >> 8, n & 0xff];
|
||||||
|
const u32 = (n: number): number[] => [...u16(n >>> 16), ...u16(n & 0xffff)];
|
||||||
|
const ascii = (s: string): number[] => [...new TextEncoder().encode(s), 0];
|
||||||
|
const rational = (num: number, den: number): number[] => [
|
||||||
|
...u32(num),
|
||||||
|
...u32(den),
|
||||||
|
];
|
||||||
|
// One IFD entry: tag, type (1 BYTE, 2 ASCII, 3 SHORT, 4 LONG, 5 RATIONAL),
|
||||||
|
// count, then the value when it fits in 4 bytes, else its offset.
|
||||||
|
const entry = (
|
||||||
|
tag: number,
|
||||||
|
type: number,
|
||||||
|
count: number,
|
||||||
|
value: number[],
|
||||||
|
): number[] => [...u16(tag), ...u16(type), ...u32(count), ...value];
|
||||||
|
|
||||||
|
// The TIFF block of a JPEG's EXIF segment, holding every field `Photo`'s typed
|
||||||
|
// methods return: the camera in the first IFD, the exposure in the Exif IFD,
|
||||||
|
// and a GPS position of 40°26'46" N, 79°58'56" W, 12.5 m below sea level.
|
||||||
|
// Offsets count from the start of this block.
|
||||||
|
const TIFF = [
|
||||||
|
...[0x4d, 0x4d, 0x00, 0x2a], // big-endian TIFF
|
||||||
|
...u32(8), // the first IFD's offset
|
||||||
|
// The first IFD, at 8: five entries, then no next IFD.
|
||||||
|
...u16(5),
|
||||||
|
...entry(0x010f, 2, 6, u32(74)), // Make
|
||||||
|
...entry(0x0110, 2, 7, u32(80)), // Model
|
||||||
|
...entry(0x0112, 3, 1, [...u16(6), 0, 0]), // Orientation
|
||||||
|
...entry(0x8769, 4, 1, u32(88)), // the Exif IFD's offset
|
||||||
|
...entry(0x8825, 4, 1, u32(246)), // the GPS IFD's offset
|
||||||
|
...u32(0),
|
||||||
|
...ascii("Canon"), // at 74
|
||||||
|
...ascii("EOS R5"), // at 80
|
||||||
|
0, // a pad byte
|
||||||
|
// The Exif IFD, at 88: seven entries, then no next IFD.
|
||||||
|
...u16(7),
|
||||||
|
...entry(0x829a, 5, 1, u32(178)), // ExposureTime
|
||||||
|
...entry(0x829d, 5, 1, u32(186)), // FNumber
|
||||||
|
...entry(0x8827, 3, 1, [...u16(400), 0, 0]), // ISOSpeedRatings
|
||||||
|
...entry(0x9003, 2, 20, u32(194)), // DateTimeOriginal
|
||||||
|
...entry(0x9011, 2, 7, u32(214)), // OffsetTimeOriginal
|
||||||
|
...entry(0x920a, 5, 1, u32(222)), // FocalLength
|
||||||
|
...entry(0xa434, 2, 16, u32(230)), // LensModel
|
||||||
|
...u32(0),
|
||||||
|
...rational(1, 250), // at 178
|
||||||
|
...rational(28, 10), // at 186
|
||||||
|
...ascii("2021:07:15 14:30:00"), // at 194
|
||||||
|
...ascii("+02:00"), // at 214
|
||||||
|
0, // a pad byte
|
||||||
|
...rational(50, 1), // at 222
|
||||||
|
...ascii("RF50mm F1.8 STM"), // at 230
|
||||||
|
// The GPS IFD, at 246: six entries, then no next IFD.
|
||||||
|
...u16(6),
|
||||||
|
...entry(0x0001, 2, 2, [...ascii("N"), 0, 0]), // GPSLatitudeRef
|
||||||
|
...entry(0x0002, 5, 3, u32(324)), // GPSLatitude
|
||||||
|
...entry(0x0003, 2, 2, [...ascii("W"), 0, 0]), // GPSLongitudeRef
|
||||||
|
...entry(0x0004, 5, 3, u32(348)), // GPSLongitude
|
||||||
|
...entry(0x0005, 1, 1, [1, 0, 0, 0]), // GPSAltitudeRef: below sea level
|
||||||
|
...entry(0x0006, 5, 1, u32(372)), // GPSAltitude
|
||||||
|
...u32(0),
|
||||||
|
...[...rational(40, 1), ...rational(26, 1), ...rational(46, 1)], // at 324
|
||||||
|
...[...rational(79, 1), ...rational(58, 1), ...rational(56, 1)], // at 348
|
||||||
|
...rational(25, 2), // at 372
|
||||||
|
];
|
||||||
|
|
||||||
|
export const JPEG_WITH_EXIF = new Uint8Array([
|
||||||
|
...[0xff, 0xd8], // start of image
|
||||||
|
...[0xff, 0xe1, ...u16(2 + 6 + TIFF.length)], // APP1 and its length
|
||||||
|
...[...ascii("Exif"), 0], // "Exif\0\0"
|
||||||
|
...TIFF,
|
||||||
|
...[0xff, 0xda, 0x00, 0x02], // start of scan
|
||||||
|
]);
|
||||||
|
|
||||||
|
// A JPEG whose EXIF segment is laid out correctly but holds "XX" where the TIFF
|
||||||
|
// byte order belongs, so exifreader cannot parse it.
|
||||||
|
export const JPEG_WITH_BAD_EXIF = new Uint8Array([
|
||||||
|
...[0xff, 0xd8], // start of image
|
||||||
|
...[0xff, 0xe1, ...u16(2 + 6 + 2)], // APP1 and its length
|
||||||
|
...[...ascii("Exif"), 0], // "Exif\0\0"
|
||||||
|
...[0x58, 0x58], // "XX"
|
||||||
|
...[0xff, 0xda, 0x00, 0x02], // start of scan
|
||||||
|
]);
|
||||||
@@ -29,6 +29,7 @@ import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
|||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
import { readPhotoExif, type PhotoExif } from "../../src/exif.js";
|
import { readPhotoExif, type PhotoExif } from "../../src/exif.js";
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||||
|
import { JPEG_WITH_BAD_EXIF, JPEG_WITH_EXIF } from "../exif-jpeg.js";
|
||||||
import {
|
import {
|
||||||
asLivePhoto,
|
asLivePhoto,
|
||||||
cdnSource,
|
cdnSource,
|
||||||
@@ -256,90 +257,6 @@ describe("Library content wiring", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// Big-endian bytes for the hand-built JPEG below.
|
|
||||||
const u16 = (n: number): number[] => [n >> 8, n & 0xff];
|
|
||||||
const u32 = (n: number): number[] => [...u16(n >>> 16), ...u16(n & 0xffff)];
|
|
||||||
const ascii = (s: string): number[] => [...new TextEncoder().encode(s), 0];
|
|
||||||
const rational = (num: number, den: number): number[] => [
|
|
||||||
...u32(num),
|
|
||||||
...u32(den),
|
|
||||||
];
|
|
||||||
// One IFD entry: tag, type (1 BYTE, 2 ASCII, 3 SHORT, 4 LONG, 5 RATIONAL),
|
|
||||||
// count, then the value when it fits in 4 bytes, else its offset.
|
|
||||||
const entry = (
|
|
||||||
tag: number,
|
|
||||||
type: number,
|
|
||||||
count: number,
|
|
||||||
value: number[],
|
|
||||||
): number[] => [...u16(tag), ...u16(type), ...u32(count), ...value];
|
|
||||||
|
|
||||||
// The TIFF block of a JPEG's EXIF segment, holding every field `Photo`'s typed
|
|
||||||
// methods return: the camera in the first IFD, the exposure in the Exif IFD,
|
|
||||||
// and a GPS position of 40°26'46" N, 79°58'56" W, 12.5 m below sea level.
|
|
||||||
// Offsets count from the start of this block.
|
|
||||||
const TIFF = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a], // big-endian TIFF
|
|
||||||
...u32(8), // the first IFD's offset
|
|
||||||
// The first IFD, at 8: five entries, then no next IFD.
|
|
||||||
...u16(5),
|
|
||||||
...entry(0x010f, 2, 6, u32(74)), // Make
|
|
||||||
...entry(0x0110, 2, 7, u32(80)), // Model
|
|
||||||
...entry(0x0112, 3, 1, [...u16(6), 0, 0]), // Orientation
|
|
||||||
...entry(0x8769, 4, 1, u32(88)), // the Exif IFD's offset
|
|
||||||
...entry(0x8825, 4, 1, u32(246)), // the GPS IFD's offset
|
|
||||||
...u32(0),
|
|
||||||
...ascii("Canon"), // at 74
|
|
||||||
...ascii("EOS R5"), // at 80
|
|
||||||
0, // a pad byte
|
|
||||||
// The Exif IFD, at 88: seven entries, then no next IFD.
|
|
||||||
...u16(7),
|
|
||||||
...entry(0x829a, 5, 1, u32(178)), // ExposureTime
|
|
||||||
...entry(0x829d, 5, 1, u32(186)), // FNumber
|
|
||||||
...entry(0x8827, 3, 1, [...u16(400), 0, 0]), // ISOSpeedRatings
|
|
||||||
...entry(0x9003, 2, 20, u32(194)), // DateTimeOriginal
|
|
||||||
...entry(0x9011, 2, 7, u32(214)), // OffsetTimeOriginal
|
|
||||||
...entry(0x920a, 5, 1, u32(222)), // FocalLength
|
|
||||||
...entry(0xa434, 2, 16, u32(230)), // LensModel
|
|
||||||
...u32(0),
|
|
||||||
...rational(1, 250), // at 178
|
|
||||||
...rational(28, 10), // at 186
|
|
||||||
...ascii("2021:07:15 14:30:00"), // at 194
|
|
||||||
...ascii("+02:00"), // at 214
|
|
||||||
0, // a pad byte
|
|
||||||
...rational(50, 1), // at 222
|
|
||||||
...ascii("RF50mm F1.8 STM"), // at 230
|
|
||||||
// The GPS IFD, at 246: six entries, then no next IFD.
|
|
||||||
...u16(6),
|
|
||||||
...entry(0x0001, 2, 2, [...ascii("N"), 0, 0]), // GPSLatitudeRef
|
|
||||||
...entry(0x0002, 5, 3, u32(324)), // GPSLatitude
|
|
||||||
...entry(0x0003, 2, 2, [...ascii("W"), 0, 0]), // GPSLongitudeRef
|
|
||||||
...entry(0x0004, 5, 3, u32(348)), // GPSLongitude
|
|
||||||
...entry(0x0005, 1, 1, [1, 0, 0, 0]), // GPSAltitudeRef: below sea level
|
|
||||||
...entry(0x0006, 5, 1, u32(372)), // GPSAltitude
|
|
||||||
...u32(0),
|
|
||||||
...[...rational(40, 1), ...rational(26, 1), ...rational(46, 1)], // at 324
|
|
||||||
...[...rational(79, 1), ...rational(58, 1), ...rational(56, 1)], // at 348
|
|
||||||
...rational(25, 2), // at 372
|
|
||||||
];
|
|
||||||
|
|
||||||
const JPEG_WITH_EXIF = new Uint8Array([
|
|
||||||
...[0xff, 0xd8], // start of image
|
|
||||||
...[0xff, 0xe1, ...u16(2 + 6 + TIFF.length)], // APP1 and its length
|
|
||||||
...[...ascii("Exif"), 0], // "Exif\0\0"
|
|
||||||
...TIFF,
|
|
||||||
...[0xff, 0xda, 0x00, 0x02], // start of scan
|
|
||||||
]);
|
|
||||||
|
|
||||||
// A JPEG whose EXIF segment is laid out correctly but holds "XX" where the TIFF
|
|
||||||
// byte order belongs, so exifreader cannot parse it.
|
|
||||||
const JPEG_WITH_BAD_EXIF = new Uint8Array([
|
|
||||||
...[0xff, 0xd8], // start of image
|
|
||||||
...[0xff, 0xe1, ...u16(2 + 6 + 2)], // APP1 and its length
|
|
||||||
...[...ascii("Exif"), 0], // "Exif\0\0"
|
|
||||||
...[0x58, 0x58], // "XX"
|
|
||||||
...[0xff, 0xda, 0x00, 0x02], // start of scan
|
|
||||||
]);
|
|
||||||
|
|
||||||
describe("Photo save path, local copy, content and EXIF", () => {
|
describe("Photo save path, local copy, content and EXIF", () => {
|
||||||
// The same account, with `files` in its album instead.
|
// The same account, with `files` in its album instead.
|
||||||
class FilesClient extends MockClient {
|
class FilesClient extends MockClient {
|
||||||
|
|||||||
Reference in New Issue
Block a user