sneak/quak
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions 1 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/quak:1ede66e5c0e9fec4a7dcd61dd0a03bf48f6a8ae5
Branches Tags
sneak/quak:main sneak/quak:next
..
compare: sneak/quak:4986ebd889500221fcbd82a0d796b3807e75513c
Branches Tags
sneak/quak:next sneak/quak:main
2 Commits
1ede66e5c0 .. 4986ebd889
Author SHA1 Message Date
sneak 4986ebd889 Photo implements one method per PhotoExif field
check / check (push) Successful in 1m46s
Details 
Photo now implements a mapped type with one method per PhotoExif field,
each taking exif()'s options and giving that field's type, so the
build's type check fails when PhotoExif has a field Photo has no method
for, whatever the test fixtures hold. The test comment and TODO.md say
so.

Model: opus-5-5
 2026-10-01 21:11:05 +00:00
sneak b669df220a Photo: one async method per EXIF field (closes #148)
check / check (push) Successful in 1m46s
Details 
Thirteen methods on Photo, make() through gpsAltitude(), each named after
its PhotoExif field and typed by it. Each calls exif() with the same opts
and returns that one field, or undefined when the file lacks it. A test
checks, on the JPEG fixture and on test/exif.heic, that every field
exif() returns has a method giving the same value. README and TODO.md
list them.

Model: opus-5-5
 2026-10-01 20:37:02 +00:00
16 changed files with 497 additions and 972 deletions
Expand all files Collapse all files
README.md
+50 -71
View File
@@ -528,36 +528,25 @@ the smallest does not.
``` ```
<dir>/ <dir>/
YYYY/YYYY-MM/YYYY-MM-DD/ originals/
YYYY-MM-DD.<fileID>.<ext> actual file content, at its save path (one <fileID>.<ext> actual file content (one per unique file,
per unique file, two for a live photo: see two for a live photo: see below)
below) <fileID>.json the file's basic metadata fields quak
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
keeps, and its private and public magic keeps, and its private and public magic
metadata metadata
YYYY-MM-DD.<fileID>.livephoto.json <fileID>.livephoto.json which of a live photo's two files is which
which of a live photo's two files is which
collections/ collections/
<name>/ <name>/
<title> -> ../../YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext> <title> -> ../../originals/<fileID>.<ext> (symlink)
(symlink)
<name>.json collection metadata + file list <name>.json collection metadata + file list
failures.json files that failed and have not yet succeeded failures.json files that failed and have not yet succeeded
``` ```
Each original is saved at its save path, the same path `photo.savePath` gives
and `photo.download()` writes (see Read surface below). The date is the photo's
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
time zone of the machine running quak. The extension is the one in the file's
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
letters and digits. When the date or the time zone changes, the next run saves
the original at its new path and leaves the old copy where it is.
`failures.json` records each failed file with the kind of failure, how many `failures.json` records each failed file with the kind of failure, how many
times it has been tried and when it was last tried. A file leaves it once it times it has been tried and when it was last tried. A file leaves it once it
succeeds, or once it is no longer in the library or in the backup's scope. The succeeds, or once it is no longer in the library or in the backup's scope. The
library's `lib.backup({ includeThumbnails: true })` also writes library's `lib.backup({ includeThumbnails: true })` also writes
`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not. `thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
A collection's directory and JSON are named after the collection, and a symlink A collection's directory and JSON are named after the collection, and a symlink
after the file's title, both with unsafe characters replaced. When two after the file's title, both with unsafe characters replaced. When two
@@ -569,21 +558,22 @@ name stays the same from run to run until such a clash appears or goes away.
A live photo, which Ente stores as one ZIP of its image and its video, is stored A live photo, which Ente stores as one ZIP of its image and its video, is stored
as those two files, which a photo viewer can open: each is as those two files, which a photo viewer can open: each is
`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP (for `originals/<fileID>.<ext>` with the extension it has inside the ZIP (for example
example `2026-03-01.12345.heic` and `2026-03-01.12345.mov`), and `12345.heic` and `12345.mov`), and `<fileID>.livephoto.json` names the two. The
`YYYY-MM-DD.<fileID>.livephoto.json` names the two. The live photo counts as live photo counts as stored only when both files are present and not empty. Its
stored only when both files are present and not empty. Its album folder links album folder links both, each named after the title with that file's extension
both, each named after the title with that file's extension (`IMG_0001.heic` and (`IMG_0001.heic` and `IMG_0001.mov`). A live photo that an earlier version of
`IMG_0001.mov`). quak stored as the ZIP, under the image's name, is replaced by its two files on
the next run, and the ZIP and its link are removed.
Each run removes the symlinks into the date folders that no longer belong in Each run removes the symlinks into `originals/` that no longer belong in their
their collection's directory, and the directories (and JSON) of collections that collection's directory, and the directories (and JSON) of collections that were
were deleted or renamed. Nothing else in `collections/` is touched: a file or a deleted or renamed. Nothing else in `collections/` is touched: a file or a
symlink you put there stays, and a directory that still holds one after its symlink you put there stays, and a directory that still holds one after its
symlinks are removed stays too, with its JSON. symlinks are removed stays too, with its JSON.
Each file is downloaded exactly once regardless of how many collections it Each file is downloaded exactly once regardless of how many collections it
appears in, and written once: straight to its save path, with no copy left in appears in, and written once: straight into `originals/`, with no copy left in
the cache. An original the cache already held is copied from there instead. On the cache. An original the cache already held is copied from there instead. On
subsequent runs, existing originals are skipped. If a download fails, the error subsequent runs, existing originals are skipped. If a download fails, the error
is logged and the backup continues with the next file. The exit code is non-zero is logged and the backup continues with the next file. The exit code is non-zero
@@ -596,15 +586,14 @@ Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even disk, and renamed into place, so an original is either complete or absent, even
after a power cut. A downloaded original's temporary file is named after a power cut. A downloaded original's temporary file is named
`.quak-<pid>-<random>.tmp`, one copied from the cache `.quak-<pid>-<random>.tmp`, one copied from the cache
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is `.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
killed can leave one of these temporary files behind; the next backup deletes one of these temporary files behind; the next backup deletes those whose process
those whose process is no longer running. The content cache uses the same is no longer running. The content cache uses the same scheme, and opening a
scheme, and opening a library deletes the temporary files in the cache whose library deletes the temporary files in the cache whose process is no longer
process is no longer running, so a download another process has in progress in running, so a download another process has in progress in the same cache is left
the same cache is left alone. The rename replaces whatever was at the alone. The rename replaces whatever was at the destination rather than writing
destination rather than writing through it: a symlink there is replaced, not through it: a symlink there is replaced, not followed, and the new file has the
followed, and the new file has the temporary file's permissions, not those of temporary file's permissions, not those of the file it replaced.
the file it replaced.
## TODO ## TODO
@@ -647,10 +636,10 @@ background, so an unreachable server does not block opening.
`LibraryOptions`: `LibraryOptions`:
| Option | Default | Meaning | | Option | Default | Meaning |
| ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------- | | ------------------------ | -------------------------------------------------- | --------------------------------------------------------------------- |
| `client` | required | the account client (a `Client`, or any `LibraryClient`) | | `client` | required | the account client (a `Client`, or any `LibraryClient`) |
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live | | `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached | | `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
| `refreshIntervalSeconds` | `3` | background refresh cadence | | `refreshIntervalSeconds` | `3` | background refresh cadence |
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first | | `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals | | `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
@@ -662,9 +651,6 @@ background, so an unreachable server does not block opening.
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) | | `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
| `contentSource` | the client's own | override the byte source (mainly for tests) | | `contentSource` | the client's own | override the byte source (mainly for tests) |
The default `downloadDirectory` is resolved against the working directory once,
when the library opens; `lib.downloadDirectory` holds the result.
Concurrency is set through `pools`: construct Concurrency is set through `pools`: construct
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` `new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below). and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
@@ -716,26 +702,20 @@ and `originalPath`, and `photo.year`, the local-time year of `takenAt`, all as
synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more
synchronous getters look at the disk and never touch the network: synchronous getters look at the disk and never touch the network:
- `photo.savePath` → `string` — where `photo.download()` and `lib.backup()` put - `photo.savePath` → `string | undefined` — where `lib.backup()` writes the
the original, whether or not it is there yet: original, `originals/<fileID>.<ext>` under the `downloadDirectory` the library
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's was opened with, whether or not it is there yet: for a live photo already
`downloadDirectory` (see Backup layout above for the date and the extension). stored, its image. For a live photo not yet stored, it carries the title's
For a live photo already stored, its image. For a live photo not yet stored, extension, and the backup may store the image under a different one, found
it carries the title's extension, and the image may be stored under a inside the live photo. `undefined` when the library has no `downloadDirectory`
different one, found inside the live photo. It needs no content source. or no content source.
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`. A - `photo.isLocal` → `boolean` — whether the whole original is at `savePath`.
copy only in the cache does not count.
These async methods may download: These async methods may download:
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the - `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
full-resolution file. For a live photo, `path` and `bytes` are its image's and full-resolution file. For a live photo, `path` and `bytes` are its image's and
`videoPath` is its video. `videoPath` is its video.
- `await photo.download()` → `{ path, bytes, videoPath? }`, as `original()` —
puts the original at `savePath`, creating its folders. When it is already
there, nothing is written. When the cache holds it, it is copied from the
cache; otherwise it is fetched straight to `savePath`, with no copy left in
the cache. Afterwards `isLocal` is true.
- `await photo.thumbnail(opts?)` → `{ path, bytes }`. - `await photo.thumbnail(opts?)` → `{ path, bytes }`.
- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read - `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read
through `original()`; for a live photo, its image's. through `original()`; for a live photo, its image's.
@@ -757,11 +737,10 @@ These async methods may download:
They serve from the on-disk content cache when the bytes are present and They serve from the on-disk content cache when the bytes are present and
otherwise fetch through the pools; `original()`, `content()` and `exif()` also otherwise fetch through the pools; `original()`, `content()` and `exif()` also
serve an original already stored at its save path. `opts.onProgress` reports serve an original the backup has already stored. `opts.onProgress` reports
per-file progress. They throw when the library was opened without a content per-file progress. They throw when the library was opened without a content
source. An original that `original()`, `content()` or `exif()` downloads lands source. An original that `content()` or `exif()` downloads lands in the cache,
in the cache, which does not make `isLocal` true; only `download()` and which does not make `isLocal` true; only `lib.backup()` does.
`lib.backup()` do.
Lower-level accessors that return decrypted model objects (which hold key Lower-level accessors that return decrypted model objects (which hold key
material) are also available: `listCollections()`, `getCollection(id)`, material) are also available: `listCollections()`, `getCollection(id)`,
@@ -804,15 +783,15 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere. a query vector the caller produced elsewhere.
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as - `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
`fresh()` does, puts every in-scope original not already at its save path `fresh()` does, fetches every in-scope original not already in the backup
there as `photo.download()` does (and, with `includeThumbnails`, fetches (and, with `includeThumbnails`, thumbnails) through the content cache, and
thumbnails) through the content cache, and rebuilds the on-disk backup tree rebuilds the on-disk backup tree with a durable failure ledger. A fetched
with a durable failure ledger. A fetched original is written straight to its original is written straight into the backup's `originals/` and not into the
save path and not into the cache, which then counts it as present; one the cache, which then counts it as present; one the cache already held is copied
cache already held is copied from there. `BackupOptions`: `downloadDirectory` from there. `BackupOptions`: `downloadDirectory` (falls back to the one
(falls back to the library's), `includeOriginals` (default `true`), `open()` was given), `includeOriginals` (default `true`), `includeThumbnails`
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See (default `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above
Backup layout above for the tree it writes. for the tree it writes.
### Request pools ### Request pools
@@ -843,7 +822,7 @@ When `metadata.json` belongs to a different account than the client's,
originals and thumbnails are kept; they are reached only through the files the originals and thumbnails are kept; they are reached only through the files the
current account's records name. current account's records name.
A live photo's original is cached as at its save path: its image and its video, A live photo's original is cached as in the backup: its image and its video,
each `originals/<fileID>.<ext>` with its own extension, and each `originals/<fileID>.<ext>` with its own extension, and
`originals/<fileID>.livephoto.json` naming them; the two are evicted together. A `originals/<fileID>.livephoto.json` naming them; the two are evicted together. A
live photo that an earlier version cached as its ZIP is not served: the library live photo that an earlier version cached as its ZIP is not served: the library
@@ -867,7 +846,7 @@ from a very old client, is stored unchecked.
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`, - `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
`LibraryClient`, `RefreshEvent` `LibraryClient`, `RefreshEvent`
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`, - `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`, `SavePathLookup` `TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`, - `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
`EnsureOptions`, `EnsureResult`, `ContentSource` `EnsureOptions`, `EnsureResult`, `ContentSource`
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`, - `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
TODO.md
-11
View File
@@ -35,17 +35,6 @@ declares one.
no method. A test checks, on the JPEG and the HEIC, that each method gives the no method. A test checks, on the JPEG and the HEIC, that each method gives the
same value as `exif()`. same value as `exif()`.
- 2026-10-01: Each original's save path is
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
download directory, which defaults to `photos` in the working directory (issue
143). The date is the photo's `takenAt` in the machine's time zone.
`photo.savePath` is always a string, with or without a content cache, and
`isLocal` is true only when the original is at its save path.
`photo.download()` puts the original there, copied from the cache when the
cache holds it and fetched otherwise. `lib.backup()` does the same for each
file, writes each file's JSON beside its original, and links `collections/` to
the save paths.
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from - 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
the other image formats `exifreader` reads (issue 145). `exifreader` replaces the other image formats `exifreader` reads (issue 145). `exifreader` replaces
src/backup.ts
+120 -110
View File
@@ -2,30 +2,29 @@
// //
// `lib.backup()` waits for a completed refresh of the library (a failed one // `lib.backup()` waits for a completed refresh of the library (a failed one
// fails the backup before any file is touched), then, for every file in scope, // fails the backup before any file is touched), then, for every file in scope,
// puts its original at its save path under `downloadDirectory`, as // gets its original bytes onto disk under `downloadDirectory` and rebuilds the
// `Photo.download()` does, and rebuilds the derived views (per-file sidecars, // derived views (per-file sidecars, per-collection symlink trees,
// per-collection symlink trees, per-collection JSON) from the model. The // per-collection JSON) from the model. The on-disk layout is the historical
// on-disk layout: // one:
// //
// <downloadDirectory>/ // <downloadDirectory>/
// YYYY/YYYY-MM/YYYY-MM-DD/ // originals/<fileID>.<ext> the decrypted bytes
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path) // originals/<fileID>.json per-file metadata sidecar
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar // collections/<name>/<title> symlink into ../../originals
// collections/<name>/<title> symlink to the original
// collections/<name>.json per-collection metadata // collections/<name>.json per-collection metadata
// failures.json durable ledger of unresolved failures // failures.json durable ledger of unresolved failures
// //
// A live photo's original is its image and its video, each with its own // A live photo's original is its image and its video, `<fileID>.<ext>` each
// extension, beside `YYYY-MM-DD.<fileID>.livephoto.json` naming them; its album // with its own extension, and `originals/<fileID>.livephoto.json` naming them;
// folders link both. // its album folders link both.
// //
// Crash-safety rests on two properties. Bytes are present-means-complete: an // Crash-safety rests on two properties. Bytes are present-means-complete: an
// original appears at its save path only via the content layer's atomic // original appears under `originals/` only via the content layer's atomic
// temp-then-rename, so a file that exists is whole and is never re-fetched — an // temp-then-rename, so a file that exists is whole and is never re-fetched — an
// interrupted run resumes by looking at the save paths. The derived views hold // interrupted run resumes by listing the directory. The derived views hold no
// no unique state, so they are rebuilt every run; that repairs stale sidecars // unique state, so they are rebuilt every run; that repairs stale sidecars and
// and missing or broken symlinks left by an earlier crash. A rebuild also // missing or broken symlinks left by an earlier crash. A rebuild also removes
// removes the symlinks to originals that no longer belong to an album, and the // the symlinks into originals/ that no longer belong to an album, and the
// directories of albums that no longer exist. // directories of albums that no longer exist.
// //
// Resilience (issue #8): no per-file condition aborts the run. A failed // Resilience (issue #8): no per-file condition aborts the run. A failed
@@ -49,25 +48,24 @@ import {
symlinkSync, symlinkSync,
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { dirname, extname, join, relative, resolve } from "node:path"; import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, extname, join, relative } from "node:path";
import { removeLeftoverTempFiles } from "./download/index.js"; import { fsyncPath, removeLeftoverTempFiles } from "./download/index.js";
import { sanitizeFileName, withExtension } from "./filename.js"; import { sanitizeFileName, withExtension } from "./filename.js";
import { import {
copyAtomic, nameInOriginals,
placeOriginal, storedOriginal,
savePath, writeLivePhotoJSON,
storedAtSavePath,
} from "./library/content.js"; } from "./library/content.js";
import { representative } from "./library/records.js";
import type { Collection, EnteFile } from "./model/types.js"; import type { Collection, EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void; export type ProgressCallback = (message: string) => void;
export interface BackupOptions { export interface BackupOptions {
// Where the backup tree lives. `lib.backup()` defaults it to the library's // Where the backup tree lives. Required: with none, `backup()` throws
// download directory; `runBackup` with none throws before any network // before any network traffic. A library opened with a `downloadDirectory`
// traffic. // supplies the default.
downloadDirectory?: string; downloadDirectory?: string;
// Fetch and store full-resolution originals. Default true. // Fetch and store full-resolution originals. Default true.
includeOriginals?: boolean; includeOriginals?: boolean;
@@ -91,7 +89,7 @@ export interface BackupResult {
totalFiles: number; totalFiles: number;
// Originals fetched (or copied from the cache) this run. // Originals fetched (or copied from the cache) this run.
downloaded: number; downloaded: number;
// Originals already at their save path and left untouched. // Originals already present and left untouched.
skipped: number; skipped: number;
// Files with an unresolved failure after this run (the ledger size); the // Files with an unresolved failure after this run (the ledger size); the
// CLI exits non-zero while this is above zero. A file can be both // CLI exits non-zero while this is above zero. A file can be both
@@ -109,8 +107,8 @@ export interface BackupLibrary {
listFiles(collectionID: number): EnteFile[]; listFiles(collectionID: number): EnteFile[];
// Get an original's bytes onto disk through the content cache/pools, // Get an original's bytes onto disk through the content cache/pools,
// returning where they landed: `destination` when they were fetched now, // returning where they landed: `destination` when they were fetched now,
// otherwise wherever they already were (the cache, or the library's save // otherwise wherever they already were (the cache, or a prior backup). A
// path). A live photo lands as its image and its video, fetched now beside // live photo lands as its image and its video, fetched now beside
// `destination`. // `destination`.
original( original(
fileID: number, fileID: number,
@@ -170,6 +168,58 @@ const classify = (err: unknown): FailureClass => {
const errorMessage = (err: unknown): string => const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err); err instanceof Error ? err.message : String(err);
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete"). As in the
// download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return;
const tmp = join(
dirname(dest),
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
.toString(36)
.slice(2)}.tmp`,
);
try {
await copyFile(src, tmp);
await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally {
await rm(tmp, { force: true });
}
};
// Put an original the library returned at `dest` in originals/, where a fresh
// fetch already wrote it. A live photo's image and video go beside `dest`: when
// they came from the cache they are copied, after removing whatever was at
// `dest` (an earlier version's ZIP of the two). Then the JSON file naming them
// is written, which is what makes the live photo count as stored.
const placeOriginal = async (
file: EnteFile,
dest: string,
got: { path: string; videoPath?: string },
): Promise<void> => {
if (got.videoPath === undefined) {
await copyAtomic(got.path, dest);
return;
}
const originalsDir = dirname(dest);
const path = join(originalsDir, basename(got.path));
const videoPath = join(originalsDir, basename(got.videoPath));
if (got.path !== path) {
await rm(dest, { force: true });
await copyAtomic(got.path, path);
await copyAtomic(got.videoPath, videoPath);
}
await writeLivePhotoJSON(originalsDir, file.id, { path, videoPath });
};
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or // Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
// non-symlink entry. Throws on failure (a directory in the way, no permission) // non-symlink entry. Throws on failure (a directory in the way, no permission)
// so the caller records it and moves on rather than aborting the run. // so the caller records it and moves on rather than aborting the run.
@@ -241,55 +291,36 @@ const linksFor = (
})); }));
}; };
// Every date folder (`YYYY/YYYY-MM/YYYY-MM-DD/`) under `root`, whether or not a // Remove the symlinks in the album directory `dir` that point into
// file in this backup is saved there. A folder that cannot be read is skipped. // `originalsDir` and are not named in `keep`. Nothing else in the directory
const dateFolders = (root: string): string[] => { // is touched: anything else there was put there by the user.
const subfolders = (dir: string, name: RegExp): string[] => {
try {
return readdirSync(dir, { withFileTypes: true })
.filter((e) => e.isDirectory() && name.test(e.name))
.map((e) => join(dir, e.name));
} catch {
return [];
}
};
return subfolders(root, /^\d{4}$/)
.flatMap((year) => subfolders(year, /^\d{4}-\d\d$/))
.flatMap((month) => subfolders(month, /^\d{4}-\d\d-\d\d$/));
};
// Whether the entry at `path` is a symlink a backup to `root` made: one to an
// original in a `YYYY/YYYY-MM/YYYY-MM-DD/` folder of `root`.
const linksToOriginal = (path: string, root: string): boolean => {
if (!lstatSync(path).isSymbolicLink()) return false;
const target = relative(root, resolve(dirname(path), readlinkSync(path)));
return /^\d{4}\/\d{4}-\d\d\/\d{4}-\d\d-\d\d\/[^/]+$/.test(target);
};
// Remove the symlinks in the album directory `dir` that point to an original
// in `root` and are not named in `keep`. Nothing else in the directory is
// touched: anything else there was put there by the user.
const removeStaleLinks = ( const removeStaleLinks = (
dir: string, dir: string,
keep: Set<string>, keep: Set<string>,
root: string, originalsDir: string,
): void => { ): void => {
const target = relative(dir, originalsDir);
for (const name of readdirSync(dir)) { for (const name of readdirSync(dir)) {
if (keep.has(name)) continue; if (keep.has(name)) continue;
const path = join(dir, name); const path = join(dir, name);
if (linksToOriginal(path, root)) rmSync(path); if (
lstatSync(path).isSymbolicLink() &&
dirname(readlinkSync(path)) === target
) {
rmSync(path);
}
} }
}; };
// Remove the directories under `collectionsDir` that an earlier run wrote for // Remove the directories under `collectionsDir` that an earlier run wrote for
// an album that is gone or renamed: a directory not named in `current` with a // an album that is gone or renamed: a directory not named in `current` with a
// `<name>.json` beside it holding an album ID, which is what a run writes. Its // `<name>.json` beside it holding an album ID, which is what a run writes. Its
// symlinks to originals in `root` are removed; if that leaves it empty, it and // symlinks into originals/ are removed; if that leaves it empty, it and its
// its JSON are deleted, otherwise both stay for what the user put there. // JSON are deleted, otherwise both stay for what the user put there.
const removeStaleAlbumDirs = ( const removeStaleAlbumDirs = (
collectionsDir: string, collectionsDir: string,
current: Set<string>, current: Set<string>,
root: string, originalsDir: string,
): void => { ): void => {
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) { for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
if (!entry.isDirectory() || current.has(entry.name)) continue; if (!entry.isDirectory() || current.has(entry.name)) continue;
@@ -303,7 +334,7 @@ const removeStaleAlbumDirs = (
continue; continue;
} }
const dir = join(collectionsDir, entry.name); const dir = join(collectionsDir, entry.name);
removeStaleLinks(dir, new Set(), root); removeStaleLinks(dir, new Set(), originalsDir);
if (readdirSync(dir).length > 0) continue; if (readdirSync(dir).length > 0) continue;
rmdirSync(dir); rmdirSync(dir);
rmSync(jsonPath); rmSync(jsonPath);
@@ -371,48 +402,34 @@ export const runBackup = async (
log("Refreshing library..."); log("Refreshing library...");
await lib.refresh(); await lib.refresh();
const originalsDir = join(downloadDirectory, "originals");
const collectionsDir = join(downloadDirectory, "collections"); const collectionsDir = join(downloadDirectory, "collections");
const thumbnailsDir = join(downloadDirectory, "thumbnails"); const thumbnailsDir = join(downloadDirectory, "thumbnails");
mkdirSync(originalsDir, { recursive: true });
mkdirSync(collectionsDir, { recursive: true }); mkdirSync(collectionsDir, { recursive: true });
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true }); if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
removeLeftoverTempFiles(originalsDir);
removeLeftoverTempFiles(thumbnailsDir); removeLeftoverTempFiles(thumbnailsDir);
for (const dir of dateFolders(downloadDirectory)) {
removeLeftoverTempFiles(dir);
}
const ledgerPath = join(downloadDirectory, "failures.json"); const ledgerPath = join(downloadDirectory, "failures.json");
const ledger = loadLedger(ledgerPath); const ledger = loadLedger(ledgerPath);
const now = Date.now(); const now = Date.now();
// Collections in scope, and the distinct files across them (a file shared // Collections in scope, and the distinct files across them (a file shared
// by two albums is one original). Each file is the membership // by two albums is one original).
// `representative` picks from all of its albums, in scope or not, so it is
// saved at the path `photo.savePath` names.
const allCollections = lib.listCollections(); const allCollections = lib.listCollections();
const collections = allCollections.filter((c) => const collections = allCollections.filter((c) =>
only ? only.has(c.name) : true, only ? only.has(c.name) : true,
); );
const collectionName = new Map<number, string>(); const collectionName = new Map<number, string>();
for (const c of allCollections) collectionName.set(c.id, c.name); for (const c of collections) collectionName.set(c.id, c.name);
const memberships = new Map<number, EnteFile[]>(); const distinct = new Map<number, EnteFile>();
const filesByCollection = new Map<number, EnteFile[]>(); const filesByCollection = new Map<number, EnteFile[]>();
for (const c of allCollections) { for (const c of collections) {
const files = lib.listFiles(c.id); const files = lib.listFiles(c.id);
filesByCollection.set(c.id, files); filesByCollection.set(c.id, files);
for (const f of files) { for (const f of files) if (!distinct.has(f.id)) distinct.set(f.id, f);
const arr = memberships.get(f.id);
if (arr) arr.push(f);
else memberships.set(f.id, [f]);
}
}
const distinct = new Map<number, EnteFile>();
for (const c of collections) {
for (const f of filesByCollection.get(c.id)!) {
if (!distinct.has(f.id)) {
distinct.set(f.id, representative(memberships.get(f.id)!));
}
}
} }
const errors: BackupError[] = []; const errors: BackupError[] = [];
@@ -448,22 +465,25 @@ export const runBackup = async (
failedThisRun.add(file.id); failedThisRun.add(file.id);
}; };
// Phase 1: get the bytes. Put each pending original at its save path // Phase 1: get the bytes. Fetch each pending original (and optional
// through the content cache/pools, as `Photo.download()` does, and fetch // thumbnail) through the content cache/pools and place it under the backup
// the optional thumbnails; a present file is left as is. // tree; a present file is left as is.
if (includeOriginals) { if (includeOriginals) {
for (const [fileID, file] of distinct) { for (const [fileID, file] of distinct) {
if (storedAtSavePath(downloadDirectory, file) !== undefined) { if (storedOriginal(originalsDir, file) !== undefined) {
skipped++; skipped++;
continue; continue;
} }
const dest = join(originalsDir, nameInOriginals(file));
try { try {
log(`Fetching original ${file.metadata.title} (${fileID})...`); log(`Fetching original ${file.metadata.title} (${fileID})...`);
// A fetched original is written straight to its save path (a // A fetched original is written straight to `dest` (a live
// live photo beside it); only one that was already cached // photo beside it); only one that was already cached elsewhere
// elsewhere is copied. // is copied.
await placeOriginal(downloadDirectory, file, (dest) => await placeOriginal(
lib.original(fileID, dest), file,
dest,
await lib.original(fileID, dest),
); );
downloaded++; downloaded++;
} catch (err) { } catch (err) {
@@ -499,10 +519,9 @@ export const runBackup = async (
// Phase 2: rebuild the derived views from the model. Sidecars first, for // Phase 2: rebuild the derived views from the model. Sidecars first, for
// every present original (this repairs stale ones). // every present original (this repairs stale ones).
if (includeOriginals) { if (includeOriginals) {
for (const file of distinct.values()) { for (const [fileID, file] of distinct) {
if (storedAtSavePath(downloadDirectory, file) !== undefined) { if (storedOriginal(originalsDir, file) !== undefined) {
const path = savePath(downloadDirectory, file); writeSidecar(join(originalsDir, `${fileID}.json`), file);
writeSidecar(withExtension(path, ".json"), file);
} }
} }
} }
@@ -524,11 +543,7 @@ export const runBackup = async (
allCollections.map((c, i) => [c.id, dirNames[i]!]), allCollections.map((c, i) => [c.id, dirNames[i]!]),
); );
try { try {
removeStaleAlbumDirs( removeStaleAlbumDirs(collectionsDir, new Set(dirNames), originalsDir);
collectionsDir,
new Set(dirNames),
downloadDirectory,
);
} catch (err) { } catch (err) {
log(`FAILED removing old album directories: ${errorMessage(err)}`); log(`FAILED removing old album directories: ${errorMessage(err)}`);
} }
@@ -538,18 +553,13 @@ export const runBackup = async (
const colDir = join(collectionsDir, colDirName); const colDir = join(collectionsDir, colDirName);
mkdirSync(colDir, { recursive: true }); mkdirSync(colDir, { recursive: true });
// Every album links the one original, saved from the file's entry in
// `distinct`.
const files = filesByCollection.get(c.id) ?? []; const files = filesByCollection.get(c.id) ?? [];
const links = files.flatMap((f) => const links = files.flatMap((f) =>
linksFor( linksFor(f, storedOriginal(originalsDir, f)),
f,
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
),
); );
const linkNames = uniqueNames(links, true); const linkNames = uniqueNames(links, true);
try { try {
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory); removeStaleLinks(colDir, new Set(linkNames), originalsDir);
} catch (err) { } catch (err) {
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`); log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
} }
src/index.ts
-1
View File
@@ -53,7 +53,6 @@ export {
type PhotoFilter, type PhotoFilter,
type TimelineGroup, type TimelineGroup,
type GroupBy, type GroupBy,
type SavePathLookup,
type ContentSource, type ContentSource,
type ContentResult, type ContentResult,
type ContentEvent, type ContentEvent,
src/library/content.ts
+69 -149
View File
@@ -23,11 +23,6 @@
// original with no recorded hash is stored unchecked, as the upstream client // original with no recorded hash is stored unchecked, as the upstream client
// does; thumbnails have none. On top of that this module refuses to record a // does; thumbnails have none. On top of that this module refuses to record a
// stored file that came out empty. // stored file that came out empty.
//
// It also names each original's save path under the download directory
// (`savePath`), where `Photo.download()` and `lib.backup()` put it. A copy in
// the cache does not count as saved there, but is copied there rather than
// fetched again.
import { import {
closeSync, closeSync,
@@ -39,10 +34,8 @@ import {
} from "node:fs"; } from "node:fs";
import { import {
chmod, chmod,
copyFile,
mkdir, mkdir,
readdir, readdir,
rename,
rm, rm,
stat, stat,
statfs, statfs,
@@ -54,15 +47,13 @@ import type { ApiClient } from "../api/client.js";
import { import {
downloadFile, downloadFile,
downloadThumbnail, downloadThumbnail,
fsyncPath,
type ProgressCallback, type ProgressCallback,
removeLeftoverTempFiles, removeLeftoverTempFiles,
writeAtomic, writeAtomic,
} from "../download/index.js"; } from "../download/index.js";
import { safeExtension, withExtension } from "../filename.js"; import { safeExtension } from "../filename.js";
import type { EnteFile } from "../model/types.js"; import type { EnteFile } from "../model/types.js";
import type { Priority, RequestPools } from "./pools.js"; import type { Priority, RequestPools } from "./pools.js";
import { takenAtOf } from "./records.js";
const DIR_MODE = 0o700; const DIR_MODE = 0o700;
const FILE_MODE = 0o600; const FILE_MODE = 0o600;
@@ -117,9 +108,11 @@ export interface ContentOptions {
export interface PhotoContent { export interface PhotoContent {
original(fileID: number, opts?: ContentOptions): Promise<ContentResult>; original(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
thumbnail(fileID: number, opts?: ContentOptions): Promise<ContentResult>; thumbnail(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
// Put the original at the save path of `file`, the copy the `Photo` holds, // Where a backup stores the original, whether or not it is there yet. For
// and return it there. // a live photo not yet stored, it carries the title's extension, and the
download(file: EnteFile): Promise<ContentResult>; // backup may store the image under a different one.
savePath(fileID: number): string | undefined;
isLocal(fileID: number): boolean;
} }
export interface EnsureResult { export interface EnsureResult {
@@ -206,10 +199,10 @@ export interface ContentCacheOptions {
pools: RequestPools; pools: RequestPools;
source: ContentSource; source: ContentSource;
cacheDirectory: string; cacheDirectory: string;
// The root of the save paths. An original already stored at its save path // The backup destination (issue-level `downloadDirectory`). An original
// counts as present, so the cache serves it rather than fetching a second // already stored there by a backup counts as present, so the cache serves
// copy. // it rather than fetching a second copy.
downloadDirectory: string; downloadDirectory?: string;
// Resolve any membership of a file; every membership shares the underlying // Resolve any membership of a file; every membership shares the underlying
// content key, so any one decrypts the same bytes. // content key, so any one decrypts the same bytes.
getFile: (fileID: number) => EnteFile | undefined; getFile: (fileID: number) => EnteFile | undefined;
@@ -236,27 +229,11 @@ class AbortDrop extends Error {
} }
} }
// The name of a file's original in the cache's originals/: `<fileID><ext>`, the // The name of a file's original in originals/: `<fileID><ext>`, the extension
// extension taken from the title (or `.bin`). // taken from the title (or `.bin`). A backup names its originals the same way.
export const nameInOriginals = (file: EnteFile): string => export const nameInOriginals = (file: EnteFile): string =>
`${file.id}${safeExtension(file.metadata.title)}`; `${file.id}${safeExtension(file.metadata.title)}`;
const pad = (n: number): string => String(n).padStart(2, "0");
// Where the original of `file` is saved under `root`:
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`, dated by the photo's
// `takenAt` in this machine's time zone, with the extension taken from the
// title (or `.bin`). A live photo is stored as its image and its video beside
// this path, each with the extension found inside the live photo.
export const savePath = (root: string, file: EnteFile): string => {
const taken = new Date(takenAtOf(file));
const year = String(taken.getFullYear());
const month = `${year}-${pad(taken.getMonth() + 1)}`;
const day = `${month}-${pad(taken.getDate())}`;
const ext = safeExtension(file.metadata.title);
return join(root, year, month, day, `${day}.${file.id}${ext}`);
};
// The fileID a cache filename encodes, or undefined when the name is not one // The fileID a cache filename encodes, or undefined when the name is not one
// the cache writes (`<digits><ext>`). // the cache writes (`<digits><ext>`).
const fileIDFromName = (name: string): number | undefined => { const fileIDFromName = (name: string): number | undefined => {
@@ -302,29 +279,28 @@ const isZip = (path: string): boolean => {
// A live photo's image and video are named with the extensions from inside its // A live photo's image and video are named with the extensions from inside its
// ZIP, so their names alone do not say which is which. Wherever the cache or a // ZIP, so their names alone do not say which is which. Wherever the cache or a
// save path stores one, a JSON file of this name beside them names both. // backup stores one, a JSON file of this name beside them names both.
// `name` is the original's name without its extension: `<fileID>` in the const livePhotoJSONName = (fileID: number): string =>
// cache, `YYYY-MM-DD.<fileID>` at the save path. `${fileID}.livephoto.json`;
const livePhotoJSONName = (name: string): string => `${name}.livephoto.json`;
// The image and video that the live photo's JSON file in `dir` names, or // The image and video that the live photo's JSON file in `dir` names, or
// undefined when there is none. Only `name` with an extension is taken, so the // undefined when there is none. Only names of the form the cache writes are
// file cannot point outside `dir`. // taken, so the file cannot point outside `dir`.
const readLivePhotoJSON = ( const readLivePhotoJSON = (
dir: string, dir: string,
name: string, fileID: number,
): { path: string; videoPath: string } | undefined => { ): { path: string; videoPath: string } | undefined => {
const valid = (part: unknown): part is string => const valid = (name: unknown): name is string =>
typeof part === "string" && part === `${name}${safeExtension(part)}`; typeof name === "string" && name === `${fileID}${safeExtension(name)}`;
try { try {
const { image, video } = JSON.parse( const { image, video } = JSON.parse(
readFileSync(join(dir, livePhotoJSONName(name)), "utf-8"), readFileSync(join(dir, livePhotoJSONName(fileID)), "utf-8"),
); );
if (valid(image) && valid(video)) { if (valid(image) && valid(video)) {
return { path: join(dir, image), videoPath: join(dir, video) }; return { path: join(dir, image), videoPath: join(dir, video) };
} }
} catch { } catch {
// No such file, or not one quak wrote. // No such file, or not one the cache wrote.
} }
return undefined; return undefined;
}; };
@@ -332,11 +308,11 @@ const readLivePhotoJSON = (
// Write the JSON file naming a live photo's image and video, both in `dir`. // Write the JSON file naming a live photo's image and video, both in `dir`.
export const writeLivePhotoJSON = ( export const writeLivePhotoJSON = (
dir: string, dir: string,
name: string, fileID: number,
stored: { path: string; videoPath: string }, stored: { path: string; videoPath: string },
): Promise<void> => ): Promise<void> =>
writeAtomic( writeAtomic(
join(dir, livePhotoJSONName(name)), join(dir, livePhotoJSONName(fileID)),
new TextEncoder().encode( new TextEncoder().encode(
JSON.stringify({ JSON.stringify({
image: basename(stored.path), image: basename(stored.path),
@@ -345,19 +321,17 @@ export const writeLivePhotoJSON = (
), ),
); );
// The original of `file` as stored in `dir` under `name` (without its // The original of `file` as the cache or a backup stored it in `dir`, when all
// extension), when all of it is there: `<name><ext>`, or a live photo's image // of it is there: `<fileID><ext>`, or a live photo's image and video.
// and video.
export const storedOriginal = ( export const storedOriginal = (
dir: string, dir: string,
name: string,
file: EnteFile, file: EnteFile,
): { path: string; videoPath?: string } | undefined => { ): { path: string; videoPath?: string } | undefined => {
if (file.metadata.fileType !== "livePhoto") { if (file.metadata.fileType !== "livePhoto") {
const path = join(dir, `${name}${safeExtension(file.metadata.title)}`); const path = join(dir, nameInOriginals(file));
return hasContent(path) ? { path } : undefined; return hasContent(path) ? { path } : undefined;
} }
const stored = readLivePhotoJSON(dir, name); const stored = readLivePhotoJSON(dir, file.id);
return stored !== undefined && return stored !== undefined &&
hasContent(stored.path) && hasContent(stored.path) &&
hasContent(stored.videoPath) hasContent(stored.videoPath)
@@ -365,76 +339,10 @@ export const storedOriginal = (
: undefined; : undefined;
}; };
// The original of `file` as stored at its save path under `root`, when all of
// it is there.
export const storedAtSavePath = (
root: string,
file: EnteFile,
): { path: string; videoPath?: string } | undefined => {
const path = savePath(root, file);
return storedOriginal(dirname(path), basename(path, extname(path)), file);
};
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete"). As in the
// download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
export const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return;
const tmp = join(
dirname(dest),
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
.toString(36)
.slice(2)}.tmp`,
);
try {
await copyFile(src, tmp);
await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally {
await rm(tmp, { force: true });
}
};
// Put the original of `file` at its save path under `root`, creating its
// folders. `get` is given the save path and returns where the original is: a
// fetch writes it there, and a copy the cache holds is copied there. A live
// photo's image and video go beside the save path, each with its own
// extension: when they came from the cache they are copied. Then the JSON file
// naming them is written, which is what makes the live photo count as stored.
export const placeOriginal = async (
root: string,
file: EnteFile,
get: (dest: string) => Promise<{ path: string; videoPath?: string }>,
): Promise<{ path: string; videoPath?: string }> => {
const dest = savePath(root, file);
await mkdir(dirname(dest), { recursive: true });
const got = await get(dest);
if (got.videoPath === undefined) {
await copyAtomic(got.path, dest);
return { path: dest };
}
const path = withExtension(dest, extname(got.path));
const videoPath = withExtension(dest, extname(got.videoPath));
if (got.path !== path) {
await copyAtomic(got.path, path);
await copyAtomic(got.videoPath, videoPath);
}
const name = basename(dest, extname(dest));
await writeLivePhotoJSON(dirname(dest), name, { path, videoPath });
return { path, videoPath };
};
export class ContentCache implements PhotoContent, ThumbnailsAPI { export class ContentCache implements PhotoContent, ThumbnailsAPI {
private readonly pools: RequestPools; private readonly pools: RequestPools;
private readonly source: ContentSource; private readonly source: ContentSource;
private readonly downloadDirectory: string; private readonly downloadDirectory?: string;
private readonly getFile: (fileID: number) => EnteFile | undefined; private readonly getFile: (fileID: number) => EnteFile | undefined;
private readonly originalsDir: string; private readonly originalsDir: string;
private readonly thumbnailsDir: string; private readonly thumbnailsDir: string;
@@ -532,23 +440,32 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress); return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
} }
// Put the original at the save path of `file` under the download directory // Where a backup to the download directory stores the file's original,
// and return it there. `file` is the copy the `Photo` holds, so the path is // whether or not it is there yet: for a live photo already stored, its
// the one its `savePath` names, even after a refresh changed the date. One // image. For a live photo not yet stored, it carries the title's
// already stored there is returned as it is; one the cache holds is copied // extension, and the backup may store the image under a different one.
// from it; any other is fetched straight to the save path, with no copy // Undefined with no download directory.
// left in the cache. savePath(fileID: number): string | undefined {
async download(file: EnteFile): Promise<ContentResult> { const file = this.getFile(fileID);
const root = this.downloadDirectory; if (this.downloadDirectory === undefined || file === undefined)
const saved = return undefined;
storedAtSavePath(root, file) ?? const dir = join(this.downloadDirectory, "originals");
(await placeOriginal(root, file, (dest) => return (
this.backupOriginal(file.id, dest), storedOriginal(dir, file)?.path ?? join(dir, nameInOriginals(file))
)); );
return { ...saved, bytes: fileSize(saved.path) ?? 0 };
} }
// Get an original for a save path. One not present anywhere is written // Whether the whole original is in the download directory, as a backup
// stores it. A copy only in the cache does not count.
isLocal(fileID: number): boolean {
const file = this.getFile(fileID);
if (this.downloadDirectory === undefined || file === undefined)
return false;
const dir = join(this.downloadDirectory, "originals");
return storedOriginal(dir, file) !== undefined;
}
// Get an original for a backup. One not present anywhere is written
// straight to `destination` and recorded there, so no second copy lands // straight to `destination` and recorded there, so no second copy lands
// in the cache; one already present is returned where it is. // in the cache; one already present is returned where it is.
async backupOriginal( async backupOriginal(
@@ -718,9 +635,12 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
known.delete(fileID); known.delete(fileID);
} }
// An original already stored at its save path counts as present. // An original a backup already stored counts as present.
if (kind === "original") { if (kind === "original" && this.downloadDirectory !== undefined) {
const stored = storedAtSavePath(this.downloadDirectory, file); const stored = storedOriginal(
join(this.downloadDirectory, "originals"),
file,
);
if (stored !== undefined) { if (stored !== undefined) {
this.originals.set(fileID, stored); this.originals.set(fileID, stored);
return { return {
@@ -756,7 +676,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
? this.beginOriginalWrite(fileID) ? this.beginOriginalWrite(fileID)
: null; : null;
try { try {
const stored = await this.fetchInto( const stored = await this.download(
file, file,
dest, dest,
kind, kind,
@@ -771,12 +691,12 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
); );
} }
} }
// `placeOriginal` records a live photo it saves. // A backup records its own live photos.
if ( if (
stored.videoPath !== undefined && stored.videoPath !== undefined &&
opts?.destination === undefined opts?.destination === undefined
) { ) {
await writeLivePhotoJSON(dir, String(fileID), { await writeLivePhotoJSON(dir, fileID, {
path: stored.path, path: stored.path,
videoPath: stored.videoPath, videoPath: stored.videoPath,
}); });
@@ -799,7 +719,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
// Fetch into `destination`, returning where the bytes landed: there, or // Fetch into `destination`, returning where the bytes landed: there, or
// for a live photo, its image and video beside it. // for a live photo, its image and video beside it.
private async fetchInto( private async download(
file: EnteFile, file: EnteFile,
destination: string, destination: string,
kind: Kind, kind: Kind,
@@ -824,10 +744,10 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
await utimes(path, now, now).catch(() => undefined); await utimes(path, now, now).catch(() => undefined);
} }
// Every stored original that lives under `originalsDir` (a save-path hit // Every stored original that lives under `originalsDir` (a backup-directory
// recorded in the map is excluded), with its size and mtime; a live // hit recorded in the map is excluded), with its size and mtime; a live
// photo's size includes its video. Entries whose file has vanished are // photo's size includes its video. Entries whose file has vanished are
// dropped from the map. Save paths and thumbnails are never counted. // dropped from the map. Backups and thumbnails are never counted.
private async measureOriginals(): Promise<{ private async measureOriginals(): Promise<{
entries: { entries: {
fileID: number; fileID: number;
@@ -933,7 +853,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
await rm( await rm(
join( join(
this.originalsDir, this.originalsDir,
livePhotoJSONName(String(e.fileID)), livePhotoJSONName(e.fileID),
), ),
{ force: true }, { force: true },
); );
@@ -987,8 +907,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
// earlier version stored under the image's name, and is removed. // earlier version stored under the image's name, and is removed.
// Any other is left alone: another process may have just stored // Any other is left alone: another process may have just stored
// it and not yet written the JSON file. // it and not yet written the JSON file.
const livePhoto = names.has(livePhotoJSONName(String(id))) const livePhoto = names.has(livePhotoJSONName(id))
? readLivePhotoJSON(dir, String(id)) ? readLivePhotoJSON(dir, id)
: undefined; : undefined;
if (livePhoto !== undefined) { if (livePhoto !== undefined) {
into.set(id, livePhoto); into.set(id, livePhoto);
src/library/index.ts
+28 -41
View File
@@ -27,7 +27,7 @@
// never masked by a subsequent empty refresh. // never masked by a subsequent empty refresh.
import { rm } from "node:fs/promises"; import { rm } from "node:fs/promises";
import { join, resolve } from "node:path"; import { join } from "node:path";
import envPaths from "env-paths"; import envPaths from "env-paths";
import { MetadataStore } from "./store.js"; import { MetadataStore } from "./store.js";
@@ -49,12 +49,9 @@ import {
type PhotosAPI, type PhotosAPI,
type TimelineAPI, type TimelineAPI,
type FreshReads, type FreshReads,
type SavePathLookup,
} from "./read.js"; } from "./read.js";
import { import {
ContentCache, ContentCache,
savePath,
storedAtSavePath,
type ContentSource, type ContentSource,
type ThumbnailsAPI, type ThumbnailsAPI,
type EnsureOptions, type EnsureOptions,
@@ -73,7 +70,6 @@ export {
type PhotoFilter, type PhotoFilter,
type TimelineGroup, type TimelineGroup,
type GroupBy, type GroupBy,
type SavePathLookup,
} from "./read.js"; } from "./read.js";
export { export {
type ContentSource, type ContentSource,
@@ -173,10 +169,8 @@ export interface LibraryOptions {
// Where `metadata.json` lives. Defaults to the env-paths cache directory // Where `metadata.json` lives. Defaults to the env-paths cache directory
// plus the user id, so each account has its own cache. // plus the user id, so each account has its own cache.
cacheDirectory?: string; cacheDirectory?: string;
// The root of every photo's save path, where `Photo.download()` and // Persistent backup destination. The refresh loop does not use it; the
// `lib.backup()` put originals. Defaults to `photos` in the working // content cache treats an original already stored there as present.
// directory at open. The content cache treats an original already stored
// at its save path as present.
downloadDirectory?: string; downloadDirectory?: string;
refreshIntervalSeconds?: number; refreshIntervalSeconds?: number;
onProgress?: RefreshProgressCallback; onProgress?: RefreshProgressCallback;
@@ -240,7 +234,7 @@ export interface LibraryStatus {
export class Library { export class Library {
readonly cacheDirectory: string; readonly cacheDirectory: string;
readonly downloadDirectory: string; readonly downloadDirectory?: string;
// The in-process read surface (issue #44). Each namespace answers // The in-process read surface (issue #44). Each namespace answers
// synchronously from the live record projection; no read touches the // synchronously from the live record projection; no read touches the
@@ -302,7 +296,7 @@ export class Library {
store: MetadataStore; store: MetadataStore;
userID: number; userID: number;
cacheDirectory: string; cacheDirectory: string;
downloadDirectory: string; downloadDirectory?: string;
intervalMs: number; intervalMs: number;
onProgress?: RefreshProgressCallback; onProgress?: RefreshProgressCallback;
pools: RequestPools; pools: RequestPools;
@@ -326,14 +320,8 @@ export class Library {
// The read namespaces derive fresh from the store on each call, so they // The read namespaces derive fresh from the store on each call, so they
// always reflect the latest refresh. // always reflect the latest refresh.
const derive = (): DerivedRecords => this.deriveNow(); const derive = (): DerivedRecords => this.deriveNow();
const root = this.downloadDirectory; this.albums = makeAlbumsAPI(derive, this.cache);
const saves: SavePathLookup = { this.photos = makePhotosAPI(derive, this.cache);
savePath: (file) =>
storedAtSavePath(root, file)?.path ?? savePath(root, file),
isLocal: (file) => storedAtSavePath(root, file) !== undefined,
};
this.albums = makeAlbumsAPI(derive, saves, this.cache);
this.photos = makePhotosAPI(derive, saves, this.cache);
this.timeline = makeTimelineAPI(derive); this.timeline = makeTimelineAPI(derive);
this.thumbnails = { this.thumbnails = {
ensure: (opts: EnsureOptions): Promise<EnsureResult[]> => { ensure: (opts: EnsureOptions): Promise<EnsureResult[]> => {
@@ -362,13 +350,6 @@ export class Library {
const { userID } = opts.client.whoami(); const { userID } = opts.client.whoami();
const cacheDirectory = const cacheDirectory =
opts.cacheDirectory ?? defaultCacheDirectory(userID); opts.cacheDirectory ?? defaultCacheDirectory(userID);
if (opts.downloadDirectory === "") {
throw new Error(
"library: downloadDirectory is empty (leave it out to save " +
"under photos/ in the working directory)",
);
}
const downloadDirectory = opts.downloadDirectory ?? resolve("photos");
const metadataPath = join(cacheDirectory, "metadata.json"); const metadataPath = join(cacheDirectory, "metadata.json");
let store = await MetadataStore.load(metadataPath); let store = await MetadataStore.load(metadataPath);
// A cache directory given explicitly can hold another account's cache. // A cache directory given explicitly can hold another account's cache.
@@ -423,7 +404,7 @@ export class Library {
pools, pools,
source, source,
cacheDirectory, cacheDirectory,
downloadDirectory, downloadDirectory: opts.downloadDirectory,
getFile: (fileID) => store.getFileByID(fileID), getFile: (fileID) => store.getFileByID(fileID),
cacheOriginalsMaxBytes: opts.cacheOriginalsMaxBytes, cacheOriginalsMaxBytes: opts.cacheOriginalsMaxBytes,
freeBelowBytes: opts.freeBelowBytes, freeBelowBytes: opts.freeBelowBytes,
@@ -445,7 +426,7 @@ export class Library {
store, store,
userID, userID,
cacheDirectory, cacheDirectory,
downloadDirectory, downloadDirectory: opts.downloadDirectory,
intervalMs, intervalMs,
onProgress: opts.onProgress, onProgress: opts.onProgress,
pools, pools,
@@ -489,10 +470,9 @@ export class Library {
return this.store.getFile(collectionID, fileID); return this.store.getFile(collectionID, fileID);
} }
// The membership of a file its record is read from, addressed by file id // Any membership of a file, addressed by file id alone. A file's own
// alone. A file's own metadata (title, creationTime) is identical across // metadata (title, creationTime) is identical across the collections it
// the collections it belongs to, so this serves the point commands that // belongs to, so this serves the point commands that hold only a fileID.
// hold only a fileID.
getFileByID(fileID: number): EnteFile | undefined { getFileByID(fileID: number): EnteFile | undefined {
return this.store.getFileByID(fileID); return this.store.getFileByID(fileID);
} }
@@ -563,20 +543,27 @@ export class Library {
}; };
} }
// Back up every in-scope file to `opts.downloadDirectory`, or else the // Back up every in-scope file to `downloadDirectory` in the historical
// library's, each original at its save path, with a durable failure // on-disk layout, with a durable failure ledger (issue #51). Waits for a
// ledger (issue #51). Waits for a completed refresh first, as `fresh()` // completed refresh first, as `fresh()` does, joining one already running,
// does, joining one already running, and rejects before touching any file // and rejects before touching any file when it fails. Then fetches pending
// when it fails. Then puts pending originals at their save paths as // originals (and optional thumbnails) through the content cache and pools,
// `Photo.download()` does (and optional thumbnails) through the content // and rebuilds the derived symlink/JSON views from the model. Throws before
// cache and pools, and rebuilds the derived symlink/JSON views from the // any network work when no download directory is available or no content
// model. Throws before any network work when no content cache backs the // cache backs the originals it must fetch.
// originals it must fetch.
backup(opts?: BackupOptions): Promise<BackupResult> { backup(opts?: BackupOptions): Promise<BackupResult> {
const downloadDirectory = const downloadDirectory =
opts?.downloadDirectory ?? this.downloadDirectory; opts?.downloadDirectory ?? this.downloadDirectory;
const includeOriginals = opts?.includeOriginals ?? true; const includeOriginals = opts?.includeOriginals ?? true;
const includeThumbnails = opts?.includeThumbnails ?? false; const includeThumbnails = opts?.includeThumbnails ?? false;
if (!downloadDirectory) {
return Promise.reject(
new Error(
"backup requires a downloadDirectory (pass one to " +
"backup() or open the library with one)",
),
);
}
if ((includeOriginals || includeThumbnails) && !this.cache) { if ((includeOriginals || includeThumbnails) && !this.cache) {
return Promise.reject( return Promise.reject(
new Error( new Error(
src/library/read.ts
+24 -51
View File
@@ -12,26 +12,19 @@
// plain records are the serializable surface, and `record()` returns one. // plain records are the serializable surface, and `record()` returns one.
// //
// A `Photo` also fetches its own bytes: `original()`, `thumbnail()`, // A `Photo` also fetches its own bytes: `original()`, `thumbnail()`,
// `download()`, `content()`, `exif()` and the methods that each return one // `content()`, `exif()` and the methods that each return one field of `exif()`
// field of `exif()` go through the on-disk content cache (issue #46), and are // go through the on-disk content cache (issue #46), and are the one place in
// the one place in this module that may touch the network. A library opened // this module that may touch the network. A library opened without a content
// without a content source leaves that cache absent, and those methods then // source leaves that cache absent, and those methods then throw. `savePath` and
// throw. `savePath` and `isLocal` look only at the disk and need no cache. // `isLocal` look only at the disk.
import { readFile } from "node:fs/promises"; import { readFile } from "node:fs/promises";
import { readPhotoExif, type PhotoExif } from "../exif.js"; import { readPhotoExif, type PhotoExif } from "../exif.js";
import type { CollectionType, EnteFile, FileType } from "../model/types.js"; import type { CollectionType, FileType } from "../model/types.js";
import type { ContentOptions, ContentResult, PhotoContent } from "./content.js"; import type { ContentOptions, ContentResult, PhotoContent } from "./content.js";
import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js"; import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js";
// Where a photo's original is saved, and whether all of it is there. The
// library answers both from the disk, with or without a content cache.
export interface SavePathLookup {
savePath(file: EnteFile): string;
isLocal(file: EnteFile): boolean;
}
// Newest first, with fileID as a stable tiebreak so equal-timed files order // Newest first, with fileID as a stable tiebreak so equal-timed files order
// deterministically — the same order the record projection uses. // deterministically — the same order the record projection uses.
const byNewest = (a: PhotoRecord, b: PhotoRecord): number => const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
@@ -50,14 +43,10 @@ type PhotoExifMethods = {
}; };
// A single photo. Field access mirrors `PhotoRecord`; `record()` returns the // A single photo. Field access mirrors `PhotoRecord`; `record()` returns the
// underlying plain record for callers that need the IPC-safe value. `file` is // underlying plain record for callers that need the IPC-safe value.
// the membership the record is read from, so the save path carries the date of
// `takenAt` and stays known after a refresh removes the file from the library.
export class Photo implements PhotoExifMethods { export class Photo implements PhotoExifMethods {
constructor( constructor(
private readonly rec: PhotoRecord, private readonly rec: PhotoRecord,
private readonly file: EnteFile,
private readonly saves: SavePathLookup,
private readonly cache?: PhotoContent, private readonly cache?: PhotoContent,
) {} ) {}
@@ -108,20 +97,20 @@ export class Photo implements PhotoExifMethods {
return this.rec.isHidden; return this.rec.isHidden;
} }
// Where `download()` and `lib.backup()` put the original under the // Where `lib.backup()` stores the original in the library's download
// library's download directory, whether or not it is there yet: // directory, whether or not it is there yet: for a live photo already
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`. For a live photo // stored, its image. For a live photo not yet stored, it carries the
// already stored, its image. For a live photo not yet stored, it carries // title's extension, and the backup may store the image under a different
// the title's extension, and the image may be stored under a different // one. Undefined when the library has no download directory or no content
// one. // cache.
get savePath(): string { get savePath(): string | undefined {
return this.saves.savePath(this.file); return this.cache?.savePath(this.rec.fileID);
} }
// Whether the whole original is at `savePath`. A copy only in the cache // Whether the whole original is at `savePath`. A copy only in the cache
// does not count. // does not count.
get isLocal(): boolean { get isLocal(): boolean {
return this.saves.isLocal(this.file); return this.cache?.isLocal(this.rec.fileID) ?? false;
} }
record(): PhotoRecord { record(): PhotoRecord {
@@ -130,20 +119,12 @@ export class Photo implements PhotoExifMethods {
// Fetch and cache the full-resolution original, returning its on-disk path // Fetch and cache the full-resolution original, returning its on-disk path
// and byte length; for a live photo, its image's, and its video's path as // and byte length; for a live photo, its image's, and its video's path as
// `videoPath`. Served from the cache (or the save path) when already // `videoPath`. Served from the cache (or the backup download directory)
// present, otherwise fetched through the content pool. // when already present, otherwise fetched through the content pool.
async original(opts?: ContentOptions): Promise<ContentResult> { async original(opts?: ContentOptions): Promise<ContentResult> {
return this.cacheOrThrow().original(this.rec.fileID, opts); return this.cacheOrThrow().original(this.rec.fileID, opts);
} }
// Put the original at `savePath` and return it there, as `original()`
// does. When it is already there, nothing is written. When the cache holds
// it, it is copied from there; otherwise it is fetched straight to
// `savePath`.
async download(): Promise<ContentResult> {
return this.cacheOrThrow().download(this.file);
}
// As `original`, for the thumbnail, through the thumbnail pool. // As `original`, for the thumbnail, through the thumbnail pool.
async thumbnail(opts?: ContentOptions): Promise<ContentResult> { async thumbnail(opts?: ContentOptions): Promise<ContentResult> {
return this.cacheOrThrow().thumbnail(this.rec.fileID, opts); return this.cacheOrThrow().thumbnail(this.rec.fileID, opts);
@@ -242,7 +223,6 @@ export class Album {
constructor( constructor(
private readonly rec: AlbumRecord, private readonly rec: AlbumRecord,
private readonly records: DerivedRecords, private readonly records: DerivedRecords,
private readonly saves: SavePathLookup,
private readonly content?: PhotoContent, private readonly content?: PhotoContent,
) {} ) {}
@@ -277,10 +257,7 @@ export class Album {
const out: Photo[] = []; const out: Photo[] = [];
for (const id of this.rec.fileIDs) { for (const id of this.rec.fileIDs) {
const p = this.records.photos.get(id); const p = this.records.photos.get(id);
const file = this.records.files.get(id); if (p) out.push(new Photo(p, this.content));
if (p && file) {
out.push(new Photo(p, file, this.saves, this.content));
}
} }
return out; return out;
} }
@@ -343,19 +320,18 @@ export interface FreshReads {
export const makeAlbumsAPI = ( export const makeAlbumsAPI = (
derive: () => DerivedRecords, derive: () => DerivedRecords,
saves: SavePathLookup,
content?: PhotoContent, content?: PhotoContent,
): AlbumsAPI => ({ ): AlbumsAPI => ({
list: (): Album[] => { list: (): Album[] => {
const records = derive(); const records = derive();
return [...records.albums.values()] return [...records.albums.values()]
.sort(byNewestAlbum) .sort(byNewestAlbum)
.map((rec) => new Album(rec, records, saves, content)); .map((rec) => new Album(rec, records, content));
}, },
byID: ({ collectionID }): Album | undefined => { byID: ({ collectionID }): Album | undefined => {
const records = derive(); const records = derive();
const rec = records.albums.get(collectionID); const rec = records.albums.get(collectionID);
return rec ? new Album(rec, records, saves, content) : undefined; return rec ? new Album(rec, records, content) : undefined;
}, },
byName: ({ albumName }): Album | undefined => { byName: ({ albumName }): Album | undefined => {
const records = derive(); const records = derive();
@@ -364,20 +340,17 @@ export const makeAlbumsAPI = (
const match = [...records.albums.values()] const match = [...records.albums.values()]
.sort(byNewestAlbum) .sort(byNewestAlbum)
.find((rec) => rec.name === albumName); .find((rec) => rec.name === albumName);
return match ? new Album(match, records, saves, content) : undefined; return match ? new Album(match, records, content) : undefined;
}, },
}); });
export const makePhotosAPI = ( export const makePhotosAPI = (
derive: () => DerivedRecords, derive: () => DerivedRecords,
saves: SavePathLookup,
content?: PhotoContent, content?: PhotoContent,
): PhotosAPI => ({ ): PhotosAPI => ({
byID: ({ fileID }): Photo | undefined => { byID: ({ fileID }): Photo | undefined => {
const records = derive(); const rec = derive().photos.get(fileID);
const rec = records.photos.get(fileID); return rec ? new Photo(rec, content) : undefined;
const file = records.files.get(fileID);
return rec && file ? new Photo(rec, file, saves, content) : undefined;
}, },
records: ({ fileIDs }): PhotoRecord[] => { records: ({ fileIDs }): PhotoRecord[] => {
const { photos } = derive(); const { photos } = derive();
src/library/records.ts
+13 -31
View File
@@ -86,10 +86,6 @@ export interface LibraryChange {
export interface DerivedRecords { export interface DerivedRecords {
albums: Map<number, AlbumRecord>; albums: Map<number, AlbumRecord>;
photos: Map<number, PhotoRecord>; photos: Map<number, PhotoRecord>;
// The membership each photo's record is read from, for its `Photo`'s save
// path. It holds the file's key, so it stays in this process: no snapshot
// or change carries it.
files: Map<number, EnteFile>;
} }
const asString = (v: unknown): string | undefined => const asString = (v: unknown): string | undefined =>
@@ -105,29 +101,10 @@ const microsToMillis = (micros: number): number => Math.floor(micros / 1000);
const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number => const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number =>
b.takenAt - a.takenAt || b.fileID - a.fileID; b.takenAt - a.takenAt || b.fileID - a.fileID;
// A photo's `takenAt` in milliseconds: `pubMagicMetadata.editedTime` when the
// user edited the date, else basic-metadata `creationTime`.
export const takenAtOf = (file: EnteFile): number =>
microsToMillis(
asNumber(file.pubMagicMetadata?.editedTime) ??
file.metadata.creationTime,
);
// The membership a file's record is read from: the most recently synced, lowest
// collection id to break ties. Whatever dates a file's save path takes this
// membership too, so the path always carries the record's `takenAt`.
export const representative = (memberships: EnteFile[]): EnteFile =>
memberships.reduce((best, m) =>
m.updationTime > best.updationTime ||
(m.updationTime === best.updationTime &&
m.collectionID < best.collectionID)
? m
: best,
);
// Build one PhotoRecord from every membership of a file. The memberships share // Build one PhotoRecord from every membership of a file. The memberships share
// the same underlying file, so metadata is read from a single representative; // the same underlying file, so metadata is read from a single representative
// `albumIDs` gathers them all. // (the most recently synced, lowest collection id to break ties); `albumIDs`
// gathers them all.
const toPhotoRecord = ( const toPhotoRecord = (
fileID: number, fileID: number,
memberships: EnteFile[], memberships: EnteFile[],
@@ -135,18 +112,25 @@ const toPhotoRecord = (
const albumIDs = memberships const albumIDs = memberships
.map((m) => m.collectionID) .map((m) => m.collectionID)
.sort((a, b) => a - b); .sort((a, b) => a - b);
const rep = representative(memberships); const rep = memberships.reduce((best, m) =>
m.updationTime > best.updationTime ||
(m.updationTime === best.updationTime &&
m.collectionID < best.collectionID)
? m
: best,
);
const pub = rep.pubMagicMetadata ?? {}; const pub = rep.pubMagicMetadata ?? {};
const priv = rep.magicMetadata ?? {}; const priv = rep.magicMetadata ?? {};
const takenAtMicros = asNumber(pub.editedTime) ?? rep.metadata.creationTime;
const visibility = asNumber(priv.visibility); const visibility = asNumber(priv.visibility);
const record: PhotoRecord = { const record: PhotoRecord = {
fileID, fileID,
albumIDs, albumIDs,
title: asString(pub.editedName) ?? rep.metadata.title, title: asString(pub.editedName) ?? rep.metadata.title,
takenAt: takenAtOf(rep), takenAt: microsToMillis(takenAtMicros),
modifiedAt: microsToMillis(rep.metadata.modificationTime), modifiedAt: microsToMillis(rep.metadata.modificationTime),
fileType: rep.metadata.fileType, fileType: rep.metadata.fileType,
isArchived: visibility === VISIBILITY_ARCHIVED, isArchived: visibility === VISIBILITY_ARCHIVED,
@@ -214,7 +198,6 @@ export const deriveRecords = (
} }
const photos = new Map<number, PhotoRecord>(); const photos = new Map<number, PhotoRecord>();
const photoFiles = new Map<number, EnteFile>();
const takenAtByFile = new Map<number, number>(); const takenAtByFile = new Map<number, number>();
for (const [fileID, memberships] of byFileID) { for (const [fileID, memberships] of byFileID) {
const record = toPhotoRecord(fileID, memberships); const record = toPhotoRecord(fileID, memberships);
@@ -226,7 +209,6 @@ export const deriveRecords = (
record.thumbnailPath = paths.thumbnailPath; record.thumbnailPath = paths.thumbnailPath;
} }
photos.set(fileID, record); photos.set(fileID, record);
photoFiles.set(fileID, representative(memberships));
takenAtByFile.set(fileID, record.takenAt); takenAtByFile.set(fileID, record.takenAt);
} }
@@ -235,7 +217,7 @@ export const deriveRecords = (
albums.set(c.id, toAlbumRecord(c, files, takenAtByFile)); albums.set(c.id, toAlbumRecord(c, files, takenAtByFile));
} }
return { albums, photos, files: photoFiles }; return { albums, photos };
}; };
// Sorted, GUI-ready arrays: albums newest updated first, photos newest first. // Sorted, GUI-ready arrays: albums newest updated first, photos newest first.
src/library/store.ts
+5 -8
View File
@@ -16,7 +16,6 @@ import { dirname } from "node:path";
import { writeAtomic } from "../download/index.js"; import { writeAtomic } from "../download/index.js";
import type { Collection, EnteFile, Microseconds } from "../model/types.js"; import type { Collection, EnteFile, Microseconds } from "../model/types.js";
import { representative } from "./records.js";
// Bumped only when the on-disk shape changes incompatibly. A file written // Bumped only when the on-disk shape changes incompatibly. A file written
// under a different version is discarded on load (see `load`): re-fetching // under a different version is discarded on load (see `load`): re-fetching
@@ -180,16 +179,14 @@ export class MetadataStore {
return this.files.get(fileKey(collectionID, fileID)); return this.files.get(fileKey(collectionID, fileID));
} }
// The membership of a file its record is read from (`representative`), or // Any membership of a file, or undefined. Every membership re-wraps the
// undefined. Any membership could fetch the bytes, but the content cache // same underlying content key, so any one is enough to fetch the bytes;
// resolves a fileID to a file this way so that it dates the save path // the content cache resolves a fileID to a file this way.
// from the same membership as `photo.savePath`.
getFileByID(fileID: number): EnteFile | undefined { getFileByID(fileID: number): EnteFile | undefined {
const memberships: EnteFile[] = [];
for (const file of this.files.values()) { for (const file of this.files.values()) {
if (file.id === fileID) memberships.push(file); if (file.id === fileID) return file;
} }
return memberships.length > 0 ? representative(memberships) : undefined; return undefined;
} }
listFiles(collectionID: number): EnteFile[] { listFiles(collectionID: number): EnteFile[] {
test/cli/backup.test.ts
+146 -178
View File
@@ -1,16 +1,16 @@
/** /**
* Tests for the `quak backup` logic, now built on the library API (issue #51). * Tests for the `quak backup` logic, now built on the library API (issue #51).
* *
* `lib.backup({ downloadDirectory })` refreshes the library, puts each pending * `lib.backup({ downloadDirectory })` refreshes the library, fetches each
* file's original at its save path through the content cache/pools, and * pending file's original through the content cache/pools, and materialises the
* materialises the on-disk layout: * unchanged on-disk layout:
* *
* <downloadDirectory>/ * <downloadDirectory>/
* YYYY/YYYY-MM/YYYY-MM-DD/ * originals/
* YYYY-MM-DD.<fileID>.<ext> the decrypted bytes ("present means complete") * <fileID>.<ext> the decrypted bytes ("present means complete")
* YYYY-MM-DD.<fileID>.json per-file metadata sidecar (rebuilt each run) * <fileID>.json per-file metadata sidecar (rebuilt each run)
* collections/ * collections/
* <name>/<title> symlink to the original (rebuilt each run) * <name>/<title> symlink into ../originals (rebuilt each run)
* <name>.json per-collection metadata (rebuilt each run) * <name>.json per-collection metadata (rebuilt each run)
* failures.json durable ledger of unresolved failures * failures.json durable ledger of unresolved failures
* *
@@ -94,18 +94,6 @@ const USER_ID = 42;
// Decrypted-byte length each stub original writes, keyed by fileID. // Decrypted-byte length each stub original writes, keyed by fileID.
const SIZE_BY_ID: Record<number, number> = { 100: 3000, 101: 2000, 200: 1500 }; const SIZE_BY_ID: Record<number, number> = { 100: 3000, 101: 2000, 200: 1500 };
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
// stores times, so the machine's time zone cannot move it to another day. Its
// original is saved in the folder `DAY`, named `2026-03-01.<fileID>.<ext>`.
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
const DAY = join("2026", "2026-03", "2026-03-01");
// Where the backup in `outDir` saves `name` (`<fileID>.<ext>`), and the target
// of an album folder's symlink to it.
const saved = (outDir: string, name: string): string =>
join(outDir, DAY, `2026-03-01.${name}`);
const linkTo = (name: string): string => `../../${DAY}/2026-03-01.${name}`;
const collection = (id: number, name: string): Collection => ({ const collection = (id: number, name: string): Collection => ({
id, id,
ownerID: USER_ID, ownerID: USER_ID,
@@ -124,7 +112,7 @@ const file = (id: number, collectionID: number, title: string): EnteFile => ({
metadata: { metadata: {
title, title,
fileType: "image", fileType: "image",
creationTime: TAKEN, creationTime: 1,
modificationTime: 1, modificationTime: 1,
}, },
file: { decryptionHeader: "aGVhZGVy" }, file: { decryptionHeader: "aGVhZGVy" },
@@ -265,11 +253,7 @@ describe("lib.backup", () => {
it("throws before any network when no downloadDirectory is given", async () => { it("throws before any network when no downloadDirectory is given", async () => {
const source = stubSource(); const source = stubSource();
const lib = await openLibrary(source); const lib = await openLibrary(source);
// The library always has a download directory, so only an empty one await expect(lib.backup()).rejects.toThrow(/downloadDirectory/i);
// given to backup() reaches runBackup as none.
await expect(lib.backup({ downloadDirectory: "" })).rejects.toThrow(
/downloadDirectory/i,
);
expect(source.originalCalls).toBe(0); expect(source.originalCalls).toBe(0);
lib.close(); lib.close();
}); });
@@ -287,22 +271,28 @@ describe("lib.backup", () => {
expect(result.failed).toBe(0); expect(result.failed).toBe(0);
expect(result.errors).toEqual([]); expect(result.errors).toEqual([]);
// Originals at YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>. // Originals under originals/<fileID>.<ext>.
expect(readFileSync(saved(outDir, "100.jpg")).length).toBe(3000); expect(readFileSync(join(outDir, "originals", "100.jpg")).length).toBe(
expect(readFileSync(saved(outDir, "101.jpg")).length).toBe(2000); 3000,
expect(readFileSync(saved(outDir, "200.png")).length).toBe(1500); );
expect(readFileSync(join(outDir, "originals", "101.jpg")).length).toBe(
2000,
);
expect(readFileSync(join(outDir, "originals", "200.png")).length).toBe(
1500,
);
// Per-file metadata sidecar. // Per-file metadata sidecar.
const sidecar = JSON.parse( const sidecar = JSON.parse(
readFileSync(saved(outDir, "100.json"), "utf-8"), readFileSync(join(outDir, "originals", "100.json"), "utf-8"),
); );
expect(sidecar.id).toBe(100); expect(sidecar.id).toBe(100);
expect(sidecar.metadata.title).toBe("beach.jpg"); expect(sidecar.metadata.title).toBe("beach.jpg");
// Collection dirs contain symlinks into the date folders. // Collection dirs contain symlinks into ../originals.
const beach = join(outDir, "collections", "Vacation", "beach.jpg"); const beach = join(outDir, "collections", "Vacation", "beach.jpg");
expect(lstatSync(beach).isSymbolicLink()).toBe(true); expect(lstatSync(beach).isSymbolicLink()).toBe(true);
expect(readlinkSync(beach)).toContain(DAY); expect(readlinkSync(beach)).toContain("originals");
expect(readFileSync(beach).length).toBe(3000); expect(readFileSync(beach).length).toBe(3000);
// Per-collection metadata JSON. // Per-collection metadata JSON.
@@ -326,7 +316,7 @@ describe("lib.backup", () => {
expect(result.failed).toBe(0); expect(result.failed).toBe(0);
// The title has no usable extension, so the original is `.bin`. // The title has no usable extension, so the original is `.bin`.
expect(existsSync(saved(outDir, "300.bin"))).toBe(true); expect(existsSync(join(outDir, "originals", "300.bin"))).toBe(true);
const link = join( const link = join(
outDir, outDir,
"collections", "collections",
@@ -376,9 +366,9 @@ describe("lib.backup", () => {
expect(result.errors[0]!.title).toBe("sunset.jpg"); expect(result.errors[0]!.title).toBe("sunset.jpg");
// The two good files are on disk; the failed one is not. // The two good files are on disk; the failed one is not.
expect(existsSync(saved(outDir, "100.jpg"))).toBe(true); expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(true);
expect(existsSync(saved(outDir, "200.png"))).toBe(true); expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
expect(existsSync(saved(outDir, "101.jpg"))).toBe(false); expect(existsSync(join(outDir, "originals", "101.jpg"))).toBe(false);
expect( expect(
existsSync(join(outDir, "collections", "Vacation", "sunset.jpg")), existsSync(join(outDir, "collections", "Vacation", "sunset.jpg")),
).toBe(false); ).toBe(false);
@@ -413,7 +403,7 @@ describe("lib.backup", () => {
const r3 = await lib.backup({ downloadDirectory: outDir }); const r3 = await lib.backup({ downloadDirectory: outDir });
expect(r3.failed).toBe(0); expect(r3.failed).toBe(0);
expect(r3.skipped).toBe(2); expect(r3.skipped).toBe(2);
expect(existsSync(saved(outDir, "101.jpg"))).toBe(true); expect(existsSync(join(outDir, "originals", "101.jpg"))).toBe(true);
// A ledger with no remaining failures is removed. // A ledger with no remaining failures is removed.
expect(existsSync(join(outDir, "failures.json"))).toBe(false); expect(existsSync(join(outDir, "failures.json"))).toBe(false);
lib.close(); lib.close();
@@ -433,8 +423,8 @@ describe("lib.backup", () => {
const result = await lib.backup({ downloadDirectory: outDir }); const result = await lib.backup({ downloadDirectory: outDir });
// Every original still downloads despite the symlink failure. // Every original still downloads despite the symlink failure.
expect(existsSync(saved(outDir, "100.jpg"))).toBe(true); expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(true);
expect(existsSync(saved(outDir, "200.png"))).toBe(true); expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
// The other symlinks are still built. // The other symlinks are still built.
expect( expect(
lstatSync( lstatSync(
@@ -464,7 +454,7 @@ describe("lib.backup", () => {
await lib.backup({ downloadDirectory: outDir }); await lib.backup({ downloadDirectory: outDir });
// Corrupt a sidecar and delete a symlink between runs. // Corrupt a sidecar and delete a symlink between runs.
writeFileSync(saved(outDir, "100.json"), "not json"); writeFileSync(join(outDir, "originals", "100.json"), "not json");
rmSync(join(outDir, "collections", "Vacation", "beach.jpg")); rmSync(join(outDir, "collections", "Vacation", "beach.jpg"));
const result = await lib.backup({ downloadDirectory: outDir }); const result = await lib.backup({ downloadDirectory: outDir });
@@ -472,7 +462,7 @@ describe("lib.backup", () => {
// The derived views are repaired from the model. // The derived views are repaired from the model.
const sidecar = JSON.parse( const sidecar = JSON.parse(
readFileSync(saved(outDir, "100.json"), "utf-8"), readFileSync(join(outDir, "originals", "100.json"), "utf-8"),
); );
expect(sidecar.metadata.title).toBe("beach.jpg"); expect(sidecar.metadata.title).toBe("beach.jpg");
expect( expect(
@@ -495,8 +485,8 @@ describe("lib.backup", () => {
expect(result.totalFiles).toBe(1); expect(result.totalFiles).toBe(1);
expect(result.downloaded).toBe(1); expect(result.downloaded).toBe(1);
expect(existsSync(saved(outDir, "200.png"))).toBe(true); expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
expect(existsSync(saved(outDir, "100.jpg"))).toBe(false); expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(false);
expect(existsSync(join(outDir, "collections", "Work.json"))).toBe(true); expect(existsSync(join(outDir, "collections", "Work.json"))).toBe(true);
expect(existsSync(join(outDir, "collections", "Vacation.json"))).toBe( expect(existsSync(join(outDir, "collections", "Vacation.json"))).toBe(
false, false,
@@ -540,7 +530,7 @@ describe("lib.backup", () => {
expect(result.totalFiles).toBe(1); expect(result.totalFiles).toBe(1);
expect(result.failed).toBe(0); expect(result.failed).toBe(0);
expect(existsSync(saved(outDir, "200.png"))).toBe(true); expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
expect(existsSync(join(outDir, "failures.json"))).toBe(false); expect(existsSync(join(outDir, "failures.json"))).toBe(false);
lib.close(); lib.close();
}); });
@@ -582,7 +572,7 @@ describe("lib.backup", () => {
it("fetches each original once and writes it only into the backup", async () => { it("fetches each original once and writes it only into the backup", async () => {
// A backup of a 500 GB account must write 500 GB, not a copy in the // A backup of a 500 GB account must write 500 GB, not a copy in the
// cache as well: an original fetched for the backup goes straight // cache as well: an original fetched for the backup goes straight
// to its save path, and the cache records it there. // into its originals/, and the cache records it there.
const source = stubSource(); const source = stubSource();
const lib = await openLibrary(source); const lib = await openLibrary(source);
const outDir = join(root, "backup"); const outDir = join(root, "backup");
@@ -592,78 +582,23 @@ describe("lib.backup", () => {
expect(result.downloaded).toBe(3); expect(result.downloaded).toBe(3);
expect(source.originalCalls).toBe(3); expect(source.originalCalls).toBe(3);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]); expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
const stored = readdirSync(join(outDir, DAY)).filter( const stored = readdirSync(join(outDir, "originals")).filter(
(name) => !name.endsWith(".json"), (name) => !name.endsWith(".json"),
); );
expect(stored.sort()).toEqual([ expect(stored.sort()).toEqual(["100.jpg", "101.jpg", "200.png"]);
"2026-03-01.100.jpg",
"2026-03-01.101.jpg",
"2026-03-01.200.png",
]);
// The cache counts the backup's copy as present: reading the // The cache counts the backup's copy as present: reading the
// original afterwards fetches nothing and answers with that copy. // original afterwards fetches nothing and answers with that copy.
const read = await lib.photos.byID({ fileID: 100 })!.original(); const read = await lib.photos.byID({ fileID: 100 })!.original();
expect(read.path).toBe(saved(outDir, "100.jpg")); expect(read.path).toBe(join(outDir, "originals", "100.jpg"));
expect(source.originalCalls).toBe(3); expect(source.originalCalls).toBe(3);
await lib.close(); await lib.close();
}); });
it("saves a file in two albums at its photo's save path and links it from both", async () => {
// The date edited in Ente has reached Work's copy, synced later, but
// Vacation's copy still has an earlier edit.
const older = file(100, 1, "beach.jpg");
older.pubMagicMetadata = {
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
};
const newer = file(100, 2, "beach.jpg");
newer.updationTime = 2;
newer.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
class SharedFileClient extends MockClient {
override async filesSince(args: {
collectionID: number;
}): Promise<FilesPage> {
const files = [args.collectionID === 1 ? older : newer];
return { files, deleted: [], cursor: 1 };
}
}
const outDir = join(root, "backup");
const lib = await Library.open({
client: new SharedFileClient(),
cacheDirectory: join(root, "cache"),
downloadDirectory: outDir,
contentSource: stubSource(),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const photo = lib.photos.byID({ fileID: 100 })!;
const result = await lib.backup();
expect(result.totalFiles).toBe(1);
expect(result.downloaded).toBe(1);
expect(result.failed).toBe(0);
expect(photo.savePath).toBe(
join(outDir, "2026", "2026-04", "2026-04-15", "2026-04-15.100.jpg"),
);
expect(photo.isLocal).toBe(true);
expect(existsSync(join(outDir, "2026", "2026-02"))).toBe(false);
const target = "../../2026/2026-04/2026-04-15/2026-04-15.100.jpg";
for (const album of ["Vacation", "Work"]) {
expect(
readlinkSync(join(outDir, "collections", album, "beach.jpg")),
).toBe(target);
}
await lib.close();
});
it("fsyncs an original copied from the cache before the rename and its directory after", async () => { it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource()); const lib = await openLibrary(stubSource());
const outDir = join(root, "backup"); const outDir = join(root, "backup");
const day = join(outDir, DAY); const originals = join(outDir, "originals");
const dest = saved(outDir, "100.jpg"); const dest = join(originals, "100.jpg");
// Only an original already in the cache is copied into the backup; // Only an original already in the cache is copied into the backup;
// one fetched for the backup is written there by the download writer. // one fetched for the backup is written there by the download writer.
await lib.photos.byID({ fileID: 100 })!.original(); await lib.photos.byID({ fileID: 100 })!.original();
@@ -674,50 +609,34 @@ describe("lib.backup", () => {
const at = fsEvents.indexOf(`rename:${dest}`); const at = fsEvents.indexOf(`rename:${dest}`);
expect(at).toBeGreaterThan(0); expect(at).toBeGreaterThan(0);
expect(fsEvents[at - 1]).toMatch( expect(fsEvents[at - 1]).toMatch(
/^sync:.*\/\.quak-backup-2026-03-01\.100\.jpg-\d+-[0-9a-z]*\.tmp$/, /^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
); );
expect(fsEvents[at + 1]).toBe(`sync:${day}`); expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
lib.close(); lib.close();
}); });
it("removes temp files left by a killed backup but not those of one still running", async () => { it("removes temp files left by a killed backup but not those of one still running", async () => {
const outDir = join(root, "backup"); const outDir = join(root, "backup");
const day = join(outDir, DAY); const originals = join(outDir, "originals");
mkdirSync(day, { recursive: true }); mkdirSync(originals, { recursive: true });
// A child that has already exited: its process ID is not running. // A child that has already exited: its process ID is not running.
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid; const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-2026-03-01.100.jpg-${exitedPID}-abc123.tmp`; const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
// This test's own process stands in for a backup running at the same // This test's own process stands in for a backup running at the same
// time. // time.
const inProgress = `.quak-backup-2026-03-01.101.jpg-${process.pid}-def456.tmp`; const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(day, leftover), "partial"); writeFileSync(join(originals, leftover), "partial");
writeFileSync(join(day, inProgress), "partial"); writeFileSync(join(originals, inProgress), "partial");
const lib = await openLibrary(stubSource()); const lib = await openLibrary(stubSource());
await lib.backup({ downloadDirectory: outDir }); await lib.backup({ downloadDirectory: outDir });
const names = readdirSync(day); const names = readdirSync(originals);
expect(names).not.toContain(leftover); expect(names).not.toContain(leftover);
expect(names).toContain(inProgress); expect(names).toContain(inProgress);
lib.close(); lib.close();
}); });
it("removes temp files left in a date folder no file in the backup is saved in", async () => {
const outDir = join(root, "backup");
// As for a file since deleted, or given another date, after a run was
// killed while writing it.
const otherDay = join(outDir, "2025", "2025-01", "2025-01-02");
mkdirSync(otherDay, { recursive: true });
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
writeFileSync(join(otherDay, `.quak-${exitedPID}-abc123.tmp`), "x");
const lib = await openLibrary(stubSource());
await lib.backup({ downloadDirectory: outDir });
expect(readdirSync(otherDay)).toEqual([]);
lib.close();
});
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => { it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
const outDir = join(root, "backup"); const outDir = join(root, "backup");
const thumbnails = join(outDir, "thumbnails"); const thumbnails = join(outDir, "thumbnails");
@@ -790,7 +709,7 @@ describe("the refresh before a backup", () => {
const result = await backup; const result = await backup;
expect(result.totalFiles).toBe(4); expect(result.totalFiles).toBe(4);
expect(existsSync(saved(outDir, "300.jpg"))).toBe(true); expect(existsSync(join(outDir, "originals", "300.jpg"))).toBe(true);
await lib.close(); await lib.close();
}); });
@@ -809,7 +728,7 @@ describe("the refresh before a backup", () => {
expect(source.originalCalls).toBe(0); expect(source.originalCalls).toBe(0);
expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore); expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore);
expect(existsSync(join(outDir, DAY))).toBe(false); expect(existsSync(join(outDir, "originals"))).toBe(false);
await lib.close(); await lib.close();
}); });
@@ -900,13 +819,13 @@ describe("backup album folders", () => {
expect(result.failed).toBe(0); expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([ expect(tree(outDir)).toEqual([
"Trip (10)/", "Trip (10)/",
`Trip (10)/IMG_0001 (1).JPG -> ${linkTo("1.JPG")}`, "Trip (10)/IMG_0001 (1).JPG -> ../../originals/1.JPG",
`Trip (10)/IMG_0001 (2).JPG -> ${linkTo("2.JPG")}`, "Trip (10)/IMG_0001 (2).JPG -> ../../originals/2.JPG",
`Trip (10)/img_0001 (4).jpg -> ${linkTo("4.jpg")}`, "Trip (10)/img_0001 (4).jpg -> ../../originals/4.jpg",
`Trip (10)/other.jpg -> ${linkTo("3.jpg")}`, "Trip (10)/other.jpg -> ../../originals/3.jpg",
"Trip (10).json", "Trip (10).json",
"Trip (11)/", "Trip (11)/",
`Trip (11)/other.jpg -> ${linkTo("3.jpg")}`, "Trip (11)/other.jpg -> ../../originals/3.jpg",
"Trip (11).json", "Trip (11).json",
]); ]);
expect(albumID(outDir, "Trip (10).json")).toBe(10); expect(albumID(outDir, "Trip (10).json")).toBe(10);
@@ -939,14 +858,14 @@ describe("backup album folders", () => {
expect(result.failed).toBe(0); expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([ expect(tree(outDir)).toEqual([
"Trip (10)/", "Trip (10)/",
`Trip (10)/IMG (6) (5).JPG -> ${linkTo("5.JPG")}`, "Trip (10)/IMG (6) (5).JPG -> ../../originals/5.JPG",
`Trip (10)/IMG (6).JPG -> ${linkTo("6.JPG")}`, "Trip (10)/IMG (6).JPG -> ../../originals/6.JPG",
`Trip (10)/IMG (7).JPG -> ${linkTo("7.JPG")}`, "Trip (10)/IMG (7).JPG -> ../../originals/7.JPG",
"Trip (10).json", "Trip (10).json",
"Trip (11)/", "Trip (11)/",
`Trip (11)/a.jpg -> ${linkTo("8.jpg")}`, "Trip (11)/a.jpg -> ../../originals/8.jpg",
"Trip (11) (12)/", "Trip (11) (12)/",
`Trip (11) (12)/b.jpg -> ${linkTo("9.jpg")}`, "Trip (11) (12)/b.jpg -> ../../originals/9.jpg",
"Trip (11) (12).json", "Trip (11) (12).json",
"Trip (11).json", "Trip (11).json",
]); ]);
@@ -1012,13 +931,13 @@ describe("backup album folders", () => {
expect(scoped.failed).toBe(0); expect(scoped.failed).toBe(0);
expect(before).toEqual([ expect(before).toEqual([
"Trip (10)/", "Trip (10)/",
`Trip (10)/a.jpg -> ${linkTo("1.jpg")}`, "Trip (10)/a.jpg -> ../../originals/1.jpg",
"Trip (10).json", "Trip (10).json",
"Work/", "Work/",
`Work/c.jpg -> ${linkTo("3.jpg")}`, "Work/c.jpg -> ../../originals/3.jpg",
"Work.json", "Work.json",
"trip (11)/", "trip (11)/",
`trip (11)/b.jpg -> ${linkTo("2.jpg")}`, "trip (11)/b.jpg -> ../../originals/2.jpg",
"trip (11).json", "trip (11).json",
]); ]);
expect(tree(outDir)).toEqual(before); expect(tree(outDir)).toEqual(before);
@@ -1071,13 +990,13 @@ describe("backup album folders", () => {
"Mine/", "Mine/",
"Mine/keep.txt", "Mine/keep.txt",
"Office/", "Office/",
`Office/a.jpg -> ${linkTo("5.jpg")}`, "Office/a.jpg -> ../../originals/5.jpg",
"Office.json", "Office.json",
"Trip/", "Trip/",
`Trip/IMG_0001.JPG -> ${linkTo("1.JPG")}`, "Trip/IMG_0001.JPG -> ../../originals/1.JPG",
"Trip/mine -> ../elsewhere", "Trip/mine -> ../elsewhere",
"Trip/notes.txt", "Trip/notes.txt",
`Trip/other.jpg -> ${linkTo("3.jpg")}`, "Trip/other.jpg -> ../../originals/3.jpg",
"Trip.json", "Trip.json",
"Work/", "Work/",
"Work/keep.txt", "Work/keep.txt",
@@ -1086,7 +1005,7 @@ describe("backup album folders", () => {
}); });
// One album backed up, then a folder the user made beside it holding a // One album backed up, then a folder the user made beside it holding a
// symlink to an original, with `json` (if given) as its sibling JSON. // symlink into originals/, with `json` (if given) as its sibling JSON.
const backupWithUserFolder = async ( const backupWithUserFolder = async (
json: string | undefined, json: string | undefined,
): Promise<{ outDir: string; failed: number }> => { ): Promise<{ outDir: string; failed: number }> => {
@@ -1100,7 +1019,10 @@ describe("backup album folders", () => {
await runBackup(lib, { downloadDirectory: outDir }); await runBackup(lib, { downloadDirectory: outDir });
const collectionsDir = join(outDir, "collections"); const collectionsDir = join(outDir, "collections");
mkdirSync(join(collectionsDir, "Mine")); mkdirSync(join(collectionsDir, "Mine"));
symlinkSync(linkTo("1.jpg"), join(collectionsDir, "Mine", "a.jpg")); symlinkSync(
"../../originals/1.jpg",
join(collectionsDir, "Mine", "a.jpg"),
);
if (json !== undefined) { if (json !== undefined) {
writeFileSync(join(collectionsDir, "Mine.json"), json); writeFileSync(join(collectionsDir, "Mine.json"), json);
} }
@@ -1114,9 +1036,9 @@ describe("backup album folders", () => {
expect(failed).toBe(0); expect(failed).toBe(0);
expect(tree(outDir)).toEqual([ expect(tree(outDir)).toEqual([
"Mine/", "Mine/",
`Mine/a.jpg -> ${linkTo("1.jpg")}`, "Mine/a.jpg -> ../../originals/1.jpg",
"Trip/", "Trip/",
`Trip/a.jpg -> ${linkTo("1.jpg")}`, "Trip/a.jpg -> ../../originals/1.jpg",
"Trip.json", "Trip.json",
]); ]);
}); });
@@ -1128,10 +1050,10 @@ describe("backup album folders", () => {
expect(failed).toBe(0); expect(failed).toBe(0);
expect(tree(outDir)).toEqual([ expect(tree(outDir)).toEqual([
"Mine/", "Mine/",
`Mine/a.jpg -> ${linkTo("1.jpg")}`, "Mine/a.jpg -> ../../originals/1.jpg",
"Mine.json", "Mine.json",
"Trip/", "Trip/",
`Trip/a.jpg -> ${linkTo("1.jpg")}`, "Trip/a.jpg -> ../../originals/1.jpg",
"Trip.json", "Trip.json",
]); ]);
expect( expect(
@@ -1165,16 +1087,23 @@ describe("backup of live photos", () => {
const open = (files: EnteFile[], bodies: Map<number, Uint8Array>) => const open = (files: EnteFile[], bodies: Map<number, Uint8Array>) =>
openLibrary(cdnSource(bodies), new TripClient(files)); openLibrary(cdnSource(bodies), new TripClient(files));
const stored = [ // What an earlier version stored for live photo 500: the ZIP under the
"2026-03-01.500.heic", // image's name, and its link.
"2026-03-01.500.json", const earlierZIP = (outDir: string): void => {
"2026-03-01.500.livephoto.json", mkdirSync(join(outDir, "originals"), { recursive: true });
"2026-03-01.500.mov", mkdirSync(join(outDir, "collections", "Trip"), { recursive: true });
]; writeFileSync(join(outDir, "originals", "500.HEIC"), livePhotoZip());
symlinkSync(
"../../originals/500.HEIC",
join(outDir, "collections", "Trip", "IMG_0500.HEIC"),
);
};
const stored = ["500.heic", "500.json", "500.livephoto.json", "500.mov"];
const linked = [ const linked = [
"Trip/", "Trip/",
`Trip/IMG_0500.heic -> ${linkTo("500.heic")}`, "Trip/IMG_0500.heic -> ../../originals/500.heic",
`Trip/IMG_0500.mov -> ${linkTo("500.mov")}`, "Trip/IMG_0500.mov -> ../../originals/500.mov",
"Trip.json", "Trip.json",
]; ];
@@ -1184,15 +1113,16 @@ describe("backup of live photos", () => {
); );
const lib = await open([live], new Map([[500, body]])); const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup"); const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const result = await lib.backup({ downloadDirectory: outDir }); const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 }); expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored); expect(readdirSync(originals).sort()).toEqual(stored);
expect(readFileSync(saved(outDir, "500.heic"))).toEqual( expect(readFileSync(join(originals, "500.heic"))).toEqual(
Buffer.from(IMAGE), Buffer.from(IMAGE),
); );
expect(readFileSync(saved(outDir, "500.mov"))).toEqual( expect(readFileSync(join(originals, "500.mov"))).toEqual(
Buffer.from(VIDEO), Buffer.from(VIDEO),
); );
expect(tree(outDir)).toEqual(linked); expect(tree(outDir)).toEqual(linked);
@@ -1219,22 +1149,39 @@ describe("backup of live photos", () => {
expect(tree(outDir)).toEqual([ expect(tree(outDir)).toEqual([
"Trip/", "Trip/",
`Trip/IMG_0001 (500).heic -> ${linkTo("500.heic")}`, "Trip/IMG_0001 (500).heic -> ../../originals/500.heic",
`Trip/IMG_0001 (500).mov -> ${linkTo("500.mov")}`, "Trip/IMG_0001 (500).mov -> ../../originals/500.mov",
`Trip/IMG_0001 (501).heic -> ${linkTo("501.heic")}`, "Trip/IMG_0001 (501).heic -> ../../originals/501.heic",
`Trip/IMG_0001 (501).mov -> ${linkTo("501.mov")}`, "Trip/IMG_0001 (501).mov -> ../../originals/501.mov",
"Trip.json", "Trip.json",
]); ]);
await lib.close(); await lib.close();
}); });
it("stores nothing for a live photo that fails its hash", async () => { it("replaces the ZIP an earlier version stored, and its link", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const outDir = join(root, "backup");
earlierZIP(outDir);
const lib = await open([live], new Map([[500, body]]));
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(tree(outDir)).toEqual(linked);
await lib.close();
});
it("stores nothing for a live photo that fails its hash, and keeps what was there", async () => {
const { file: live, body } = await asLivePhoto( const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"), file(500, 10, "IMG_0500.HEIC"),
livePhotoZip(), livePhotoZip(),
"not:the recorded hash", "not:the recorded hash",
); );
const outDir = join(root, "backup"); const outDir = join(root, "backup");
earlierZIP(outDir);
const lib = await open([live], new Map([[500, body]])); const lib = await open([live], new Map([[500, body]]));
const result = await lib.backup({ downloadDirectory: outDir }); const result = await lib.backup({ downloadDirectory: outDir });
@@ -1242,8 +1189,12 @@ describe("backup of live photos", () => {
expect(result).toMatchObject({ downloaded: 0, failed: 1 }); expect(result).toMatchObject({ downloaded: 0, failed: 1 });
expect(result.errors.map((e) => e.fileID)).toEqual([500]); expect(result.errors.map((e) => e.fileID)).toEqual([500]);
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]); expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
expect(readdirSync(join(outDir, DAY))).toEqual([]); expect(readdirSync(join(outDir, "originals"))).toEqual(["500.HEIC"]);
expect(tree(outDir)).toEqual(["Trip/", "Trip.json"]); expect(tree(outDir)).toEqual([
"Trip/",
"Trip/IMG_0500.HEIC -> ../../originals/500.HEIC",
"Trip.json",
]);
await lib.close(); await lib.close();
}); });
@@ -1258,8 +1209,8 @@ describe("backup of live photos", () => {
const result = await lib.backup({ downloadDirectory: outDir }); const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 }); expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored); expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(readFileSync(saved(outDir, "500.mov"))).toEqual( expect(readFileSync(join(outDir, "originals", "500.mov"))).toEqual(
Buffer.from(VIDEO), Buffer.from(VIDEO),
); );
expect(tree(outDir)).toEqual(linked); expect(tree(outDir)).toEqual(linked);
@@ -1268,6 +1219,23 @@ describe("backup of live photos", () => {
await lib.close(); await lib.close();
}); });
it("replaces an earlier ZIP and its link with the image and video the cache holds", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const lib = await open([live], new Map([[500, body]]));
await lib.photos.byID({ fileID: 500 })!.original();
const outDir = join(root, "backup");
earlierZIP(outDir);
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(tree(outDir)).toEqual(linked);
await lib.close();
});
it.each(["missing", "empty"])( it.each(["missing", "empty"])(
"fetches a live photo again when the video its JSON file names is %s", "fetches a live photo again when the video its JSON file names is %s",
async (state) => { async (state) => {
@@ -1277,7 +1245,7 @@ describe("backup of live photos", () => {
const lib = await open([live], new Map([[500, body]])); const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup"); const outDir = join(root, "backup");
await lib.backup({ downloadDirectory: outDir }); await lib.backup({ downloadDirectory: outDir });
const video = saved(outDir, "500.mov"); const video = join(outDir, "originals", "500.mov");
if (state === "missing") rmSync(video); if (state === "missing") rmSync(video);
else writeFileSync(video, ""); else writeFileSync(video, "");
@@ -1303,7 +1271,7 @@ describe("backup of live photos", () => {
const lib = await open([live], new Map([[500, body]])); const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup"); const outDir = join(root, "backup");
await lib.backup({ downloadDirectory: outDir }); await lib.backup({ downloadDirectory: outDir });
const image = saved(outDir, "500.heic"); const image = join(outDir, "originals", "500.heic");
if (state === "missing") rmSync(image); if (state === "missing") rmSync(image);
else writeFileSync(image, ""); else writeFileSync(image, "");
@@ -1342,8 +1310,8 @@ describe("backup of live photos", () => {
const read = await reader.photos.byID({ fileID: 500 })!.original(); const read = await reader.photos.byID({ fileID: 500 })!.original();
expect(read).toEqual({ expect(read).toEqual({
path: saved(outDir, "500.heic"), path: join(outDir, "originals", "500.heic"),
videoPath: saved(outDir, "500.mov"), videoPath: join(outDir, "originals", "500.mov"),
bytes: IMAGE.length, bytes: IMAGE.length,
}); });
await reader.close(); await reader.close();
test/cli/commands.test.ts
+1 -1
View File
@@ -757,7 +757,7 @@ describe("backup", () => {
expect(code).toBe(1); expect(code).toBe(1);
expect(runText).toBe("quak: HTTP 401 from server\n"); expect(runText).toBe("quak: HTTP 401 from server\n");
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n"); expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
expect(existsSync(dir)).toBe(false); expect(existsSync(join(dir, "originals"))).toBe(false);
}); });
}); });
test/library/content-eviction.test.ts
-1
View File
@@ -142,7 +142,6 @@ const buildCache = (args: {
pools: new RequestPools(), pools: new RequestPools(),
source, source,
cacheDirectory: cacheDir, cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id), getFile: (id) => byID.get(id),
statfs: args.statfs, statfs: args.statfs,
cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes, cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes,
test/library/content-library.test.ts
+13 -237
View File
@@ -6,18 +6,16 @@
* `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and * `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and
* a cached path shows up on the projected record. A library opened without a * a cached path shows up on the projected record. A library opened without a
* content source leaves those methods throwing rather than silently doing * content source leaves those methods throwing rather than silently doing
* nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `download()`, * nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `content()`,
* `content()`, `exif()` and the methods that each return one field of `exif()`. * `exif()` and the methods that each return one field of `exif()`.
*/ */
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { import {
mkdtempSync, mkdtempSync,
readdirSync, readdirSync,
readFileSync,
rmSync, rmSync,
existsSync, existsSync,
statSync,
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
@@ -40,12 +38,6 @@ import {
const USER_ID = 7; const USER_ID = 7;
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
// stores times, so the machine's time zone cannot move it to another day; it
// is saved in the folder `DAY`.
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
const DAY = join("2026", "2026-03", "2026-03-01");
const collection = (id: number): Collection => ({ const collection = (id: number): Collection => ({
id, id,
ownerID: USER_ID, ownerID: USER_ID,
@@ -64,7 +56,7 @@ const file = (id: number, collectionID: number): EnteFile => ({
metadata: { metadata: {
title: `file-${id}.jpg`, title: `file-${id}.jpg`,
fileType: "image", fileType: "image",
creationTime: TAKEN, creationTime: 1,
modificationTime: 1, modificationTime: 1,
}, },
file: { decryptionHeader: "aGVhZGVy" }, file: { decryptionHeader: "aGVhZGVy" },
@@ -232,9 +224,6 @@ describe("Library content wiring", () => {
await expect( await expect(
lib.photos.byID({ fileID: 1 })!.thumbnail(), lib.photos.byID({ fileID: 1 })!.thumbnail(),
).rejects.toThrow(/content cache/i); ).rejects.toThrow(/content cache/i);
await expect(
lib.photos.byID({ fileID: 1 })!.download(),
).rejects.toThrow(/content cache/i);
await expect( await expect(
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }), lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
).rejects.toThrow(/content cache/i); ).rejects.toThrow(/content cache/i);
@@ -352,7 +341,7 @@ describe("Photo save path, local copy, content and EXIF", () => {
it("names where a backup writes the original, which is local once the backup has written it", async () => { it("names where a backup writes the original, which is local once the backup has written it", async () => {
const lib = await open(); const lib = await open();
const photo = lib.photos.byID({ fileID: 1 })!; const photo = lib.photos.byID({ fileID: 1 })!;
const savePath = join(root, "backup", DAY, "2026-03-01.1.jpg"); const savePath = join(root, "backup", "originals", "1.jpg");
expect(photo.savePath).toBe(savePath); expect(photo.savePath).toBe(savePath);
expect(photo.isLocal).toBe(false); expect(photo.isLocal).toBe(false);
@@ -370,150 +359,23 @@ describe("Photo save path, local copy, content and EXIF", () => {
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe( expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
true, true,
); );
expect(existsSync(photo.savePath)).toBe(false); expect(existsSync(photo.savePath!)).toBe(false);
expect(photo.isLocal).toBe(false); expect(photo.isLocal).toBe(false);
await lib.close(); await lib.close();
}); });
it("saves under photos/ in the working directory at open without a download directory", async () => { it("has no save path and is not local without a download directory", async () => {
const cwd = vi.spyOn(process, "cwd").mockReturnValue(root);
const lib = await open({ downloadDirectory: undefined }); const lib = await open({ downloadDirectory: undefined });
cwd.mockRestore();
const photo = lib.photos.byID({ fileID: 1 })!; const photo = lib.photos.byID({ fileID: 1 })!;
expect(lib.downloadDirectory).toBe(join(root, "photos")); expect(photo.savePath).toBeUndefined();
expect(photo.savePath).toBe(
join(root, "photos", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(false); expect(photo.isLocal).toBe(false);
await lib.close(); await lib.close();
}); });
it("refuses an empty download directory", async () => { it("has no save path and is not local without a content source", async () => {
await expect(open({ downloadDirectory: "" })).rejects.toThrow(
/downloadDirectory is empty/,
);
});
it("keeps a photo's save path after a refresh removes its file", async () => {
// The same account, whose album is deleted on the second refresh.
class AlbumDeletedClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
if (!this.served) return super.collectionsSince();
return { collections: [], deleted: [1], cursor: 2 };
}
}
const lib = await open({ client: new AlbumDeletedClient() });
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.download();
await lib.fresh();
expect(lib.photos.byID({ fileID: 1 })).toBeUndefined();
expect(photo.savePath).toBe(
join(root, "backup", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("dates the save path and download() by the membership its takenAt comes from", async () => {
// One file in two albums. The date edited in Ente has reached album 2's
// copy, synced later, but album 1 still has an earlier edit.
const older = file(1, 1);
older.pubMagicMetadata = {
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
};
const newer = file(1, 2);
newer.updationTime = 2;
newer.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
const client = {
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
collectionsSince: async (): Promise<CollectionsPage> => ({
collections: [collection(1), collection(2)],
deleted: [],
cursor: 1,
}),
filesSince: async (args: {
collectionID: number;
}): Promise<FilesPage> => ({
files: [args.collectionID === 1 ? older : newer],
deleted: [],
cursor: 1,
}),
};
const lib = await open({ client });
const photo = lib.photos.byID({ fileID: 1 })!;
expect(photo.takenAt).toBe(new Date(2026, 3, 15, 12).getTime());
expect(photo.savePath).toBe(
join(
root,
"backup",
"2026",
"2026-04",
"2026-04-15",
"2026-04-15.1.jpg",
),
);
// download() writes to that same path, so the photo is then local.
const saved = await photo.download();
expect(saved.path).toBe(photo.savePath);
expect(photo.isLocal).toBe(true);
expect(existsSync(join(root, "backup", "2026", "2026-02"))).toBe(false);
await lib.close();
});
it("downloads a photo held across a refresh that edits its date to the save path it names", async () => {
// The same account, whose second refresh brings a date edited in Ente.
const edited = file(1, 1);
edited.updationTime = 2;
edited.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
class DateEditedClient extends MockClient {
refreshes = 0;
override async collectionsSince(): Promise<CollectionsPage> {
this.refreshes++;
return {
collections: [
{ ...collection(1), updationTime: this.refreshes },
],
deleted: [],
cursor: this.refreshes,
};
}
override async filesSince(): Promise<FilesPage> {
return {
files: [this.refreshes === 1 ? file(1, 1) : edited],
deleted: [],
cursor: this.refreshes,
};
}
}
const lib = await open({ client: new DateEditedClient() });
const photo = lib.photos.byID({ fileID: 1 })!;
await lib.fresh();
expect(lib.photos.byID({ fileID: 1 })!.takenAt).toBe(
new Date(2026, 3, 15, 12).getTime(),
);
const saved = await photo.download();
expect(saved.path).toBe(photo.savePath);
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("has a save path, and is not local, without a content source", async () => {
const lib = await open({ contentSource: undefined }); const lib = await open({ contentSource: undefined });
const photo = lib.photos.byID({ fileID: 1 })!; const photo = lib.photos.byID({ fileID: 1 })!;
expect(photo.savePath).toBe( expect(photo.savePath).toBeUndefined();
join(root, "backup", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(false); expect(photo.isLocal).toBe(false);
await lib.close(); await lib.close();
}); });
@@ -525,104 +387,18 @@ describe("Photo save path, local copy, content and EXIF", () => {
contentSource: cdnSource(new Map([[1, body]])), contentSource: cdnSource(new Map([[1, body]])),
}); });
const photo = lib.photos.byID({ fileID: 1 })!; const photo = lib.photos.byID({ fileID: 1 })!;
const day = join(root, "backup", DAY); const originals = join(root, "backup", "originals");
// Until then the name comes from the title, file-1.jpg; the backup // Until then the name comes from the title, file-1.jpg; the backup
// stores the image with the extension found inside the live photo. // stores the image with the extension found inside the live photo.
expect(photo.savePath).toBe(join(day, "2026-03-01.1.jpg")); expect(photo.savePath).toBe(join(originals, "1.jpg"));
await lib.backup(); await lib.backup();
expect(photo.savePath).toBe(join(day, "2026-03-01.1.heic")); expect(photo.savePath).toBe(join(originals, "1.heic"));
expect(photo.isLocal).toBe(true); expect(photo.isLocal).toBe(true);
expect(await photo.content()).toEqual(Buffer.from(IMAGE)); expect(await photo.content()).toEqual(Buffer.from(IMAGE));
await lib.close(); await lib.close();
}); });
it("downloads an original the cache holds by copying it, without fetching it again", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.original();
expect(source.originalCalls()).toBe(1);
expect(photo.isLocal).toBe(false);
const saved = await photo.download();
expect(source.originalCalls()).toBe(1);
expect(saved).toEqual({
path: photo.savePath,
bytes: "orig-bytes".length,
});
expect(readFileSync(photo.savePath, "utf-8")).toBe("orig-bytes");
expect(photo.isLocal).toBe(true);
// The cache keeps its own copy.
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
true,
);
await lib.close();
});
it("downloads an original the cache does not hold straight to its save path", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
const saved = await photo.download();
expect(source.originalCalls()).toBe(1);
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
expect(readFileSync(saved.path, "utf-8")).toBe("orig-bytes");
expect(photo.isLocal).toBe(true);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
await lib.close();
});
it("does nothing when download() finds the original already at its save path", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
const first = await photo.download();
const written = statSync(first.path).ino;
const second = await photo.download();
expect(second).toEqual(first);
expect(source.originalCalls()).toBe(1);
expect(statSync(second.path).ino).toBe(written);
await lib.close();
});
it("downloads a live photo the cache holds as its image, its video and the JSON file naming them", async () => {
const { file: live, body } = await asLivePhoto(file(1, 1));
const lib = await open({
client: new FilesClient([live]),
contentSource: cdnSource(new Map([[1, body]])),
});
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.original();
const day = join(root, "backup", DAY);
const saved = await photo.download();
expect(saved).toEqual({
path: join(day, "2026-03-01.1.heic"),
videoPath: join(day, "2026-03-01.1.mov"),
bytes: IMAGE.length,
});
expect(readdirSync(day).sort()).toEqual([
"2026-03-01.1.heic",
"2026-03-01.1.livephoto.json",
"2026-03-01.1.mov",
]);
expect(
JSON.parse(
readFileSync(join(day, "2026-03-01.1.livephoto.json"), "utf-8"),
),
).toEqual({ image: "2026-03-01.1.heic", video: "2026-03-01.1.mov" });
expect(photo.savePath).toBe(saved.path);
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("reads the common EXIF fields of a JPEG original", async () => { it("reads the common EXIF fields of a JPEG original", async () => {
const lib = await open({ contentSource: stubSource(JPEG_WITH_EXIF) }); const lib = await open({ contentSource: stubSource(JPEG_WITH_EXIF) });
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({ expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({
test/library/content.test.ts
+6 -50
View File
@@ -7,8 +7,8 @@
* 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail` * 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail`
* fetches through the request pool and stores the bytes; the next finds the * fetches through the request pool and stores the bytes; the next finds the
* file present and returns its path with a single `skipped` event and no * file present and returns its path with a single `skipped` event and no
* network. A file already stored at its save path under the * network. A file already sitting in the backup `downloadDirectory` counts
* `downloadDirectory` counts as present too. * as present too.
* 2. **Present-means-complete.** Content appears only by the streaming atomic * 2. **Present-means-complete.** Content appears only by the streaming atomic
* writer's rename, so a file that exists is whole. The directory listing * writer's rename, so a file that exists is whole. The directory listing
* taken at `open()` is the record of what is cached, and the orphan temp * taken at `open()` is the record of what is cached, and the orphan temp
@@ -41,7 +41,6 @@ import { join } from "node:path";
import { import {
ContentCache, ContentCache,
savePath,
type ContentSource, type ContentSource,
type EnsureEvent, type EnsureEvent,
} from "../../src/library/content.js"; } from "../../src/library/content.js";
@@ -153,48 +152,12 @@ const buildCache = (
pools: args.pools ?? new RequestPools(), pools: args.pools ?? new RequestPools(),
source, source,
cacheDirectory: cacheDir, cacheDirectory: cacheDir,
downloadDirectory: args.downloadDirectory ?? join(root, "photos"), downloadDirectory: args.downloadDirectory,
getFile: (id) => byID.get(id), getFile: (id) => byID.get(id),
}); });
return { cache, source }; return { cache, source };
}; };
// Microseconds, as Ente stores times, for noon local time on a day, so the
// machine's time zone cannot move the photo to another day.
const noon = (year: number, month: number, day: number): number =>
new Date(year, month - 1, day, 12).getTime() * 1000;
describe("savePath", () => {
it("files an original by year, month and day under the root", () => {
const f = file(12345, "IMG_0001.HEIC");
f.metadata.creationTime = noon(2026, 3, 1);
expect(savePath("/photos", f)).toBe(
"/photos/2026/2026-03/2026-03-01/2026-03-01.12345.HEIC",
);
});
it("dates an original by the date the user set, when there is one", () => {
const f = file(7, "a.jpg");
f.metadata.creationTime = noon(2026, 3, 1);
f.pubMagicMetadata = { editedTime: noon(1999, 12, 31) };
expect(savePath("/photos", f)).toBe(
"/photos/1999/1999-12/1999-12-31/1999-12-31.7.jpg",
);
});
it("takes the extension from the title it was uploaded with, not a new name", () => {
const f = file(8, "upload");
f.metadata.creationTime = noon(2026, 3, 1);
f.pubMagicMetadata = { editedName: "renamed.png" };
expect(savePath("/photos", f)).toBe(
"/photos/2026/2026-03/2026-03-01/2026-03-01.8.bin",
);
});
});
describe("ContentCache.open", () => { describe("ContentCache.open", () => {
it("creates the cache directories with 0700 permissions", async () => { it("creates the cache directories with 0700 permissions", async () => {
const { cache } = buildCache(); const { cache } = buildCache();
@@ -311,17 +274,11 @@ describe("ContentCache.original / thumbnail", () => {
it("serves a file already present in the download directory without fetching", async () => { it("serves a file already present in the download directory without fetching", async () => {
const downloadDirectory = join(root, "backup"); const downloadDirectory = join(root, "backup");
const day = join(downloadDirectory, "2026", "2026-03", "2026-03-01"); mkdirSync(join(downloadDirectory, "originals"), { recursive: true });
mkdirSync(day, { recursive: true }); const backupPath = join(downloadDirectory, "originals", "1.jpg");
const backupPath = join(day, "2026-03-01.1.jpg");
writeFileSync(backupPath, "from-backup"); writeFileSync(backupPath, "from-backup");
const f = file(1);
f.metadata.creationTime = noon(2026, 3, 1);
const { cache, source } = buildCache({ const { cache, source } = buildCache({ downloadDirectory });
downloadDirectory,
files: [f],
});
await cache.open(); await cache.open();
const events: EnsureEvent["status"][] = []; const events: EnsureEvent["status"][] = [];
@@ -692,7 +649,6 @@ describe("ContentCache live photos", () => {
]), ]),
), ),
cacheDirectory: cacheDir, cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => [a.file, b.file].find((f) => f.id === id), getFile: (id) => [a.file, b.file].find((f) => f.id === id),
// Room for one live photo, on a disk with plenty free. // Room for one live photo, on a disk with plenty free.
cacheOriginalsMaxBytes: size, cacheOriginalsMaxBytes: size,
test/library/precache.test.ts
-2
View File
@@ -280,7 +280,6 @@ describe("Precache eviction integration", () => {
pools: new RequestPools(), pools: new RequestPools(),
source, source,
cacheDirectory: cacheDir, cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id), getFile: (id) => byID.get(id),
statfs, statfs,
cacheOriginalsMaxBytes: 25, // holds two 10-byte originals cacheOriginalsMaxBytes: 25, // holds two 10-byte originals
@@ -349,7 +348,6 @@ describe("Precache preemption", () => {
pools: new RequestPools({ contentConcurrency: 1 }), pools: new RequestPools({ contentConcurrency: 1 }),
source, source,
cacheDirectory: join(root, "cache"), cacheDirectory: join(root, "cache"),
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id), getFile: (id) => byID.get(id),
statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }), statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }),
freeBelowBytes: 0, freeBelowBytes: 0,
test/library/read.test.ts
+3 -11
View File
@@ -36,7 +36,6 @@ import {
makeAlbumsAPI, makeAlbumsAPI,
makePhotosAPI, makePhotosAPI,
makeTimelineAPI, makeTimelineAPI,
type SavePathLookup,
type TimelineGroup, type TimelineGroup,
} from "../../src/library/read.js"; } from "../../src/library/read.js";
import { Library } from "../../src/library/index.js"; import { Library } from "../../src/library/index.js";
@@ -102,19 +101,12 @@ const file = (
}; };
// Build the three API objects over one fixed projection, the way `Library` // Build the three API objects over one fixed projection, the way `Library`
// wires them over its live store. Save paths are covered in // wires them over its live store.
// content-library.test.ts; these tests never ask for one.
const apis = (records: DerivedRecords) => { const apis = (records: DerivedRecords) => {
const derive = () => records; const derive = () => records;
const saves: SavePathLookup = {
savePath: () => {
throw new Error("no save paths in these tests");
},
isLocal: () => false,
};
return { return {
albums: makeAlbumsAPI(derive, saves), albums: makeAlbumsAPI(derive),
photos: makePhotosAPI(derive, saves), photos: makePhotosAPI(derive),
timeline: makeTimelineAPI(derive), timeline: makeTimelineAPI(derive),
}; };
}; };