Compare commits

..
7 Commits
Author SHA1 Message Date
clawbot 9e94542a57 docker build . stamps the git tag or short commit, not dev (closes #154)
check / check (push) Waiting to run
A plain `docker build .` now stamps the version from git rather than `dev`/`0.0.0`. After `tsc`, `script/build` writes into `dist/package.json` the version `script/version` decides: `VERSION` when given, otherwise `git describe --tags --always` (the tag; or tag, commits since and short commit; or the short commit), otherwise `package.json`'s. A checkout with `.git` that yields an empty, `dev` or `unknown` version fails the build. `.dockerignore` sends `.git` but not `.git/config`, so no remote URL or credential reaches the image. `ARG VERSION` has no default, and the host scripts' version still wins.

Not changed: `REPO_POLICIES.md` still says `ARG VERSION=dev` until the shared policy changes.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-10-02 06:12:56 +02:00
clawbot e50d2a78c8 exif() returns every EXIF tag in the file (closes #156)
check / check (push) Waiting to run
`photo.exif()` now returns every EXIF tag in the file, as the owner ruled, typed `ExifTags`: each tag keyed by name with `exifreader`'s `id`, `value`, `description` and `computed`. A tag with no name is keyed `undefined-` plus its number, and the embedded thumbnail's tags sit under `Thumbnail`. The thirteen typed methods stay, with the same names and types; each now picks its field from `exif()`'s tags. GPS latitude and longitude are worked out from the GPS tags and their reference tags, and a position with no reference tags gives neither. `backup-metadata --exif` output is unchanged.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-10-02 05:30:05 +02:00
clawbot 0c995a8c4f Remove the content cache's handling of an earlier version's live-photo ZIP (closes #151)
check / check (push) Failing after 44s
quak is pre-1.0 and keeps no handling of old data. The content cache no longer recognises or removes a live photo that an earlier quak version cached as one ZIP. A live-photo download no longer removes what was at its destination before renaming the image and video into place; the rename already replaces it. The README sentences and the tests about that old ZIP are gone. The two removed tests that also covered current behaviour are replaced by tests with no ZIP: the cache re-fetching a live photo recorded with no video, and the library telling the cache which files are live photos when it opens.

Model: opus-5-5
2026-10-02 04:14:30 +02:00
clawbot d788c5457d Thumbnail test: re-encode a small image so it cannot time out (closes #153)
check / check (push) Failing after 45s
The test "re-encodes smaller until the thumbnail fits the recorded size" built a noisy 400x300 JPEG and encoded it several times. That came close to vitest's 5 s limit and timed out on a busy host, which made `script/cibuild` fail on `next` and `main`. It now uses a noisy 64x48 JPEG, still too big for the first, default-quality encoding to fit, and keeps every assertion. It runs in about 1 s.

Model: opus-5-5
2026-10-02 03:49:23 +02:00
clawbot 10afa7a7f4 Example script: download every album's photos and metadata (closes #144)
check / check (push) Successful in 1m20s
`examples/download-albums.ts` logs in from `QUAK_EMAIL` and `QUAK_PASSWORD`, then walks every album from `await lib.fresh()`. For each photo it runs `photo.download()` to the photo's save path and writes `{savePath}.json`, which holds the photo's record and EXIF. For each album it writes `{dir}/albums/{collectionID}.json` with the album's name and its photos' save paths. A file is written only when its content changes, so a second run downloads and rewrites nothing.

The script opens the library with prefetching off, as `quak backup` does. The build type-checks `examples/`. The README says how to run it.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-10-02 00:33:00 +02:00
clawbot d40f339c0b Photo: one async method per EXIF field (closes #148)
check / check (push) Successful in 1m37s
`Photo` gains thirteen async methods, one per `PhotoExif` field and named after it: `make()`, `model()`, `lensModel()`, `dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`, `iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()` and `gpsAltitude()`. Each calls `exif()` and returns its one field, or `undefined` when the file lacks it. `exif()` is unchanged.

`Photo` implements a type built from `PhotoExif`'s keys, so the type check fails when a field has no method. Each call reads the original again; a caller that wants several fields calls `exif()` once.

Model: opus-5-5
2026-10-02 00:09:24 +02:00
clawbot 10e1a9ef39 Save path ./photos/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.fileID.ext; download() from the cache first (closes #143)
check / check (push) Successful in 1m25s
Originals are saved at `{downloadDirectory}/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.{fileID}{ext}`. The date is the photo's `takenAt` in local time, and `downloadDirectory` defaults to `./photos`, resolved when the library opens. The old `originals/` layout is gone.

`photo.download()` writes the original to `savePath`. It copies from the cache when the cache holds the original, and fetches otherwise. `lib.backup()` uses the same path and rule, and every album in `collections/` links to it. `isLocal` is true only when the original is at `savePath`.

For a file in several albums, one rule picks the copy everything uses: the most recently synced, with the lowest album ID breaking a tie.

Model: opus-5-5
2026-10-01 23:20:58 +02:00
33 changed files with 2162 additions and 734 deletions
+6 -3
View File
@@ -1,9 +1,12 @@
# Mirrors .gitignore, with one deliberate exception: .gitignore itself stays
# in the build context, because prettier 3 reads it as a default ignore file
# and dropping it would change what the lint phase's prettier check sees.
# VCS
.git
#
# .git is deliberately NOT excluded: the build derives the version it stamps
# from it (script/version). It is sent without its config, which holds the
# clone's remote URL and any credential in it, and which the build stage, the
# final image, would otherwise carry. git describe does not need it.
.git/config
# OS
.DS_Store
+6 -3
View File
@@ -61,9 +61,12 @@ RUN script/bootstrap
COPY . .
# The version is computed on the host and passed in, because
# .dockerignore excludes .git.
ARG VERSION=dev
# Version stamped into the build: the VERSION build arg when one is given,
# otherwise what script/version derives from the .git the build context
# carries (script/bootstrap installed git), so any `docker build .` of a
# clone stamps its commit. The label can only carry the build arg, and is
# empty without one.
ARG VERSION
LABEL org.opencontainers.image.version="${VERSION}"
RUN make build
+4 -2
View File
@@ -26,8 +26,10 @@ check:
build:
@script/build
build-bin:
nix-shell -p bun --run "bun build bin/quak.ts --compile --outfile bin/quak"
# Bundles the built dist/, so the binary reports the version script/build
# stamped.
build-bin: build
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
install: build-bin
mkdir -p ~/bin
+162 -74
View File
@@ -80,6 +80,37 @@ await lib.close();
The lower-level `Client` (login, session serialization, and the raw
enumeration/download calls) is exported too and documented under Design below.
## Examples
`examples/download-albums.ts` downloads every album's photos and their metadata
into a directory, `photos` in the working directory unless you name another. The
build compiles it; run it after `yarn install`:
```bash
yarn build
QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir]
```
It opens the library with `precacheThumbnails` and `precacheOriginals` off, as
`quak backup` does, so the only file content it fetches is the originals it
saves. It asks on the terminal for a two-factor or email code when the account
requires one, and writes:
- each photo's original at its save path under `dir`, as `photo.download()`
writes it: `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, and for a live
photo its image, its video and the `.livephoto.json` file naming them
- beside each original, a JSON file named after it with `.json` added, for
example `2026-03-01.12345.jpg.json`: the photo's record (`photo.record()`)
without its cache paths, and every EXIF tag of the photo (`photo.exif()`)
under `exif`
- `albums/<collectionID>.json` for each album: its `collectionID`, its `name`,
and under `savePaths` the save paths of its photos relative to `dir`, newest
first
A photo in several albums is downloaded once. A second run downloads nothing and
rewrites only the JSON files whose content changed. A failed download stops the
run; running it again carries on, since every photo already saved is skipped.
## Entrypoints
This repository adheres to the
@@ -95,9 +126,12 @@ alpine. We provide:
`script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (our own extension); used by
`script/docker` for the image tag
- `script/build` — compile the TypeScript sources into `dist/`, then verify that
the entrypoints `package.json` declares (`main`, `types`, `bin`) are among the
files the compiler wrote, and make the CLI executable (our own extension)
- `script/build` — compile the TypeScript sources into `dist/`, stamp the
version into `dist/package.json`, then verify that the entrypoints
`package.json` declares (`main`, `types`, `bin`) are among the files the
compiler wrote, and make the CLI executable (our own extension)
- `script/version` — print the version `script/build` stamps (our own
extension); see Version below
- `script/test` — run the test suite, by building the `test` phase of the
`Dockerfile` (vitest, 90s timeout, verbose rerun on failure); requires docker
- `script/lint` — run eslint and a prettier check, by building the `lint` phase
@@ -145,6 +179,34 @@ an exact version, installed from `yarn.lock` under `--frozen-lockfile` in both
places, and reads `.gitignore` as its default ignore file — which is why
`.dockerignore` keeps `.gitignore` in the build context.
### Version
`quak --version` reports the `version` of `dist/package.json`, which
`script/build` writes after compiling; the repo's own `package.json` keeps
`0.0.0`, and that is what the tests, which run from source, report.
`script/version` decides what is written:
- the `VERSION` environment variable, or the `Dockerfile`'s `VERSION` build arg
(`--build-arg VERSION=...`), when one is given and not empty;
- otherwise, in a checkout with `.git`, `git describe --tags --always`: the tag
on a tagged commit; the tag, the commits since it and the short commit on a
later commit (`v1.2.3-4-gabc1234`); the short commit when no tag is reachable;
- otherwise, as in a source tarball, the version `package.json` declares.
The build fails if the checkout has `.git` and the version still comes out
empty, `dev` or `unknown`: such a build could not be traced back to its commit.
`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of
a clone stamps the commit it was built from; a shallow clone stamps a tag only
when the cloned commit itself carries one, and otherwise the short commit. It
leaves out `.git/config`, which holds the clone's remote URL and any credential
in it, so the image carries `.git` without its config; `git describe` does not
need that file. `script/docker` (and so `make docker`) and `script/cibuild` pass
the version they resolve on the host, with `--dirty`, as the build arg, which
takes precedence. The image's `org.opencontainers.image.version` label carries
that build arg only, so a build given none leaves it empty. `make build-bin`
bundles the built `dist/`, so the single binary reports the stamped version too.
## Rationale
Ente is one of very few photo services with a credible end-to-end encryption
@@ -239,6 +301,9 @@ quak/
index.ts public library exports
bin/
quak.ts CLI entrypoint (commander.js)
examples/
download-albums.ts
download every album's photos and metadata
test/ unit + integration tests (vitest)
Makefile
Dockerfile lint phase, test phase, compile
@@ -247,10 +312,11 @@ quak/
```
`make build` compiles that tree into `dist/`, preserving its shape: the library
lands in `dist/src/` and the CLI in `dist/bin/quak.js`, which is what
`package.json` points `main`, `types` and `bin` at. The compiler's `rootDir` is
the repository root rather than `src/`, because `bin/` is compiled too and
`rootDir` has to contain everything that is compiled.
lands in `dist/src/`, the examples in `dist/examples/`, and the CLI in
`dist/bin/quak.js`, which is what `package.json` points `main`, `types` and
`bin` at. The compiler's `rootDir` is the repository root rather than `src/`,
because `bin/` is compiled too and `rootDir` has to contain everything that is
compiled.
### Cryptography
@@ -528,25 +594,36 @@ the smallest does not.
```
<dir>/
originals/
<fileID>.<ext> actual file content (one per unique file,
two for a live photo: see below)
<fileID>.json the file's basic metadata fields quak
YYYY/YYYY-MM/YYYY-MM-DD/
YYYY-MM-DD.<fileID>.<ext> actual file content, at its save path (one
per unique file, two for a live photo: see
below)
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
keeps, and its private and public magic
metadata
<fileID>.livephoto.json which of a live photo's two files is which
YYYY-MM-DD.<fileID>.livephoto.json
which of a live photo's two files is which
collections/
<name>/
<title> -> ../../originals/<fileID>.<ext> (symlink)
<title> -> ../../YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>
(symlink)
<name>.json collection metadata + file list
failures.json files that failed and have not yet succeeded
```
Each original is saved at its save path, the same path `photo.savePath` gives
and `photo.download()` writes (see Read surface below). The date is the photo's
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
time zone of the machine running quak. The extension is the one in the file's
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
letters and digits. When the date or the time zone changes, the next run saves
the original at its new path and leaves the old copy where it is.
`failures.json` records each failed file with the kind of failure, how many
times it has been tried and when it was last tried. A file leaves it once it
succeeds, or once it is no longer in the library or in the backup's scope. The
library's `lib.backup({ includeThumbnails: true })` also writes
`thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not.
A collection's directory and JSON are named after the collection, and a symlink
after the file's title, both with unsafe characters replaced. When two
@@ -558,22 +635,21 @@ name stays the same from run to run until such a clash appears or goes away.
A live photo, which Ente stores as one ZIP of its image and its video, is stored
as those two files, which a photo viewer can open: each is
`originals/<fileID>.<ext>` with the extension it has inside the ZIP (for example
`12345.heic` and `12345.mov`), and `<fileID>.livephoto.json` names the two. The
live photo counts as stored only when both files are present and not empty. Its
album folder links both, each named after the title with that file's extension
(`IMG_0001.heic` and `IMG_0001.mov`). A live photo that an earlier version of
quak stored as the ZIP, under the image's name, is replaced by its two files on
the next run, and the ZIP and its link are removed.
`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP (for
example `2026-03-01.12345.heic` and `2026-03-01.12345.mov`), and
`YYYY-MM-DD.<fileID>.livephoto.json` names the two. The live photo counts as
stored only when both files are present and not empty. Its album folder links
both, each named after the title with that file's extension (`IMG_0001.heic` and
`IMG_0001.mov`).
Each run removes the symlinks into `originals/` that no longer belong in their
collection's directory, and the directories (and JSON) of collections that were
deleted or renamed. Nothing else in `collections/` is touched: a file or a
Each run removes the symlinks into the date folders that no longer belong in
their collection's directory, and the directories (and JSON) of collections that
were deleted or renamed. Nothing else in `collections/` is touched: a file or a
symlink you put there stays, and a directory that still holds one after its
symlinks are removed stays too, with its JSON.
Each file is downloaded exactly once regardless of how many collections it
appears in, and written once: straight into `originals/`, with no copy left in
appears in, and written once: straight to its save path, with no copy left in
the cache. An original the cache already held is copied from there instead. On
subsequent runs, existing originals are skipped. If a download fails, the error
is logged and the backup continues with the next file. The exit code is non-zero
@@ -586,14 +662,15 @@ Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even
after a power cut. A downloaded original's temporary file is named
`.quak-<pid>-<random>.tmp`, one copied from the cache
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
one of these temporary files behind; the next backup deletes those whose process
is no longer running. The content cache uses the same scheme, and opening a
library deletes the temporary files in the cache whose process is no longer
running, so a download another process has in progress in the same cache is left
alone. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is
killed can leave one of these temporary files behind; the next backup deletes
those whose process is no longer running. The content cache uses the same
scheme, and opening a library deletes the temporary files in the cache whose
process is no longer running, so a download another process has in progress in
the same cache is left alone. The rename replaces whatever was at the
destination rather than writing through it: a symlink there is replaced, not
followed, and the new file has the temporary file's permissions, not those of
the file it replaced.
## TODO
@@ -636,10 +713,10 @@ background, so an unreachable server does not block opening.
`LibraryOptions`:
| Option | Default | Meaning |
| ------------------------ | -------------------------------------------------- | --------------------------------------------------------------------- |
| ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------- |
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached |
| `refreshIntervalSeconds` | `3` | background refresh cadence |
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
@@ -651,6 +728,9 @@ background, so an unreachable server does not block opening.
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
| `contentSource` | the client's own | override the byte source (mainly for tests) |
The default `downloadDirectory` is resolved against the working directory once,
when the library opens; `lib.downloadDirectory` holds the result.
Concurrency is set through `pools`: construct
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
@@ -702,45 +782,56 @@ and `originalPath`, and `photo.year`, the local-time year of `takenAt`, all as
synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more
synchronous getters look at the disk and never touch the network:
- `photo.savePath` → `string | undefined` — where `lib.backup()` writes the
original, `originals/<fileID>.<ext>` under the `downloadDirectory` the library
was opened with, whether or not it is there yet: for a live photo already
stored, its image. For a live photo not yet stored, it carries the title's
extension, and the backup may store the image under a different one, found
inside the live photo. `undefined` when the library has no `downloadDirectory`
or no content source.
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`.
- `photo.savePath` → `string` — where `photo.download()` and `lib.backup()` put
the original, whether or not it is there yet:
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
`downloadDirectory` (see Backup layout above for the date and the extension).
For a live photo already stored, its image. For a live photo not yet stored,
it carries the title's extension, and the image may be stored under a
different one, found inside the live photo. It needs no content source.
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`. A
copy only in the cache does not count.
These async methods may download:
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
full-resolution file. For a live photo, `path` and `bytes` are its image's and
`videoPath` is its video.
- `await photo.download()` → `{ path, bytes, videoPath? }`, as `original()` —
puts the original at `savePath`, creating its folders. When it is already
there, nothing is written. When the cache holds it, it is copied from the
cache; otherwise it is fetched straight to `savePath`, with no copy left in
the cache. Afterwards `isLocal` is true.
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read
through `original()`; for a live photo, its image's.
- `await photo.exif(opts?)` → `PhotoExif` — `make`, `model`, `lensModel`,
`dateTimeOriginal`, `offsetTimeOriginal`, `exposureTime`, `fNumber`, `iso`,
`focalLength`, `orientation`, `gpsLatitude`, `gpsLongitude` and `gpsAltitude`,
each absent when the file lacks it. GPS values are signed decimal degrees and
metres. `dateTimeOriginal` is the camera's clock reading held in the `Date`'s
UTC fields; `offsetTimeOriginal`, when present, is that clock's offset from
UTC. EXIF is read from any image format exifreader reads (such as JPEG,
HEIC/HEIF, AVIF, PNG, WebP and TIFF), a live photo's image included. Any other
original gives `{}`, and a video gives `{}` without being downloaded.
- `await photo.exif(opts?)` → `ExifTags` — every EXIF tag in the file, keyed by
tag name, each as exifreader decodes it, with its `id`, `value`, `description`
and `computed` value: for example `Make` is
`{ id: 271, value: ["Canon"], description: "Canon", computed: "Canon" }`. A
tag exifreader has no name for is keyed `undefined-<tag number>`. The embedded
thumbnail's tags are under `Thumbnail`, so they cannot hide the main image's
tags of the same name; the thumbnail image itself is left out. EXIF is read
from any image format exifreader reads (such as JPEG, HEIC/HEIF, AVIF, PNG,
WebP and TIFF), a live photo's image included. Any other original gives `{}`,
and a video gives `{}` without being downloaded.
- `await photo.make(opts?)`, and likewise `model()`, `lensModel()`,
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
and `gpsAltitude()` → one field of `exif()` each, typed as in `PhotoExif`, or
`undefined` when the file lacks it. Each calls `exif()` with its `opts`, so
each call reads the original again.
and `gpsAltitude()` → one common field each, picked from the tags `exif()`
returns and typed as in `PhotoExif`, or `undefined` when the file lacks it.
GPS values are signed decimal degrees and metres. `dateTimeOriginal()` is the
camera's clock reading held in the `Date`'s UTC fields;
`offsetTimeOriginal()`, when present, is that clock's offset from UTC. Each
calls `exif()` with its `opts`, so each call reads the original again.
They serve from the on-disk content cache when the bytes are present and
otherwise fetch through the pools; `original()`, `content()` and `exif()` also
serve an original the backup has already stored. `opts.onProgress` reports
serve an original already stored at its save path. `opts.onProgress` reports
per-file progress. They throw when the library was opened without a content
source. An original that `content()` or `exif()` downloads lands in the cache,
which does not make `isLocal` true; only `lib.backup()` does.
source. An original that `original()`, `content()` or `exif()` downloads lands
in the cache, which does not make `isLocal` true; only `download()` and
`lib.backup()` do.
Lower-level accessors that return decrypted model objects (which hold key
material) are also available: `listCollections()`, `getCollection(id)`,
@@ -783,15 +874,15 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere.
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
`fresh()` does, fetches every in-scope original not already in the backup
(and, with `includeThumbnails`, thumbnails) through the content cache, and
rebuilds the on-disk backup tree with a durable failure ledger. A fetched
original is written straight into the backup's `originals/` and not into the
cache, which then counts it as present; one the cache already held is copied
from there. `BackupOptions`: `downloadDirectory` (falls back to the one
`open()` was given), `includeOriginals` (default `true`), `includeThumbnails`
(default `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above
for the tree it writes.
`fresh()` does, puts every in-scope original not already at its save path
there as `photo.download()` does (and, with `includeThumbnails`, fetches
thumbnails) through the content cache, and rebuilds the on-disk backup tree
with a durable failure ledger. A fetched original is written straight to its
save path and not into the cache, which then counts it as present; one the
cache already held is copied from there. `BackupOptions`: `downloadDirectory`
(falls back to the library's), `includeOriginals` (default `true`),
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See
Backup layout above for the tree it writes.
### Request pools
@@ -822,12 +913,9 @@ When `metadata.json` belongs to a different account than the client's,
originals and thumbnails are kept; they are reached only through the files the
current account's records name.
A live photo's original is cached as in the backup: its image and its video,
A live photo's original is cached as at its save path: its image and its video,
each `originals/<fileID>.<ext>` with its own extension, and
`originals/<fileID>.livephoto.json` naming them; the two are evicted together. A
live photo that an earlier version cached as its ZIP is not served: the library
removes the ZIP when it opens the cache, and fetches the two files when the
photo is next read or precached.
`originals/<fileID>.livephoto.json` naming them; the two are evicted together.
A stored file appears only via an atomic temp-then-rename, so its presence means
it is complete. Every downloaded original (by `quak get`, the cache, or
@@ -846,13 +934,13 @@ from a very old client, is stored unchecked.
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
`LibraryClient`, `RefreshEvent`
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`, `SavePathLookup`
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
`EnsureOptions`, `EnsureResult`, `ContentSource`
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
`LibraryChange`
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
- `src/exif.ts`: `PhotoExif`
- `src/exif.ts`: `ExifTags`, `PhotoExif`
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
+37
View File
@@ -25,6 +25,32 @@ declares one.
# Completed Steps
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
`dev` (issue 154). `script/build` writes the version `script/version` prints
into `dist/package.json`: the `VERSION` environment variable or build arg when
one is given, otherwise `git describe --tags --always`, otherwise the version
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
`make build-bin` bundles the built `dist/`, so the single binary reports the
same version.
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
keyed by tag name, each as exifreader decodes it, not only the thirteen common
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
without the thumbnail image. The thirteen typed methods stay, each picking its
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
script's JSON files now carry every tag.
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
version cached as one ZIP (issue 151). When the cache opens, a live photo's
file that no JSON file names is now always left alone.
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
every album downloads each photo to its save path, writes the photo's record
and EXIF fields to a JSON file beside it, and writes the album's photos to
`albums/<collectionID>.json` (issue 144). The build compiles it to
`dist/examples/`; the README's "Examples" section says how to run it.
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
@@ -35,6 +61,17 @@ declares one.
no method. A test checks, on the JPEG and the HEIC, that each method gives the
same value as `exif()`.
- 2026-10-01: Each original's save path is
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
download directory, which defaults to `photos` in the working directory (issue
143). The date is the photo's `takenAt` in the machine's time zone.
`photo.savePath` is always a string, with or without a content cache, and
`isLocal` is true only when the original is at its save path.
`photo.download()` puts the original there, copied from the cache when the
cache holds it and fetched otherwise. `lib.backup()` does the same for each
file, writes each file's JSON beside its original, and links `collections/` to
the save paths.
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
+129
View File
@@ -0,0 +1,129 @@
// Download every album's photos to a directory, with each photo's metadata
// beside it, using only quak's public API. The README's "Examples" section
// describes the files it writes.
//
// yarn build
// QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir]
import { realpathSync } from "node:fs";
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { join, relative } from "node:path";
import { stdin, stdout } from "node:process";
import { createInterface } from "node:readline/promises";
import { pathToFileURL } from "node:url";
import { Client, Library } from "../src/index.js";
// Write `text` to `path` unless the file already holds exactly that, so a
// second run rewrites nothing.
async function writeIfChanged(path: string, text: string): Promise<void> {
const current = await readFile(path, "utf-8").catch(() => undefined);
if (current !== text) await writeFile(path, text);
}
const pretty = (value: unknown): string =>
JSON.stringify(value, null, 2) + "\n";
// For every album in `lib`, download each photo to its save path, write the
// photo's metadata to `{savePath}.json`, and write the album's photos to
// `{dir}/albums/{collectionID}.json`. Returns how many photos it downloaded
// and how many were already at their save paths.
export async function downloadAlbums(
lib: Library,
dir: string,
): Promise<{ downloaded: number; alreadyLocal: number }> {
let downloaded = 0;
let alreadyLocal = 0;
// A photo in several albums is handled once.
const done = new Set<number>();
// fresh() waits for a refresh from the server and throws if it fails, so
// albums and photos added since the cache was last written are included.
const { albums } = await lib.fresh();
for (const album of albums.list()) {
const savePaths: string[] = [];
for (const photo of album.photos.list()) {
if (!done.has(photo.fileID)) {
done.add(photo.fileID);
if (photo.isLocal) alreadyLocal++;
else downloaded++;
await photo.download();
// The cache paths say where quak's cache keeps copies, not
// anything about the photo.
const record = { ...photo.record() };
delete record.thumbnailPath;
delete record.originalPath;
const exif = await photo.exif();
// savePath is read after download(): a live photo's names its
// image only once the image is stored.
await writeIfChanged(
`${photo.savePath}.json`,
pretty({ ...record, exif }),
);
}
savePaths.push(relative(dir, photo.savePath));
}
await mkdir(join(dir, "albums"), { recursive: true });
await writeIfChanged(
join(dir, "albums", `${album.collectionID}.json`),
pretty({
collectionID: album.collectionID,
name: album.name,
savePaths,
}),
);
}
return { downloaded, alreadyLocal };
}
// Ask for a login code on the terminal. Client.login calls this only when the
// account requires a code.
async function ask(question: string): Promise<string> {
const terminal = createInterface({ input: stdin, output: stdout });
try {
return await terminal.question(question);
} finally {
terminal.close();
}
}
async function main(): Promise<void> {
const email = process.env.QUAK_EMAIL;
const password = process.env.QUAK_PASSWORD;
if (!email || !password) {
console.error(
"Set QUAK_EMAIL and QUAK_PASSWORD to the account's email and password.",
);
process.exit(1);
}
const dir = process.argv[2] ?? "photos";
const client = await Client.login({
email,
password,
totp: () => ask("Two-factor code: "),
emailOTP: () => ask("Code sent to your email: "),
});
const lib = await Library.open({
client,
downloadDirectory: dir,
// As in `quak backup`: fetch only the originals this script saves,
// not every thumbnail and the recent originals into the cache too.
precacheThumbnails: false,
precacheOriginals: false,
});
try {
const { downloaded, alreadyLocal } = await downloadAlbums(lib, dir);
console.log(
`${downloaded} photos downloaded, ${alreadyLocal} already local, in ${dir}`,
);
} finally {
await lib.close();
}
}
// Run main() when node runs this file, not when a test imports it. argv[1] is
// the path as given, and import.meta.url has symlinks resolved.
const script = process.argv[1];
if (script && pathToFileURL(realpathSync(script)).href === import.meta.url) {
await main();
}
+23 -9
View File
@@ -1,7 +1,7 @@
#!/bin/sh
# script/build: compile the TypeScript sources into dist/, then verify that
# the artifacts package.json advertises are among the files the compiler
# actually wrote. tsc reports success by exit status alone and knows nothing
# script/build: compile the TypeScript sources into dist/, stamp the version
# script/version prints into it, then verify that the artifacts package.json
# advertises are among the files the compiler actually wrote. tsc reports success by exit status alone and knows nothing
# about the manifest, so without this step a green build can still ship a
# package whose main, types or bin resolve to nothing. Our own extension to
# scripts-to-rule-them-all.
@@ -46,13 +46,24 @@ for (const bin of bins) {
}
# src/index.ts imports ../package.json for the version, which tsc copies to
# dist/package.json. Running the built CLI proves that import resolves from
# dist/ and reports the version package.json declares.
# dist/package.json. The version script/version prints is written into that
# copy only; the repo's own package.json is left as it is.
stamp_version() {
node -e '
const { readFileSync, writeFileSync } = require("node:fs");
const pkg = JSON.parse(readFileSync("dist/package.json", "utf-8"));
pkg.version = process.argv[1];
writeFileSync("dist/package.json", JSON.stringify(pkg, null, 4) + "\n");
' "$1"
}
# Running the built CLI proves the import resolves from dist/ and reports
# the stamped version.
verify_version() {
built="$(node dist/bin/quak.js --version)"
declared="$(node -p 'require("./package.json").version')"
if [ "$built" != "$declared" ]; then
echo "build: dist/bin/quak.js reports $built, package.json declares $declared" >&2
if [ "$built" != "$1" ]; then
echo "build: dist/bin/quak.js reports $built, the build stamped $1" >&2
exit 1
fi
echo "build: dist/bin/quak.js reports version $built"
@@ -60,9 +71,12 @@ verify_version() {
main() {
cd "$ROOT"
# Own line, so that a failing script/version stops the build.
version="$("$ROOT/script/version")"
yarn run tsc
stamp_version "$version"
verify_entrypoints
verify_version
verify_version "$version"
}
main "$@"
+3 -3
View File
@@ -15,9 +15,9 @@ main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. VERSION is computed here because .dockerignore
# excludes .git, so `git describe` in a build stage yields an empty
# version without failing.
# empty constant. The version resolved here goes in as the VERSION
# build arg, which takes precedence over what the build would derive
# from the .git in its context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
+3 -3
View File
@@ -12,9 +12,9 @@ main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. VERSION is computed here because .dockerignore
# excludes .git, so `git describe` in a build stage yields an empty
# version without failing.
# empty constant. The version resolved here goes in as the VERSION
# build arg, which takes precedence over what the build would derive
# from the .git in its context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
Executable
+41
View File
@@ -0,0 +1,41 @@
#!/bin/sh
# script/version: print the version script/build stamps into the built
# package. Our own extension to scripts-to-rule-them-all.
#
# Order of precedence:
#
# 1. $VERSION, if set and not empty: an explicit value, such as the
# Dockerfile's VERSION build arg.
# 2. If this checkout has .git, `git describe --tags --always`: the tag
# on a tagged commit; the tag, the commits since it and the short
# commit on a later commit (v1.2.3-4-gabc1234); the short commit when
# no tag is reachable.
# 3. Otherwise, as in a source tarball, the version package.json declares.
#
# A checkout with .git whose version still comes out empty, dev or unknown
# fails: git is missing or could not read the checkout, and the build could
# not be traced back to its commit.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
version="${VERSION:-}"
if [ -e .git ]; then
if [ -z "$version" ]; then
version="$(git describe --tags --always || true)"
fi
case "$version" in
"" | dev | unknown)
echo "version: $ROOT has .git, but the version came out '$version'" >&2
exit 1
;;
esac
elif [ -z "$version" ]; then
version="$(node -p 'require("./package.json").version')"
fi
echo "$version"
}
main "$@"
+110 -120
View File
@@ -2,29 +2,30 @@
//
// `lib.backup()` waits for a completed refresh of the library (a failed one
// fails the backup before any file is touched), then, for every file in scope,
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
// derived views (per-file sidecars, per-collection symlink trees,
// per-collection JSON) from the model. The on-disk layout is the historical
// one:
// puts its original at its save path under `downloadDirectory`, as
// `Photo.download()` does, and rebuilds the derived views (per-file sidecars,
// per-collection symlink trees, per-collection JSON) from the model. The
// on-disk layout:
//
// <downloadDirectory>/
// originals/<fileID>.<ext> the decrypted bytes
// originals/<fileID>.json per-file metadata sidecar
// collections/<name>/<title> symlink into ../../originals
// YYYY/YYYY-MM/YYYY-MM-DD/
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar
// collections/<name>/<title> symlink to the original
// collections/<name>.json per-collection metadata
// failures.json durable ledger of unresolved failures
//
// A live photo's original is its image and its video, `<fileID>.<ext>` each
// with its own extension, and `originals/<fileID>.livephoto.json` naming them;
// its album folders link both.
// A live photo's original is its image and its video, each with its own
// extension, beside `YYYY-MM-DD.<fileID>.livephoto.json` naming them; its album
// folders link both.
//
// Crash-safety rests on two properties. Bytes are present-means-complete: an
// original appears under `originals/` only via the content layer's atomic
// original appears at its save path only via the content layer's atomic
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
// interrupted run resumes by listing the directory. The derived views hold no
// unique state, so they are rebuilt every run; that repairs stale sidecars and
// missing or broken symlinks left by an earlier crash. A rebuild also removes
// the symlinks into originals/ that no longer belong to an album, and the
// interrupted run resumes by looking at the save paths. The derived views hold
// no unique state, so they are rebuilt every run; that repairs stale sidecars
// and missing or broken symlinks left by an earlier crash. A rebuild also
// removes the symlinks to originals that no longer belong to an album, and the
// directories of albums that no longer exist.
//
// Resilience (issue #8): no per-file condition aborts the run. A failed
@@ -48,24 +49,25 @@ import {
symlinkSync,
writeFileSync,
} from "node:fs";
import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, extname, join, relative } from "node:path";
import { dirname, extname, join, relative, resolve } from "node:path";
import { fsyncPath, removeLeftoverTempFiles } from "./download/index.js";
import { removeLeftoverTempFiles } from "./download/index.js";
import { sanitizeFileName, withExtension } from "./filename.js";
import {
nameInOriginals,
storedOriginal,
writeLivePhotoJSON,
copyAtomic,
placeOriginal,
savePath,
storedAtSavePath,
} from "./library/content.js";
import { representative } from "./library/records.js";
import type { Collection, EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void;
export interface BackupOptions {
// Where the backup tree lives. Required: with none, `backup()` throws
// before any network traffic. A library opened with a `downloadDirectory`
// supplies the default.
// Where the backup tree lives. `lib.backup()` defaults it to the library's
// download directory; `runBackup` with none throws before any network
// traffic.
downloadDirectory?: string;
// Fetch and store full-resolution originals. Default true.
includeOriginals?: boolean;
@@ -89,7 +91,7 @@ export interface BackupResult {
totalFiles: number;
// Originals fetched (or copied from the cache) this run.
downloaded: number;
// Originals already present and left untouched.
// Originals already at their save path and left untouched.
skipped: number;
// Files with an unresolved failure after this run (the ledger size); the
// CLI exits non-zero while this is above zero. A file can be both
@@ -107,8 +109,8 @@ export interface BackupLibrary {
listFiles(collectionID: number): EnteFile[];
// Get an original's bytes onto disk through the content cache/pools,
// returning where they landed: `destination` when they were fetched now,
// otherwise wherever they already were (the cache, or a prior backup). A
// live photo lands as its image and its video, fetched now beside
// otherwise wherever they already were (the cache, or the library's save
// path). A live photo lands as its image and its video, fetched now beside
// `destination`.
original(
fileID: number,
@@ -168,58 +170,6 @@ const classify = (err: unknown): FailureClass => {
const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err);
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete"). As in the
// download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return;
const tmp = join(
dirname(dest),
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
.toString(36)
.slice(2)}.tmp`,
);
try {
await copyFile(src, tmp);
await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally {
await rm(tmp, { force: true });
}
};
// Put an original the library returned at `dest` in originals/, where a fresh
// fetch already wrote it. A live photo's image and video go beside `dest`: when
// they came from the cache they are copied, after removing whatever was at
// `dest` (an earlier version's ZIP of the two). Then the JSON file naming them
// is written, which is what makes the live photo count as stored.
const placeOriginal = async (
file: EnteFile,
dest: string,
got: { path: string; videoPath?: string },
): Promise<void> => {
if (got.videoPath === undefined) {
await copyAtomic(got.path, dest);
return;
}
const originalsDir = dirname(dest);
const path = join(originalsDir, basename(got.path));
const videoPath = join(originalsDir, basename(got.videoPath));
if (got.path !== path) {
await rm(dest, { force: true });
await copyAtomic(got.path, path);
await copyAtomic(got.videoPath, videoPath);
}
await writeLivePhotoJSON(originalsDir, file.id, { path, videoPath });
};
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
// non-symlink entry. Throws on failure (a directory in the way, no permission)
// so the caller records it and moves on rather than aborting the run.
@@ -291,36 +241,55 @@ const linksFor = (
}));
};
// Remove the symlinks in the album directory `dir` that point into
// `originalsDir` and are not named in `keep`. Nothing else in the directory
// is touched: anything else there was put there by the user.
// Every date folder (`YYYY/YYYY-MM/YYYY-MM-DD/`) under `root`, whether or not a
// file in this backup is saved there. A folder that cannot be read is skipped.
const dateFolders = (root: string): string[] => {
const subfolders = (dir: string, name: RegExp): string[] => {
try {
return readdirSync(dir, { withFileTypes: true })
.filter((e) => e.isDirectory() && name.test(e.name))
.map((e) => join(dir, e.name));
} catch {
return [];
}
};
return subfolders(root, /^\d{4}$/)
.flatMap((year) => subfolders(year, /^\d{4}-\d\d$/))
.flatMap((month) => subfolders(month, /^\d{4}-\d\d-\d\d$/));
};
// Whether the entry at `path` is a symlink a backup to `root` made: one to an
// original in a `YYYY/YYYY-MM/YYYY-MM-DD/` folder of `root`.
const linksToOriginal = (path: string, root: string): boolean => {
if (!lstatSync(path).isSymbolicLink()) return false;
const target = relative(root, resolve(dirname(path), readlinkSync(path)));
return /^\d{4}\/\d{4}-\d\d\/\d{4}-\d\d-\d\d\/[^/]+$/.test(target);
};
// Remove the symlinks in the album directory `dir` that point to an original
// in `root` and are not named in `keep`. Nothing else in the directory is
// touched: anything else there was put there by the user.
const removeStaleLinks = (
dir: string,
keep: Set<string>,
originalsDir: string,
root: string,
): void => {
const target = relative(dir, originalsDir);
for (const name of readdirSync(dir)) {
if (keep.has(name)) continue;
const path = join(dir, name);
if (
lstatSync(path).isSymbolicLink() &&
dirname(readlinkSync(path)) === target
) {
rmSync(path);
}
if (linksToOriginal(path, root)) rmSync(path);
}
};
// Remove the directories under `collectionsDir` that an earlier run wrote for
// an album that is gone or renamed: a directory not named in `current` with a
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
// symlinks into originals/ are removed; if that leaves it empty, it and its
// JSON are deleted, otherwise both stay for what the user put there.
// symlinks to originals in `root` are removed; if that leaves it empty, it and
// its JSON are deleted, otherwise both stay for what the user put there.
const removeStaleAlbumDirs = (
collectionsDir: string,
current: Set<string>,
originalsDir: string,
root: string,
): void => {
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
if (!entry.isDirectory() || current.has(entry.name)) continue;
@@ -334,7 +303,7 @@ const removeStaleAlbumDirs = (
continue;
}
const dir = join(collectionsDir, entry.name);
removeStaleLinks(dir, new Set(), originalsDir);
removeStaleLinks(dir, new Set(), root);
if (readdirSync(dir).length > 0) continue;
rmdirSync(dir);
rmSync(jsonPath);
@@ -402,34 +371,48 @@ export const runBackup = async (
log("Refreshing library...");
await lib.refresh();
const originalsDir = join(downloadDirectory, "originals");
const collectionsDir = join(downloadDirectory, "collections");
const thumbnailsDir = join(downloadDirectory, "thumbnails");
mkdirSync(originalsDir, { recursive: true });
mkdirSync(collectionsDir, { recursive: true });
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
removeLeftoverTempFiles(originalsDir);
removeLeftoverTempFiles(thumbnailsDir);
for (const dir of dateFolders(downloadDirectory)) {
removeLeftoverTempFiles(dir);
}
const ledgerPath = join(downloadDirectory, "failures.json");
const ledger = loadLedger(ledgerPath);
const now = Date.now();
// Collections in scope, and the distinct files across them (a file shared
// by two albums is one original).
// by two albums is one original). Each file is the membership
// `representative` picks from all of its albums, in scope or not, so it is
// saved at the path `photo.savePath` names.
const allCollections = lib.listCollections();
const collections = allCollections.filter((c) =>
only ? only.has(c.name) : true,
);
const collectionName = new Map<number, string>();
for (const c of collections) collectionName.set(c.id, c.name);
for (const c of allCollections) collectionName.set(c.id, c.name);
const distinct = new Map<number, EnteFile>();
const memberships = new Map<number, EnteFile[]>();
const filesByCollection = new Map<number, EnteFile[]>();
for (const c of collections) {
for (const c of allCollections) {
const files = lib.listFiles(c.id);
filesByCollection.set(c.id, files);
for (const f of files) if (!distinct.has(f.id)) distinct.set(f.id, f);
for (const f of files) {
const arr = memberships.get(f.id);
if (arr) arr.push(f);
else memberships.set(f.id, [f]);
}
}
const distinct = new Map<number, EnteFile>();
for (const c of collections) {
for (const f of filesByCollection.get(c.id)!) {
if (!distinct.has(f.id)) {
distinct.set(f.id, representative(memberships.get(f.id)!));
}
}
}
const errors: BackupError[] = [];
@@ -465,25 +448,22 @@ export const runBackup = async (
failedThisRun.add(file.id);
};
// Phase 1: get the bytes. Fetch each pending original (and optional
// thumbnail) through the content cache/pools and place it under the backup
// tree; a present file is left as is.
// Phase 1: get the bytes. Put each pending original at its save path
// through the content cache/pools, as `Photo.download()` does, and fetch
// the optional thumbnails; a present file is left as is.
if (includeOriginals) {
for (const [fileID, file] of distinct) {
if (storedOriginal(originalsDir, file) !== undefined) {
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
skipped++;
continue;
}
const dest = join(originalsDir, nameInOriginals(file));
try {
log(`Fetching original ${file.metadata.title} (${fileID})...`);
// A fetched original is written straight to `dest` (a live
// photo beside it); only one that was already cached elsewhere
// is copied.
await placeOriginal(
file,
dest,
await lib.original(fileID, dest),
// A fetched original is written straight to its save path (a
// live photo beside it); only one that was already cached
// elsewhere is copied.
await placeOriginal(downloadDirectory, file, (dest) =>
lib.original(fileID, dest),
);
downloaded++;
} catch (err) {
@@ -519,9 +499,10 @@ export const runBackup = async (
// Phase 2: rebuild the derived views from the model. Sidecars first, for
// every present original (this repairs stale ones).
if (includeOriginals) {
for (const [fileID, file] of distinct) {
if (storedOriginal(originalsDir, file) !== undefined) {
writeSidecar(join(originalsDir, `${fileID}.json`), file);
for (const file of distinct.values()) {
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
const path = savePath(downloadDirectory, file);
writeSidecar(withExtension(path, ".json"), file);
}
}
}
@@ -543,7 +524,11 @@ export const runBackup = async (
allCollections.map((c, i) => [c.id, dirNames[i]!]),
);
try {
removeStaleAlbumDirs(collectionsDir, new Set(dirNames), originalsDir);
removeStaleAlbumDirs(
collectionsDir,
new Set(dirNames),
downloadDirectory,
);
} catch (err) {
log(`FAILED removing old album directories: ${errorMessage(err)}`);
}
@@ -553,13 +538,18 @@ export const runBackup = async (
const colDir = join(collectionsDir, colDirName);
mkdirSync(colDir, { recursive: true });
// Every album links the one original, saved from the file's entry in
// `distinct`.
const files = filesByCollection.get(c.id) ?? [];
const links = files.flatMap((f) =>
linksFor(f, storedOriginal(originalsDir, f)),
linksFor(
f,
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
),
);
const linkNames = uniqueNames(links, true);
try {
removeStaleLinks(colDir, new Set(linkNames), originalsDir);
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
} catch (err) {
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
}
+3 -5
View File
@@ -361,8 +361,8 @@ const openPart = async (
// written unpacked: each part is named `destination` with the extension
// replaced by its own entry's, and the two must differ ignoring case. When the
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
// part's own bytes. Only then is whatever was at `destination` removed and the
// image, then the video, renamed into place; on any failure neither is stored.
// part's own bytes. Only then are the image, then the video, renamed into
// place; on any failure neither is stored.
//
// The ZIP is chosen by its uploader and may expand enormously, so each part is
// written as it decompresses and never held, and the ZIP is refused once the
@@ -486,7 +486,6 @@ const decryptLivePhoto = async (
await part.handle.sync();
await part.handle.close();
}
await rm(destination, { force: true });
await rename(image.tmpPath, path);
try {
await rename(video.tmpPath, videoPath);
@@ -570,8 +569,7 @@ const fetchAndDecrypt = async (
}, api.getRetryOptions());
// Write `file`'s original to `outPath`. A live photo is written as its image
// and its video beside `outPath` instead, and whatever was at `outPath` is
// removed (see `decryptLivePhoto`).
// and its video beside `outPath` instead (see `decryptLivePhoto`).
export const downloadFile = async (
api: ApiClient,
file: EnteFile,
+80 -31
View File
@@ -1,19 +1,19 @@
// EXIF in an original's bytes, read with exifreader, which reads it from JPEG,
// HEIC/HEIF, AVIF, PNG, WebP and the other image formats it supports.
// `backup-metadata --exif` records every EXIF tag it finds except the
// thumbnail's; `Photo.exif()` returns the common fields picked from them here.
// thumbnail's. `Photo.exif()` returns every tag, the thumbnail's included, and
// `Photo`'s typed methods return the common fields picked from them here.
import ExifReader, { type ExpandedTags } from "exifreader";
// The EXIF tags in `bytes` (`exif`), the GPS position exifreader computes from
// them (`gps`), and where the EXIF block lies in `bytes` (`metadataRange`).
// The EXIF tags in `bytes` (`exif`), the embedded thumbnail's tags
// (`Thumbnail`), and where the EXIF block lies in `bytes` (`metadataRange`).
// Undefined when exifreader cannot read the file at all, such as a video. An
// EXIF block it finds but reads no tag from comes back as an empty `exif`.
// `exif` holds every tag except the thumbnail's; a tag exifreader has no name
// for is keyed `undefined-<tag number>`. Each tag's `computed` holds its value
// as a string or number, or as an array of them for a tag with several values,
// such as `GPSLatitude`'s `[40, 26, 46]`. A fraction with a zero denominator
// computes to null.
// EXIF block it finds but reads no tag from comes back as an empty `exif`. A
// tag exifreader has no name for is keyed `undefined-<tag number>`. Each tag's
// `computed` holds its value as a string or number, or as an array of them for
// a tag with several values, such as `GPSLatitude`'s `[40, 26, 46]`. A
// fraction with a zero denominator computes to null.
export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
try {
return ExifReader.loadView(
@@ -23,7 +23,7 @@ export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
computed: true,
includeOffsets: true,
includeUnknown: true,
includeTags: { exif: true, gps: true },
includeTags: { exif: true, thumbnail: true },
},
);
} catch {
@@ -31,7 +31,35 @@ export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
}
};
// The common EXIF fields of an original. Each is absent when the file lacks it.
// Every EXIF tag of an original, keyed by name, each as exifreader decodes it
// (see `readExifTags`). The embedded thumbnail's own tags are under
// `Thumbnail`, so its `Orientation` or `ImageWidth` cannot hide the main
// image's.
export type ExifTags = Omit<NonNullable<ExpandedTags["exif"]>, "Thumbnail"> & {
Thumbnail?: Omit<
NonNullable<ExpandedTags["Thumbnail"]>,
"type" | "image" | "base64"
>;
};
// Every EXIF tag in `bytes`: `{}` when the file has no EXIF, exifreader cannot
// read its EXIF, or it is not an image exifreader reads.
export const readAllExifTags = (bytes: Uint8Array): ExifTags => {
const tags = readExifTags(bytes);
if (!tags?.Thumbnail) return tags?.exif ?? {};
// exifreader puts the thumbnail's JPEG image beside its tags, as `type`,
// `image` and `base64`. The image is not a tag, so it is left out.
const {
type: _type,
image: _image,
base64: _base64,
...thumbnail
} = tags.Thumbnail;
return { ...tags.exif, Thumbnail: thumbnail };
};
// The common EXIF fields of an original, one for each of `Photo`'s typed
// methods. Each is absent when the file lacks it.
export interface PhotoExif {
make?: string;
model?: string;
@@ -80,30 +108,51 @@ const asDate = (v: unknown): Date | undefined => {
return Number.isNaN(date.getTime()) ? undefined : date;
};
// The common fields of an original's EXIF: `{}` when the file has no EXIF,
// exifreader cannot read its EXIF, or it is not an image exifreader reads.
export const readPhotoExif = (bytes: Uint8Array): PhotoExif => {
const tags = readExifTags(bytes);
const exif = tags?.exif;
const gps = tags?.gps;
const altitude = asNumber(exif?.GPSAltitude?.computed);
// GPSLatitude and GPSLongitude hold degrees, minutes and seconds, computed as
// three numbers. This is them in decimal degrees, negative when `ref`, the
// GPSLatitudeRef or GPSLongitudeRef tag, is `negativeRef` ("S" or "W").
// Without that tag the hemisphere is unknown, so it is undefined.
const asDegrees = (
dms: unknown,
ref: unknown,
negativeRef: string,
): number | undefined => {
if (!Array.isArray(dms) || ref === undefined) return undefined;
const [d, m, s] = dms.map(asNumber);
if (d === undefined || m === undefined || s === undefined) return undefined;
const degrees = d + m / 60 + s / 3600;
return ref === negativeRef ? -degrees : degrees;
};
// The common fields picked from an original's EXIF tags, `readAllExifTags`'s
// result: `{}` when there are none.
export const readPhotoExif = (tags: ExifTags): PhotoExif => {
const altitude = asNumber(tags.GPSAltitude?.computed);
const fields: PhotoExif = {
make: asString(exif?.Make?.computed),
model: asString(exif?.Model?.computed),
lensModel: asString(exif?.LensModel?.computed),
dateTimeOriginal: asDate(exif?.DateTimeOriginal?.computed),
offsetTimeOriginal: asString(exif?.OffsetTimeOriginal?.computed),
exposureTime: asNumber(exif?.ExposureTime?.computed),
fNumber: asNumber(exif?.FNumber?.computed),
make: asString(tags.Make?.computed),
model: asString(tags.Model?.computed),
lensModel: asString(tags.LensModel?.computed),
dateTimeOriginal: asDate(tags.DateTimeOriginal?.computed),
offsetTimeOriginal: asString(tags.OffsetTimeOriginal?.computed),
exposureTime: asNumber(tags.ExposureTime?.computed),
fNumber: asNumber(tags.FNumber?.computed),
// Only when the tag holds a single number, as most cameras write it.
iso: asNumber(exif?.ISOSpeedRatings?.computed),
focalLength: asNumber(exif?.FocalLength?.computed),
orientation: asNumber(exif?.Orientation?.computed),
gpsLatitude: asNumber(gps?.Latitude),
gpsLongitude: asNumber(gps?.Longitude),
iso: asNumber(tags.ISOSpeedRatings?.computed),
focalLength: asNumber(tags.FocalLength?.computed),
orientation: asNumber(tags.Orientation?.computed),
gpsLatitude: asDegrees(
tags.GPSLatitude?.computed,
tags.GPSLatitudeRef?.computed,
"S",
),
gpsLongitude: asDegrees(
tags.GPSLongitude?.computed,
tags.GPSLongitudeRef?.computed,
"W",
),
// A GPSAltitudeRef of 1 means the altitude is below sea level.
gpsAltitude:
altitude !== undefined && exif?.GPSAltitudeRef?.value === 1
altitude !== undefined && tags.GPSAltitudeRef?.value === 1
? -altitude
: altitude,
};
+6 -3
View File
@@ -1,5 +1,7 @@
// package.json is the one place the version is written. tsc copies it to
// dist/package.json, so this path resolves from source and from dist/src/.
// A build reports the version script/build stamps into dist/package.json;
// package.json's own version is reported only when running from source. tsc
// copies package.json to dist/package.json, so this path resolves from source
// and from dist/src/.
import pkg from "../package.json" with { type: "json" };
export const VERSION: string = pkg.version;
@@ -53,6 +55,7 @@ export {
type PhotoFilter,
type TimelineGroup,
type GroupBy,
type SavePathLookup,
type ContentSource,
type ContentResult,
type ContentEvent,
@@ -84,7 +87,7 @@ export type {
LibrarySnapshot,
LibraryChange,
} from "./library/records.js";
export type { PhotoExif } from "./exif.js";
export type { ExifTags, PhotoExif } from "./exif.js";
export { decryptCollection, decryptFile } from "./model/index.js";
export { downloadFile, downloadThumbnail } from "./download/index.js";
export type {
+157 -107
View File
@@ -23,19 +23,19 @@
// original with no recorded hash is stored unchecked, as the upstream client
// does; thumbnails have none. On top of that this module refuses to record a
// stored file that came out empty.
//
// It also names each original's save path under the download directory
// (`savePath`), where `Photo.download()` and `lib.backup()` put it. A copy in
// the cache does not count as saved there, but is copied there rather than
// fetched again.
import {
closeSync,
existsSync,
openSync,
readFileSync,
readSync,
statSync,
} from "node:fs";
import { existsSync, readFileSync, statSync } from "node:fs";
import {
chmod,
copyFile,
mkdir,
readdir,
rename,
rm,
stat,
statfs,
@@ -47,13 +47,15 @@ import type { ApiClient } from "../api/client.js";
import {
downloadFile,
downloadThumbnail,
fsyncPath,
type ProgressCallback,
removeLeftoverTempFiles,
writeAtomic,
} from "../download/index.js";
import { safeExtension } from "../filename.js";
import { safeExtension, withExtension } from "../filename.js";
import type { EnteFile } from "../model/types.js";
import type { Priority, RequestPools } from "./pools.js";
import { takenAtOf } from "./records.js";
const DIR_MODE = 0o700;
const FILE_MODE = 0o600;
@@ -108,11 +110,9 @@ export interface ContentOptions {
export interface PhotoContent {
original(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
thumbnail(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
// Where a backup stores the original, whether or not it is there yet. For
// a live photo not yet stored, it carries the title's extension, and the
// backup may store the image under a different one.
savePath(fileID: number): string | undefined;
isLocal(fileID: number): boolean;
// Put the original at the save path of `file`, the copy the `Photo` holds,
// and return it there.
download(file: EnteFile): Promise<ContentResult>;
}
export interface EnsureResult {
@@ -199,10 +199,10 @@ export interface ContentCacheOptions {
pools: RequestPools;
source: ContentSource;
cacheDirectory: string;
// The backup destination (issue-level `downloadDirectory`). An original
// already stored there by a backup counts as present, so the cache serves
// it rather than fetching a second copy.
downloadDirectory?: string;
// The root of the save paths. An original already stored at its save path
// counts as present, so the cache serves it rather than fetching a second
// copy.
downloadDirectory: string;
// Resolve any membership of a file; every membership shares the underlying
// content key, so any one decrypts the same bytes.
getFile: (fileID: number) => EnteFile | undefined;
@@ -229,11 +229,27 @@ class AbortDrop extends Error {
}
}
// The name of a file's original in originals/: `<fileID><ext>`, the extension
// taken from the title (or `.bin`). A backup names its originals the same way.
// The name of a file's original in the cache's originals/: `<fileID><ext>`, the
// extension taken from the title (or `.bin`).
export const nameInOriginals = (file: EnteFile): string =>
`${file.id}${safeExtension(file.metadata.title)}`;
const pad = (n: number): string => String(n).padStart(2, "0");
// Where the original of `file` is saved under `root`:
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`, dated by the photo's
// `takenAt` in this machine's time zone, with the extension taken from the
// title (or `.bin`). A live photo is stored as its image and its video beside
// this path, each with the extension found inside the live photo.
export const savePath = (root: string, file: EnteFile): string => {
const taken = new Date(takenAtOf(file));
const year = String(taken.getFullYear());
const month = `${year}-${pad(taken.getMonth() + 1)}`;
const day = `${month}-${pad(taken.getDate())}`;
const ext = safeExtension(file.metadata.title);
return join(root, year, month, day, `${day}.${file.id}${ext}`);
};
// The fileID a cache filename encodes, or undefined when the name is not one
// the cache writes (`<digits><ext>`).
const fileIDFromName = (name: string): number | undefined => {
@@ -258,49 +274,31 @@ const fileSize = (path: string): number | undefined => {
const hasContent = (path: string | undefined): boolean =>
path !== undefined && (fileSize(path) ?? 0) > 0;
// Whether the file at `path` begins as a ZIP does, with `PK\x03\x04`. False
// when it cannot be read.
const isZip = (path: string): boolean => {
try {
const fd = openSync(path, "r");
try {
const head = Buffer.alloc(4);
return (
readSync(fd, head, 0, 4, 0) === 4 &&
head.toString("latin1") === "PK\x03\x04"
);
} finally {
closeSync(fd);
}
} catch {
return false;
}
};
// A live photo's image and video are named with the extensions from inside its
// ZIP, so their names alone do not say which is which. Wherever the cache or a
// backup stores one, a JSON file of this name beside them names both.
const livePhotoJSONName = (fileID: number): string =>
`${fileID}.livephoto.json`;
// save path stores one, a JSON file of this name beside them names both.
// `name` is the original's name without its extension: `<fileID>` in the
// cache, `YYYY-MM-DD.<fileID>` at the save path.
const livePhotoJSONName = (name: string): string => `${name}.livephoto.json`;
// The image and video that the live photo's JSON file in `dir` names, or
// undefined when there is none. Only names of the form the cache writes are
// taken, so the file cannot point outside `dir`.
// undefined when there is none. Only `name` with an extension is taken, so the
// file cannot point outside `dir`.
const readLivePhotoJSON = (
dir: string,
fileID: number,
name: string,
): { path: string; videoPath: string } | undefined => {
const valid = (name: unknown): name is string =>
typeof name === "string" && name === `${fileID}${safeExtension(name)}`;
const valid = (part: unknown): part is string =>
typeof part === "string" && part === `${name}${safeExtension(part)}`;
try {
const { image, video } = JSON.parse(
readFileSync(join(dir, livePhotoJSONName(fileID)), "utf-8"),
readFileSync(join(dir, livePhotoJSONName(name)), "utf-8"),
);
if (valid(image) && valid(video)) {
return { path: join(dir, image), videoPath: join(dir, video) };
}
} catch {
// No such file, or not one the cache wrote.
// No such file, or not one quak wrote.
}
return undefined;
};
@@ -308,11 +306,11 @@ const readLivePhotoJSON = (
// Write the JSON file naming a live photo's image and video, both in `dir`.
export const writeLivePhotoJSON = (
dir: string,
fileID: number,
name: string,
stored: { path: string; videoPath: string },
): Promise<void> =>
writeAtomic(
join(dir, livePhotoJSONName(fileID)),
join(dir, livePhotoJSONName(name)),
new TextEncoder().encode(
JSON.stringify({
image: basename(stored.path),
@@ -321,17 +319,19 @@ export const writeLivePhotoJSON = (
),
);
// The original of `file` as the cache or a backup stored it in `dir`, when all
// of it is there: `<fileID><ext>`, or a live photo's image and video.
// The original of `file` as stored in `dir` under `name` (without its
// extension), when all of it is there: `<name><ext>`, or a live photo's image
// and video.
export const storedOriginal = (
dir: string,
name: string,
file: EnteFile,
): { path: string; videoPath?: string } | undefined => {
if (file.metadata.fileType !== "livePhoto") {
const path = join(dir, nameInOriginals(file));
const path = join(dir, `${name}${safeExtension(file.metadata.title)}`);
return hasContent(path) ? { path } : undefined;
}
const stored = readLivePhotoJSON(dir, file.id);
const stored = readLivePhotoJSON(dir, name);
return stored !== undefined &&
hasContent(stored.path) &&
hasContent(stored.videoPath)
@@ -339,10 +339,76 @@ export const storedOriginal = (
: undefined;
};
// The original of `file` as stored at its save path under `root`, when all of
// it is there.
export const storedAtSavePath = (
root: string,
file: EnteFile,
): { path: string; videoPath?: string } | undefined => {
const path = savePath(root, file);
return storedOriginal(dirname(path), basename(path, extname(path)), file);
};
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete"). As in the
// download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
export const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return;
const tmp = join(
dirname(dest),
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
.toString(36)
.slice(2)}.tmp`,
);
try {
await copyFile(src, tmp);
await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally {
await rm(tmp, { force: true });
}
};
// Put the original of `file` at its save path under `root`, creating its
// folders. `get` is given the save path and returns where the original is: a
// fetch writes it there, and a copy the cache holds is copied there. A live
// photo's image and video go beside the save path, each with its own
// extension: when they came from the cache they are copied. Then the JSON file
// naming them is written, which is what makes the live photo count as stored.
export const placeOriginal = async (
root: string,
file: EnteFile,
get: (dest: string) => Promise<{ path: string; videoPath?: string }>,
): Promise<{ path: string; videoPath?: string }> => {
const dest = savePath(root, file);
await mkdir(dirname(dest), { recursive: true });
const got = await get(dest);
if (got.videoPath === undefined) {
await copyAtomic(got.path, dest);
return { path: dest };
}
const path = withExtension(dest, extname(got.path));
const videoPath = withExtension(dest, extname(got.videoPath));
if (got.path !== path) {
await copyAtomic(got.path, path);
await copyAtomic(got.videoPath, videoPath);
}
const name = basename(dest, extname(dest));
await writeLivePhotoJSON(dirname(dest), name, { path, videoPath });
return { path, videoPath };
};
export class ContentCache implements PhotoContent, ThumbnailsAPI {
private readonly pools: RequestPools;
private readonly source: ContentSource;
private readonly downloadDirectory?: string;
private readonly downloadDirectory: string;
private readonly getFile: (fileID: number) => EnteFile | undefined;
private readonly originalsDir: string;
private readonly thumbnailsDir: string;
@@ -440,32 +506,23 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
}
// Where a backup to the download directory stores the file's original,
// whether or not it is there yet: for a live photo already stored, its
// image. For a live photo not yet stored, it carries the title's
// extension, and the backup may store the image under a different one.
// Undefined with no download directory.
savePath(fileID: number): string | undefined {
const file = this.getFile(fileID);
if (this.downloadDirectory === undefined || file === undefined)
return undefined;
const dir = join(this.downloadDirectory, "originals");
return (
storedOriginal(dir, file)?.path ?? join(dir, nameInOriginals(file))
);
// Put the original at the save path of `file` under the download directory
// and return it there. `file` is the copy the `Photo` holds, so the path is
// the one its `savePath` names, even after a refresh changed the date. One
// already stored there is returned as it is; one the cache holds is copied
// from it; any other is fetched straight to the save path, with no copy
// left in the cache.
async download(file: EnteFile): Promise<ContentResult> {
const root = this.downloadDirectory;
const saved =
storedAtSavePath(root, file) ??
(await placeOriginal(root, file, (dest) =>
this.backupOriginal(file.id, dest),
));
return { ...saved, bytes: fileSize(saved.path) ?? 0 };
}
// Whether the whole original is in the download directory, as a backup
// stores it. A copy only in the cache does not count.
isLocal(fileID: number): boolean {
const file = this.getFile(fileID);
if (this.downloadDirectory === undefined || file === undefined)
return false;
const dir = join(this.downloadDirectory, "originals");
return storedOriginal(dir, file) !== undefined;
}
// Get an original for a backup. One not present anywhere is written
// Get an original for a save path. One not present anywhere is written
// straight to `destination` and recorded there, so no second copy lands
// in the cache; one already present is returned where it is.
async backupOriginal(
@@ -630,17 +687,16 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
await this.touch(cached.path);
return { ...cached, bytes: size, cached: true };
}
// A recorded file that has since gone, or a live photo an earlier
// version stored as one ZIP, re-fetches below.
// A recorded file that has since gone re-fetches below. So does a
// live photo recorded with no video: the cache opened before the
// library's records said it is a live photo, while its image and
// video had no JSON file beside them yet.
known.delete(fileID);
}
// An original a backup already stored counts as present.
if (kind === "original" && this.downloadDirectory !== undefined) {
const stored = storedOriginal(
join(this.downloadDirectory, "originals"),
file,
);
// An original already stored at its save path counts as present.
if (kind === "original") {
const stored = storedAtSavePath(this.downloadDirectory, file);
if (stored !== undefined) {
this.originals.set(fileID, stored);
return {
@@ -676,7 +732,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
? this.beginOriginalWrite(fileID)
: null;
try {
const stored = await this.download(
const stored = await this.fetchInto(
file,
dest,
kind,
@@ -691,12 +747,12 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
);
}
}
// A backup records its own live photos.
// `placeOriginal` records a live photo it saves.
if (
stored.videoPath !== undefined &&
opts?.destination === undefined
) {
await writeLivePhotoJSON(dir, fileID, {
await writeLivePhotoJSON(dir, String(fileID), {
path: stored.path,
videoPath: stored.videoPath,
});
@@ -719,7 +775,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
// Fetch into `destination`, returning where the bytes landed: there, or
// for a live photo, its image and video beside it.
private async download(
private async fetchInto(
file: EnteFile,
destination: string,
kind: Kind,
@@ -744,10 +800,10 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
await utimes(path, now, now).catch(() => undefined);
}
// Every stored original that lives under `originalsDir` (a backup-directory
// hit recorded in the map is excluded), with its size and mtime; a live
// Every stored original that lives under `originalsDir` (a save-path hit
// recorded in the map is excluded), with its size and mtime; a live
// photo's size includes its video. Entries whose file has vanished are
// dropped from the map. Backups and thumbnails are never counted.
// dropped from the map. Save paths and thumbnails are never counted.
private async measureOriginals(): Promise<{
entries: {
fileID: number;
@@ -853,7 +909,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
await rm(
join(
this.originalsDir,
livePhotoJSONName(e.fileID),
livePhotoJSONName(String(e.fileID)),
),
{ force: true },
);
@@ -903,20 +959,14 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
if (id === undefined || !existsSync(path)) continue;
// A live photo's image and video are one entry, as the JSON file
// beside them names them. A live photo's file with no such JSON
// file is not its original. If it is a ZIP, it is the one an
// earlier version stored under the image's name, and is removed.
// Any other is left alone: another process may have just stored
// it and not yet written the JSON file.
const livePhoto = names.has(livePhotoJSONName(id))
? readLivePhotoJSON(dir, id)
// file is not its original and is left alone: another process may
// have just stored it and not yet written the JSON file.
const livePhoto = names.has(livePhotoJSONName(String(id)))
? readLivePhotoJSON(dir, String(id))
: undefined;
if (livePhoto !== undefined) {
into.set(id, livePhoto);
} else if (isLivePhoto(id)) {
if (isZip(path)) {
await rm(path, { force: true }).catch(() => undefined);
}
} else {
} else if (!isLivePhoto(id)) {
into.set(id, { path });
}
}
+41 -28
View File
@@ -27,7 +27,7 @@
// never masked by a subsequent empty refresh.
import { rm } from "node:fs/promises";
import { join } from "node:path";
import { join, resolve } from "node:path";
import envPaths from "env-paths";
import { MetadataStore } from "./store.js";
@@ -49,9 +49,12 @@ import {
type PhotosAPI,
type TimelineAPI,
type FreshReads,
type SavePathLookup,
} from "./read.js";
import {
ContentCache,
savePath,
storedAtSavePath,
type ContentSource,
type ThumbnailsAPI,
type EnsureOptions,
@@ -70,6 +73,7 @@ export {
type PhotoFilter,
type TimelineGroup,
type GroupBy,
type SavePathLookup,
} from "./read.js";
export {
type ContentSource,
@@ -169,8 +173,10 @@ export interface LibraryOptions {
// Where `metadata.json` lives. Defaults to the env-paths cache directory
// plus the user id, so each account has its own cache.
cacheDirectory?: string;
// Persistent backup destination. The refresh loop does not use it; the
// content cache treats an original already stored there as present.
// The root of every photo's save path, where `Photo.download()` and
// `lib.backup()` put originals. Defaults to `photos` in the working
// directory at open. The content cache treats an original already stored
// at its save path as present.
downloadDirectory?: string;
refreshIntervalSeconds?: number;
onProgress?: RefreshProgressCallback;
@@ -234,7 +240,7 @@ export interface LibraryStatus {
export class Library {
readonly cacheDirectory: string;
readonly downloadDirectory?: string;
readonly downloadDirectory: string;
// The in-process read surface (issue #44). Each namespace answers
// synchronously from the live record projection; no read touches the
@@ -296,7 +302,7 @@ export class Library {
store: MetadataStore;
userID: number;
cacheDirectory: string;
downloadDirectory?: string;
downloadDirectory: string;
intervalMs: number;
onProgress?: RefreshProgressCallback;
pools: RequestPools;
@@ -320,8 +326,14 @@ export class Library {
// The read namespaces derive fresh from the store on each call, so they
// always reflect the latest refresh.
const derive = (): DerivedRecords => this.deriveNow();
this.albums = makeAlbumsAPI(derive, this.cache);
this.photos = makePhotosAPI(derive, this.cache);
const root = this.downloadDirectory;
const saves: SavePathLookup = {
savePath: (file) =>
storedAtSavePath(root, file)?.path ?? savePath(root, file),
isLocal: (file) => storedAtSavePath(root, file) !== undefined,
};
this.albums = makeAlbumsAPI(derive, saves, this.cache);
this.photos = makePhotosAPI(derive, saves, this.cache);
this.timeline = makeTimelineAPI(derive);
this.thumbnails = {
ensure: (opts: EnsureOptions): Promise<EnsureResult[]> => {
@@ -350,6 +362,13 @@ export class Library {
const { userID } = opts.client.whoami();
const cacheDirectory =
opts.cacheDirectory ?? defaultCacheDirectory(userID);
if (opts.downloadDirectory === "") {
throw new Error(
"library: downloadDirectory is empty (leave it out to save " +
"under photos/ in the working directory)",
);
}
const downloadDirectory = opts.downloadDirectory ?? resolve("photos");
const metadataPath = join(cacheDirectory, "metadata.json");
let store = await MetadataStore.load(metadataPath);
// A cache directory given explicitly can hold another account's cache.
@@ -404,7 +423,7 @@ export class Library {
pools,
source,
cacheDirectory,
downloadDirectory: opts.downloadDirectory,
downloadDirectory,
getFile: (fileID) => store.getFileByID(fileID),
cacheOriginalsMaxBytes: opts.cacheOriginalsMaxBytes,
freeBelowBytes: opts.freeBelowBytes,
@@ -426,7 +445,7 @@ export class Library {
store,
userID,
cacheDirectory,
downloadDirectory: opts.downloadDirectory,
downloadDirectory,
intervalMs,
onProgress: opts.onProgress,
pools,
@@ -470,9 +489,10 @@ export class Library {
return this.store.getFile(collectionID, fileID);
}
// Any membership of a file, addressed by file id alone. A file's own
// metadata (title, creationTime) is identical across the collections it
// belongs to, so this serves the point commands that hold only a fileID.
// The membership of a file its record is read from, addressed by file id
// alone. A file's own metadata (title, creationTime) is identical across
// the collections it belongs to, so this serves the point commands that
// hold only a fileID.
getFileByID(fileID: number): EnteFile | undefined {
return this.store.getFileByID(fileID);
}
@@ -543,27 +563,20 @@ export class Library {
};
}
// Back up every in-scope file to `downloadDirectory` in the historical
// on-disk layout, with a durable failure ledger (issue #51). Waits for a
// completed refresh first, as `fresh()` does, joining one already running,
// and rejects before touching any file when it fails. Then fetches pending
// originals (and optional thumbnails) through the content cache and pools,
// and rebuilds the derived symlink/JSON views from the model. Throws before
// any network work when no download directory is available or no content
// cache backs the originals it must fetch.
// Back up every in-scope file to `opts.downloadDirectory`, or else the
// library's, each original at its save path, with a durable failure
// ledger (issue #51). Waits for a completed refresh first, as `fresh()`
// does, joining one already running, and rejects before touching any file
// when it fails. Then puts pending originals at their save paths as
// `Photo.download()` does (and optional thumbnails) through the content
// cache and pools, and rebuilds the derived symlink/JSON views from the
// model. Throws before any network work when no content cache backs the
// originals it must fetch.
backup(opts?: BackupOptions): Promise<BackupResult> {
const downloadDirectory =
opts?.downloadDirectory ?? this.downloadDirectory;
const includeOriginals = opts?.includeOriginals ?? true;
const includeThumbnails = opts?.includeThumbnails ?? false;
if (!downloadDirectory) {
return Promise.reject(
new Error(
"backup requires a downloadDirectory (pass one to " +
"backup() or open the library with one)",
),
);
}
if ((includeOriginals || includeThumbnails) && !this.cache) {
return Promise.reject(
new Error(
+81 -48
View File
@@ -12,19 +12,31 @@
// plain records are the serializable surface, and `record()` returns one.
//
// A `Photo` also fetches its own bytes: `original()`, `thumbnail()`,
// `content()`, `exif()` and the methods that each return one field of `exif()`
// go through the on-disk content cache (issue #46), and are the one place in
// this module that may touch the network. A library opened without a content
// source leaves that cache absent, and those methods then throw. `savePath` and
// `isLocal` look only at the disk.
// `download()`, `content()`, `exif()` and the methods that each return one
// EXIF field go through the on-disk content cache (issue #46), and are
// the one place in this module that may touch the network. A library opened
// without a content source leaves that cache absent, and those methods then
// throw. `savePath` and `isLocal` look only at the disk and need no cache.
import { readFile } from "node:fs/promises";
import { readPhotoExif, type PhotoExif } from "../exif.js";
import type { CollectionType, FileType } from "../model/types.js";
import {
readAllExifTags,
readPhotoExif,
type ExifTags,
type PhotoExif,
} from "../exif.js";
import type { CollectionType, EnteFile, FileType } from "../model/types.js";
import type { ContentOptions, ContentResult, PhotoContent } from "./content.js";
import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js";
// Where a photo's original is saved, and whether all of it is there. The
// library answers both from the disk, with or without a content cache.
export interface SavePathLookup {
savePath(file: EnteFile): string;
isLocal(file: EnteFile): boolean;
}
// Newest first, with fileID as a stable tiebreak so equal-timed files order
// deterministically — the same order the record projection uses.
const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
@@ -43,10 +55,14 @@ type PhotoExifMethods = {
};
// A single photo. Field access mirrors `PhotoRecord`; `record()` returns the
// underlying plain record for callers that need the IPC-safe value.
// underlying plain record for callers that need the IPC-safe value. `file` is
// the membership the record is read from, so the save path carries the date of
// `takenAt` and stays known after a refresh removes the file from the library.
export class Photo implements PhotoExifMethods {
constructor(
private readonly rec: PhotoRecord,
private readonly file: EnteFile,
private readonly saves: SavePathLookup,
private readonly cache?: PhotoContent,
) {}
@@ -97,20 +113,20 @@ export class Photo implements PhotoExifMethods {
return this.rec.isHidden;
}
// Where `lib.backup()` stores the original in the library's download
// directory, whether or not it is there yet: for a live photo already
// stored, its image. For a live photo not yet stored, it carries the
// title's extension, and the backup may store the image under a different
// one. Undefined when the library has no download directory or no content
// cache.
get savePath(): string | undefined {
return this.cache?.savePath(this.rec.fileID);
// Where `download()` and `lib.backup()` put the original under the
// library's download directory, whether or not it is there yet:
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`. For a live photo
// already stored, its image. For a live photo not yet stored, it carries
// the title's extension, and the image may be stored under a different
// one.
get savePath(): string {
return this.saves.savePath(this.file);
}
// Whether the whole original is at `savePath`. A copy only in the cache
// does not count.
get isLocal(): boolean {
return this.cache?.isLocal(this.rec.fileID) ?? false;
return this.saves.isLocal(this.file);
}
record(): PhotoRecord {
@@ -119,12 +135,20 @@ export class Photo implements PhotoExifMethods {
// Fetch and cache the full-resolution original, returning its on-disk path
// and byte length; for a live photo, its image's, and its video's path as
// `videoPath`. Served from the cache (or the backup download directory)
// when already present, otherwise fetched through the content pool.
// `videoPath`. Served from the cache (or the save path) when already
// present, otherwise fetched through the content pool.
async original(opts?: ContentOptions): Promise<ContentResult> {
return this.cacheOrThrow().original(this.rec.fileID, opts);
}
// Put the original at `savePath` and return it there, as `original()`
// does. When it is already there, nothing is written. When the cache holds
// it, it is copied from there; otherwise it is fetched straight to
// `savePath`.
async download(): Promise<ContentResult> {
return this.cacheOrThrow().download(this.file);
}
// As `original`, for the thumbnail, through the thumbnail pool.
async thumbnail(opts?: ContentOptions): Promise<ContentResult> {
return this.cacheOrThrow().thumbnail(this.rec.fileID, opts);
@@ -137,74 +161,75 @@ export class Photo implements PhotoExifMethods {
return readFile(path);
}
// The common EXIF fields of the original, read from `content()`, so this
// may download it. EXIF is read from any image format exifreader reads,
// JPEG and HEIC/HEIF among them; any other file gives `{}`, and a video
// gives it without fetching anything. Like the other content methods, it
// throws when there is no content cache, video or not.
async exif(opts?: ContentOptions): Promise<PhotoExif> {
// Every EXIF tag of the original, keyed by name (see `ExifTags`), read
// from `content()`, so this may download it. EXIF is read from any image
// format exifreader reads, JPEG and HEIC/HEIF among them; any other file
// gives `{}`, and a video gives it without fetching anything. Like the
// other content methods, it throws when there is no content cache, video
// or not.
async exif(opts?: ContentOptions): Promise<ExifTags> {
this.cacheOrThrow();
if (this.rec.fileType === "video") return {};
return readPhotoExif(await this.content(opts));
return readAllExifTags(await this.content(opts));
}
// One field of `exif()` each, named and typed as in `PhotoExif`, and
// undefined when the file lacks it. Each call runs `exif()`, which reads
// the original again.
// One field each, named and typed as in `PhotoExif`, picked from the tags
// `exif()` returns, and undefined when the file lacks it. Each call runs
// `exif()`, which reads the original again.
async make(opts?: ContentOptions): Promise<PhotoExif["make"]> {
return (await this.exif(opts)).make;
return readPhotoExif(await this.exif(opts)).make;
}
async model(opts?: ContentOptions): Promise<PhotoExif["model"]> {
return (await this.exif(opts)).model;
return readPhotoExif(await this.exif(opts)).model;
}
async lensModel(opts?: ContentOptions): Promise<PhotoExif["lensModel"]> {
return (await this.exif(opts)).lensModel;
return readPhotoExif(await this.exif(opts)).lensModel;
}
async dateTimeOriginal(
opts?: ContentOptions,
): Promise<PhotoExif["dateTimeOriginal"]> {
return (await this.exif(opts)).dateTimeOriginal;
return readPhotoExif(await this.exif(opts)).dateTimeOriginal;
}
async offsetTimeOriginal(
opts?: ContentOptions,
): Promise<PhotoExif["offsetTimeOriginal"]> {
return (await this.exif(opts)).offsetTimeOriginal;
return readPhotoExif(await this.exif(opts)).offsetTimeOriginal;
}
async exposureTime(
opts?: ContentOptions,
): Promise<PhotoExif["exposureTime"]> {
return (await this.exif(opts)).exposureTime;
return readPhotoExif(await this.exif(opts)).exposureTime;
}
async fNumber(opts?: ContentOptions): Promise<PhotoExif["fNumber"]> {
return (await this.exif(opts)).fNumber;
return readPhotoExif(await this.exif(opts)).fNumber;
}
async iso(opts?: ContentOptions): Promise<PhotoExif["iso"]> {
return (await this.exif(opts)).iso;
return readPhotoExif(await this.exif(opts)).iso;
}
async focalLength(
opts?: ContentOptions,
): Promise<PhotoExif["focalLength"]> {
return (await this.exif(opts)).focalLength;
return readPhotoExif(await this.exif(opts)).focalLength;
}
async orientation(
opts?: ContentOptions,
): Promise<PhotoExif["orientation"]> {
return (await this.exif(opts)).orientation;
return readPhotoExif(await this.exif(opts)).orientation;
}
async gpsLatitude(
opts?: ContentOptions,
): Promise<PhotoExif["gpsLatitude"]> {
return (await this.exif(opts)).gpsLatitude;
return readPhotoExif(await this.exif(opts)).gpsLatitude;
}
async gpsLongitude(
opts?: ContentOptions,
): Promise<PhotoExif["gpsLongitude"]> {
return (await this.exif(opts)).gpsLongitude;
return readPhotoExif(await this.exif(opts)).gpsLongitude;
}
async gpsAltitude(
opts?: ContentOptions,
): Promise<PhotoExif["gpsAltitude"]> {
return (await this.exif(opts)).gpsAltitude;
return readPhotoExif(await this.exif(opts)).gpsAltitude;
}
private cacheOrThrow(): PhotoContent {
@@ -223,6 +248,7 @@ export class Album {
constructor(
private readonly rec: AlbumRecord,
private readonly records: DerivedRecords,
private readonly saves: SavePathLookup,
private readonly content?: PhotoContent,
) {}
@@ -257,7 +283,10 @@ export class Album {
const out: Photo[] = [];
for (const id of this.rec.fileIDs) {
const p = this.records.photos.get(id);
if (p) out.push(new Photo(p, this.content));
const file = this.records.files.get(id);
if (p && file) {
out.push(new Photo(p, file, this.saves, this.content));
}
}
return out;
}
@@ -320,18 +349,19 @@ export interface FreshReads {
export const makeAlbumsAPI = (
derive: () => DerivedRecords,
saves: SavePathLookup,
content?: PhotoContent,
): AlbumsAPI => ({
list: (): Album[] => {
const records = derive();
return [...records.albums.values()]
.sort(byNewestAlbum)
.map((rec) => new Album(rec, records, content));
.map((rec) => new Album(rec, records, saves, content));
},
byID: ({ collectionID }): Album | undefined => {
const records = derive();
const rec = records.albums.get(collectionID);
return rec ? new Album(rec, records, content) : undefined;
return rec ? new Album(rec, records, saves, content) : undefined;
},
byName: ({ albumName }): Album | undefined => {
const records = derive();
@@ -340,17 +370,20 @@ export const makeAlbumsAPI = (
const match = [...records.albums.values()]
.sort(byNewestAlbum)
.find((rec) => rec.name === albumName);
return match ? new Album(match, records, content) : undefined;
return match ? new Album(match, records, saves, content) : undefined;
},
});
export const makePhotosAPI = (
derive: () => DerivedRecords,
saves: SavePathLookup,
content?: PhotoContent,
): PhotosAPI => ({
byID: ({ fileID }): Photo | undefined => {
const rec = derive().photos.get(fileID);
return rec ? new Photo(rec, content) : undefined;
const records = derive();
const rec = records.photos.get(fileID);
const file = records.files.get(fileID);
return rec && file ? new Photo(rec, file, saves, content) : undefined;
},
records: ({ fileIDs }): PhotoRecord[] => {
const { photos } = derive();
+33 -15
View File
@@ -86,6 +86,10 @@ export interface LibraryChange {
export interface DerivedRecords {
albums: Map<number, AlbumRecord>;
photos: Map<number, PhotoRecord>;
// The membership each photo's record is read from, for its `Photo`'s save
// path. It holds the file's key, so it stays in this process: no snapshot
// or change carries it.
files: Map<number, EnteFile>;
}
const asString = (v: unknown): string | undefined =>
@@ -101,18 +105,19 @@ const microsToMillis = (micros: number): number => Math.floor(micros / 1000);
const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number =>
b.takenAt - a.takenAt || b.fileID - a.fileID;
// Build one PhotoRecord from every membership of a file. The memberships share
// the same underlying file, so metadata is read from a single representative
// (the most recently synced, lowest collection id to break ties); `albumIDs`
// gathers them all.
const toPhotoRecord = (
fileID: number,
memberships: EnteFile[],
): PhotoRecord => {
const albumIDs = memberships
.map((m) => m.collectionID)
.sort((a, b) => a - b);
const rep = memberships.reduce((best, m) =>
// A photo's `takenAt` in milliseconds: `pubMagicMetadata.editedTime` when the
// user edited the date, else basic-metadata `creationTime`.
export const takenAtOf = (file: EnteFile): number =>
microsToMillis(
asNumber(file.pubMagicMetadata?.editedTime) ??
file.metadata.creationTime,
);
// The membership a file's record is read from: the most recently synced, lowest
// collection id to break ties. Whatever dates a file's save path takes this
// membership too, so the path always carries the record's `takenAt`.
export const representative = (memberships: EnteFile[]): EnteFile =>
memberships.reduce((best, m) =>
m.updationTime > best.updationTime ||
(m.updationTime === best.updationTime &&
m.collectionID < best.collectionID)
@@ -120,17 +125,28 @@ const toPhotoRecord = (
: best,
);
// Build one PhotoRecord from every membership of a file. The memberships share
// the same underlying file, so metadata is read from a single representative;
// `albumIDs` gathers them all.
const toPhotoRecord = (
fileID: number,
memberships: EnteFile[],
): PhotoRecord => {
const albumIDs = memberships
.map((m) => m.collectionID)
.sort((a, b) => a - b);
const rep = representative(memberships);
const pub = rep.pubMagicMetadata ?? {};
const priv = rep.magicMetadata ?? {};
const takenAtMicros = asNumber(pub.editedTime) ?? rep.metadata.creationTime;
const visibility = asNumber(priv.visibility);
const record: PhotoRecord = {
fileID,
albumIDs,
title: asString(pub.editedName) ?? rep.metadata.title,
takenAt: microsToMillis(takenAtMicros),
takenAt: takenAtOf(rep),
modifiedAt: microsToMillis(rep.metadata.modificationTime),
fileType: rep.metadata.fileType,
isArchived: visibility === VISIBILITY_ARCHIVED,
@@ -198,6 +214,7 @@ export const deriveRecords = (
}
const photos = new Map<number, PhotoRecord>();
const photoFiles = new Map<number, EnteFile>();
const takenAtByFile = new Map<number, number>();
for (const [fileID, memberships] of byFileID) {
const record = toPhotoRecord(fileID, memberships);
@@ -209,6 +226,7 @@ export const deriveRecords = (
record.thumbnailPath = paths.thumbnailPath;
}
photos.set(fileID, record);
photoFiles.set(fileID, representative(memberships));
takenAtByFile.set(fileID, record.takenAt);
}
@@ -217,7 +235,7 @@ export const deriveRecords = (
albums.set(c.id, toAlbumRecord(c, files, takenAtByFile));
}
return { albums, photos };
return { albums, photos, files: photoFiles };
};
// Sorted, GUI-ready arrays: albums newest updated first, photos newest first.
+8 -5
View File
@@ -16,6 +16,7 @@ import { dirname } from "node:path";
import { writeAtomic } from "../download/index.js";
import type { Collection, EnteFile, Microseconds } from "../model/types.js";
import { representative } from "./records.js";
// Bumped only when the on-disk shape changes incompatibly. A file written
// under a different version is discarded on load (see `load`): re-fetching
@@ -179,14 +180,16 @@ export class MetadataStore {
return this.files.get(fileKey(collectionID, fileID));
}
// Any membership of a file, or undefined. Every membership re-wraps the
// same underlying content key, so any one is enough to fetch the bytes;
// the content cache resolves a fileID to a file this way.
// The membership of a file its record is read from (`representative`), or
// undefined. Any membership could fetch the bytes, but the content cache
// resolves a fileID to a file this way so that it dates the save path
// from the same membership as `photo.savePath`.
getFileByID(fileID: number): EnteFile | undefined {
const memberships: EnteFile[] = [];
for (const file of this.files.values()) {
if (file.id === fileID) return file;
if (file.id === fileID) memberships.push(file);
}
return undefined;
return memberships.length > 0 ? representative(memberships) : undefined;
}
listFiles(collectionID: number): EnteFile[] {
+178 -146
View File
@@ -1,16 +1,16 @@
/**
* Tests for the `quak backup` logic, now built on the library API (issue #51).
*
* `lib.backup({ downloadDirectory })` refreshes the library, fetches each
* pending file's original through the content cache/pools, and materialises the
* unchanged on-disk layout:
* `lib.backup({ downloadDirectory })` refreshes the library, puts each pending
* file's original at its save path through the content cache/pools, and
* materialises the on-disk layout:
*
* <downloadDirectory>/
* originals/
* <fileID>.<ext> the decrypted bytes ("present means complete")
* <fileID>.json per-file metadata sidecar (rebuilt each run)
* YYYY/YYYY-MM/YYYY-MM-DD/
* YYYY-MM-DD.<fileID>.<ext> the decrypted bytes ("present means complete")
* YYYY-MM-DD.<fileID>.json per-file metadata sidecar (rebuilt each run)
* collections/
* <name>/<title> symlink into ../originals (rebuilt each run)
* <name>/<title> symlink to the original (rebuilt each run)
* <name>.json per-collection metadata (rebuilt each run)
* failures.json durable ledger of unresolved failures
*
@@ -94,6 +94,18 @@ const USER_ID = 42;
// Decrypted-byte length each stub original writes, keyed by fileID.
const SIZE_BY_ID: Record<number, number> = { 100: 3000, 101: 2000, 200: 1500 };
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
// stores times, so the machine's time zone cannot move it to another day. Its
// original is saved in the folder `DAY`, named `2026-03-01.<fileID>.<ext>`.
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
const DAY = join("2026", "2026-03", "2026-03-01");
// Where the backup in `outDir` saves `name` (`<fileID>.<ext>`), and the target
// of an album folder's symlink to it.
const saved = (outDir: string, name: string): string =>
join(outDir, DAY, `2026-03-01.${name}`);
const linkTo = (name: string): string => `../../${DAY}/2026-03-01.${name}`;
const collection = (id: number, name: string): Collection => ({
id,
ownerID: USER_ID,
@@ -112,7 +124,7 @@ const file = (id: number, collectionID: number, title: string): EnteFile => ({
metadata: {
title,
fileType: "image",
creationTime: 1,
creationTime: TAKEN,
modificationTime: 1,
},
file: { decryptionHeader: "aGVhZGVy" },
@@ -253,7 +265,11 @@ describe("lib.backup", () => {
it("throws before any network when no downloadDirectory is given", async () => {
const source = stubSource();
const lib = await openLibrary(source);
await expect(lib.backup()).rejects.toThrow(/downloadDirectory/i);
// The library always has a download directory, so only an empty one
// given to backup() reaches runBackup as none.
await expect(lib.backup({ downloadDirectory: "" })).rejects.toThrow(
/downloadDirectory/i,
);
expect(source.originalCalls).toBe(0);
lib.close();
});
@@ -271,28 +287,22 @@ describe("lib.backup", () => {
expect(result.failed).toBe(0);
expect(result.errors).toEqual([]);
// Originals under originals/<fileID>.<ext>.
expect(readFileSync(join(outDir, "originals", "100.jpg")).length).toBe(
3000,
);
expect(readFileSync(join(outDir, "originals", "101.jpg")).length).toBe(
2000,
);
expect(readFileSync(join(outDir, "originals", "200.png")).length).toBe(
1500,
);
// Originals at YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>.
expect(readFileSync(saved(outDir, "100.jpg")).length).toBe(3000);
expect(readFileSync(saved(outDir, "101.jpg")).length).toBe(2000);
expect(readFileSync(saved(outDir, "200.png")).length).toBe(1500);
// Per-file metadata sidecar.
const sidecar = JSON.parse(
readFileSync(join(outDir, "originals", "100.json"), "utf-8"),
readFileSync(saved(outDir, "100.json"), "utf-8"),
);
expect(sidecar.id).toBe(100);
expect(sidecar.metadata.title).toBe("beach.jpg");
// Collection dirs contain symlinks into ../originals.
// Collection dirs contain symlinks into the date folders.
const beach = join(outDir, "collections", "Vacation", "beach.jpg");
expect(lstatSync(beach).isSymbolicLink()).toBe(true);
expect(readlinkSync(beach)).toContain("originals");
expect(readlinkSync(beach)).toContain(DAY);
expect(readFileSync(beach).length).toBe(3000);
// Per-collection metadata JSON.
@@ -316,7 +326,7 @@ describe("lib.backup", () => {
expect(result.failed).toBe(0);
// The title has no usable extension, so the original is `.bin`.
expect(existsSync(join(outDir, "originals", "300.bin"))).toBe(true);
expect(existsSync(saved(outDir, "300.bin"))).toBe(true);
const link = join(
outDir,
"collections",
@@ -366,9 +376,9 @@ describe("lib.backup", () => {
expect(result.errors[0]!.title).toBe("sunset.jpg");
// The two good files are on disk; the failed one is not.
expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(true);
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
expect(existsSync(join(outDir, "originals", "101.jpg"))).toBe(false);
expect(existsSync(saved(outDir, "100.jpg"))).toBe(true);
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
expect(existsSync(saved(outDir, "101.jpg"))).toBe(false);
expect(
existsSync(join(outDir, "collections", "Vacation", "sunset.jpg")),
).toBe(false);
@@ -403,7 +413,7 @@ describe("lib.backup", () => {
const r3 = await lib.backup({ downloadDirectory: outDir });
expect(r3.failed).toBe(0);
expect(r3.skipped).toBe(2);
expect(existsSync(join(outDir, "originals", "101.jpg"))).toBe(true);
expect(existsSync(saved(outDir, "101.jpg"))).toBe(true);
// A ledger with no remaining failures is removed.
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
lib.close();
@@ -423,8 +433,8 @@ describe("lib.backup", () => {
const result = await lib.backup({ downloadDirectory: outDir });
// Every original still downloads despite the symlink failure.
expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(true);
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
expect(existsSync(saved(outDir, "100.jpg"))).toBe(true);
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
// The other symlinks are still built.
expect(
lstatSync(
@@ -454,7 +464,7 @@ describe("lib.backup", () => {
await lib.backup({ downloadDirectory: outDir });
// Corrupt a sidecar and delete a symlink between runs.
writeFileSync(join(outDir, "originals", "100.json"), "not json");
writeFileSync(saved(outDir, "100.json"), "not json");
rmSync(join(outDir, "collections", "Vacation", "beach.jpg"));
const result = await lib.backup({ downloadDirectory: outDir });
@@ -462,7 +472,7 @@ describe("lib.backup", () => {
// The derived views are repaired from the model.
const sidecar = JSON.parse(
readFileSync(join(outDir, "originals", "100.json"), "utf-8"),
readFileSync(saved(outDir, "100.json"), "utf-8"),
);
expect(sidecar.metadata.title).toBe("beach.jpg");
expect(
@@ -485,8 +495,8 @@ describe("lib.backup", () => {
expect(result.totalFiles).toBe(1);
expect(result.downloaded).toBe(1);
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(false);
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
expect(existsSync(saved(outDir, "100.jpg"))).toBe(false);
expect(existsSync(join(outDir, "collections", "Work.json"))).toBe(true);
expect(existsSync(join(outDir, "collections", "Vacation.json"))).toBe(
false,
@@ -530,7 +540,7 @@ describe("lib.backup", () => {
expect(result.totalFiles).toBe(1);
expect(result.failed).toBe(0);
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
lib.close();
});
@@ -572,7 +582,7 @@ describe("lib.backup", () => {
it("fetches each original once and writes it only into the backup", async () => {
// A backup of a 500 GB account must write 500 GB, not a copy in the
// cache as well: an original fetched for the backup goes straight
// into its originals/, and the cache records it there.
// to its save path, and the cache records it there.
const source = stubSource();
const lib = await openLibrary(source);
const outDir = join(root, "backup");
@@ -582,23 +592,78 @@ describe("lib.backup", () => {
expect(result.downloaded).toBe(3);
expect(source.originalCalls).toBe(3);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
const stored = readdirSync(join(outDir, "originals")).filter(
const stored = readdirSync(join(outDir, DAY)).filter(
(name) => !name.endsWith(".json"),
);
expect(stored.sort()).toEqual(["100.jpg", "101.jpg", "200.png"]);
expect(stored.sort()).toEqual([
"2026-03-01.100.jpg",
"2026-03-01.101.jpg",
"2026-03-01.200.png",
]);
// The cache counts the backup's copy as present: reading the
// original afterwards fetches nothing and answers with that copy.
const read = await lib.photos.byID({ fileID: 100 })!.original();
expect(read.path).toBe(join(outDir, "originals", "100.jpg"));
expect(read.path).toBe(saved(outDir, "100.jpg"));
expect(source.originalCalls).toBe(3);
await lib.close();
});
it("saves a file in two albums at its photo's save path and links it from both", async () => {
// The date edited in Ente has reached Work's copy, synced later, but
// Vacation's copy still has an earlier edit.
const older = file(100, 1, "beach.jpg");
older.pubMagicMetadata = {
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
};
const newer = file(100, 2, "beach.jpg");
newer.updationTime = 2;
newer.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
class SharedFileClient extends MockClient {
override async filesSince(args: {
collectionID: number;
}): Promise<FilesPage> {
const files = [args.collectionID === 1 ? older : newer];
return { files, deleted: [], cursor: 1 };
}
}
const outDir = join(root, "backup");
const lib = await Library.open({
client: new SharedFileClient(),
cacheDirectory: join(root, "cache"),
downloadDirectory: outDir,
contentSource: stubSource(),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const photo = lib.photos.byID({ fileID: 100 })!;
const result = await lib.backup();
expect(result.totalFiles).toBe(1);
expect(result.downloaded).toBe(1);
expect(result.failed).toBe(0);
expect(photo.savePath).toBe(
join(outDir, "2026", "2026-04", "2026-04-15", "2026-04-15.100.jpg"),
);
expect(photo.isLocal).toBe(true);
expect(existsSync(join(outDir, "2026", "2026-02"))).toBe(false);
const target = "../../2026/2026-04/2026-04-15/2026-04-15.100.jpg";
for (const album of ["Vacation", "Work"]) {
expect(
readlinkSync(join(outDir, "collections", album, "beach.jpg")),
).toBe(target);
}
await lib.close();
});
it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource());
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const dest = join(originals, "100.jpg");
const day = join(outDir, DAY);
const dest = saved(outDir, "100.jpg");
// Only an original already in the cache is copied into the backup;
// one fetched for the backup is written there by the download writer.
await lib.photos.byID({ fileID: 100 })!.original();
@@ -609,34 +674,50 @@ describe("lib.backup", () => {
const at = fsEvents.indexOf(`rename:${dest}`);
expect(at).toBeGreaterThan(0);
expect(fsEvents[at - 1]).toMatch(
/^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
/^sync:.*\/\.quak-backup-2026-03-01\.100\.jpg-\d+-[0-9a-z]*\.tmp$/,
);
expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
expect(fsEvents[at + 1]).toBe(`sync:${day}`);
lib.close();
});
it("removes temp files left by a killed backup but not those of one still running", async () => {
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
mkdirSync(originals, { recursive: true });
const day = join(outDir, DAY);
mkdirSync(day, { recursive: true });
// A child that has already exited: its process ID is not running.
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
const leftover = `.quak-backup-2026-03-01.100.jpg-${exitedPID}-abc123.tmp`;
// This test's own process stands in for a backup running at the same
// time.
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(originals, leftover), "partial");
writeFileSync(join(originals, inProgress), "partial");
const inProgress = `.quak-backup-2026-03-01.101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(day, leftover), "partial");
writeFileSync(join(day, inProgress), "partial");
const lib = await openLibrary(stubSource());
await lib.backup({ downloadDirectory: outDir });
const names = readdirSync(originals);
const names = readdirSync(day);
expect(names).not.toContain(leftover);
expect(names).toContain(inProgress);
lib.close();
});
it("removes temp files left in a date folder no file in the backup is saved in", async () => {
const outDir = join(root, "backup");
// As for a file since deleted, or given another date, after a run was
// killed while writing it.
const otherDay = join(outDir, "2025", "2025-01", "2025-01-02");
mkdirSync(otherDay, { recursive: true });
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
writeFileSync(join(otherDay, `.quak-${exitedPID}-abc123.tmp`), "x");
const lib = await openLibrary(stubSource());
await lib.backup({ downloadDirectory: outDir });
expect(readdirSync(otherDay)).toEqual([]);
lib.close();
});
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
const outDir = join(root, "backup");
const thumbnails = join(outDir, "thumbnails");
@@ -709,7 +790,7 @@ describe("the refresh before a backup", () => {
const result = await backup;
expect(result.totalFiles).toBe(4);
expect(existsSync(join(outDir, "originals", "300.jpg"))).toBe(true);
expect(existsSync(saved(outDir, "300.jpg"))).toBe(true);
await lib.close();
});
@@ -728,7 +809,7 @@ describe("the refresh before a backup", () => {
expect(source.originalCalls).toBe(0);
expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore);
expect(existsSync(join(outDir, "originals"))).toBe(false);
expect(existsSync(join(outDir, DAY))).toBe(false);
await lib.close();
});
@@ -819,13 +900,13 @@ describe("backup album folders", () => {
expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([
"Trip (10)/",
"Trip (10)/IMG_0001 (1).JPG -> ../../originals/1.JPG",
"Trip (10)/IMG_0001 (2).JPG -> ../../originals/2.JPG",
"Trip (10)/img_0001 (4).jpg -> ../../originals/4.jpg",
"Trip (10)/other.jpg -> ../../originals/3.jpg",
`Trip (10)/IMG_0001 (1).JPG -> ${linkTo("1.JPG")}`,
`Trip (10)/IMG_0001 (2).JPG -> ${linkTo("2.JPG")}`,
`Trip (10)/img_0001 (4).jpg -> ${linkTo("4.jpg")}`,
`Trip (10)/other.jpg -> ${linkTo("3.jpg")}`,
"Trip (10).json",
"Trip (11)/",
"Trip (11)/other.jpg -> ../../originals/3.jpg",
`Trip (11)/other.jpg -> ${linkTo("3.jpg")}`,
"Trip (11).json",
]);
expect(albumID(outDir, "Trip (10).json")).toBe(10);
@@ -858,14 +939,14 @@ describe("backup album folders", () => {
expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([
"Trip (10)/",
"Trip (10)/IMG (6) (5).JPG -> ../../originals/5.JPG",
"Trip (10)/IMG (6).JPG -> ../../originals/6.JPG",
"Trip (10)/IMG (7).JPG -> ../../originals/7.JPG",
`Trip (10)/IMG (6) (5).JPG -> ${linkTo("5.JPG")}`,
`Trip (10)/IMG (6).JPG -> ${linkTo("6.JPG")}`,
`Trip (10)/IMG (7).JPG -> ${linkTo("7.JPG")}`,
"Trip (10).json",
"Trip (11)/",
"Trip (11)/a.jpg -> ../../originals/8.jpg",
`Trip (11)/a.jpg -> ${linkTo("8.jpg")}`,
"Trip (11) (12)/",
"Trip (11) (12)/b.jpg -> ../../originals/9.jpg",
`Trip (11) (12)/b.jpg -> ${linkTo("9.jpg")}`,
"Trip (11) (12).json",
"Trip (11).json",
]);
@@ -931,13 +1012,13 @@ describe("backup album folders", () => {
expect(scoped.failed).toBe(0);
expect(before).toEqual([
"Trip (10)/",
"Trip (10)/a.jpg -> ../../originals/1.jpg",
`Trip (10)/a.jpg -> ${linkTo("1.jpg")}`,
"Trip (10).json",
"Work/",
"Work/c.jpg -> ../../originals/3.jpg",
`Work/c.jpg -> ${linkTo("3.jpg")}`,
"Work.json",
"trip (11)/",
"trip (11)/b.jpg -> ../../originals/2.jpg",
`trip (11)/b.jpg -> ${linkTo("2.jpg")}`,
"trip (11).json",
]);
expect(tree(outDir)).toEqual(before);
@@ -990,13 +1071,13 @@ describe("backup album folders", () => {
"Mine/",
"Mine/keep.txt",
"Office/",
"Office/a.jpg -> ../../originals/5.jpg",
`Office/a.jpg -> ${linkTo("5.jpg")}`,
"Office.json",
"Trip/",
"Trip/IMG_0001.JPG -> ../../originals/1.JPG",
`Trip/IMG_0001.JPG -> ${linkTo("1.JPG")}`,
"Trip/mine -> ../elsewhere",
"Trip/notes.txt",
"Trip/other.jpg -> ../../originals/3.jpg",
`Trip/other.jpg -> ${linkTo("3.jpg")}`,
"Trip.json",
"Work/",
"Work/keep.txt",
@@ -1005,7 +1086,7 @@ describe("backup album folders", () => {
});
// One album backed up, then a folder the user made beside it holding a
// symlink into originals/, with `json` (if given) as its sibling JSON.
// symlink to an original, with `json` (if given) as its sibling JSON.
const backupWithUserFolder = async (
json: string | undefined,
): Promise<{ outDir: string; failed: number }> => {
@@ -1019,10 +1100,7 @@ describe("backup album folders", () => {
await runBackup(lib, { downloadDirectory: outDir });
const collectionsDir = join(outDir, "collections");
mkdirSync(join(collectionsDir, "Mine"));
symlinkSync(
"../../originals/1.jpg",
join(collectionsDir, "Mine", "a.jpg"),
);
symlinkSync(linkTo("1.jpg"), join(collectionsDir, "Mine", "a.jpg"));
if (json !== undefined) {
writeFileSync(join(collectionsDir, "Mine.json"), json);
}
@@ -1036,9 +1114,9 @@ describe("backup album folders", () => {
expect(failed).toBe(0);
expect(tree(outDir)).toEqual([
"Mine/",
"Mine/a.jpg -> ../../originals/1.jpg",
`Mine/a.jpg -> ${linkTo("1.jpg")}`,
"Trip/",
"Trip/a.jpg -> ../../originals/1.jpg",
`Trip/a.jpg -> ${linkTo("1.jpg")}`,
"Trip.json",
]);
});
@@ -1050,10 +1128,10 @@ describe("backup album folders", () => {
expect(failed).toBe(0);
expect(tree(outDir)).toEqual([
"Mine/",
"Mine/a.jpg -> ../../originals/1.jpg",
`Mine/a.jpg -> ${linkTo("1.jpg")}`,
"Mine.json",
"Trip/",
"Trip/a.jpg -> ../../originals/1.jpg",
`Trip/a.jpg -> ${linkTo("1.jpg")}`,
"Trip.json",
]);
expect(
@@ -1087,23 +1165,16 @@ describe("backup of live photos", () => {
const open = (files: EnteFile[], bodies: Map<number, Uint8Array>) =>
openLibrary(cdnSource(bodies), new TripClient(files));
// What an earlier version stored for live photo 500: the ZIP under the
// image's name, and its link.
const earlierZIP = (outDir: string): void => {
mkdirSync(join(outDir, "originals"), { recursive: true });
mkdirSync(join(outDir, "collections", "Trip"), { recursive: true });
writeFileSync(join(outDir, "originals", "500.HEIC"), livePhotoZip());
symlinkSync(
"../../originals/500.HEIC",
join(outDir, "collections", "Trip", "IMG_0500.HEIC"),
);
};
const stored = ["500.heic", "500.json", "500.livephoto.json", "500.mov"];
const stored = [
"2026-03-01.500.heic",
"2026-03-01.500.json",
"2026-03-01.500.livephoto.json",
"2026-03-01.500.mov",
];
const linked = [
"Trip/",
"Trip/IMG_0500.heic -> ../../originals/500.heic",
"Trip/IMG_0500.mov -> ../../originals/500.mov",
`Trip/IMG_0500.heic -> ${linkTo("500.heic")}`,
`Trip/IMG_0500.mov -> ${linkTo("500.mov")}`,
"Trip.json",
];
@@ -1113,16 +1184,15 @@ describe("backup of live photos", () => {
);
const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(originals).sort()).toEqual(stored);
expect(readFileSync(join(originals, "500.heic"))).toEqual(
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
expect(readFileSync(saved(outDir, "500.heic"))).toEqual(
Buffer.from(IMAGE),
);
expect(readFileSync(join(originals, "500.mov"))).toEqual(
expect(readFileSync(saved(outDir, "500.mov"))).toEqual(
Buffer.from(VIDEO),
);
expect(tree(outDir)).toEqual(linked);
@@ -1149,39 +1219,22 @@ describe("backup of live photos", () => {
expect(tree(outDir)).toEqual([
"Trip/",
"Trip/IMG_0001 (500).heic -> ../../originals/500.heic",
"Trip/IMG_0001 (500).mov -> ../../originals/500.mov",
"Trip/IMG_0001 (501).heic -> ../../originals/501.heic",
"Trip/IMG_0001 (501).mov -> ../../originals/501.mov",
`Trip/IMG_0001 (500).heic -> ${linkTo("500.heic")}`,
`Trip/IMG_0001 (500).mov -> ${linkTo("500.mov")}`,
`Trip/IMG_0001 (501).heic -> ${linkTo("501.heic")}`,
`Trip/IMG_0001 (501).mov -> ${linkTo("501.mov")}`,
"Trip.json",
]);
await lib.close();
});
it("replaces the ZIP an earlier version stored, and its link", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const outDir = join(root, "backup");
earlierZIP(outDir);
const lib = await open([live], new Map([[500, body]]));
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(tree(outDir)).toEqual(linked);
await lib.close();
});
it("stores nothing for a live photo that fails its hash, and keeps what was there", async () => {
it("stores nothing for a live photo that fails its hash", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
livePhotoZip(),
"not:the recorded hash",
);
const outDir = join(root, "backup");
earlierZIP(outDir);
const lib = await open([live], new Map([[500, body]]));
const result = await lib.backup({ downloadDirectory: outDir });
@@ -1189,12 +1242,8 @@ describe("backup of live photos", () => {
expect(result).toMatchObject({ downloaded: 0, failed: 1 });
expect(result.errors.map((e) => e.fileID)).toEqual([500]);
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
expect(readdirSync(join(outDir, "originals"))).toEqual(["500.HEIC"]);
expect(tree(outDir)).toEqual([
"Trip/",
"Trip/IMG_0500.HEIC -> ../../originals/500.HEIC",
"Trip.json",
]);
expect(readdirSync(join(outDir, DAY))).toEqual([]);
expect(tree(outDir)).toEqual(["Trip/", "Trip.json"]);
await lib.close();
});
@@ -1209,8 +1258,8 @@ describe("backup of live photos", () => {
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(readFileSync(join(outDir, "originals", "500.mov"))).toEqual(
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
expect(readFileSync(saved(outDir, "500.mov"))).toEqual(
Buffer.from(VIDEO),
);
expect(tree(outDir)).toEqual(linked);
@@ -1219,23 +1268,6 @@ describe("backup of live photos", () => {
await lib.close();
});
it("replaces an earlier ZIP and its link with the image and video the cache holds", async () => {
const { file: live, body } = await asLivePhoto(
file(500, 10, "IMG_0500.HEIC"),
);
const lib = await open([live], new Map([[500, body]]));
await lib.photos.byID({ fileID: 500 })!.original();
const outDir = join(root, "backup");
earlierZIP(outDir);
const result = await lib.backup({ downloadDirectory: outDir });
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
expect(tree(outDir)).toEqual(linked);
await lib.close();
});
it.each(["missing", "empty"])(
"fetches a live photo again when the video its JSON file names is %s",
async (state) => {
@@ -1245,7 +1277,7 @@ describe("backup of live photos", () => {
const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup");
await lib.backup({ downloadDirectory: outDir });
const video = join(outDir, "originals", "500.mov");
const video = saved(outDir, "500.mov");
if (state === "missing") rmSync(video);
else writeFileSync(video, "");
@@ -1271,7 +1303,7 @@ describe("backup of live photos", () => {
const lib = await open([live], new Map([[500, body]]));
const outDir = join(root, "backup");
await lib.backup({ downloadDirectory: outDir });
const image = join(outDir, "originals", "500.heic");
const image = saved(outDir, "500.heic");
if (state === "missing") rmSync(image);
else writeFileSync(image, "");
@@ -1310,8 +1342,8 @@ describe("backup of live photos", () => {
const read = await reader.photos.byID({ fileID: 500 })!.original();
expect(read).toEqual({
path: join(outDir, "originals", "500.heic"),
videoPath: join(outDir, "originals", "500.mov"),
path: saved(outDir, "500.heic"),
videoPath: saved(outDir, "500.mov"),
bytes: IMAGE.length,
});
await reader.close();
+1 -1
View File
@@ -757,7 +757,7 @@ describe("backup", () => {
expect(code).toBe(1);
expect(runText).toBe("quak: HTTP 401 from server\n");
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
expect(existsSync(join(dir, "originals"))).toBe(false);
expect(existsSync(dir)).toBe(false);
});
});
+138 -8
View File
@@ -3,14 +3,18 @@
* `quak backup-metadata --exif` records.
*
* The originals come from users' libraries, so a truncated or corrupt file
* must neither hang the read nor throw out of it: `readPhotoExif` gives `{}`,
* must neither hang the read nor throw out of it: `readAllExifTags` gives `{}`,
* and `backup-metadata` tells an EXIF block it cannot read apart from a file
* that simply has no EXIF, carrying the reason in `exifError`. Each JPEG below
* is a short hand-built byte array; the HEIC is a real file.
*/
import { describe, expect, it } from "vitest";
import { readPhotoExif } from "../../src/exif.js";
import {
readAllExifTags,
readExifTags,
readPhotoExif,
} from "../../src/exif.js";
import { extractImageMetadata } from "../../src/metadata-backup.js";
import { HEIC_WITH_EXIF } from "../exif-heic.js";
@@ -63,6 +67,51 @@ const TIFF_ALTITUDE_WITHOUT_REF = [
...[0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x02], // 25/2, at 44
];
// A big-endian TIFF block holding a GPSLatitude of 33° 30' 0" and a
// GPSLatitudeRef of "S".
const TIFF_SOUTHERN_LATITUDE = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then no next IFD.
...[0x00, 0x01],
// The GPS IFD's offset (0x8825), LONG, 26.
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
...[0x00, 0x00, 0x00, 0x00],
// The GPS IFD, at 26: two entries, then no next IFD.
...[0x00, 0x02],
// GPSLatitudeRef (0x0001), 2 ASCII bytes: "S".
...[0x00, 0x01, 0x00, 0x02, 0x00, 0x00, 0x00, 0x02, 0x53, 0x00, 0x00, 0x00],
// GPSLatitude (0x0002), three RATIONALs at 56.
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
...[0x00, 0x00, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, 0x01], // 33/1, at 56
...[0x00, 0x00, 0x00, 0x1e, 0x00, 0x00, 0x00, 0x01], // 30/1
...[0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01], // 0/1
];
// A big-endian TIFF block holding a GPSLatitude of 40° 26' 46" and a
// GPSLongitude of 79° 58' 56", and neither GPSLatitudeRef nor GPSLongitudeRef.
const TIFF_POSITION_WITHOUT_REFS = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then no next IFD.
...[0x00, 0x01],
// The GPS IFD's offset (0x8825), LONG, 26.
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
...[0x00, 0x00, 0x00, 0x00],
// The GPS IFD, at 26: two entries, then no next IFD.
...[0x00, 0x02],
// GPSLatitude (0x0002), three RATIONALs at 56.
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
// GPSLongitude (0x0004), three RATIONALs at 80.
...[0x00, 0x04, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x50],
...[0x00, 0x00, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x28, 0x00, 0x00, 0x00, 0x01], // 40/1, at 56
...[0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x01], // 26/1
...[0x00, 0x00, 0x00, 0x2e, 0x00, 0x00, 0x00, 0x01], // 46/1
...[0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, 0x01], // 79/1, at 80
...[0x00, 0x00, 0x00, 0x3a, 0x00, 0x00, 0x00, 0x01], // 58/1
...[0x00, 0x00, 0x00, 0x38, 0x00, 0x00, 0x00, 0x01], // 56/1
];
// A big-endian TIFF block holding Orientation 6 and a Make whose value lies
// past the end of the file.
const TIFF_MAKE_PAST_END = [
@@ -87,6 +136,27 @@ const TIFF_UNNAMED_TAG = [
...[0x00, 0x00, 0x00, 0x00],
];
// A big-endian TIFF block holding Orientation 6, and a thumbnail IFD holding
// its own Orientation 1 and a 4-byte JPEG thumbnail.
const TIFF_WITH_THUMBNAIL = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then the thumbnail IFD at 26.
...[0x00, 0x01],
// Orientation (0x0112), SHORT, 6.
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x1a],
// The thumbnail IFD, at 26: three entries, then no next IFD.
...[0x00, 0x03],
// Orientation (0x0112), SHORT, 1.
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00],
// JPEGInterchangeFormat (0x0201), LONG: the thumbnail is at 68.
...[0x02, 0x01, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x44],
// JPEGInterchangeFormatLength (0x0202), LONG: 4 bytes.
...[0x02, 0x02, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04],
...[0x00, 0x00, 0x00, 0x00],
...[0xff, 0xd8, 0xff, 0xd9], // the thumbnail, at 68: an empty JPEG
];
// An APP1 segment whose length field matches its data.
const app1 = (data: number[]): number[] => {
const len = data.length + 2;
@@ -96,31 +166,90 @@ const app1 = (data: number[]): number[] => {
const bytes = (...parts: number[][]): Uint8Array =>
new Uint8Array(parts.flat());
describe("readAllExifTags", () => {
it("keys a tag exifreader has no name for by its number", () => {
const data = [...EXIF_HEADER, ...TIFF_UNNAMED_TAG];
expect(readAllExifTags(bytes(SOI, app1(data), SOS))).toStrictEqual({
"undefined-49152": {
id: 49152,
value: 7,
description: 7,
computed: 7,
},
});
});
it("puts the thumbnail's tags under Thumbnail, without its image", () => {
const input = bytes(
SOI,
app1([...EXIF_HEADER, ...TIFF_WITH_THUMBNAIL]),
SOS,
);
// exifreader finds the thumbnail's image.
expect(readExifTags(input)?.Thumbnail?.type).toBe("image/jpeg");
const tags = readAllExifTags(input);
expect(tags.Orientation?.value).toBe(6);
expect(Object.keys(tags.Thumbnail ?? {}).sort()).toEqual([
"JPEGInterchangeFormat",
"JPEGInterchangeFormatLength",
"Orientation",
]);
expect(tags.Thumbnail?.Orientation?.value).toBe(1);
expect(readPhotoExif(tags)).toStrictEqual({ orientation: 6 });
});
});
describe("readPhotoExif", () => {
it("reads the common fields of a valid JPEG", () => {
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
orientation: 6,
});
});
it("gives no dateTimeOriginal for a DateTimeOriginal of 0000:00:00 00:00:00", () => {
const data = [...EXIF_HEADER, ...TIFF_UNSET_DATE];
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
orientation: 6,
});
});
it("reads a GPSAltitude without GPSAltitudeRef as above sea level", () => {
const data = [...EXIF_HEADER, ...TIFF_ALTITUDE_WITHOUT_REF];
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
gpsAltitude: 12.5,
});
});
it("reads a GPSLatitude with GPSLatitudeRef S as south of the equator", () => {
const data = [...EXIF_HEADER, ...TIFF_SOUTHERN_LATITUDE];
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
gpsLatitude: -33.5,
});
});
it("gives no gpsLatitude or gpsLongitude without their reference tags", () => {
const data = [...EXIF_HEADER, ...TIFF_POSITION_WITHOUT_REFS];
const tags = readAllExifTags(bytes(SOI, app1(data), SOS));
// The position is read; only its hemisphere is unknown.
expect(tags.GPSLatitude?.computed).toStrictEqual([40, 26, 46]);
expect(tags.GPSLongitude?.computed).toStrictEqual([79, 58, 56]);
expect(readPhotoExif(tags)).toStrictEqual({});
});
it("gives no make for a Make whose value lies past the end of the file", () => {
const data = [...EXIF_HEADER, ...TIFF_MAKE_PAST_END];
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
orientation: 6,
});
});
@@ -175,8 +304,9 @@ describe("readPhotoExif", () => {
"an EXIF block that cannot be parsed",
bytes(SOI, app1([...EXIF_HEADER, 0x58, 0x58]), SOS),
],
])("returns no fields for %s", (_, input) => {
expect(readPhotoExif(input)).toStrictEqual({});
])("returns no tags and no fields for %s", (_, input) => {
expect(readAllExifTags(input)).toStrictEqual({});
expect(readPhotoExif(readAllExifTags(input))).toStrictEqual({});
});
});
-9
View File
@@ -1883,15 +1883,6 @@ describe("downloadFile live photos", () => {
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
});
it("replaces what was at the destination, such as an earlier ZIP of the two", async () => {
const t = setup(livePhotoZip(), livePhoto);
writeFileSync(t.outPath, livePhotoZip());
await t.run();
expect(readdirSync(t.dir).sort()).toEqual(["f.heic", "f.mov"]);
});
it("renames the image and then the video into place, each from its own temp file", async () => {
const t = setup(livePhotoZip(), livePhoto);
+303
View File
@@ -0,0 +1,303 @@
/**
* The example script `examples/download-albums.ts` (issue #144), run twice
* against a stand-in account, as a user would run it twice.
*
* The account has two albums sharing one photo, and one of its photos is a
* live photo whose image is a HEIC with EXIF. The first run puts every
* original at its save path, writes each photo's metadata beside it and each
* album's photos under `albums/`. Before the second run the account gains an
* album holding a new photo. The second run downloads that photo and writes
* its files and the new album's, and fetches nothing else and changes no
* other file.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import {
existsSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
statSync,
utimesSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { downloadAlbums } from "../../examples/download-albums.js";
import { Library, type ContentSource } from "../../src/index.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { readAllExifTags } from "../../src/exif.js";
import { HEIC_WITH_EXIF } from "../exif-heic.js";
import {
asLivePhoto,
cdnSource,
livePhotoHash,
livePhotoZip,
VIDEO,
} from "../live-photo.js";
const USER_ID = 7;
// Every photo is taken at noon local time on 2026-03-01, so the machine's time
// zone cannot move it to another day; it is saved in the folder `DAY`. Ente
// stores times in microseconds.
const TAKEN_MS = new Date(2026, 2, 1, 12).getTime();
const DAY = join("2026", "2026-03", "2026-03-01");
const collection = (id: number, name: string): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id]),
name,
type: "album",
updationTime: 1,
isShared: false,
});
const file = (id: number, collectionID: number): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: TAKEN_MS * 1000,
modificationTime: TAKEN_MS * 1000,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 1,
});
let root: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-download-albums-"));
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
// Every file and directory under `dir`, by path.
const entries = (dir: string): string[] =>
readdirSync(dir, { recursive: true, encoding: "utf-8" });
// Set the modification time of everything under `dir` to the epoch, so that
// anything written there afterwards has a later one, however soon it comes.
const backdate = (dir: string): void => {
for (const name of entries(dir)) utimesSync(join(dir, name), 0, 0);
};
// The modification time of everything under `dir`, by path.
const mtimes = (dir: string): Map<string, number> =>
new Map(
entries(dir).map((name) => [name, statSync(join(dir, name)).mtimeMs]),
);
const readJSON = (path: string): unknown =>
JSON.parse(readFileSync(path, "utf-8"));
describe("examples/download-albums.ts", () => {
it("downloads every album's photos with their metadata, and on a second run only what the account gained", async () => {
// Album 1, "Trip", holds photos 1 and 2. Album 2, "Family", holds
// photo 2 and photo 3, a live photo.
const live = await asLivePhoto(
file(3, 2),
livePhotoZip({ "image.heic": HEIC_WITH_EXIF, "video.mov": VIDEO }),
livePhotoHash(HEIC_WITH_EXIF, VIDEO),
);
const collections = [collection(1, "Trip"), collection(2, "Family")];
const filesByAlbum = new Map<number, EnteFile[]>([
[1, [file(1, 1), file(2, 1)]],
[2, [file(2, 2), live.file]],
]);
const client = {
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
collectionsSince: async (): Promise<CollectionsPage> => ({
collections: [...collections],
deleted: [],
cursor: 1,
}),
filesSince: async (args: {
collectionID: number;
}): Promise<FilesPage> => ({
files: filesByAlbum.get(args.collectionID) ?? [],
deleted: [],
cursor: 1,
}),
};
// Photo 3 comes from a stand-in server, encrypted as Ente serves a
// live photo. Any other original is a few bytes naming its photo.
// `calls` counts every fetch, thumbnails included.
const server = cdnSource(new Map([[3, live.body]]));
let calls = 0;
const source: ContentSource = {
original: async (args) => {
calls++;
if (args.file.id === 3) return server.original(args);
const bytes = `original-${args.file.id}`;
writeFileSync(args.destination, bytes);
return { bytesWritten: bytes.length };
},
thumbnail: async (args) => {
calls++;
return server.thumbnail(args);
},
};
const dir = join(root, "photos");
const open = (): Promise<Library> =>
Library.open({
client,
cacheDirectory: join(root, "cache"),
downloadDirectory: dir,
contentSource: source,
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const first = await open();
// The cache already holds photo 1's original, so its record names a
// cache path, which the metadata leaves out. download() copies it
// from the cache rather than fetching it again.
await first.photos.byID({ fileID: 1 })!.original();
expect(await downloadAlbums(first, dir)).toEqual({
downloaded: 3,
alreadyLocal: 0,
});
await first.close();
expect(calls).toBe(3);
// Each original at its save path, a live photo as its image, its video
// and the file naming them, and each photo's metadata beside it.
const day = join(dir, DAY);
expect(readdirSync(day).sort()).toEqual([
"2026-03-01.1.jpg",
"2026-03-01.1.jpg.json",
"2026-03-01.2.jpg",
"2026-03-01.2.jpg.json",
"2026-03-01.3.heic",
"2026-03-01.3.heic.json",
"2026-03-01.3.livephoto.json",
"2026-03-01.3.mov",
]);
expect(readFileSync(join(day, "2026-03-01.1.jpg"), "utf-8")).toBe(
"original-1",
);
expect(readFileSync(join(day, "2026-03-01.2.jpg"), "utf-8")).toBe(
"original-2",
);
expect(readFileSync(join(day, "2026-03-01.3.heic"))).toEqual(
Buffer.from(HEIC_WITH_EXIF),
);
expect(readFileSync(join(day, "2026-03-01.3.mov"))).toEqual(
Buffer.from(VIDEO),
);
// The metadata is the photo's record and its EXIF tags. The originals
// of photos 1 and 2 are not image data, so they have no EXIF tags.
// Photo 3's are every tag of its image, as `photo.exif()` returns
// them. `record` holds the fields the three records share.
const record = {
takenAt: TAKEN_MS,
modifiedAt: TAKEN_MS,
fileType: "image",
isArchived: false,
isHidden: false,
};
expect(readJSON(join(day, "2026-03-01.1.jpg.json"))).toEqual({
...record,
fileID: 1,
albumIDs: [1],
title: "file-1.jpg",
exif: {},
});
expect(readJSON(join(day, "2026-03-01.2.jpg.json"))).toEqual({
...record,
fileID: 2,
albumIDs: [1, 2],
title: "file-2.jpg",
exif: {},
});
expect(readJSON(join(day, "2026-03-01.3.heic.json"))).toEqual({
...record,
fileID: 3,
albumIDs: [2],
title: "file-3.jpg",
fileType: "livePhoto",
hash: livePhotoHash(HEIC_WITH_EXIF, VIDEO),
exif: readAllExifTags(HEIC_WITH_EXIF),
});
// Each album's photos, newest first, by save path relative to `dir`.
// Photo 2 is in both.
expect(readdirSync(join(dir, "albums")).sort()).toEqual([
"1.json",
"2.json",
]);
expect(readJSON(join(dir, "albums", "1.json"))).toEqual({
collectionID: 1,
name: "Trip",
savePaths: [
join(DAY, "2026-03-01.2.jpg"),
join(DAY, "2026-03-01.1.jpg"),
],
});
expect(readJSON(join(dir, "albums", "2.json"))).toEqual({
collectionID: 2,
name: "Family",
savePaths: [
join(DAY, "2026-03-01.3.heic"),
join(DAY, "2026-03-01.2.jpg"),
],
});
// Before the second run the account gains album 3, "Garden", holding
// a new photo 4. The second run, with a newly opened library, opens
// the cache written by the first and still downloads photo 4. It finds
// the other photos already local and fetches nothing else.
collections.push(collection(3, "Garden"));
filesByAlbum.set(3, [file(4, 3)]);
backdate(dir);
const before = mtimes(dir);
const second = await open();
expect(await downloadAlbums(second, dir)).toEqual({
downloaded: 1,
alreadyLocal: 3,
});
await second.close();
expect(calls).toBe(4);
expect(readFileSync(join(day, "2026-03-01.4.jpg"), "utf-8")).toBe(
"original-4",
);
expect(readJSON(join(dir, "albums", "3.json"))).toEqual({
collectionID: 3,
name: "Garden",
savePaths: [join(DAY, "2026-03-01.4.jpg")],
});
// The second run adds only photo 4's files and album 3's, and
// rewrites, renames or removes no file from the first run. The two
// directories that gain a file are the only other changes.
const after = mtimes(dir);
expect(
[...after.keys()].filter((name) => !before.has(name)).sort(),
).toEqual([
join(DAY, "2026-03-01.4.jpg"),
join(DAY, "2026-03-01.4.jpg.json"),
join("albums", "3.json"),
]);
for (const [name, mtime] of before) {
if (name === DAY || name === "albums") continue;
expect(after.get(name), name).toBe(mtime);
}
});
});
+1
View File
@@ -142,6 +142,7 @@ const buildCache = (args: {
pools: new RequestPools(),
source,
cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id),
statfs: args.statfs,
cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes,
+339 -40
View File
@@ -6,16 +6,18 @@
* `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and
* a cached path shows up on the projected record. A library opened without a
* content source leaves those methods throwing rather than silently doing
* nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `content()`,
* `exif()` and the methods that each return one field of `exif()`.
* nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `download()`,
* `content()`, `exif()` and the methods that each return one EXIF field.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import {
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
existsSync,
statSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
@@ -25,7 +27,7 @@ import { Library, type LibraryOptions } from "../../src/library/index.js";
import type { ContentSource } from "../../src/library/content.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import type { PhotoExif } from "../../src/exif.js";
import { readPhotoExif, type PhotoExif } from "../../src/exif.js";
import { HEIC_WITH_EXIF } from "../exif-heic.js";
import {
asLivePhoto,
@@ -38,6 +40,12 @@ import {
const USER_ID = 7;
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
// stores times, so the machine's time zone cannot move it to another day; it
// is saved in the folder `DAY`.
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
const DAY = join("2026", "2026-03", "2026-03-01");
const collection = (id: number): Collection => ({
id,
ownerID: USER_ID,
@@ -56,7 +64,7 @@ const file = (id: number, collectionID: number): EnteFile => ({
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: 1,
creationTime: TAKEN,
modificationTime: 1,
},
file: { decryptionHeader: "aGVhZGVy" },
@@ -168,7 +176,7 @@ describe("Library content wiring", () => {
await lib.close();
});
it("removes a live photo's ZIP an earlier version cached when it opens, and precaches its image and video", async () => {
it("does not take a live photo's image or video with no JSON file as its original when it opens, and precaches both", async () => {
const { file: live, body } = await asLivePhoto(file(1, 1));
class LiveClient extends MockClient {
override async filesSince(): Promise<FilesPage> {
@@ -189,7 +197,8 @@ describe("Library content wiring", () => {
// A first run records the library, so the next one knows that file 1
// is a live photo when it opens the cache.
await (await open({})).close();
writeFileSync(join(originals, "1.jpg"), livePhotoZip());
writeFileSync(join(originals, "1.heic"), "an image");
writeFileSync(join(originals, "1.mov"), "a video");
let precached!: () => void;
const done = new Promise<void>((r) => (precached = r));
@@ -200,7 +209,9 @@ describe("Library content wiring", () => {
precached();
},
});
expect(existsSync(join(originals, "1.jpg"))).toBe(false);
expect(
lib.photos.byID({ fileID: 1 })!.record().originalPath,
).toBeUndefined();
await done;
expect(readdirSync(originals).sort()).toEqual([
@@ -208,6 +219,17 @@ describe("Library content wiring", () => {
"1.livephoto.json",
"1.mov",
]);
expect(readFileSync(join(originals, "1.heic"))).toEqual(
Buffer.from(IMAGE),
);
expect(readFileSync(join(originals, "1.mov"))).toEqual(
Buffer.from(VIDEO),
);
expect(
JSON.parse(
readFileSync(join(originals, "1.livephoto.json"), "utf-8"),
),
).toEqual({ image: "1.heic", video: "1.mov" });
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
join(originals, "1.heic"),
);
@@ -224,6 +246,9 @@ describe("Library content wiring", () => {
await expect(
lib.photos.byID({ fileID: 1 })!.thumbnail(),
).rejects.toThrow(/content cache/i);
await expect(
lib.photos.byID({ fileID: 1 })!.download(),
).rejects.toThrow(/content cache/i);
await expect(
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
).rejects.toThrow(/content cache/i);
@@ -248,10 +273,10 @@ const entry = (
value: number[],
): number[] => [...u16(tag), ...u16(type), ...u32(count), ...value];
// The TIFF block of a JPEG's EXIF segment, holding every field `exif()` picks:
// the camera in the first IFD, the exposure in the Exif IFD, and a GPS position
// of 40°26'46" N, 79°58'56" W, 12.5 m below sea level. Offsets count from the
// start of this block.
// The TIFF block of a JPEG's EXIF segment, holding every field `Photo`'s typed
// methods return: the camera in the first IFD, the exposure in the Exif IFD,
// and a GPS position of 40°26'46" N, 79°58'56" W, 12.5 m below sea level.
// Offsets count from the start of this block.
const TIFF = [
...[0x4d, 0x4d, 0x00, 0x2a], // big-endian TIFF
...u32(8), // the first IFD's offset
@@ -341,7 +366,7 @@ describe("Photo save path, local copy, content and EXIF", () => {
it("names where a backup writes the original, which is local once the backup has written it", async () => {
const lib = await open();
const photo = lib.photos.byID({ fileID: 1 })!;
const savePath = join(root, "backup", "originals", "1.jpg");
const savePath = join(root, "backup", DAY, "2026-03-01.1.jpg");
expect(photo.savePath).toBe(savePath);
expect(photo.isLocal).toBe(false);
@@ -359,23 +384,150 @@ describe("Photo save path, local copy, content and EXIF", () => {
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
true,
);
expect(existsSync(photo.savePath!)).toBe(false);
expect(existsSync(photo.savePath)).toBe(false);
expect(photo.isLocal).toBe(false);
await lib.close();
});
it("has no save path and is not local without a download directory", async () => {
it("saves under photos/ in the working directory at open without a download directory", async () => {
const cwd = vi.spyOn(process, "cwd").mockReturnValue(root);
const lib = await open({ downloadDirectory: undefined });
cwd.mockRestore();
const photo = lib.photos.byID({ fileID: 1 })!;
expect(photo.savePath).toBeUndefined();
expect(lib.downloadDirectory).toBe(join(root, "photos"));
expect(photo.savePath).toBe(
join(root, "photos", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(false);
await lib.close();
});
it("has no save path and is not local without a content source", async () => {
it("refuses an empty download directory", async () => {
await expect(open({ downloadDirectory: "" })).rejects.toThrow(
/downloadDirectory is empty/,
);
});
it("keeps a photo's save path after a refresh removes its file", async () => {
// The same account, whose album is deleted on the second refresh.
class AlbumDeletedClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
if (!this.served) return super.collectionsSince();
return { collections: [], deleted: [1], cursor: 2 };
}
}
const lib = await open({ client: new AlbumDeletedClient() });
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.download();
await lib.fresh();
expect(lib.photos.byID({ fileID: 1 })).toBeUndefined();
expect(photo.savePath).toBe(
join(root, "backup", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("dates the save path and download() by the membership its takenAt comes from", async () => {
// One file in two albums. The date edited in Ente has reached album 2's
// copy, synced later, but album 1 still has an earlier edit.
const older = file(1, 1);
older.pubMagicMetadata = {
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
};
const newer = file(1, 2);
newer.updationTime = 2;
newer.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
const client = {
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
collectionsSince: async (): Promise<CollectionsPage> => ({
collections: [collection(1), collection(2)],
deleted: [],
cursor: 1,
}),
filesSince: async (args: {
collectionID: number;
}): Promise<FilesPage> => ({
files: [args.collectionID === 1 ? older : newer],
deleted: [],
cursor: 1,
}),
};
const lib = await open({ client });
const photo = lib.photos.byID({ fileID: 1 })!;
expect(photo.takenAt).toBe(new Date(2026, 3, 15, 12).getTime());
expect(photo.savePath).toBe(
join(
root,
"backup",
"2026",
"2026-04",
"2026-04-15",
"2026-04-15.1.jpg",
),
);
// download() writes to that same path, so the photo is then local.
const saved = await photo.download();
expect(saved.path).toBe(photo.savePath);
expect(photo.isLocal).toBe(true);
expect(existsSync(join(root, "backup", "2026", "2026-02"))).toBe(false);
await lib.close();
});
it("downloads a photo held across a refresh that edits its date to the save path it names", async () => {
// The same account, whose second refresh brings a date edited in Ente.
const edited = file(1, 1);
edited.updationTime = 2;
edited.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
class DateEditedClient extends MockClient {
refreshes = 0;
override async collectionsSince(): Promise<CollectionsPage> {
this.refreshes++;
return {
collections: [
{ ...collection(1), updationTime: this.refreshes },
],
deleted: [],
cursor: this.refreshes,
};
}
override async filesSince(): Promise<FilesPage> {
return {
files: [this.refreshes === 1 ? file(1, 1) : edited],
deleted: [],
cursor: this.refreshes,
};
}
}
const lib = await open({ client: new DateEditedClient() });
const photo = lib.photos.byID({ fileID: 1 })!;
await lib.fresh();
expect(lib.photos.byID({ fileID: 1 })!.takenAt).toBe(
new Date(2026, 3, 15, 12).getTime(),
);
const saved = await photo.download();
expect(saved.path).toBe(photo.savePath);
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("has a save path, and is not local, without a content source", async () => {
const lib = await open({ contentSource: undefined });
const photo = lib.photos.byID({ fileID: 1 })!;
expect(photo.savePath).toBeUndefined();
expect(photo.savePath).toBe(
join(root, "backup", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(false);
await lib.close();
});
@@ -387,21 +539,170 @@ describe("Photo save path, local copy, content and EXIF", () => {
contentSource: cdnSource(new Map([[1, body]])),
});
const photo = lib.photos.byID({ fileID: 1 })!;
const originals = join(root, "backup", "originals");
const day = join(root, "backup", DAY);
// Until then the name comes from the title, file-1.jpg; the backup
// stores the image with the extension found inside the live photo.
expect(photo.savePath).toBe(join(originals, "1.jpg"));
expect(photo.savePath).toBe(join(day, "2026-03-01.1.jpg"));
await lib.backup();
expect(photo.savePath).toBe(join(originals, "1.heic"));
expect(photo.savePath).toBe(join(day, "2026-03-01.1.heic"));
expect(photo.isLocal).toBe(true);
expect(await photo.content()).toEqual(Buffer.from(IMAGE));
await lib.close();
});
it("reads the common EXIF fields of a JPEG original", async () => {
it("downloads an original the cache holds by copying it, without fetching it again", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.original();
expect(source.originalCalls()).toBe(1);
expect(photo.isLocal).toBe(false);
const saved = await photo.download();
expect(source.originalCalls()).toBe(1);
expect(saved).toEqual({
path: photo.savePath,
bytes: "orig-bytes".length,
});
expect(readFileSync(photo.savePath, "utf-8")).toBe("orig-bytes");
expect(photo.isLocal).toBe(true);
// The cache keeps its own copy.
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
true,
);
await lib.close();
});
it("downloads an original the cache does not hold straight to its save path", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
const saved = await photo.download();
expect(source.originalCalls()).toBe(1);
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
expect(readFileSync(saved.path, "utf-8")).toBe("orig-bytes");
expect(photo.isLocal).toBe(true);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
await lib.close();
});
it("does nothing when download() finds the original already at its save path", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
const first = await photo.download();
const written = statSync(first.path).ino;
const second = await photo.download();
expect(second).toEqual(first);
expect(source.originalCalls()).toBe(1);
expect(statSync(second.path).ino).toBe(written);
await lib.close();
});
it("downloads a live photo the cache holds as its image, its video and the JSON file naming them", async () => {
const { file: live, body } = await asLivePhoto(file(1, 1));
const lib = await open({
client: new FilesClient([live]),
contentSource: cdnSource(new Map([[1, body]])),
});
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.original();
const day = join(root, "backup", DAY);
const saved = await photo.download();
expect(saved).toEqual({
path: join(day, "2026-03-01.1.heic"),
videoPath: join(day, "2026-03-01.1.mov"),
bytes: IMAGE.length,
});
expect(readdirSync(day).sort()).toEqual([
"2026-03-01.1.heic",
"2026-03-01.1.livephoto.json",
"2026-03-01.1.mov",
]);
expect(
JSON.parse(
readFileSync(join(day, "2026-03-01.1.livephoto.json"), "utf-8"),
),
).toEqual({ image: "2026-03-01.1.heic", video: "2026-03-01.1.mov" });
expect(photo.savePath).toBe(saved.path);
expect(photo.isLocal).toBe(true);
await lib.close();
});
// Every tag in JPEG_WITH_EXIF, by name, in the order of its IFDs.
const jpegTags = [
"Make",
"Model",
"Orientation",
"Exif IFD Pointer",
"GPS Info IFD Pointer",
"ExposureTime",
"FNumber",
"ISOSpeedRatings",
"DateTimeOriginal",
"OffsetTimeOriginal",
"FocalLength",
"LensModel",
"GPSLatitudeRef",
"GPSLatitude",
"GPSLongitudeRef",
"GPSLongitude",
"GPSAltitudeRef",
"GPSAltitude",
];
// HEIC_WITH_EXIF holds those and the tags exiftool adds to every file.
const heicTags = [
...jpegTags,
"YCbCrPositioning",
"ExifVersion",
"ComponentsConfiguration",
"ColorSpace",
"GPSVersionID",
];
it.each([
[
"JPEG",
JPEG_WITH_EXIF,
jpegTags,
{
"Exif IFD Pointer": { value: 88 },
GPSLatitudeRef: { value: ["N"], description: "North latitude" },
},
],
[
"HEIC",
HEIC_WITH_EXIF,
heicTags,
{
ColorSpace: { value: 0xffff, description: "Uncalibrated" },
ExifVersion: { description: "0232" },
},
],
])(
"returns every EXIF tag of a %s original, by name",
async (_, bytes, names, others) => {
const lib = await open({ contentSource: stubSource(bytes) });
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(Object.keys(exif).sort()).toEqual([...names].sort());
// Tags outside the thirteen fields, as exifreader decodes them.
expect(exif).toMatchObject(others);
await lib.close();
},
);
it("picks the common EXIF fields from a JPEG original's tags", async () => {
const lib = await open({ contentSource: stubSource(JPEG_WITH_EXIF) });
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(readPhotoExif(exif)).toStrictEqual({
make: "Canon",
model: "EOS R5",
lensModel: "RF50mm F1.8 STM",
@@ -420,8 +721,8 @@ describe("Photo save path, local copy, content and EXIF", () => {
await lib.close();
});
// What exif() returns for HEIC_WITH_EXIF, and for JPEG_WITH_EXIF, which
// holds the same values.
// The fields picked from HEIC_WITH_EXIF's tags, and from JPEG_WITH_EXIF's,
// which hold the same values.
const heicFields: PhotoExif = {
make: "Canon",
model: "EOS R5",
@@ -438,11 +739,10 @@ describe("Photo save path, local copy, content and EXIF", () => {
gpsAltitude: -12.5,
};
it("reads the same common EXIF fields from a HEIC original", async () => {
it("picks the same common EXIF fields from a HEIC original's tags", async () => {
const lib = await open({ contentSource: stubSource(HEIC_WITH_EXIF) });
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual(
heicFields,
);
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
await lib.close();
});
@@ -456,28 +756,27 @@ describe("Photo save path, local copy, content and EXIF", () => {
client: new FilesClient([live]),
contentSource: cdnSource(new Map([[1, body]])),
});
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual(
heicFields,
);
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
await lib.close();
});
// The build's type check, not this test, makes sure `Photo` has a method
// for every `PhotoExif` field, whatever the fixtures hold: `Photo`
// implements a type with one method per field. This test checks that each
// method gives the same value as exif().
// method gives the field picked from the tags exif() returns.
it.each([
["JPEG", JPEG_WITH_EXIF],
["HEIC", HEIC_WITH_EXIF],
])(
"has a method for each field exif() returns, giving the same value, for a %s",
"has a method for each field, agreeing with the tags exif() returns, for a %s",
async (_, bytes) => {
const lib = await open({ contentSource: stubSource(bytes) });
const photo = lib.photos.byID({ fileID: 1 })!;
const exif = await photo.exif();
const fields = readPhotoExif(await photo.exif());
// The file holds every field, so every method is checked.
expect(exif).toStrictEqual(heicFields);
for (const [field, value] of Object.entries(exif)) {
expect(fields).toStrictEqual(heicFields);
for (const [field, value] of Object.entries(fields)) {
expect(await photo[field as keyof PhotoExif]()).toStrictEqual(
value,
);
@@ -486,7 +785,7 @@ describe("Photo save path, local copy, content and EXIF", () => {
},
);
it("returns no EXIF fields for an original that is not an image", async () => {
it("returns no EXIF tags for an original that is not an image", async () => {
const lib = await open();
const photo = lib.photos.byID({ fileID: 1 })!;
expect(await photo.exif()).toStrictEqual({});
@@ -494,7 +793,7 @@ describe("Photo save path, local copy, content and EXIF", () => {
await lib.close();
});
it("returns no EXIF fields for a JPEG whose EXIF cannot be parsed", async () => {
it("returns no EXIF tags for a JPEG whose EXIF cannot be parsed", async () => {
const lib = await open({
contentSource: stubSource(JPEG_WITH_BAD_EXIF),
});
@@ -515,7 +814,7 @@ describe("Photo save path, local copy, content and EXIF", () => {
await lib.close();
});
it("returns no EXIF fields for a video, without fetching it", async () => {
it("returns no EXIF tags for a video, without fetching it", async () => {
const video = file(1, 1);
video.metadata.fileType = "video";
const source = stubSource(JPEG_WITH_EXIF);
+80 -43
View File
@@ -7,8 +7,8 @@
* 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail`
* fetches through the request pool and stores the bytes; the next finds the
* file present and returns its path with a single `skipped` event and no
* network. A file already sitting in the backup `downloadDirectory` counts
* as present too.
* network. A file already stored at its save path under the
* `downloadDirectory` counts as present too.
* 2. **Present-means-complete.** Content appears only by the streaming atomic
* writer's rename, so a file that exists is whole. The directory listing
* taken at `open()` is the record of what is cached, and the orphan temp
@@ -41,6 +41,7 @@ import { join } from "node:path";
import {
ContentCache,
savePath,
type ContentSource,
type EnsureEvent,
} from "../../src/library/content.js";
@@ -152,12 +153,48 @@ const buildCache = (
pools: args.pools ?? new RequestPools(),
source,
cacheDirectory: cacheDir,
downloadDirectory: args.downloadDirectory,
downloadDirectory: args.downloadDirectory ?? join(root, "photos"),
getFile: (id) => byID.get(id),
});
return { cache, source };
};
// Microseconds, as Ente stores times, for noon local time on a day, so the
// machine's time zone cannot move the photo to another day.
const noon = (year: number, month: number, day: number): number =>
new Date(year, month - 1, day, 12).getTime() * 1000;
describe("savePath", () => {
it("files an original by year, month and day under the root", () => {
const f = file(12345, "IMG_0001.HEIC");
f.metadata.creationTime = noon(2026, 3, 1);
expect(savePath("/photos", f)).toBe(
"/photos/2026/2026-03/2026-03-01/2026-03-01.12345.HEIC",
);
});
it("dates an original by the date the user set, when there is one", () => {
const f = file(7, "a.jpg");
f.metadata.creationTime = noon(2026, 3, 1);
f.pubMagicMetadata = { editedTime: noon(1999, 12, 31) };
expect(savePath("/photos", f)).toBe(
"/photos/1999/1999-12/1999-12-31/1999-12-31.7.jpg",
);
});
it("takes the extension from the title it was uploaded with, not a new name", () => {
const f = file(8, "upload");
f.metadata.creationTime = noon(2026, 3, 1);
f.pubMagicMetadata = { editedName: "renamed.png" };
expect(savePath("/photos", f)).toBe(
"/photos/2026/2026-03/2026-03-01/2026-03-01.8.bin",
);
});
});
describe("ContentCache.open", () => {
it("creates the cache directories with 0700 permissions", async () => {
const { cache } = buildCache();
@@ -274,11 +311,17 @@ describe("ContentCache.original / thumbnail", () => {
it("serves a file already present in the download directory without fetching", async () => {
const downloadDirectory = join(root, "backup");
mkdirSync(join(downloadDirectory, "originals"), { recursive: true });
const backupPath = join(downloadDirectory, "originals", "1.jpg");
const day = join(downloadDirectory, "2026", "2026-03", "2026-03-01");
mkdirSync(day, { recursive: true });
const backupPath = join(day, "2026-03-01.1.jpg");
writeFileSync(backupPath, "from-backup");
const f = file(1);
f.metadata.creationTime = noon(2026, 3, 1);
const { cache, source } = buildCache({ downloadDirectory });
const { cache, source } = buildCache({
downloadDirectory,
files: [f],
});
await cache.open();
const events: EnsureEvent["status"][] = [];
@@ -523,43 +566,6 @@ describe("ContentCache live photos", () => {
expect(events).toEqual(["skipped"]);
});
it("replaces a live photo an earlier version stored as a ZIP under the image's name", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
mkdirSync(originals(), { recursive: true });
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
const cache = cacheOf([live], new Map([[5, body]]));
// Opened without being told that file 5 is a live photo, the cache
// records the ZIP, and does not serve it.
await cache.open();
const result = await cache.original(5);
expect(result.videoPath).toBe(join(originals(), "5.mov"));
expect(readdirSync(originals()).sort()).toEqual([
"5.heic",
"5.livephoto.json",
"5.mov",
]);
});
it("removes a live photo's ZIP an earlier version stored when it opens, so the precache fetches the image and video", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
mkdirSync(originals(), { recursive: true });
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open((fileID) => fileID === 5);
expect(readdirSync(originals())).toEqual([]);
expect(cache.pathsFor(5)).toEqual({});
const [fetched] = await cache.ensureOriginals({ fileIDs: [5] });
expect(fetched).toEqual({
fileID: 5,
path: join(originals(), "5.heic"),
});
expect(cache.pathsFor(5)).toEqual({ originalPath: fetched!.path });
});
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
const server = cdnSource(new Map([[5, body]]));
@@ -590,6 +596,36 @@ describe("ContentCache live photos", () => {
expect(second!.pathsFor(5)).toEqual({});
});
it("fetches a live photo's image and video again when the cache opened before knowing it is a live photo and no JSON file names them", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
mkdirSync(originals(), { recursive: true });
writeFileSync(join(originals(), "5.heic"), "an image");
writeFileSync(join(originals(), "5.mov"), "a video");
const cache = cacheOf([live], new Map([[5, body]]));
// Opened without being told that file 5 is a live photo, the cache
// records one of the two files as its original, with no video.
await cache.open();
const events: string[] = [];
const result = await cache.original(5, {
onProgress: (e) => events.push(e.status),
});
expect(events.at(-1)).toBe("done");
expect(result).toEqual({
path: join(originals(), "5.heic"),
videoPath: join(originals(), "5.mov"),
bytes: IMAGE.length,
});
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
expect(
JSON.parse(
readFileSync(join(originals(), "5.livephoto.json"), "utf-8"),
),
).toEqual({ image: "5.heic", video: "5.mov" });
});
it.each(["missing", "empty"])(
"fetches a live photo again when the video its JSON file names is %s",
async (state) => {
@@ -649,6 +685,7 @@ describe("ContentCache live photos", () => {
]),
),
cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => [a.file, b.file].find((f) => f.id === id),
// Room for one live photo, on a disk with plenty free.
cacheOriginalsMaxBytes: size,
+2
View File
@@ -280,6 +280,7 @@ describe("Precache eviction integration", () => {
pools: new RequestPools(),
source,
cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id),
statfs,
cacheOriginalsMaxBytes: 25, // holds two 10-byte originals
@@ -348,6 +349,7 @@ describe("Precache preemption", () => {
pools: new RequestPools({ contentConcurrency: 1 }),
source,
cacheDirectory: join(root, "cache"),
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id),
statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }),
freeBelowBytes: 0,
+11 -3
View File
@@ -36,6 +36,7 @@ import {
makeAlbumsAPI,
makePhotosAPI,
makeTimelineAPI,
type SavePathLookup,
type TimelineGroup,
} from "../../src/library/read.js";
import { Library } from "../../src/library/index.js";
@@ -101,12 +102,19 @@ const file = (
};
// Build the three API objects over one fixed projection, the way `Library`
// wires them over its live store.
// wires them over its live store. Save paths are covered in
// content-library.test.ts; these tests never ask for one.
const apis = (records: DerivedRecords) => {
const derive = () => records;
const saves: SavePathLookup = {
savePath: () => {
throw new Error("no save paths in these tests");
},
isLocal: () => false,
};
return {
albums: makeAlbumsAPI(derive),
photos: makePhotosAPI(derive),
albums: makeAlbumsAPI(derive, saves),
photos: makePhotosAPI(derive, saves),
timeline: makeTimelineAPI(derive),
};
};
+14 -1
View File
@@ -9,8 +9,10 @@
// Excluding too much: Prettier 3 reads `.gitignore` as a default ignore file,
// so dropping it from the context silently changes which files the lint
// phase's prettier check looks at compared to `make fmt-check` on the host.
// And without `.git`, a `docker build .` given no `VERSION` build arg cannot
// derive the version (`script/version`) and stamps `package.json`'s instead.
//
// Neither shows up as a build failure, so they are asserted here.
// None of these shows up as a build failure, so they are asserted here.
import { describe, expect, it } from "vitest";
import { existsSync, readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
@@ -47,6 +49,17 @@ describe(".dockerignore", () => {
expect(dockerignore).not.toContain(".gitignore");
});
it("leaves .git in the build context for the version", () => {
expect(dockerignore).not.toContain(".git");
expect(dockerignore).not.toContain(".git/");
});
// The build stage is the final image, so a .git/config sent in would
// ship the clone's remote URL and any credential in it.
it("sends .git without its config", () => {
expect(dockerignore).toContain(".git/config");
});
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
// file would silently give the build a different, unreviewed context —
// and eslint's flat config does not ignore dot-directories, so a stray
+137
View File
@@ -0,0 +1,137 @@
// `script/version` prints the version `script/build` stamps into
// `dist/package.json`, which is what `quak --version` reports from a build.
// A `docker build .` of a clone is given no `VERSION` build arg, so the
// version has to come from the `.git` in its context: the tag on a tagged
// commit; the tag, the commits since it and the short commit on a later commit;
// the short commit when no tag is reachable. A checkout with `.git` that still
// yields no usable version must fail the build, not ship a version nobody can
// trace back to its commit.
//
// Each test copies the script into a fresh directory, which the script then
// treats as the checkout, and executes it there.
import { afterEach, describe, expect, it } from "vitest";
import { execFileSync, spawnSync } from "node:child_process";
import {
chmodSync,
copyFileSync,
mkdirSync,
mkdtempSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
let checkout = "";
afterEach(() => {
rmSync(checkout, { recursive: true, force: true });
});
// A checkout holding the script and a package.json that declares 0.0.0, with
// no .git yet.
const makeCheckout = (): void => {
checkout = mkdtempSync(join(tmpdir(), "quak-version-"));
mkdirSync(join(checkout, "script"));
copyFileSync(
join(repoRoot, "script/version"),
join(checkout, "script/version"),
);
chmodSync(join(checkout, "script/version"), 0o755);
writeFileSync(join(checkout, "package.json"), '{ "version": "0.0.0" }\n');
};
// git in the checkout, with an identity and no commit signing, whatever the
// host's own git config says.
const git = (...args: string[]): string =>
execFileSync(
"git",
[
"-c",
"user.name=quak",
"-c",
"user.email=quak@example.invalid",
"-c",
"commit.gpgsign=false",
...args,
],
{ cwd: checkout, encoding: "utf-8", stdio: ["ignore", "pipe", "pipe"] },
).trim();
const makeCommittedCheckout = (): void => {
makeCheckout();
git("init", "-q");
git("add", "package.json");
git("commit", "-q", "-m", "first");
};
// Runs the script with nothing in its environment but PATH and, when given,
// VERSION.
const runVersion = (version?: string) =>
spawnSync(join(checkout, "script/version"), {
cwd: checkout,
encoding: "utf-8",
env: { PATH: process.env.PATH, VERSION: version },
});
describe("script/version", () => {
it("prints the short commit of an untagged commit", () => {
makeCommittedCheckout();
expect(runVersion().stdout.trim()).toBe(
git("rev-parse", "--short", "HEAD"),
);
});
it("prints the tag of a tagged commit", () => {
makeCommittedCheckout();
git("tag", "v1.2.3");
expect(runVersion().stdout.trim()).toBe("v1.2.3");
});
// script/docker and script/cibuild pass the version they resolve on the
// host as the VERSION build arg.
it("prints the VERSION it is given over what git would derive", () => {
makeCommittedCheckout();
expect(runVersion("x").stdout.trim()).toBe("x");
});
// `--build-arg VERSION=` must not stamp an empty version.
it("treats an empty VERSION as unset", () => {
makeCommittedCheckout();
expect(runVersion("").stdout.trim()).toBe(
git("rev-parse", "--short", "HEAD"),
);
});
// A source tarball has no .git: it keeps the version package.json
// declares, and must still build.
it("prints package.json's version where there is no .git", () => {
makeCheckout();
const result = runVersion();
expect(result.status).toBe(0);
expect(result.stdout.trim()).toBe("0.0.0");
});
// A repository with no commits stands in for any .git that git cannot
// describe: git missing from the image, or refusing to read the checkout.
it("fails where .git yields no version", () => {
makeCheckout();
git("init", "-q");
const result = runVersion();
expect(result.status).not.toBe(0);
expect(result.stdout).toBe("");
});
it.each(["dev", "unknown"])(
"fails where there is .git and the version is %s",
(version) => {
makeCommittedCheckout();
const result = runVersion(version);
expect(result.status).not.toBe(0);
expect(result.stdout).toBe("");
},
);
});
+5 -4
View File
@@ -787,12 +787,13 @@ describe("fixMissingThumbnails", () => {
});
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
// A noisy 400x300 JPEG, which the default encoding (quality 50, not
// A noisy 64x48 JPEG, which the default encoding (quality 50, not
// resized because it is under 720 px) cannot compress below the size
// recorded here: one byte less than that encoding's ciphertext.
// recorded here: one byte less than that encoding's ciphertext. It is
// small so that each encode is quick even on a busy host.
const fixMock = await buildThumbMock();
const w = 400;
const h = 300;
const w = 64;
const h = 48;
const noisy = new Uint8Array(
jpegJs.encode(
{
+1 -1
View File
@@ -18,5 +18,5 @@
"sourceMap": true,
"resolveJsonModule": true
},
"include": ["src/**/*", "bin/**/*"]
"include": ["src/**/*", "bin/**/*", "examples/**/*"]
}