Compare commits
7
Commits
4986ebd889
...
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9e94542a57 | ||
|
|
e50d2a78c8 | ||
|
|
0c995a8c4f | ||
|
|
d788c5457d | ||
|
|
10afa7a7f4 | ||
|
|
d40f339c0b | ||
|
|
10e1a9ef39 |
+6
-3
@@ -1,9 +1,12 @@
|
|||||||
# Mirrors .gitignore, with one deliberate exception: .gitignore itself stays
|
# Mirrors .gitignore, with one deliberate exception: .gitignore itself stays
|
||||||
# in the build context, because prettier 3 reads it as a default ignore file
|
# in the build context, because prettier 3 reads it as a default ignore file
|
||||||
# and dropping it would change what the lint phase's prettier check sees.
|
# and dropping it would change what the lint phase's prettier check sees.
|
||||||
|
#
|
||||||
# VCS
|
# .git is deliberately NOT excluded: the build derives the version it stamps
|
||||||
.git
|
# from it (script/version). It is sent without its config, which holds the
|
||||||
|
# clone's remote URL and any credential in it, and which the build stage, the
|
||||||
|
# final image, would otherwise carry. git describe does not need it.
|
||||||
|
.git/config
|
||||||
|
|
||||||
# OS
|
# OS
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|||||||
+6
-3
@@ -61,9 +61,12 @@ RUN script/bootstrap
|
|||||||
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# The version is computed on the host and passed in, because
|
# Version stamped into the build: the VERSION build arg when one is given,
|
||||||
# .dockerignore excludes .git.
|
# otherwise what script/version derives from the .git the build context
|
||||||
ARG VERSION=dev
|
# carries (script/bootstrap installed git), so any `docker build .` of a
|
||||||
|
# clone stamps its commit. The label can only carry the build arg, and is
|
||||||
|
# empty without one.
|
||||||
|
ARG VERSION
|
||||||
LABEL org.opencontainers.image.version="${VERSION}"
|
LABEL org.opencontainers.image.version="${VERSION}"
|
||||||
|
|
||||||
RUN make build
|
RUN make build
|
||||||
|
|||||||
@@ -26,8 +26,10 @@ check:
|
|||||||
build:
|
build:
|
||||||
@script/build
|
@script/build
|
||||||
|
|
||||||
build-bin:
|
# Bundles the built dist/, so the binary reports the version script/build
|
||||||
nix-shell -p bun --run "bun build bin/quak.ts --compile --outfile bin/quak"
|
# stamped.
|
||||||
|
build-bin: build
|
||||||
|
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
|
||||||
|
|
||||||
install: build-bin
|
install: build-bin
|
||||||
mkdir -p ~/bin
|
mkdir -p ~/bin
|
||||||
|
|||||||
@@ -80,6 +80,37 @@ await lib.close();
|
|||||||
The lower-level `Client` (login, session serialization, and the raw
|
The lower-level `Client` (login, session serialization, and the raw
|
||||||
enumeration/download calls) is exported too and documented under Design below.
|
enumeration/download calls) is exported too and documented under Design below.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
`examples/download-albums.ts` downloads every album's photos and their metadata
|
||||||
|
into a directory, `photos` in the working directory unless you name another. The
|
||||||
|
build compiles it; run it after `yarn install`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yarn build
|
||||||
|
QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir]
|
||||||
|
```
|
||||||
|
|
||||||
|
It opens the library with `precacheThumbnails` and `precacheOriginals` off, as
|
||||||
|
`quak backup` does, so the only file content it fetches is the originals it
|
||||||
|
saves. It asks on the terminal for a two-factor or email code when the account
|
||||||
|
requires one, and writes:
|
||||||
|
|
||||||
|
- each photo's original at its save path under `dir`, as `photo.download()`
|
||||||
|
writes it: `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, and for a live
|
||||||
|
photo its image, its video and the `.livephoto.json` file naming them
|
||||||
|
- beside each original, a JSON file named after it with `.json` added, for
|
||||||
|
example `2026-03-01.12345.jpg.json`: the photo's record (`photo.record()`)
|
||||||
|
without its cache paths, and every EXIF tag of the photo (`photo.exif()`)
|
||||||
|
under `exif`
|
||||||
|
- `albums/<collectionID>.json` for each album: its `collectionID`, its `name`,
|
||||||
|
and under `savePaths` the save paths of its photos relative to `dir`, newest
|
||||||
|
first
|
||||||
|
|
||||||
|
A photo in several albums is downloaded once. A second run downloads nothing and
|
||||||
|
rewrites only the JSON files whose content changed. A failed download stops the
|
||||||
|
run; running it again carries on, since every photo already saved is skipped.
|
||||||
|
|
||||||
## Entrypoints
|
## Entrypoints
|
||||||
|
|
||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
@@ -95,9 +126,12 @@ alpine. We provide:
|
|||||||
`script/bootstrap`, then `script/install-precommit`
|
`script/bootstrap`, then `script/install-precommit`
|
||||||
- `script/projectname` — output the project name (our own extension); used by
|
- `script/projectname` — output the project name (our own extension); used by
|
||||||
`script/docker` for the image tag
|
`script/docker` for the image tag
|
||||||
- `script/build` — compile the TypeScript sources into `dist/`, then verify that
|
- `script/build` — compile the TypeScript sources into `dist/`, stamp the
|
||||||
the entrypoints `package.json` declares (`main`, `types`, `bin`) are among the
|
version into `dist/package.json`, then verify that the entrypoints
|
||||||
files the compiler wrote, and make the CLI executable (our own extension)
|
`package.json` declares (`main`, `types`, `bin`) are among the files the
|
||||||
|
compiler wrote, and make the CLI executable (our own extension)
|
||||||
|
- `script/version` — print the version `script/build` stamps (our own
|
||||||
|
extension); see Version below
|
||||||
- `script/test` — run the test suite, by building the `test` phase of the
|
- `script/test` — run the test suite, by building the `test` phase of the
|
||||||
`Dockerfile` (vitest, 90s timeout, verbose rerun on failure); requires docker
|
`Dockerfile` (vitest, 90s timeout, verbose rerun on failure); requires docker
|
||||||
- `script/lint` — run eslint and a prettier check, by building the `lint` phase
|
- `script/lint` — run eslint and a prettier check, by building the `lint` phase
|
||||||
@@ -145,6 +179,34 @@ an exact version, installed from `yarn.lock` under `--frozen-lockfile` in both
|
|||||||
places, and reads `.gitignore` as its default ignore file — which is why
|
places, and reads `.gitignore` as its default ignore file — which is why
|
||||||
`.dockerignore` keeps `.gitignore` in the build context.
|
`.dockerignore` keeps `.gitignore` in the build context.
|
||||||
|
|
||||||
|
### Version
|
||||||
|
|
||||||
|
`quak --version` reports the `version` of `dist/package.json`, which
|
||||||
|
`script/build` writes after compiling; the repo's own `package.json` keeps
|
||||||
|
`0.0.0`, and that is what the tests, which run from source, report.
|
||||||
|
`script/version` decides what is written:
|
||||||
|
|
||||||
|
- the `VERSION` environment variable, or the `Dockerfile`'s `VERSION` build arg
|
||||||
|
(`--build-arg VERSION=...`), when one is given and not empty;
|
||||||
|
- otherwise, in a checkout with `.git`, `git describe --tags --always`: the tag
|
||||||
|
on a tagged commit; the tag, the commits since it and the short commit on a
|
||||||
|
later commit (`v1.2.3-4-gabc1234`); the short commit when no tag is reachable;
|
||||||
|
- otherwise, as in a source tarball, the version `package.json` declares.
|
||||||
|
|
||||||
|
The build fails if the checkout has `.git` and the version still comes out
|
||||||
|
empty, `dev` or `unknown`: such a build could not be traced back to its commit.
|
||||||
|
|
||||||
|
`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of
|
||||||
|
a clone stamps the commit it was built from; a shallow clone stamps a tag only
|
||||||
|
when the cloned commit itself carries one, and otherwise the short commit. It
|
||||||
|
leaves out `.git/config`, which holds the clone's remote URL and any credential
|
||||||
|
in it, so the image carries `.git` without its config; `git describe` does not
|
||||||
|
need that file. `script/docker` (and so `make docker`) and `script/cibuild` pass
|
||||||
|
the version they resolve on the host, with `--dirty`, as the build arg, which
|
||||||
|
takes precedence. The image's `org.opencontainers.image.version` label carries
|
||||||
|
that build arg only, so a build given none leaves it empty. `make build-bin`
|
||||||
|
bundles the built `dist/`, so the single binary reports the stamped version too.
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
Ente is one of very few photo services with a credible end-to-end encryption
|
Ente is one of very few photo services with a credible end-to-end encryption
|
||||||
@@ -239,6 +301,9 @@ quak/
|
|||||||
index.ts public library exports
|
index.ts public library exports
|
||||||
bin/
|
bin/
|
||||||
quak.ts CLI entrypoint (commander.js)
|
quak.ts CLI entrypoint (commander.js)
|
||||||
|
examples/
|
||||||
|
download-albums.ts
|
||||||
|
download every album's photos and metadata
|
||||||
test/ unit + integration tests (vitest)
|
test/ unit + integration tests (vitest)
|
||||||
Makefile
|
Makefile
|
||||||
Dockerfile lint phase, test phase, compile
|
Dockerfile lint phase, test phase, compile
|
||||||
@@ -247,10 +312,11 @@ quak/
|
|||||||
```
|
```
|
||||||
|
|
||||||
`make build` compiles that tree into `dist/`, preserving its shape: the library
|
`make build` compiles that tree into `dist/`, preserving its shape: the library
|
||||||
lands in `dist/src/` and the CLI in `dist/bin/quak.js`, which is what
|
lands in `dist/src/`, the examples in `dist/examples/`, and the CLI in
|
||||||
`package.json` points `main`, `types` and `bin` at. The compiler's `rootDir` is
|
`dist/bin/quak.js`, which is what `package.json` points `main`, `types` and
|
||||||
the repository root rather than `src/`, because `bin/` is compiled too and
|
`bin` at. The compiler's `rootDir` is the repository root rather than `src/`,
|
||||||
`rootDir` has to contain everything that is compiled.
|
because `bin/` is compiled too and `rootDir` has to contain everything that is
|
||||||
|
compiled.
|
||||||
|
|
||||||
### Cryptography
|
### Cryptography
|
||||||
|
|
||||||
@@ -528,25 +594,36 @@ the smallest does not.
|
|||||||
|
|
||||||
```
|
```
|
||||||
<dir>/
|
<dir>/
|
||||||
originals/
|
YYYY/YYYY-MM/YYYY-MM-DD/
|
||||||
<fileID>.<ext> actual file content (one per unique file,
|
YYYY-MM-DD.<fileID>.<ext> actual file content, at its save path (one
|
||||||
two for a live photo: see below)
|
per unique file, two for a live photo: see
|
||||||
<fileID>.json the file's basic metadata fields quak
|
below)
|
||||||
|
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
||||||
keeps, and its private and public magic
|
keeps, and its private and public magic
|
||||||
metadata
|
metadata
|
||||||
<fileID>.livephoto.json which of a live photo's two files is which
|
YYYY-MM-DD.<fileID>.livephoto.json
|
||||||
|
which of a live photo's two files is which
|
||||||
collections/
|
collections/
|
||||||
<name>/
|
<name>/
|
||||||
<title> -> ../../originals/<fileID>.<ext> (symlink)
|
<title> -> ../../YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>
|
||||||
|
(symlink)
|
||||||
<name>.json collection metadata + file list
|
<name>.json collection metadata + file list
|
||||||
failures.json files that failed and have not yet succeeded
|
failures.json files that failed and have not yet succeeded
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Each original is saved at its save path, the same path `photo.savePath` gives
|
||||||
|
and `photo.download()` writes (see Read surface below). The date is the photo's
|
||||||
|
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
|
||||||
|
time zone of the machine running quak. The extension is the one in the file's
|
||||||
|
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
|
||||||
|
letters and digits. When the date or the time zone changes, the next run saves
|
||||||
|
the original at its new path and leaves the old copy where it is.
|
||||||
|
|
||||||
`failures.json` records each failed file with the kind of failure, how many
|
`failures.json` records each failed file with the kind of failure, how many
|
||||||
times it has been tried and when it was last tried. A file leaves it once it
|
times it has been tried and when it was last tried. A file leaves it once it
|
||||||
succeeds, or once it is no longer in the library or in the backup's scope. The
|
succeeds, or once it is no longer in the library or in the backup's scope. The
|
||||||
library's `lib.backup({ includeThumbnails: true })` also writes
|
library's `lib.backup({ includeThumbnails: true })` also writes
|
||||||
`thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
|
`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not.
|
||||||
|
|
||||||
A collection's directory and JSON are named after the collection, and a symlink
|
A collection's directory and JSON are named after the collection, and a symlink
|
||||||
after the file's title, both with unsafe characters replaced. When two
|
after the file's title, both with unsafe characters replaced. When two
|
||||||
@@ -558,22 +635,21 @@ name stays the same from run to run until such a clash appears or goes away.
|
|||||||
|
|
||||||
A live photo, which Ente stores as one ZIP of its image and its video, is stored
|
A live photo, which Ente stores as one ZIP of its image and its video, is stored
|
||||||
as those two files, which a photo viewer can open: each is
|
as those two files, which a photo viewer can open: each is
|
||||||
`originals/<fileID>.<ext>` with the extension it has inside the ZIP (for example
|
`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP (for
|
||||||
`12345.heic` and `12345.mov`), and `<fileID>.livephoto.json` names the two. The
|
example `2026-03-01.12345.heic` and `2026-03-01.12345.mov`), and
|
||||||
live photo counts as stored only when both files are present and not empty. Its
|
`YYYY-MM-DD.<fileID>.livephoto.json` names the two. The live photo counts as
|
||||||
album folder links both, each named after the title with that file's extension
|
stored only when both files are present and not empty. Its album folder links
|
||||||
(`IMG_0001.heic` and `IMG_0001.mov`). A live photo that an earlier version of
|
both, each named after the title with that file's extension (`IMG_0001.heic` and
|
||||||
quak stored as the ZIP, under the image's name, is replaced by its two files on
|
`IMG_0001.mov`).
|
||||||
the next run, and the ZIP and its link are removed.
|
|
||||||
|
|
||||||
Each run removes the symlinks into `originals/` that no longer belong in their
|
Each run removes the symlinks into the date folders that no longer belong in
|
||||||
collection's directory, and the directories (and JSON) of collections that were
|
their collection's directory, and the directories (and JSON) of collections that
|
||||||
deleted or renamed. Nothing else in `collections/` is touched: a file or a
|
were deleted or renamed. Nothing else in `collections/` is touched: a file or a
|
||||||
symlink you put there stays, and a directory that still holds one after its
|
symlink you put there stays, and a directory that still holds one after its
|
||||||
symlinks are removed stays too, with its JSON.
|
symlinks are removed stays too, with its JSON.
|
||||||
|
|
||||||
Each file is downloaded exactly once regardless of how many collections it
|
Each file is downloaded exactly once regardless of how many collections it
|
||||||
appears in, and written once: straight into `originals/`, with no copy left in
|
appears in, and written once: straight to its save path, with no copy left in
|
||||||
the cache. An original the cache already held is copied from there instead. On
|
the cache. An original the cache already held is copied from there instead. On
|
||||||
subsequent runs, existing originals are skipped. If a download fails, the error
|
subsequent runs, existing originals are skipped. If a download fails, the error
|
||||||
is logged and the backup continues with the next file. The exit code is non-zero
|
is logged and the backup continues with the next file. The exit code is non-zero
|
||||||
@@ -586,14 +662,15 @@ Each original is written to a temporary file in the same directory, synced to
|
|||||||
disk, and renamed into place, so an original is either complete or absent, even
|
disk, and renamed into place, so an original is either complete or absent, even
|
||||||
after a power cut. A downloaded original's temporary file is named
|
after a power cut. A downloaded original's temporary file is named
|
||||||
`.quak-<pid>-<random>.tmp`, one copied from the cache
|
`.quak-<pid>-<random>.tmp`, one copied from the cache
|
||||||
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
|
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is
|
||||||
one of these temporary files behind; the next backup deletes those whose process
|
killed can leave one of these temporary files behind; the next backup deletes
|
||||||
is no longer running. The content cache uses the same scheme, and opening a
|
those whose process is no longer running. The content cache uses the same
|
||||||
library deletes the temporary files in the cache whose process is no longer
|
scheme, and opening a library deletes the temporary files in the cache whose
|
||||||
running, so a download another process has in progress in the same cache is left
|
process is no longer running, so a download another process has in progress in
|
||||||
alone. The rename replaces whatever was at the destination rather than writing
|
the same cache is left alone. The rename replaces whatever was at the
|
||||||
through it: a symlink there is replaced, not followed, and the new file has the
|
destination rather than writing through it: a symlink there is replaced, not
|
||||||
temporary file's permissions, not those of the file it replaced.
|
followed, and the new file has the temporary file's permissions, not those of
|
||||||
|
the file it replaced.
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
@@ -636,10 +713,10 @@ background, so an unreachable server does not block opening.
|
|||||||
`LibraryOptions`:
|
`LibraryOptions`:
|
||||||
|
|
||||||
| Option | Default | Meaning |
|
| Option | Default | Meaning |
|
||||||
| ------------------------ | -------------------------------------------------- | --------------------------------------------------------------------- |
|
| ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------- |
|
||||||
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
||||||
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
|
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
|
||||||
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
|
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached |
|
||||||
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
||||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||||
@@ -651,6 +728,9 @@ background, so an unreachable server does not block opening.
|
|||||||
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
|
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
|
||||||
| `contentSource` | the client's own | override the byte source (mainly for tests) |
|
| `contentSource` | the client's own | override the byte source (mainly for tests) |
|
||||||
|
|
||||||
|
The default `downloadDirectory` is resolved against the working directory once,
|
||||||
|
when the library opens; `lib.downloadDirectory` holds the result.
|
||||||
|
|
||||||
Concurrency is set through `pools`: construct
|
Concurrency is set through `pools`: construct
|
||||||
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
||||||
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
|
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
|
||||||
@@ -702,39 +782,56 @@ and `originalPath`, and `photo.year`, the local-time year of `takenAt`, all as
|
|||||||
synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more
|
synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more
|
||||||
synchronous getters look at the disk and never touch the network:
|
synchronous getters look at the disk and never touch the network:
|
||||||
|
|
||||||
- `photo.savePath` → `string | undefined` — where `lib.backup()` writes the
|
- `photo.savePath` → `string` — where `photo.download()` and `lib.backup()` put
|
||||||
original, `originals/<fileID>.<ext>` under the `downloadDirectory` the library
|
the original, whether or not it is there yet:
|
||||||
was opened with, whether or not it is there yet: for a live photo already
|
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
||||||
stored, its image. For a live photo not yet stored, it carries the title's
|
`downloadDirectory` (see Backup layout above for the date and the extension).
|
||||||
extension, and the backup may store the image under a different one, found
|
For a live photo already stored, its image. For a live photo not yet stored,
|
||||||
inside the live photo. `undefined` when the library has no `downloadDirectory`
|
it carries the title's extension, and the image may be stored under a
|
||||||
or no content source.
|
different one, found inside the live photo. It needs no content source.
|
||||||
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`.
|
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`. A
|
||||||
|
copy only in the cache does not count.
|
||||||
|
|
||||||
Four async methods may download:
|
These async methods may download:
|
||||||
|
|
||||||
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
|
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
|
||||||
full-resolution file. For a live photo, `path` and `bytes` are its image's and
|
full-resolution file. For a live photo, `path` and `bytes` are its image's and
|
||||||
`videoPath` is its video.
|
`videoPath` is its video.
|
||||||
|
- `await photo.download()` → `{ path, bytes, videoPath? }`, as `original()` —
|
||||||
|
puts the original at `savePath`, creating its folders. When it is already
|
||||||
|
there, nothing is written. When the cache holds it, it is copied from the
|
||||||
|
cache; otherwise it is fetched straight to `savePath`, with no copy left in
|
||||||
|
the cache. Afterwards `isLocal` is true.
|
||||||
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
|
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
|
||||||
- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read
|
- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read
|
||||||
through `original()`; for a live photo, its image's.
|
through `original()`; for a live photo, its image's.
|
||||||
- `await photo.exif(opts?)` → `PhotoExif` — `make`, `model`, `lensModel`,
|
- `await photo.exif(opts?)` → `ExifTags` — every EXIF tag in the file, keyed by
|
||||||
`dateTimeOriginal`, `offsetTimeOriginal`, `exposureTime`, `fNumber`, `iso`,
|
tag name, each as exifreader decodes it, with its `id`, `value`, `description`
|
||||||
`focalLength`, `orientation`, `gpsLatitude`, `gpsLongitude` and `gpsAltitude`,
|
and `computed` value: for example `Make` is
|
||||||
each absent when the file lacks it. GPS values are signed decimal degrees and
|
`{ id: 271, value: ["Canon"], description: "Canon", computed: "Canon" }`. A
|
||||||
metres. `dateTimeOriginal` is the camera's clock reading held in the `Date`'s
|
tag exifreader has no name for is keyed `undefined-<tag number>`. The embedded
|
||||||
UTC fields; `offsetTimeOriginal`, when present, is that clock's offset from
|
thumbnail's tags are under `Thumbnail`, so they cannot hide the main image's
|
||||||
UTC. EXIF is read from any image format exifreader reads (such as JPEG,
|
tags of the same name; the thumbnail image itself is left out. EXIF is read
|
||||||
HEIC/HEIF, AVIF, PNG, WebP and TIFF), a live photo's image included. Any other
|
from any image format exifreader reads (such as JPEG, HEIC/HEIF, AVIF, PNG,
|
||||||
original gives `{}`, and a video gives `{}` without being downloaded.
|
WebP and TIFF), a live photo's image included. Any other original gives `{}`,
|
||||||
|
and a video gives `{}` without being downloaded.
|
||||||
|
- `await photo.make(opts?)`, and likewise `model()`, `lensModel()`,
|
||||||
|
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
||||||
|
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
||||||
|
and `gpsAltitude()` → one common field each, picked from the tags `exif()`
|
||||||
|
returns and typed as in `PhotoExif`, or `undefined` when the file lacks it.
|
||||||
|
GPS values are signed decimal degrees and metres. `dateTimeOriginal()` is the
|
||||||
|
camera's clock reading held in the `Date`'s UTC fields;
|
||||||
|
`offsetTimeOriginal()`, when present, is that clock's offset from UTC. Each
|
||||||
|
calls `exif()` with its `opts`, so each call reads the original again.
|
||||||
|
|
||||||
They serve from the on-disk content cache when the bytes are present and
|
They serve from the on-disk content cache when the bytes are present and
|
||||||
otherwise fetch through the pools; `original()`, `content()` and `exif()` also
|
otherwise fetch through the pools; `original()`, `content()` and `exif()` also
|
||||||
serve an original the backup has already stored. `opts.onProgress` reports
|
serve an original already stored at its save path. `opts.onProgress` reports
|
||||||
per-file progress. They throw when the library was opened without a content
|
per-file progress. They throw when the library was opened without a content
|
||||||
source. An original that `content()` or `exif()` downloads lands in the cache,
|
source. An original that `original()`, `content()` or `exif()` downloads lands
|
||||||
which does not make `isLocal` true; only `lib.backup()` does.
|
in the cache, which does not make `isLocal` true; only `download()` and
|
||||||
|
`lib.backup()` do.
|
||||||
|
|
||||||
Lower-level accessors that return decrypted model objects (which hold key
|
Lower-level accessors that return decrypted model objects (which hold key
|
||||||
material) are also available: `listCollections()`, `getCollection(id)`,
|
material) are also available: `listCollections()`, `getCollection(id)`,
|
||||||
@@ -777,15 +874,15 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
|||||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||||
a query vector the caller produced elsewhere.
|
a query vector the caller produced elsewhere.
|
||||||
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
|
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
|
||||||
`fresh()` does, fetches every in-scope original not already in the backup
|
`fresh()` does, puts every in-scope original not already at its save path
|
||||||
(and, with `includeThumbnails`, thumbnails) through the content cache, and
|
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||||
rebuilds the on-disk backup tree with a durable failure ledger. A fetched
|
thumbnails) through the content cache, and rebuilds the on-disk backup tree
|
||||||
original is written straight into the backup's `originals/` and not into the
|
with a durable failure ledger. A fetched original is written straight to its
|
||||||
cache, which then counts it as present; one the cache already held is copied
|
save path and not into the cache, which then counts it as present; one the
|
||||||
from there. `BackupOptions`: `downloadDirectory` (falls back to the one
|
cache already held is copied from there. `BackupOptions`: `downloadDirectory`
|
||||||
`open()` was given), `includeOriginals` (default `true`), `includeThumbnails`
|
(falls back to the library's), `includeOriginals` (default `true`),
|
||||||
(default `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above
|
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See
|
||||||
for the tree it writes.
|
Backup layout above for the tree it writes.
|
||||||
|
|
||||||
### Request pools
|
### Request pools
|
||||||
|
|
||||||
@@ -816,12 +913,9 @@ When `metadata.json` belongs to a different account than the client's,
|
|||||||
originals and thumbnails are kept; they are reached only through the files the
|
originals and thumbnails are kept; they are reached only through the files the
|
||||||
current account's records name.
|
current account's records name.
|
||||||
|
|
||||||
A live photo's original is cached as in the backup: its image and its video,
|
A live photo's original is cached as at its save path: its image and its video,
|
||||||
each `originals/<fileID>.<ext>` with its own extension, and
|
each `originals/<fileID>.<ext>` with its own extension, and
|
||||||
`originals/<fileID>.livephoto.json` naming them; the two are evicted together. A
|
`originals/<fileID>.livephoto.json` naming them; the two are evicted together.
|
||||||
live photo that an earlier version cached as its ZIP is not served: the library
|
|
||||||
removes the ZIP when it opens the cache, and fetches the two files when the
|
|
||||||
photo is next read or precached.
|
|
||||||
|
|
||||||
A stored file appears only via an atomic temp-then-rename, so its presence means
|
A stored file appears only via an atomic temp-then-rename, so its presence means
|
||||||
it is complete. Every downloaded original (by `quak get`, the cache, or
|
it is complete. Every downloaded original (by `quak get`, the cache, or
|
||||||
@@ -840,13 +934,13 @@ from a very old client, is stored unchecked.
|
|||||||
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
|
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
|
||||||
`LibraryClient`, `RefreshEvent`
|
`LibraryClient`, `RefreshEvent`
|
||||||
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
|
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
|
||||||
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
|
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`, `SavePathLookup`
|
||||||
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
|
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
|
||||||
`EnsureOptions`, `EnsureResult`, `ContentSource`
|
`EnsureOptions`, `EnsureResult`, `ContentSource`
|
||||||
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
|
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
|
||||||
`LibraryChange`
|
`LibraryChange`
|
||||||
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
||||||
- `src/exif.ts`: `PhotoExif`
|
- `src/exif.ts`: `ExifTags`, `PhotoExif`
|
||||||
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
||||||
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
||||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||||
|
|||||||
@@ -25,6 +25,53 @@ declares one.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
|
||||||
|
`dev` (issue 154). `script/build` writes the version `script/version` prints
|
||||||
|
into `dist/package.json`: the `VERSION` environment variable or build arg when
|
||||||
|
one is given, otherwise `git describe --tags --always`, otherwise the version
|
||||||
|
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
|
||||||
|
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
|
||||||
|
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
||||||
|
same version.
|
||||||
|
|
||||||
|
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
|
||||||
|
keyed by tag name, each as exifreader decodes it, not only the thirteen common
|
||||||
|
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
|
||||||
|
without the thumbnail image. The thirteen typed methods stay, each picking its
|
||||||
|
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
|
||||||
|
script's JSON files now carry every tag.
|
||||||
|
|
||||||
|
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
|
||||||
|
version cached as one ZIP (issue 151). When the cache opens, a live photo's
|
||||||
|
file that no JSON file names is now always left alone.
|
||||||
|
|
||||||
|
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
||||||
|
every album downloads each photo to its save path, writes the photo's record
|
||||||
|
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
||||||
|
`albums/<collectionID>.json` (issue 144). The build compiles it to
|
||||||
|
`dist/examples/`; the README's "Examples" section says how to run it.
|
||||||
|
|
||||||
|
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
|
||||||
|
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
|
||||||
|
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
||||||
|
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
||||||
|
and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
|
||||||
|
field, or undefined when the file lacks it. `Photo` implements a type with one
|
||||||
|
method per `PhotoExif` field, so the build's type check fails when a field has
|
||||||
|
no method. A test checks, on the JPEG and the HEIC, that each method gives the
|
||||||
|
same value as `exif()`.
|
||||||
|
|
||||||
|
- 2026-10-01: Each original's save path is
|
||||||
|
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
||||||
|
download directory, which defaults to `photos` in the working directory (issue
|
||||||
|
143). The date is the photo's `takenAt` in the machine's time zone.
|
||||||
|
`photo.savePath` is always a string, with or without a content cache, and
|
||||||
|
`isLocal` is true only when the original is at its save path.
|
||||||
|
`photo.download()` puts the original there, copied from the cache when the
|
||||||
|
cache holds it and fetched otherwise. `lib.backup()` does the same for each
|
||||||
|
file, writes each file's JSON beside its original, and links `collections/` to
|
||||||
|
the save paths.
|
||||||
|
|
||||||
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
|
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
|
||||||
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
|
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
|
||||||
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
|
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
|
||||||
|
|||||||
@@ -0,0 +1,129 @@
|
|||||||
|
// Download every album's photos to a directory, with each photo's metadata
|
||||||
|
// beside it, using only quak's public API. The README's "Examples" section
|
||||||
|
// describes the files it writes.
|
||||||
|
//
|
||||||
|
// yarn build
|
||||||
|
// QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir]
|
||||||
|
|
||||||
|
import { realpathSync } from "node:fs";
|
||||||
|
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||||
|
import { join, relative } from "node:path";
|
||||||
|
import { stdin, stdout } from "node:process";
|
||||||
|
import { createInterface } from "node:readline/promises";
|
||||||
|
import { pathToFileURL } from "node:url";
|
||||||
|
|
||||||
|
import { Client, Library } from "../src/index.js";
|
||||||
|
|
||||||
|
// Write `text` to `path` unless the file already holds exactly that, so a
|
||||||
|
// second run rewrites nothing.
|
||||||
|
async function writeIfChanged(path: string, text: string): Promise<void> {
|
||||||
|
const current = await readFile(path, "utf-8").catch(() => undefined);
|
||||||
|
if (current !== text) await writeFile(path, text);
|
||||||
|
}
|
||||||
|
|
||||||
|
const pretty = (value: unknown): string =>
|
||||||
|
JSON.stringify(value, null, 2) + "\n";
|
||||||
|
|
||||||
|
// For every album in `lib`, download each photo to its save path, write the
|
||||||
|
// photo's metadata to `{savePath}.json`, and write the album's photos to
|
||||||
|
// `{dir}/albums/{collectionID}.json`. Returns how many photos it downloaded
|
||||||
|
// and how many were already at their save paths.
|
||||||
|
export async function downloadAlbums(
|
||||||
|
lib: Library,
|
||||||
|
dir: string,
|
||||||
|
): Promise<{ downloaded: number; alreadyLocal: number }> {
|
||||||
|
let downloaded = 0;
|
||||||
|
let alreadyLocal = 0;
|
||||||
|
// A photo in several albums is handled once.
|
||||||
|
const done = new Set<number>();
|
||||||
|
// fresh() waits for a refresh from the server and throws if it fails, so
|
||||||
|
// albums and photos added since the cache was last written are included.
|
||||||
|
const { albums } = await lib.fresh();
|
||||||
|
for (const album of albums.list()) {
|
||||||
|
const savePaths: string[] = [];
|
||||||
|
for (const photo of album.photos.list()) {
|
||||||
|
if (!done.has(photo.fileID)) {
|
||||||
|
done.add(photo.fileID);
|
||||||
|
if (photo.isLocal) alreadyLocal++;
|
||||||
|
else downloaded++;
|
||||||
|
await photo.download();
|
||||||
|
// The cache paths say where quak's cache keeps copies, not
|
||||||
|
// anything about the photo.
|
||||||
|
const record = { ...photo.record() };
|
||||||
|
delete record.thumbnailPath;
|
||||||
|
delete record.originalPath;
|
||||||
|
const exif = await photo.exif();
|
||||||
|
// savePath is read after download(): a live photo's names its
|
||||||
|
// image only once the image is stored.
|
||||||
|
await writeIfChanged(
|
||||||
|
`${photo.savePath}.json`,
|
||||||
|
pretty({ ...record, exif }),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
savePaths.push(relative(dir, photo.savePath));
|
||||||
|
}
|
||||||
|
await mkdir(join(dir, "albums"), { recursive: true });
|
||||||
|
await writeIfChanged(
|
||||||
|
join(dir, "albums", `${album.collectionID}.json`),
|
||||||
|
pretty({
|
||||||
|
collectionID: album.collectionID,
|
||||||
|
name: album.name,
|
||||||
|
savePaths,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return { downloaded, alreadyLocal };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ask for a login code on the terminal. Client.login calls this only when the
|
||||||
|
// account requires a code.
|
||||||
|
async function ask(question: string): Promise<string> {
|
||||||
|
const terminal = createInterface({ input: stdin, output: stdout });
|
||||||
|
try {
|
||||||
|
return await terminal.question(question);
|
||||||
|
} finally {
|
||||||
|
terminal.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main(): Promise<void> {
|
||||||
|
const email = process.env.QUAK_EMAIL;
|
||||||
|
const password = process.env.QUAK_PASSWORD;
|
||||||
|
if (!email || !password) {
|
||||||
|
console.error(
|
||||||
|
"Set QUAK_EMAIL and QUAK_PASSWORD to the account's email and password.",
|
||||||
|
);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
const dir = process.argv[2] ?? "photos";
|
||||||
|
|
||||||
|
const client = await Client.login({
|
||||||
|
email,
|
||||||
|
password,
|
||||||
|
totp: () => ask("Two-factor code: "),
|
||||||
|
emailOTP: () => ask("Code sent to your email: "),
|
||||||
|
});
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
downloadDirectory: dir,
|
||||||
|
// As in `quak backup`: fetch only the originals this script saves,
|
||||||
|
// not every thumbnail and the recent originals into the cache too.
|
||||||
|
precacheThumbnails: false,
|
||||||
|
precacheOriginals: false,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const { downloaded, alreadyLocal } = await downloadAlbums(lib, dir);
|
||||||
|
console.log(
|
||||||
|
`${downloaded} photos downloaded, ${alreadyLocal} already local, in ${dir}`,
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run main() when node runs this file, not when a test imports it. argv[1] is
|
||||||
|
// the path as given, and import.meta.url has symlinks resolved.
|
||||||
|
const script = process.argv[1];
|
||||||
|
if (script && pathToFileURL(realpathSync(script)).href === import.meta.url) {
|
||||||
|
await main();
|
||||||
|
}
|
||||||
+23
-9
@@ -1,7 +1,7 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/build: compile the TypeScript sources into dist/, then verify that
|
# script/build: compile the TypeScript sources into dist/, stamp the version
|
||||||
# the artifacts package.json advertises are among the files the compiler
|
# script/version prints into it, then verify that the artifacts package.json
|
||||||
# actually wrote. tsc reports success by exit status alone and knows nothing
|
# advertises are among the files the compiler actually wrote. tsc reports success by exit status alone and knows nothing
|
||||||
# about the manifest, so without this step a green build can still ship a
|
# about the manifest, so without this step a green build can still ship a
|
||||||
# package whose main, types or bin resolve to nothing. Our own extension to
|
# package whose main, types or bin resolve to nothing. Our own extension to
|
||||||
# scripts-to-rule-them-all.
|
# scripts-to-rule-them-all.
|
||||||
@@ -46,13 +46,24 @@ for (const bin of bins) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
# src/index.ts imports ../package.json for the version, which tsc copies to
|
# src/index.ts imports ../package.json for the version, which tsc copies to
|
||||||
# dist/package.json. Running the built CLI proves that import resolves from
|
# dist/package.json. The version script/version prints is written into that
|
||||||
# dist/ and reports the version package.json declares.
|
# copy only; the repo's own package.json is left as it is.
|
||||||
|
stamp_version() {
|
||||||
|
node -e '
|
||||||
|
const { readFileSync, writeFileSync } = require("node:fs");
|
||||||
|
|
||||||
|
const pkg = JSON.parse(readFileSync("dist/package.json", "utf-8"));
|
||||||
|
pkg.version = process.argv[1];
|
||||||
|
writeFileSync("dist/package.json", JSON.stringify(pkg, null, 4) + "\n");
|
||||||
|
' "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Running the built CLI proves the import resolves from dist/ and reports
|
||||||
|
# the stamped version.
|
||||||
verify_version() {
|
verify_version() {
|
||||||
built="$(node dist/bin/quak.js --version)"
|
built="$(node dist/bin/quak.js --version)"
|
||||||
declared="$(node -p 'require("./package.json").version')"
|
if [ "$built" != "$1" ]; then
|
||||||
if [ "$built" != "$declared" ]; then
|
echo "build: dist/bin/quak.js reports $built, the build stamped $1" >&2
|
||||||
echo "build: dist/bin/quak.js reports $built, package.json declares $declared" >&2
|
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
echo "build: dist/bin/quak.js reports version $built"
|
echo "build: dist/bin/quak.js reports version $built"
|
||||||
@@ -60,9 +71,12 @@ verify_version() {
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
|
# Own line, so that a failing script/version stops the build.
|
||||||
|
version="$("$ROOT/script/version")"
|
||||||
yarn run tsc
|
yarn run tsc
|
||||||
|
stamp_version "$version"
|
||||||
verify_entrypoints
|
verify_entrypoints
|
||||||
verify_version
|
verify_version "$version"
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+3
-3
@@ -15,9 +15,9 @@ main() {
|
|||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# Own line: a failing command substitution inside an argument does
|
# Own line: a failing command substitution inside an argument does
|
||||||
# not trip `set -e`, so the inline form degrades silently to an
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# empty constant. VERSION is computed here because .dockerignore
|
# empty constant. The version resolved here goes in as the VERSION
|
||||||
# excludes .git, so `git describe` in a build stage yields an empty
|
# build arg, which takes precedence over what the build would derive
|
||||||
# version without failing.
|
# from the .git in its context.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
docker build --no-cache \
|
||||||
|
|||||||
+3
-3
@@ -12,9 +12,9 @@ main() {
|
|||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# Own line: a failing command substitution inside an argument does
|
# Own line: a failing command substitution inside an argument does
|
||||||
# not trip `set -e`, so the inline form degrades silently to an
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# empty constant. VERSION is computed here because .dockerignore
|
# empty constant. The version resolved here goes in as the VERSION
|
||||||
# excludes .git, so `git describe` in a build stage yields an empty
|
# build arg, which takes precedence over what the build would derive
|
||||||
# version without failing.
|
# from the .git in its context.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
docker build --no-cache \
|
||||||
|
|||||||
Executable
+41
@@ -0,0 +1,41 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# script/version: print the version script/build stamps into the built
|
||||||
|
# package. Our own extension to scripts-to-rule-them-all.
|
||||||
|
#
|
||||||
|
# Order of precedence:
|
||||||
|
#
|
||||||
|
# 1. $VERSION, if set and not empty: an explicit value, such as the
|
||||||
|
# Dockerfile's VERSION build arg.
|
||||||
|
# 2. If this checkout has .git, `git describe --tags --always`: the tag
|
||||||
|
# on a tagged commit; the tag, the commits since it and the short
|
||||||
|
# commit on a later commit (v1.2.3-4-gabc1234); the short commit when
|
||||||
|
# no tag is reachable.
|
||||||
|
# 3. Otherwise, as in a source tarball, the version package.json declares.
|
||||||
|
#
|
||||||
|
# A checkout with .git whose version still comes out empty, dev or unknown
|
||||||
|
# fails: git is missing or could not read the checkout, and the build could
|
||||||
|
# not be traced back to its commit.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
main() {
|
||||||
|
cd "$ROOT"
|
||||||
|
version="${VERSION:-}"
|
||||||
|
if [ -e .git ]; then
|
||||||
|
if [ -z "$version" ]; then
|
||||||
|
version="$(git describe --tags --always || true)"
|
||||||
|
fi
|
||||||
|
case "$version" in
|
||||||
|
"" | dev | unknown)
|
||||||
|
echo "version: $ROOT has .git, but the version came out '$version'" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
elif [ -z "$version" ]; then
|
||||||
|
version="$(node -p 'require("./package.json").version')"
|
||||||
|
fi
|
||||||
|
echo "$version"
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
+110
-120
@@ -2,29 +2,30 @@
|
|||||||
//
|
//
|
||||||
// `lib.backup()` waits for a completed refresh of the library (a failed one
|
// `lib.backup()` waits for a completed refresh of the library (a failed one
|
||||||
// fails the backup before any file is touched), then, for every file in scope,
|
// fails the backup before any file is touched), then, for every file in scope,
|
||||||
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
|
// puts its original at its save path under `downloadDirectory`, as
|
||||||
// derived views (per-file sidecars, per-collection symlink trees,
|
// `Photo.download()` does, and rebuilds the derived views (per-file sidecars,
|
||||||
// per-collection JSON) from the model. The on-disk layout is the historical
|
// per-collection symlink trees, per-collection JSON) from the model. The
|
||||||
// one:
|
// on-disk layout:
|
||||||
//
|
//
|
||||||
// <downloadDirectory>/
|
// <downloadDirectory>/
|
||||||
// originals/<fileID>.<ext> the decrypted bytes
|
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||||
// originals/<fileID>.json per-file metadata sidecar
|
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
||||||
// collections/<name>/<title> symlink into ../../originals
|
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar
|
||||||
|
// collections/<name>/<title> symlink to the original
|
||||||
// collections/<name>.json per-collection metadata
|
// collections/<name>.json per-collection metadata
|
||||||
// failures.json durable ledger of unresolved failures
|
// failures.json durable ledger of unresolved failures
|
||||||
//
|
//
|
||||||
// A live photo's original is its image and its video, `<fileID>.<ext>` each
|
// A live photo's original is its image and its video, each with its own
|
||||||
// with its own extension, and `originals/<fileID>.livephoto.json` naming them;
|
// extension, beside `YYYY-MM-DD.<fileID>.livephoto.json` naming them; its album
|
||||||
// its album folders link both.
|
// folders link both.
|
||||||
//
|
//
|
||||||
// Crash-safety rests on two properties. Bytes are present-means-complete: an
|
// Crash-safety rests on two properties. Bytes are present-means-complete: an
|
||||||
// original appears under `originals/` only via the content layer's atomic
|
// original appears at its save path only via the content layer's atomic
|
||||||
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
|
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
|
||||||
// interrupted run resumes by listing the directory. The derived views hold no
|
// interrupted run resumes by looking at the save paths. The derived views hold
|
||||||
// unique state, so they are rebuilt every run; that repairs stale sidecars and
|
// no unique state, so they are rebuilt every run; that repairs stale sidecars
|
||||||
// missing or broken symlinks left by an earlier crash. A rebuild also removes
|
// and missing or broken symlinks left by an earlier crash. A rebuild also
|
||||||
// the symlinks into originals/ that no longer belong to an album, and the
|
// removes the symlinks to originals that no longer belong to an album, and the
|
||||||
// directories of albums that no longer exist.
|
// directories of albums that no longer exist.
|
||||||
//
|
//
|
||||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||||
@@ -48,24 +49,25 @@ import {
|
|||||||
symlinkSync,
|
symlinkSync,
|
||||||
writeFileSync,
|
writeFileSync,
|
||||||
} from "node:fs";
|
} from "node:fs";
|
||||||
import { copyFile, rename, rm } from "node:fs/promises";
|
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||||
import { basename, dirname, extname, join, relative } from "node:path";
|
|
||||||
|
|
||||||
import { fsyncPath, removeLeftoverTempFiles } from "./download/index.js";
|
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||||
import { sanitizeFileName, withExtension } from "./filename.js";
|
import { sanitizeFileName, withExtension } from "./filename.js";
|
||||||
import {
|
import {
|
||||||
nameInOriginals,
|
copyAtomic,
|
||||||
storedOriginal,
|
placeOriginal,
|
||||||
writeLivePhotoJSON,
|
savePath,
|
||||||
|
storedAtSavePath,
|
||||||
} from "./library/content.js";
|
} from "./library/content.js";
|
||||||
|
import { representative } from "./library/records.js";
|
||||||
import type { Collection, EnteFile } from "./model/types.js";
|
import type { Collection, EnteFile } from "./model/types.js";
|
||||||
|
|
||||||
export type ProgressCallback = (message: string) => void;
|
export type ProgressCallback = (message: string) => void;
|
||||||
|
|
||||||
export interface BackupOptions {
|
export interface BackupOptions {
|
||||||
// Where the backup tree lives. Required: with none, `backup()` throws
|
// Where the backup tree lives. `lib.backup()` defaults it to the library's
|
||||||
// before any network traffic. A library opened with a `downloadDirectory`
|
// download directory; `runBackup` with none throws before any network
|
||||||
// supplies the default.
|
// traffic.
|
||||||
downloadDirectory?: string;
|
downloadDirectory?: string;
|
||||||
// Fetch and store full-resolution originals. Default true.
|
// Fetch and store full-resolution originals. Default true.
|
||||||
includeOriginals?: boolean;
|
includeOriginals?: boolean;
|
||||||
@@ -89,7 +91,7 @@ export interface BackupResult {
|
|||||||
totalFiles: number;
|
totalFiles: number;
|
||||||
// Originals fetched (or copied from the cache) this run.
|
// Originals fetched (or copied from the cache) this run.
|
||||||
downloaded: number;
|
downloaded: number;
|
||||||
// Originals already present and left untouched.
|
// Originals already at their save path and left untouched.
|
||||||
skipped: number;
|
skipped: number;
|
||||||
// Files with an unresolved failure after this run (the ledger size); the
|
// Files with an unresolved failure after this run (the ledger size); the
|
||||||
// CLI exits non-zero while this is above zero. A file can be both
|
// CLI exits non-zero while this is above zero. A file can be both
|
||||||
@@ -107,8 +109,8 @@ export interface BackupLibrary {
|
|||||||
listFiles(collectionID: number): EnteFile[];
|
listFiles(collectionID: number): EnteFile[];
|
||||||
// Get an original's bytes onto disk through the content cache/pools,
|
// Get an original's bytes onto disk through the content cache/pools,
|
||||||
// returning where they landed: `destination` when they were fetched now,
|
// returning where they landed: `destination` when they were fetched now,
|
||||||
// otherwise wherever they already were (the cache, or a prior backup). A
|
// otherwise wherever they already were (the cache, or the library's save
|
||||||
// live photo lands as its image and its video, fetched now beside
|
// path). A live photo lands as its image and its video, fetched now beside
|
||||||
// `destination`.
|
// `destination`.
|
||||||
original(
|
original(
|
||||||
fileID: number,
|
fileID: number,
|
||||||
@@ -168,58 +170,6 @@ const classify = (err: unknown): FailureClass => {
|
|||||||
const errorMessage = (err: unknown): string =>
|
const errorMessage = (err: unknown): string =>
|
||||||
err instanceof Error ? err.message : String(err);
|
err instanceof Error ? err.message : String(err);
|
||||||
|
|
||||||
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
|
||||||
// `dest` appears only once it is whole ("present means complete"). As in the
|
|
||||||
// download writer, the temp file is fsynced before the rename and the directory
|
|
||||||
// after it, so a power cut cannot leave a correctly named but short original.
|
|
||||||
// The temp name carries this process's ID so a later run can tell a leftover
|
|
||||||
// from a copy still in progress (see `removeLeftoverTempFiles`).
|
|
||||||
const copyAtomic = async (src: string, dest: string): Promise<void> => {
|
|
||||||
if (src === dest) return;
|
|
||||||
const tmp = join(
|
|
||||||
dirname(dest),
|
|
||||||
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
|
|
||||||
.toString(36)
|
|
||||||
.slice(2)}.tmp`,
|
|
||||||
);
|
|
||||||
try {
|
|
||||||
await copyFile(src, tmp);
|
|
||||||
await fsyncPath(tmp);
|
|
||||||
// `rename` replaces the destination's directory entry: an existing
|
|
||||||
// symlink at `dest` is replaced, not followed, and the new file has
|
|
||||||
// the temp file's permissions (copied from `src`).
|
|
||||||
await rename(tmp, dest);
|
|
||||||
await fsyncPath(dirname(dest));
|
|
||||||
} finally {
|
|
||||||
await rm(tmp, { force: true });
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// Put an original the library returned at `dest` in originals/, where a fresh
|
|
||||||
// fetch already wrote it. A live photo's image and video go beside `dest`: when
|
|
||||||
// they came from the cache they are copied, after removing whatever was at
|
|
||||||
// `dest` (an earlier version's ZIP of the two). Then the JSON file naming them
|
|
||||||
// is written, which is what makes the live photo count as stored.
|
|
||||||
const placeOriginal = async (
|
|
||||||
file: EnteFile,
|
|
||||||
dest: string,
|
|
||||||
got: { path: string; videoPath?: string },
|
|
||||||
): Promise<void> => {
|
|
||||||
if (got.videoPath === undefined) {
|
|
||||||
await copyAtomic(got.path, dest);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const originalsDir = dirname(dest);
|
|
||||||
const path = join(originalsDir, basename(got.path));
|
|
||||||
const videoPath = join(originalsDir, basename(got.videoPath));
|
|
||||||
if (got.path !== path) {
|
|
||||||
await rm(dest, { force: true });
|
|
||||||
await copyAtomic(got.path, path);
|
|
||||||
await copyAtomic(got.videoPath, videoPath);
|
|
||||||
}
|
|
||||||
await writeLivePhotoJSON(originalsDir, file.id, { path, videoPath });
|
|
||||||
};
|
|
||||||
|
|
||||||
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
|
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
|
||||||
// non-symlink entry. Throws on failure (a directory in the way, no permission)
|
// non-symlink entry. Throws on failure (a directory in the way, no permission)
|
||||||
// so the caller records it and moves on rather than aborting the run.
|
// so the caller records it and moves on rather than aborting the run.
|
||||||
@@ -291,36 +241,55 @@ const linksFor = (
|
|||||||
}));
|
}));
|
||||||
};
|
};
|
||||||
|
|
||||||
// Remove the symlinks in the album directory `dir` that point into
|
// Every date folder (`YYYY/YYYY-MM/YYYY-MM-DD/`) under `root`, whether or not a
|
||||||
// `originalsDir` and are not named in `keep`. Nothing else in the directory
|
// file in this backup is saved there. A folder that cannot be read is skipped.
|
||||||
// is touched: anything else there was put there by the user.
|
const dateFolders = (root: string): string[] => {
|
||||||
|
const subfolders = (dir: string, name: RegExp): string[] => {
|
||||||
|
try {
|
||||||
|
return readdirSync(dir, { withFileTypes: true })
|
||||||
|
.filter((e) => e.isDirectory() && name.test(e.name))
|
||||||
|
.map((e) => join(dir, e.name));
|
||||||
|
} catch {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
};
|
||||||
|
return subfolders(root, /^\d{4}$/)
|
||||||
|
.flatMap((year) => subfolders(year, /^\d{4}-\d\d$/))
|
||||||
|
.flatMap((month) => subfolders(month, /^\d{4}-\d\d-\d\d$/));
|
||||||
|
};
|
||||||
|
|
||||||
|
// Whether the entry at `path` is a symlink a backup to `root` made: one to an
|
||||||
|
// original in a `YYYY/YYYY-MM/YYYY-MM-DD/` folder of `root`.
|
||||||
|
const linksToOriginal = (path: string, root: string): boolean => {
|
||||||
|
if (!lstatSync(path).isSymbolicLink()) return false;
|
||||||
|
const target = relative(root, resolve(dirname(path), readlinkSync(path)));
|
||||||
|
return /^\d{4}\/\d{4}-\d\d\/\d{4}-\d\d-\d\d\/[^/]+$/.test(target);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Remove the symlinks in the album directory `dir` that point to an original
|
||||||
|
// in `root` and are not named in `keep`. Nothing else in the directory is
|
||||||
|
// touched: anything else there was put there by the user.
|
||||||
const removeStaleLinks = (
|
const removeStaleLinks = (
|
||||||
dir: string,
|
dir: string,
|
||||||
keep: Set<string>,
|
keep: Set<string>,
|
||||||
originalsDir: string,
|
root: string,
|
||||||
): void => {
|
): void => {
|
||||||
const target = relative(dir, originalsDir);
|
|
||||||
for (const name of readdirSync(dir)) {
|
for (const name of readdirSync(dir)) {
|
||||||
if (keep.has(name)) continue;
|
if (keep.has(name)) continue;
|
||||||
const path = join(dir, name);
|
const path = join(dir, name);
|
||||||
if (
|
if (linksToOriginal(path, root)) rmSync(path);
|
||||||
lstatSync(path).isSymbolicLink() &&
|
|
||||||
dirname(readlinkSync(path)) === target
|
|
||||||
) {
|
|
||||||
rmSync(path);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
// Remove the directories under `collectionsDir` that an earlier run wrote for
|
// Remove the directories under `collectionsDir` that an earlier run wrote for
|
||||||
// an album that is gone or renamed: a directory not named in `current` with a
|
// an album that is gone or renamed: a directory not named in `current` with a
|
||||||
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
|
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
|
||||||
// symlinks into originals/ are removed; if that leaves it empty, it and its
|
// symlinks to originals in `root` are removed; if that leaves it empty, it and
|
||||||
// JSON are deleted, otherwise both stay for what the user put there.
|
// its JSON are deleted, otherwise both stay for what the user put there.
|
||||||
const removeStaleAlbumDirs = (
|
const removeStaleAlbumDirs = (
|
||||||
collectionsDir: string,
|
collectionsDir: string,
|
||||||
current: Set<string>,
|
current: Set<string>,
|
||||||
originalsDir: string,
|
root: string,
|
||||||
): void => {
|
): void => {
|
||||||
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
|
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
|
||||||
if (!entry.isDirectory() || current.has(entry.name)) continue;
|
if (!entry.isDirectory() || current.has(entry.name)) continue;
|
||||||
@@ -334,7 +303,7 @@ const removeStaleAlbumDirs = (
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const dir = join(collectionsDir, entry.name);
|
const dir = join(collectionsDir, entry.name);
|
||||||
removeStaleLinks(dir, new Set(), originalsDir);
|
removeStaleLinks(dir, new Set(), root);
|
||||||
if (readdirSync(dir).length > 0) continue;
|
if (readdirSync(dir).length > 0) continue;
|
||||||
rmdirSync(dir);
|
rmdirSync(dir);
|
||||||
rmSync(jsonPath);
|
rmSync(jsonPath);
|
||||||
@@ -402,34 +371,48 @@ export const runBackup = async (
|
|||||||
log("Refreshing library...");
|
log("Refreshing library...");
|
||||||
await lib.refresh();
|
await lib.refresh();
|
||||||
|
|
||||||
const originalsDir = join(downloadDirectory, "originals");
|
|
||||||
const collectionsDir = join(downloadDirectory, "collections");
|
const collectionsDir = join(downloadDirectory, "collections");
|
||||||
const thumbnailsDir = join(downloadDirectory, "thumbnails");
|
const thumbnailsDir = join(downloadDirectory, "thumbnails");
|
||||||
mkdirSync(originalsDir, { recursive: true });
|
|
||||||
mkdirSync(collectionsDir, { recursive: true });
|
mkdirSync(collectionsDir, { recursive: true });
|
||||||
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
|
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
|
||||||
removeLeftoverTempFiles(originalsDir);
|
|
||||||
removeLeftoverTempFiles(thumbnailsDir);
|
removeLeftoverTempFiles(thumbnailsDir);
|
||||||
|
for (const dir of dateFolders(downloadDirectory)) {
|
||||||
|
removeLeftoverTempFiles(dir);
|
||||||
|
}
|
||||||
|
|
||||||
const ledgerPath = join(downloadDirectory, "failures.json");
|
const ledgerPath = join(downloadDirectory, "failures.json");
|
||||||
const ledger = loadLedger(ledgerPath);
|
const ledger = loadLedger(ledgerPath);
|
||||||
const now = Date.now();
|
const now = Date.now();
|
||||||
|
|
||||||
// Collections in scope, and the distinct files across them (a file shared
|
// Collections in scope, and the distinct files across them (a file shared
|
||||||
// by two albums is one original).
|
// by two albums is one original). Each file is the membership
|
||||||
|
// `representative` picks from all of its albums, in scope or not, so it is
|
||||||
|
// saved at the path `photo.savePath` names.
|
||||||
const allCollections = lib.listCollections();
|
const allCollections = lib.listCollections();
|
||||||
const collections = allCollections.filter((c) =>
|
const collections = allCollections.filter((c) =>
|
||||||
only ? only.has(c.name) : true,
|
only ? only.has(c.name) : true,
|
||||||
);
|
);
|
||||||
const collectionName = new Map<number, string>();
|
const collectionName = new Map<number, string>();
|
||||||
for (const c of collections) collectionName.set(c.id, c.name);
|
for (const c of allCollections) collectionName.set(c.id, c.name);
|
||||||
|
|
||||||
const distinct = new Map<number, EnteFile>();
|
const memberships = new Map<number, EnteFile[]>();
|
||||||
const filesByCollection = new Map<number, EnteFile[]>();
|
const filesByCollection = new Map<number, EnteFile[]>();
|
||||||
for (const c of collections) {
|
for (const c of allCollections) {
|
||||||
const files = lib.listFiles(c.id);
|
const files = lib.listFiles(c.id);
|
||||||
filesByCollection.set(c.id, files);
|
filesByCollection.set(c.id, files);
|
||||||
for (const f of files) if (!distinct.has(f.id)) distinct.set(f.id, f);
|
for (const f of files) {
|
||||||
|
const arr = memberships.get(f.id);
|
||||||
|
if (arr) arr.push(f);
|
||||||
|
else memberships.set(f.id, [f]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const distinct = new Map<number, EnteFile>();
|
||||||
|
for (const c of collections) {
|
||||||
|
for (const f of filesByCollection.get(c.id)!) {
|
||||||
|
if (!distinct.has(f.id)) {
|
||||||
|
distinct.set(f.id, representative(memberships.get(f.id)!));
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const errors: BackupError[] = [];
|
const errors: BackupError[] = [];
|
||||||
@@ -465,25 +448,22 @@ export const runBackup = async (
|
|||||||
failedThisRun.add(file.id);
|
failedThisRun.add(file.id);
|
||||||
};
|
};
|
||||||
|
|
||||||
// Phase 1: get the bytes. Fetch each pending original (and optional
|
// Phase 1: get the bytes. Put each pending original at its save path
|
||||||
// thumbnail) through the content cache/pools and place it under the backup
|
// through the content cache/pools, as `Photo.download()` does, and fetch
|
||||||
// tree; a present file is left as is.
|
// the optional thumbnails; a present file is left as is.
|
||||||
if (includeOriginals) {
|
if (includeOriginals) {
|
||||||
for (const [fileID, file] of distinct) {
|
for (const [fileID, file] of distinct) {
|
||||||
if (storedOriginal(originalsDir, file) !== undefined) {
|
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
|
||||||
skipped++;
|
skipped++;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const dest = join(originalsDir, nameInOriginals(file));
|
|
||||||
try {
|
try {
|
||||||
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
||||||
// A fetched original is written straight to `dest` (a live
|
// A fetched original is written straight to its save path (a
|
||||||
// photo beside it); only one that was already cached elsewhere
|
// live photo beside it); only one that was already cached
|
||||||
// is copied.
|
// elsewhere is copied.
|
||||||
await placeOriginal(
|
await placeOriginal(downloadDirectory, file, (dest) =>
|
||||||
file,
|
lib.original(fileID, dest),
|
||||||
dest,
|
|
||||||
await lib.original(fileID, dest),
|
|
||||||
);
|
);
|
||||||
downloaded++;
|
downloaded++;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
@@ -519,9 +499,10 @@ export const runBackup = async (
|
|||||||
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
||||||
// every present original (this repairs stale ones).
|
// every present original (this repairs stale ones).
|
||||||
if (includeOriginals) {
|
if (includeOriginals) {
|
||||||
for (const [fileID, file] of distinct) {
|
for (const file of distinct.values()) {
|
||||||
if (storedOriginal(originalsDir, file) !== undefined) {
|
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
|
||||||
writeSidecar(join(originalsDir, `${fileID}.json`), file);
|
const path = savePath(downloadDirectory, file);
|
||||||
|
writeSidecar(withExtension(path, ".json"), file);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -543,7 +524,11 @@ export const runBackup = async (
|
|||||||
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
||||||
);
|
);
|
||||||
try {
|
try {
|
||||||
removeStaleAlbumDirs(collectionsDir, new Set(dirNames), originalsDir);
|
removeStaleAlbumDirs(
|
||||||
|
collectionsDir,
|
||||||
|
new Set(dirNames),
|
||||||
|
downloadDirectory,
|
||||||
|
);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(`FAILED removing old album directories: ${errorMessage(err)}`);
|
log(`FAILED removing old album directories: ${errorMessage(err)}`);
|
||||||
}
|
}
|
||||||
@@ -553,13 +538,18 @@ export const runBackup = async (
|
|||||||
const colDir = join(collectionsDir, colDirName);
|
const colDir = join(collectionsDir, colDirName);
|
||||||
mkdirSync(colDir, { recursive: true });
|
mkdirSync(colDir, { recursive: true });
|
||||||
|
|
||||||
|
// Every album links the one original, saved from the file's entry in
|
||||||
|
// `distinct`.
|
||||||
const files = filesByCollection.get(c.id) ?? [];
|
const files = filesByCollection.get(c.id) ?? [];
|
||||||
const links = files.flatMap((f) =>
|
const links = files.flatMap((f) =>
|
||||||
linksFor(f, storedOriginal(originalsDir, f)),
|
linksFor(
|
||||||
|
f,
|
||||||
|
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
const linkNames = uniqueNames(links, true);
|
const linkNames = uniqueNames(links, true);
|
||||||
try {
|
try {
|
||||||
removeStaleLinks(colDir, new Set(linkNames), originalsDir);
|
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
|
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -361,8 +361,8 @@ const openPart = async (
|
|||||||
// written unpacked: each part is named `destination` with the extension
|
// written unpacked: each part is named `destination` with the extension
|
||||||
// replaced by its own entry's, and the two must differ ignoring case. When the
|
// replaced by its own entry's, and the two must differ ignoring case. When the
|
||||||
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
||||||
// part's own bytes. Only then is whatever was at `destination` removed and the
|
// part's own bytes. Only then are the image, then the video, renamed into
|
||||||
// image, then the video, renamed into place; on any failure neither is stored.
|
// place; on any failure neither is stored.
|
||||||
//
|
//
|
||||||
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
||||||
// written as it decompresses and never held, and the ZIP is refused once the
|
// written as it decompresses and never held, and the ZIP is refused once the
|
||||||
@@ -486,7 +486,6 @@ const decryptLivePhoto = async (
|
|||||||
await part.handle.sync();
|
await part.handle.sync();
|
||||||
await part.handle.close();
|
await part.handle.close();
|
||||||
}
|
}
|
||||||
await rm(destination, { force: true });
|
|
||||||
await rename(image.tmpPath, path);
|
await rename(image.tmpPath, path);
|
||||||
try {
|
try {
|
||||||
await rename(video.tmpPath, videoPath);
|
await rename(video.tmpPath, videoPath);
|
||||||
@@ -570,8 +569,7 @@ const fetchAndDecrypt = async (
|
|||||||
}, api.getRetryOptions());
|
}, api.getRetryOptions());
|
||||||
|
|
||||||
// Write `file`'s original to `outPath`. A live photo is written as its image
|
// Write `file`'s original to `outPath`. A live photo is written as its image
|
||||||
// and its video beside `outPath` instead, and whatever was at `outPath` is
|
// and its video beside `outPath` instead (see `decryptLivePhoto`).
|
||||||
// removed (see `decryptLivePhoto`).
|
|
||||||
export const downloadFile = async (
|
export const downloadFile = async (
|
||||||
api: ApiClient,
|
api: ApiClient,
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
|
|||||||
+80
-31
@@ -1,19 +1,19 @@
|
|||||||
// EXIF in an original's bytes, read with exifreader, which reads it from JPEG,
|
// EXIF in an original's bytes, read with exifreader, which reads it from JPEG,
|
||||||
// HEIC/HEIF, AVIF, PNG, WebP and the other image formats it supports.
|
// HEIC/HEIF, AVIF, PNG, WebP and the other image formats it supports.
|
||||||
// `backup-metadata --exif` records every EXIF tag it finds except the
|
// `backup-metadata --exif` records every EXIF tag it finds except the
|
||||||
// thumbnail's; `Photo.exif()` returns the common fields picked from them here.
|
// thumbnail's. `Photo.exif()` returns every tag, the thumbnail's included, and
|
||||||
|
// `Photo`'s typed methods return the common fields picked from them here.
|
||||||
|
|
||||||
import ExifReader, { type ExpandedTags } from "exifreader";
|
import ExifReader, { type ExpandedTags } from "exifreader";
|
||||||
|
|
||||||
// The EXIF tags in `bytes` (`exif`), the GPS position exifreader computes from
|
// The EXIF tags in `bytes` (`exif`), the embedded thumbnail's tags
|
||||||
// them (`gps`), and where the EXIF block lies in `bytes` (`metadataRange`).
|
// (`Thumbnail`), and where the EXIF block lies in `bytes` (`metadataRange`).
|
||||||
// Undefined when exifreader cannot read the file at all, such as a video. An
|
// Undefined when exifreader cannot read the file at all, such as a video. An
|
||||||
// EXIF block it finds but reads no tag from comes back as an empty `exif`.
|
// EXIF block it finds but reads no tag from comes back as an empty `exif`. A
|
||||||
// `exif` holds every tag except the thumbnail's; a tag exifreader has no name
|
// tag exifreader has no name for is keyed `undefined-<tag number>`. Each tag's
|
||||||
// for is keyed `undefined-<tag number>`. Each tag's `computed` holds its value
|
// `computed` holds its value as a string or number, or as an array of them for
|
||||||
// as a string or number, or as an array of them for a tag with several values,
|
// a tag with several values, such as `GPSLatitude`'s `[40, 26, 46]`. A
|
||||||
// such as `GPSLatitude`'s `[40, 26, 46]`. A fraction with a zero denominator
|
// fraction with a zero denominator computes to null.
|
||||||
// computes to null.
|
|
||||||
export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
|
export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
|
||||||
try {
|
try {
|
||||||
return ExifReader.loadView(
|
return ExifReader.loadView(
|
||||||
@@ -23,7 +23,7 @@ export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
|
|||||||
computed: true,
|
computed: true,
|
||||||
includeOffsets: true,
|
includeOffsets: true,
|
||||||
includeUnknown: true,
|
includeUnknown: true,
|
||||||
includeTags: { exif: true, gps: true },
|
includeTags: { exif: true, thumbnail: true },
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
} catch {
|
} catch {
|
||||||
@@ -31,7 +31,35 @@ export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
// The common EXIF fields of an original. Each is absent when the file lacks it.
|
// Every EXIF tag of an original, keyed by name, each as exifreader decodes it
|
||||||
|
// (see `readExifTags`). The embedded thumbnail's own tags are under
|
||||||
|
// `Thumbnail`, so its `Orientation` or `ImageWidth` cannot hide the main
|
||||||
|
// image's.
|
||||||
|
export type ExifTags = Omit<NonNullable<ExpandedTags["exif"]>, "Thumbnail"> & {
|
||||||
|
Thumbnail?: Omit<
|
||||||
|
NonNullable<ExpandedTags["Thumbnail"]>,
|
||||||
|
"type" | "image" | "base64"
|
||||||
|
>;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Every EXIF tag in `bytes`: `{}` when the file has no EXIF, exifreader cannot
|
||||||
|
// read its EXIF, or it is not an image exifreader reads.
|
||||||
|
export const readAllExifTags = (bytes: Uint8Array): ExifTags => {
|
||||||
|
const tags = readExifTags(bytes);
|
||||||
|
if (!tags?.Thumbnail) return tags?.exif ?? {};
|
||||||
|
// exifreader puts the thumbnail's JPEG image beside its tags, as `type`,
|
||||||
|
// `image` and `base64`. The image is not a tag, so it is left out.
|
||||||
|
const {
|
||||||
|
type: _type,
|
||||||
|
image: _image,
|
||||||
|
base64: _base64,
|
||||||
|
...thumbnail
|
||||||
|
} = tags.Thumbnail;
|
||||||
|
return { ...tags.exif, Thumbnail: thumbnail };
|
||||||
|
};
|
||||||
|
|
||||||
|
// The common EXIF fields of an original, one for each of `Photo`'s typed
|
||||||
|
// methods. Each is absent when the file lacks it.
|
||||||
export interface PhotoExif {
|
export interface PhotoExif {
|
||||||
make?: string;
|
make?: string;
|
||||||
model?: string;
|
model?: string;
|
||||||
@@ -80,30 +108,51 @@ const asDate = (v: unknown): Date | undefined => {
|
|||||||
return Number.isNaN(date.getTime()) ? undefined : date;
|
return Number.isNaN(date.getTime()) ? undefined : date;
|
||||||
};
|
};
|
||||||
|
|
||||||
// The common fields of an original's EXIF: `{}` when the file has no EXIF,
|
// GPSLatitude and GPSLongitude hold degrees, minutes and seconds, computed as
|
||||||
// exifreader cannot read its EXIF, or it is not an image exifreader reads.
|
// three numbers. This is them in decimal degrees, negative when `ref`, the
|
||||||
export const readPhotoExif = (bytes: Uint8Array): PhotoExif => {
|
// GPSLatitudeRef or GPSLongitudeRef tag, is `negativeRef` ("S" or "W").
|
||||||
const tags = readExifTags(bytes);
|
// Without that tag the hemisphere is unknown, so it is undefined.
|
||||||
const exif = tags?.exif;
|
const asDegrees = (
|
||||||
const gps = tags?.gps;
|
dms: unknown,
|
||||||
const altitude = asNumber(exif?.GPSAltitude?.computed);
|
ref: unknown,
|
||||||
|
negativeRef: string,
|
||||||
|
): number | undefined => {
|
||||||
|
if (!Array.isArray(dms) || ref === undefined) return undefined;
|
||||||
|
const [d, m, s] = dms.map(asNumber);
|
||||||
|
if (d === undefined || m === undefined || s === undefined) return undefined;
|
||||||
|
const degrees = d + m / 60 + s / 3600;
|
||||||
|
return ref === negativeRef ? -degrees : degrees;
|
||||||
|
};
|
||||||
|
|
||||||
|
// The common fields picked from an original's EXIF tags, `readAllExifTags`'s
|
||||||
|
// result: `{}` when there are none.
|
||||||
|
export const readPhotoExif = (tags: ExifTags): PhotoExif => {
|
||||||
|
const altitude = asNumber(tags.GPSAltitude?.computed);
|
||||||
const fields: PhotoExif = {
|
const fields: PhotoExif = {
|
||||||
make: asString(exif?.Make?.computed),
|
make: asString(tags.Make?.computed),
|
||||||
model: asString(exif?.Model?.computed),
|
model: asString(tags.Model?.computed),
|
||||||
lensModel: asString(exif?.LensModel?.computed),
|
lensModel: asString(tags.LensModel?.computed),
|
||||||
dateTimeOriginal: asDate(exif?.DateTimeOriginal?.computed),
|
dateTimeOriginal: asDate(tags.DateTimeOriginal?.computed),
|
||||||
offsetTimeOriginal: asString(exif?.OffsetTimeOriginal?.computed),
|
offsetTimeOriginal: asString(tags.OffsetTimeOriginal?.computed),
|
||||||
exposureTime: asNumber(exif?.ExposureTime?.computed),
|
exposureTime: asNumber(tags.ExposureTime?.computed),
|
||||||
fNumber: asNumber(exif?.FNumber?.computed),
|
fNumber: asNumber(tags.FNumber?.computed),
|
||||||
// Only when the tag holds a single number, as most cameras write it.
|
// Only when the tag holds a single number, as most cameras write it.
|
||||||
iso: asNumber(exif?.ISOSpeedRatings?.computed),
|
iso: asNumber(tags.ISOSpeedRatings?.computed),
|
||||||
focalLength: asNumber(exif?.FocalLength?.computed),
|
focalLength: asNumber(tags.FocalLength?.computed),
|
||||||
orientation: asNumber(exif?.Orientation?.computed),
|
orientation: asNumber(tags.Orientation?.computed),
|
||||||
gpsLatitude: asNumber(gps?.Latitude),
|
gpsLatitude: asDegrees(
|
||||||
gpsLongitude: asNumber(gps?.Longitude),
|
tags.GPSLatitude?.computed,
|
||||||
|
tags.GPSLatitudeRef?.computed,
|
||||||
|
"S",
|
||||||
|
),
|
||||||
|
gpsLongitude: asDegrees(
|
||||||
|
tags.GPSLongitude?.computed,
|
||||||
|
tags.GPSLongitudeRef?.computed,
|
||||||
|
"W",
|
||||||
|
),
|
||||||
// A GPSAltitudeRef of 1 means the altitude is below sea level.
|
// A GPSAltitudeRef of 1 means the altitude is below sea level.
|
||||||
gpsAltitude:
|
gpsAltitude:
|
||||||
altitude !== undefined && exif?.GPSAltitudeRef?.value === 1
|
altitude !== undefined && tags.GPSAltitudeRef?.value === 1
|
||||||
? -altitude
|
? -altitude
|
||||||
: altitude,
|
: altitude,
|
||||||
};
|
};
|
||||||
|
|||||||
+6
-3
@@ -1,5 +1,7 @@
|
|||||||
// package.json is the one place the version is written. tsc copies it to
|
// A build reports the version script/build stamps into dist/package.json;
|
||||||
// dist/package.json, so this path resolves from source and from dist/src/.
|
// package.json's own version is reported only when running from source. tsc
|
||||||
|
// copies package.json to dist/package.json, so this path resolves from source
|
||||||
|
// and from dist/src/.
|
||||||
import pkg from "../package.json" with { type: "json" };
|
import pkg from "../package.json" with { type: "json" };
|
||||||
|
|
||||||
export const VERSION: string = pkg.version;
|
export const VERSION: string = pkg.version;
|
||||||
@@ -53,6 +55,7 @@ export {
|
|||||||
type PhotoFilter,
|
type PhotoFilter,
|
||||||
type TimelineGroup,
|
type TimelineGroup,
|
||||||
type GroupBy,
|
type GroupBy,
|
||||||
|
type SavePathLookup,
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
type ContentResult,
|
type ContentResult,
|
||||||
type ContentEvent,
|
type ContentEvent,
|
||||||
@@ -84,7 +87,7 @@ export type {
|
|||||||
LibrarySnapshot,
|
LibrarySnapshot,
|
||||||
LibraryChange,
|
LibraryChange,
|
||||||
} from "./library/records.js";
|
} from "./library/records.js";
|
||||||
export type { PhotoExif } from "./exif.js";
|
export type { ExifTags, PhotoExif } from "./exif.js";
|
||||||
export { decryptCollection, decryptFile } from "./model/index.js";
|
export { decryptCollection, decryptFile } from "./model/index.js";
|
||||||
export { downloadFile, downloadThumbnail } from "./download/index.js";
|
export { downloadFile, downloadThumbnail } from "./download/index.js";
|
||||||
export type {
|
export type {
|
||||||
|
|||||||
+157
-107
@@ -23,19 +23,19 @@
|
|||||||
// original with no recorded hash is stored unchecked, as the upstream client
|
// original with no recorded hash is stored unchecked, as the upstream client
|
||||||
// does; thumbnails have none. On top of that this module refuses to record a
|
// does; thumbnails have none. On top of that this module refuses to record a
|
||||||
// stored file that came out empty.
|
// stored file that came out empty.
|
||||||
|
//
|
||||||
|
// It also names each original's save path under the download directory
|
||||||
|
// (`savePath`), where `Photo.download()` and `lib.backup()` put it. A copy in
|
||||||
|
// the cache does not count as saved there, but is copied there rather than
|
||||||
|
// fetched again.
|
||||||
|
|
||||||
import {
|
import { existsSync, readFileSync, statSync } from "node:fs";
|
||||||
closeSync,
|
|
||||||
existsSync,
|
|
||||||
openSync,
|
|
||||||
readFileSync,
|
|
||||||
readSync,
|
|
||||||
statSync,
|
|
||||||
} from "node:fs";
|
|
||||||
import {
|
import {
|
||||||
chmod,
|
chmod,
|
||||||
|
copyFile,
|
||||||
mkdir,
|
mkdir,
|
||||||
readdir,
|
readdir,
|
||||||
|
rename,
|
||||||
rm,
|
rm,
|
||||||
stat,
|
stat,
|
||||||
statfs,
|
statfs,
|
||||||
@@ -47,13 +47,15 @@ import type { ApiClient } from "../api/client.js";
|
|||||||
import {
|
import {
|
||||||
downloadFile,
|
downloadFile,
|
||||||
downloadThumbnail,
|
downloadThumbnail,
|
||||||
|
fsyncPath,
|
||||||
type ProgressCallback,
|
type ProgressCallback,
|
||||||
removeLeftoverTempFiles,
|
removeLeftoverTempFiles,
|
||||||
writeAtomic,
|
writeAtomic,
|
||||||
} from "../download/index.js";
|
} from "../download/index.js";
|
||||||
import { safeExtension } from "../filename.js";
|
import { safeExtension, withExtension } from "../filename.js";
|
||||||
import type { EnteFile } from "../model/types.js";
|
import type { EnteFile } from "../model/types.js";
|
||||||
import type { Priority, RequestPools } from "./pools.js";
|
import type { Priority, RequestPools } from "./pools.js";
|
||||||
|
import { takenAtOf } from "./records.js";
|
||||||
|
|
||||||
const DIR_MODE = 0o700;
|
const DIR_MODE = 0o700;
|
||||||
const FILE_MODE = 0o600;
|
const FILE_MODE = 0o600;
|
||||||
@@ -108,11 +110,9 @@ export interface ContentOptions {
|
|||||||
export interface PhotoContent {
|
export interface PhotoContent {
|
||||||
original(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
original(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
||||||
thumbnail(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
thumbnail(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
||||||
// Where a backup stores the original, whether or not it is there yet. For
|
// Put the original at the save path of `file`, the copy the `Photo` holds,
|
||||||
// a live photo not yet stored, it carries the title's extension, and the
|
// and return it there.
|
||||||
// backup may store the image under a different one.
|
download(file: EnteFile): Promise<ContentResult>;
|
||||||
savePath(fileID: number): string | undefined;
|
|
||||||
isLocal(fileID: number): boolean;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface EnsureResult {
|
export interface EnsureResult {
|
||||||
@@ -199,10 +199,10 @@ export interface ContentCacheOptions {
|
|||||||
pools: RequestPools;
|
pools: RequestPools;
|
||||||
source: ContentSource;
|
source: ContentSource;
|
||||||
cacheDirectory: string;
|
cacheDirectory: string;
|
||||||
// The backup destination (issue-level `downloadDirectory`). An original
|
// The root of the save paths. An original already stored at its save path
|
||||||
// already stored there by a backup counts as present, so the cache serves
|
// counts as present, so the cache serves it rather than fetching a second
|
||||||
// it rather than fetching a second copy.
|
// copy.
|
||||||
downloadDirectory?: string;
|
downloadDirectory: string;
|
||||||
// Resolve any membership of a file; every membership shares the underlying
|
// Resolve any membership of a file; every membership shares the underlying
|
||||||
// content key, so any one decrypts the same bytes.
|
// content key, so any one decrypts the same bytes.
|
||||||
getFile: (fileID: number) => EnteFile | undefined;
|
getFile: (fileID: number) => EnteFile | undefined;
|
||||||
@@ -229,11 +229,27 @@ class AbortDrop extends Error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The name of a file's original in originals/: `<fileID><ext>`, the extension
|
// The name of a file's original in the cache's originals/: `<fileID><ext>`, the
|
||||||
// taken from the title (or `.bin`). A backup names its originals the same way.
|
// extension taken from the title (or `.bin`).
|
||||||
export const nameInOriginals = (file: EnteFile): string =>
|
export const nameInOriginals = (file: EnteFile): string =>
|
||||||
`${file.id}${safeExtension(file.metadata.title)}`;
|
`${file.id}${safeExtension(file.metadata.title)}`;
|
||||||
|
|
||||||
|
const pad = (n: number): string => String(n).padStart(2, "0");
|
||||||
|
|
||||||
|
// Where the original of `file` is saved under `root`:
|
||||||
|
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`, dated by the photo's
|
||||||
|
// `takenAt` in this machine's time zone, with the extension taken from the
|
||||||
|
// title (or `.bin`). A live photo is stored as its image and its video beside
|
||||||
|
// this path, each with the extension found inside the live photo.
|
||||||
|
export const savePath = (root: string, file: EnteFile): string => {
|
||||||
|
const taken = new Date(takenAtOf(file));
|
||||||
|
const year = String(taken.getFullYear());
|
||||||
|
const month = `${year}-${pad(taken.getMonth() + 1)}`;
|
||||||
|
const day = `${month}-${pad(taken.getDate())}`;
|
||||||
|
const ext = safeExtension(file.metadata.title);
|
||||||
|
return join(root, year, month, day, `${day}.${file.id}${ext}`);
|
||||||
|
};
|
||||||
|
|
||||||
// The fileID a cache filename encodes, or undefined when the name is not one
|
// The fileID a cache filename encodes, or undefined when the name is not one
|
||||||
// the cache writes (`<digits><ext>`).
|
// the cache writes (`<digits><ext>`).
|
||||||
const fileIDFromName = (name: string): number | undefined => {
|
const fileIDFromName = (name: string): number | undefined => {
|
||||||
@@ -258,49 +274,31 @@ const fileSize = (path: string): number | undefined => {
|
|||||||
const hasContent = (path: string | undefined): boolean =>
|
const hasContent = (path: string | undefined): boolean =>
|
||||||
path !== undefined && (fileSize(path) ?? 0) > 0;
|
path !== undefined && (fileSize(path) ?? 0) > 0;
|
||||||
|
|
||||||
// Whether the file at `path` begins as a ZIP does, with `PK\x03\x04`. False
|
|
||||||
// when it cannot be read.
|
|
||||||
const isZip = (path: string): boolean => {
|
|
||||||
try {
|
|
||||||
const fd = openSync(path, "r");
|
|
||||||
try {
|
|
||||||
const head = Buffer.alloc(4);
|
|
||||||
return (
|
|
||||||
readSync(fd, head, 0, 4, 0) === 4 &&
|
|
||||||
head.toString("latin1") === "PK\x03\x04"
|
|
||||||
);
|
|
||||||
} finally {
|
|
||||||
closeSync(fd);
|
|
||||||
}
|
|
||||||
} catch {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// A live photo's image and video are named with the extensions from inside its
|
// A live photo's image and video are named with the extensions from inside its
|
||||||
// ZIP, so their names alone do not say which is which. Wherever the cache or a
|
// ZIP, so their names alone do not say which is which. Wherever the cache or a
|
||||||
// backup stores one, a JSON file of this name beside them names both.
|
// save path stores one, a JSON file of this name beside them names both.
|
||||||
const livePhotoJSONName = (fileID: number): string =>
|
// `name` is the original's name without its extension: `<fileID>` in the
|
||||||
`${fileID}.livephoto.json`;
|
// cache, `YYYY-MM-DD.<fileID>` at the save path.
|
||||||
|
const livePhotoJSONName = (name: string): string => `${name}.livephoto.json`;
|
||||||
|
|
||||||
// The image and video that the live photo's JSON file in `dir` names, or
|
// The image and video that the live photo's JSON file in `dir` names, or
|
||||||
// undefined when there is none. Only names of the form the cache writes are
|
// undefined when there is none. Only `name` with an extension is taken, so the
|
||||||
// taken, so the file cannot point outside `dir`.
|
// file cannot point outside `dir`.
|
||||||
const readLivePhotoJSON = (
|
const readLivePhotoJSON = (
|
||||||
dir: string,
|
dir: string,
|
||||||
fileID: number,
|
name: string,
|
||||||
): { path: string; videoPath: string } | undefined => {
|
): { path: string; videoPath: string } | undefined => {
|
||||||
const valid = (name: unknown): name is string =>
|
const valid = (part: unknown): part is string =>
|
||||||
typeof name === "string" && name === `${fileID}${safeExtension(name)}`;
|
typeof part === "string" && part === `${name}${safeExtension(part)}`;
|
||||||
try {
|
try {
|
||||||
const { image, video } = JSON.parse(
|
const { image, video } = JSON.parse(
|
||||||
readFileSync(join(dir, livePhotoJSONName(fileID)), "utf-8"),
|
readFileSync(join(dir, livePhotoJSONName(name)), "utf-8"),
|
||||||
);
|
);
|
||||||
if (valid(image) && valid(video)) {
|
if (valid(image) && valid(video)) {
|
||||||
return { path: join(dir, image), videoPath: join(dir, video) };
|
return { path: join(dir, image), videoPath: join(dir, video) };
|
||||||
}
|
}
|
||||||
} catch {
|
} catch {
|
||||||
// No such file, or not one the cache wrote.
|
// No such file, or not one quak wrote.
|
||||||
}
|
}
|
||||||
return undefined;
|
return undefined;
|
||||||
};
|
};
|
||||||
@@ -308,11 +306,11 @@ const readLivePhotoJSON = (
|
|||||||
// Write the JSON file naming a live photo's image and video, both in `dir`.
|
// Write the JSON file naming a live photo's image and video, both in `dir`.
|
||||||
export const writeLivePhotoJSON = (
|
export const writeLivePhotoJSON = (
|
||||||
dir: string,
|
dir: string,
|
||||||
fileID: number,
|
name: string,
|
||||||
stored: { path: string; videoPath: string },
|
stored: { path: string; videoPath: string },
|
||||||
): Promise<void> =>
|
): Promise<void> =>
|
||||||
writeAtomic(
|
writeAtomic(
|
||||||
join(dir, livePhotoJSONName(fileID)),
|
join(dir, livePhotoJSONName(name)),
|
||||||
new TextEncoder().encode(
|
new TextEncoder().encode(
|
||||||
JSON.stringify({
|
JSON.stringify({
|
||||||
image: basename(stored.path),
|
image: basename(stored.path),
|
||||||
@@ -321,17 +319,19 @@ export const writeLivePhotoJSON = (
|
|||||||
),
|
),
|
||||||
);
|
);
|
||||||
|
|
||||||
// The original of `file` as the cache or a backup stored it in `dir`, when all
|
// The original of `file` as stored in `dir` under `name` (without its
|
||||||
// of it is there: `<fileID><ext>`, or a live photo's image and video.
|
// extension), when all of it is there: `<name><ext>`, or a live photo's image
|
||||||
|
// and video.
|
||||||
export const storedOriginal = (
|
export const storedOriginal = (
|
||||||
dir: string,
|
dir: string,
|
||||||
|
name: string,
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
): { path: string; videoPath?: string } | undefined => {
|
): { path: string; videoPath?: string } | undefined => {
|
||||||
if (file.metadata.fileType !== "livePhoto") {
|
if (file.metadata.fileType !== "livePhoto") {
|
||||||
const path = join(dir, nameInOriginals(file));
|
const path = join(dir, `${name}${safeExtension(file.metadata.title)}`);
|
||||||
return hasContent(path) ? { path } : undefined;
|
return hasContent(path) ? { path } : undefined;
|
||||||
}
|
}
|
||||||
const stored = readLivePhotoJSON(dir, file.id);
|
const stored = readLivePhotoJSON(dir, name);
|
||||||
return stored !== undefined &&
|
return stored !== undefined &&
|
||||||
hasContent(stored.path) &&
|
hasContent(stored.path) &&
|
||||||
hasContent(stored.videoPath)
|
hasContent(stored.videoPath)
|
||||||
@@ -339,10 +339,76 @@ export const storedOriginal = (
|
|||||||
: undefined;
|
: undefined;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// The original of `file` as stored at its save path under `root`, when all of
|
||||||
|
// it is there.
|
||||||
|
export const storedAtSavePath = (
|
||||||
|
root: string,
|
||||||
|
file: EnteFile,
|
||||||
|
): { path: string; videoPath?: string } | undefined => {
|
||||||
|
const path = savePath(root, file);
|
||||||
|
return storedOriginal(dirname(path), basename(path, extname(path)), file);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
||||||
|
// `dest` appears only once it is whole ("present means complete"). As in the
|
||||||
|
// download writer, the temp file is fsynced before the rename and the directory
|
||||||
|
// after it, so a power cut cannot leave a correctly named but short original.
|
||||||
|
// The temp name carries this process's ID so a later run can tell a leftover
|
||||||
|
// from a copy still in progress (see `removeLeftoverTempFiles`).
|
||||||
|
export const copyAtomic = async (src: string, dest: string): Promise<void> => {
|
||||||
|
if (src === dest) return;
|
||||||
|
const tmp = join(
|
||||||
|
dirname(dest),
|
||||||
|
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
|
||||||
|
.toString(36)
|
||||||
|
.slice(2)}.tmp`,
|
||||||
|
);
|
||||||
|
try {
|
||||||
|
await copyFile(src, tmp);
|
||||||
|
await fsyncPath(tmp);
|
||||||
|
// `rename` replaces the destination's directory entry: an existing
|
||||||
|
// symlink at `dest` is replaced, not followed, and the new file has
|
||||||
|
// the temp file's permissions (copied from `src`).
|
||||||
|
await rename(tmp, dest);
|
||||||
|
await fsyncPath(dirname(dest));
|
||||||
|
} finally {
|
||||||
|
await rm(tmp, { force: true });
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Put the original of `file` at its save path under `root`, creating its
|
||||||
|
// folders. `get` is given the save path and returns where the original is: a
|
||||||
|
// fetch writes it there, and a copy the cache holds is copied there. A live
|
||||||
|
// photo's image and video go beside the save path, each with its own
|
||||||
|
// extension: when they came from the cache they are copied. Then the JSON file
|
||||||
|
// naming them is written, which is what makes the live photo count as stored.
|
||||||
|
export const placeOriginal = async (
|
||||||
|
root: string,
|
||||||
|
file: EnteFile,
|
||||||
|
get: (dest: string) => Promise<{ path: string; videoPath?: string }>,
|
||||||
|
): Promise<{ path: string; videoPath?: string }> => {
|
||||||
|
const dest = savePath(root, file);
|
||||||
|
await mkdir(dirname(dest), { recursive: true });
|
||||||
|
const got = await get(dest);
|
||||||
|
if (got.videoPath === undefined) {
|
||||||
|
await copyAtomic(got.path, dest);
|
||||||
|
return { path: dest };
|
||||||
|
}
|
||||||
|
const path = withExtension(dest, extname(got.path));
|
||||||
|
const videoPath = withExtension(dest, extname(got.videoPath));
|
||||||
|
if (got.path !== path) {
|
||||||
|
await copyAtomic(got.path, path);
|
||||||
|
await copyAtomic(got.videoPath, videoPath);
|
||||||
|
}
|
||||||
|
const name = basename(dest, extname(dest));
|
||||||
|
await writeLivePhotoJSON(dirname(dest), name, { path, videoPath });
|
||||||
|
return { path, videoPath };
|
||||||
|
};
|
||||||
|
|
||||||
export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||||
private readonly pools: RequestPools;
|
private readonly pools: RequestPools;
|
||||||
private readonly source: ContentSource;
|
private readonly source: ContentSource;
|
||||||
private readonly downloadDirectory?: string;
|
private readonly downloadDirectory: string;
|
||||||
private readonly getFile: (fileID: number) => EnteFile | undefined;
|
private readonly getFile: (fileID: number) => EnteFile | undefined;
|
||||||
private readonly originalsDir: string;
|
private readonly originalsDir: string;
|
||||||
private readonly thumbnailsDir: string;
|
private readonly thumbnailsDir: string;
|
||||||
@@ -440,32 +506,23 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Where a backup to the download directory stores the file's original,
|
// Put the original at the save path of `file` under the download directory
|
||||||
// whether or not it is there yet: for a live photo already stored, its
|
// and return it there. `file` is the copy the `Photo` holds, so the path is
|
||||||
// image. For a live photo not yet stored, it carries the title's
|
// the one its `savePath` names, even after a refresh changed the date. One
|
||||||
// extension, and the backup may store the image under a different one.
|
// already stored there is returned as it is; one the cache holds is copied
|
||||||
// Undefined with no download directory.
|
// from it; any other is fetched straight to the save path, with no copy
|
||||||
savePath(fileID: number): string | undefined {
|
// left in the cache.
|
||||||
const file = this.getFile(fileID);
|
async download(file: EnteFile): Promise<ContentResult> {
|
||||||
if (this.downloadDirectory === undefined || file === undefined)
|
const root = this.downloadDirectory;
|
||||||
return undefined;
|
const saved =
|
||||||
const dir = join(this.downloadDirectory, "originals");
|
storedAtSavePath(root, file) ??
|
||||||
return (
|
(await placeOriginal(root, file, (dest) =>
|
||||||
storedOriginal(dir, file)?.path ?? join(dir, nameInOriginals(file))
|
this.backupOriginal(file.id, dest),
|
||||||
);
|
));
|
||||||
|
return { ...saved, bytes: fileSize(saved.path) ?? 0 };
|
||||||
}
|
}
|
||||||
|
|
||||||
// Whether the whole original is in the download directory, as a backup
|
// Get an original for a save path. One not present anywhere is written
|
||||||
// stores it. A copy only in the cache does not count.
|
|
||||||
isLocal(fileID: number): boolean {
|
|
||||||
const file = this.getFile(fileID);
|
|
||||||
if (this.downloadDirectory === undefined || file === undefined)
|
|
||||||
return false;
|
|
||||||
const dir = join(this.downloadDirectory, "originals");
|
|
||||||
return storedOriginal(dir, file) !== undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get an original for a backup. One not present anywhere is written
|
|
||||||
// straight to `destination` and recorded there, so no second copy lands
|
// straight to `destination` and recorded there, so no second copy lands
|
||||||
// in the cache; one already present is returned where it is.
|
// in the cache; one already present is returned where it is.
|
||||||
async backupOriginal(
|
async backupOriginal(
|
||||||
@@ -630,17 +687,16 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
await this.touch(cached.path);
|
await this.touch(cached.path);
|
||||||
return { ...cached, bytes: size, cached: true };
|
return { ...cached, bytes: size, cached: true };
|
||||||
}
|
}
|
||||||
// A recorded file that has since gone, or a live photo an earlier
|
// A recorded file that has since gone re-fetches below. So does a
|
||||||
// version stored as one ZIP, re-fetches below.
|
// live photo recorded with no video: the cache opened before the
|
||||||
|
// library's records said it is a live photo, while its image and
|
||||||
|
// video had no JSON file beside them yet.
|
||||||
known.delete(fileID);
|
known.delete(fileID);
|
||||||
}
|
}
|
||||||
|
|
||||||
// An original a backup already stored counts as present.
|
// An original already stored at its save path counts as present.
|
||||||
if (kind === "original" && this.downloadDirectory !== undefined) {
|
if (kind === "original") {
|
||||||
const stored = storedOriginal(
|
const stored = storedAtSavePath(this.downloadDirectory, file);
|
||||||
join(this.downloadDirectory, "originals"),
|
|
||||||
file,
|
|
||||||
);
|
|
||||||
if (stored !== undefined) {
|
if (stored !== undefined) {
|
||||||
this.originals.set(fileID, stored);
|
this.originals.set(fileID, stored);
|
||||||
return {
|
return {
|
||||||
@@ -676,7 +732,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
? this.beginOriginalWrite(fileID)
|
? this.beginOriginalWrite(fileID)
|
||||||
: null;
|
: null;
|
||||||
try {
|
try {
|
||||||
const stored = await this.download(
|
const stored = await this.fetchInto(
|
||||||
file,
|
file,
|
||||||
dest,
|
dest,
|
||||||
kind,
|
kind,
|
||||||
@@ -691,12 +747,12 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// A backup records its own live photos.
|
// `placeOriginal` records a live photo it saves.
|
||||||
if (
|
if (
|
||||||
stored.videoPath !== undefined &&
|
stored.videoPath !== undefined &&
|
||||||
opts?.destination === undefined
|
opts?.destination === undefined
|
||||||
) {
|
) {
|
||||||
await writeLivePhotoJSON(dir, fileID, {
|
await writeLivePhotoJSON(dir, String(fileID), {
|
||||||
path: stored.path,
|
path: stored.path,
|
||||||
videoPath: stored.videoPath,
|
videoPath: stored.videoPath,
|
||||||
});
|
});
|
||||||
@@ -719,7 +775,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
|
|
||||||
// Fetch into `destination`, returning where the bytes landed: there, or
|
// Fetch into `destination`, returning where the bytes landed: there, or
|
||||||
// for a live photo, its image and video beside it.
|
// for a live photo, its image and video beside it.
|
||||||
private async download(
|
private async fetchInto(
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
destination: string,
|
destination: string,
|
||||||
kind: Kind,
|
kind: Kind,
|
||||||
@@ -744,10 +800,10 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
await utimes(path, now, now).catch(() => undefined);
|
await utimes(path, now, now).catch(() => undefined);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Every stored original that lives under `originalsDir` (a backup-directory
|
// Every stored original that lives under `originalsDir` (a save-path hit
|
||||||
// hit recorded in the map is excluded), with its size and mtime; a live
|
// recorded in the map is excluded), with its size and mtime; a live
|
||||||
// photo's size includes its video. Entries whose file has vanished are
|
// photo's size includes its video. Entries whose file has vanished are
|
||||||
// dropped from the map. Backups and thumbnails are never counted.
|
// dropped from the map. Save paths and thumbnails are never counted.
|
||||||
private async measureOriginals(): Promise<{
|
private async measureOriginals(): Promise<{
|
||||||
entries: {
|
entries: {
|
||||||
fileID: number;
|
fileID: number;
|
||||||
@@ -853,7 +909,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
await rm(
|
await rm(
|
||||||
join(
|
join(
|
||||||
this.originalsDir,
|
this.originalsDir,
|
||||||
livePhotoJSONName(e.fileID),
|
livePhotoJSONName(String(e.fileID)),
|
||||||
),
|
),
|
||||||
{ force: true },
|
{ force: true },
|
||||||
);
|
);
|
||||||
@@ -903,20 +959,14 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
if (id === undefined || !existsSync(path)) continue;
|
if (id === undefined || !existsSync(path)) continue;
|
||||||
// A live photo's image and video are one entry, as the JSON file
|
// A live photo's image and video are one entry, as the JSON file
|
||||||
// beside them names them. A live photo's file with no such JSON
|
// beside them names them. A live photo's file with no such JSON
|
||||||
// file is not its original. If it is a ZIP, it is the one an
|
// file is not its original and is left alone: another process may
|
||||||
// earlier version stored under the image's name, and is removed.
|
// have just stored it and not yet written the JSON file.
|
||||||
// Any other is left alone: another process may have just stored
|
const livePhoto = names.has(livePhotoJSONName(String(id)))
|
||||||
// it and not yet written the JSON file.
|
? readLivePhotoJSON(dir, String(id))
|
||||||
const livePhoto = names.has(livePhotoJSONName(id))
|
|
||||||
? readLivePhotoJSON(dir, id)
|
|
||||||
: undefined;
|
: undefined;
|
||||||
if (livePhoto !== undefined) {
|
if (livePhoto !== undefined) {
|
||||||
into.set(id, livePhoto);
|
into.set(id, livePhoto);
|
||||||
} else if (isLivePhoto(id)) {
|
} else if (!isLivePhoto(id)) {
|
||||||
if (isZip(path)) {
|
|
||||||
await rm(path, { force: true }).catch(() => undefined);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
into.set(id, { path });
|
into.set(id, { path });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+41
-28
@@ -27,7 +27,7 @@
|
|||||||
// never masked by a subsequent empty refresh.
|
// never masked by a subsequent empty refresh.
|
||||||
|
|
||||||
import { rm } from "node:fs/promises";
|
import { rm } from "node:fs/promises";
|
||||||
import { join } from "node:path";
|
import { join, resolve } from "node:path";
|
||||||
import envPaths from "env-paths";
|
import envPaths from "env-paths";
|
||||||
|
|
||||||
import { MetadataStore } from "./store.js";
|
import { MetadataStore } from "./store.js";
|
||||||
@@ -49,9 +49,12 @@ import {
|
|||||||
type PhotosAPI,
|
type PhotosAPI,
|
||||||
type TimelineAPI,
|
type TimelineAPI,
|
||||||
type FreshReads,
|
type FreshReads,
|
||||||
|
type SavePathLookup,
|
||||||
} from "./read.js";
|
} from "./read.js";
|
||||||
import {
|
import {
|
||||||
ContentCache,
|
ContentCache,
|
||||||
|
savePath,
|
||||||
|
storedAtSavePath,
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
type ThumbnailsAPI,
|
type ThumbnailsAPI,
|
||||||
type EnsureOptions,
|
type EnsureOptions,
|
||||||
@@ -70,6 +73,7 @@ export {
|
|||||||
type PhotoFilter,
|
type PhotoFilter,
|
||||||
type TimelineGroup,
|
type TimelineGroup,
|
||||||
type GroupBy,
|
type GroupBy,
|
||||||
|
type SavePathLookup,
|
||||||
} from "./read.js";
|
} from "./read.js";
|
||||||
export {
|
export {
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
@@ -169,8 +173,10 @@ export interface LibraryOptions {
|
|||||||
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
||||||
// plus the user id, so each account has its own cache.
|
// plus the user id, so each account has its own cache.
|
||||||
cacheDirectory?: string;
|
cacheDirectory?: string;
|
||||||
// Persistent backup destination. The refresh loop does not use it; the
|
// The root of every photo's save path, where `Photo.download()` and
|
||||||
// content cache treats an original already stored there as present.
|
// `lib.backup()` put originals. Defaults to `photos` in the working
|
||||||
|
// directory at open. The content cache treats an original already stored
|
||||||
|
// at its save path as present.
|
||||||
downloadDirectory?: string;
|
downloadDirectory?: string;
|
||||||
refreshIntervalSeconds?: number;
|
refreshIntervalSeconds?: number;
|
||||||
onProgress?: RefreshProgressCallback;
|
onProgress?: RefreshProgressCallback;
|
||||||
@@ -234,7 +240,7 @@ export interface LibraryStatus {
|
|||||||
|
|
||||||
export class Library {
|
export class Library {
|
||||||
readonly cacheDirectory: string;
|
readonly cacheDirectory: string;
|
||||||
readonly downloadDirectory?: string;
|
readonly downloadDirectory: string;
|
||||||
|
|
||||||
// The in-process read surface (issue #44). Each namespace answers
|
// The in-process read surface (issue #44). Each namespace answers
|
||||||
// synchronously from the live record projection; no read touches the
|
// synchronously from the live record projection; no read touches the
|
||||||
@@ -296,7 +302,7 @@ export class Library {
|
|||||||
store: MetadataStore;
|
store: MetadataStore;
|
||||||
userID: number;
|
userID: number;
|
||||||
cacheDirectory: string;
|
cacheDirectory: string;
|
||||||
downloadDirectory?: string;
|
downloadDirectory: string;
|
||||||
intervalMs: number;
|
intervalMs: number;
|
||||||
onProgress?: RefreshProgressCallback;
|
onProgress?: RefreshProgressCallback;
|
||||||
pools: RequestPools;
|
pools: RequestPools;
|
||||||
@@ -320,8 +326,14 @@ export class Library {
|
|||||||
// The read namespaces derive fresh from the store on each call, so they
|
// The read namespaces derive fresh from the store on each call, so they
|
||||||
// always reflect the latest refresh.
|
// always reflect the latest refresh.
|
||||||
const derive = (): DerivedRecords => this.deriveNow();
|
const derive = (): DerivedRecords => this.deriveNow();
|
||||||
this.albums = makeAlbumsAPI(derive, this.cache);
|
const root = this.downloadDirectory;
|
||||||
this.photos = makePhotosAPI(derive, this.cache);
|
const saves: SavePathLookup = {
|
||||||
|
savePath: (file) =>
|
||||||
|
storedAtSavePath(root, file)?.path ?? savePath(root, file),
|
||||||
|
isLocal: (file) => storedAtSavePath(root, file) !== undefined,
|
||||||
|
};
|
||||||
|
this.albums = makeAlbumsAPI(derive, saves, this.cache);
|
||||||
|
this.photos = makePhotosAPI(derive, saves, this.cache);
|
||||||
this.timeline = makeTimelineAPI(derive);
|
this.timeline = makeTimelineAPI(derive);
|
||||||
this.thumbnails = {
|
this.thumbnails = {
|
||||||
ensure: (opts: EnsureOptions): Promise<EnsureResult[]> => {
|
ensure: (opts: EnsureOptions): Promise<EnsureResult[]> => {
|
||||||
@@ -350,6 +362,13 @@ export class Library {
|
|||||||
const { userID } = opts.client.whoami();
|
const { userID } = opts.client.whoami();
|
||||||
const cacheDirectory =
|
const cacheDirectory =
|
||||||
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
||||||
|
if (opts.downloadDirectory === "") {
|
||||||
|
throw new Error(
|
||||||
|
"library: downloadDirectory is empty (leave it out to save " +
|
||||||
|
"under photos/ in the working directory)",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const downloadDirectory = opts.downloadDirectory ?? resolve("photos");
|
||||||
const metadataPath = join(cacheDirectory, "metadata.json");
|
const metadataPath = join(cacheDirectory, "metadata.json");
|
||||||
let store = await MetadataStore.load(metadataPath);
|
let store = await MetadataStore.load(metadataPath);
|
||||||
// A cache directory given explicitly can hold another account's cache.
|
// A cache directory given explicitly can hold another account's cache.
|
||||||
@@ -404,7 +423,7 @@ export class Library {
|
|||||||
pools,
|
pools,
|
||||||
source,
|
source,
|
||||||
cacheDirectory,
|
cacheDirectory,
|
||||||
downloadDirectory: opts.downloadDirectory,
|
downloadDirectory,
|
||||||
getFile: (fileID) => store.getFileByID(fileID),
|
getFile: (fileID) => store.getFileByID(fileID),
|
||||||
cacheOriginalsMaxBytes: opts.cacheOriginalsMaxBytes,
|
cacheOriginalsMaxBytes: opts.cacheOriginalsMaxBytes,
|
||||||
freeBelowBytes: opts.freeBelowBytes,
|
freeBelowBytes: opts.freeBelowBytes,
|
||||||
@@ -426,7 +445,7 @@ export class Library {
|
|||||||
store,
|
store,
|
||||||
userID,
|
userID,
|
||||||
cacheDirectory,
|
cacheDirectory,
|
||||||
downloadDirectory: opts.downloadDirectory,
|
downloadDirectory,
|
||||||
intervalMs,
|
intervalMs,
|
||||||
onProgress: opts.onProgress,
|
onProgress: opts.onProgress,
|
||||||
pools,
|
pools,
|
||||||
@@ -470,9 +489,10 @@ export class Library {
|
|||||||
return this.store.getFile(collectionID, fileID);
|
return this.store.getFile(collectionID, fileID);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Any membership of a file, addressed by file id alone. A file's own
|
// The membership of a file its record is read from, addressed by file id
|
||||||
// metadata (title, creationTime) is identical across the collections it
|
// alone. A file's own metadata (title, creationTime) is identical across
|
||||||
// belongs to, so this serves the point commands that hold only a fileID.
|
// the collections it belongs to, so this serves the point commands that
|
||||||
|
// hold only a fileID.
|
||||||
getFileByID(fileID: number): EnteFile | undefined {
|
getFileByID(fileID: number): EnteFile | undefined {
|
||||||
return this.store.getFileByID(fileID);
|
return this.store.getFileByID(fileID);
|
||||||
}
|
}
|
||||||
@@ -543,27 +563,20 @@ export class Library {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Back up every in-scope file to `downloadDirectory` in the historical
|
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||||
// on-disk layout, with a durable failure ledger (issue #51). Waits for a
|
// library's, each original at its save path, with a durable failure
|
||||||
// completed refresh first, as `fresh()` does, joining one already running,
|
// ledger (issue #51). Waits for a completed refresh first, as `fresh()`
|
||||||
// and rejects before touching any file when it fails. Then fetches pending
|
// does, joining one already running, and rejects before touching any file
|
||||||
// originals (and optional thumbnails) through the content cache and pools,
|
// when it fails. Then puts pending originals at their save paths as
|
||||||
// and rebuilds the derived symlink/JSON views from the model. Throws before
|
// `Photo.download()` does (and optional thumbnails) through the content
|
||||||
// any network work when no download directory is available or no content
|
// cache and pools, and rebuilds the derived symlink/JSON views from the
|
||||||
// cache backs the originals it must fetch.
|
// model. Throws before any network work when no content cache backs the
|
||||||
|
// originals it must fetch.
|
||||||
backup(opts?: BackupOptions): Promise<BackupResult> {
|
backup(opts?: BackupOptions): Promise<BackupResult> {
|
||||||
const downloadDirectory =
|
const downloadDirectory =
|
||||||
opts?.downloadDirectory ?? this.downloadDirectory;
|
opts?.downloadDirectory ?? this.downloadDirectory;
|
||||||
const includeOriginals = opts?.includeOriginals ?? true;
|
const includeOriginals = opts?.includeOriginals ?? true;
|
||||||
const includeThumbnails = opts?.includeThumbnails ?? false;
|
const includeThumbnails = opts?.includeThumbnails ?? false;
|
||||||
if (!downloadDirectory) {
|
|
||||||
return Promise.reject(
|
|
||||||
new Error(
|
|
||||||
"backup requires a downloadDirectory (pass one to " +
|
|
||||||
"backup() or open the library with one)",
|
|
||||||
),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if ((includeOriginals || includeThumbnails) && !this.cache) {
|
if ((includeOriginals || includeThumbnails) && !this.cache) {
|
||||||
return Promise.reject(
|
return Promise.reject(
|
||||||
new Error(
|
new Error(
|
||||||
|
|||||||
+132
-32
@@ -12,18 +12,31 @@
|
|||||||
// plain records are the serializable surface, and `record()` returns one.
|
// plain records are the serializable surface, and `record()` returns one.
|
||||||
//
|
//
|
||||||
// A `Photo` also fetches its own bytes: `original()`, `thumbnail()`,
|
// A `Photo` also fetches its own bytes: `original()`, `thumbnail()`,
|
||||||
// `content()` and `exif()` go through the on-disk content cache (issue #46),
|
// `download()`, `content()`, `exif()` and the methods that each return one
|
||||||
// and are the one place in this module that may touch the network. A library
|
// EXIF field go through the on-disk content cache (issue #46), and are
|
||||||
// opened without a content source leaves that cache absent, and those methods
|
// the one place in this module that may touch the network. A library opened
|
||||||
// then throw. `savePath` and `isLocal` look only at the disk.
|
// without a content source leaves that cache absent, and those methods then
|
||||||
|
// throw. `savePath` and `isLocal` look only at the disk and need no cache.
|
||||||
|
|
||||||
import { readFile } from "node:fs/promises";
|
import { readFile } from "node:fs/promises";
|
||||||
|
|
||||||
import { readPhotoExif, type PhotoExif } from "../exif.js";
|
import {
|
||||||
import type { CollectionType, FileType } from "../model/types.js";
|
readAllExifTags,
|
||||||
|
readPhotoExif,
|
||||||
|
type ExifTags,
|
||||||
|
type PhotoExif,
|
||||||
|
} from "../exif.js";
|
||||||
|
import type { CollectionType, EnteFile, FileType } from "../model/types.js";
|
||||||
import type { ContentOptions, ContentResult, PhotoContent } from "./content.js";
|
import type { ContentOptions, ContentResult, PhotoContent } from "./content.js";
|
||||||
import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js";
|
import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js";
|
||||||
|
|
||||||
|
// Where a photo's original is saved, and whether all of it is there. The
|
||||||
|
// library answers both from the disk, with or without a content cache.
|
||||||
|
export interface SavePathLookup {
|
||||||
|
savePath(file: EnteFile): string;
|
||||||
|
isLocal(file: EnteFile): boolean;
|
||||||
|
}
|
||||||
|
|
||||||
// Newest first, with fileID as a stable tiebreak so equal-timed files order
|
// Newest first, with fileID as a stable tiebreak so equal-timed files order
|
||||||
// deterministically — the same order the record projection uses.
|
// deterministically — the same order the record projection uses.
|
||||||
const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
|
const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
|
||||||
@@ -34,11 +47,22 @@ const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
|
|||||||
const byNewestAlbum = (a: AlbumRecord, b: AlbumRecord): number =>
|
const byNewestAlbum = (a: AlbumRecord, b: AlbumRecord): number =>
|
||||||
b.updationTime - a.updationTime || b.collectionID - a.collectionID;
|
b.updationTime - a.updationTime || b.collectionID - a.collectionID;
|
||||||
|
|
||||||
|
// A method for each `PhotoExif` field, named after it, taking the options
|
||||||
|
// `exif()` takes and giving that field. `Photo` implements it, so the build's
|
||||||
|
// type check fails when `PhotoExif` has a field `Photo` has no method for.
|
||||||
|
type PhotoExifMethods = {
|
||||||
|
[K in keyof PhotoExif]-?: (opts?: ContentOptions) => Promise<PhotoExif[K]>;
|
||||||
|
};
|
||||||
|
|
||||||
// A single photo. Field access mirrors `PhotoRecord`; `record()` returns the
|
// A single photo. Field access mirrors `PhotoRecord`; `record()` returns the
|
||||||
// underlying plain record for callers that need the IPC-safe value.
|
// underlying plain record for callers that need the IPC-safe value. `file` is
|
||||||
export class Photo {
|
// the membership the record is read from, so the save path carries the date of
|
||||||
|
// `takenAt` and stays known after a refresh removes the file from the library.
|
||||||
|
export class Photo implements PhotoExifMethods {
|
||||||
constructor(
|
constructor(
|
||||||
private readonly rec: PhotoRecord,
|
private readonly rec: PhotoRecord,
|
||||||
|
private readonly file: EnteFile,
|
||||||
|
private readonly saves: SavePathLookup,
|
||||||
private readonly cache?: PhotoContent,
|
private readonly cache?: PhotoContent,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
@@ -89,20 +113,20 @@ export class Photo {
|
|||||||
return this.rec.isHidden;
|
return this.rec.isHidden;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Where `lib.backup()` stores the original in the library's download
|
// Where `download()` and `lib.backup()` put the original under the
|
||||||
// directory, whether or not it is there yet: for a live photo already
|
// library's download directory, whether or not it is there yet:
|
||||||
// stored, its image. For a live photo not yet stored, it carries the
|
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`. For a live photo
|
||||||
// title's extension, and the backup may store the image under a different
|
// already stored, its image. For a live photo not yet stored, it carries
|
||||||
// one. Undefined when the library has no download directory or no content
|
// the title's extension, and the image may be stored under a different
|
||||||
// cache.
|
// one.
|
||||||
get savePath(): string | undefined {
|
get savePath(): string {
|
||||||
return this.cache?.savePath(this.rec.fileID);
|
return this.saves.savePath(this.file);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Whether the whole original is at `savePath`. A copy only in the cache
|
// Whether the whole original is at `savePath`. A copy only in the cache
|
||||||
// does not count.
|
// does not count.
|
||||||
get isLocal(): boolean {
|
get isLocal(): boolean {
|
||||||
return this.cache?.isLocal(this.rec.fileID) ?? false;
|
return this.saves.isLocal(this.file);
|
||||||
}
|
}
|
||||||
|
|
||||||
record(): PhotoRecord {
|
record(): PhotoRecord {
|
||||||
@@ -111,12 +135,20 @@ export class Photo {
|
|||||||
|
|
||||||
// Fetch and cache the full-resolution original, returning its on-disk path
|
// Fetch and cache the full-resolution original, returning its on-disk path
|
||||||
// and byte length; for a live photo, its image's, and its video's path as
|
// and byte length; for a live photo, its image's, and its video's path as
|
||||||
// `videoPath`. Served from the cache (or the backup download directory)
|
// `videoPath`. Served from the cache (or the save path) when already
|
||||||
// when already present, otherwise fetched through the content pool.
|
// present, otherwise fetched through the content pool.
|
||||||
async original(opts?: ContentOptions): Promise<ContentResult> {
|
async original(opts?: ContentOptions): Promise<ContentResult> {
|
||||||
return this.cacheOrThrow().original(this.rec.fileID, opts);
|
return this.cacheOrThrow().original(this.rec.fileID, opts);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Put the original at `savePath` and return it there, as `original()`
|
||||||
|
// does. When it is already there, nothing is written. When the cache holds
|
||||||
|
// it, it is copied from there; otherwise it is fetched straight to
|
||||||
|
// `savePath`.
|
||||||
|
async download(): Promise<ContentResult> {
|
||||||
|
return this.cacheOrThrow().download(this.file);
|
||||||
|
}
|
||||||
|
|
||||||
// As `original`, for the thumbnail, through the thumbnail pool.
|
// As `original`, for the thumbnail, through the thumbnail pool.
|
||||||
async thumbnail(opts?: ContentOptions): Promise<ContentResult> {
|
async thumbnail(opts?: ContentOptions): Promise<ContentResult> {
|
||||||
return this.cacheOrThrow().thumbnail(this.rec.fileID, opts);
|
return this.cacheOrThrow().thumbnail(this.rec.fileID, opts);
|
||||||
@@ -129,15 +161,75 @@ export class Photo {
|
|||||||
return readFile(path);
|
return readFile(path);
|
||||||
}
|
}
|
||||||
|
|
||||||
// The common EXIF fields of the original, read from `content()`, so this
|
// Every EXIF tag of the original, keyed by name (see `ExifTags`), read
|
||||||
// may download it. EXIF is read from any image format exifreader reads,
|
// from `content()`, so this may download it. EXIF is read from any image
|
||||||
// JPEG and HEIC/HEIF among them; any other file gives `{}`, and a video
|
// format exifreader reads, JPEG and HEIC/HEIF among them; any other file
|
||||||
// gives it without fetching anything. Like the other content methods, it
|
// gives `{}`, and a video gives it without fetching anything. Like the
|
||||||
// throws when there is no content cache, video or not.
|
// other content methods, it throws when there is no content cache, video
|
||||||
async exif(opts?: ContentOptions): Promise<PhotoExif> {
|
// or not.
|
||||||
|
async exif(opts?: ContentOptions): Promise<ExifTags> {
|
||||||
this.cacheOrThrow();
|
this.cacheOrThrow();
|
||||||
if (this.rec.fileType === "video") return {};
|
if (this.rec.fileType === "video") return {};
|
||||||
return readPhotoExif(await this.content(opts));
|
return readAllExifTags(await this.content(opts));
|
||||||
|
}
|
||||||
|
|
||||||
|
// One field each, named and typed as in `PhotoExif`, picked from the tags
|
||||||
|
// `exif()` returns, and undefined when the file lacks it. Each call runs
|
||||||
|
// `exif()`, which reads the original again.
|
||||||
|
async make(opts?: ContentOptions): Promise<PhotoExif["make"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).make;
|
||||||
|
}
|
||||||
|
async model(opts?: ContentOptions): Promise<PhotoExif["model"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).model;
|
||||||
|
}
|
||||||
|
async lensModel(opts?: ContentOptions): Promise<PhotoExif["lensModel"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).lensModel;
|
||||||
|
}
|
||||||
|
async dateTimeOriginal(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["dateTimeOriginal"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).dateTimeOriginal;
|
||||||
|
}
|
||||||
|
async offsetTimeOriginal(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["offsetTimeOriginal"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).offsetTimeOriginal;
|
||||||
|
}
|
||||||
|
async exposureTime(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["exposureTime"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).exposureTime;
|
||||||
|
}
|
||||||
|
async fNumber(opts?: ContentOptions): Promise<PhotoExif["fNumber"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).fNumber;
|
||||||
|
}
|
||||||
|
async iso(opts?: ContentOptions): Promise<PhotoExif["iso"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).iso;
|
||||||
|
}
|
||||||
|
async focalLength(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["focalLength"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).focalLength;
|
||||||
|
}
|
||||||
|
async orientation(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["orientation"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).orientation;
|
||||||
|
}
|
||||||
|
async gpsLatitude(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["gpsLatitude"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).gpsLatitude;
|
||||||
|
}
|
||||||
|
async gpsLongitude(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["gpsLongitude"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).gpsLongitude;
|
||||||
|
}
|
||||||
|
async gpsAltitude(
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<PhotoExif["gpsAltitude"]> {
|
||||||
|
return readPhotoExif(await this.exif(opts)).gpsAltitude;
|
||||||
}
|
}
|
||||||
|
|
||||||
private cacheOrThrow(): PhotoContent {
|
private cacheOrThrow(): PhotoContent {
|
||||||
@@ -156,6 +248,7 @@ export class Album {
|
|||||||
constructor(
|
constructor(
|
||||||
private readonly rec: AlbumRecord,
|
private readonly rec: AlbumRecord,
|
||||||
private readonly records: DerivedRecords,
|
private readonly records: DerivedRecords,
|
||||||
|
private readonly saves: SavePathLookup,
|
||||||
private readonly content?: PhotoContent,
|
private readonly content?: PhotoContent,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
@@ -190,7 +283,10 @@ export class Album {
|
|||||||
const out: Photo[] = [];
|
const out: Photo[] = [];
|
||||||
for (const id of this.rec.fileIDs) {
|
for (const id of this.rec.fileIDs) {
|
||||||
const p = this.records.photos.get(id);
|
const p = this.records.photos.get(id);
|
||||||
if (p) out.push(new Photo(p, this.content));
|
const file = this.records.files.get(id);
|
||||||
|
if (p && file) {
|
||||||
|
out.push(new Photo(p, file, this.saves, this.content));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
@@ -253,18 +349,19 @@ export interface FreshReads {
|
|||||||
|
|
||||||
export const makeAlbumsAPI = (
|
export const makeAlbumsAPI = (
|
||||||
derive: () => DerivedRecords,
|
derive: () => DerivedRecords,
|
||||||
|
saves: SavePathLookup,
|
||||||
content?: PhotoContent,
|
content?: PhotoContent,
|
||||||
): AlbumsAPI => ({
|
): AlbumsAPI => ({
|
||||||
list: (): Album[] => {
|
list: (): Album[] => {
|
||||||
const records = derive();
|
const records = derive();
|
||||||
return [...records.albums.values()]
|
return [...records.albums.values()]
|
||||||
.sort(byNewestAlbum)
|
.sort(byNewestAlbum)
|
||||||
.map((rec) => new Album(rec, records, content));
|
.map((rec) => new Album(rec, records, saves, content));
|
||||||
},
|
},
|
||||||
byID: ({ collectionID }): Album | undefined => {
|
byID: ({ collectionID }): Album | undefined => {
|
||||||
const records = derive();
|
const records = derive();
|
||||||
const rec = records.albums.get(collectionID);
|
const rec = records.albums.get(collectionID);
|
||||||
return rec ? new Album(rec, records, content) : undefined;
|
return rec ? new Album(rec, records, saves, content) : undefined;
|
||||||
},
|
},
|
||||||
byName: ({ albumName }): Album | undefined => {
|
byName: ({ albumName }): Album | undefined => {
|
||||||
const records = derive();
|
const records = derive();
|
||||||
@@ -273,17 +370,20 @@ export const makeAlbumsAPI = (
|
|||||||
const match = [...records.albums.values()]
|
const match = [...records.albums.values()]
|
||||||
.sort(byNewestAlbum)
|
.sort(byNewestAlbum)
|
||||||
.find((rec) => rec.name === albumName);
|
.find((rec) => rec.name === albumName);
|
||||||
return match ? new Album(match, records, content) : undefined;
|
return match ? new Album(match, records, saves, content) : undefined;
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
export const makePhotosAPI = (
|
export const makePhotosAPI = (
|
||||||
derive: () => DerivedRecords,
|
derive: () => DerivedRecords,
|
||||||
|
saves: SavePathLookup,
|
||||||
content?: PhotoContent,
|
content?: PhotoContent,
|
||||||
): PhotosAPI => ({
|
): PhotosAPI => ({
|
||||||
byID: ({ fileID }): Photo | undefined => {
|
byID: ({ fileID }): Photo | undefined => {
|
||||||
const rec = derive().photos.get(fileID);
|
const records = derive();
|
||||||
return rec ? new Photo(rec, content) : undefined;
|
const rec = records.photos.get(fileID);
|
||||||
|
const file = records.files.get(fileID);
|
||||||
|
return rec && file ? new Photo(rec, file, saves, content) : undefined;
|
||||||
},
|
},
|
||||||
records: ({ fileIDs }): PhotoRecord[] => {
|
records: ({ fileIDs }): PhotoRecord[] => {
|
||||||
const { photos } = derive();
|
const { photos } = derive();
|
||||||
|
|||||||
+33
-15
@@ -86,6 +86,10 @@ export interface LibraryChange {
|
|||||||
export interface DerivedRecords {
|
export interface DerivedRecords {
|
||||||
albums: Map<number, AlbumRecord>;
|
albums: Map<number, AlbumRecord>;
|
||||||
photos: Map<number, PhotoRecord>;
|
photos: Map<number, PhotoRecord>;
|
||||||
|
// The membership each photo's record is read from, for its `Photo`'s save
|
||||||
|
// path. It holds the file's key, so it stays in this process: no snapshot
|
||||||
|
// or change carries it.
|
||||||
|
files: Map<number, EnteFile>;
|
||||||
}
|
}
|
||||||
|
|
||||||
const asString = (v: unknown): string | undefined =>
|
const asString = (v: unknown): string | undefined =>
|
||||||
@@ -101,18 +105,19 @@ const microsToMillis = (micros: number): number => Math.floor(micros / 1000);
|
|||||||
const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number =>
|
const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number =>
|
||||||
b.takenAt - a.takenAt || b.fileID - a.fileID;
|
b.takenAt - a.takenAt || b.fileID - a.fileID;
|
||||||
|
|
||||||
// Build one PhotoRecord from every membership of a file. The memberships share
|
// A photo's `takenAt` in milliseconds: `pubMagicMetadata.editedTime` when the
|
||||||
// the same underlying file, so metadata is read from a single representative
|
// user edited the date, else basic-metadata `creationTime`.
|
||||||
// (the most recently synced, lowest collection id to break ties); `albumIDs`
|
export const takenAtOf = (file: EnteFile): number =>
|
||||||
// gathers them all.
|
microsToMillis(
|
||||||
const toPhotoRecord = (
|
asNumber(file.pubMagicMetadata?.editedTime) ??
|
||||||
fileID: number,
|
file.metadata.creationTime,
|
||||||
memberships: EnteFile[],
|
);
|
||||||
): PhotoRecord => {
|
|
||||||
const albumIDs = memberships
|
// The membership a file's record is read from: the most recently synced, lowest
|
||||||
.map((m) => m.collectionID)
|
// collection id to break ties. Whatever dates a file's save path takes this
|
||||||
.sort((a, b) => a - b);
|
// membership too, so the path always carries the record's `takenAt`.
|
||||||
const rep = memberships.reduce((best, m) =>
|
export const representative = (memberships: EnteFile[]): EnteFile =>
|
||||||
|
memberships.reduce((best, m) =>
|
||||||
m.updationTime > best.updationTime ||
|
m.updationTime > best.updationTime ||
|
||||||
(m.updationTime === best.updationTime &&
|
(m.updationTime === best.updationTime &&
|
||||||
m.collectionID < best.collectionID)
|
m.collectionID < best.collectionID)
|
||||||
@@ -120,17 +125,28 @@ const toPhotoRecord = (
|
|||||||
: best,
|
: best,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Build one PhotoRecord from every membership of a file. The memberships share
|
||||||
|
// the same underlying file, so metadata is read from a single representative;
|
||||||
|
// `albumIDs` gathers them all.
|
||||||
|
const toPhotoRecord = (
|
||||||
|
fileID: number,
|
||||||
|
memberships: EnteFile[],
|
||||||
|
): PhotoRecord => {
|
||||||
|
const albumIDs = memberships
|
||||||
|
.map((m) => m.collectionID)
|
||||||
|
.sort((a, b) => a - b);
|
||||||
|
const rep = representative(memberships);
|
||||||
|
|
||||||
const pub = rep.pubMagicMetadata ?? {};
|
const pub = rep.pubMagicMetadata ?? {};
|
||||||
const priv = rep.magicMetadata ?? {};
|
const priv = rep.magicMetadata ?? {};
|
||||||
|
|
||||||
const takenAtMicros = asNumber(pub.editedTime) ?? rep.metadata.creationTime;
|
|
||||||
const visibility = asNumber(priv.visibility);
|
const visibility = asNumber(priv.visibility);
|
||||||
|
|
||||||
const record: PhotoRecord = {
|
const record: PhotoRecord = {
|
||||||
fileID,
|
fileID,
|
||||||
albumIDs,
|
albumIDs,
|
||||||
title: asString(pub.editedName) ?? rep.metadata.title,
|
title: asString(pub.editedName) ?? rep.metadata.title,
|
||||||
takenAt: microsToMillis(takenAtMicros),
|
takenAt: takenAtOf(rep),
|
||||||
modifiedAt: microsToMillis(rep.metadata.modificationTime),
|
modifiedAt: microsToMillis(rep.metadata.modificationTime),
|
||||||
fileType: rep.metadata.fileType,
|
fileType: rep.metadata.fileType,
|
||||||
isArchived: visibility === VISIBILITY_ARCHIVED,
|
isArchived: visibility === VISIBILITY_ARCHIVED,
|
||||||
@@ -198,6 +214,7 @@ export const deriveRecords = (
|
|||||||
}
|
}
|
||||||
|
|
||||||
const photos = new Map<number, PhotoRecord>();
|
const photos = new Map<number, PhotoRecord>();
|
||||||
|
const photoFiles = new Map<number, EnteFile>();
|
||||||
const takenAtByFile = new Map<number, number>();
|
const takenAtByFile = new Map<number, number>();
|
||||||
for (const [fileID, memberships] of byFileID) {
|
for (const [fileID, memberships] of byFileID) {
|
||||||
const record = toPhotoRecord(fileID, memberships);
|
const record = toPhotoRecord(fileID, memberships);
|
||||||
@@ -209,6 +226,7 @@ export const deriveRecords = (
|
|||||||
record.thumbnailPath = paths.thumbnailPath;
|
record.thumbnailPath = paths.thumbnailPath;
|
||||||
}
|
}
|
||||||
photos.set(fileID, record);
|
photos.set(fileID, record);
|
||||||
|
photoFiles.set(fileID, representative(memberships));
|
||||||
takenAtByFile.set(fileID, record.takenAt);
|
takenAtByFile.set(fileID, record.takenAt);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -217,7 +235,7 @@ export const deriveRecords = (
|
|||||||
albums.set(c.id, toAlbumRecord(c, files, takenAtByFile));
|
albums.set(c.id, toAlbumRecord(c, files, takenAtByFile));
|
||||||
}
|
}
|
||||||
|
|
||||||
return { albums, photos };
|
return { albums, photos, files: photoFiles };
|
||||||
};
|
};
|
||||||
|
|
||||||
// Sorted, GUI-ready arrays: albums newest updated first, photos newest first.
|
// Sorted, GUI-ready arrays: albums newest updated first, photos newest first.
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import { dirname } from "node:path";
|
|||||||
|
|
||||||
import { writeAtomic } from "../download/index.js";
|
import { writeAtomic } from "../download/index.js";
|
||||||
import type { Collection, EnteFile, Microseconds } from "../model/types.js";
|
import type { Collection, EnteFile, Microseconds } from "../model/types.js";
|
||||||
|
import { representative } from "./records.js";
|
||||||
|
|
||||||
// Bumped only when the on-disk shape changes incompatibly. A file written
|
// Bumped only when the on-disk shape changes incompatibly. A file written
|
||||||
// under a different version is discarded on load (see `load`): re-fetching
|
// under a different version is discarded on load (see `load`): re-fetching
|
||||||
@@ -179,14 +180,16 @@ export class MetadataStore {
|
|||||||
return this.files.get(fileKey(collectionID, fileID));
|
return this.files.get(fileKey(collectionID, fileID));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Any membership of a file, or undefined. Every membership re-wraps the
|
// The membership of a file its record is read from (`representative`), or
|
||||||
// same underlying content key, so any one is enough to fetch the bytes;
|
// undefined. Any membership could fetch the bytes, but the content cache
|
||||||
// the content cache resolves a fileID to a file this way.
|
// resolves a fileID to a file this way so that it dates the save path
|
||||||
|
// from the same membership as `photo.savePath`.
|
||||||
getFileByID(fileID: number): EnteFile | undefined {
|
getFileByID(fileID: number): EnteFile | undefined {
|
||||||
|
const memberships: EnteFile[] = [];
|
||||||
for (const file of this.files.values()) {
|
for (const file of this.files.values()) {
|
||||||
if (file.id === fileID) return file;
|
if (file.id === fileID) memberships.push(file);
|
||||||
}
|
}
|
||||||
return undefined;
|
return memberships.length > 0 ? representative(memberships) : undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
listFiles(collectionID: number): EnteFile[] {
|
listFiles(collectionID: number): EnteFile[] {
|
||||||
|
|||||||
+178
-146
@@ -1,16 +1,16 @@
|
|||||||
/**
|
/**
|
||||||
* Tests for the `quak backup` logic, now built on the library API (issue #51).
|
* Tests for the `quak backup` logic, now built on the library API (issue #51).
|
||||||
*
|
*
|
||||||
* `lib.backup({ downloadDirectory })` refreshes the library, fetches each
|
* `lib.backup({ downloadDirectory })` refreshes the library, puts each pending
|
||||||
* pending file's original through the content cache/pools, and materialises the
|
* file's original at its save path through the content cache/pools, and
|
||||||
* unchanged on-disk layout:
|
* materialises the on-disk layout:
|
||||||
*
|
*
|
||||||
* <downloadDirectory>/
|
* <downloadDirectory>/
|
||||||
* originals/
|
* YYYY/YYYY-MM/YYYY-MM-DD/
|
||||||
* <fileID>.<ext> the decrypted bytes ("present means complete")
|
* YYYY-MM-DD.<fileID>.<ext> the decrypted bytes ("present means complete")
|
||||||
* <fileID>.json per-file metadata sidecar (rebuilt each run)
|
* YYYY-MM-DD.<fileID>.json per-file metadata sidecar (rebuilt each run)
|
||||||
* collections/
|
* collections/
|
||||||
* <name>/<title> symlink into ../originals (rebuilt each run)
|
* <name>/<title> symlink to the original (rebuilt each run)
|
||||||
* <name>.json per-collection metadata (rebuilt each run)
|
* <name>.json per-collection metadata (rebuilt each run)
|
||||||
* failures.json durable ledger of unresolved failures
|
* failures.json durable ledger of unresolved failures
|
||||||
*
|
*
|
||||||
@@ -94,6 +94,18 @@ const USER_ID = 42;
|
|||||||
// Decrypted-byte length each stub original writes, keyed by fileID.
|
// Decrypted-byte length each stub original writes, keyed by fileID.
|
||||||
const SIZE_BY_ID: Record<number, number> = { 100: 3000, 101: 2000, 200: 1500 };
|
const SIZE_BY_ID: Record<number, number> = { 100: 3000, 101: 2000, 200: 1500 };
|
||||||
|
|
||||||
|
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
|
||||||
|
// stores times, so the machine's time zone cannot move it to another day. Its
|
||||||
|
// original is saved in the folder `DAY`, named `2026-03-01.<fileID>.<ext>`.
|
||||||
|
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
|
||||||
|
const DAY = join("2026", "2026-03", "2026-03-01");
|
||||||
|
|
||||||
|
// Where the backup in `outDir` saves `name` (`<fileID>.<ext>`), and the target
|
||||||
|
// of an album folder's symlink to it.
|
||||||
|
const saved = (outDir: string, name: string): string =>
|
||||||
|
join(outDir, DAY, `2026-03-01.${name}`);
|
||||||
|
const linkTo = (name: string): string => `../../${DAY}/2026-03-01.${name}`;
|
||||||
|
|
||||||
const collection = (id: number, name: string): Collection => ({
|
const collection = (id: number, name: string): Collection => ({
|
||||||
id,
|
id,
|
||||||
ownerID: USER_ID,
|
ownerID: USER_ID,
|
||||||
@@ -112,7 +124,7 @@ const file = (id: number, collectionID: number, title: string): EnteFile => ({
|
|||||||
metadata: {
|
metadata: {
|
||||||
title,
|
title,
|
||||||
fileType: "image",
|
fileType: "image",
|
||||||
creationTime: 1,
|
creationTime: TAKEN,
|
||||||
modificationTime: 1,
|
modificationTime: 1,
|
||||||
},
|
},
|
||||||
file: { decryptionHeader: "aGVhZGVy" },
|
file: { decryptionHeader: "aGVhZGVy" },
|
||||||
@@ -253,7 +265,11 @@ describe("lib.backup", () => {
|
|||||||
it("throws before any network when no downloadDirectory is given", async () => {
|
it("throws before any network when no downloadDirectory is given", async () => {
|
||||||
const source = stubSource();
|
const source = stubSource();
|
||||||
const lib = await openLibrary(source);
|
const lib = await openLibrary(source);
|
||||||
await expect(lib.backup()).rejects.toThrow(/downloadDirectory/i);
|
// The library always has a download directory, so only an empty one
|
||||||
|
// given to backup() reaches runBackup as none.
|
||||||
|
await expect(lib.backup({ downloadDirectory: "" })).rejects.toThrow(
|
||||||
|
/downloadDirectory/i,
|
||||||
|
);
|
||||||
expect(source.originalCalls).toBe(0);
|
expect(source.originalCalls).toBe(0);
|
||||||
lib.close();
|
lib.close();
|
||||||
});
|
});
|
||||||
@@ -271,28 +287,22 @@ describe("lib.backup", () => {
|
|||||||
expect(result.failed).toBe(0);
|
expect(result.failed).toBe(0);
|
||||||
expect(result.errors).toEqual([]);
|
expect(result.errors).toEqual([]);
|
||||||
|
|
||||||
// Originals under originals/<fileID>.<ext>.
|
// Originals at YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>.
|
||||||
expect(readFileSync(join(outDir, "originals", "100.jpg")).length).toBe(
|
expect(readFileSync(saved(outDir, "100.jpg")).length).toBe(3000);
|
||||||
3000,
|
expect(readFileSync(saved(outDir, "101.jpg")).length).toBe(2000);
|
||||||
);
|
expect(readFileSync(saved(outDir, "200.png")).length).toBe(1500);
|
||||||
expect(readFileSync(join(outDir, "originals", "101.jpg")).length).toBe(
|
|
||||||
2000,
|
|
||||||
);
|
|
||||||
expect(readFileSync(join(outDir, "originals", "200.png")).length).toBe(
|
|
||||||
1500,
|
|
||||||
);
|
|
||||||
|
|
||||||
// Per-file metadata sidecar.
|
// Per-file metadata sidecar.
|
||||||
const sidecar = JSON.parse(
|
const sidecar = JSON.parse(
|
||||||
readFileSync(join(outDir, "originals", "100.json"), "utf-8"),
|
readFileSync(saved(outDir, "100.json"), "utf-8"),
|
||||||
);
|
);
|
||||||
expect(sidecar.id).toBe(100);
|
expect(sidecar.id).toBe(100);
|
||||||
expect(sidecar.metadata.title).toBe("beach.jpg");
|
expect(sidecar.metadata.title).toBe("beach.jpg");
|
||||||
|
|
||||||
// Collection dirs contain symlinks into ../originals.
|
// Collection dirs contain symlinks into the date folders.
|
||||||
const beach = join(outDir, "collections", "Vacation", "beach.jpg");
|
const beach = join(outDir, "collections", "Vacation", "beach.jpg");
|
||||||
expect(lstatSync(beach).isSymbolicLink()).toBe(true);
|
expect(lstatSync(beach).isSymbolicLink()).toBe(true);
|
||||||
expect(readlinkSync(beach)).toContain("originals");
|
expect(readlinkSync(beach)).toContain(DAY);
|
||||||
expect(readFileSync(beach).length).toBe(3000);
|
expect(readFileSync(beach).length).toBe(3000);
|
||||||
|
|
||||||
// Per-collection metadata JSON.
|
// Per-collection metadata JSON.
|
||||||
@@ -316,7 +326,7 @@ describe("lib.backup", () => {
|
|||||||
|
|
||||||
expect(result.failed).toBe(0);
|
expect(result.failed).toBe(0);
|
||||||
// The title has no usable extension, so the original is `.bin`.
|
// The title has no usable extension, so the original is `.bin`.
|
||||||
expect(existsSync(join(outDir, "originals", "300.bin"))).toBe(true);
|
expect(existsSync(saved(outDir, "300.bin"))).toBe(true);
|
||||||
const link = join(
|
const link = join(
|
||||||
outDir,
|
outDir,
|
||||||
"collections",
|
"collections",
|
||||||
@@ -366,9 +376,9 @@ describe("lib.backup", () => {
|
|||||||
expect(result.errors[0]!.title).toBe("sunset.jpg");
|
expect(result.errors[0]!.title).toBe("sunset.jpg");
|
||||||
|
|
||||||
// The two good files are on disk; the failed one is not.
|
// The two good files are on disk; the failed one is not.
|
||||||
expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(true);
|
expect(existsSync(saved(outDir, "100.jpg"))).toBe(true);
|
||||||
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
|
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
|
||||||
expect(existsSync(join(outDir, "originals", "101.jpg"))).toBe(false);
|
expect(existsSync(saved(outDir, "101.jpg"))).toBe(false);
|
||||||
expect(
|
expect(
|
||||||
existsSync(join(outDir, "collections", "Vacation", "sunset.jpg")),
|
existsSync(join(outDir, "collections", "Vacation", "sunset.jpg")),
|
||||||
).toBe(false);
|
).toBe(false);
|
||||||
@@ -403,7 +413,7 @@ describe("lib.backup", () => {
|
|||||||
const r3 = await lib.backup({ downloadDirectory: outDir });
|
const r3 = await lib.backup({ downloadDirectory: outDir });
|
||||||
expect(r3.failed).toBe(0);
|
expect(r3.failed).toBe(0);
|
||||||
expect(r3.skipped).toBe(2);
|
expect(r3.skipped).toBe(2);
|
||||||
expect(existsSync(join(outDir, "originals", "101.jpg"))).toBe(true);
|
expect(existsSync(saved(outDir, "101.jpg"))).toBe(true);
|
||||||
// A ledger with no remaining failures is removed.
|
// A ledger with no remaining failures is removed.
|
||||||
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
|
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
|
||||||
lib.close();
|
lib.close();
|
||||||
@@ -423,8 +433,8 @@ describe("lib.backup", () => {
|
|||||||
const result = await lib.backup({ downloadDirectory: outDir });
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
// Every original still downloads despite the symlink failure.
|
// Every original still downloads despite the symlink failure.
|
||||||
expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(true);
|
expect(existsSync(saved(outDir, "100.jpg"))).toBe(true);
|
||||||
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
|
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
|
||||||
// The other symlinks are still built.
|
// The other symlinks are still built.
|
||||||
expect(
|
expect(
|
||||||
lstatSync(
|
lstatSync(
|
||||||
@@ -454,7 +464,7 @@ describe("lib.backup", () => {
|
|||||||
await lib.backup({ downloadDirectory: outDir });
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
// Corrupt a sidecar and delete a symlink between runs.
|
// Corrupt a sidecar and delete a symlink between runs.
|
||||||
writeFileSync(join(outDir, "originals", "100.json"), "not json");
|
writeFileSync(saved(outDir, "100.json"), "not json");
|
||||||
rmSync(join(outDir, "collections", "Vacation", "beach.jpg"));
|
rmSync(join(outDir, "collections", "Vacation", "beach.jpg"));
|
||||||
|
|
||||||
const result = await lib.backup({ downloadDirectory: outDir });
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
@@ -462,7 +472,7 @@ describe("lib.backup", () => {
|
|||||||
|
|
||||||
// The derived views are repaired from the model.
|
// The derived views are repaired from the model.
|
||||||
const sidecar = JSON.parse(
|
const sidecar = JSON.parse(
|
||||||
readFileSync(join(outDir, "originals", "100.json"), "utf-8"),
|
readFileSync(saved(outDir, "100.json"), "utf-8"),
|
||||||
);
|
);
|
||||||
expect(sidecar.metadata.title).toBe("beach.jpg");
|
expect(sidecar.metadata.title).toBe("beach.jpg");
|
||||||
expect(
|
expect(
|
||||||
@@ -485,8 +495,8 @@ describe("lib.backup", () => {
|
|||||||
|
|
||||||
expect(result.totalFiles).toBe(1);
|
expect(result.totalFiles).toBe(1);
|
||||||
expect(result.downloaded).toBe(1);
|
expect(result.downloaded).toBe(1);
|
||||||
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
|
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
|
||||||
expect(existsSync(join(outDir, "originals", "100.jpg"))).toBe(false);
|
expect(existsSync(saved(outDir, "100.jpg"))).toBe(false);
|
||||||
expect(existsSync(join(outDir, "collections", "Work.json"))).toBe(true);
|
expect(existsSync(join(outDir, "collections", "Work.json"))).toBe(true);
|
||||||
expect(existsSync(join(outDir, "collections", "Vacation.json"))).toBe(
|
expect(existsSync(join(outDir, "collections", "Vacation.json"))).toBe(
|
||||||
false,
|
false,
|
||||||
@@ -530,7 +540,7 @@ describe("lib.backup", () => {
|
|||||||
|
|
||||||
expect(result.totalFiles).toBe(1);
|
expect(result.totalFiles).toBe(1);
|
||||||
expect(result.failed).toBe(0);
|
expect(result.failed).toBe(0);
|
||||||
expect(existsSync(join(outDir, "originals", "200.png"))).toBe(true);
|
expect(existsSync(saved(outDir, "200.png"))).toBe(true);
|
||||||
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
|
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
|
||||||
lib.close();
|
lib.close();
|
||||||
});
|
});
|
||||||
@@ -572,7 +582,7 @@ describe("lib.backup", () => {
|
|||||||
it("fetches each original once and writes it only into the backup", async () => {
|
it("fetches each original once and writes it only into the backup", async () => {
|
||||||
// A backup of a 500 GB account must write 500 GB, not a copy in the
|
// A backup of a 500 GB account must write 500 GB, not a copy in the
|
||||||
// cache as well: an original fetched for the backup goes straight
|
// cache as well: an original fetched for the backup goes straight
|
||||||
// into its originals/, and the cache records it there.
|
// to its save path, and the cache records it there.
|
||||||
const source = stubSource();
|
const source = stubSource();
|
||||||
const lib = await openLibrary(source);
|
const lib = await openLibrary(source);
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
@@ -582,23 +592,78 @@ describe("lib.backup", () => {
|
|||||||
expect(result.downloaded).toBe(3);
|
expect(result.downloaded).toBe(3);
|
||||||
expect(source.originalCalls).toBe(3);
|
expect(source.originalCalls).toBe(3);
|
||||||
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
|
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
|
||||||
const stored = readdirSync(join(outDir, "originals")).filter(
|
const stored = readdirSync(join(outDir, DAY)).filter(
|
||||||
(name) => !name.endsWith(".json"),
|
(name) => !name.endsWith(".json"),
|
||||||
);
|
);
|
||||||
expect(stored.sort()).toEqual(["100.jpg", "101.jpg", "200.png"]);
|
expect(stored.sort()).toEqual([
|
||||||
|
"2026-03-01.100.jpg",
|
||||||
|
"2026-03-01.101.jpg",
|
||||||
|
"2026-03-01.200.png",
|
||||||
|
]);
|
||||||
// The cache counts the backup's copy as present: reading the
|
// The cache counts the backup's copy as present: reading the
|
||||||
// original afterwards fetches nothing and answers with that copy.
|
// original afterwards fetches nothing and answers with that copy.
|
||||||
const read = await lib.photos.byID({ fileID: 100 })!.original();
|
const read = await lib.photos.byID({ fileID: 100 })!.original();
|
||||||
expect(read.path).toBe(join(outDir, "originals", "100.jpg"));
|
expect(read.path).toBe(saved(outDir, "100.jpg"));
|
||||||
expect(source.originalCalls).toBe(3);
|
expect(source.originalCalls).toBe(3);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("saves a file in two albums at its photo's save path and links it from both", async () => {
|
||||||
|
// The date edited in Ente has reached Work's copy, synced later, but
|
||||||
|
// Vacation's copy still has an earlier edit.
|
||||||
|
const older = file(100, 1, "beach.jpg");
|
||||||
|
older.pubMagicMetadata = {
|
||||||
|
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
|
||||||
|
};
|
||||||
|
const newer = file(100, 2, "beach.jpg");
|
||||||
|
newer.updationTime = 2;
|
||||||
|
newer.pubMagicMetadata = {
|
||||||
|
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
|
||||||
|
};
|
||||||
|
class SharedFileClient extends MockClient {
|
||||||
|
override async filesSince(args: {
|
||||||
|
collectionID: number;
|
||||||
|
}): Promise<FilesPage> {
|
||||||
|
const files = [args.collectionID === 1 ? older : newer];
|
||||||
|
return { files, deleted: [], cursor: 1 };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
const lib = await Library.open({
|
||||||
|
client: new SharedFileClient(),
|
||||||
|
cacheDirectory: join(root, "cache"),
|
||||||
|
downloadDirectory: outDir,
|
||||||
|
contentSource: stubSource(),
|
||||||
|
refreshIntervalSeconds: 3600,
|
||||||
|
precacheThumbnails: false,
|
||||||
|
precacheOriginals: false,
|
||||||
|
});
|
||||||
|
const photo = lib.photos.byID({ fileID: 100 })!;
|
||||||
|
|
||||||
|
const result = await lib.backup();
|
||||||
|
|
||||||
|
expect(result.totalFiles).toBe(1);
|
||||||
|
expect(result.downloaded).toBe(1);
|
||||||
|
expect(result.failed).toBe(0);
|
||||||
|
expect(photo.savePath).toBe(
|
||||||
|
join(outDir, "2026", "2026-04", "2026-04-15", "2026-04-15.100.jpg"),
|
||||||
|
);
|
||||||
|
expect(photo.isLocal).toBe(true);
|
||||||
|
expect(existsSync(join(outDir, "2026", "2026-02"))).toBe(false);
|
||||||
|
const target = "../../2026/2026-04/2026-04-15/2026-04-15.100.jpg";
|
||||||
|
for (const album of ["Vacation", "Work"]) {
|
||||||
|
expect(
|
||||||
|
readlinkSync(join(outDir, "collections", album, "beach.jpg")),
|
||||||
|
).toBe(target);
|
||||||
|
}
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
|
it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
|
||||||
const lib = await openLibrary(stubSource());
|
const lib = await openLibrary(stubSource());
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
const originals = join(outDir, "originals");
|
const day = join(outDir, DAY);
|
||||||
const dest = join(originals, "100.jpg");
|
const dest = saved(outDir, "100.jpg");
|
||||||
// Only an original already in the cache is copied into the backup;
|
// Only an original already in the cache is copied into the backup;
|
||||||
// one fetched for the backup is written there by the download writer.
|
// one fetched for the backup is written there by the download writer.
|
||||||
await lib.photos.byID({ fileID: 100 })!.original();
|
await lib.photos.byID({ fileID: 100 })!.original();
|
||||||
@@ -609,34 +674,50 @@ describe("lib.backup", () => {
|
|||||||
const at = fsEvents.indexOf(`rename:${dest}`);
|
const at = fsEvents.indexOf(`rename:${dest}`);
|
||||||
expect(at).toBeGreaterThan(0);
|
expect(at).toBeGreaterThan(0);
|
||||||
expect(fsEvents[at - 1]).toMatch(
|
expect(fsEvents[at - 1]).toMatch(
|
||||||
/^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
|
/^sync:.*\/\.quak-backup-2026-03-01\.100\.jpg-\d+-[0-9a-z]*\.tmp$/,
|
||||||
);
|
);
|
||||||
expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
|
expect(fsEvents[at + 1]).toBe(`sync:${day}`);
|
||||||
lib.close();
|
lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("removes temp files left by a killed backup but not those of one still running", async () => {
|
it("removes temp files left by a killed backup but not those of one still running", async () => {
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
const originals = join(outDir, "originals");
|
const day = join(outDir, DAY);
|
||||||
mkdirSync(originals, { recursive: true });
|
mkdirSync(day, { recursive: true });
|
||||||
// A child that has already exited: its process ID is not running.
|
// A child that has already exited: its process ID is not running.
|
||||||
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||||
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
|
const leftover = `.quak-backup-2026-03-01.100.jpg-${exitedPID}-abc123.tmp`;
|
||||||
// This test's own process stands in for a backup running at the same
|
// This test's own process stands in for a backup running at the same
|
||||||
// time.
|
// time.
|
||||||
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
|
const inProgress = `.quak-backup-2026-03-01.101.jpg-${process.pid}-def456.tmp`;
|
||||||
writeFileSync(join(originals, leftover), "partial");
|
writeFileSync(join(day, leftover), "partial");
|
||||||
writeFileSync(join(originals, inProgress), "partial");
|
writeFileSync(join(day, inProgress), "partial");
|
||||||
const lib = await openLibrary(stubSource());
|
const lib = await openLibrary(stubSource());
|
||||||
|
|
||||||
await lib.backup({ downloadDirectory: outDir });
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
const names = readdirSync(originals);
|
const names = readdirSync(day);
|
||||||
expect(names).not.toContain(leftover);
|
expect(names).not.toContain(leftover);
|
||||||
expect(names).toContain(inProgress);
|
expect(names).toContain(inProgress);
|
||||||
lib.close();
|
lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("removes temp files left in a date folder no file in the backup is saved in", async () => {
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
// As for a file since deleted, or given another date, after a run was
|
||||||
|
// killed while writing it.
|
||||||
|
const otherDay = join(outDir, "2025", "2025-01", "2025-01-02");
|
||||||
|
mkdirSync(otherDay, { recursive: true });
|
||||||
|
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||||
|
writeFileSync(join(otherDay, `.quak-${exitedPID}-abc123.tmp`), "x");
|
||||||
|
const lib = await openLibrary(stubSource());
|
||||||
|
|
||||||
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(readdirSync(otherDay)).toEqual([]);
|
||||||
|
lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
|
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
const thumbnails = join(outDir, "thumbnails");
|
const thumbnails = join(outDir, "thumbnails");
|
||||||
@@ -709,7 +790,7 @@ describe("the refresh before a backup", () => {
|
|||||||
const result = await backup;
|
const result = await backup;
|
||||||
|
|
||||||
expect(result.totalFiles).toBe(4);
|
expect(result.totalFiles).toBe(4);
|
||||||
expect(existsSync(join(outDir, "originals", "300.jpg"))).toBe(true);
|
expect(existsSync(saved(outDir, "300.jpg"))).toBe(true);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -728,7 +809,7 @@ describe("the refresh before a backup", () => {
|
|||||||
|
|
||||||
expect(source.originalCalls).toBe(0);
|
expect(source.originalCalls).toBe(0);
|
||||||
expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore);
|
expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore);
|
||||||
expect(existsSync(join(outDir, "originals"))).toBe(false);
|
expect(existsSync(join(outDir, DAY))).toBe(false);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -819,13 +900,13 @@ describe("backup album folders", () => {
|
|||||||
expect(result.failed).toBe(0);
|
expect(result.failed).toBe(0);
|
||||||
expect(tree(outDir)).toEqual([
|
expect(tree(outDir)).toEqual([
|
||||||
"Trip (10)/",
|
"Trip (10)/",
|
||||||
"Trip (10)/IMG_0001 (1).JPG -> ../../originals/1.JPG",
|
`Trip (10)/IMG_0001 (1).JPG -> ${linkTo("1.JPG")}`,
|
||||||
"Trip (10)/IMG_0001 (2).JPG -> ../../originals/2.JPG",
|
`Trip (10)/IMG_0001 (2).JPG -> ${linkTo("2.JPG")}`,
|
||||||
"Trip (10)/img_0001 (4).jpg -> ../../originals/4.jpg",
|
`Trip (10)/img_0001 (4).jpg -> ${linkTo("4.jpg")}`,
|
||||||
"Trip (10)/other.jpg -> ../../originals/3.jpg",
|
`Trip (10)/other.jpg -> ${linkTo("3.jpg")}`,
|
||||||
"Trip (10).json",
|
"Trip (10).json",
|
||||||
"Trip (11)/",
|
"Trip (11)/",
|
||||||
"Trip (11)/other.jpg -> ../../originals/3.jpg",
|
`Trip (11)/other.jpg -> ${linkTo("3.jpg")}`,
|
||||||
"Trip (11).json",
|
"Trip (11).json",
|
||||||
]);
|
]);
|
||||||
expect(albumID(outDir, "Trip (10).json")).toBe(10);
|
expect(albumID(outDir, "Trip (10).json")).toBe(10);
|
||||||
@@ -858,14 +939,14 @@ describe("backup album folders", () => {
|
|||||||
expect(result.failed).toBe(0);
|
expect(result.failed).toBe(0);
|
||||||
expect(tree(outDir)).toEqual([
|
expect(tree(outDir)).toEqual([
|
||||||
"Trip (10)/",
|
"Trip (10)/",
|
||||||
"Trip (10)/IMG (6) (5).JPG -> ../../originals/5.JPG",
|
`Trip (10)/IMG (6) (5).JPG -> ${linkTo("5.JPG")}`,
|
||||||
"Trip (10)/IMG (6).JPG -> ../../originals/6.JPG",
|
`Trip (10)/IMG (6).JPG -> ${linkTo("6.JPG")}`,
|
||||||
"Trip (10)/IMG (7).JPG -> ../../originals/7.JPG",
|
`Trip (10)/IMG (7).JPG -> ${linkTo("7.JPG")}`,
|
||||||
"Trip (10).json",
|
"Trip (10).json",
|
||||||
"Trip (11)/",
|
"Trip (11)/",
|
||||||
"Trip (11)/a.jpg -> ../../originals/8.jpg",
|
`Trip (11)/a.jpg -> ${linkTo("8.jpg")}`,
|
||||||
"Trip (11) (12)/",
|
"Trip (11) (12)/",
|
||||||
"Trip (11) (12)/b.jpg -> ../../originals/9.jpg",
|
`Trip (11) (12)/b.jpg -> ${linkTo("9.jpg")}`,
|
||||||
"Trip (11) (12).json",
|
"Trip (11) (12).json",
|
||||||
"Trip (11).json",
|
"Trip (11).json",
|
||||||
]);
|
]);
|
||||||
@@ -931,13 +1012,13 @@ describe("backup album folders", () => {
|
|||||||
expect(scoped.failed).toBe(0);
|
expect(scoped.failed).toBe(0);
|
||||||
expect(before).toEqual([
|
expect(before).toEqual([
|
||||||
"Trip (10)/",
|
"Trip (10)/",
|
||||||
"Trip (10)/a.jpg -> ../../originals/1.jpg",
|
`Trip (10)/a.jpg -> ${linkTo("1.jpg")}`,
|
||||||
"Trip (10).json",
|
"Trip (10).json",
|
||||||
"Work/",
|
"Work/",
|
||||||
"Work/c.jpg -> ../../originals/3.jpg",
|
`Work/c.jpg -> ${linkTo("3.jpg")}`,
|
||||||
"Work.json",
|
"Work.json",
|
||||||
"trip (11)/",
|
"trip (11)/",
|
||||||
"trip (11)/b.jpg -> ../../originals/2.jpg",
|
`trip (11)/b.jpg -> ${linkTo("2.jpg")}`,
|
||||||
"trip (11).json",
|
"trip (11).json",
|
||||||
]);
|
]);
|
||||||
expect(tree(outDir)).toEqual(before);
|
expect(tree(outDir)).toEqual(before);
|
||||||
@@ -990,13 +1071,13 @@ describe("backup album folders", () => {
|
|||||||
"Mine/",
|
"Mine/",
|
||||||
"Mine/keep.txt",
|
"Mine/keep.txt",
|
||||||
"Office/",
|
"Office/",
|
||||||
"Office/a.jpg -> ../../originals/5.jpg",
|
`Office/a.jpg -> ${linkTo("5.jpg")}`,
|
||||||
"Office.json",
|
"Office.json",
|
||||||
"Trip/",
|
"Trip/",
|
||||||
"Trip/IMG_0001.JPG -> ../../originals/1.JPG",
|
`Trip/IMG_0001.JPG -> ${linkTo("1.JPG")}`,
|
||||||
"Trip/mine -> ../elsewhere",
|
"Trip/mine -> ../elsewhere",
|
||||||
"Trip/notes.txt",
|
"Trip/notes.txt",
|
||||||
"Trip/other.jpg -> ../../originals/3.jpg",
|
`Trip/other.jpg -> ${linkTo("3.jpg")}`,
|
||||||
"Trip.json",
|
"Trip.json",
|
||||||
"Work/",
|
"Work/",
|
||||||
"Work/keep.txt",
|
"Work/keep.txt",
|
||||||
@@ -1005,7 +1086,7 @@ describe("backup album folders", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
// One album backed up, then a folder the user made beside it holding a
|
// One album backed up, then a folder the user made beside it holding a
|
||||||
// symlink into originals/, with `json` (if given) as its sibling JSON.
|
// symlink to an original, with `json` (if given) as its sibling JSON.
|
||||||
const backupWithUserFolder = async (
|
const backupWithUserFolder = async (
|
||||||
json: string | undefined,
|
json: string | undefined,
|
||||||
): Promise<{ outDir: string; failed: number }> => {
|
): Promise<{ outDir: string; failed: number }> => {
|
||||||
@@ -1019,10 +1100,7 @@ describe("backup album folders", () => {
|
|||||||
await runBackup(lib, { downloadDirectory: outDir });
|
await runBackup(lib, { downloadDirectory: outDir });
|
||||||
const collectionsDir = join(outDir, "collections");
|
const collectionsDir = join(outDir, "collections");
|
||||||
mkdirSync(join(collectionsDir, "Mine"));
|
mkdirSync(join(collectionsDir, "Mine"));
|
||||||
symlinkSync(
|
symlinkSync(linkTo("1.jpg"), join(collectionsDir, "Mine", "a.jpg"));
|
||||||
"../../originals/1.jpg",
|
|
||||||
join(collectionsDir, "Mine", "a.jpg"),
|
|
||||||
);
|
|
||||||
if (json !== undefined) {
|
if (json !== undefined) {
|
||||||
writeFileSync(join(collectionsDir, "Mine.json"), json);
|
writeFileSync(join(collectionsDir, "Mine.json"), json);
|
||||||
}
|
}
|
||||||
@@ -1036,9 +1114,9 @@ describe("backup album folders", () => {
|
|||||||
expect(failed).toBe(0);
|
expect(failed).toBe(0);
|
||||||
expect(tree(outDir)).toEqual([
|
expect(tree(outDir)).toEqual([
|
||||||
"Mine/",
|
"Mine/",
|
||||||
"Mine/a.jpg -> ../../originals/1.jpg",
|
`Mine/a.jpg -> ${linkTo("1.jpg")}`,
|
||||||
"Trip/",
|
"Trip/",
|
||||||
"Trip/a.jpg -> ../../originals/1.jpg",
|
`Trip/a.jpg -> ${linkTo("1.jpg")}`,
|
||||||
"Trip.json",
|
"Trip.json",
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
@@ -1050,10 +1128,10 @@ describe("backup album folders", () => {
|
|||||||
expect(failed).toBe(0);
|
expect(failed).toBe(0);
|
||||||
expect(tree(outDir)).toEqual([
|
expect(tree(outDir)).toEqual([
|
||||||
"Mine/",
|
"Mine/",
|
||||||
"Mine/a.jpg -> ../../originals/1.jpg",
|
`Mine/a.jpg -> ${linkTo("1.jpg")}`,
|
||||||
"Mine.json",
|
"Mine.json",
|
||||||
"Trip/",
|
"Trip/",
|
||||||
"Trip/a.jpg -> ../../originals/1.jpg",
|
`Trip/a.jpg -> ${linkTo("1.jpg")}`,
|
||||||
"Trip.json",
|
"Trip.json",
|
||||||
]);
|
]);
|
||||||
expect(
|
expect(
|
||||||
@@ -1087,23 +1165,16 @@ describe("backup of live photos", () => {
|
|||||||
const open = (files: EnteFile[], bodies: Map<number, Uint8Array>) =>
|
const open = (files: EnteFile[], bodies: Map<number, Uint8Array>) =>
|
||||||
openLibrary(cdnSource(bodies), new TripClient(files));
|
openLibrary(cdnSource(bodies), new TripClient(files));
|
||||||
|
|
||||||
// What an earlier version stored for live photo 500: the ZIP under the
|
const stored = [
|
||||||
// image's name, and its link.
|
"2026-03-01.500.heic",
|
||||||
const earlierZIP = (outDir: string): void => {
|
"2026-03-01.500.json",
|
||||||
mkdirSync(join(outDir, "originals"), { recursive: true });
|
"2026-03-01.500.livephoto.json",
|
||||||
mkdirSync(join(outDir, "collections", "Trip"), { recursive: true });
|
"2026-03-01.500.mov",
|
||||||
writeFileSync(join(outDir, "originals", "500.HEIC"), livePhotoZip());
|
];
|
||||||
symlinkSync(
|
|
||||||
"../../originals/500.HEIC",
|
|
||||||
join(outDir, "collections", "Trip", "IMG_0500.HEIC"),
|
|
||||||
);
|
|
||||||
};
|
|
||||||
|
|
||||||
const stored = ["500.heic", "500.json", "500.livephoto.json", "500.mov"];
|
|
||||||
const linked = [
|
const linked = [
|
||||||
"Trip/",
|
"Trip/",
|
||||||
"Trip/IMG_0500.heic -> ../../originals/500.heic",
|
`Trip/IMG_0500.heic -> ${linkTo("500.heic")}`,
|
||||||
"Trip/IMG_0500.mov -> ../../originals/500.mov",
|
`Trip/IMG_0500.mov -> ${linkTo("500.mov")}`,
|
||||||
"Trip.json",
|
"Trip.json",
|
||||||
];
|
];
|
||||||
|
|
||||||
@@ -1113,16 +1184,15 @@ describe("backup of live photos", () => {
|
|||||||
);
|
);
|
||||||
const lib = await open([live], new Map([[500, body]]));
|
const lib = await open([live], new Map([[500, body]]));
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
const originals = join(outDir, "originals");
|
|
||||||
|
|
||||||
const result = await lib.backup({ downloadDirectory: outDir });
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
|
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
|
||||||
expect(readdirSync(originals).sort()).toEqual(stored);
|
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
|
||||||
expect(readFileSync(join(originals, "500.heic"))).toEqual(
|
expect(readFileSync(saved(outDir, "500.heic"))).toEqual(
|
||||||
Buffer.from(IMAGE),
|
Buffer.from(IMAGE),
|
||||||
);
|
);
|
||||||
expect(readFileSync(join(originals, "500.mov"))).toEqual(
|
expect(readFileSync(saved(outDir, "500.mov"))).toEqual(
|
||||||
Buffer.from(VIDEO),
|
Buffer.from(VIDEO),
|
||||||
);
|
);
|
||||||
expect(tree(outDir)).toEqual(linked);
|
expect(tree(outDir)).toEqual(linked);
|
||||||
@@ -1149,39 +1219,22 @@ describe("backup of live photos", () => {
|
|||||||
|
|
||||||
expect(tree(outDir)).toEqual([
|
expect(tree(outDir)).toEqual([
|
||||||
"Trip/",
|
"Trip/",
|
||||||
"Trip/IMG_0001 (500).heic -> ../../originals/500.heic",
|
`Trip/IMG_0001 (500).heic -> ${linkTo("500.heic")}`,
|
||||||
"Trip/IMG_0001 (500).mov -> ../../originals/500.mov",
|
`Trip/IMG_0001 (500).mov -> ${linkTo("500.mov")}`,
|
||||||
"Trip/IMG_0001 (501).heic -> ../../originals/501.heic",
|
`Trip/IMG_0001 (501).heic -> ${linkTo("501.heic")}`,
|
||||||
"Trip/IMG_0001 (501).mov -> ../../originals/501.mov",
|
`Trip/IMG_0001 (501).mov -> ${linkTo("501.mov")}`,
|
||||||
"Trip.json",
|
"Trip.json",
|
||||||
]);
|
]);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("replaces the ZIP an earlier version stored, and its link", async () => {
|
it("stores nothing for a live photo that fails its hash", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(
|
|
||||||
file(500, 10, "IMG_0500.HEIC"),
|
|
||||||
);
|
|
||||||
const outDir = join(root, "backup");
|
|
||||||
earlierZIP(outDir);
|
|
||||||
const lib = await open([live], new Map([[500, body]]));
|
|
||||||
|
|
||||||
const result = await lib.backup({ downloadDirectory: outDir });
|
|
||||||
|
|
||||||
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
|
|
||||||
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
|
|
||||||
expect(tree(outDir)).toEqual(linked);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("stores nothing for a live photo that fails its hash, and keeps what was there", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(
|
const { file: live, body } = await asLivePhoto(
|
||||||
file(500, 10, "IMG_0500.HEIC"),
|
file(500, 10, "IMG_0500.HEIC"),
|
||||||
livePhotoZip(),
|
livePhotoZip(),
|
||||||
"not:the recorded hash",
|
"not:the recorded hash",
|
||||||
);
|
);
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
earlierZIP(outDir);
|
|
||||||
const lib = await open([live], new Map([[500, body]]));
|
const lib = await open([live], new Map([[500, body]]));
|
||||||
|
|
||||||
const result = await lib.backup({ downloadDirectory: outDir });
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
@@ -1189,12 +1242,8 @@ describe("backup of live photos", () => {
|
|||||||
expect(result).toMatchObject({ downloaded: 0, failed: 1 });
|
expect(result).toMatchObject({ downloaded: 0, failed: 1 });
|
||||||
expect(result.errors.map((e) => e.fileID)).toEqual([500]);
|
expect(result.errors.map((e) => e.fileID)).toEqual([500]);
|
||||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
|
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
|
||||||
expect(readdirSync(join(outDir, "originals"))).toEqual(["500.HEIC"]);
|
expect(readdirSync(join(outDir, DAY))).toEqual([]);
|
||||||
expect(tree(outDir)).toEqual([
|
expect(tree(outDir)).toEqual(["Trip/", "Trip.json"]);
|
||||||
"Trip/",
|
|
||||||
"Trip/IMG_0500.HEIC -> ../../originals/500.HEIC",
|
|
||||||
"Trip.json",
|
|
||||||
]);
|
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -1209,8 +1258,8 @@ describe("backup of live photos", () => {
|
|||||||
const result = await lib.backup({ downloadDirectory: outDir });
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
|
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
|
||||||
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
|
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
|
||||||
expect(readFileSync(join(outDir, "originals", "500.mov"))).toEqual(
|
expect(readFileSync(saved(outDir, "500.mov"))).toEqual(
|
||||||
Buffer.from(VIDEO),
|
Buffer.from(VIDEO),
|
||||||
);
|
);
|
||||||
expect(tree(outDir)).toEqual(linked);
|
expect(tree(outDir)).toEqual(linked);
|
||||||
@@ -1219,23 +1268,6 @@ describe("backup of live photos", () => {
|
|||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("replaces an earlier ZIP and its link with the image and video the cache holds", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(
|
|
||||||
file(500, 10, "IMG_0500.HEIC"),
|
|
||||||
);
|
|
||||||
const lib = await open([live], new Map([[500, body]]));
|
|
||||||
await lib.photos.byID({ fileID: 500 })!.original();
|
|
||||||
const outDir = join(root, "backup");
|
|
||||||
earlierZIP(outDir);
|
|
||||||
|
|
||||||
const result = await lib.backup({ downloadDirectory: outDir });
|
|
||||||
|
|
||||||
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
|
|
||||||
expect(readdirSync(join(outDir, "originals")).sort()).toEqual(stored);
|
|
||||||
expect(tree(outDir)).toEqual(linked);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it.each(["missing", "empty"])(
|
it.each(["missing", "empty"])(
|
||||||
"fetches a live photo again when the video its JSON file names is %s",
|
"fetches a live photo again when the video its JSON file names is %s",
|
||||||
async (state) => {
|
async (state) => {
|
||||||
@@ -1245,7 +1277,7 @@ describe("backup of live photos", () => {
|
|||||||
const lib = await open([live], new Map([[500, body]]));
|
const lib = await open([live], new Map([[500, body]]));
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
await lib.backup({ downloadDirectory: outDir });
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
const video = join(outDir, "originals", "500.mov");
|
const video = saved(outDir, "500.mov");
|
||||||
if (state === "missing") rmSync(video);
|
if (state === "missing") rmSync(video);
|
||||||
else writeFileSync(video, "");
|
else writeFileSync(video, "");
|
||||||
|
|
||||||
@@ -1271,7 +1303,7 @@ describe("backup of live photos", () => {
|
|||||||
const lib = await open([live], new Map([[500, body]]));
|
const lib = await open([live], new Map([[500, body]]));
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
await lib.backup({ downloadDirectory: outDir });
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
const image = join(outDir, "originals", "500.heic");
|
const image = saved(outDir, "500.heic");
|
||||||
if (state === "missing") rmSync(image);
|
if (state === "missing") rmSync(image);
|
||||||
else writeFileSync(image, "");
|
else writeFileSync(image, "");
|
||||||
|
|
||||||
@@ -1310,8 +1342,8 @@ describe("backup of live photos", () => {
|
|||||||
const read = await reader.photos.byID({ fileID: 500 })!.original();
|
const read = await reader.photos.byID({ fileID: 500 })!.original();
|
||||||
|
|
||||||
expect(read).toEqual({
|
expect(read).toEqual({
|
||||||
path: join(outDir, "originals", "500.heic"),
|
path: saved(outDir, "500.heic"),
|
||||||
videoPath: join(outDir, "originals", "500.mov"),
|
videoPath: saved(outDir, "500.mov"),
|
||||||
bytes: IMAGE.length,
|
bytes: IMAGE.length,
|
||||||
});
|
});
|
||||||
await reader.close();
|
await reader.close();
|
||||||
|
|||||||
@@ -757,7 +757,7 @@ describe("backup", () => {
|
|||||||
expect(code).toBe(1);
|
expect(code).toBe(1);
|
||||||
expect(runText).toBe("quak: HTTP 401 from server\n");
|
expect(runText).toBe("quak: HTTP 401 from server\n");
|
||||||
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
||||||
expect(existsSync(join(dir, "originals"))).toBe(false);
|
expect(existsSync(dir)).toBe(false);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -3,14 +3,18 @@
|
|||||||
* `quak backup-metadata --exif` records.
|
* `quak backup-metadata --exif` records.
|
||||||
*
|
*
|
||||||
* The originals come from users' libraries, so a truncated or corrupt file
|
* The originals come from users' libraries, so a truncated or corrupt file
|
||||||
* must neither hang the read nor throw out of it: `readPhotoExif` gives `{}`,
|
* must neither hang the read nor throw out of it: `readAllExifTags` gives `{}`,
|
||||||
* and `backup-metadata` tells an EXIF block it cannot read apart from a file
|
* and `backup-metadata` tells an EXIF block it cannot read apart from a file
|
||||||
* that simply has no EXIF, carrying the reason in `exifError`. Each JPEG below
|
* that simply has no EXIF, carrying the reason in `exifError`. Each JPEG below
|
||||||
* is a short hand-built byte array; the HEIC is a real file.
|
* is a short hand-built byte array; the HEIC is a real file.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import { readPhotoExif } from "../../src/exif.js";
|
import {
|
||||||
|
readAllExifTags,
|
||||||
|
readExifTags,
|
||||||
|
readPhotoExif,
|
||||||
|
} from "../../src/exif.js";
|
||||||
import { extractImageMetadata } from "../../src/metadata-backup.js";
|
import { extractImageMetadata } from "../../src/metadata-backup.js";
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||||
|
|
||||||
@@ -63,6 +67,51 @@ const TIFF_ALTITUDE_WITHOUT_REF = [
|
|||||||
...[0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x02], // 25/2, at 44
|
...[0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x02], // 25/2, at 44
|
||||||
];
|
];
|
||||||
|
|
||||||
|
// A big-endian TIFF block holding a GPSLatitude of 33° 30' 0" and a
|
||||||
|
// GPSLatitudeRef of "S".
|
||||||
|
const TIFF_SOUTHERN_LATITUDE = [
|
||||||
|
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
||||||
|
// The first IFD, at 8: one entry, then no next IFD.
|
||||||
|
...[0x00, 0x01],
|
||||||
|
// The GPS IFD's offset (0x8825), LONG, 26.
|
||||||
|
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
|
||||||
|
...[0x00, 0x00, 0x00, 0x00],
|
||||||
|
// The GPS IFD, at 26: two entries, then no next IFD.
|
||||||
|
...[0x00, 0x02],
|
||||||
|
// GPSLatitudeRef (0x0001), 2 ASCII bytes: "S".
|
||||||
|
...[0x00, 0x01, 0x00, 0x02, 0x00, 0x00, 0x00, 0x02, 0x53, 0x00, 0x00, 0x00],
|
||||||
|
// GPSLatitude (0x0002), three RATIONALs at 56.
|
||||||
|
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
|
||||||
|
...[0x00, 0x00, 0x00, 0x00],
|
||||||
|
...[0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, 0x01], // 33/1, at 56
|
||||||
|
...[0x00, 0x00, 0x00, 0x1e, 0x00, 0x00, 0x00, 0x01], // 30/1
|
||||||
|
...[0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01], // 0/1
|
||||||
|
];
|
||||||
|
|
||||||
|
// A big-endian TIFF block holding a GPSLatitude of 40° 26' 46" and a
|
||||||
|
// GPSLongitude of 79° 58' 56", and neither GPSLatitudeRef nor GPSLongitudeRef.
|
||||||
|
const TIFF_POSITION_WITHOUT_REFS = [
|
||||||
|
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
||||||
|
// The first IFD, at 8: one entry, then no next IFD.
|
||||||
|
...[0x00, 0x01],
|
||||||
|
// The GPS IFD's offset (0x8825), LONG, 26.
|
||||||
|
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
|
||||||
|
...[0x00, 0x00, 0x00, 0x00],
|
||||||
|
// The GPS IFD, at 26: two entries, then no next IFD.
|
||||||
|
...[0x00, 0x02],
|
||||||
|
// GPSLatitude (0x0002), three RATIONALs at 56.
|
||||||
|
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
|
||||||
|
// GPSLongitude (0x0004), three RATIONALs at 80.
|
||||||
|
...[0x00, 0x04, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x50],
|
||||||
|
...[0x00, 0x00, 0x00, 0x00],
|
||||||
|
...[0x00, 0x00, 0x00, 0x28, 0x00, 0x00, 0x00, 0x01], // 40/1, at 56
|
||||||
|
...[0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x01], // 26/1
|
||||||
|
...[0x00, 0x00, 0x00, 0x2e, 0x00, 0x00, 0x00, 0x01], // 46/1
|
||||||
|
...[0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, 0x01], // 79/1, at 80
|
||||||
|
...[0x00, 0x00, 0x00, 0x3a, 0x00, 0x00, 0x00, 0x01], // 58/1
|
||||||
|
...[0x00, 0x00, 0x00, 0x38, 0x00, 0x00, 0x00, 0x01], // 56/1
|
||||||
|
];
|
||||||
|
|
||||||
// A big-endian TIFF block holding Orientation 6 and a Make whose value lies
|
// A big-endian TIFF block holding Orientation 6 and a Make whose value lies
|
||||||
// past the end of the file.
|
// past the end of the file.
|
||||||
const TIFF_MAKE_PAST_END = [
|
const TIFF_MAKE_PAST_END = [
|
||||||
@@ -87,6 +136,27 @@ const TIFF_UNNAMED_TAG = [
|
|||||||
...[0x00, 0x00, 0x00, 0x00],
|
...[0x00, 0x00, 0x00, 0x00],
|
||||||
];
|
];
|
||||||
|
|
||||||
|
// A big-endian TIFF block holding Orientation 6, and a thumbnail IFD holding
|
||||||
|
// its own Orientation 1 and a 4-byte JPEG thumbnail.
|
||||||
|
const TIFF_WITH_THUMBNAIL = [
|
||||||
|
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
||||||
|
// The first IFD, at 8: one entry, then the thumbnail IFD at 26.
|
||||||
|
...[0x00, 0x01],
|
||||||
|
// Orientation (0x0112), SHORT, 6.
|
||||||
|
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
|
||||||
|
...[0x00, 0x00, 0x00, 0x1a],
|
||||||
|
// The thumbnail IFD, at 26: three entries, then no next IFD.
|
||||||
|
...[0x00, 0x03],
|
||||||
|
// Orientation (0x0112), SHORT, 1.
|
||||||
|
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00],
|
||||||
|
// JPEGInterchangeFormat (0x0201), LONG: the thumbnail is at 68.
|
||||||
|
...[0x02, 0x01, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x44],
|
||||||
|
// JPEGInterchangeFormatLength (0x0202), LONG: 4 bytes.
|
||||||
|
...[0x02, 0x02, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04],
|
||||||
|
...[0x00, 0x00, 0x00, 0x00],
|
||||||
|
...[0xff, 0xd8, 0xff, 0xd9], // the thumbnail, at 68: an empty JPEG
|
||||||
|
];
|
||||||
|
|
||||||
// An APP1 segment whose length field matches its data.
|
// An APP1 segment whose length field matches its data.
|
||||||
const app1 = (data: number[]): number[] => {
|
const app1 = (data: number[]): number[] => {
|
||||||
const len = data.length + 2;
|
const len = data.length + 2;
|
||||||
@@ -96,31 +166,90 @@ const app1 = (data: number[]): number[] => {
|
|||||||
const bytes = (...parts: number[][]): Uint8Array =>
|
const bytes = (...parts: number[][]): Uint8Array =>
|
||||||
new Uint8Array(parts.flat());
|
new Uint8Array(parts.flat());
|
||||||
|
|
||||||
|
describe("readAllExifTags", () => {
|
||||||
|
it("keys a tag exifreader has no name for by its number", () => {
|
||||||
|
const data = [...EXIF_HEADER, ...TIFF_UNNAMED_TAG];
|
||||||
|
expect(readAllExifTags(bytes(SOI, app1(data), SOS))).toStrictEqual({
|
||||||
|
"undefined-49152": {
|
||||||
|
id: 49152,
|
||||||
|
value: 7,
|
||||||
|
description: 7,
|
||||||
|
computed: 7,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("puts the thumbnail's tags under Thumbnail, without its image", () => {
|
||||||
|
const input = bytes(
|
||||||
|
SOI,
|
||||||
|
app1([...EXIF_HEADER, ...TIFF_WITH_THUMBNAIL]),
|
||||||
|
SOS,
|
||||||
|
);
|
||||||
|
// exifreader finds the thumbnail's image.
|
||||||
|
expect(readExifTags(input)?.Thumbnail?.type).toBe("image/jpeg");
|
||||||
|
const tags = readAllExifTags(input);
|
||||||
|
expect(tags.Orientation?.value).toBe(6);
|
||||||
|
expect(Object.keys(tags.Thumbnail ?? {}).sort()).toEqual([
|
||||||
|
"JPEGInterchangeFormat",
|
||||||
|
"JPEGInterchangeFormatLength",
|
||||||
|
"Orientation",
|
||||||
|
]);
|
||||||
|
expect(tags.Thumbnail?.Orientation?.value).toBe(1);
|
||||||
|
expect(readPhotoExif(tags)).toStrictEqual({ orientation: 6 });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("readPhotoExif", () => {
|
describe("readPhotoExif", () => {
|
||||||
it("reads the common fields of a valid JPEG", () => {
|
it("reads the common fields of a valid JPEG", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
|
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
|
||||||
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
|
expect(
|
||||||
|
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
||||||
|
).toStrictEqual({
|
||||||
orientation: 6,
|
orientation: 6,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("gives no dateTimeOriginal for a DateTimeOriginal of 0000:00:00 00:00:00", () => {
|
it("gives no dateTimeOriginal for a DateTimeOriginal of 0000:00:00 00:00:00", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_UNSET_DATE];
|
const data = [...EXIF_HEADER, ...TIFF_UNSET_DATE];
|
||||||
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
|
expect(
|
||||||
|
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
||||||
|
).toStrictEqual({
|
||||||
orientation: 6,
|
orientation: 6,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("reads a GPSAltitude without GPSAltitudeRef as above sea level", () => {
|
it("reads a GPSAltitude without GPSAltitudeRef as above sea level", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_ALTITUDE_WITHOUT_REF];
|
const data = [...EXIF_HEADER, ...TIFF_ALTITUDE_WITHOUT_REF];
|
||||||
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
|
expect(
|
||||||
|
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
||||||
|
).toStrictEqual({
|
||||||
gpsAltitude: 12.5,
|
gpsAltitude: 12.5,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("reads a GPSLatitude with GPSLatitudeRef S as south of the equator", () => {
|
||||||
|
const data = [...EXIF_HEADER, ...TIFF_SOUTHERN_LATITUDE];
|
||||||
|
expect(
|
||||||
|
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
||||||
|
).toStrictEqual({
|
||||||
|
gpsLatitude: -33.5,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("gives no gpsLatitude or gpsLongitude without their reference tags", () => {
|
||||||
|
const data = [...EXIF_HEADER, ...TIFF_POSITION_WITHOUT_REFS];
|
||||||
|
const tags = readAllExifTags(bytes(SOI, app1(data), SOS));
|
||||||
|
// The position is read; only its hemisphere is unknown.
|
||||||
|
expect(tags.GPSLatitude?.computed).toStrictEqual([40, 26, 46]);
|
||||||
|
expect(tags.GPSLongitude?.computed).toStrictEqual([79, 58, 56]);
|
||||||
|
expect(readPhotoExif(tags)).toStrictEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
it("gives no make for a Make whose value lies past the end of the file", () => {
|
it("gives no make for a Make whose value lies past the end of the file", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_MAKE_PAST_END];
|
const data = [...EXIF_HEADER, ...TIFF_MAKE_PAST_END];
|
||||||
expect(readPhotoExif(bytes(SOI, app1(data), SOS))).toStrictEqual({
|
expect(
|
||||||
|
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
||||||
|
).toStrictEqual({
|
||||||
orientation: 6,
|
orientation: 6,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
@@ -175,8 +304,9 @@ describe("readPhotoExif", () => {
|
|||||||
"an EXIF block that cannot be parsed",
|
"an EXIF block that cannot be parsed",
|
||||||
bytes(SOI, app1([...EXIF_HEADER, 0x58, 0x58]), SOS),
|
bytes(SOI, app1([...EXIF_HEADER, 0x58, 0x58]), SOS),
|
||||||
],
|
],
|
||||||
])("returns no fields for %s", (_, input) => {
|
])("returns no tags and no fields for %s", (_, input) => {
|
||||||
expect(readPhotoExif(input)).toStrictEqual({});
|
expect(readAllExifTags(input)).toStrictEqual({});
|
||||||
|
expect(readPhotoExif(readAllExifTags(input))).toStrictEqual({});
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -1883,15 +1883,6 @@ describe("downloadFile live photos", () => {
|
|||||||
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
|
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("replaces what was at the destination, such as an earlier ZIP of the two", async () => {
|
|
||||||
const t = setup(livePhotoZip(), livePhoto);
|
|
||||||
writeFileSync(t.outPath, livePhotoZip());
|
|
||||||
|
|
||||||
await t.run();
|
|
||||||
|
|
||||||
expect(readdirSync(t.dir).sort()).toEqual(["f.heic", "f.mov"]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("renames the image and then the video into place, each from its own temp file", async () => {
|
it("renames the image and then the video into place, each from its own temp file", async () => {
|
||||||
const t = setup(livePhotoZip(), livePhoto);
|
const t = setup(livePhotoZip(), livePhoto);
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,303 @@
|
|||||||
|
/**
|
||||||
|
* The example script `examples/download-albums.ts` (issue #144), run twice
|
||||||
|
* against a stand-in account, as a user would run it twice.
|
||||||
|
*
|
||||||
|
* The account has two albums sharing one photo, and one of its photos is a
|
||||||
|
* live photo whose image is a HEIC with EXIF. The first run puts every
|
||||||
|
* original at its save path, writes each photo's metadata beside it and each
|
||||||
|
* album's photos under `albums/`. Before the second run the account gains an
|
||||||
|
* album holding a new photo. The second run downloads that photo and writes
|
||||||
|
* its files and the new album's, and fetches nothing else and changes no
|
||||||
|
* other file.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||||
|
import {
|
||||||
|
existsSync,
|
||||||
|
mkdtempSync,
|
||||||
|
readdirSync,
|
||||||
|
readFileSync,
|
||||||
|
rmSync,
|
||||||
|
statSync,
|
||||||
|
utimesSync,
|
||||||
|
writeFileSync,
|
||||||
|
} from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
|
||||||
|
import { downloadAlbums } from "../../examples/download-albums.js";
|
||||||
|
import { Library, type ContentSource } from "../../src/index.js";
|
||||||
|
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||||
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
|
import { readAllExifTags } from "../../src/exif.js";
|
||||||
|
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||||
|
import {
|
||||||
|
asLivePhoto,
|
||||||
|
cdnSource,
|
||||||
|
livePhotoHash,
|
||||||
|
livePhotoZip,
|
||||||
|
VIDEO,
|
||||||
|
} from "../live-photo.js";
|
||||||
|
|
||||||
|
const USER_ID = 7;
|
||||||
|
|
||||||
|
// Every photo is taken at noon local time on 2026-03-01, so the machine's time
|
||||||
|
// zone cannot move it to another day; it is saved in the folder `DAY`. Ente
|
||||||
|
// stores times in microseconds.
|
||||||
|
const TAKEN_MS = new Date(2026, 2, 1, 12).getTime();
|
||||||
|
const DAY = join("2026", "2026-03", "2026-03-01");
|
||||||
|
|
||||||
|
const collection = (id: number, name: string): Collection => ({
|
||||||
|
id,
|
||||||
|
ownerID: USER_ID,
|
||||||
|
key: new Uint8Array([id]),
|
||||||
|
name,
|
||||||
|
type: "album",
|
||||||
|
updationTime: 1,
|
||||||
|
isShared: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
const file = (id: number, collectionID: number): EnteFile => ({
|
||||||
|
id,
|
||||||
|
collectionID,
|
||||||
|
ownerID: USER_ID,
|
||||||
|
key: new Uint8Array([id]),
|
||||||
|
metadata: {
|
||||||
|
title: `file-${id}.jpg`,
|
||||||
|
fileType: "image",
|
||||||
|
creationTime: TAKEN_MS * 1000,
|
||||||
|
modificationTime: TAKEN_MS * 1000,
|
||||||
|
},
|
||||||
|
file: { decryptionHeader: "aGVhZGVy" },
|
||||||
|
thumbnail: { decryptionHeader: "dGh1bWI=" },
|
||||||
|
updationTime: 1,
|
||||||
|
});
|
||||||
|
|
||||||
|
let root: string;
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
root = mkdtempSync(join(tmpdir(), "quak-download-albums-"));
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
if (root && existsSync(root))
|
||||||
|
rmSync(root, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
// Every file and directory under `dir`, by path.
|
||||||
|
const entries = (dir: string): string[] =>
|
||||||
|
readdirSync(dir, { recursive: true, encoding: "utf-8" });
|
||||||
|
|
||||||
|
// Set the modification time of everything under `dir` to the epoch, so that
|
||||||
|
// anything written there afterwards has a later one, however soon it comes.
|
||||||
|
const backdate = (dir: string): void => {
|
||||||
|
for (const name of entries(dir)) utimesSync(join(dir, name), 0, 0);
|
||||||
|
};
|
||||||
|
|
||||||
|
// The modification time of everything under `dir`, by path.
|
||||||
|
const mtimes = (dir: string): Map<string, number> =>
|
||||||
|
new Map(
|
||||||
|
entries(dir).map((name) => [name, statSync(join(dir, name)).mtimeMs]),
|
||||||
|
);
|
||||||
|
|
||||||
|
const readJSON = (path: string): unknown =>
|
||||||
|
JSON.parse(readFileSync(path, "utf-8"));
|
||||||
|
|
||||||
|
describe("examples/download-albums.ts", () => {
|
||||||
|
it("downloads every album's photos with their metadata, and on a second run only what the account gained", async () => {
|
||||||
|
// Album 1, "Trip", holds photos 1 and 2. Album 2, "Family", holds
|
||||||
|
// photo 2 and photo 3, a live photo.
|
||||||
|
const live = await asLivePhoto(
|
||||||
|
file(3, 2),
|
||||||
|
livePhotoZip({ "image.heic": HEIC_WITH_EXIF, "video.mov": VIDEO }),
|
||||||
|
livePhotoHash(HEIC_WITH_EXIF, VIDEO),
|
||||||
|
);
|
||||||
|
const collections = [collection(1, "Trip"), collection(2, "Family")];
|
||||||
|
const filesByAlbum = new Map<number, EnteFile[]>([
|
||||||
|
[1, [file(1, 1), file(2, 1)]],
|
||||||
|
[2, [file(2, 2), live.file]],
|
||||||
|
]);
|
||||||
|
const client = {
|
||||||
|
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
|
||||||
|
collectionsSince: async (): Promise<CollectionsPage> => ({
|
||||||
|
collections: [...collections],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
}),
|
||||||
|
filesSince: async (args: {
|
||||||
|
collectionID: number;
|
||||||
|
}): Promise<FilesPage> => ({
|
||||||
|
files: filesByAlbum.get(args.collectionID) ?? [],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
|
||||||
|
// Photo 3 comes from a stand-in server, encrypted as Ente serves a
|
||||||
|
// live photo. Any other original is a few bytes naming its photo.
|
||||||
|
// `calls` counts every fetch, thumbnails included.
|
||||||
|
const server = cdnSource(new Map([[3, live.body]]));
|
||||||
|
let calls = 0;
|
||||||
|
const source: ContentSource = {
|
||||||
|
original: async (args) => {
|
||||||
|
calls++;
|
||||||
|
if (args.file.id === 3) return server.original(args);
|
||||||
|
const bytes = `original-${args.file.id}`;
|
||||||
|
writeFileSync(args.destination, bytes);
|
||||||
|
return { bytesWritten: bytes.length };
|
||||||
|
},
|
||||||
|
thumbnail: async (args) => {
|
||||||
|
calls++;
|
||||||
|
return server.thumbnail(args);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const dir = join(root, "photos");
|
||||||
|
const open = (): Promise<Library> =>
|
||||||
|
Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory: join(root, "cache"),
|
||||||
|
downloadDirectory: dir,
|
||||||
|
contentSource: source,
|
||||||
|
refreshIntervalSeconds: 3600,
|
||||||
|
precacheThumbnails: false,
|
||||||
|
precacheOriginals: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
const first = await open();
|
||||||
|
// The cache already holds photo 1's original, so its record names a
|
||||||
|
// cache path, which the metadata leaves out. download() copies it
|
||||||
|
// from the cache rather than fetching it again.
|
||||||
|
await first.photos.byID({ fileID: 1 })!.original();
|
||||||
|
expect(await downloadAlbums(first, dir)).toEqual({
|
||||||
|
downloaded: 3,
|
||||||
|
alreadyLocal: 0,
|
||||||
|
});
|
||||||
|
await first.close();
|
||||||
|
expect(calls).toBe(3);
|
||||||
|
|
||||||
|
// Each original at its save path, a live photo as its image, its video
|
||||||
|
// and the file naming them, and each photo's metadata beside it.
|
||||||
|
const day = join(dir, DAY);
|
||||||
|
expect(readdirSync(day).sort()).toEqual([
|
||||||
|
"2026-03-01.1.jpg",
|
||||||
|
"2026-03-01.1.jpg.json",
|
||||||
|
"2026-03-01.2.jpg",
|
||||||
|
"2026-03-01.2.jpg.json",
|
||||||
|
"2026-03-01.3.heic",
|
||||||
|
"2026-03-01.3.heic.json",
|
||||||
|
"2026-03-01.3.livephoto.json",
|
||||||
|
"2026-03-01.3.mov",
|
||||||
|
]);
|
||||||
|
expect(readFileSync(join(day, "2026-03-01.1.jpg"), "utf-8")).toBe(
|
||||||
|
"original-1",
|
||||||
|
);
|
||||||
|
expect(readFileSync(join(day, "2026-03-01.2.jpg"), "utf-8")).toBe(
|
||||||
|
"original-2",
|
||||||
|
);
|
||||||
|
expect(readFileSync(join(day, "2026-03-01.3.heic"))).toEqual(
|
||||||
|
Buffer.from(HEIC_WITH_EXIF),
|
||||||
|
);
|
||||||
|
expect(readFileSync(join(day, "2026-03-01.3.mov"))).toEqual(
|
||||||
|
Buffer.from(VIDEO),
|
||||||
|
);
|
||||||
|
|
||||||
|
// The metadata is the photo's record and its EXIF tags. The originals
|
||||||
|
// of photos 1 and 2 are not image data, so they have no EXIF tags.
|
||||||
|
// Photo 3's are every tag of its image, as `photo.exif()` returns
|
||||||
|
// them. `record` holds the fields the three records share.
|
||||||
|
const record = {
|
||||||
|
takenAt: TAKEN_MS,
|
||||||
|
modifiedAt: TAKEN_MS,
|
||||||
|
fileType: "image",
|
||||||
|
isArchived: false,
|
||||||
|
isHidden: false,
|
||||||
|
};
|
||||||
|
expect(readJSON(join(day, "2026-03-01.1.jpg.json"))).toEqual({
|
||||||
|
...record,
|
||||||
|
fileID: 1,
|
||||||
|
albumIDs: [1],
|
||||||
|
title: "file-1.jpg",
|
||||||
|
exif: {},
|
||||||
|
});
|
||||||
|
expect(readJSON(join(day, "2026-03-01.2.jpg.json"))).toEqual({
|
||||||
|
...record,
|
||||||
|
fileID: 2,
|
||||||
|
albumIDs: [1, 2],
|
||||||
|
title: "file-2.jpg",
|
||||||
|
exif: {},
|
||||||
|
});
|
||||||
|
expect(readJSON(join(day, "2026-03-01.3.heic.json"))).toEqual({
|
||||||
|
...record,
|
||||||
|
fileID: 3,
|
||||||
|
albumIDs: [2],
|
||||||
|
title: "file-3.jpg",
|
||||||
|
fileType: "livePhoto",
|
||||||
|
hash: livePhotoHash(HEIC_WITH_EXIF, VIDEO),
|
||||||
|
exif: readAllExifTags(HEIC_WITH_EXIF),
|
||||||
|
});
|
||||||
|
|
||||||
|
// Each album's photos, newest first, by save path relative to `dir`.
|
||||||
|
// Photo 2 is in both.
|
||||||
|
expect(readdirSync(join(dir, "albums")).sort()).toEqual([
|
||||||
|
"1.json",
|
||||||
|
"2.json",
|
||||||
|
]);
|
||||||
|
expect(readJSON(join(dir, "albums", "1.json"))).toEqual({
|
||||||
|
collectionID: 1,
|
||||||
|
name: "Trip",
|
||||||
|
savePaths: [
|
||||||
|
join(DAY, "2026-03-01.2.jpg"),
|
||||||
|
join(DAY, "2026-03-01.1.jpg"),
|
||||||
|
],
|
||||||
|
});
|
||||||
|
expect(readJSON(join(dir, "albums", "2.json"))).toEqual({
|
||||||
|
collectionID: 2,
|
||||||
|
name: "Family",
|
||||||
|
savePaths: [
|
||||||
|
join(DAY, "2026-03-01.3.heic"),
|
||||||
|
join(DAY, "2026-03-01.2.jpg"),
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
// Before the second run the account gains album 3, "Garden", holding
|
||||||
|
// a new photo 4. The second run, with a newly opened library, opens
|
||||||
|
// the cache written by the first and still downloads photo 4. It finds
|
||||||
|
// the other photos already local and fetches nothing else.
|
||||||
|
collections.push(collection(3, "Garden"));
|
||||||
|
filesByAlbum.set(3, [file(4, 3)]);
|
||||||
|
backdate(dir);
|
||||||
|
const before = mtimes(dir);
|
||||||
|
const second = await open();
|
||||||
|
expect(await downloadAlbums(second, dir)).toEqual({
|
||||||
|
downloaded: 1,
|
||||||
|
alreadyLocal: 3,
|
||||||
|
});
|
||||||
|
await second.close();
|
||||||
|
expect(calls).toBe(4);
|
||||||
|
expect(readFileSync(join(day, "2026-03-01.4.jpg"), "utf-8")).toBe(
|
||||||
|
"original-4",
|
||||||
|
);
|
||||||
|
expect(readJSON(join(dir, "albums", "3.json"))).toEqual({
|
||||||
|
collectionID: 3,
|
||||||
|
name: "Garden",
|
||||||
|
savePaths: [join(DAY, "2026-03-01.4.jpg")],
|
||||||
|
});
|
||||||
|
|
||||||
|
// The second run adds only photo 4's files and album 3's, and
|
||||||
|
// rewrites, renames or removes no file from the first run. The two
|
||||||
|
// directories that gain a file are the only other changes.
|
||||||
|
const after = mtimes(dir);
|
||||||
|
expect(
|
||||||
|
[...after.keys()].filter((name) => !before.has(name)).sort(),
|
||||||
|
).toEqual([
|
||||||
|
join(DAY, "2026-03-01.4.jpg"),
|
||||||
|
join(DAY, "2026-03-01.4.jpg.json"),
|
||||||
|
join("albums", "3.json"),
|
||||||
|
]);
|
||||||
|
for (const [name, mtime] of before) {
|
||||||
|
if (name === DAY || name === "albums") continue;
|
||||||
|
expect(after.get(name), name).toBe(mtime);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -142,6 +142,7 @@ const buildCache = (args: {
|
|||||||
pools: new RequestPools(),
|
pools: new RequestPools(),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
|
downloadDirectory: join(root, "photos"),
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
statfs: args.statfs,
|
statfs: args.statfs,
|
||||||
cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes,
|
cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes,
|
||||||
|
|||||||
@@ -6,16 +6,18 @@
|
|||||||
* `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and
|
* `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and
|
||||||
* a cached path shows up on the projected record. A library opened without a
|
* a cached path shows up on the projected record. A library opened without a
|
||||||
* content source leaves those methods throwing rather than silently doing
|
* content source leaves those methods throwing rather than silently doing
|
||||||
* nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `content()` and
|
* nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `download()`,
|
||||||
* `exif()`.
|
* `content()`, `exif()` and the methods that each return one EXIF field.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||||
import {
|
import {
|
||||||
mkdtempSync,
|
mkdtempSync,
|
||||||
readdirSync,
|
readdirSync,
|
||||||
|
readFileSync,
|
||||||
rmSync,
|
rmSync,
|
||||||
existsSync,
|
existsSync,
|
||||||
|
statSync,
|
||||||
writeFileSync,
|
writeFileSync,
|
||||||
} from "node:fs";
|
} from "node:fs";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
@@ -25,7 +27,7 @@ import { Library, type LibraryOptions } from "../../src/library/index.js";
|
|||||||
import type { ContentSource } from "../../src/library/content.js";
|
import type { ContentSource } from "../../src/library/content.js";
|
||||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
import type { PhotoExif } from "../../src/exif.js";
|
import { readPhotoExif, type PhotoExif } from "../../src/exif.js";
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||||
import {
|
import {
|
||||||
asLivePhoto,
|
asLivePhoto,
|
||||||
@@ -38,6 +40,12 @@ import {
|
|||||||
|
|
||||||
const USER_ID = 7;
|
const USER_ID = 7;
|
||||||
|
|
||||||
|
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
|
||||||
|
// stores times, so the machine's time zone cannot move it to another day; it
|
||||||
|
// is saved in the folder `DAY`.
|
||||||
|
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
|
||||||
|
const DAY = join("2026", "2026-03", "2026-03-01");
|
||||||
|
|
||||||
const collection = (id: number): Collection => ({
|
const collection = (id: number): Collection => ({
|
||||||
id,
|
id,
|
||||||
ownerID: USER_ID,
|
ownerID: USER_ID,
|
||||||
@@ -56,7 +64,7 @@ const file = (id: number, collectionID: number): EnteFile => ({
|
|||||||
metadata: {
|
metadata: {
|
||||||
title: `file-${id}.jpg`,
|
title: `file-${id}.jpg`,
|
||||||
fileType: "image",
|
fileType: "image",
|
||||||
creationTime: 1,
|
creationTime: TAKEN,
|
||||||
modificationTime: 1,
|
modificationTime: 1,
|
||||||
},
|
},
|
||||||
file: { decryptionHeader: "aGVhZGVy" },
|
file: { decryptionHeader: "aGVhZGVy" },
|
||||||
@@ -168,7 +176,7 @@ describe("Library content wiring", () => {
|
|||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("removes a live photo's ZIP an earlier version cached when it opens, and precaches its image and video", async () => {
|
it("does not take a live photo's image or video with no JSON file as its original when it opens, and precaches both", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(file(1, 1));
|
const { file: live, body } = await asLivePhoto(file(1, 1));
|
||||||
class LiveClient extends MockClient {
|
class LiveClient extends MockClient {
|
||||||
override async filesSince(): Promise<FilesPage> {
|
override async filesSince(): Promise<FilesPage> {
|
||||||
@@ -189,7 +197,8 @@ describe("Library content wiring", () => {
|
|||||||
// A first run records the library, so the next one knows that file 1
|
// A first run records the library, so the next one knows that file 1
|
||||||
// is a live photo when it opens the cache.
|
// is a live photo when it opens the cache.
|
||||||
await (await open({})).close();
|
await (await open({})).close();
|
||||||
writeFileSync(join(originals, "1.jpg"), livePhotoZip());
|
writeFileSync(join(originals, "1.heic"), "an image");
|
||||||
|
writeFileSync(join(originals, "1.mov"), "a video");
|
||||||
|
|
||||||
let precached!: () => void;
|
let precached!: () => void;
|
||||||
const done = new Promise<void>((r) => (precached = r));
|
const done = new Promise<void>((r) => (precached = r));
|
||||||
@@ -200,7 +209,9 @@ describe("Library content wiring", () => {
|
|||||||
precached();
|
precached();
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
expect(existsSync(join(originals, "1.jpg"))).toBe(false);
|
expect(
|
||||||
|
lib.photos.byID({ fileID: 1 })!.record().originalPath,
|
||||||
|
).toBeUndefined();
|
||||||
await done;
|
await done;
|
||||||
|
|
||||||
expect(readdirSync(originals).sort()).toEqual([
|
expect(readdirSync(originals).sort()).toEqual([
|
||||||
@@ -208,6 +219,17 @@ describe("Library content wiring", () => {
|
|||||||
"1.livephoto.json",
|
"1.livephoto.json",
|
||||||
"1.mov",
|
"1.mov",
|
||||||
]);
|
]);
|
||||||
|
expect(readFileSync(join(originals, "1.heic"))).toEqual(
|
||||||
|
Buffer.from(IMAGE),
|
||||||
|
);
|
||||||
|
expect(readFileSync(join(originals, "1.mov"))).toEqual(
|
||||||
|
Buffer.from(VIDEO),
|
||||||
|
);
|
||||||
|
expect(
|
||||||
|
JSON.parse(
|
||||||
|
readFileSync(join(originals, "1.livephoto.json"), "utf-8"),
|
||||||
|
),
|
||||||
|
).toEqual({ image: "1.heic", video: "1.mov" });
|
||||||
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
|
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
|
||||||
join(originals, "1.heic"),
|
join(originals, "1.heic"),
|
||||||
);
|
);
|
||||||
@@ -224,6 +246,9 @@ describe("Library content wiring", () => {
|
|||||||
await expect(
|
await expect(
|
||||||
lib.photos.byID({ fileID: 1 })!.thumbnail(),
|
lib.photos.byID({ fileID: 1 })!.thumbnail(),
|
||||||
).rejects.toThrow(/content cache/i);
|
).rejects.toThrow(/content cache/i);
|
||||||
|
await expect(
|
||||||
|
lib.photos.byID({ fileID: 1 })!.download(),
|
||||||
|
).rejects.toThrow(/content cache/i);
|
||||||
await expect(
|
await expect(
|
||||||
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
||||||
).rejects.toThrow(/content cache/i);
|
).rejects.toThrow(/content cache/i);
|
||||||
@@ -248,10 +273,10 @@ const entry = (
|
|||||||
value: number[],
|
value: number[],
|
||||||
): number[] => [...u16(tag), ...u16(type), ...u32(count), ...value];
|
): number[] => [...u16(tag), ...u16(type), ...u32(count), ...value];
|
||||||
|
|
||||||
// The TIFF block of a JPEG's EXIF segment, holding every field `exif()` picks:
|
// The TIFF block of a JPEG's EXIF segment, holding every field `Photo`'s typed
|
||||||
// the camera in the first IFD, the exposure in the Exif IFD, and a GPS position
|
// methods return: the camera in the first IFD, the exposure in the Exif IFD,
|
||||||
// of 40°26'46" N, 79°58'56" W, 12.5 m below sea level. Offsets count from the
|
// and a GPS position of 40°26'46" N, 79°58'56" W, 12.5 m below sea level.
|
||||||
// start of this block.
|
// Offsets count from the start of this block.
|
||||||
const TIFF = [
|
const TIFF = [
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a], // big-endian TIFF
|
...[0x4d, 0x4d, 0x00, 0x2a], // big-endian TIFF
|
||||||
...u32(8), // the first IFD's offset
|
...u32(8), // the first IFD's offset
|
||||||
@@ -341,7 +366,7 @@ describe("Photo save path, local copy, content and EXIF", () => {
|
|||||||
it("names where a backup writes the original, which is local once the backup has written it", async () => {
|
it("names where a backup writes the original, which is local once the backup has written it", async () => {
|
||||||
const lib = await open();
|
const lib = await open();
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
const savePath = join(root, "backup", "originals", "1.jpg");
|
const savePath = join(root, "backup", DAY, "2026-03-01.1.jpg");
|
||||||
expect(photo.savePath).toBe(savePath);
|
expect(photo.savePath).toBe(savePath);
|
||||||
expect(photo.isLocal).toBe(false);
|
expect(photo.isLocal).toBe(false);
|
||||||
|
|
||||||
@@ -359,23 +384,150 @@ describe("Photo save path, local copy, content and EXIF", () => {
|
|||||||
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
|
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
|
||||||
true,
|
true,
|
||||||
);
|
);
|
||||||
expect(existsSync(photo.savePath!)).toBe(false);
|
expect(existsSync(photo.savePath)).toBe(false);
|
||||||
expect(photo.isLocal).toBe(false);
|
expect(photo.isLocal).toBe(false);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("has no save path and is not local without a download directory", async () => {
|
it("saves under photos/ in the working directory at open without a download directory", async () => {
|
||||||
|
const cwd = vi.spyOn(process, "cwd").mockReturnValue(root);
|
||||||
const lib = await open({ downloadDirectory: undefined });
|
const lib = await open({ downloadDirectory: undefined });
|
||||||
|
cwd.mockRestore();
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
expect(photo.savePath).toBeUndefined();
|
expect(lib.downloadDirectory).toBe(join(root, "photos"));
|
||||||
|
expect(photo.savePath).toBe(
|
||||||
|
join(root, "photos", DAY, "2026-03-01.1.jpg"),
|
||||||
|
);
|
||||||
expect(photo.isLocal).toBe(false);
|
expect(photo.isLocal).toBe(false);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("has no save path and is not local without a content source", async () => {
|
it("refuses an empty download directory", async () => {
|
||||||
|
await expect(open({ downloadDirectory: "" })).rejects.toThrow(
|
||||||
|
/downloadDirectory is empty/,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps a photo's save path after a refresh removes its file", async () => {
|
||||||
|
// The same account, whose album is deleted on the second refresh.
|
||||||
|
class AlbumDeletedClient extends MockClient {
|
||||||
|
override async collectionsSince(): Promise<CollectionsPage> {
|
||||||
|
if (!this.served) return super.collectionsSince();
|
||||||
|
return { collections: [], deleted: [1], cursor: 2 };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const lib = await open({ client: new AlbumDeletedClient() });
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
await photo.download();
|
||||||
|
|
||||||
|
await lib.fresh();
|
||||||
|
|
||||||
|
expect(lib.photos.byID({ fileID: 1 })).toBeUndefined();
|
||||||
|
expect(photo.savePath).toBe(
|
||||||
|
join(root, "backup", DAY, "2026-03-01.1.jpg"),
|
||||||
|
);
|
||||||
|
expect(photo.isLocal).toBe(true);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("dates the save path and download() by the membership its takenAt comes from", async () => {
|
||||||
|
// One file in two albums. The date edited in Ente has reached album 2's
|
||||||
|
// copy, synced later, but album 1 still has an earlier edit.
|
||||||
|
const older = file(1, 1);
|
||||||
|
older.pubMagicMetadata = {
|
||||||
|
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
|
||||||
|
};
|
||||||
|
const newer = file(1, 2);
|
||||||
|
newer.updationTime = 2;
|
||||||
|
newer.pubMagicMetadata = {
|
||||||
|
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
|
||||||
|
};
|
||||||
|
const client = {
|
||||||
|
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
|
||||||
|
collectionsSince: async (): Promise<CollectionsPage> => ({
|
||||||
|
collections: [collection(1), collection(2)],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
}),
|
||||||
|
filesSince: async (args: {
|
||||||
|
collectionID: number;
|
||||||
|
}): Promise<FilesPage> => ({
|
||||||
|
files: [args.collectionID === 1 ? older : newer],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
const lib = await open({ client });
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
|
||||||
|
expect(photo.takenAt).toBe(new Date(2026, 3, 15, 12).getTime());
|
||||||
|
expect(photo.savePath).toBe(
|
||||||
|
join(
|
||||||
|
root,
|
||||||
|
"backup",
|
||||||
|
"2026",
|
||||||
|
"2026-04",
|
||||||
|
"2026-04-15",
|
||||||
|
"2026-04-15.1.jpg",
|
||||||
|
),
|
||||||
|
);
|
||||||
|
|
||||||
|
// download() writes to that same path, so the photo is then local.
|
||||||
|
const saved = await photo.download();
|
||||||
|
expect(saved.path).toBe(photo.savePath);
|
||||||
|
expect(photo.isLocal).toBe(true);
|
||||||
|
expect(existsSync(join(root, "backup", "2026", "2026-02"))).toBe(false);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("downloads a photo held across a refresh that edits its date to the save path it names", async () => {
|
||||||
|
// The same account, whose second refresh brings a date edited in Ente.
|
||||||
|
const edited = file(1, 1);
|
||||||
|
edited.updationTime = 2;
|
||||||
|
edited.pubMagicMetadata = {
|
||||||
|
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
|
||||||
|
};
|
||||||
|
class DateEditedClient extends MockClient {
|
||||||
|
refreshes = 0;
|
||||||
|
override async collectionsSince(): Promise<CollectionsPage> {
|
||||||
|
this.refreshes++;
|
||||||
|
return {
|
||||||
|
collections: [
|
||||||
|
{ ...collection(1), updationTime: this.refreshes },
|
||||||
|
],
|
||||||
|
deleted: [],
|
||||||
|
cursor: this.refreshes,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
override async filesSince(): Promise<FilesPage> {
|
||||||
|
return {
|
||||||
|
files: [this.refreshes === 1 ? file(1, 1) : edited],
|
||||||
|
deleted: [],
|
||||||
|
cursor: this.refreshes,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const lib = await open({ client: new DateEditedClient() });
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
|
||||||
|
await lib.fresh();
|
||||||
|
expect(lib.photos.byID({ fileID: 1 })!.takenAt).toBe(
|
||||||
|
new Date(2026, 3, 15, 12).getTime(),
|
||||||
|
);
|
||||||
|
|
||||||
|
const saved = await photo.download();
|
||||||
|
expect(saved.path).toBe(photo.savePath);
|
||||||
|
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
|
||||||
|
expect(photo.isLocal).toBe(true);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("has a save path, and is not local, without a content source", async () => {
|
||||||
const lib = await open({ contentSource: undefined });
|
const lib = await open({ contentSource: undefined });
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
expect(photo.savePath).toBeUndefined();
|
expect(photo.savePath).toBe(
|
||||||
|
join(root, "backup", DAY, "2026-03-01.1.jpg"),
|
||||||
|
);
|
||||||
expect(photo.isLocal).toBe(false);
|
expect(photo.isLocal).toBe(false);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
@@ -387,21 +539,170 @@ describe("Photo save path, local copy, content and EXIF", () => {
|
|||||||
contentSource: cdnSource(new Map([[1, body]])),
|
contentSource: cdnSource(new Map([[1, body]])),
|
||||||
});
|
});
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
const originals = join(root, "backup", "originals");
|
const day = join(root, "backup", DAY);
|
||||||
// Until then the name comes from the title, file-1.jpg; the backup
|
// Until then the name comes from the title, file-1.jpg; the backup
|
||||||
// stores the image with the extension found inside the live photo.
|
// stores the image with the extension found inside the live photo.
|
||||||
expect(photo.savePath).toBe(join(originals, "1.jpg"));
|
expect(photo.savePath).toBe(join(day, "2026-03-01.1.jpg"));
|
||||||
|
|
||||||
await lib.backup();
|
await lib.backup();
|
||||||
expect(photo.savePath).toBe(join(originals, "1.heic"));
|
expect(photo.savePath).toBe(join(day, "2026-03-01.1.heic"));
|
||||||
expect(photo.isLocal).toBe(true);
|
expect(photo.isLocal).toBe(true);
|
||||||
expect(await photo.content()).toEqual(Buffer.from(IMAGE));
|
expect(await photo.content()).toEqual(Buffer.from(IMAGE));
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("reads the common EXIF fields of a JPEG original", async () => {
|
it("downloads an original the cache holds by copying it, without fetching it again", async () => {
|
||||||
|
const source = stubSource();
|
||||||
|
const lib = await open({ contentSource: source });
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
await photo.original();
|
||||||
|
expect(source.originalCalls()).toBe(1);
|
||||||
|
expect(photo.isLocal).toBe(false);
|
||||||
|
|
||||||
|
const saved = await photo.download();
|
||||||
|
|
||||||
|
expect(source.originalCalls()).toBe(1);
|
||||||
|
expect(saved).toEqual({
|
||||||
|
path: photo.savePath,
|
||||||
|
bytes: "orig-bytes".length,
|
||||||
|
});
|
||||||
|
expect(readFileSync(photo.savePath, "utf-8")).toBe("orig-bytes");
|
||||||
|
expect(photo.isLocal).toBe(true);
|
||||||
|
// The cache keeps its own copy.
|
||||||
|
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
|
||||||
|
true,
|
||||||
|
);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("downloads an original the cache does not hold straight to its save path", async () => {
|
||||||
|
const source = stubSource();
|
||||||
|
const lib = await open({ contentSource: source });
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
|
||||||
|
const saved = await photo.download();
|
||||||
|
|
||||||
|
expect(source.originalCalls()).toBe(1);
|
||||||
|
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
|
||||||
|
expect(readFileSync(saved.path, "utf-8")).toBe("orig-bytes");
|
||||||
|
expect(photo.isLocal).toBe(true);
|
||||||
|
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does nothing when download() finds the original already at its save path", async () => {
|
||||||
|
const source = stubSource();
|
||||||
|
const lib = await open({ contentSource: source });
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
const first = await photo.download();
|
||||||
|
const written = statSync(first.path).ino;
|
||||||
|
|
||||||
|
const second = await photo.download();
|
||||||
|
|
||||||
|
expect(second).toEqual(first);
|
||||||
|
expect(source.originalCalls()).toBe(1);
|
||||||
|
expect(statSync(second.path).ino).toBe(written);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("downloads a live photo the cache holds as its image, its video and the JSON file naming them", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(file(1, 1));
|
||||||
|
const lib = await open({
|
||||||
|
client: new FilesClient([live]),
|
||||||
|
contentSource: cdnSource(new Map([[1, body]])),
|
||||||
|
});
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
await photo.original();
|
||||||
|
const day = join(root, "backup", DAY);
|
||||||
|
|
||||||
|
const saved = await photo.download();
|
||||||
|
|
||||||
|
expect(saved).toEqual({
|
||||||
|
path: join(day, "2026-03-01.1.heic"),
|
||||||
|
videoPath: join(day, "2026-03-01.1.mov"),
|
||||||
|
bytes: IMAGE.length,
|
||||||
|
});
|
||||||
|
expect(readdirSync(day).sort()).toEqual([
|
||||||
|
"2026-03-01.1.heic",
|
||||||
|
"2026-03-01.1.livephoto.json",
|
||||||
|
"2026-03-01.1.mov",
|
||||||
|
]);
|
||||||
|
expect(
|
||||||
|
JSON.parse(
|
||||||
|
readFileSync(join(day, "2026-03-01.1.livephoto.json"), "utf-8"),
|
||||||
|
),
|
||||||
|
).toEqual({ image: "2026-03-01.1.heic", video: "2026-03-01.1.mov" });
|
||||||
|
expect(photo.savePath).toBe(saved.path);
|
||||||
|
expect(photo.isLocal).toBe(true);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Every tag in JPEG_WITH_EXIF, by name, in the order of its IFDs.
|
||||||
|
const jpegTags = [
|
||||||
|
"Make",
|
||||||
|
"Model",
|
||||||
|
"Orientation",
|
||||||
|
"Exif IFD Pointer",
|
||||||
|
"GPS Info IFD Pointer",
|
||||||
|
"ExposureTime",
|
||||||
|
"FNumber",
|
||||||
|
"ISOSpeedRatings",
|
||||||
|
"DateTimeOriginal",
|
||||||
|
"OffsetTimeOriginal",
|
||||||
|
"FocalLength",
|
||||||
|
"LensModel",
|
||||||
|
"GPSLatitudeRef",
|
||||||
|
"GPSLatitude",
|
||||||
|
"GPSLongitudeRef",
|
||||||
|
"GPSLongitude",
|
||||||
|
"GPSAltitudeRef",
|
||||||
|
"GPSAltitude",
|
||||||
|
];
|
||||||
|
// HEIC_WITH_EXIF holds those and the tags exiftool adds to every file.
|
||||||
|
const heicTags = [
|
||||||
|
...jpegTags,
|
||||||
|
"YCbCrPositioning",
|
||||||
|
"ExifVersion",
|
||||||
|
"ComponentsConfiguration",
|
||||||
|
"ColorSpace",
|
||||||
|
"GPSVersionID",
|
||||||
|
];
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
[
|
||||||
|
"JPEG",
|
||||||
|
JPEG_WITH_EXIF,
|
||||||
|
jpegTags,
|
||||||
|
{
|
||||||
|
"Exif IFD Pointer": { value: 88 },
|
||||||
|
GPSLatitudeRef: { value: ["N"], description: "North latitude" },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"HEIC",
|
||||||
|
HEIC_WITH_EXIF,
|
||||||
|
heicTags,
|
||||||
|
{
|
||||||
|
ColorSpace: { value: 0xffff, description: "Uncalibrated" },
|
||||||
|
ExifVersion: { description: "0232" },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
])(
|
||||||
|
"returns every EXIF tag of a %s original, by name",
|
||||||
|
async (_, bytes, names, others) => {
|
||||||
|
const lib = await open({ contentSource: stubSource(bytes) });
|
||||||
|
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
||||||
|
expect(Object.keys(exif).sort()).toEqual([...names].sort());
|
||||||
|
// Tags outside the thirteen fields, as exifreader decodes them.
|
||||||
|
expect(exif).toMatchObject(others);
|
||||||
|
await lib.close();
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
it("picks the common EXIF fields from a JPEG original's tags", async () => {
|
||||||
const lib = await open({ contentSource: stubSource(JPEG_WITH_EXIF) });
|
const lib = await open({ contentSource: stubSource(JPEG_WITH_EXIF) });
|
||||||
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({
|
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
||||||
|
expect(readPhotoExif(exif)).toStrictEqual({
|
||||||
make: "Canon",
|
make: "Canon",
|
||||||
model: "EOS R5",
|
model: "EOS R5",
|
||||||
lensModel: "RF50mm F1.8 STM",
|
lensModel: "RF50mm F1.8 STM",
|
||||||
@@ -420,8 +721,8 @@ describe("Photo save path, local copy, content and EXIF", () => {
|
|||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
// What exif() returns for HEIC_WITH_EXIF, which holds the same values as
|
// The fields picked from HEIC_WITH_EXIF's tags, and from JPEG_WITH_EXIF's,
|
||||||
// JPEG_WITH_EXIF.
|
// which hold the same values.
|
||||||
const heicFields: PhotoExif = {
|
const heicFields: PhotoExif = {
|
||||||
make: "Canon",
|
make: "Canon",
|
||||||
model: "EOS R5",
|
model: "EOS R5",
|
||||||
@@ -438,11 +739,10 @@ describe("Photo save path, local copy, content and EXIF", () => {
|
|||||||
gpsAltitude: -12.5,
|
gpsAltitude: -12.5,
|
||||||
};
|
};
|
||||||
|
|
||||||
it("reads the same common EXIF fields from a HEIC original", async () => {
|
it("picks the same common EXIF fields from a HEIC original's tags", async () => {
|
||||||
const lib = await open({ contentSource: stubSource(HEIC_WITH_EXIF) });
|
const lib = await open({ contentSource: stubSource(HEIC_WITH_EXIF) });
|
||||||
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual(
|
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
||||||
heicFields,
|
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
|
||||||
);
|
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -456,19 +756,44 @@ describe("Photo save path, local copy, content and EXIF", () => {
|
|||||||
client: new FilesClient([live]),
|
client: new FilesClient([live]),
|
||||||
contentSource: cdnSource(new Map([[1, body]])),
|
contentSource: cdnSource(new Map([[1, body]])),
|
||||||
});
|
});
|
||||||
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual(
|
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
||||||
heicFields,
|
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
// The build's type check, not this test, makes sure `Photo` has a method
|
||||||
|
// for every `PhotoExif` field, whatever the fixtures hold: `Photo`
|
||||||
|
// implements a type with one method per field. This test checks that each
|
||||||
|
// method gives the field picked from the tags exif() returns.
|
||||||
|
it.each([
|
||||||
|
["JPEG", JPEG_WITH_EXIF],
|
||||||
|
["HEIC", HEIC_WITH_EXIF],
|
||||||
|
])(
|
||||||
|
"has a method for each field, agreeing with the tags exif() returns, for a %s",
|
||||||
|
async (_, bytes) => {
|
||||||
|
const lib = await open({ contentSource: stubSource(bytes) });
|
||||||
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
const fields = readPhotoExif(await photo.exif());
|
||||||
|
// The file holds every field, so every method is checked.
|
||||||
|
expect(fields).toStrictEqual(heicFields);
|
||||||
|
for (const [field, value] of Object.entries(fields)) {
|
||||||
|
expect(await photo[field as keyof PhotoExif]()).toStrictEqual(
|
||||||
|
value,
|
||||||
);
|
);
|
||||||
|
}
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
},
|
||||||
|
);
|
||||||
|
|
||||||
it("returns no EXIF fields for an original that is not an image", async () => {
|
it("returns no EXIF tags for an original that is not an image", async () => {
|
||||||
const lib = await open();
|
const lib = await open();
|
||||||
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({});
|
const photo = lib.photos.byID({ fileID: 1 })!;
|
||||||
|
expect(await photo.exif()).toStrictEqual({});
|
||||||
|
expect(await photo.dateTimeOriginal()).toBeUndefined();
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("returns no EXIF fields for a JPEG whose EXIF cannot be parsed", async () => {
|
it("returns no EXIF tags for a JPEG whose EXIF cannot be parsed", async () => {
|
||||||
const lib = await open({
|
const lib = await open({
|
||||||
contentSource: stubSource(JPEG_WITH_BAD_EXIF),
|
contentSource: stubSource(JPEG_WITH_BAD_EXIF),
|
||||||
});
|
});
|
||||||
@@ -489,7 +814,7 @@ describe("Photo save path, local copy, content and EXIF", () => {
|
|||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("returns no EXIF fields for a video, without fetching it", async () => {
|
it("returns no EXIF tags for a video, without fetching it", async () => {
|
||||||
const video = file(1, 1);
|
const video = file(1, 1);
|
||||||
video.metadata.fileType = "video";
|
video.metadata.fileType = "video";
|
||||||
const source = stubSource(JPEG_WITH_EXIF);
|
const source = stubSource(JPEG_WITH_EXIF);
|
||||||
|
|||||||
@@ -7,8 +7,8 @@
|
|||||||
* 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail`
|
* 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail`
|
||||||
* fetches through the request pool and stores the bytes; the next finds the
|
* fetches through the request pool and stores the bytes; the next finds the
|
||||||
* file present and returns its path with a single `skipped` event and no
|
* file present and returns its path with a single `skipped` event and no
|
||||||
* network. A file already sitting in the backup `downloadDirectory` counts
|
* network. A file already stored at its save path under the
|
||||||
* as present too.
|
* `downloadDirectory` counts as present too.
|
||||||
* 2. **Present-means-complete.** Content appears only by the streaming atomic
|
* 2. **Present-means-complete.** Content appears only by the streaming atomic
|
||||||
* writer's rename, so a file that exists is whole. The directory listing
|
* writer's rename, so a file that exists is whole. The directory listing
|
||||||
* taken at `open()` is the record of what is cached, and the orphan temp
|
* taken at `open()` is the record of what is cached, and the orphan temp
|
||||||
@@ -41,6 +41,7 @@ import { join } from "node:path";
|
|||||||
|
|
||||||
import {
|
import {
|
||||||
ContentCache,
|
ContentCache,
|
||||||
|
savePath,
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
type EnsureEvent,
|
type EnsureEvent,
|
||||||
} from "../../src/library/content.js";
|
} from "../../src/library/content.js";
|
||||||
@@ -152,12 +153,48 @@ const buildCache = (
|
|||||||
pools: args.pools ?? new RequestPools(),
|
pools: args.pools ?? new RequestPools(),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
downloadDirectory: args.downloadDirectory,
|
downloadDirectory: args.downloadDirectory ?? join(root, "photos"),
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
});
|
});
|
||||||
return { cache, source };
|
return { cache, source };
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// Microseconds, as Ente stores times, for noon local time on a day, so the
|
||||||
|
// machine's time zone cannot move the photo to another day.
|
||||||
|
const noon = (year: number, month: number, day: number): number =>
|
||||||
|
new Date(year, month - 1, day, 12).getTime() * 1000;
|
||||||
|
|
||||||
|
describe("savePath", () => {
|
||||||
|
it("files an original by year, month and day under the root", () => {
|
||||||
|
const f = file(12345, "IMG_0001.HEIC");
|
||||||
|
f.metadata.creationTime = noon(2026, 3, 1);
|
||||||
|
|
||||||
|
expect(savePath("/photos", f)).toBe(
|
||||||
|
"/photos/2026/2026-03/2026-03-01/2026-03-01.12345.HEIC",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("dates an original by the date the user set, when there is one", () => {
|
||||||
|
const f = file(7, "a.jpg");
|
||||||
|
f.metadata.creationTime = noon(2026, 3, 1);
|
||||||
|
f.pubMagicMetadata = { editedTime: noon(1999, 12, 31) };
|
||||||
|
|
||||||
|
expect(savePath("/photos", f)).toBe(
|
||||||
|
"/photos/1999/1999-12/1999-12-31/1999-12-31.7.jpg",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("takes the extension from the title it was uploaded with, not a new name", () => {
|
||||||
|
const f = file(8, "upload");
|
||||||
|
f.metadata.creationTime = noon(2026, 3, 1);
|
||||||
|
f.pubMagicMetadata = { editedName: "renamed.png" };
|
||||||
|
|
||||||
|
expect(savePath("/photos", f)).toBe(
|
||||||
|
"/photos/2026/2026-03/2026-03-01/2026-03-01.8.bin",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("ContentCache.open", () => {
|
describe("ContentCache.open", () => {
|
||||||
it("creates the cache directories with 0700 permissions", async () => {
|
it("creates the cache directories with 0700 permissions", async () => {
|
||||||
const { cache } = buildCache();
|
const { cache } = buildCache();
|
||||||
@@ -274,11 +311,17 @@ describe("ContentCache.original / thumbnail", () => {
|
|||||||
|
|
||||||
it("serves a file already present in the download directory without fetching", async () => {
|
it("serves a file already present in the download directory without fetching", async () => {
|
||||||
const downloadDirectory = join(root, "backup");
|
const downloadDirectory = join(root, "backup");
|
||||||
mkdirSync(join(downloadDirectory, "originals"), { recursive: true });
|
const day = join(downloadDirectory, "2026", "2026-03", "2026-03-01");
|
||||||
const backupPath = join(downloadDirectory, "originals", "1.jpg");
|
mkdirSync(day, { recursive: true });
|
||||||
|
const backupPath = join(day, "2026-03-01.1.jpg");
|
||||||
writeFileSync(backupPath, "from-backup");
|
writeFileSync(backupPath, "from-backup");
|
||||||
|
const f = file(1);
|
||||||
|
f.metadata.creationTime = noon(2026, 3, 1);
|
||||||
|
|
||||||
const { cache, source } = buildCache({ downloadDirectory });
|
const { cache, source } = buildCache({
|
||||||
|
downloadDirectory,
|
||||||
|
files: [f],
|
||||||
|
});
|
||||||
await cache.open();
|
await cache.open();
|
||||||
|
|
||||||
const events: EnsureEvent["status"][] = [];
|
const events: EnsureEvent["status"][] = [];
|
||||||
@@ -523,43 +566,6 @@ describe("ContentCache live photos", () => {
|
|||||||
expect(events).toEqual(["skipped"]);
|
expect(events).toEqual(["skipped"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("replaces a live photo an earlier version stored as a ZIP under the image's name", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
|
||||||
mkdirSync(originals(), { recursive: true });
|
|
||||||
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
|
|
||||||
const cache = cacheOf([live], new Map([[5, body]]));
|
|
||||||
// Opened without being told that file 5 is a live photo, the cache
|
|
||||||
// records the ZIP, and does not serve it.
|
|
||||||
await cache.open();
|
|
||||||
|
|
||||||
const result = await cache.original(5);
|
|
||||||
|
|
||||||
expect(result.videoPath).toBe(join(originals(), "5.mov"));
|
|
||||||
expect(readdirSync(originals()).sort()).toEqual([
|
|
||||||
"5.heic",
|
|
||||||
"5.livephoto.json",
|
|
||||||
"5.mov",
|
|
||||||
]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("removes a live photo's ZIP an earlier version stored when it opens, so the precache fetches the image and video", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
|
||||||
mkdirSync(originals(), { recursive: true });
|
|
||||||
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
|
|
||||||
const cache = cacheOf([live], new Map([[5, body]]));
|
|
||||||
|
|
||||||
await cache.open((fileID) => fileID === 5);
|
|
||||||
|
|
||||||
expect(readdirSync(originals())).toEqual([]);
|
|
||||||
expect(cache.pathsFor(5)).toEqual({});
|
|
||||||
const [fetched] = await cache.ensureOriginals({ fileIDs: [5] });
|
|
||||||
expect(fetched).toEqual({
|
|
||||||
fileID: 5,
|
|
||||||
path: join(originals(), "5.heic"),
|
|
||||||
});
|
|
||||||
expect(cache.pathsFor(5)).toEqual({ originalPath: fetched!.path });
|
|
||||||
});
|
|
||||||
|
|
||||||
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
|
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
const server = cdnSource(new Map([[5, body]]));
|
const server = cdnSource(new Map([[5, body]]));
|
||||||
@@ -590,6 +596,36 @@ describe("ContentCache live photos", () => {
|
|||||||
expect(second!.pathsFor(5)).toEqual({});
|
expect(second!.pathsFor(5)).toEqual({});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("fetches a live photo's image and video again when the cache opened before knowing it is a live photo and no JSON file names them", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
|
mkdirSync(originals(), { recursive: true });
|
||||||
|
writeFileSync(join(originals(), "5.heic"), "an image");
|
||||||
|
writeFileSync(join(originals(), "5.mov"), "a video");
|
||||||
|
const cache = cacheOf([live], new Map([[5, body]]));
|
||||||
|
// Opened without being told that file 5 is a live photo, the cache
|
||||||
|
// records one of the two files as its original, with no video.
|
||||||
|
await cache.open();
|
||||||
|
|
||||||
|
const events: string[] = [];
|
||||||
|
const result = await cache.original(5, {
|
||||||
|
onProgress: (e) => events.push(e.status),
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(events.at(-1)).toBe("done");
|
||||||
|
expect(result).toEqual({
|
||||||
|
path: join(originals(), "5.heic"),
|
||||||
|
videoPath: join(originals(), "5.mov"),
|
||||||
|
bytes: IMAGE.length,
|
||||||
|
});
|
||||||
|
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
|
||||||
|
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
|
||||||
|
expect(
|
||||||
|
JSON.parse(
|
||||||
|
readFileSync(join(originals(), "5.livephoto.json"), "utf-8"),
|
||||||
|
),
|
||||||
|
).toEqual({ image: "5.heic", video: "5.mov" });
|
||||||
|
});
|
||||||
|
|
||||||
it.each(["missing", "empty"])(
|
it.each(["missing", "empty"])(
|
||||||
"fetches a live photo again when the video its JSON file names is %s",
|
"fetches a live photo again when the video its JSON file names is %s",
|
||||||
async (state) => {
|
async (state) => {
|
||||||
@@ -649,6 +685,7 @@ describe("ContentCache live photos", () => {
|
|||||||
]),
|
]),
|
||||||
),
|
),
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
|
downloadDirectory: join(root, "photos"),
|
||||||
getFile: (id) => [a.file, b.file].find((f) => f.id === id),
|
getFile: (id) => [a.file, b.file].find((f) => f.id === id),
|
||||||
// Room for one live photo, on a disk with plenty free.
|
// Room for one live photo, on a disk with plenty free.
|
||||||
cacheOriginalsMaxBytes: size,
|
cacheOriginalsMaxBytes: size,
|
||||||
|
|||||||
@@ -280,6 +280,7 @@ describe("Precache eviction integration", () => {
|
|||||||
pools: new RequestPools(),
|
pools: new RequestPools(),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
|
downloadDirectory: join(root, "photos"),
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
statfs,
|
statfs,
|
||||||
cacheOriginalsMaxBytes: 25, // holds two 10-byte originals
|
cacheOriginalsMaxBytes: 25, // holds two 10-byte originals
|
||||||
@@ -348,6 +349,7 @@ describe("Precache preemption", () => {
|
|||||||
pools: new RequestPools({ contentConcurrency: 1 }),
|
pools: new RequestPools({ contentConcurrency: 1 }),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: join(root, "cache"),
|
cacheDirectory: join(root, "cache"),
|
||||||
|
downloadDirectory: join(root, "photos"),
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }),
|
statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }),
|
||||||
freeBelowBytes: 0,
|
freeBelowBytes: 0,
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ import {
|
|||||||
makeAlbumsAPI,
|
makeAlbumsAPI,
|
||||||
makePhotosAPI,
|
makePhotosAPI,
|
||||||
makeTimelineAPI,
|
makeTimelineAPI,
|
||||||
|
type SavePathLookup,
|
||||||
type TimelineGroup,
|
type TimelineGroup,
|
||||||
} from "../../src/library/read.js";
|
} from "../../src/library/read.js";
|
||||||
import { Library } from "../../src/library/index.js";
|
import { Library } from "../../src/library/index.js";
|
||||||
@@ -101,12 +102,19 @@ const file = (
|
|||||||
};
|
};
|
||||||
|
|
||||||
// Build the three API objects over one fixed projection, the way `Library`
|
// Build the three API objects over one fixed projection, the way `Library`
|
||||||
// wires them over its live store.
|
// wires them over its live store. Save paths are covered in
|
||||||
|
// content-library.test.ts; these tests never ask for one.
|
||||||
const apis = (records: DerivedRecords) => {
|
const apis = (records: DerivedRecords) => {
|
||||||
const derive = () => records;
|
const derive = () => records;
|
||||||
|
const saves: SavePathLookup = {
|
||||||
|
savePath: () => {
|
||||||
|
throw new Error("no save paths in these tests");
|
||||||
|
},
|
||||||
|
isLocal: () => false,
|
||||||
|
};
|
||||||
return {
|
return {
|
||||||
albums: makeAlbumsAPI(derive),
|
albums: makeAlbumsAPI(derive, saves),
|
||||||
photos: makePhotosAPI(derive),
|
photos: makePhotosAPI(derive, saves),
|
||||||
timeline: makeTimelineAPI(derive),
|
timeline: makeTimelineAPI(derive),
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -9,8 +9,10 @@
|
|||||||
// Excluding too much: Prettier 3 reads `.gitignore` as a default ignore file,
|
// Excluding too much: Prettier 3 reads `.gitignore` as a default ignore file,
|
||||||
// so dropping it from the context silently changes which files the lint
|
// so dropping it from the context silently changes which files the lint
|
||||||
// phase's prettier check looks at compared to `make fmt-check` on the host.
|
// phase's prettier check looks at compared to `make fmt-check` on the host.
|
||||||
|
// And without `.git`, a `docker build .` given no `VERSION` build arg cannot
|
||||||
|
// derive the version (`script/version`) and stamps `package.json`'s instead.
|
||||||
//
|
//
|
||||||
// Neither shows up as a build failure, so they are asserted here.
|
// None of these shows up as a build failure, so they are asserted here.
|
||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import { existsSync, readFileSync } from "node:fs";
|
import { existsSync, readFileSync } from "node:fs";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
@@ -47,6 +49,17 @@ describe(".dockerignore", () => {
|
|||||||
expect(dockerignore).not.toContain(".gitignore");
|
expect(dockerignore).not.toContain(".gitignore");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("leaves .git in the build context for the version", () => {
|
||||||
|
expect(dockerignore).not.toContain(".git");
|
||||||
|
expect(dockerignore).not.toContain(".git/");
|
||||||
|
});
|
||||||
|
|
||||||
|
// The build stage is the final image, so a .git/config sent in would
|
||||||
|
// ship the clone's remote URL and any credential in it.
|
||||||
|
it("sends .git without its config", () => {
|
||||||
|
expect(dockerignore).toContain(".git/config");
|
||||||
|
});
|
||||||
|
|
||||||
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
|
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
|
||||||
// file would silently give the build a different, unreviewed context —
|
// file would silently give the build a different, unreviewed context —
|
||||||
// and eslint's flat config does not ignore dot-directories, so a stray
|
// and eslint's flat config does not ignore dot-directories, so a stray
|
||||||
|
|||||||
@@ -0,0 +1,137 @@
|
|||||||
|
// `script/version` prints the version `script/build` stamps into
|
||||||
|
// `dist/package.json`, which is what `quak --version` reports from a build.
|
||||||
|
// A `docker build .` of a clone is given no `VERSION` build arg, so the
|
||||||
|
// version has to come from the `.git` in its context: the tag on a tagged
|
||||||
|
// commit; the tag, the commits since it and the short commit on a later commit;
|
||||||
|
// the short commit when no tag is reachable. A checkout with `.git` that still
|
||||||
|
// yields no usable version must fail the build, not ship a version nobody can
|
||||||
|
// trace back to its commit.
|
||||||
|
//
|
||||||
|
// Each test copies the script into a fresh directory, which the script then
|
||||||
|
// treats as the checkout, and executes it there.
|
||||||
|
import { afterEach, describe, expect, it } from "vitest";
|
||||||
|
import { execFileSync, spawnSync } from "node:child_process";
|
||||||
|
import {
|
||||||
|
chmodSync,
|
||||||
|
copyFileSync,
|
||||||
|
mkdirSync,
|
||||||
|
mkdtempSync,
|
||||||
|
rmSync,
|
||||||
|
writeFileSync,
|
||||||
|
} from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
|
||||||
|
|
||||||
|
let checkout = "";
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
rmSync(checkout, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
// A checkout holding the script and a package.json that declares 0.0.0, with
|
||||||
|
// no .git yet.
|
||||||
|
const makeCheckout = (): void => {
|
||||||
|
checkout = mkdtempSync(join(tmpdir(), "quak-version-"));
|
||||||
|
mkdirSync(join(checkout, "script"));
|
||||||
|
copyFileSync(
|
||||||
|
join(repoRoot, "script/version"),
|
||||||
|
join(checkout, "script/version"),
|
||||||
|
);
|
||||||
|
chmodSync(join(checkout, "script/version"), 0o755);
|
||||||
|
writeFileSync(join(checkout, "package.json"), '{ "version": "0.0.0" }\n');
|
||||||
|
};
|
||||||
|
|
||||||
|
// git in the checkout, with an identity and no commit signing, whatever the
|
||||||
|
// host's own git config says.
|
||||||
|
const git = (...args: string[]): string =>
|
||||||
|
execFileSync(
|
||||||
|
"git",
|
||||||
|
[
|
||||||
|
"-c",
|
||||||
|
"user.name=quak",
|
||||||
|
"-c",
|
||||||
|
"user.email=quak@example.invalid",
|
||||||
|
"-c",
|
||||||
|
"commit.gpgsign=false",
|
||||||
|
...args,
|
||||||
|
],
|
||||||
|
{ cwd: checkout, encoding: "utf-8", stdio: ["ignore", "pipe", "pipe"] },
|
||||||
|
).trim();
|
||||||
|
|
||||||
|
const makeCommittedCheckout = (): void => {
|
||||||
|
makeCheckout();
|
||||||
|
git("init", "-q");
|
||||||
|
git("add", "package.json");
|
||||||
|
git("commit", "-q", "-m", "first");
|
||||||
|
};
|
||||||
|
|
||||||
|
// Runs the script with nothing in its environment but PATH and, when given,
|
||||||
|
// VERSION.
|
||||||
|
const runVersion = (version?: string) =>
|
||||||
|
spawnSync(join(checkout, "script/version"), {
|
||||||
|
cwd: checkout,
|
||||||
|
encoding: "utf-8",
|
||||||
|
env: { PATH: process.env.PATH, VERSION: version },
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("script/version", () => {
|
||||||
|
it("prints the short commit of an untagged commit", () => {
|
||||||
|
makeCommittedCheckout();
|
||||||
|
expect(runVersion().stdout.trim()).toBe(
|
||||||
|
git("rev-parse", "--short", "HEAD"),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("prints the tag of a tagged commit", () => {
|
||||||
|
makeCommittedCheckout();
|
||||||
|
git("tag", "v1.2.3");
|
||||||
|
expect(runVersion().stdout.trim()).toBe("v1.2.3");
|
||||||
|
});
|
||||||
|
|
||||||
|
// script/docker and script/cibuild pass the version they resolve on the
|
||||||
|
// host as the VERSION build arg.
|
||||||
|
it("prints the VERSION it is given over what git would derive", () => {
|
||||||
|
makeCommittedCheckout();
|
||||||
|
expect(runVersion("x").stdout.trim()).toBe("x");
|
||||||
|
});
|
||||||
|
|
||||||
|
// `--build-arg VERSION=` must not stamp an empty version.
|
||||||
|
it("treats an empty VERSION as unset", () => {
|
||||||
|
makeCommittedCheckout();
|
||||||
|
expect(runVersion("").stdout.trim()).toBe(
|
||||||
|
git("rev-parse", "--short", "HEAD"),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// A source tarball has no .git: it keeps the version package.json
|
||||||
|
// declares, and must still build.
|
||||||
|
it("prints package.json's version where there is no .git", () => {
|
||||||
|
makeCheckout();
|
||||||
|
const result = runVersion();
|
||||||
|
expect(result.status).toBe(0);
|
||||||
|
expect(result.stdout.trim()).toBe("0.0.0");
|
||||||
|
});
|
||||||
|
|
||||||
|
// A repository with no commits stands in for any .git that git cannot
|
||||||
|
// describe: git missing from the image, or refusing to read the checkout.
|
||||||
|
it("fails where .git yields no version", () => {
|
||||||
|
makeCheckout();
|
||||||
|
git("init", "-q");
|
||||||
|
const result = runVersion();
|
||||||
|
expect(result.status).not.toBe(0);
|
||||||
|
expect(result.stdout).toBe("");
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each(["dev", "unknown"])(
|
||||||
|
"fails where there is .git and the version is %s",
|
||||||
|
(version) => {
|
||||||
|
makeCommittedCheckout();
|
||||||
|
const result = runVersion(version);
|
||||||
|
expect(result.status).not.toBe(0);
|
||||||
|
expect(result.stdout).toBe("");
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
@@ -787,12 +787,13 @@ describe("fixMissingThumbnails", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
|
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
|
||||||
// A noisy 400x300 JPEG, which the default encoding (quality 50, not
|
// A noisy 64x48 JPEG, which the default encoding (quality 50, not
|
||||||
// resized because it is under 720 px) cannot compress below the size
|
// resized because it is under 720 px) cannot compress below the size
|
||||||
// recorded here: one byte less than that encoding's ciphertext.
|
// recorded here: one byte less than that encoding's ciphertext. It is
|
||||||
|
// small so that each encode is quick even on a busy host.
|
||||||
const fixMock = await buildThumbMock();
|
const fixMock = await buildThumbMock();
|
||||||
const w = 400;
|
const w = 64;
|
||||||
const h = 300;
|
const h = 48;
|
||||||
const noisy = new Uint8Array(
|
const noisy = new Uint8Array(
|
||||||
jpegJs.encode(
|
jpegJs.encode(
|
||||||
{
|
{
|
||||||
|
|||||||
+1
-1
@@ -18,5 +18,5 @@
|
|||||||
"sourceMap": true,
|
"sourceMap": true,
|
||||||
"resolveJsonModule": true
|
"resolveJsonModule": true
|
||||||
},
|
},
|
||||||
"include": ["src/**/*", "bin/**/*"]
|
"include": ["src/**/*", "bin/**/*", "examples/**/*"]
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user