Compare commits
1
Commits
1bd6d72876
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
c5a588177c |
+39
-75
@@ -1,86 +1,50 @@
|
|||||||
# .dockerignore does NOT use .gitignore semantics. Docker matches with
|
# Mirrors .gitignore, with one deliberate exception: .gitignore itself stays
|
||||||
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
|
# in the build context, because prettier 3 reads it as a default ignore file
|
||||||
# `/` and an unprefixed pattern is anchored at the context root. Every
|
# and dropping it would change what the lint phase's prettier check sees.
|
||||||
# depth-independent pattern therefore needs `**/`, or `config/.env` and
|
|
||||||
# `certs/server.key` still ship while this file reads as solved. Only
|
|
||||||
# genuinely root-anchored entries go unprefixed. Never transplant these
|
|
||||||
# into .gitignore, where `**/` is wrong.
|
|
||||||
#
|
|
||||||
# Matching is case-sensitive, so secrets use character ranges rather
|
|
||||||
# than an ALL-CAPS twin, which would still miss `Server.Key`.
|
|
||||||
#
|
|
||||||
# Extend with this repo's own host-built artifacts, written anchored:
|
|
||||||
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
|
|
||||||
# deletes the package directory from the context.
|
|
||||||
|
|
||||||
# .git is sent without its config. Without a VERSION build argument the
|
# VCS
|
||||||
# stage that compiles runs `git describe --tags --always` on .git, which
|
.git
|
||||||
# does not need .git/config; that file can hold a credential, such as a
|
|
||||||
# password in a remote URL or the token the CI checkout step stores there.
|
|
||||||
# Each submodule keeps a config with the same exposure in its git directory
|
|
||||||
# under .git/modules/, nested again for a submodule's own submodules, or in
|
|
||||||
# its own .git directory when it keeps one.
|
|
||||||
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
|
|
||||||
# `deploy/config`, `config/lib`) loses its whole git directory, because
|
|
||||||
# `**/.git/modules/**/config` also matches that segment's directory
|
|
||||||
# under .git/modules/. Go's version stamping then fails the build;
|
|
||||||
# nothing leaks. Name such a submodule without that segment:
|
|
||||||
# `git submodule add --name`.
|
|
||||||
**/.git/config
|
|
||||||
**/.git/modules/**/config
|
|
||||||
|
|
||||||
# Agent scratch: one full checkout of the repo per in-flight agent.
|
# OS
|
||||||
# Anchored because it occurs once where agents run at the repo root.
|
.DS_Store
|
||||||
# KNOWN GAP: a repo running agents in subdirectories still ships
|
Thumbs.db
|
||||||
# `services/api/.claude/` and must add its own anchored entry.
|
|
||||||
.claude
|
|
||||||
|
|
||||||
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
# Editors
|
||||||
# convention. Re-include a committed template with a negation if the
|
*.swp
|
||||||
# build needs one: `!docs/example.env`.
|
*.swo
|
||||||
**/*.[eE][nN][vV]
|
*~
|
||||||
**/.[eE][nN][vV].*
|
*.bak
|
||||||
**/.[eE][nN][vV][rR][cC]
|
.idea/
|
||||||
|
.vscode/
|
||||||
|
*.sublime-*
|
||||||
|
|
||||||
# Private keys and the bundles carrying them. Public certificates
|
# Node
|
||||||
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
|
node_modules
|
||||||
**/*.[pP][eE][mM]
|
|
||||||
**/*.[kK][eE][yY]
|
|
||||||
**/*.[pP]12
|
|
||||||
**/*.[pP][fF][xX]
|
|
||||||
**/[iI][dD]_[rR][sS][aA]
|
|
||||||
**/[iI][dD]_[dD][sS][aA]
|
|
||||||
**/[iI][dD]_[eE][cC][dD][sS][aA]
|
|
||||||
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
|
||||||
**/[iI][dD]_[eE][dD]25519
|
|
||||||
**/[iI][dD]_[eE][dD]25519_[sS][kK]
|
|
||||||
|
|
||||||
# Dependencies: restored inside the image, never copied in.
|
# TypeScript / build artifacts
|
||||||
**/node_modules
|
dist
|
||||||
|
build
|
||||||
|
*.tsbuildinfo
|
||||||
|
coverage
|
||||||
|
.nyc_output/
|
||||||
|
|
||||||
# OS metadata.
|
# Vitest
|
||||||
**/.DS_Store
|
.vitest-cache/
|
||||||
**/Thumbs.db
|
|
||||||
|
|
||||||
# Editor state: never a build input, and it churns COPY.
|
# Environment / secrets
|
||||||
**/*.swp
|
.env
|
||||||
**/*.swo
|
.env.*
|
||||||
**/*~
|
*.pem
|
||||||
**/*.bak
|
*.key
|
||||||
**/.idea
|
|
||||||
**/.vscode
|
|
||||||
**/*.sublime-*
|
|
||||||
|
|
||||||
# TypeScript / build artifacts: the image compiles its own.
|
|
||||||
/dist
|
|
||||||
/build
|
|
||||||
/*.tsbuildinfo
|
|
||||||
/coverage
|
|
||||||
/.nyc_output
|
|
||||||
/.vitest-cache
|
|
||||||
|
|
||||||
# Compiled binary (built by make build-bin); around 100 MB
|
# Compiled binary (built by make build-bin); around 100 MB
|
||||||
/bin/quak
|
bin/quak
|
||||||
|
|
||||||
# quak runtime data (in case anyone runs the CLI from inside the repo)
|
# quak runtime data (in case anyone runs the CLI from inside the repo)
|
||||||
/.quak
|
.quak/
|
||||||
|
|
||||||
|
# Local per-developer tool state, including agent worktrees. Correctness,
|
||||||
|
# not context size: a worktree copied in here has its own test/ tree, which
|
||||||
|
# vitest globs alongside the real one, so the containerised suite runs N+1
|
||||||
|
# times over and still reports success.
|
||||||
|
.claude/
|
||||||
|
|||||||
+10
-32
@@ -11,41 +11,9 @@ Thumbs.db
|
|||||||
.vscode/
|
.vscode/
|
||||||
*.sublime-*
|
*.sublime-*
|
||||||
|
|
||||||
# Agent scratch (worktrees of this repo, created and destroyed by
|
|
||||||
# in-flight tooling). Unanchored: .gitignore patterns already match at
|
|
||||||
# every depth, so no prefix is wanted here. This is not a .dockerignore
|
|
||||||
# entry and must not be given a `**/` prefix on the way into one.
|
|
||||||
.claude/
|
|
||||||
|
|
||||||
# Node
|
# Node
|
||||||
node_modules/
|
node_modules/
|
||||||
|
|
||||||
# Secrets. Unanchored like every entry above, so each matches at every
|
|
||||||
# depth. Matching is case-sensitive on Linux, so names use character
|
|
||||||
# ranges rather than a lowercase form that misses `Server.Key`.
|
|
||||||
|
|
||||||
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
|
||||||
# convention. Only the templates `example.env` and `sample.env` are
|
|
||||||
# re-included below. A repository that commits any other template adds
|
|
||||||
# its own negation after these lines, for example `!.env.example`.
|
|
||||||
*.[eE][nN][vV]
|
|
||||||
.[eE][nN][vV].*
|
|
||||||
.[eE][nN][vV][rR][cC]
|
|
||||||
!example.env
|
|
||||||
!sample.env
|
|
||||||
|
|
||||||
# Private keys and the bundles carrying them.
|
|
||||||
*.[pP][eE][mM]
|
|
||||||
*.[kK][eE][yY]
|
|
||||||
*.[pP]12
|
|
||||||
*.[pP][fF][xX]
|
|
||||||
[iI][dD]_[rR][sS][aA]
|
|
||||||
[iI][dD]_[dD][sS][aA]
|
|
||||||
[iI][dD]_[eE][cC][dD][sS][aA]
|
|
||||||
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
|
||||||
[iI][dD]_[eE][dD]25519
|
|
||||||
[iI][dD]_[eE][dD]25519_[sS][kK]
|
|
||||||
|
|
||||||
# TypeScript / build artifacts
|
# TypeScript / build artifacts
|
||||||
dist/
|
dist/
|
||||||
build/
|
build/
|
||||||
@@ -56,8 +24,18 @@ coverage/
|
|||||||
# Vitest
|
# Vitest
|
||||||
.vitest-cache/
|
.vitest-cache/
|
||||||
|
|
||||||
|
# Environment / secrets
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
*.pem
|
||||||
|
*.key
|
||||||
|
|
||||||
# Compiled binary (built by make build-bin)
|
# Compiled binary (built by make build-bin)
|
||||||
bin/quak
|
bin/quak
|
||||||
|
|
||||||
# quak runtime data (in case anyone runs the CLI from inside the repo)
|
# quak runtime data (in case anyone runs the CLI from inside the repo)
|
||||||
.quak/
|
.quak/
|
||||||
|
|
||||||
|
# Local per-developer tool settings and scratch state, including the
|
||||||
|
# worktrees agents check out under this directory
|
||||||
|
.claude/
|
||||||
|
|||||||
@@ -1,2 +1,5 @@
|
|||||||
node_modules/
|
node_modules/
|
||||||
yarn.lock
|
yarn.lock
|
||||||
|
dist/
|
||||||
|
build/
|
||||||
|
coverage/
|
||||||
|
|||||||
+3
-8
@@ -58,17 +58,12 @@ COPY --from=test /app/package.json /dev/null
|
|||||||
COPY script/ script/
|
COPY script/ script/
|
||||||
COPY package.json yarn.lock ./
|
COPY package.json yarn.lock ./
|
||||||
RUN script/bootstrap
|
RUN script/bootstrap
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
|
||||||
RUN git config --system --add safe.directory /app
|
|
||||||
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Version stamped into the build: the VERSION build arg when one is given,
|
# The version is computed on the host and passed in, because
|
||||||
# otherwise what script/version derives from the .git the build context
|
# .dockerignore excludes .git.
|
||||||
# carries (script/bootstrap installed git), so any `docker build .` of a
|
ARG VERSION=dev
|
||||||
# clone stamps its commit. The label can only carry the build arg, and is
|
|
||||||
# empty without one.
|
|
||||||
ARG VERSION
|
|
||||||
LABEL org.opencontainers.image.version="${VERSION}"
|
LABEL org.opencontainers.image.version="${VERSION}"
|
||||||
|
|
||||||
RUN make build
|
RUN make build
|
||||||
|
|||||||
@@ -26,10 +26,8 @@ check:
|
|||||||
build:
|
build:
|
||||||
@script/build
|
@script/build
|
||||||
|
|
||||||
# Bundles the built dist/, so the binary reports the version script/build
|
build-bin:
|
||||||
# stamped.
|
nix-shell -p bun --run "bun build bin/quak.ts --compile --outfile bin/quak"
|
||||||
build-bin: build
|
|
||||||
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
|
|
||||||
|
|
||||||
install: build-bin
|
install: build-bin
|
||||||
mkdir -p ~/bin
|
mkdir -p ~/bin
|
||||||
|
|||||||
@@ -8,9 +8,9 @@ and downloads individual images while decrypting them on the way to disk.
|
|||||||
quak also includes a resilient backup command that downloads every file in the
|
quak also includes a resilient backup command that downloads every file in the
|
||||||
account into a deduplicated local directory tree, skipping files that already
|
account into a deduplicated local directory tree, skipping files that already
|
||||||
exist on disk and continuing past individual download failures instead of
|
exist on disk and continuing past individual download failures instead of
|
||||||
crashing. For each file it persists the basic metadata fields quak keeps (title,
|
crashing. It decrypts and persists all three metadata layers (basic, private
|
||||||
file type, creation and modification time, latitude, longitude, content hash),
|
magic, public magic) per file, including camera info, GPS coordinates, captions,
|
||||||
and the private and public magic metadata in full. A helper subcommand can
|
and any face/keyword labels the Ente clients have added. A helper subcommand can
|
||||||
detect and regenerate missing thumbnails, encrypting and uploading them back to
|
detect and regenerate missing thumbnails, encrypting and uploading them back to
|
||||||
the server.
|
the server.
|
||||||
|
|
||||||
@@ -51,8 +51,7 @@ const client = await Client.login({
|
|||||||
|
|
||||||
// Open a cache-backed library. On an empty cache this awaits one server
|
// Open a cache-backed library. On an empty cache this awaits one server
|
||||||
// refresh; on an existing cache it returns immediately and refreshes in the
|
// refresh; on an existing cache it returns immediately and refreshes in the
|
||||||
// background. Later refreshes start `refreshIntervalSeconds` (default 3) after
|
// background every `refreshIntervalSeconds` (default 3).
|
||||||
// the previous one ends.
|
|
||||||
const lib = await Library.open({ client });
|
const lib = await Library.open({ client });
|
||||||
|
|
||||||
// Default reads answer synchronously from the local cache — no network.
|
// Default reads answer synchronously from the local cache — no network.
|
||||||
@@ -63,7 +62,7 @@ for (const album of lib.albums.list()) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fresh reads await a server round-trip, joining one already running.
|
// Fresh reads await a server round-trip and answer with current state.
|
||||||
const { albums } = await lib.fresh();
|
const { albums } = await lib.fresh();
|
||||||
console.log(`${albums.list().length} albums as of now`);
|
console.log(`${albums.list().length} albums as of now`);
|
||||||
|
|
||||||
@@ -80,43 +79,12 @@ await lib.close();
|
|||||||
The lower-level `Client` (login, session serialization, and the raw
|
The lower-level `Client` (login, session serialization, and the raw
|
||||||
enumeration/download calls) is exported too and documented under Design below.
|
enumeration/download calls) is exported too and documented under Design below.
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
`examples/download-albums.ts` downloads every album's photos and their metadata
|
|
||||||
into a directory, `photos` in the working directory unless you name another. The
|
|
||||||
build compiles it; run it after `yarn install`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
yarn build
|
|
||||||
QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir]
|
|
||||||
```
|
|
||||||
|
|
||||||
It opens the library with `precacheThumbnails` and `precacheOriginals` off, as
|
|
||||||
`quak backup` does, so the only file content it fetches is the originals it
|
|
||||||
saves. It asks on the terminal for a two-factor or email code when the account
|
|
||||||
requires one, and writes:
|
|
||||||
|
|
||||||
- each photo's original at its save path under `dir`, as `photo.download()`
|
|
||||||
writes it: `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, and for a live
|
|
||||||
photo its image, its video and the `.livephoto.json` file naming them
|
|
||||||
- beside each original, a JSON file named after it with `.json` added, for
|
|
||||||
example `2026-03-01.12345.jpg.json`: the photo's record (`photo.record()`)
|
|
||||||
without its cache paths, and every EXIF tag of the photo (`photo.exif()`)
|
|
||||||
under `exif`
|
|
||||||
- `albums/<collectionID>.json` for each album: its `collectionID`, its `name`,
|
|
||||||
and under `savePaths` the save paths of its photos relative to `dir`, newest
|
|
||||||
first
|
|
||||||
|
|
||||||
A photo in several albums is downloaded once. A second run downloads nothing and
|
|
||||||
rewrites only the JSON files whose content changed. A failed download stops the
|
|
||||||
run; running it again carries on, since every photo already saved is skipped.
|
|
||||||
|
|
||||||
## Entrypoints
|
## Entrypoints
|
||||||
|
|
||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||||
standard: normalized scripts in `script/` are the entrypoints for the
|
standard: normalized scripts in `script/` are the entrypoints for the
|
||||||
development workflow, and most Makefile targets are thin shims that call them.
|
development workflow, and the Makefile targets are thin shims that call them.
|
||||||
The scripts are POSIX sh (not bash) so they run in minimal containers such as
|
The scripts are POSIX sh (not bash) so they run in minimal containers such as
|
||||||
alpine. We provide:
|
alpine. We provide:
|
||||||
|
|
||||||
@@ -126,26 +94,24 @@ alpine. We provide:
|
|||||||
`script/bootstrap`, then `script/install-precommit`
|
`script/bootstrap`, then `script/install-precommit`
|
||||||
- `script/projectname` — output the project name (our own extension); used by
|
- `script/projectname` — output the project name (our own extension); used by
|
||||||
`script/docker` for the image tag
|
`script/docker` for the image tag
|
||||||
- `script/build` — compile the TypeScript sources into `dist/`, stamp the
|
- `script/build` — compile the TypeScript sources into `dist/`, then verify that
|
||||||
version into `dist/package.json`, then verify that the entrypoints
|
the entrypoints `package.json` declares (`main`, `types`, `bin`) are among the
|
||||||
`package.json` declares (`main`, `types`, `bin`) are among the files the
|
files the compiler wrote, and make the CLI executable (our own extension)
|
||||||
compiler wrote, and make the CLI executable (our own extension)
|
|
||||||
- `script/version` — print the version `script/build` stamps (our own
|
|
||||||
extension); see Version below
|
|
||||||
- `script/test` — run the test suite, by building the `test` phase of the
|
- `script/test` — run the test suite, by building the `test` phase of the
|
||||||
`Dockerfile` (vitest, 90s timeout, verbose rerun on failure); requires docker
|
`Dockerfile` (vitest, 90s timeout, verbose rerun on failure); requires docker
|
||||||
- `script/lint` — run eslint and a prettier check, by building the `lint` phase
|
- `script/lint` — run eslint and a prettier check, by building the `lint` phase
|
||||||
of the `Dockerfile`; requires docker (see Linting and testing below)
|
of the `Dockerfile`; requires docker (see Linting and testing below)
|
||||||
- `script/fmt` — format all files with prettier (writes)
|
- `script/fmt` — format all files with prettier (writes)
|
||||||
- `script/fmt-check` — check formatting on the host (read-only)
|
- `script/fmt-check` — check formatting on the host (read-only); standalone, and
|
||||||
- `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own
|
not called by `script/check` or `script/precommit`, because `script/lint`
|
||||||
extension)
|
already checks formatting in the container
|
||||||
|
- `script/check` — run all checks: `test`, `lint` (our own extension)
|
||||||
- `script/docker` — build the image, tagged via `script/projectname`
|
- `script/docker` — build the image, tagged via `script/projectname`
|
||||||
- `script/cibuild` — what CI runs: `script/bootstrap`, then `script/check`, then
|
- `script/cibuild` — build the image (what CI runs); its last stage depends on
|
||||||
the image build
|
the `lint` and `test` phases, so this one build lints, tests and compiles
|
||||||
- `script/precommit` — run by the git pre-commit hook (our own extension); runs
|
- `script/precommit` — run by the git pre-commit hook (our own extension); runs
|
||||||
`script/lint` and `script/fmt-check` but deliberately not the tests, so the
|
`script/lint`, which checks both lint and formatting, but deliberately not the
|
||||||
TDD red-phase commit can land
|
tests, so the TDD red-phase commit can land
|
||||||
- `script/install-precommit` — installs the git pre-commit hook (our own
|
- `script/install-precommit` — installs the git pre-commit hook (our own
|
||||||
extension); `make hooks` shims to it
|
extension); `make hooks` shims to it
|
||||||
|
|
||||||
@@ -156,52 +122,27 @@ alpine. We provide:
|
|||||||
Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the
|
Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the
|
||||||
repo into a digest-pinned node image and runs eslint and `prettier --check .`;
|
repo into a digest-pinned node image and runs eslint and `prettier --check .`;
|
||||||
the `test` phase does the same with the suite. `script/lint` and `script/test`
|
the `test` phase does the same with the suite. `script/lint` and `script/test`
|
||||||
each build one phase with `docker build --no-cache --target <phase>`. Neither
|
each build one phase with `docker build --no-cache --target <phase>`. There is
|
||||||
script runs the tools on the host: docker is required, and that also works where
|
no host lint or test path: docker is required, and that also works where the
|
||||||
the docker daemon is remote and bind mounts are impossible.
|
docker daemon is remote and bind mounts are impossible.
|
||||||
|
|
||||||
The last stage of the `Dockerfile` compiles the package, and it copies a file
|
The last stage of the `Dockerfile` compiles the package, and it copies a file
|
||||||
from each phase, so it cannot be built unless lint and the tests pass. The image
|
from each phase, so it cannot be built unless lint and the tests pass. That is
|
||||||
build in `script/cibuild` therefore runs lint and the tests a second time, after
|
why `script/cibuild` is a single `docker build`: it runs lint and the tests once
|
||||||
`script/check` has run them.
|
each and then compiles.
|
||||||
|
|
||||||
Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree
|
Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree
|
||||||
Docker would otherwise serve the lint and test steps from cache, nothing would
|
Docker would otherwise serve the lint and test steps from cache, nothing would
|
||||||
run, and the build would still exit 0.
|
run, and the build would still exit 0.
|
||||||
|
|
||||||
`script/fmt-check` runs prettier on the host. Its verdict matches the `lint`
|
The formatting check is part of the `lint` phase, not a step beside it, so
|
||||||
phase's: prettier is pinned to an exact version, installed from `yarn.lock`
|
`script/check` and `script/precommit` do not call `script/fmt-check` as well;
|
||||||
under `--frozen-lockfile` in both places, and reads `.gitignore` as its default
|
that would run prettier a second time over the same tree for the same verdict.
|
||||||
ignore file — which is why `.dockerignore` keeps `.gitignore` in the build
|
`script/fmt-check` remains as a standalone entrypoint for asking the formatting
|
||||||
context.
|
question on the host. Its verdict matches the container's: prettier is pinned to
|
||||||
|
an exact version, installed from `yarn.lock` under `--frozen-lockfile` in both
|
||||||
### Version
|
places, and reads `.gitignore` as its default ignore file — which is why
|
||||||
|
`.dockerignore` keeps `.gitignore` in the build context.
|
||||||
`quak --version` reports the `version` of `dist/package.json`, which
|
|
||||||
`script/build` writes after compiling; the repo's own `package.json` keeps
|
|
||||||
`0.0.0`, and that is what the tests, which run from source, report.
|
|
||||||
`script/version` decides what is written:
|
|
||||||
|
|
||||||
- the `VERSION` environment variable, or the `Dockerfile`'s `VERSION` build arg
|
|
||||||
(`--build-arg VERSION=...`), when one is given and not empty;
|
|
||||||
- otherwise, in a checkout with `.git`, `git describe --tags --always`: the tag
|
|
||||||
on a tagged commit; the tag, the commits since it and the short commit on a
|
|
||||||
later commit (`v1.2.3-4-gabc1234`); the short commit when no tag is reachable;
|
|
||||||
- otherwise, as in a source tarball, the version `package.json` declares.
|
|
||||||
|
|
||||||
The build fails if the checkout has `.git` and the version still comes out
|
|
||||||
empty, `dev` or `unknown`: such a build could not be traced back to its commit.
|
|
||||||
|
|
||||||
`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of
|
|
||||||
a clone stamps the commit it was built from; a shallow clone stamps a tag only
|
|
||||||
when the cloned commit itself carries one, and otherwise the short commit. It
|
|
||||||
leaves out `.git/config`, which holds the clone's remote URL and any credential
|
|
||||||
in it, so the image carries `.git` without its config; `git describe` does not
|
|
||||||
need that file. `script/docker` (and so `make docker`) and `script/cibuild` pass
|
|
||||||
the version they resolve on the host, with `--dirty`, as the build arg, which
|
|
||||||
takes precedence. The image's `org.opencontainers.image.version` label carries
|
|
||||||
that build arg only, so a build given none leaves it empty. `make build-bin`
|
|
||||||
bundles the built `dist/`, so the single binary reports the stamped version too.
|
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
@@ -229,19 +170,16 @@ local cache is reliable, and the UI is responsive on a five-year-old laptop.
|
|||||||
|
|
||||||
All work on quak is test-driven. No exceptions.
|
All work on quak is test-driven. No exceptions.
|
||||||
|
|
||||||
1. Every change starts on a feature branch off `next`, and its pull request
|
1. Every change starts on a feature branch off `main`.
|
||||||
targets `next`.
|
|
||||||
2. The first commit on the branch is the test suite for what is being added or
|
2. The first commit on the branch is the test suite for what is being added or
|
||||||
changed. Those tests must fail at that commit; the branch is red until the
|
changed. Those tests must fail at that commit; the branch is red until the
|
||||||
implementation lands.
|
implementation lands.
|
||||||
3. Subsequent commits add the implementation and any refactors needed to make
|
3. Subsequent commits add the implementation and any refactors needed to make
|
||||||
the tests pass.
|
the tests pass.
|
||||||
4. A pull request can only be merged into `next` when `make check` is green.
|
4. A feature branch can only be merged into `main` when `make check` is green.
|
||||||
Once it has passed review, the repository manager squash-merges it into
|
`main` is always green. CI runs `script/cibuild`, which builds the
|
||||||
`next`. Only sneak merges `next` into `main`. `main` and `next` are always
|
`Dockerfile`: its `lint` and `test` phases, then the compile, so neither a
|
||||||
green. CI runs `script/cibuild`, which builds the `Dockerfile`: its `lint`
|
red branch nor one that does not compile can pass CI.
|
||||||
and `test` phases, then the compile, so neither a red branch nor one that
|
|
||||||
does not compile can pass CI.
|
|
||||||
5. Tests are the canonical API documentation for this library. Every test file
|
5. Tests are the canonical API documentation for this library. Every test file
|
||||||
is commented thoroughly enough that a reader who has never seen quak can
|
is commented thoroughly enough that a reader who has never seen quak can
|
||||||
learn how to use it from the tests alone. Comments explain why a behavior
|
learn how to use it from the tests alone. Comments explain why a behavior
|
||||||
@@ -255,10 +193,11 @@ All work on quak is test-driven. No exceptions.
|
|||||||
history must still show tests landing before (or with) the matching
|
history must still show tests landing before (or with) the matching
|
||||||
implementation.
|
implementation.
|
||||||
8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which
|
8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which
|
||||||
runs `script/lint` and `script/fmt-check` but not the tests, and so not the
|
runs `script/lint` — eslint and the prettier check, in the container — but
|
||||||
full `make check`. This is deliberate so the TDD red-phase commit (failing
|
not the tests, and so not the full `make check`. This is deliberate so the
|
||||||
tests, no implementation yet) can land. CI executes `script/cibuild`, which
|
TDD red-phase commit (failing tests, no implementation yet) can land. The
|
||||||
runs the tests, so a red branch still cannot reach `next`.
|
`test` phase is part of the image build, which is what CI executes via
|
||||||
|
`script/cibuild`, so a red branch still cannot reach `main`.
|
||||||
|
|
||||||
## Design
|
## Design
|
||||||
|
|
||||||
@@ -270,7 +209,7 @@ the CLI is for humans.
|
|||||||
```
|
```
|
||||||
quak/
|
quak/
|
||||||
src/
|
src/
|
||||||
crypto/ libsodium primitives (boxes, secretstreams, KDF, hash)
|
crypto/ libsodium primitives (boxes, secretstreams, KDF, SRP)
|
||||||
api/ HTTP client (ApiClient class)
|
api/ HTTP client (ApiClient class)
|
||||||
auth/ login flow (SRP + email OTP + TOTP), key unwrap
|
auth/ login flow (SRP + email OTP + TOTP), key unwrap
|
||||||
model/ decrypted Collection, File, Metadata types + decrypt fns
|
model/ decrypted Collection, File, Metadata types + decrypt fns
|
||||||
@@ -280,8 +219,7 @@ quak/
|
|||||||
and search, request pools
|
and search, request pools
|
||||||
backup.ts resilient full-account backup with dedup
|
backup.ts resilient full-account backup with dedup
|
||||||
metadata-backup.ts
|
metadata-backup.ts
|
||||||
backup-metadata: the metadata quak keeps, as JSON
|
backup-metadata: all decrypted metadata as JSON
|
||||||
exif.ts EXIF read from an image's bytes with exifreader
|
|
||||||
mldata-fetch.ts fetch + decrypt per-file ML data
|
mldata-fetch.ts fetch + decrypt per-file ML data
|
||||||
filename.ts safe file names from server metadata
|
filename.ts safe file names from server metadata
|
||||||
errors.ts error types shared across layers
|
errors.ts error types shared across layers
|
||||||
@@ -296,9 +234,6 @@ quak/
|
|||||||
index.ts public library exports
|
index.ts public library exports
|
||||||
bin/
|
bin/
|
||||||
quak.ts CLI entrypoint (commander.js)
|
quak.ts CLI entrypoint (commander.js)
|
||||||
examples/
|
|
||||||
download-albums.ts
|
|
||||||
download every album's photos and metadata
|
|
||||||
test/ unit + integration tests (vitest)
|
test/ unit + integration tests (vitest)
|
||||||
Makefile
|
Makefile
|
||||||
Dockerfile lint phase, test phase, compile
|
Dockerfile lint phase, test phase, compile
|
||||||
@@ -307,18 +242,15 @@ quak/
|
|||||||
```
|
```
|
||||||
|
|
||||||
`make build` compiles that tree into `dist/`, preserving its shape: the library
|
`make build` compiles that tree into `dist/`, preserving its shape: the library
|
||||||
lands in `dist/src/`, the examples in `dist/examples/`, and the CLI in
|
lands in `dist/src/` and the CLI in `dist/bin/quak.js`, which is what
|
||||||
`dist/bin/quak.js`, which is what `package.json` points `main`, `types` and
|
`package.json` points `main`, `types` and `bin` at. The compiler's `rootDir` is
|
||||||
`bin` at. The compiler's `rootDir` is the repository root rather than `src/`,
|
the repository root rather than `src/`, because `bin/` is compiled too and
|
||||||
because `bin/` is compiled too and `rootDir` has to contain everything that is
|
`rootDir` has to contain everything that is compiled.
|
||||||
compiled.
|
|
||||||
|
|
||||||
### Cryptography
|
### Cryptography
|
||||||
|
|
||||||
All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
|
All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
|
||||||
required for `crypto_pwhash` / Argon2id), except the SRP handshake, which uses
|
required for `crypto_pwhash` / Argon2id). No hand-rolled crypto.
|
||||||
`fast-srp-hap`, and the MD5 checksum sent with a thumbnail upload, which uses
|
|
||||||
Node's built-in `node:crypto`. No hand-rolled crypto.
|
|
||||||
|
|
||||||
The key hierarchy, derived during login, is:
|
The key hierarchy, derived during login, is:
|
||||||
|
|
||||||
@@ -326,23 +258,19 @@ The key hierarchy, derived during login, is:
|
|||||||
2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`,
|
2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`,
|
||||||
with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key
|
with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key
|
||||||
Encryption Key (KEK).
|
Encryption Key (KEK).
|
||||||
3. SRP login: `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte subkey
|
3. SRP login: a 16-byte SRP login subkey is derived from the KEK using
|
||||||
from the KEK with subkey id 1 and context `loginctx`. Its first 16 bytes are
|
`crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context
|
||||||
the SRP password.
|
`loginctx`. That 16-byte value is the SRP password.
|
||||||
4. When SRP completes, the server returns a blob of "key attributes" plus an
|
4. After SRP completes (or after email-OTP fallback), the server returns a blob
|
||||||
encrypted auth token, or first asks for a second factor. quak answers a TOTP
|
of "key attributes" plus an encrypted auth token.
|
||||||
request with the code (`POST /users/two-factor/verify`), after which the
|
|
||||||
server returns them, and cannot answer a passkey request. When the account
|
|
||||||
has email MFA on (`isEmailMFAEnabled`), an email OTP replaces SRP and the
|
|
||||||
server returns them after it.
|
|
||||||
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
||||||
yields the 32-byte master key.
|
yields the 32-byte master key.
|
||||||
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
||||||
key yields the user's X25519 private key. The matching public key is
|
key yields the user's X25519 private key. The matching public key is
|
||||||
delivered in cleartext.
|
delivered in cleartext.
|
||||||
7. `crypto_box_seal_open` over the encrypted token with the user's keypair
|
7. `crypto_box_seal_open` over the encrypted token with the user's keypair
|
||||||
yields the auth token's bytes. Encoded as URL-safe base64 with padding, they
|
yields the URL-safe base64 auth token used in `X-Auth-Token` for all
|
||||||
are the `X-Auth-Token` value for all subsequent calls.
|
subsequent calls.
|
||||||
|
|
||||||
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
|
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
|
||||||
master key (for owned collections). Per-file keys are decrypted with
|
master key (for owned collections). Per-file keys are decrypted with
|
||||||
@@ -378,8 +306,7 @@ Endpoints used:
|
|||||||
- `POST /users/srp/create-session`: begin SRP handshake.
|
- `POST /users/srp/create-session`: begin SRP handshake.
|
||||||
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the
|
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the
|
||||||
encrypted token plus key attributes.
|
encrypted token plus key attributes.
|
||||||
- `POST /users/ott` and `POST /users/verify-email`: email OTP, used instead of
|
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
|
||||||
SRP when the SRP attributes have `isEmailMFAEnabled` set.
|
|
||||||
- `POST /users/two-factor/verify`: TOTP second factor.
|
- `POST /users/two-factor/verify`: TOTP second factor.
|
||||||
- `POST /users/logout`: end the calling token's session (`quak logout`).
|
- `POST /users/logout`: end the calling token's session (`quak logout`).
|
||||||
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
|
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
|
||||||
@@ -401,9 +328,8 @@ request is repeated only when repeating it could produce a different answer:
|
|||||||
- Every other 4xx: not retried. A 404 in particular is an answer, and
|
- Every other 4xx: not retried. A 404 in particular is an answer, and
|
||||||
`listMissingThumbnails` depends on getting it promptly and once.
|
`listMissingThumbnails` depends on getting it promptly and once.
|
||||||
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
|
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
|
||||||
TLS failure — and deadline aborts: retried. Node's `fetch` rejects with a
|
TLS failure — and deadline aborts: retried. The errno is looked for in the
|
||||||
plain `TypeError`, so every `TypeError` is retried. The errno is looked for in
|
error's `cause` chain, because that is where Node's `fetch` puts it.
|
||||||
the error's `cause` chain, because that is where Node's `fetch` puts it.
|
|
||||||
- A truncated download: retried.
|
- A truncated download: retried.
|
||||||
- Anything else, including a secretstream authentication failure that is not
|
- Anything else, including a secretstream authentication failure that is not
|
||||||
truncation: not retried. The default answer is no. For a backup tool, retrying
|
truncation: not retried. The default answer is no. For a backup tool, retrying
|
||||||
@@ -415,24 +341,18 @@ Backoff is exponential with full jitter: the delay before retry _n_ is
|
|||||||
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
|
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
|
||||||
is the ceiling and the wait is drawn below it, so a client that lost many
|
is the ceiling and the wait is drawn below it, so a client that lost many
|
||||||
parallel downloads to one CDN blip does not send them all again at the same
|
parallel downloads to one CDN blip does not send them all again at the same
|
||||||
instant. The numbers, configurable through `ApiClientOptions.retry`:
|
instant. Defaults, configurable through `ApiClientOptions.retry`:
|
||||||
|
|
||||||
| Option | Default | `quak backup` | Meaning |
|
| Option | Default | Meaning |
|
||||||
| ------------- | ------- | ------------- | ----------------------------------- |
|
| ------------- | ------- | ----------------------------------- |
|
||||||
| `attempts` | `4` | `10` | total calls, not retries |
|
| `attempts` | `4` | total calls, not retries |
|
||||||
| `baseDelayMs` | `500` | `1000` | ceiling for the first retry's delay |
|
| `baseDelayMs` | `500` | ceiling for the first retry's delay |
|
||||||
| `maxDelayMs` | `10000` | `60000` | upper bound on that ceiling |
|
| `maxDelayMs` | `10000` | upper bound on that ceiling |
|
||||||
|
|
||||||
With the defaults a file that is going to fail gives up after at most three and
|
With those defaults a file that is going to fail gives up after at most three
|
||||||
a half seconds of waiting. `quak backup` usually runs from cron with nobody
|
and a half seconds of waiting. `sleep` and `random` are injectable through the
|
||||||
watching, so every request it makes uses the `quak backup` column instead,
|
same option, which is how the test suite exercises the whole policy without
|
||||||
exported as `UNATTENDED_RETRY_OPTIONS`: a request that keeps failing gives up
|
waiting.
|
||||||
after at most 243 seconds of waiting, and usually after about half that, since
|
|
||||||
each wait is drawn at random below its ceiling. Every other command uses the
|
|
||||||
defaults. A library user gets the same budget by passing
|
|
||||||
`UNATTENDED_RETRY_OPTIONS` as `ApiClientOptions.retry`. `sleep` and `random` are
|
|
||||||
injectable through the same option, which is how the test suite exercises the
|
|
||||||
whole policy without waiting.
|
|
||||||
|
|
||||||
Two deadlines, renewed for each attempt:
|
Two deadlines, renewed for each attempt:
|
||||||
|
|
||||||
@@ -475,13 +395,10 @@ because a socket reset after the response headers have arrived surfaces in the
|
|||||||
download layer rather than in `ApiClient`, and that is the common failure for
|
download layer rather than in `ApiClient`, and that is the common failure for
|
||||||
multi-megabyte photos over a CDN. The secretstream pull state is not resumable
|
multi-megabyte photos over a CDN. The secretstream pull state is not resumable
|
||||||
and these endpoints have no Range support, so a retry starts the file over. The
|
and these endpoints have no Range support, so a retry starts the file over. The
|
||||||
atomic write is part of the retried unit: each attempt writes its own temporary
|
atomic write stays outside the retry, so a download that needed three attempts
|
||||||
files, one for most files and two for a live photo (its image and its video),
|
still performs exactly one write and one rename. `runBackup` and
|
||||||
and removes them if it fails. Only the attempt that completes renames anything
|
`runMetadataBackup` are unchanged: the retry sits below them, and a file that
|
||||||
into place. The retry sits below `runBackup` and `runMetadataBackup`.
|
fails after exhausting it is still logged, counted, and stepped over.
|
||||||
`runBackup` logs, counts and steps over a file that still fails after its
|
|
||||||
retries; `runMetadataBackup`, which downloads only with `--exif`, records the
|
|
||||||
error in that file's JSON as `imageMetadataError` and goes on.
|
|
||||||
|
|
||||||
One imprecision is deliberate and worth knowing about. When a body ends part-way
|
One imprecision is deliberate and worth knowing about. When a body ends part-way
|
||||||
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
|
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
|
||||||
@@ -505,11 +422,10 @@ base64-encoded keys) that the consumer can write to disk, a database, or
|
|||||||
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
||||||
working client from that snapshot without re-authenticating; it checks every
|
working client from that snapshot without re-authenticating; it checks every
|
||||||
field and each key's length first, and throws an error naming the bad field.
|
field and each key's length first, and throws an error naming the bad field.
|
||||||
`client.logout()` clears the token and zeroes the key buffers in place; after
|
`client.logout()` clears the token and zeroes the key buffers in place; every
|
||||||
it, every other method on that client throws. It does not contact the server, so
|
later call on that client throws. It does not contact the server, so the token
|
||||||
the token stays valid there and in any saved snapshot;
|
stays valid there and in any saved snapshot; `await client.logoutOnServer()`
|
||||||
`await client.logoutOnServer()` first ends the session on the server
|
first ends the session on the server (`POST /users/logout`).
|
||||||
(`POST /users/logout`).
|
|
||||||
|
|
||||||
The CLI stores the snapshot at the platform-appropriate data directory via
|
The CLI stores the snapshot at the platform-appropriate data directory via
|
||||||
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
||||||
@@ -517,53 +433,41 @@ The CLI stores the snapshot at the platform-appropriate data directory via
|
|||||||
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
||||||
you would treat the password itself. A missing file is reported as "not logged
|
you would treat the password itself. A missing file is reported as "not logged
|
||||||
in"; a file that exists but is corrupt is reported as such, naming the bad
|
in"; a file that exists but is corrupt is reported as such, naming the bad
|
||||||
field. When the refresh a command starts with gets HTTP 401 from the server,
|
field. Both exit with status 1.
|
||||||
because it no longer accepts the saved session's token, the command prints one
|
|
||||||
line, `quak: the saved session is no longer valid; run "quak login"`, with no
|
|
||||||
stack trace. All three exit with status 3, which means the user must run
|
|
||||||
`quak login` again. `quak logout` is the exception: with no file it says there
|
|
||||||
is no session and exits 0, and it handles a corrupt file or a failed server call
|
|
||||||
as described below. `quak backup` meets an expired session on the refresh that
|
|
||||||
starts every run, before it touches any file but its lock (see "Backup layout").
|
|
||||||
A session that stops working partway through a backup instead fails each
|
|
||||||
remaining file into `failures.json`, so that run exits 1 and the next one stops
|
|
||||||
at its refresh with status 3. No command but `quak login` ever prompts.
|
|
||||||
|
|
||||||
`quak logout` ends the session on the server, so the token in `session.json`
|
`quak logout` ends the session on the server, so the token in `session.json`
|
||||||
stops working even in a copy of the file, and then deletes the file. If the
|
stops working even in a copy of the file, and then deletes the file. If the
|
||||||
server call fails (or the file is corrupt), the file is still deleted, the
|
server call fails (or the file is corrupt), the file is still deleted, the
|
||||||
command says the server session could not be ended, and it exits with status 1.
|
command says the server session could not be ended, and it exits with status 1.
|
||||||
It does not delete the cache. When it knows the cache directory, from
|
It does not delete the cache: it prints the account's cache directory and says
|
||||||
`--cache-dir` or from a session file it could read, it prints it and says it
|
it still holds decrypted data (file keys in `metadata.json`, cached originals
|
||||||
still holds decrypted data (file keys in `metadata.json`, cached originals and
|
and thumbnails), for the user to delete if they want it gone.
|
||||||
thumbnails), for the user to delete if they want it gone.
|
|
||||||
|
|
||||||
### CLI surface
|
### CLI surface
|
||||||
|
|
||||||
```
|
```
|
||||||
quak [--cache-dir <path>] <command> global: local metadata/content cache location
|
quak [--cache-dir <path>] <command> global: local metadata/content cache location
|
||||||
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
|
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
|
||||||
quak whoami print logged-in account as JSON
|
quak whoami print logged-in account as JSON
|
||||||
quak logout end the session, delete it
|
quak logout end the session, delete it
|
||||||
quak collections [--json] list all collections
|
quak collections [--json] list all collections
|
||||||
quak files --collection <id> [--json] list files in a collection
|
quak files --collection <id> [--json] list files in a collection
|
||||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||||
quak backup <dir> [--json] [--verify] full incremental backup
|
quak backup <dir> [--json] full incremental backup
|
||||||
quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON
|
quak backup-metadata <dir> [--exif] dump all decrypted metadata as JSON
|
||||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||||
quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails
|
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
|
||||||
```
|
```
|
||||||
|
|
||||||
Every command except `login`, `whoami` and `logout` runs on the cache-backed
|
Every command runs on the same cache-backed library. The read commands —
|
||||||
library. The read commands — `collections`, `files`, `get`, `get-thumb`,
|
`collections`, `files`, `get`, `get-thumb`, `backup-metadata`,
|
||||||
`backup-metadata`, `helper list-missing-thumbnails` and
|
`helper list-missing-thumbnails` and `helper fix-missing-thumbnails` — force a
|
||||||
`helper fix-missing-thumbnails` — force a fresh server round-trip before they
|
fresh server round-trip before they answer, so they report current account state
|
||||||
answer, so they report current account state rather than whatever the cache last
|
rather than whatever the cache last held. If that round-trip fails, the command
|
||||||
held. If that round-trip fails, the command prints the error on one line and
|
prints the error on one line and exits 1. `--cache-dir` overrides where the
|
||||||
exits 1, or 3 when the server no longer accepts the saved session (see "Session
|
cache lives; without it each account gets its own directory under the per-user
|
||||||
handling"). `--cache-dir` overrides where the cache lives; without it each
|
cache path.
|
||||||
account gets its own directory under the per-user cache path.
|
|
||||||
|
|
||||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||||
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
||||||
@@ -571,29 +475,18 @@ its image and its video, each named after the title with its own extension, as
|
|||||||
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
|
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
|
||||||
`--out PATH`, the image is written to `PATH` and the video beside it, with
|
`--out PATH`, the image is written to `PATH` and the video beside it, with
|
||||||
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
|
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
|
||||||
refused. `backup-metadata --exif` (alias `--all`) additionally fetches each
|
refused. `backup-metadata --exif` (alias `--all`) additionally downloads each
|
||||||
file's original through the cache and records, from it or a live photo's image,
|
file to extract full EXIF/IPTC/XMP metadata. The listing and backup commands
|
||||||
its XMP metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is
|
support `--json` for machine-readable output.
|
||||||
read with [exifreader](https://github.com/mattiasw/ExifReader) from any image
|
|
||||||
format it reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's
|
|
||||||
`exif` field is exifreader's EXIF tag output: each tag by name, with its
|
|
||||||
`value`, `description` and `computed` value. An EXIF block exifreader finds but
|
|
||||||
reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
|
||||||
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
|
|
||||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
|
||||||
|
|
||||||
`backup --verify` also hashes the originals already in the backup and replaces
|
|
||||||
any that do not match the content hash Ente records; one it cannot replace goes
|
|
||||||
into `failures.json` (see "Backup layout").
|
|
||||||
|
|
||||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||||
fails, the error is logged, each of its files is written with the reason in an
|
still fails after its retries, the error is logged, each of its files is written
|
||||||
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
with the reason in an `mlDataError` field instead of `mlData`, and the dump goes
|
||||||
non-zero if any ML data request failed.
|
on. The exit code is non-zero if any ML data request failed.
|
||||||
|
|
||||||
`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only,
|
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
|
||||||
because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image
|
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
|
||||||
(PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
||||||
distinct from a `failed` repair, and does not affect the exit code; a genuine
|
distinct from a `failed` repair, and does not affect the exit code; a genuine
|
||||||
failure still exits non-zero. The server accepts a new thumbnail only from the
|
failure still exits non-zero. The server accepts a new thumbnail only from the
|
||||||
file's owner and only when it is no larger than the thumbnail size it records
|
file's owner and only when it is no larger than the thumbnail size it records
|
||||||
@@ -609,83 +502,23 @@ the smallest does not.
|
|||||||
|
|
||||||
```
|
```
|
||||||
<dir>/
|
<dir>/
|
||||||
YYYY/YYYY-MM/YYYY-MM-DD/
|
originals/
|
||||||
YYYY-MM-DD.<fileID>.<ext> actual file content, at its save path (one
|
<fileID>.<ext> actual file content (one per unique file,
|
||||||
per unique file, two for a live photo: see
|
two for a live photo: see below)
|
||||||
below)
|
<fileID>.json all decrypted metadata for that file
|
||||||
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
<fileID>.livephoto.json which of a live photo's two files is which
|
||||||
keeps, its update time, its private and
|
|
||||||
public magic metadata, its ML data, and
|
|
||||||
its original's EXIF, XMP and dimensions
|
|
||||||
YYYY-MM-DD.<fileID>.livephoto.json
|
|
||||||
which of a live photo's two files is which
|
|
||||||
collections/
|
collections/
|
||||||
<name>/
|
<name>/
|
||||||
<title> -> ../../YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>
|
<title> -> ../../originals/<fileID>.<ext> (symlink)
|
||||||
(symlink)
|
<name>.json collection metadata + file list
|
||||||
<name>.json collection metadata + file list
|
failures.json files that failed and have not yet succeeded
|
||||||
account.json the account's email and user ID
|
|
||||||
backup.lock the lock a running backup holds (see below)
|
|
||||||
failures.json files that failed and have not yet succeeded
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Each original is saved at its save path, the same path `photo.savePath` gives
|
|
||||||
and `photo.download()` writes (see Read surface below). The date is the photo's
|
|
||||||
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
|
|
||||||
time zone of the machine running quak. The extension is the one in the file's
|
|
||||||
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
|
|
||||||
letters and digits. When the date or the time zone changes, the next run saves
|
|
||||||
the original at its new path and leaves the old copy where it is.
|
|
||||||
|
|
||||||
`account.json` holds the account's `email` and `userID`, as `backup-metadata`
|
|
||||||
writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`,
|
|
||||||
`isShared` and `updationTime`, its `magicMetadata`, `pubMagicMetadata` and
|
|
||||||
`sharedMagicMetadata` when it has them, as `backup-metadata`'s
|
|
||||||
`_collection.json` does, and `files`, each file's `id` and `metadata`. A file's
|
|
||||||
JSON holds its `id`, `collectionID`, `ownerID`, `metadata` and `updationTime`,
|
|
||||||
and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times
|
|
||||||
are in microseconds, as Ente records them.
|
|
||||||
|
|
||||||
A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
|
|
||||||
`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data
|
|
||||||
for has no `mlData`. The backup waits for the library's ML data fetch to finish
|
|
||||||
before it writes the JSON files. If that fetch fails, each file whose ML data is
|
|
||||||
not in the cache gets the reason in `mlDataError` instead and counts as failed,
|
|
||||||
and the next run fetches it again. The JSON files are rewritten on every run, so
|
|
||||||
ML data that arrived since the last run appears.
|
|
||||||
|
|
||||||
A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
|
|
||||||
from its original, or from a live photo's image: `format`, `width` and `height`
|
|
||||||
for a JPEG, `exif` (or `exifRaw` and `exifError`), and `xmp`. An original with
|
|
||||||
none of these gets `{}`. A video gets no `imageMetadata`: reading a whole video
|
|
||||||
to look for tags is not worth it. When the original cannot be read, the reason
|
|
||||||
is in `imageMetadataError` instead; neither the file nor the run fails. The
|
|
||||||
original is read when the run stores it, or when the JSON beside it has neither
|
|
||||||
field, as one an earlier version wrote. Otherwise the field is taken from that
|
|
||||||
JSON when it is rewritten, so a run does not read every stored original again.
|
|
||||||
|
|
||||||
`failures.json` records each failed file with the kind of failure, how many
|
`failures.json` records each failed file with the kind of failure, how many
|
||||||
times it has been tried and when it was last tried. A file leaves it once it
|
times it has been tried and when it was last tried. A file leaves it once it
|
||||||
succeeds, or once it is no longer in the library or in the backup's scope. The
|
succeeds, or once it is no longer in the library or in the backup's scope. The
|
||||||
library's `lib.backup({ includeThumbnails: true })` also writes
|
library's `lib.backup({ includeThumbnails: true })` also writes
|
||||||
`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not.
|
`thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
|
||||||
|
|
||||||
`backup.lock` keeps two backups of the same directory from running at once, such
|
|
||||||
as a cron run that starts while the previous one is still going. A backup
|
|
||||||
creates `<dir>` if it is missing and takes the lock before its refresh, and
|
|
||||||
removes the lock when it ends, whether it succeeds or fails; `quak backup` takes
|
|
||||||
it before it opens its library, so a refused run sends no request. The lock is a
|
|
||||||
directory that
|
|
||||||
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates
|
|
||||||
and keeps touching while the backup runs. A second backup of the directory, from
|
|
||||||
another process or the same one, fails at once: `quak backup` prints
|
|
||||||
`quak: another backup of <dir> is running` and exits with status 2. A run
|
|
||||||
stopped with Ctrl-C or `kill` removes the lock as it exits. Only a run that
|
|
||||||
cannot, such as one killed with SIGKILL or cut off by a crash or power loss,
|
|
||||||
leaves it behind; once it has gone 10 seconds untouched, the next run takes it
|
|
||||||
over, so nobody has to remove it. The lock is outside the date folders and
|
|
||||||
`collections/`, so it is never taken for an original, and the removal of old
|
|
||||||
album directories never touches it.
|
|
||||||
|
|
||||||
A collection's directory and JSON are named after the collection, and a symlink
|
A collection's directory and JSON are named after the collection, and a symlink
|
||||||
after the file's title, both with unsafe characters replaced. When two
|
after the file's title, both with unsafe characters replaced. When two
|
||||||
@@ -697,65 +530,45 @@ name stays the same from run to run until such a clash appears or goes away.
|
|||||||
|
|
||||||
A live photo, which Ente stores as one ZIP of its image and its video, is stored
|
A live photo, which Ente stores as one ZIP of its image and its video, is stored
|
||||||
as those two files, which a photo viewer can open: each is
|
as those two files, which a photo viewer can open: each is
|
||||||
`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP (for
|
`originals/<fileID>.<ext>` with the extension it has inside the ZIP (for example
|
||||||
example `2026-03-01.12345.heic` and `2026-03-01.12345.mov`), and
|
`12345.heic` and `12345.mov`), and `<fileID>.livephoto.json` names the two. The
|
||||||
`YYYY-MM-DD.<fileID>.livephoto.json` names the two. The live photo counts as
|
live photo counts as stored only when both files are present and not empty. Its
|
||||||
stored only when both files are present and not empty. Its album folder links
|
album folder links both, each named after the title with that file's extension
|
||||||
both, each named after the title with that file's extension (`IMG_0001.heic` and
|
(`IMG_0001.heic` and `IMG_0001.mov`). A live photo that an earlier version of
|
||||||
`IMG_0001.mov`).
|
quak stored as the ZIP, under the image's name, is replaced by its two files on
|
||||||
|
the next run, and the ZIP and its link are removed.
|
||||||
|
|
||||||
Each run removes the symlinks into the date folders that no longer belong in
|
Each run removes the symlinks into `originals/` that no longer belong in their
|
||||||
their collection's directory, and the directories (and JSON) of collections that
|
collection's directory, and the directories (and JSON) of collections that were
|
||||||
were deleted or renamed. Nothing else in `collections/` is touched: a file or a
|
deleted or renamed. Nothing else in `collections/` is touched: a file or a
|
||||||
symlink you put there stays, and a directory that still holds one after its
|
symlink you put there stays, and a directory that still holds one after its
|
||||||
symlinks are removed stays too, with its JSON.
|
symlinks are removed stays too, with its JSON.
|
||||||
|
|
||||||
Each file is downloaded exactly once regardless of how many collections it
|
Each file is downloaded exactly once regardless of how many collections it
|
||||||
appears in, and written once: straight to its save path, with no copy left in
|
appears in, and written once: straight into `originals/`, with no copy left in
|
||||||
the cache. An original the cache already held is copied from there instead. On
|
the cache. An original the cache already held is copied from there instead. On
|
||||||
subsequent runs, existing originals are skipped. If a download fails, the error
|
subsequent runs, existing originals are skipped. If a download fails, the error
|
||||||
is logged and the backup continues with the next file. The exit code is non-zero
|
is logged and the backup continues with the next file. The exit code is non-zero
|
||||||
if any files failed. `quak backup` opens its library with the thumbnail and
|
if any files failed. `quak backup` opens its library with the thumbnail and
|
||||||
originals precache off, so the only file content it fetches is the originals the
|
originals precache off, so it fetches only what the backup stores.
|
||||||
backup stores.
|
|
||||||
|
|
||||||
With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each
|
|
||||||
original already at its save path the way a download is checked (see "On-disk
|
|
||||||
cache layout" below): its bytes, read in chunks, or a live photo's image and
|
|
||||||
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
|
|
||||||
hash its metadata records is left as it is. One that does not is logged on one
|
|
||||||
line naming the file, deleted (a live photo's image and video both), and put
|
|
||||||
back in the same run like a missing one: downloaded, or copied from the cache if
|
|
||||||
the cache holds it. What is put back is hashed too, because a copy from the
|
|
||||||
cache is not checked as a download is. If it still does not match, it stays at
|
|
||||||
its save path and the file goes into `failures.json`, as it does when the
|
|
||||||
download fails. A file whose metadata records no hash is left as it is and
|
|
||||||
counted as unchecked. A stored original that cannot be read is left as it is and
|
|
||||||
counts as failed. The summary and `--json` add the counts `verified`,
|
|
||||||
`mismatched` and `unchecked`, all of originals that were already stored; one
|
|
||||||
first downloaded in this run is in none of them. A mismatch that was put back
|
|
||||||
with matching bytes does not make the exit code non-zero. Without `--verify`
|
|
||||||
nothing is hashed, the summary is unchanged, and the three counts are 0 in
|
|
||||||
`--json`.
|
|
||||||
|
|
||||||
Each original is written to a temporary file in the same directory, synced to
|
Each original is written to a temporary file in the same directory, synced to
|
||||||
disk, and renamed into place, so an original is either complete or absent, even
|
disk, and renamed into place, so an original is either complete or absent, even
|
||||||
after a power cut. A downloaded original's temporary file is named
|
after a power cut. A downloaded original's temporary file is named
|
||||||
`.quak-<pid>-<random>.tmp`, one copied from the cache
|
`.quak-<pid>-<random>.tmp`, one copied from the cache
|
||||||
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is
|
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
|
||||||
killed can leave one of these temporary files behind; the next backup deletes
|
one of these temporary files behind; the next backup deletes those whose process
|
||||||
those whose process is no longer running. The content cache uses the same
|
is no longer running. The content cache uses the same scheme, and opening a
|
||||||
scheme, and opening a library deletes the temporary files in the cache whose
|
library deletes the temporary files in the cache whose process is no longer
|
||||||
process is no longer running, so a download another process has in progress in
|
running, so a download another process has in progress in the same cache is left
|
||||||
the same cache is left alone. The rename replaces whatever was at the
|
alone. The rename replaces whatever was at the destination rather than writing
|
||||||
destination rather than writing through it: a symlink there is replaced, not
|
through it: a symlink there is replaced, not followed, and the new file has the
|
||||||
followed, and the new file has the temporary file's permissions, not those of
|
temporary file's permissions, not those of the file it replaced.
|
||||||
the file it replaced.
|
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
- [x] Retry policy: no retry on 4xx (except `408` and `429`), exponential
|
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||||
backoff on 5xx and network errors
|
errors
|
||||||
- [x] Update the API reference section below to match the current implementation
|
- [x] Update the API reference section below to match the current implementation
|
||||||
- [x] `make docker` green
|
- [x] `make docker` green
|
||||||
- [x] Store live photos in a form a photo viewer can open
|
- [x] Store live photos in a form a photo viewer can open
|
||||||
@@ -778,8 +591,7 @@ Future (desktop client, separate repo):
|
|||||||
The library's primary surface is the cache-backed `Library`; the lower-level
|
The library's primary surface is the cache-backed `Library`; the lower-level
|
||||||
`Client` sits underneath it and is covered by the Design sections above. The
|
`Client` sits underneath it and is covered by the Design sections above. The
|
||||||
test suite is the canonical, executable documentation — `test/library/` and
|
test suite is the canonical, executable documentation — `test/library/` and
|
||||||
`test/client/usage.test.ts` walk most operations, `test/cli/backup.test.ts`
|
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.
|
||||||
walks `lib.backup()`, and `yarn test` verifies them.
|
|
||||||
|
|
||||||
### Opening a library
|
### Opening a library
|
||||||
|
|
||||||
@@ -792,24 +604,21 @@ background, so an unreachable server does not block opening.
|
|||||||
|
|
||||||
`LibraryOptions`:
|
`LibraryOptions`:
|
||||||
|
|
||||||
| Option | Default | Meaning |
|
| Option | Default | Meaning |
|
||||||
| ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------- |
|
| ------------------------ | --------------------------- | --------------------------------------------------------------------- |
|
||||||
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
||||||
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
|
| `cacheDirectory` | `<XDG cache>/quak/<userID>` | where `metadata.json` and the content cache live |
|
||||||
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached |
|
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
|
||||||
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
||||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||||
| `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it |
|
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache |
|
||||||
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
||||||
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
||||||
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
||||||
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
|
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
|
||||||
| `contentSource` | the client's own | override the byte source (mainly for tests) |
|
| `contentSource` | the client's own | override the byte source (mainly for tests) |
|
||||||
|
|
||||||
The default `downloadDirectory` is resolved against the working directory once,
|
|
||||||
when the library opens; `lib.downloadDirectory` holds the result.
|
|
||||||
|
|
||||||
Concurrency is set through `pools`: construct
|
Concurrency is set through `pools`: construct
|
||||||
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
||||||
@@ -826,21 +635,17 @@ can then be removed.
|
|||||||
### Default reads vs. fresh reads
|
### Default reads vs. fresh reads
|
||||||
|
|
||||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||||
synchronously from the copy held in RAM and never touch the network. A refresh
|
synchronously from the last refreshed copy held in RAM and never touch the
|
||||||
changes that copy only once all its server requests have succeeded, and the
|
network. The background timer refreshes that copy every
|
||||||
background timer starts the next refresh `refreshIntervalSeconds` after the
|
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
|
||||||
previous one ends. So a default read is immediate, but only as current as the
|
interval stale.
|
||||||
last refresh whose requests all succeeded; right after opening an existing
|
|
||||||
cache, it is the copy on disk.
|
|
||||||
|
|
||||||
`await lib.fresh()` waits for a refresh to complete and persist, and returns the
|
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
|
||||||
same `{ albums, photos, timeline }` namespaces, which then reflect a completed
|
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
|
||||||
server round-trip. When a refresh is already running, background or not,
|
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
|
||||||
`fresh()` waits for that one, so its answer can come from requests made before
|
one refresh, and a refresh that fails rejects the caller (default reads stay
|
||||||
the call; only when none is running does it start one. A refresh that fails
|
silent and keep serving the last good copy). The CLI's read commands use fresh
|
||||||
rejects the caller (default reads stay silent and keep serving the last good
|
reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||||
copy). The CLI's read commands use fresh reads (issue
|
|
||||||
https://git.eeqj.de/sneak/quak/issues/75).
|
|
||||||
|
|
||||||
### Read surface
|
### Read surface
|
||||||
|
|
||||||
@@ -857,61 +662,17 @@ https://git.eeqj.de/sneak/quak/issues/75).
|
|||||||
`includeArchived`; hidden photos are always excluded.
|
`includeArchived`; hidden photos are always excluded.
|
||||||
|
|
||||||
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
||||||
(newest first). A `Photo` exposes its record fields other than `thumbnailPath`
|
(newest first). A `Photo` exposes its record fields, `photo.record()` →
|
||||||
and `originalPath`, and `photo.year`, the local-time year of `takenAt`, all as
|
`PhotoRecord`, and two content methods:
|
||||||
synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more
|
|
||||||
synchronous getters look at the disk and never touch the network:
|
|
||||||
|
|
||||||
- `photo.savePath` → `string` — where `photo.download()` and `lib.backup()` put
|
|
||||||
the original, whether or not it is there yet:
|
|
||||||
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
|
||||||
`downloadDirectory` (see Backup layout above for the date and the extension).
|
|
||||||
For a live photo already stored, its image. For a live photo not yet stored,
|
|
||||||
it carries the title's extension, and the image may be stored under a
|
|
||||||
different one, found inside the live photo. It needs no content source.
|
|
||||||
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`. A
|
|
||||||
copy only in the cache does not count.
|
|
||||||
|
|
||||||
These async methods may download:
|
|
||||||
|
|
||||||
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
|
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
|
||||||
full-resolution file. For a live photo, `path` and `bytes` are its image's and
|
full-resolution file. For a live photo, `path` and `bytes` are its image's and
|
||||||
`videoPath` is its video.
|
`videoPath` is its video.
|
||||||
- `await photo.download()` → `{ path, bytes, videoPath? }`, as `original()` —
|
|
||||||
puts the original at `savePath`, creating its folders. When it is already
|
|
||||||
there, nothing is written. When the cache holds it, it is copied from the
|
|
||||||
cache; otherwise it is fetched straight to `savePath`, with no copy left in
|
|
||||||
the cache. Afterwards `isLocal` is true.
|
|
||||||
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
|
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
|
||||||
- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read
|
|
||||||
through `original()`; for a live photo, its image's.
|
|
||||||
- `await photo.exif(opts?)` → `ExifTags` — every EXIF tag in the file, keyed by
|
|
||||||
tag name, each as exifreader decodes it, with its `id`, `value`, `description`
|
|
||||||
and `computed` value: for example `Make` is
|
|
||||||
`{ id: 271, value: ["Canon"], description: "Canon", computed: "Canon" }`. A
|
|
||||||
tag exifreader has no name for is keyed `undefined-<tag number>`. The embedded
|
|
||||||
thumbnail's tags are under `Thumbnail`, so they cannot hide the main image's
|
|
||||||
tags of the same name; the thumbnail image itself is left out. EXIF is read
|
|
||||||
from any image format exifreader reads (such as JPEG, HEIC/HEIF, AVIF, PNG,
|
|
||||||
WebP and TIFF), a live photo's image included. Any other original gives `{}`,
|
|
||||||
and a video gives `{}` without being downloaded.
|
|
||||||
- `await photo.make(opts?)`, and likewise `model()`, `lensModel()`,
|
|
||||||
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
|
||||||
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
|
||||||
and `gpsAltitude()` → one common field each, picked from the tags `exif()`
|
|
||||||
returns and typed as in `PhotoExif`, or `undefined` when the file lacks it.
|
|
||||||
GPS values are signed decimal degrees and metres. `dateTimeOriginal()` is the
|
|
||||||
camera's clock reading held in the `Date`'s UTC fields;
|
|
||||||
`offsetTimeOriginal()`, when present, is that clock's offset from UTC. Each
|
|
||||||
calls `exif()` with its `opts`, so each call reads the original again.
|
|
||||||
|
|
||||||
They serve from the on-disk content cache when the bytes are present and
|
Both serve from the on-disk content cache when the bytes are present and
|
||||||
otherwise fetch through the pools; `original()`, `content()` and `exif()` also
|
otherwise fetch through the pools; `opts.onProgress` reports per-file progress.
|
||||||
serve an original already stored at its save path. `opts.onProgress` reports
|
They throw when the library was opened without a content source.
|
||||||
per-file progress. They throw when the library was opened without a content
|
|
||||||
source. An original that `original()`, `content()` or `exif()` downloads lands
|
|
||||||
in the cache, which does not make `isLocal` true; only `download()` and
|
|
||||||
`lib.backup()` do.
|
|
||||||
|
|
||||||
Lower-level accessors that return decrypted model objects (which hold key
|
Lower-level accessors that return decrypted model objects (which hold key
|
||||||
material) are also available: `listCollections()`, `getCollection(id)`,
|
material) are also available: `listCollections()`, `getCollection(id)`,
|
||||||
@@ -923,12 +684,10 @@ material) are also available: `listCollections()`, `getCollection(id)`,
|
|||||||
The GUI-facing records hold no key material and no binary, so they survive
|
The GUI-facing records hold no key material and no binary, so they survive
|
||||||
`structuredClone`/JSON across the Electron IPC boundary:
|
`structuredClone`/JSON across the Electron IPC boundary:
|
||||||
|
|
||||||
- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` and `modifiedAt`
|
- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds),
|
||||||
(milliseconds), `fileType`, optional `caption` / `width` / `height` /
|
`fileType`, optional `caption` / `width` / `height` / `latitude` /
|
||||||
`latitude` / `longitude`, optional `hash` (the content hash recorded at
|
`longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath`
|
||||||
upload; very old files have none), `isArchived`, `isHidden`, and
|
once the bytes are cached (for a live photo, `originalPath` is its image).
|
||||||
`thumbnailPath` / `originalPath` once the bytes are cached (for a live photo,
|
|
||||||
`originalPath` is its image).
|
|
||||||
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
|
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
|
||||||
`fileIDs` (newest first).
|
`fileIDs` (newest first).
|
||||||
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
||||||
@@ -953,23 +712,16 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
|||||||
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
||||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||||
a query vector the caller produced elsewhere.
|
a query vector the caller produced elsewhere.
|
||||||
- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
|
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
||||||
directory, and fails at once with an error whose `code` is `ELOCKED` while
|
in-scope original not already in the backup (and, with `includeThumbnails`,
|
||||||
another backup of it runs. A caller can instead take the lock itself, before
|
thumbnails) through the content cache, and rebuilds the on-disk backup tree
|
||||||
it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)`
|
with a durable failure ledger. A fetched original is written straight into the
|
||||||
takes it, failing the same way, and returns the function that releases it, and
|
backup's `originals/` and not into the cache, which then counts it as present;
|
||||||
the caller passes `lockHeld: true` to the backup. It waits for a refresh as
|
one the cache already held is copied from there. `BackupOptions`:
|
||||||
`fresh()` does, puts every in-scope original not already at its save path
|
`downloadDirectory` (falls back to the one `open()` was given),
|
||||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
|
||||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
|
||||||
rebuilds the on-disk backup tree, each file's JSON with its ML data and its
|
|
||||||
original's EXIF, XMP and dimensions, with a durable failure ledger. A fetched
|
|
||||||
original is written straight to its save path and not into the cache, which
|
|
||||||
then counts it as present; one the cache already held is copied from there.
|
|
||||||
`BackupOptions`: `downloadDirectory` (falls back to the library's),
|
|
||||||
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
||||||
`onlyAlbumNames`, `verify` (default `false`), `onProgress`, and `lockHeld`
|
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
|
||||||
(default `false`). See Backup layout above for the tree it writes.
|
writes.
|
||||||
|
|
||||||
### Request pools
|
### Request pools
|
||||||
|
|
||||||
@@ -1000,9 +752,12 @@ When `metadata.json` belongs to a different account than the client's,
|
|||||||
originals and thumbnails are kept; they are reached only through the files the
|
originals and thumbnails are kept; they are reached only through the files the
|
||||||
current account's records name.
|
current account's records name.
|
||||||
|
|
||||||
A live photo's original is cached as at its save path: its image and its video,
|
A live photo's original is cached as in the backup: its image and its video,
|
||||||
each `originals/<fileID>.<ext>` with its own extension, and
|
each `originals/<fileID>.<ext>` with its own extension, and
|
||||||
`originals/<fileID>.livephoto.json` naming them; the two are evicted together.
|
`originals/<fileID>.livephoto.json` naming them; the two are evicted together. A
|
||||||
|
live photo that an earlier version cached as its ZIP is not served: the library
|
||||||
|
removes the ZIP when it opens the cache, and fetches the two files when the
|
||||||
|
photo is next read or precached.
|
||||||
|
|
||||||
A stored file appears only via an atomic temp-then-rename, so its presence means
|
A stored file appears only via an atomic temp-then-rename, so its presence means
|
||||||
it is complete. Every downloaded original (by `quak get`, the cache, or
|
it is complete. Every downloaded original (by `quak get`, the cache, or
|
||||||
@@ -1021,13 +776,12 @@ from a very old client, is stored unchecked.
|
|||||||
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
|
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
|
||||||
`LibraryClient`, `RefreshEvent`
|
`LibraryClient`, `RefreshEvent`
|
||||||
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
|
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
|
||||||
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`, `SavePathLookup`
|
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
|
||||||
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
|
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
|
||||||
`EnsureOptions`, `EnsureResult`, `ContentSource`
|
`EnsureOptions`, `EnsureResult`, `ContentSource`
|
||||||
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
|
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
|
||||||
`LibraryChange`
|
`LibraryChange`
|
||||||
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
||||||
- `src/exif.ts`: `ExifTags`, `PhotoExif`
|
|
||||||
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
||||||
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
||||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||||
@@ -1059,17 +813,18 @@ documents:
|
|||||||
`yarn.lock`. Never `git add -A`. Never force-push to main.
|
`yarn.lock`. Never `git add -A`. Never force-push to main.
|
||||||
|
|
||||||
- **The "Development workflow" section above.** All changes go on feature
|
- **The "Development workflow" section above.** All changes go on feature
|
||||||
branches off `next`, and every pull request targets `next`; only sneak merges
|
branches. Tests are written first and committed in a failing state before the
|
||||||
`next` into `main`. Tests are written first and committed in a failing state
|
implementation. Tests are the canonical API documentation and must be
|
||||||
before the implementation. Tests are the canonical API documentation and must
|
commented thoroughly. `main` is always green.
|
||||||
be commented thoroughly. `main` and `next` are always green.
|
|
||||||
|
|
||||||
- **Required checks before every commit:** `make lint` and `make fmt-check` must
|
- **Required checks before every commit:** `make lint` must pass — that is
|
||||||
pass. `make lint` is eslint plus the prettier check, and it builds the `lint`
|
eslint plus the prettier check, and it builds the `lint` phase of the
|
||||||
phase of the `Dockerfile`, so it needs docker. The pre-commit hook enforces
|
`Dockerfile`, so it needs docker. The pre-commit hook enforces exactly that.
|
||||||
exactly that. `make check` (which also runs the tests) must pass before
|
`make check` (which also runs the tests) must pass before merging to `main`.
|
||||||
merging into `next`. Never invoke eslint or prettier directly; linting runs in
|
`make fmt-check` is available for a host-side formatting check on its own, but
|
||||||
the container only.
|
it is not a separate requirement: `make lint` already covers it, and running
|
||||||
|
both would check formatting twice. Never invoke eslint or prettier directly;
|
||||||
|
linting runs in the container only.
|
||||||
|
|
||||||
- **Formatting:** prettier with 4-space indents and `proseWrap: always` for
|
- **Formatting:** prettier with 4-space indents and `proseWrap: always` for
|
||||||
markdown. Use `make fmt` to format. Use `yarn` not `npm`.
|
markdown. Use `make fmt` to format. Use `yarn` not `npm`.
|
||||||
|
|||||||
+44
-120
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-10-04
|
last_modified: 2026-09-08
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -104,14 +104,10 @@ style conventions are in separate documents:
|
|||||||
`lint` phase and a `test` phase, with the final stage depending on both so the
|
`lint` phase and a `test` phase, with the final stage depending on both so the
|
||||||
image cannot be built unless they pass. For non-server repos the final stage
|
image cannot be built unless they pass. For non-server repos the final stage
|
||||||
brings up a development environment; for server repos it is the runtime image.
|
brings up a development environment; for server repos it is the runtime image.
|
||||||
The gate phases and the build stage start from their pinned base images and
|
Dockerfiles install development prerequisites by running `script/bootstrap`
|
||||||
install what those images lack either inline, as the canonical Go `Dockerfile`
|
rather than duplicating installs inline; COPY `script/` and the dependency
|
||||||
below does for `git`, or by running `script/bootstrap`, as the `prompts`
|
manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before
|
||||||
repo's own `Dockerfile` does for its yarn packages. The development
|
running it.
|
||||||
environment stage installs development prerequisites by running
|
|
||||||
`script/bootstrap` rather than duplicating its installs inline. A stage that
|
|
||||||
runs `script/bootstrap` COPYs `script/` and the dependency manifests
|
|
||||||
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
|
|
||||||
|
|
||||||
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
||||||
no separate lint file. `script/lint` and `script/test` each build one phase
|
no separate lint file. `script/lint` and `script/test` each build one phase
|
||||||
@@ -160,14 +156,11 @@ style conventions are in separate documents:
|
|||||||
not evidence that anything ran: a sub-second build reporting success is a
|
not evidence that anything ran: a sub-second build reporting success is a
|
||||||
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
||||||
and friends destroy a build cache shared with every other build on the host.
|
and friends destroy a build cache shared with every other build on the host.
|
||||||
When a check is added or changed, prove it works by planting a defect it must
|
|
||||||
catch and watching the run fail on it, then revert the defect. A green run
|
|
||||||
alone shows neither that the check ran nor that it covers what it should.
|
|
||||||
|
|
||||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
hash), so lint failures surface in seconds rather than after a full compile,
|
hash), so lint failures surface in seconds rather than after a full compile,
|
||||||
and the test phase is based on the Debian Go image. The canonical Go repo
|
and the test phase is based on the Go image. The canonical Go repo
|
||||||
`Dockerfile`:
|
`Dockerfile`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
@@ -180,9 +173,8 @@ style conventions are in separate documents:
|
|||||||
COPY . .
|
COPY . .
|
||||||
RUN golangci-lint run --config .golangci.yml ./...
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
|
|
||||||
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
# Test phase
|
||||||
# image ships and the alpine one does not.
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
# golang:1.x, YYYY-MM-DD
|
|
||||||
FROM golang@sha256:... AS test
|
FROM golang@sha256:... AS test
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
@@ -199,29 +191,15 @@ style conventions are in separate documents:
|
|||||||
FROM golang@sha256:... AS builder
|
FROM golang@sha256:... AS builder
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
COPY --from=test /src/go.sum /dev/null
|
COPY --from=test /src/go.sum /dev/null
|
||||||
RUN apk add --no-cache git
|
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
|
||||||
RUN git config --system --add safe.directory /src
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# The VERSION build arg when one is given, otherwise
|
ARG VERSION=dev
|
||||||
# `git describe --tags --always` on the .git in the build context. With
|
RUN CGO_ENABLED=0 go build -trimpath \
|
||||||
# .git present, a version that is still empty, dev or unknown fails the
|
-ldflags="-s -w -X main.Version=${VERSION}" \
|
||||||
# build: git is missing or could not read the checkout.
|
-o /app ./cmd/app/
|
||||||
ARG VERSION
|
|
||||||
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
|
||||||
if [ -e .git ]; then \
|
|
||||||
case "$VERSION" in ""|dev|unknown) \
|
|
||||||
echo "version is '$VERSION' although .git is present" >&2; \
|
|
||||||
exit 1 ;; \
|
|
||||||
esac; \
|
|
||||||
fi; \
|
|
||||||
CGO_ENABLED=0 go build -trimpath \
|
|
||||||
-ldflags="-s -w -X main.Version=${VERSION}" \
|
|
||||||
-o /app ./cmd/app/
|
|
||||||
|
|
||||||
# Runtime stage, and the last one
|
# Runtime stage, and the last one
|
||||||
FROM alpine@sha256:...
|
FROM alpine@sha256:...
|
||||||
@@ -243,41 +221,10 @@ style conventions are in separate documents:
|
|||||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
||||||
- If the project requires CGO or system libraries for linting, install them
|
- If the project requires CGO or system libraries for linting (e.g.
|
||||||
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
`vips-dev`), install them in the lint phase with `apk add`.
|
||||||
has no `apk`, so install with `apt-get` under the Debian package name
|
- `ARG VERSION=dev` is declared in the stage that compiles and supplied by
|
||||||
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
`script/docker` and `script/cibuild`; no stage may call `git describe`.
|
||||||
lists in the same `RUN`, so the layer does not keep them:
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
RUN apt-get update \
|
|
||||||
&& apt-get install -y --no-install-recommends libvips-dev \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
```
|
|
||||||
|
|
||||||
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
|
||||||
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
|
|
||||||
repository's own, each submodule's under `.git/modules/`, and that of a
|
|
||||||
submodule keeping its own `.git` directory. `git describe` does not need
|
|
||||||
them, and each can hold a credential: a password in a remote URL, or the
|
|
||||||
token the CI checkout step stores there. A submodule whose name has a
|
|
||||||
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
|
|
||||||
git directory to `**/.git/modules/**/config`, and Go's version stamping
|
|
||||||
then fails the build: give it a name without that segment
|
|
||||||
(`git submodule add --name`). The stage that compiles has `git` (the
|
|
||||||
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
|
|
||||||
takes the version from the `VERSION` build argument when one is given,
|
|
||||||
otherwise from `git describe --tags --always`. That gives the tag on a
|
|
||||||
tagged commit; on a later commit, the tag, the number of commits since it
|
|
||||||
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
|
|
||||||
tag is reachable. The stage that compiles also marks its working directory
|
|
||||||
safe for git (`git config --system --add safe.directory /src`): a context
|
|
||||||
sent as a tar stream keeps the sender's file owners, and git refuses a
|
|
||||||
checkout owned by another user, so the version would come out empty.
|
|
||||||
`ARG VERSION` has no default, and the build fails if the context carries
|
|
||||||
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
|
||||||
`docker build .` with no build arguments must succeed; a Dockerfile that
|
|
||||||
refuses an empty build argument drops that refusal and keeps the argument.
|
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
||||||
@@ -286,12 +233,7 @@ style conventions are in separate documents:
|
|||||||
carry the same guarantee, because its gate phases may come from the cache. The
|
carry the same guarantee, because its gate phases may come from the cache. The
|
||||||
image build is uncached and so runs the gate phases a second time. That is the
|
image build is uncached and so runs the gate phases a second time. That is the
|
||||||
price of the rule above, and it is worth paying: the image that ships is built
|
price of the rule above, and it is worth paying: the image that ships is built
|
||||||
from a run of its own gates rather than from a cache entry. A separate
|
from a run of its own gates rather than from a cache entry.
|
||||||
workflow limited to `main` by a `branches` list under `on: push` cannot be
|
|
||||||
checked by review: to try a change to it, add the feature branch to that list
|
|
||||||
and push, then remove the branch from the list again before merging. Keep any
|
|
||||||
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
|
|
||||||
from the feature branch publishes nothing.
|
|
||||||
|
|
||||||
- Use platform-standard formatters: `black` for Python, `prettier` for
|
- Use platform-standard formatters: `black` for Python, `prettier` for
|
||||||
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
||||||
@@ -344,19 +286,17 @@ style conventions are in separate documents:
|
|||||||
```
|
```
|
||||||
|
|
||||||
`-count=1` is required on both invocations: it defeats Go's test _result_
|
`-count=1` is required on both invocations: it defeats Go's test _result_
|
||||||
cache, so neither run can report a stored pass in place of running the
|
cache, so the target cannot report a pass it did not earn, and the rerun
|
||||||
tests. It leaves the build cache alone, so it costs the runtime of the suite
|
reproduces a failure instead of replaying it. It leaves the build cache
|
||||||
and no recompilation.
|
alone, so it costs the runtime of the suite and no recompilation.
|
||||||
|
|
||||||
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
Note that this is a second, independent cache, stacked below the Docker
|
||||||
passing result in its cache directory (`GOCACHE`), and when the same tests
|
layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26)
|
||||||
run again on unchanged code it prints that result, marked `(cached)`,
|
addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes;
|
||||||
without running them. That matters on a developer's machine, where this
|
it does not guarantee `go test` inside that step does any work, because the
|
||||||
target runs and the directory lasts from one run to the next. The `test`
|
`GOCACHE` baked into earlier image layers survives into the re-executed
|
||||||
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
|
step. They are two separate defects requiring two separate fixes, and a fix
|
||||||
result for this repo's tests and nothing before its `go test` step runs a
|
for one must not be recorded as covering the other.
|
||||||
test, so there is nothing to replay. `--no-cache` (above) is what makes that
|
|
||||||
step run on an unchanged tree.
|
|
||||||
|
|
||||||
Python example:
|
Python example:
|
||||||
|
|
||||||
@@ -400,7 +340,7 @@ style conventions are in separate documents:
|
|||||||
— which is more dangerous than a short file with no secret patterns at all,
|
— which is more dangerous than a short file with no secret patterns at all,
|
||||||
because it reads as solved and stops anyone looking. Give every
|
because it reads as solved and stops anyone looking. Give every
|
||||||
depth-independent pattern the `**/` prefix and leave only genuinely
|
depth-independent pattern the `**/` prefix and leave only genuinely
|
||||||
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
|
root-anchored entries unprefixed: `.git`, and the repo's own host-built
|
||||||
binary, written `/myapp` and never `**/myapp`, which would also match
|
binary, written `/myapp` and never `**/myapp`, which would also match
|
||||||
`cmd/myapp/` and delete the package directory from the context. Matching is
|
`cmd/myapp/` and delete the package directory from the context. Matching is
|
||||||
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
||||||
@@ -425,13 +365,12 @@ style conventions are in separate documents:
|
|||||||
directory, so a repo running agents in subdirectories still ships
|
directory, so a repo running agents in subdirectories still ships
|
||||||
`services/api/.claude/` and must add its own anchored entry there.
|
`services/api/.claude/` and must add its own anchored entry there.
|
||||||
|
|
||||||
- **A plain `docker build .` of a clone stamps the version that
|
- **Excluding `.git` means `git describe` cannot run inside any build stage, and
|
||||||
`git describe --tags --always` gives**, derived from the `.git` in the build
|
it fails quietly there.** In a build stage there is no repository, so
|
||||||
context as the canonical `Dockerfile` above shows. Without its failure check,
|
`git describe` writes nothing to stdout, `-X main.Version=` comes out empty,
|
||||||
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
|
the binary reports no version at all, and the build still exits 0. Compute the
|
||||||
and the build would still exit 0. `script/docker` and `script/cibuild` pass
|
version on the host and thread it in as a build arg. `script/docker` and
|
||||||
the version they compute on the host; it takes precedence. They do this
|
`script/cibuild` do this, byte-identically across repos:
|
||||||
byte-identically across repos:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# Own line: a failing command substitution inside an argument does not
|
# Own line: a failing command substitution inside an argument does not
|
||||||
@@ -448,7 +387,7 @@ style conventions are in separate documents:
|
|||||||
fallback is applied — a live check that fires on a build from an export with
|
fallback is applied — a live check that fires on a build from an export with
|
||||||
no `.git` and on a repository with no commits yet. Do not fold it into the
|
no `.git` and on a repository with no commits yet. Do not fold it into the
|
||||||
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
||||||
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
|
Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared
|
||||||
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
||||||
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
||||||
the scripts stay byte-identical. One consequence for CI: the standard
|
the scripts stay byte-identical. One consequence for CI: the standard
|
||||||
@@ -487,18 +426,12 @@ style conventions are in separate documents:
|
|||||||
`test-support` depguard rule, where a repo names its own test-support packages
|
`test-support` depguard rule, where a repo names its own test-support packages
|
||||||
by full import path. A repo adds entries there and changes nothing else, and a
|
by full import path. A repo adds entries there and changes nothing else, and a
|
||||||
re-vendor carries its entries forward. The canonical golangci-lint version is
|
re-vendor carries its entries forward. The canonical golangci-lint version is
|
||||||
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
|
v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base
|
||||||
image
|
image
|
||||||
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
|
(`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`,
|
||||||
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
|
which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the
|
||||||
directive must not name a newer Go minor version than the one golangci-lint
|
only pin, since no repo installs golangci-lint on the host: bumping the
|
||||||
was built with, or golangci-lint refuses to lint it: this release lints
|
version means changing it and nothing else.
|
||||||
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
|
|
||||||
installs golangci-lint on the host. A repo sets the lint phase digest to the
|
|
||||||
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
|
|
||||||
the two prompted the change: the canonical copy can name linters that an older
|
|
||||||
golangci-lint rejects, and a newer golangci-lint can add linters that
|
|
||||||
`default: all` switches on until the canonical copy disables them.
|
|
||||||
|
|
||||||
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
||||||
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
||||||
@@ -522,11 +455,6 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
||||||
|
|
||||||
A Go tool a repo needs on the host is installed with `go install` pinned to
|
|
||||||
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
|
|
||||||
a `go.mod` tool dependency or through a `tools.go` file, either of which
|
|
||||||
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
|
|
||||||
|
|
||||||
- When pinning images or packages by hash, add a comment above the reference
|
- When pinning images or packages by hash, add a comment above the reference
|
||||||
with the version and date (YYYY-MM-DD).
|
with the version and date (YYYY-MM-DD).
|
||||||
|
|
||||||
@@ -639,10 +567,10 @@ style conventions are in separate documents:
|
|||||||
settings.
|
settings.
|
||||||
|
|
||||||
- Avoid putting files in the repo root unless necessary. Root should contain
|
- Avoid putting files in the repo root unless necessary. Root should contain
|
||||||
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
|
||||||
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
|
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
|
||||||
and language-specific config). Everything else goes in a subdirectory.
|
language-specific config). Everything else goes in a subdirectory. Canonical
|
||||||
Canonical subdirectory names:
|
subdirectory names:
|
||||||
- `bin/` — executable scripts and tools
|
- `bin/` — executable scripts and tools
|
||||||
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
||||||
body is a single call into `internal/` or `pkg/`, no project logic in
|
body is a single call into `internal/` or `pkg/`, no project logic in
|
||||||
@@ -673,7 +601,3 @@ style conventions are in separate documents:
|
|||||||
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
||||||
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
||||||
- Python: `pyproject.toml`
|
- Python: `pyproject.toml`
|
||||||
|
|
||||||
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
|
|
||||||
is never committed under a file or directory named after one agent tool, such
|
|
||||||
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
|
|
||||||
|
|||||||
@@ -1,15 +1,12 @@
|
|||||||
# Workflow
|
# Workflow
|
||||||
|
|
||||||
- branch from `next`
|
- branch (from `main`)
|
||||||
- do the work in Next Step
|
- do the work in Next Step
|
||||||
- move Next Step to the top of Completed Steps
|
- move Next Step to the top of Completed Steps
|
||||||
- move the top item of Future Steps into Next Step
|
- move the top item of Future Steps into Next Step
|
||||||
- commit (`TODO.md` changes in the same commit as the work)
|
- commit (`TODO.md` changes in the same commit as the work)
|
||||||
|
- merge to `main` if the branch is not protected, otherwise open a PR
|
||||||
- push
|
- push
|
||||||
- open a pull request that targets `next`
|
|
||||||
- once the pull request has passed review, the repository manager squash-merges
|
|
||||||
it into `next`
|
|
||||||
- only sneak merges `next` into `main`
|
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
@@ -17,192 +14,13 @@ pre-1.0
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
None: no implementation work is open. The cache design,
|
None: every issue still open is done on `next` and waits for it to reach `main`.
|
||||||
https://git.eeqj.de/sneak/quak/issues/36, waits on sneak's review.
|
|
||||||
|
|
||||||
Tagging and releases are decided by sneak alone, and happen only when he
|
Tagging and releases are decided by sneak alone, and happen only when he
|
||||||
declares one.
|
declares one.
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
|
|
||||||
each original already at its save path as the download check does, streamed, a
|
|
||||||
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
|
|
||||||
the content hash its metadata records is logged, removed (both files of a live
|
|
||||||
photo) and put back in the same run, downloaded or copied from the cache, and
|
|
||||||
what is put back is hashed too. One that still does not match, or whose
|
|
||||||
download fails, goes into `failures.json`. One with no recorded hash is left
|
|
||||||
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
|
|
||||||
`unchecked`. Without `--verify` nothing is hashed.
|
|
||||||
|
|
||||||
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
|
||||||
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
|
||||||
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
|
||||||
whether it succeeds or fails. A second backup of the directory, from another
|
|
||||||
process or the same one, fails at once with an error naming the directory.
|
|
||||||
`quak backup` takes the lock before it opens its library, so a refused run
|
|
||||||
sends no request; it prints the error as one line and exits 2. A lock that has
|
|
||||||
gone 10 seconds untouched, left by a run that could not remove it, is taken
|
|
||||||
over by the next run.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
|
|
||||||
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
|
|
||||||
(issue 167): for a live photo from its image, for a video nothing, and `{}`
|
|
||||||
for an original with none of them. A failed read puts the reason in
|
|
||||||
`imageMetadataError` and fails neither the file nor the run. An original is
|
|
||||||
read when the run stores it or when its JSON has neither field; otherwise the
|
|
||||||
field is taken from that JSON. The hand-built JPEGs moved to
|
|
||||||
`test/exif-jpeg.ts`, beside the HEIC.
|
|
||||||
|
|
||||||
- 2026-10-06: When the server answers HTTP 401 and that ends a command that
|
|
||||||
loads the saved session, because the server no longer accepts its token, the
|
|
||||||
command prints one line,
|
|
||||||
`quak: the saved session is no longer valid; run "quak login"`, and exits 3
|
|
||||||
(issue 164). `quak backup` meets that 401 on the refresh it starts with,
|
|
||||||
before it touches any file. For the same commands, a missing or corrupt
|
|
||||||
session file keeps its message and also exits 3, so a cron job can tell that
|
|
||||||
the user must log in again. `quak logout` is unchanged. `run` in
|
|
||||||
`src/cli-run.ts` recognises the 401, which reaches it unchanged.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` retries a failed request for longer than the other
|
|
||||||
commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
|
|
||||||
beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
|
|
||||||
a request that keeps failing waits at most 243 s before it gives up.
|
|
||||||
`bin/quak.ts` loads the backup's session with them, so its refresh, ML data
|
|
||||||
and downloads all use them. What is retried and the backoff formula are
|
|
||||||
unchanged.
|
|
||||||
|
|
||||||
- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
|
|
||||||
again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
|
|
||||||
keep out more secret files, and this repository's build artifacts follow the
|
|
||||||
copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
|
|
||||||
runs `script/bootstrap` and `script/check` before the image build,
|
|
||||||
`script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
|
|
||||||
image's last stage marks `/app` safe for git.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` writes the account and album records
|
|
||||||
`backup-metadata` writes (issue 166): `account.json` with the account's
|
|
||||||
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
|
|
||||||
`updationTime` and, when present, its three layers of magic metadata. Each
|
|
||||||
file's JSON gains its `updationTime`.
|
|
||||||
|
|
||||||
- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
|
|
||||||
embedding) into the file's JSON as `mlData`, the payload
|
|
||||||
`lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
|
|
||||||
data fetch before it writes the JSON files. When that fetch fails, each file
|
|
||||||
whose ML data is not cached gets the reason in `mlDataError` and counts as
|
|
||||||
failed, and the next run fetches it again.
|
|
||||||
|
|
||||||
- 2026-10-03: The download-albums example test no longer fails on a disk with
|
|
||||||
under 50 GiB free (issue 160). It opens its libraries with
|
|
||||||
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
|
|
||||||
the cache, evict the original it cached and add a fetch to the count. No other
|
|
||||||
test's result depends on that free space. The 50 GiB default is unchanged.
|
|
||||||
|
|
||||||
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
|
|
||||||
`dev` (issue 154). `script/build` writes the version `script/version` prints
|
|
||||||
into `dist/package.json`: the `VERSION` environment variable or build arg when
|
|
||||||
one is given, otherwise `git describe --tags --always`, otherwise the version
|
|
||||||
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
|
|
||||||
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
|
|
||||||
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
|
||||||
same version.
|
|
||||||
|
|
||||||
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
|
|
||||||
keyed by tag name, each as exifreader decodes it, not only the thirteen common
|
|
||||||
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
|
|
||||||
without the thumbnail image. The thirteen typed methods stay, each picking its
|
|
||||||
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
|
|
||||||
script's JSON files now carry every tag.
|
|
||||||
|
|
||||||
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
|
|
||||||
version cached as one ZIP (issue 151). When the cache opens, a live photo's
|
|
||||||
file that no JSON file names is now always left alone.
|
|
||||||
|
|
||||||
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
|
||||||
every album downloads each photo to its save path, writes the photo's record
|
|
||||||
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
|
||||||
`albums/<collectionID>.json` (issue 144). The build compiles it to
|
|
||||||
`dist/examples/`; the README's "Examples" section says how to run it.
|
|
||||||
|
|
||||||
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
|
|
||||||
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
|
|
||||||
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
|
||||||
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
|
||||||
and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
|
|
||||||
field, or undefined when the file lacks it. `Photo` implements a type with one
|
|
||||||
method per `PhotoExif` field, so the build's type check fails when a field has
|
|
||||||
no method. A test checks, on the JPEG and the HEIC, that each method gives the
|
|
||||||
same value as `exif()`.
|
|
||||||
|
|
||||||
- 2026-10-01: Each original's save path is
|
|
||||||
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
|
||||||
download directory, which defaults to `photos` in the working directory (issue
|
|
||||||
143). The date is the photo's `takenAt` in the machine's time zone.
|
|
||||||
`photo.savePath` is always a string, with or without a content cache, and
|
|
||||||
`isLocal` is true only when the original is at its save path.
|
|
||||||
`photo.download()` puts the original there, copied from the cache when the
|
|
||||||
cache holds it and fetched otherwise. `lib.backup()` does the same for each
|
|
||||||
file, writes each file's JSON beside its original, and links `collections/` to
|
|
||||||
the save paths.
|
|
||||||
|
|
||||||
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
|
|
||||||
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
|
|
||||||
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
|
|
||||||
`exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
|
|
||||||
field of `backup-metadata --exif` is now exifreader's tag output, and
|
|
||||||
`exifRaw` holds the whole EXIF block it could not read. The tests use a real
|
|
||||||
HEIC, `test/exif.heic`.
|
|
||||||
|
|
||||||
- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
|
|
||||||
`modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
|
|
||||||
`lib.backup()` writes the original under the library's download directory; for
|
|
||||||
a live photo not yet stored, it carries the title's extension, and the backup
|
|
||||||
may store the image under a different one. `isLocal` says whether the whole
|
|
||||||
original is there. Both look only at the disk. `content()` returns the
|
|
||||||
original's bytes and `exif()` the common EXIF fields of a JPEG; both may
|
|
||||||
download the original, and `exif()` downloads no video. `PhotoRecord` gains
|
|
||||||
`modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
|
|
||||||
|
|
||||||
- 2026-09-29: The "Workflow" list at the top of this file now says to branch
|
|
||||||
from `next` and open a pull request that targets `next`, that the repository
|
|
||||||
manager squash-merges a reviewed pull request into `next`, and that only sneak
|
|
||||||
merges `next` into `main`, as the README does (issue 137).
|
|
||||||
|
|
||||||
- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
|
|
||||||
say work branches from `next`, every pull request targets `next`, the
|
|
||||||
repository manager squash-merges reviewed pull requests into `next`, and only
|
|
||||||
sneak merges `next` into `main` (issue 135).
|
|
||||||
|
|
||||||
- 2026-09-29: Brought the README and this file in line with `next` after the
|
|
||||||
milestone merge (issue 132). The Next Step says no implementation work is open
|
|
||||||
and the cache design (issue 36) waits on sneak's review. README corrections:
|
|
||||||
the Getting Started comments say when the library refreshes; most, not all,
|
|
||||||
Makefile targets call a script; `script/lint` and `script/test` have no host
|
|
||||||
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
|
|
||||||
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
|
|
||||||
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
|
|
||||||
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
|
|
||||||
replaces SRP when the account has email MFA on, and quak cannot answer a
|
|
||||||
passkey; the auth token is sent as URL-safe base64 with padding; every
|
|
||||||
`TypeError` is retried; each download attempt writes its own temporary files,
|
|
||||||
two for a live photo, and only the attempt that completes renames them into
|
|
||||||
place; `runMetadataBackup` records a failed download in the file's JSON; a
|
|
||||||
second `client.logout()` does not throw; `quak logout` with no session exits
|
|
||||||
0, and names the cache directory only when it knows it; `login`, `whoami` and
|
|
||||||
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
|
|
||||||
IPTC; which commands take `--json`; a failed ML data request may not have been
|
|
||||||
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
|
|
||||||
still fetches ML data; a backup's JSON holds the basic metadata fields quak
|
|
||||||
keeps and the private and public magic metadata, not every decrypted field,
|
|
||||||
and the README no longer lists what the magic metadata holds; the default
|
|
||||||
cache directory is the per-user one, not an XDG path on macOS; pinned
|
|
||||||
originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
|
|
||||||
operations; a default read is only as current as the last refresh whose
|
|
||||||
requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
|
|
||||||
running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
|
|
||||||
`429` are retried.
|
|
||||||
|
|
||||||
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
||||||
that the image's and the video's temp files are fsynced before either is
|
that the image's and the video's temp files are fsynced before either is
|
||||||
renamed into place, and the directory after both renames, as the `writeAtomic`
|
renamed into place, and the directory after both renames, as the `writeAtomic`
|
||||||
|
|||||||
+2
-19
@@ -23,7 +23,6 @@ import { run as runCommand } from "../src/cli-run.js";
|
|||||||
import { loadSession } from "../src/cli-session.js";
|
import { loadSession } from "../src/cli-session.js";
|
||||||
import { Client } from "../src/client.js";
|
import { Client } from "../src/client.js";
|
||||||
import { VERSION } from "../src/index.js";
|
import { VERSION } from "../src/index.js";
|
||||||
import { UNATTENDED_RETRY_OPTIONS } from "../src/retry.js";
|
|
||||||
|
|
||||||
const paths = envPaths("quak", { suffix: "" });
|
const paths = envPaths("quak", { suffix: "" });
|
||||||
|
|
||||||
@@ -130,24 +129,8 @@ program
|
|||||||
)
|
)
|
||||||
.argument("<dir>", "Output directory")
|
.argument("<dir>", "Output directory")
|
||||||
.option("--json", "Print result as JSON instead of human-readable summary")
|
.option("--json", "Print result as JSON instead of human-readable summary")
|
||||||
.option(
|
.action((dir: string, opts: { json?: boolean }) =>
|
||||||
"--verify",
|
run(backupCommand(context(), dir, opts)),
|
||||||
"Re-hash stored originals and download again any that do not match",
|
|
||||||
)
|
|
||||||
// A backup usually runs from cron with nobody watching, so every request
|
|
||||||
// it makes retries for longer than the other commands' requests do.
|
|
||||||
.action((dir: string, opts: { json?: boolean; verify?: boolean }) =>
|
|
||||||
run(
|
|
||||||
backupCommand(
|
|
||||||
{
|
|
||||||
...context(),
|
|
||||||
loadSession: (path) =>
|
|
||||||
loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
|
|
||||||
},
|
|
||||||
dir,
|
|
||||||
opts,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
);
|
);
|
||||||
|
|
||||||
const helper = program
|
const helper = program
|
||||||
|
|||||||
@@ -1,129 +0,0 @@
|
|||||||
// Download every album's photos to a directory, with each photo's metadata
|
|
||||||
// beside it, using only quak's public API. The README's "Examples" section
|
|
||||||
// describes the files it writes.
|
|
||||||
//
|
|
||||||
// yarn build
|
|
||||||
// QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir]
|
|
||||||
|
|
||||||
import { realpathSync } from "node:fs";
|
|
||||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
|
||||||
import { join, relative } from "node:path";
|
|
||||||
import { stdin, stdout } from "node:process";
|
|
||||||
import { createInterface } from "node:readline/promises";
|
|
||||||
import { pathToFileURL } from "node:url";
|
|
||||||
|
|
||||||
import { Client, Library } from "../src/index.js";
|
|
||||||
|
|
||||||
// Write `text` to `path` unless the file already holds exactly that, so a
|
|
||||||
// second run rewrites nothing.
|
|
||||||
async function writeIfChanged(path: string, text: string): Promise<void> {
|
|
||||||
const current = await readFile(path, "utf-8").catch(() => undefined);
|
|
||||||
if (current !== text) await writeFile(path, text);
|
|
||||||
}
|
|
||||||
|
|
||||||
const pretty = (value: unknown): string =>
|
|
||||||
JSON.stringify(value, null, 2) + "\n";
|
|
||||||
|
|
||||||
// For every album in `lib`, download each photo to its save path, write the
|
|
||||||
// photo's metadata to `{savePath}.json`, and write the album's photos to
|
|
||||||
// `{dir}/albums/{collectionID}.json`. Returns how many photos it downloaded
|
|
||||||
// and how many were already at their save paths.
|
|
||||||
export async function downloadAlbums(
|
|
||||||
lib: Library,
|
|
||||||
dir: string,
|
|
||||||
): Promise<{ downloaded: number; alreadyLocal: number }> {
|
|
||||||
let downloaded = 0;
|
|
||||||
let alreadyLocal = 0;
|
|
||||||
// A photo in several albums is handled once.
|
|
||||||
const done = new Set<number>();
|
|
||||||
// fresh() waits for a refresh from the server and throws if it fails, so
|
|
||||||
// albums and photos added since the cache was last written are included.
|
|
||||||
const { albums } = await lib.fresh();
|
|
||||||
for (const album of albums.list()) {
|
|
||||||
const savePaths: string[] = [];
|
|
||||||
for (const photo of album.photos.list()) {
|
|
||||||
if (!done.has(photo.fileID)) {
|
|
||||||
done.add(photo.fileID);
|
|
||||||
if (photo.isLocal) alreadyLocal++;
|
|
||||||
else downloaded++;
|
|
||||||
await photo.download();
|
|
||||||
// The cache paths say where quak's cache keeps copies, not
|
|
||||||
// anything about the photo.
|
|
||||||
const record = { ...photo.record() };
|
|
||||||
delete record.thumbnailPath;
|
|
||||||
delete record.originalPath;
|
|
||||||
const exif = await photo.exif();
|
|
||||||
// savePath is read after download(): a live photo's names its
|
|
||||||
// image only once the image is stored.
|
|
||||||
await writeIfChanged(
|
|
||||||
`${photo.savePath}.json`,
|
|
||||||
pretty({ ...record, exif }),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
savePaths.push(relative(dir, photo.savePath));
|
|
||||||
}
|
|
||||||
await mkdir(join(dir, "albums"), { recursive: true });
|
|
||||||
await writeIfChanged(
|
|
||||||
join(dir, "albums", `${album.collectionID}.json`),
|
|
||||||
pretty({
|
|
||||||
collectionID: album.collectionID,
|
|
||||||
name: album.name,
|
|
||||||
savePaths,
|
|
||||||
}),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
return { downloaded, alreadyLocal };
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ask for a login code on the terminal. Client.login calls this only when the
|
|
||||||
// account requires a code.
|
|
||||||
async function ask(question: string): Promise<string> {
|
|
||||||
const terminal = createInterface({ input: stdin, output: stdout });
|
|
||||||
try {
|
|
||||||
return await terminal.question(question);
|
|
||||||
} finally {
|
|
||||||
terminal.close();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function main(): Promise<void> {
|
|
||||||
const email = process.env.QUAK_EMAIL;
|
|
||||||
const password = process.env.QUAK_PASSWORD;
|
|
||||||
if (!email || !password) {
|
|
||||||
console.error(
|
|
||||||
"Set QUAK_EMAIL and QUAK_PASSWORD to the account's email and password.",
|
|
||||||
);
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
const dir = process.argv[2] ?? "photos";
|
|
||||||
|
|
||||||
const client = await Client.login({
|
|
||||||
email,
|
|
||||||
password,
|
|
||||||
totp: () => ask("Two-factor code: "),
|
|
||||||
emailOTP: () => ask("Code sent to your email: "),
|
|
||||||
});
|
|
||||||
const lib = await Library.open({
|
|
||||||
client,
|
|
||||||
downloadDirectory: dir,
|
|
||||||
// As in `quak backup`: fetch only the originals this script saves,
|
|
||||||
// not every thumbnail and the recent originals into the cache too.
|
|
||||||
precacheThumbnails: false,
|
|
||||||
precacheOriginals: false,
|
|
||||||
});
|
|
||||||
try {
|
|
||||||
const { downloaded, alreadyLocal } = await downloadAlbums(lib, dir);
|
|
||||||
console.log(
|
|
||||||
`${downloaded} photos downloaded, ${alreadyLocal} already local, in ${dir}`,
|
|
||||||
);
|
|
||||||
} finally {
|
|
||||||
await lib.close();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Run main() when node runs this file, not when a test imports it. argv[1] is
|
|
||||||
// the path as given, and import.meta.url has symlinks resolved.
|
|
||||||
const script = process.argv[1];
|
|
||||||
if (script && pathToFileURL(realpathSync(script)).href === import.meta.url) {
|
|
||||||
await main();
|
|
||||||
}
|
|
||||||
+2
-4
@@ -36,7 +36,6 @@
|
|||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@eslint/js": "9.38.0",
|
"@eslint/js": "9.38.0",
|
||||||
"@types/node": "22.18.13",
|
"@types/node": "22.18.13",
|
||||||
"@types/proper-lockfile": "4.1.4",
|
|
||||||
"eslint": "9.38.0",
|
"eslint": "9.38.0",
|
||||||
"prettier": "3.8.1",
|
"prettier": "3.8.1",
|
||||||
"typescript": "5.9.3",
|
"typescript": "5.9.3",
|
||||||
@@ -47,11 +46,10 @@
|
|||||||
"@inquirer/prompts": "8.5.2",
|
"@inquirer/prompts": "8.5.2",
|
||||||
"commander": "14.0.3",
|
"commander": "14.0.3",
|
||||||
"env-paths": "4.0.0",
|
"env-paths": "4.0.0",
|
||||||
"exifreader": "4.46.0",
|
"exif-reader": "2.0.3",
|
||||||
"fast-srp-hap": "2.0.4",
|
"fast-srp-hap": "2.0.4",
|
||||||
"fflate": "0.8.3",
|
"fflate": "0.8.3",
|
||||||
"jpeg-js": "0.4.4",
|
"jpeg-js": "0.4.4",
|
||||||
"libsodium-wrappers-sumo": "0.8.4",
|
"libsodium-wrappers-sumo": "0.8.4"
|
||||||
"proper-lockfile": "4.1.2"
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+9
-23
@@ -1,7 +1,7 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/build: compile the TypeScript sources into dist/, stamp the version
|
# script/build: compile the TypeScript sources into dist/, then verify that
|
||||||
# script/version prints into it, then verify that the artifacts package.json
|
# the artifacts package.json advertises are among the files the compiler
|
||||||
# advertises are among the files the compiler actually wrote. tsc reports success by exit status alone and knows nothing
|
# actually wrote. tsc reports success by exit status alone and knows nothing
|
||||||
# about the manifest, so without this step a green build can still ship a
|
# about the manifest, so without this step a green build can still ship a
|
||||||
# package whose main, types or bin resolve to nothing. Our own extension to
|
# package whose main, types or bin resolve to nothing. Our own extension to
|
||||||
# scripts-to-rule-them-all.
|
# scripts-to-rule-them-all.
|
||||||
@@ -46,24 +46,13 @@ for (const bin of bins) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
# src/index.ts imports ../package.json for the version, which tsc copies to
|
# src/index.ts imports ../package.json for the version, which tsc copies to
|
||||||
# dist/package.json. The version script/version prints is written into that
|
# dist/package.json. Running the built CLI proves that import resolves from
|
||||||
# copy only; the repo's own package.json is left as it is.
|
# dist/ and reports the version package.json declares.
|
||||||
stamp_version() {
|
|
||||||
node -e '
|
|
||||||
const { readFileSync, writeFileSync } = require("node:fs");
|
|
||||||
|
|
||||||
const pkg = JSON.parse(readFileSync("dist/package.json", "utf-8"));
|
|
||||||
pkg.version = process.argv[1];
|
|
||||||
writeFileSync("dist/package.json", JSON.stringify(pkg, null, 4) + "\n");
|
|
||||||
' "$1"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Running the built CLI proves the import resolves from dist/ and reports
|
|
||||||
# the stamped version.
|
|
||||||
verify_version() {
|
verify_version() {
|
||||||
built="$(node dist/bin/quak.js --version)"
|
built="$(node dist/bin/quak.js --version)"
|
||||||
if [ "$built" != "$1" ]; then
|
declared="$(node -p 'require("./package.json").version')"
|
||||||
echo "build: dist/bin/quak.js reports $built, the build stamped $1" >&2
|
if [ "$built" != "$declared" ]; then
|
||||||
|
echo "build: dist/bin/quak.js reports $built, package.json declares $declared" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
echo "build: dist/bin/quak.js reports version $built"
|
echo "build: dist/bin/quak.js reports version $built"
|
||||||
@@ -71,12 +60,9 @@ verify_version() {
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# Own line, so that a failing script/version stops the build.
|
|
||||||
version="$("$ROOT/script/version")"
|
|
||||||
yarn run tsc
|
yarn run tsc
|
||||||
stamp_version "$version"
|
|
||||||
verify_entrypoints
|
verify_entrypoints
|
||||||
verify_version "$version"
|
verify_version
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+7
-5
@@ -1,8 +1,11 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/check: run all checks (test, lint, fmt-check). Our own
|
# script/check: run all checks (test, lint). Our own extension to
|
||||||
# extension to scripts-to-rule-them-all. test and lint are Docker
|
# scripts-to-rule-them-all. Both are Docker phases. Must not modify any
|
||||||
# phases; fmt-check is native, because a formatter writes the working
|
# files.
|
||||||
# tree. Must not modify any files.
|
#
|
||||||
|
# script/fmt-check is not called here, unlike the template: the lint
|
||||||
|
# phase already runs `prettier --check .`, so calling it would run
|
||||||
|
# prettier a second time over the same tree for the same verdict.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -10,7 +13,6 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
|||||||
main() {
|
main() {
|
||||||
"$SCRIPT_DIR/test"
|
"$SCRIPT_DIR/test"
|
||||||
"$SCRIPT_DIR/lint"
|
"$SCRIPT_DIR/lint"
|
||||||
"$SCRIPT_DIR/fmt-check"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+7
-7
@@ -1,7 +1,8 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
# script/cibuild: run the CI build. The image's last stage depends on the
|
||||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
# lint and test phases, so this one build runs eslint, prettier and the
|
||||||
# the formatter on the host, which a pristine checkout cannot do.
|
# suite once each and then compiles. Unlike the template it does not run
|
||||||
|
# script/check first, which would run lint and the tests a second time.
|
||||||
# --no-cache for the same reason as script/docker: the gate phases the
|
# --no-cache for the same reason as script/docker: the gate phases the
|
||||||
# final stage depends on are RUN steps, and a cached one is a check that
|
# final stage depends on are RUN steps, and a cached one is a check that
|
||||||
# did not run.
|
# did not run.
|
||||||
@@ -12,12 +13,11 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$SCRIPT_DIR/bootstrap"
|
|
||||||
"$SCRIPT_DIR/check"
|
|
||||||
# Own line: a failing command substitution inside an argument does
|
# Own line: a failing command substitution inside an argument does
|
||||||
# not trip `set -e`, so the inline form degrades silently to an
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# empty constant. The VERSION build argument takes precedence over
|
# empty constant. VERSION is computed here because .dockerignore
|
||||||
# the version a build stage derives from the .git in the context.
|
# excludes .git, so `git describe` in a build stage yields an empty
|
||||||
|
# version without failing.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
docker build --no-cache \
|
||||||
|
|||||||
+3
-2
@@ -12,8 +12,9 @@ main() {
|
|||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# Own line: a failing command substitution inside an argument does
|
# Own line: a failing command substitution inside an argument does
|
||||||
# not trip `set -e`, so the inline form degrades silently to an
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# empty constant. The VERSION build argument takes precedence over
|
# empty constant. VERSION is computed here because .dockerignore
|
||||||
# the version a build stage derives from the .git in the context.
|
# excludes .git, so `git describe` in a build stage yields an empty
|
||||||
|
# version without failing.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
docker build --no-cache \
|
||||||
|
|||||||
+1
-20
@@ -4,28 +4,9 @@ set -eu
|
|||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
# Must match the pin in script/bootstrap.
|
|
||||||
NODE_VERSION="22.17.0"
|
|
||||||
|
|
||||||
# script/bootstrap installs node and yarn under nvm and leaves neither
|
|
||||||
# on the PATH of the shell that called it, so resolve the pinned
|
|
||||||
# toolchain here the way bootstrap's own install step does. nvm is a
|
|
||||||
# bash script, hence the subshell.
|
|
||||||
run_yarn() {
|
|
||||||
if command -v yarn >/dev/null 2>&1; then
|
|
||||||
exec yarn "$@"
|
|
||||||
fi
|
|
||||||
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
|
||||||
echo "fmt: no yarn; run script/bootstrap first" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
|
||||||
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
run_yarn run prettier --write .
|
yarn run prettier --write .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+1
-20
@@ -4,28 +4,9 @@ set -eu
|
|||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
# Must match the pin in script/bootstrap.
|
|
||||||
NODE_VERSION="22.17.0"
|
|
||||||
|
|
||||||
# script/bootstrap installs node and yarn under nvm and leaves neither
|
|
||||||
# on the PATH of the shell that called it, so resolve the pinned
|
|
||||||
# toolchain here the way bootstrap's own install step does. nvm is a
|
|
||||||
# bash script, hence the subshell.
|
|
||||||
run_yarn() {
|
|
||||||
if command -v yarn >/dev/null 2>&1; then
|
|
||||||
exec yarn "$@"
|
|
||||||
fi
|
|
||||||
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
|
||||||
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
|
||||||
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
run_yarn run prettier --check .
|
yarn run prettier --check .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+5
-5
@@ -2,17 +2,17 @@
|
|||||||
# script/precommit: run by the git pre-commit hook; fails the commit if
|
# script/precommit: run by the git pre-commit hook; fails the commit if
|
||||||
# checks fail. Our own extension to scripts-to-rule-them-all.
|
# checks fail. Our own extension to scripts-to-rule-them-all.
|
||||||
#
|
#
|
||||||
# Runs lint and fmt-check but deliberately NOT the tests, so the TDD
|
# Runs lint but deliberately NOT the tests, so the TDD red-phase commit
|
||||||
# red-phase commit (failing tests, no implementation yet) can land. CI
|
# (failing tests, no implementation yet) can land. CI runs
|
||||||
# runs script/cibuild, which runs the tests, and so catches any branch
|
# script/cibuild, whose image build includes the test phase, and so
|
||||||
# that ships red.
|
# catches any branch that ships red. The lint phase includes the
|
||||||
|
# prettier check, so a badly formatted tree still fails the commit.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
"$SCRIPT_DIR/lint"
|
"$SCRIPT_DIR/lint"
|
||||||
"$SCRIPT_DIR/fmt-check"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -1,41 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# script/version: print the version script/build stamps into the built
|
|
||||||
# package. Our own extension to scripts-to-rule-them-all.
|
|
||||||
#
|
|
||||||
# Order of precedence:
|
|
||||||
#
|
|
||||||
# 1. $VERSION, if set and not empty: an explicit value, such as the
|
|
||||||
# Dockerfile's VERSION build arg.
|
|
||||||
# 2. If this checkout has .git, `git describe --tags --always`: the tag
|
|
||||||
# on a tagged commit; the tag, the commits since it and the short
|
|
||||||
# commit on a later commit (v1.2.3-4-gabc1234); the short commit when
|
|
||||||
# no tag is reachable.
|
|
||||||
# 3. Otherwise, as in a source tarball, the version package.json declares.
|
|
||||||
#
|
|
||||||
# A checkout with .git whose version still comes out empty, dev or unknown
|
|
||||||
# fails: git is missing or could not read the checkout, and the build could
|
|
||||||
# not be traced back to its commit.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|
||||||
|
|
||||||
main() {
|
|
||||||
cd "$ROOT"
|
|
||||||
version="${VERSION:-}"
|
|
||||||
if [ -e .git ]; then
|
|
||||||
if [ -z "$version" ]; then
|
|
||||||
version="$(git describe --tags --always || true)"
|
|
||||||
fi
|
|
||||||
case "$version" in
|
|
||||||
"" | dev | unknown)
|
|
||||||
echo "version: $ROOT has .git, but the version came out '$version'" >&2
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
elif [ -z "$version" ]; then
|
|
||||||
version="$(node -p 'require("./package.json").version')"
|
|
||||||
fi
|
|
||||||
echo "$version"
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
+148
-422
@@ -1,61 +1,42 @@
|
|||||||
// The backup command, rebuilt on the library API (issue #51).
|
// The backup command, rebuilt on the library API (issue #51).
|
||||||
//
|
//
|
||||||
// `lib.backup()` takes the lock in `downloadDirectory` (unless its caller
|
// `lib.backup()` waits for a completed refresh of the library (a failed one
|
||||||
// already holds it), and fails at once when another backup of it holds the
|
// fails the backup before any file is touched), then, for every file in scope,
|
||||||
// lock. It waits for a completed refresh of the library (a failed one fails the
|
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
|
||||||
// backup before any file is touched), then, for every file in scope, puts its
|
// derived views (per-file sidecars, per-collection symlink trees,
|
||||||
// original at its save path under `downloadDirectory`, as `Photo.download()`
|
// per-collection JSON) from the model. The on-disk layout is the historical
|
||||||
// does, waits for an ML data fetch, and rebuilds the derived views (per-file
|
// one:
|
||||||
// sidecars, per-collection symlink trees, per-collection JSON) from the model.
|
|
||||||
// The on-disk layout:
|
|
||||||
//
|
//
|
||||||
// <downloadDirectory>/
|
// <downloadDirectory>/
|
||||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
// originals/<fileID>.<ext> the decrypted bytes
|
||||||
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
// originals/<fileID>.json per-file metadata sidecar
|
||||||
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
|
// collections/<name>/<title> symlink into ../../originals
|
||||||
// the file's ML data and its
|
// collections/<name>.json per-collection metadata
|
||||||
// original's EXIF, XMP and
|
// failures.json durable ledger of unresolved failures
|
||||||
// dimensions
|
|
||||||
// collections/<name>/<title> symlink to the original
|
|
||||||
// collections/<name>.json per-collection metadata
|
|
||||||
// account.json the account's email and user ID
|
|
||||||
// backup.lock the lock, while a backup runs
|
|
||||||
// failures.json durable ledger of unresolved failures
|
|
||||||
//
|
//
|
||||||
// A live photo's original is its image and its video, each with its own
|
// A live photo's original is its image and its video, `<fileID>.<ext>` each
|
||||||
// extension, beside `YYYY-MM-DD.<fileID>.livephoto.json` naming them; its album
|
// with its own extension, and `originals/<fileID>.livephoto.json` naming them;
|
||||||
// folders link both.
|
// its album folders link both.
|
||||||
//
|
//
|
||||||
// Crash-safety rests on two properties. Bytes are present-means-complete: an
|
// Crash-safety rests on two properties. Bytes are present-means-complete: an
|
||||||
// original appears at its save path only via the content layer's atomic
|
// original appears under `originals/` only via the content layer's atomic
|
||||||
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
|
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
|
||||||
// interrupted run resumes by looking at the save paths. The derived views hold
|
// interrupted run resumes by listing the directory. The derived views hold no
|
||||||
// no unique state, so they are rebuilt every run; that repairs stale sidecars
|
// unique state, so they are rebuilt every run; that repairs stale sidecars and
|
||||||
// and missing or broken symlinks left by an earlier crash. A rebuild also
|
// missing or broken symlinks left by an earlier crash. A rebuild also removes
|
||||||
// removes the symlinks to originals that no longer belong to an album, and the
|
// the symlinks into originals/ that no longer belong to an album, and the
|
||||||
// directories of albums that no longer exist. The one thing a sidecar takes
|
// directories of albums that no longer exist.
|
||||||
// from the sidecar it replaces is its original's EXIF, XMP and dimensions (or
|
|
||||||
// why they could not be read), so that a run does not read every stored
|
|
||||||
// original again; a sidecar without them gets them read from the original.
|
|
||||||
//
|
|
||||||
// With `verify`, each original already at its save path is hashed as the
|
|
||||||
// download check hashes it, and one that does not match the content hash its
|
|
||||||
// metadata records is removed and fetched again in the same run. What is put
|
|
||||||
// back is hashed too, and recorded as failed if it still does not match.
|
|
||||||
//
|
//
|
||||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
// download or a failed symlink is caught, recorded in `failures.json` with a
|
||||||
// failed is caught, recorded in `failures.json` with a classification, a
|
// classification, a running attempt count, and the last-tried time, and the run
|
||||||
// running attempt count, and the last-tried time, and the run continues.
|
// continues. `result.failed` — and thus the CLI's exit code — stays non-zero
|
||||||
// `result.failed` — and thus the CLI's exit code — stays non-zero while any
|
// while any failure remains unresolved and clears once every one succeeds. Each
|
||||||
// failure remains unresolved and clears once every one succeeds. Each run
|
// run reconciles the ledger against the files it attempted, so an entry for a
|
||||||
// reconciles the ledger against the files it attempted, so an entry for a file
|
// file that has since left the library (deleted) or this run's scope is dropped
|
||||||
// that has since left the library (deleted) or this run's scope is dropped
|
// rather than counted forever, which would poison a scheduled backup's exit code.
|
||||||
// rather than counted forever, which would poison a scheduled backup's exit
|
|
||||||
// code.
|
|
||||||
|
|
||||||
import {
|
import {
|
||||||
createReadStream,
|
|
||||||
lstatSync,
|
lstatSync,
|
||||||
mkdirSync,
|
mkdirSync,
|
||||||
readdirSync,
|
readdirSync,
|
||||||
@@ -67,35 +48,24 @@ import {
|
|||||||
symlinkSync,
|
symlinkSync,
|
||||||
writeFileSync,
|
writeFileSync,
|
||||||
} from "node:fs";
|
} from "node:fs";
|
||||||
import { readFile } from "node:fs/promises";
|
import { copyFile, rename, rm } from "node:fs/promises";
|
||||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
import { basename, dirname, extname, join, relative } from "node:path";
|
||||||
import lockfile from "proper-lockfile";
|
|
||||||
|
|
||||||
import {
|
import { fsyncPath, removeLeftoverTempFiles } from "./download/index.js";
|
||||||
chunkHashFinal,
|
|
||||||
chunkHashInit,
|
|
||||||
chunkHashUpdate,
|
|
||||||
init,
|
|
||||||
} from "./crypto/index.js";
|
|
||||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
|
||||||
import { sanitizeFileName, withExtension } from "./filename.js";
|
import { sanitizeFileName, withExtension } from "./filename.js";
|
||||||
import {
|
import {
|
||||||
copyAtomic,
|
nameInOriginals,
|
||||||
placeOriginal,
|
storedOriginal,
|
||||||
savePath,
|
writeLivePhotoJSON,
|
||||||
storedAtSavePath,
|
|
||||||
} from "./library/content.js";
|
} from "./library/content.js";
|
||||||
import { representative } from "./library/records.js";
|
import type { Collection, EnteFile } from "./model/types.js";
|
||||||
import { extractImageMetadata } from "./metadata-backup.js";
|
|
||||||
import type { MLData } from "./mldata-fetch.js";
|
|
||||||
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
|
||||||
|
|
||||||
export type ProgressCallback = (message: string) => void;
|
export type ProgressCallback = (message: string) => void;
|
||||||
|
|
||||||
export interface BackupOptions {
|
export interface BackupOptions {
|
||||||
// Where the backup tree lives. `lib.backup()` defaults it to the library's
|
// Where the backup tree lives. Required: with none, `backup()` throws
|
||||||
// download directory; `runBackup` with none throws before any network
|
// before any network traffic. A library opened with a `downloadDirectory`
|
||||||
// traffic.
|
// supplies the default.
|
||||||
downloadDirectory?: string;
|
downloadDirectory?: string;
|
||||||
// Fetch and store full-resolution originals. Default true.
|
// Fetch and store full-resolution originals. Default true.
|
||||||
includeOriginals?: boolean;
|
includeOriginals?: boolean;
|
||||||
@@ -104,14 +74,7 @@ export interface BackupOptions {
|
|||||||
includeThumbnails?: boolean;
|
includeThumbnails?: boolean;
|
||||||
// Restrict the backup to albums with these names; others are left untouched.
|
// Restrict the backup to albums with these names; others are left untouched.
|
||||||
onlyAlbumNames?: string[];
|
onlyAlbumNames?: string[];
|
||||||
// Hash each original already at its save path, and fetch again any whose
|
|
||||||
// bytes do not match the content hash its metadata records. Default false.
|
|
||||||
verify?: boolean;
|
|
||||||
onProgress?: ProgressCallback;
|
onProgress?: ProgressCallback;
|
||||||
// The caller already holds the lock in `downloadDirectory`, taken with
|
|
||||||
// `lockBackupDirectory`, and releases it itself, so the backup does not
|
|
||||||
// take it. `quak backup` takes it before it opens its library.
|
|
||||||
lockHeld?: boolean;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface BackupError {
|
export interface BackupError {
|
||||||
@@ -126,15 +89,8 @@ export interface BackupResult {
|
|||||||
totalFiles: number;
|
totalFiles: number;
|
||||||
// Originals fetched (or copied from the cache) this run.
|
// Originals fetched (or copied from the cache) this run.
|
||||||
downloaded: number;
|
downloaded: number;
|
||||||
// Originals already at their save path and left untouched.
|
// Originals already present and left untouched.
|
||||||
skipped: number;
|
skipped: number;
|
||||||
// With `verify`, the originals already at their save path whose hash
|
|
||||||
// matched, those whose hash did not (each removed and fetched again), and
|
|
||||||
// those whose metadata records no hash (left as they are). All three are
|
|
||||||
// zero without `verify`.
|
|
||||||
verified: number;
|
|
||||||
mismatched: number;
|
|
||||||
unchecked: number;
|
|
||||||
// Files with an unresolved failure after this run (the ledger size); the
|
// Files with an unresolved failure after this run (the ledger size); the
|
||||||
// CLI exits non-zero while this is above zero. A file can be both
|
// CLI exits non-zero while this is above zero. A file can be both
|
||||||
// downloaded and failed if its bytes landed but its symlink did not.
|
// downloaded and failed if its bytes landed but its symlink did not.
|
||||||
@@ -146,28 +102,19 @@ export interface BackupResult {
|
|||||||
// The slice of the library that backup drives. `Library` implements it; a test
|
// The slice of the library that backup drives. `Library` implements it; a test
|
||||||
// can drive backup with a stand-in.
|
// can drive backup with a stand-in.
|
||||||
export interface BackupLibrary {
|
export interface BackupLibrary {
|
||||||
// The account the library belongs to.
|
|
||||||
whoami(): { email: string; userID: number };
|
|
||||||
refresh(): Promise<void>;
|
refresh(): Promise<void>;
|
||||||
listCollections(): Collection[];
|
listCollections(): Collection[];
|
||||||
listFiles(collectionID: number): EnteFile[];
|
listFiles(collectionID: number): EnteFile[];
|
||||||
// Get an original's bytes onto disk through the content cache/pools,
|
// Get an original's bytes onto disk through the content cache/pools,
|
||||||
// returning where they landed: `destination` when they were fetched now,
|
// returning where they landed: `destination` when they were fetched now,
|
||||||
// otherwise wherever they already were (the cache, or the library's save
|
// otherwise wherever they already were (the cache, or a prior backup). A
|
||||||
// path). A live photo lands as its image and its video, fetched now beside
|
// live photo lands as its image and its video, fetched now beside
|
||||||
// `destination`.
|
// `destination`.
|
||||||
original(
|
original(
|
||||||
fileID: number,
|
fileID: number,
|
||||||
destination: string,
|
destination: string,
|
||||||
): Promise<{ path: string; videoPath?: string }>;
|
): Promise<{ path: string; videoPath?: string }>;
|
||||||
thumbnail(fileID: number): Promise<{ path: string }>;
|
thumbnail(fileID: number): Promise<{ path: string }>;
|
||||||
// Wait for an ML data fetch to complete, joining one already running or
|
|
||||||
// starting one. Rejects with the reason when it fails; resolves at once
|
|
||||||
// when the library cannot fetch ML data.
|
|
||||||
fetchMLData(): Promise<void>;
|
|
||||||
// A file's cached ML data, as `lib.mldata.forFile()` returns it, or
|
|
||||||
// undefined when none is cached.
|
|
||||||
mlData(fileID: number): Promise<MLData | undefined>;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type FailureClass = "transient" | "permanent" | "unknown";
|
type FailureClass = "transient" | "permanent" | "unknown";
|
||||||
@@ -221,6 +168,58 @@ const classify = (err: unknown): FailureClass => {
|
|||||||
const errorMessage = (err: unknown): string =>
|
const errorMessage = (err: unknown): string =>
|
||||||
err instanceof Error ? err.message : String(err);
|
err instanceof Error ? err.message : String(err);
|
||||||
|
|
||||||
|
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
||||||
|
// `dest` appears only once it is whole ("present means complete"). As in the
|
||||||
|
// download writer, the temp file is fsynced before the rename and the directory
|
||||||
|
// after it, so a power cut cannot leave a correctly named but short original.
|
||||||
|
// The temp name carries this process's ID so a later run can tell a leftover
|
||||||
|
// from a copy still in progress (see `removeLeftoverTempFiles`).
|
||||||
|
const copyAtomic = async (src: string, dest: string): Promise<void> => {
|
||||||
|
if (src === dest) return;
|
||||||
|
const tmp = join(
|
||||||
|
dirname(dest),
|
||||||
|
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
|
||||||
|
.toString(36)
|
||||||
|
.slice(2)}.tmp`,
|
||||||
|
);
|
||||||
|
try {
|
||||||
|
await copyFile(src, tmp);
|
||||||
|
await fsyncPath(tmp);
|
||||||
|
// `rename` replaces the destination's directory entry: an existing
|
||||||
|
// symlink at `dest` is replaced, not followed, and the new file has
|
||||||
|
// the temp file's permissions (copied from `src`).
|
||||||
|
await rename(tmp, dest);
|
||||||
|
await fsyncPath(dirname(dest));
|
||||||
|
} finally {
|
||||||
|
await rm(tmp, { force: true });
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Put an original the library returned at `dest` in originals/, where a fresh
|
||||||
|
// fetch already wrote it. A live photo's image and video go beside `dest`: when
|
||||||
|
// they came from the cache they are copied, after removing whatever was at
|
||||||
|
// `dest` (an earlier version's ZIP of the two). Then the JSON file naming them
|
||||||
|
// is written, which is what makes the live photo count as stored.
|
||||||
|
const placeOriginal = async (
|
||||||
|
file: EnteFile,
|
||||||
|
dest: string,
|
||||||
|
got: { path: string; videoPath?: string },
|
||||||
|
): Promise<void> => {
|
||||||
|
if (got.videoPath === undefined) {
|
||||||
|
await copyAtomic(got.path, dest);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const originalsDir = dirname(dest);
|
||||||
|
const path = join(originalsDir, basename(got.path));
|
||||||
|
const videoPath = join(originalsDir, basename(got.videoPath));
|
||||||
|
if (got.path !== path) {
|
||||||
|
await rm(dest, { force: true });
|
||||||
|
await copyAtomic(got.path, path);
|
||||||
|
await copyAtomic(got.videoPath, videoPath);
|
||||||
|
}
|
||||||
|
await writeLivePhotoJSON(originalsDir, file.id, { path, videoPath });
|
||||||
|
};
|
||||||
|
|
||||||
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
|
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
|
||||||
// non-symlink entry. Throws on failure (a directory in the way, no permission)
|
// non-symlink entry. Throws on failure (a directory in the way, no permission)
|
||||||
// so the caller records it and moves on rather than aborting the run.
|
// so the caller records it and moves on rather than aborting the run.
|
||||||
@@ -292,55 +291,36 @@ const linksFor = (
|
|||||||
}));
|
}));
|
||||||
};
|
};
|
||||||
|
|
||||||
// Every date folder (`YYYY/YYYY-MM/YYYY-MM-DD/`) under `root`, whether or not a
|
// Remove the symlinks in the album directory `dir` that point into
|
||||||
// file in this backup is saved there. A folder that cannot be read is skipped.
|
// `originalsDir` and are not named in `keep`. Nothing else in the directory
|
||||||
const dateFolders = (root: string): string[] => {
|
// is touched: anything else there was put there by the user.
|
||||||
const subfolders = (dir: string, name: RegExp): string[] => {
|
|
||||||
try {
|
|
||||||
return readdirSync(dir, { withFileTypes: true })
|
|
||||||
.filter((e) => e.isDirectory() && name.test(e.name))
|
|
||||||
.map((e) => join(dir, e.name));
|
|
||||||
} catch {
|
|
||||||
return [];
|
|
||||||
}
|
|
||||||
};
|
|
||||||
return subfolders(root, /^\d{4}$/)
|
|
||||||
.flatMap((year) => subfolders(year, /^\d{4}-\d\d$/))
|
|
||||||
.flatMap((month) => subfolders(month, /^\d{4}-\d\d-\d\d$/));
|
|
||||||
};
|
|
||||||
|
|
||||||
// Whether the entry at `path` is a symlink a backup to `root` made: one to an
|
|
||||||
// original in a `YYYY/YYYY-MM/YYYY-MM-DD/` folder of `root`.
|
|
||||||
const linksToOriginal = (path: string, root: string): boolean => {
|
|
||||||
if (!lstatSync(path).isSymbolicLink()) return false;
|
|
||||||
const target = relative(root, resolve(dirname(path), readlinkSync(path)));
|
|
||||||
return /^\d{4}\/\d{4}-\d\d\/\d{4}-\d\d-\d\d\/[^/]+$/.test(target);
|
|
||||||
};
|
|
||||||
|
|
||||||
// Remove the symlinks in the album directory `dir` that point to an original
|
|
||||||
// in `root` and are not named in `keep`. Nothing else in the directory is
|
|
||||||
// touched: anything else there was put there by the user.
|
|
||||||
const removeStaleLinks = (
|
const removeStaleLinks = (
|
||||||
dir: string,
|
dir: string,
|
||||||
keep: Set<string>,
|
keep: Set<string>,
|
||||||
root: string,
|
originalsDir: string,
|
||||||
): void => {
|
): void => {
|
||||||
|
const target = relative(dir, originalsDir);
|
||||||
for (const name of readdirSync(dir)) {
|
for (const name of readdirSync(dir)) {
|
||||||
if (keep.has(name)) continue;
|
if (keep.has(name)) continue;
|
||||||
const path = join(dir, name);
|
const path = join(dir, name);
|
||||||
if (linksToOriginal(path, root)) rmSync(path);
|
if (
|
||||||
|
lstatSync(path).isSymbolicLink() &&
|
||||||
|
dirname(readlinkSync(path)) === target
|
||||||
|
) {
|
||||||
|
rmSync(path);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
// Remove the directories under `collectionsDir` that an earlier run wrote for
|
// Remove the directories under `collectionsDir` that an earlier run wrote for
|
||||||
// an album that is gone or renamed: a directory not named in `current` with a
|
// an album that is gone or renamed: a directory not named in `current` with a
|
||||||
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
|
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
|
||||||
// symlinks to originals in `root` are removed; if that leaves it empty, it and
|
// symlinks into originals/ are removed; if that leaves it empty, it and its
|
||||||
// its JSON are deleted, otherwise both stay for what the user put there.
|
// JSON are deleted, otherwise both stay for what the user put there.
|
||||||
const removeStaleAlbumDirs = (
|
const removeStaleAlbumDirs = (
|
||||||
collectionsDir: string,
|
collectionsDir: string,
|
||||||
current: Set<string>,
|
current: Set<string>,
|
||||||
root: string,
|
originalsDir: string,
|
||||||
): void => {
|
): void => {
|
||||||
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
|
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
|
||||||
if (!entry.isDirectory() || current.has(entry.name)) continue;
|
if (!entry.isDirectory() || current.has(entry.name)) continue;
|
||||||
@@ -354,7 +334,7 @@ const removeStaleAlbumDirs = (
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const dir = join(collectionsDir, entry.name);
|
const dir = join(collectionsDir, entry.name);
|
||||||
removeStaleLinks(dir, new Set(), root);
|
removeStaleLinks(dir, new Set(), originalsDir);
|
||||||
if (readdirSync(dir).length > 0) continue;
|
if (readdirSync(dir).length > 0) continue;
|
||||||
rmdirSync(dir);
|
rmdirSync(dir);
|
||||||
rmSync(jsonPath);
|
rmSync(jsonPath);
|
||||||
@@ -391,187 +371,71 @@ const saveLedger = (path: string, ledger: Map<number, FailureEntry>): void => {
|
|||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
// The content hash of the original stored at `stored`, computed as the download
|
const writeSidecar = (path: string, file: EnteFile): void => {
|
||||||
// check computes it: over each file's bytes, read in chunks, and for a live
|
|
||||||
// photo `<imageHash>:<videoHash>`.
|
|
||||||
const storedHash = async (stored: {
|
|
||||||
path: string;
|
|
||||||
videoPath?: string;
|
|
||||||
}): Promise<string> => {
|
|
||||||
await init();
|
|
||||||
const hashFile = async (path: string): Promise<string> => {
|
|
||||||
const state = chunkHashInit();
|
|
||||||
for await (const chunk of createReadStream(path)) {
|
|
||||||
chunkHashUpdate(state, chunk as Buffer);
|
|
||||||
}
|
|
||||||
return chunkHashFinal(state);
|
|
||||||
};
|
|
||||||
const hash = await hashFile(stored.path);
|
|
||||||
if (stored.videoPath === undefined) return hash;
|
|
||||||
return `${hash}:${await hashFile(stored.videoPath)}`;
|
|
||||||
};
|
|
||||||
|
|
||||||
// A file's EXIF, XMP and dimensions as its JSON holds them: what
|
|
||||||
// `extractImageMetadata` found in its original, or why the original could not
|
|
||||||
// be read.
|
|
||||||
interface ImageMetadata {
|
|
||||||
imageMetadata?: Record<string, unknown>;
|
|
||||||
imageMetadataError?: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
// The image metadata for the file whose original is at `originalPath` (for a
|
|
||||||
// live photo, its image) and whose JSON is at `jsonPath`. A video gets none,
|
|
||||||
// as `photo.exif()` reads none. An original stored before this run is not read
|
|
||||||
// again when its JSON already holds image metadata: that is kept. A failed
|
|
||||||
// read gives the reason, and fails neither the file nor the run. An original
|
|
||||||
// with no EXIF, XMP or JPEG dimensions gets `{}`, so it is not read again.
|
|
||||||
const imageMetadataFor = async (
|
|
||||||
file: EnteFile,
|
|
||||||
originalPath: string,
|
|
||||||
jsonPath: string,
|
|
||||||
storedThisRun: boolean,
|
|
||||||
): Promise<ImageMetadata> => {
|
|
||||||
if (file.metadata.fileType === "video") return {};
|
|
||||||
if (!storedThisRun) {
|
|
||||||
try {
|
|
||||||
const { imageMetadata, imageMetadataError } = JSON.parse(
|
|
||||||
readFileSync(jsonPath, "utf-8"),
|
|
||||||
) as ImageMetadata;
|
|
||||||
if (imageMetadata !== undefined || imageMetadataError !== undefined)
|
|
||||||
return { imageMetadata, imageMetadataError };
|
|
||||||
} catch {
|
|
||||||
// No JSON yet, or one that cannot be parsed: read the original.
|
|
||||||
}
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
const bytes = await readFile(originalPath);
|
|
||||||
return { imageMetadata: extractImageMetadata(bytes) ?? {} };
|
|
||||||
} catch (err) {
|
|
||||||
return { imageMetadataError: errorMessage(err) };
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// The file's JSON: its basic fields, its magic metadata, its ML data or the
|
|
||||||
// reason the ML data is missing, and its image metadata.
|
|
||||||
const writeSidecar = (
|
|
||||||
path: string,
|
|
||||||
file: EnteFile,
|
|
||||||
ml: { mlData?: MLData; mlDataError?: string },
|
|
||||||
image: ImageMetadata,
|
|
||||||
): void => {
|
|
||||||
const meta: Record<string, unknown> = {
|
const meta: Record<string, unknown> = {
|
||||||
id: file.id,
|
id: file.id,
|
||||||
collectionID: file.collectionID,
|
collectionID: file.collectionID,
|
||||||
ownerID: file.ownerID,
|
ownerID: file.ownerID,
|
||||||
metadata: file.metadata,
|
metadata: file.metadata,
|
||||||
updationTime: file.updationTime,
|
|
||||||
};
|
};
|
||||||
if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
|
if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
|
||||||
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
||||||
if (ml.mlData) meta.mlData = ml.mlData;
|
|
||||||
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
|
|
||||||
if (image.imageMetadata) meta.imageMetadata = image.imageMetadata;
|
|
||||||
if (image.imageMetadataError) {
|
|
||||||
meta.imageMetadataError = image.imageMetadataError;
|
|
||||||
}
|
|
||||||
writeFileSync(path, JSON.stringify(meta, null, 2));
|
writeFileSync(path, JSON.stringify(meta, null, 2));
|
||||||
};
|
};
|
||||||
|
|
||||||
// The album's JSON: its basic fields, its magic metadata, and its files.
|
export const runBackup = async (
|
||||||
const writeAlbumJSON = (
|
|
||||||
path: string,
|
|
||||||
c: Collection,
|
|
||||||
files: { id: number; metadata: FileMetadata }[],
|
|
||||||
): void => {
|
|
||||||
const album: Record<string, unknown> = {
|
|
||||||
id: c.id,
|
|
||||||
name: c.name,
|
|
||||||
type: c.type,
|
|
||||||
ownerID: c.ownerID,
|
|
||||||
isShared: c.isShared,
|
|
||||||
updationTime: c.updationTime,
|
|
||||||
};
|
|
||||||
if (c.magicMetadata) album.magicMetadata = c.magicMetadata;
|
|
||||||
if (c.pubMagicMetadata) album.pubMagicMetadata = c.pubMagicMetadata;
|
|
||||||
if (c.sharedMagicMetadata) {
|
|
||||||
album.sharedMagicMetadata = c.sharedMagicMetadata;
|
|
||||||
}
|
|
||||||
album.files = files;
|
|
||||||
writeFileSync(path, JSON.stringify(album, null, 2));
|
|
||||||
};
|
|
||||||
|
|
||||||
// The backup itself, which `runBackup` below runs while the lock is held.
|
|
||||||
const runLockedBackup = async (
|
|
||||||
lib: BackupLibrary,
|
lib: BackupLibrary,
|
||||||
opts: BackupOptions,
|
opts: BackupOptions,
|
||||||
downloadDirectory: string,
|
|
||||||
): Promise<BackupResult> => {
|
): Promise<BackupResult> => {
|
||||||
|
const downloadDirectory = opts.downloadDirectory;
|
||||||
|
if (!downloadDirectory) {
|
||||||
|
throw new Error(
|
||||||
|
"backup requires a downloadDirectory (pass one to backup() or " +
|
||||||
|
"open the library with one)",
|
||||||
|
);
|
||||||
|
}
|
||||||
const includeOriginals = opts.includeOriginals ?? true;
|
const includeOriginals = opts.includeOriginals ?? true;
|
||||||
const includeThumbnails = opts.includeThumbnails ?? false;
|
const includeThumbnails = opts.includeThumbnails ?? false;
|
||||||
const log = opts.onProgress ?? (() => {});
|
const log = opts.onProgress ?? (() => {});
|
||||||
const only = opts.onlyAlbumNames ? new Set(opts.onlyAlbumNames) : undefined;
|
const only = opts.onlyAlbumNames ? new Set(opts.onlyAlbumNames) : undefined;
|
||||||
const verify = opts.verify ?? false;
|
|
||||||
|
|
||||||
log("Refreshing library...");
|
log("Refreshing library...");
|
||||||
await lib.refresh();
|
await lib.refresh();
|
||||||
|
|
||||||
|
const originalsDir = join(downloadDirectory, "originals");
|
||||||
const collectionsDir = join(downloadDirectory, "collections");
|
const collectionsDir = join(downloadDirectory, "collections");
|
||||||
const thumbnailsDir = join(downloadDirectory, "thumbnails");
|
const thumbnailsDir = join(downloadDirectory, "thumbnails");
|
||||||
|
mkdirSync(originalsDir, { recursive: true });
|
||||||
mkdirSync(collectionsDir, { recursive: true });
|
mkdirSync(collectionsDir, { recursive: true });
|
||||||
const { email, userID } = lib.whoami();
|
|
||||||
writeFileSync(
|
|
||||||
join(downloadDirectory, "account.json"),
|
|
||||||
JSON.stringify({ email, userID }, null, 2),
|
|
||||||
);
|
|
||||||
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
|
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
|
||||||
|
removeLeftoverTempFiles(originalsDir);
|
||||||
removeLeftoverTempFiles(thumbnailsDir);
|
removeLeftoverTempFiles(thumbnailsDir);
|
||||||
for (const dir of dateFolders(downloadDirectory)) {
|
|
||||||
removeLeftoverTempFiles(dir);
|
|
||||||
}
|
|
||||||
|
|
||||||
const ledgerPath = join(downloadDirectory, "failures.json");
|
const ledgerPath = join(downloadDirectory, "failures.json");
|
||||||
const ledger = loadLedger(ledgerPath);
|
const ledger = loadLedger(ledgerPath);
|
||||||
const now = Date.now();
|
const now = Date.now();
|
||||||
|
|
||||||
// Collections in scope, and the distinct files across them (a file shared
|
// Collections in scope, and the distinct files across them (a file shared
|
||||||
// by two albums is one original). Each file is the membership
|
// by two albums is one original).
|
||||||
// `representative` picks from all of its albums, in scope or not, so it is
|
|
||||||
// saved at the path `photo.savePath` names.
|
|
||||||
const allCollections = lib.listCollections();
|
const allCollections = lib.listCollections();
|
||||||
const collections = allCollections.filter((c) =>
|
const collections = allCollections.filter((c) =>
|
||||||
only ? only.has(c.name) : true,
|
only ? only.has(c.name) : true,
|
||||||
);
|
);
|
||||||
const collectionName = new Map<number, string>();
|
const collectionName = new Map<number, string>();
|
||||||
for (const c of allCollections) collectionName.set(c.id, c.name);
|
for (const c of collections) collectionName.set(c.id, c.name);
|
||||||
|
|
||||||
const memberships = new Map<number, EnteFile[]>();
|
const distinct = new Map<number, EnteFile>();
|
||||||
const filesByCollection = new Map<number, EnteFile[]>();
|
const filesByCollection = new Map<number, EnteFile[]>();
|
||||||
for (const c of allCollections) {
|
for (const c of collections) {
|
||||||
const files = lib.listFiles(c.id);
|
const files = lib.listFiles(c.id);
|
||||||
filesByCollection.set(c.id, files);
|
filesByCollection.set(c.id, files);
|
||||||
for (const f of files) {
|
for (const f of files) if (!distinct.has(f.id)) distinct.set(f.id, f);
|
||||||
const arr = memberships.get(f.id);
|
|
||||||
if (arr) arr.push(f);
|
|
||||||
else memberships.set(f.id, [f]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const distinct = new Map<number, EnteFile>();
|
|
||||||
for (const c of collections) {
|
|
||||||
for (const f of filesByCollection.get(c.id)!) {
|
|
||||||
if (!distinct.has(f.id)) {
|
|
||||||
distinct.set(f.id, representative(memberships.get(f.id)!));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const errors: BackupError[] = [];
|
const errors: BackupError[] = [];
|
||||||
const failedThisRun = new Set<number>();
|
const failedThisRun = new Set<number>();
|
||||||
const storedThisRun = new Set<number>();
|
|
||||||
let downloaded = 0;
|
let downloaded = 0;
|
||||||
let skipped = 0;
|
let skipped = 0;
|
||||||
let verified = 0;
|
|
||||||
let mismatched = 0;
|
|
||||||
let unchecked = 0;
|
|
||||||
|
|
||||||
const recordFailure = (
|
const recordFailure = (
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
@@ -601,75 +465,26 @@ const runLockedBackup = async (
|
|||||||
failedThisRun.add(file.id);
|
failedThisRun.add(file.id);
|
||||||
};
|
};
|
||||||
|
|
||||||
// Phase 1: get the bytes. Put each pending original at its save path
|
// Phase 1: get the bytes. Fetch each pending original (and optional
|
||||||
// through the content cache/pools, as `Photo.download()` does, and fetch
|
// thumbnail) through the content cache/pools and place it under the backup
|
||||||
// the optional thumbnails; a present file is left as is. With `verify`, a
|
// tree; a present file is left as is.
|
||||||
// present original is hashed first, and one that does not match the hash
|
|
||||||
// its metadata records is removed and fetched like a missing one. One
|
|
||||||
// that cannot be read is recorded as failed and left where it is.
|
|
||||||
if (includeOriginals) {
|
if (includeOriginals) {
|
||||||
for (const [fileID, file] of distinct) {
|
for (const [fileID, file] of distinct) {
|
||||||
let stored = storedAtSavePath(downloadDirectory, file);
|
if (storedOriginal(originalsDir, file) !== undefined) {
|
||||||
let mismatch = false;
|
|
||||||
if (stored !== undefined && verify) {
|
|
||||||
try {
|
|
||||||
if (file.metadata.hash === undefined) {
|
|
||||||
unchecked++;
|
|
||||||
} else if (
|
|
||||||
(await storedHash(stored)) === file.metadata.hash
|
|
||||||
) {
|
|
||||||
verified++;
|
|
||||||
} else {
|
|
||||||
log(
|
|
||||||
`MISMATCH original ${file.metadata.title} (${fileID}): its bytes do not match its content hash`,
|
|
||||||
);
|
|
||||||
mismatched++;
|
|
||||||
mismatch = true;
|
|
||||||
rmSync(stored.path);
|
|
||||||
if (stored.videoPath !== undefined) {
|
|
||||||
rmSync(stored.videoPath);
|
|
||||||
}
|
|
||||||
stored = undefined;
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
log(
|
|
||||||
`FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
|
|
||||||
);
|
|
||||||
recordFailure(
|
|
||||||
file,
|
|
||||||
collectionName.get(file.collectionID) ?? "",
|
|
||||||
err,
|
|
||||||
);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (stored !== undefined) {
|
|
||||||
skipped++;
|
skipped++;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
const dest = join(originalsDir, nameInOriginals(file));
|
||||||
try {
|
try {
|
||||||
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
||||||
// A fetched original is written straight to its save path (a
|
// A fetched original is written straight to `dest` (a live
|
||||||
// live photo beside it); only one that was already cached
|
// photo beside it); only one that was already cached elsewhere
|
||||||
// elsewhere is copied.
|
// is copied.
|
||||||
const placed = await placeOriginal(
|
await placeOriginal(
|
||||||
downloadDirectory,
|
|
||||||
file,
|
file,
|
||||||
(dest) => lib.original(fileID, dest),
|
dest,
|
||||||
|
await lib.original(fileID, dest),
|
||||||
);
|
);
|
||||||
storedThisRun.add(fileID);
|
|
||||||
// A copy from the cache is not checked as a download is and can
|
|
||||||
// hold the same bad bytes, so what is put back after a
|
|
||||||
// mismatch is hashed too. A bad copy stays where it is and the
|
|
||||||
// file fails.
|
|
||||||
if (
|
|
||||||
mismatch &&
|
|
||||||
(await storedHash(placed)) !== file.metadata.hash
|
|
||||||
) {
|
|
||||||
throw new Error(
|
|
||||||
"the original put back does not match its content hash either",
|
|
||||||
);
|
|
||||||
}
|
|
||||||
downloaded++;
|
downloaded++;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(
|
log(
|
||||||
@@ -702,43 +517,11 @@ const runLockedBackup = async (
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
||||||
// every present original (this repairs stale ones), each with the file's
|
// every present original (this repairs stale ones).
|
||||||
// ML data once an ML data fetch has completed, and its image metadata.
|
|
||||||
// When the fetch fails, a file with no cached ML data gets the reason
|
|
||||||
// instead and is recorded as failed. The next run fetches its ML data
|
|
||||||
// again because none is cached.
|
|
||||||
if (includeOriginals) {
|
if (includeOriginals) {
|
||||||
let mlDataError: string | undefined;
|
for (const [fileID, file] of distinct) {
|
||||||
try {
|
if (storedOriginal(originalsDir, file) !== undefined) {
|
||||||
log("Fetching ML data...");
|
writeSidecar(join(originalsDir, `${fileID}.json`), file);
|
||||||
await lib.fetchMLData();
|
|
||||||
} catch (err) {
|
|
||||||
mlDataError = errorMessage(err);
|
|
||||||
log(`FAILED ML data: ${mlDataError}`);
|
|
||||||
}
|
|
||||||
for (const file of distinct.values()) {
|
|
||||||
const stored = storedAtSavePath(downloadDirectory, file);
|
|
||||||
if (stored === undefined) continue;
|
|
||||||
const path = withExtension(
|
|
||||||
savePath(downloadDirectory, file),
|
|
||||||
".json",
|
|
||||||
);
|
|
||||||
const image = await imageMetadataFor(
|
|
||||||
file,
|
|
||||||
stored.path,
|
|
||||||
path,
|
|
||||||
storedThisRun.has(file.id),
|
|
||||||
);
|
|
||||||
const mlData = await lib.mlData(file.id);
|
|
||||||
if (mlData === undefined && mlDataError !== undefined) {
|
|
||||||
writeSidecar(path, file, { mlDataError }, image);
|
|
||||||
recordFailure(
|
|
||||||
file,
|
|
||||||
collectionName.get(file.collectionID) ?? "",
|
|
||||||
new Error(`ML data: ${mlDataError}`),
|
|
||||||
);
|
|
||||||
} else {
|
|
||||||
writeSidecar(path, file, { mlData }, image);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -760,11 +543,7 @@ const runLockedBackup = async (
|
|||||||
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
||||||
);
|
);
|
||||||
try {
|
try {
|
||||||
removeStaleAlbumDirs(
|
removeStaleAlbumDirs(collectionsDir, new Set(dirNames), originalsDir);
|
||||||
collectionsDir,
|
|
||||||
new Set(dirNames),
|
|
||||||
downloadDirectory,
|
|
||||||
);
|
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(`FAILED removing old album directories: ${errorMessage(err)}`);
|
log(`FAILED removing old album directories: ${errorMessage(err)}`);
|
||||||
}
|
}
|
||||||
@@ -774,18 +553,13 @@ const runLockedBackup = async (
|
|||||||
const colDir = join(collectionsDir, colDirName);
|
const colDir = join(collectionsDir, colDirName);
|
||||||
mkdirSync(colDir, { recursive: true });
|
mkdirSync(colDir, { recursive: true });
|
||||||
|
|
||||||
// Every album links the one original, saved from the file's entry in
|
|
||||||
// `distinct`.
|
|
||||||
const files = filesByCollection.get(c.id) ?? [];
|
const files = filesByCollection.get(c.id) ?? [];
|
||||||
const links = files.flatMap((f) =>
|
const links = files.flatMap((f) =>
|
||||||
linksFor(
|
linksFor(f, storedOriginal(originalsDir, f)),
|
||||||
f,
|
|
||||||
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
|
|
||||||
),
|
|
||||||
);
|
);
|
||||||
const linkNames = uniqueNames(links, true);
|
const linkNames = uniqueNames(links, true);
|
||||||
try {
|
try {
|
||||||
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
removeStaleLinks(colDir, new Set(linkNames), originalsDir);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
|
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
|
||||||
}
|
}
|
||||||
@@ -810,10 +584,13 @@ const runLockedBackup = async (
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
writeAlbumJSON(
|
writeFileSync(
|
||||||
join(collectionsDir, `${colDirName}.json`),
|
join(collectionsDir, `${colDirName}.json`),
|
||||||
c,
|
JSON.stringify(
|
||||||
metaFiles,
|
{ id: c.id, name: c.name, type: c.type, files: metaFiles },
|
||||||
|
null,
|
||||||
|
2,
|
||||||
|
),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -832,58 +609,7 @@ const runLockedBackup = async (
|
|||||||
totalFiles: distinct.size,
|
totalFiles: distinct.size,
|
||||||
downloaded,
|
downloaded,
|
||||||
skipped,
|
skipped,
|
||||||
verified,
|
|
||||||
mismatched,
|
|
||||||
unchecked,
|
|
||||||
failed: ledger.size,
|
failed: ledger.size,
|
||||||
errors,
|
errors,
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
// Only one backup of a directory runs at a time, in this process or another.
|
|
||||||
// Creates `downloadDirectory` if it is missing, takes the lock in it and
|
|
||||||
// returns the function that releases it; while another backup holds the lock,
|
|
||||||
// fails at once with an error whose `code` is `ELOCKED`. The lock is the
|
|
||||||
// directory `backup.lock`, whose modification time proper-lockfile keeps
|
|
||||||
// current while it is held. One it has not touched for 10 seconds was left by
|
|
||||||
// a run that could not remove it, and is taken over.
|
|
||||||
export const lockBackupDirectory = async (
|
|
||||||
downloadDirectory: string,
|
|
||||||
): Promise<() => Promise<void>> => {
|
|
||||||
mkdirSync(downloadDirectory, { recursive: true });
|
|
||||||
try {
|
|
||||||
return await lockfile.lock(downloadDirectory, {
|
|
||||||
lockfilePath: join(downloadDirectory, "backup.lock"),
|
|
||||||
});
|
|
||||||
} catch (err) {
|
|
||||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
|
||||||
throw Object.assign(
|
|
||||||
new Error(`another backup of ${downloadDirectory} is running`),
|
|
||||||
{ code: "ELOCKED" },
|
|
||||||
);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// Runs the backup holding the lock, which it takes and releases itself unless
|
|
||||||
// the caller already holds it (`opts.lockHeld`).
|
|
||||||
export const runBackup = async (
|
|
||||||
lib: BackupLibrary,
|
|
||||||
opts: BackupOptions,
|
|
||||||
): Promise<BackupResult> => {
|
|
||||||
const downloadDirectory = opts.downloadDirectory;
|
|
||||||
if (!downloadDirectory) {
|
|
||||||
throw new Error(
|
|
||||||
"backup requires a downloadDirectory (pass one to backup() or " +
|
|
||||||
"open the library with one)",
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if (opts.lockHeld) {
|
|
||||||
return runLockedBackup(lib, opts, downloadDirectory);
|
|
||||||
}
|
|
||||||
const release = await lockBackupDirectory(downloadDirectory);
|
|
||||||
try {
|
|
||||||
return await runLockedBackup(lib, opts, downloadDirectory);
|
|
||||||
} finally {
|
|
||||||
await release();
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|||||||
+41
-65
@@ -20,7 +20,6 @@ import {
|
|||||||
type ClientSnapshot,
|
type ClientSnapshot,
|
||||||
type LoginOptions,
|
type LoginOptions,
|
||||||
} from "./client.js";
|
} from "./client.js";
|
||||||
import { lockBackupDirectory } from "./backup.js";
|
|
||||||
import { init } from "./crypto/index.js";
|
import { init } from "./crypto/index.js";
|
||||||
import {
|
import {
|
||||||
defaultCacheDirectory,
|
defaultCacheDirectory,
|
||||||
@@ -74,9 +73,7 @@ export const saveSession = (
|
|||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
// The saved client, or undefined after telling the user why there is none. The
|
// The saved client, or undefined after telling the user why there is none.
|
||||||
// command then exits 3, the code for "log in again", as `run` in `cli-run.ts`
|
|
||||||
// does when the server no longer accepts the saved session.
|
|
||||||
const requireSession = (ctx: CliContext): Client | undefined => {
|
const requireSession = (ctx: CliContext): Client | undefined => {
|
||||||
let client: Client | null;
|
let client: Client | null;
|
||||||
try {
|
try {
|
||||||
@@ -153,7 +150,7 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
|||||||
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const info = client.whoami();
|
const info = client.whoami();
|
||||||
ctx.stdout.write(JSON.stringify(info) + "\n");
|
ctx.stdout.write(JSON.stringify(info) + "\n");
|
||||||
return 0;
|
return 0;
|
||||||
@@ -204,7 +201,7 @@ export const collectionsCommand = async (
|
|||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const lib = await openReadLibrary(ctx, client);
|
const lib = await openReadLibrary(ctx, client);
|
||||||
try {
|
try {
|
||||||
// Force a server round-trip and list in enumeration order (issue #36
|
// Force a server round-trip and list in enumeration order (issue #36
|
||||||
@@ -246,7 +243,7 @@ export const filesCommand = async (
|
|||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const collectionID = Number(opts.collection);
|
const collectionID = Number(opts.collection);
|
||||||
if (!Number.isFinite(collectionID)) {
|
if (!Number.isFinite(collectionID)) {
|
||||||
ctx.stderr.write("Invalid collection ID\n");
|
ctx.stderr.write("Invalid collection ID\n");
|
||||||
@@ -288,7 +285,7 @@ export const getCommand = async (
|
|||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const fileID = Number(fileIDStr);
|
const fileID = Number(fileIDStr);
|
||||||
if (!Number.isFinite(fileID)) {
|
if (!Number.isFinite(fileID)) {
|
||||||
ctx.stderr.write("Invalid file ID\n");
|
ctx.stderr.write("Invalid file ID\n");
|
||||||
@@ -346,7 +343,7 @@ export const getThumbCommand = async (
|
|||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const fileID = Number(fileIDStr);
|
const fileID = Number(fileIDStr);
|
||||||
if (!Number.isFinite(fileID)) {
|
if (!Number.isFinite(fileID)) {
|
||||||
ctx.stderr.write("Invalid file ID\n");
|
ctx.stderr.write("Invalid file ID\n");
|
||||||
@@ -383,7 +380,7 @@ export const backupMetadataCommand = async (
|
|||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const lib = await openReadLibrary(ctx, client);
|
const lib = await openReadLibrary(ctx, client);
|
||||||
try {
|
try {
|
||||||
// Refresh first so the dump holds current account state, not what the
|
// Refresh first so the dump holds current account state, not what the
|
||||||
@@ -402,72 +399,51 @@ export const backupMetadataCommand = async (
|
|||||||
export const backupCommand = async (
|
export const backupCommand = async (
|
||||||
ctx: CliContext,
|
ctx: CliContext,
|
||||||
dir: string,
|
dir: string,
|
||||||
opts: { json?: boolean; verify?: boolean },
|
opts: { json?: boolean },
|
||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
|
|
||||||
ctx.stderr.write("Starting backup...\n");
|
ctx.stderr.write("Starting backup...\n");
|
||||||
// The lock is taken before the library opens and starts its refresh, so a
|
// The precache is off: the backup fetches what it needs, and must not
|
||||||
// run refused while another backup of `dir` runs sends no request.
|
// also fill the cache with every thumbnail and the recent originals.
|
||||||
let release: () => Promise<void>;
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
downloadDirectory: dir,
|
||||||
|
cacheDirectory: ctx.cacheDir,
|
||||||
|
precacheThumbnails: false,
|
||||||
|
precacheOriginals: false,
|
||||||
|
});
|
||||||
try {
|
try {
|
||||||
release = await lockBackupDirectory(dir);
|
const result = await lib.backup({
|
||||||
} catch (err) {
|
|
||||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
|
||||||
ctx.stderr.write(`quak: ${(err as Error).message}\n`);
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
// The precache is off: the backup fetches what it needs, and must not
|
|
||||||
// also fill the cache with every thumbnail and the recent originals.
|
|
||||||
const lib = await Library.open({
|
|
||||||
client,
|
|
||||||
downloadDirectory: dir,
|
downloadDirectory: dir,
|
||||||
cacheDirectory: ctx.cacheDir,
|
onProgress: (msg) => {
|
||||||
precacheThumbnails: false,
|
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||||
precacheOriginals: false,
|
},
|
||||||
});
|
});
|
||||||
try {
|
|
||||||
const result = await lib.backup({
|
|
||||||
downloadDirectory: dir,
|
|
||||||
lockHeld: true,
|
|
||||||
verify: opts.verify,
|
|
||||||
onProgress: (msg) => {
|
|
||||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
if (opts.json) {
|
if (opts.json) {
|
||||||
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||||
} else {
|
} else {
|
||||||
ctx.stderr.write("\n--- Backup complete ---\n");
|
ctx.stderr.write("\n--- Backup complete ---\n");
|
||||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||||
if (opts.verify) {
|
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||||
ctx.stderr.write(` Verified: ${result.verified}\n`);
|
if (result.errors.length > 0) {
|
||||||
ctx.stderr.write(` Mismatched: ${result.mismatched}\n`);
|
ctx.stderr.write("\nFailed files:\n");
|
||||||
ctx.stderr.write(` Unchecked: ${result.unchecked}\n`);
|
for (const e of result.errors) {
|
||||||
}
|
ctx.stderr.write(
|
||||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||||
if (result.errors.length > 0) {
|
);
|
||||||
ctx.stderr.write("\nFailed files:\n");
|
|
||||||
for (const e of result.errors) {
|
|
||||||
ctx.stderr.write(
|
|
||||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return result.failed > 0 ? 1 : 0;
|
|
||||||
} finally {
|
|
||||||
await lib.close();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return result.failed > 0 ? 1 : 0;
|
||||||
} finally {
|
} finally {
|
||||||
await release();
|
await lib.close();
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -477,7 +453,7 @@ export const listMissingThumbnailsCommand = async (
|
|||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const lib = await openReadLibrary(ctx, client);
|
const lib = await openReadLibrary(ctx, client);
|
||||||
try {
|
try {
|
||||||
// Refresh first so files added since the cache was written are
|
// Refresh first so files added since the cache was written are
|
||||||
@@ -515,7 +491,7 @@ export const fixMissingThumbnailsCommand = async (
|
|||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 1;
|
||||||
const lib = await openReadLibrary(ctx, client);
|
const lib = await openReadLibrary(ctx, client);
|
||||||
try {
|
try {
|
||||||
// Refresh first so files added since the cache was written are found;
|
// Refresh first so files added since the cache was written are found;
|
||||||
|
|||||||
+5
-15
@@ -1,15 +1,12 @@
|
|||||||
// Runs one CLI command for `bin/quak.ts` and exits with its code.
|
// Runs one CLI command for `bin/quak.ts` and exits with its code.
|
||||||
|
|
||||||
import type { Writable } from "node:stream";
|
import type { Writable } from "node:stream";
|
||||||
import { ApiError } from "./api/client.js";
|
|
||||||
|
|
||||||
// Run a command and exit with its code once stdout/stderr have drained.
|
// Run a command and exit with its code once stdout/stderr have drained.
|
||||||
// Exiting before the drain can truncate piped output, and the library can keep
|
// Exiting before the drain can truncate piped output, and the library can keep
|
||||||
// the event loop alive after a command returns, so a plain return could hang.
|
// the event loop alive after a command returns, so a plain return could hang.
|
||||||
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
||||||
// the stack trace, and exits 1. A 401 from the server means it no longer
|
// the stack trace, and exits 1.
|
||||||
// accepts the saved session: that prints one line saying to log in again and
|
|
||||||
// exits 3, as a missing or corrupt session file does.
|
|
||||||
export const run = async (
|
export const run = async (
|
||||||
command: Promise<number>,
|
command: Promise<number>,
|
||||||
stdout: Writable,
|
stdout: Writable,
|
||||||
@@ -20,17 +17,10 @@ export const run = async (
|
|||||||
try {
|
try {
|
||||||
code = await command;
|
code = await command;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
if (err instanceof ApiError && err.status === 401) {
|
stderr.write(
|
||||||
stderr.write(
|
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
|
||||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
);
|
||||||
);
|
code = 1;
|
||||||
code = 3;
|
|
||||||
} else {
|
|
||||||
stderr.write(
|
|
||||||
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
|
|
||||||
);
|
|
||||||
code = 1;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
|
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
|
||||||
if (pending.length === 0) {
|
if (pending.length === 0) {
|
||||||
|
|||||||
@@ -361,8 +361,8 @@ const openPart = async (
|
|||||||
// written unpacked: each part is named `destination` with the extension
|
// written unpacked: each part is named `destination` with the extension
|
||||||
// replaced by its own entry's, and the two must differ ignoring case. When the
|
// replaced by its own entry's, and the two must differ ignoring case. When the
|
||||||
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
||||||
// part's own bytes. Only then are the image, then the video, renamed into
|
// part's own bytes. Only then is whatever was at `destination` removed and the
|
||||||
// place; on any failure neither is stored.
|
// image, then the video, renamed into place; on any failure neither is stored.
|
||||||
//
|
//
|
||||||
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
||||||
// written as it decompresses and never held, and the ZIP is refused once the
|
// written as it decompresses and never held, and the ZIP is refused once the
|
||||||
@@ -486,6 +486,7 @@ const decryptLivePhoto = async (
|
|||||||
await part.handle.sync();
|
await part.handle.sync();
|
||||||
await part.handle.close();
|
await part.handle.close();
|
||||||
}
|
}
|
||||||
|
await rm(destination, { force: true });
|
||||||
await rename(image.tmpPath, path);
|
await rename(image.tmpPath, path);
|
||||||
try {
|
try {
|
||||||
await rename(video.tmpPath, videoPath);
|
await rename(video.tmpPath, videoPath);
|
||||||
@@ -569,7 +570,8 @@ const fetchAndDecrypt = async (
|
|||||||
}, api.getRetryOptions());
|
}, api.getRetryOptions());
|
||||||
|
|
||||||
// Write `file`'s original to `outPath`. A live photo is written as its image
|
// Write `file`'s original to `outPath`. A live photo is written as its image
|
||||||
// and its video beside `outPath` instead (see `decryptLivePhoto`).
|
// and its video beside `outPath` instead, and whatever was at `outPath` is
|
||||||
|
// removed (see `decryptLivePhoto`).
|
||||||
export const downloadFile = async (
|
export const downloadFile = async (
|
||||||
api: ApiClient,
|
api: ApiClient,
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
|
|||||||
-164
@@ -1,164 +0,0 @@
|
|||||||
// EXIF in an original's bytes, read with exifreader, which reads it from JPEG,
|
|
||||||
// HEIC/HEIF, AVIF, PNG, WebP and the other image formats it supports.
|
|
||||||
// `backup-metadata --exif` records every EXIF tag it finds except the
|
|
||||||
// thumbnail's. `Photo.exif()` returns every tag, the thumbnail's included, and
|
|
||||||
// `Photo`'s typed methods return the common fields picked from them here.
|
|
||||||
|
|
||||||
import ExifReader, { type ExpandedTags } from "exifreader";
|
|
||||||
|
|
||||||
// The EXIF tags in `bytes` (`exif`), the embedded thumbnail's tags
|
|
||||||
// (`Thumbnail`), and where the EXIF block lies in `bytes` (`metadataRange`).
|
|
||||||
// Undefined when exifreader cannot read the file at all, such as a video. An
|
|
||||||
// EXIF block it finds but reads no tag from comes back as an empty `exif`. A
|
|
||||||
// tag exifreader has no name for is keyed `undefined-<tag number>`. Each tag's
|
|
||||||
// `computed` holds its value as a string or number, or as an array of them for
|
|
||||||
// a tag with several values, such as `GPSLatitude`'s `[40, 26, 46]`. A
|
|
||||||
// fraction with a zero denominator computes to null.
|
|
||||||
export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
|
|
||||||
try {
|
|
||||||
return ExifReader.loadView(
|
|
||||||
new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength),
|
|
||||||
{
|
|
||||||
expanded: true,
|
|
||||||
computed: true,
|
|
||||||
includeOffsets: true,
|
|
||||||
includeUnknown: true,
|
|
||||||
includeTags: { exif: true, thumbnail: true },
|
|
||||||
},
|
|
||||||
);
|
|
||||||
} catch {
|
|
||||||
return undefined;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// Every EXIF tag of an original, keyed by name, each as exifreader decodes it
|
|
||||||
// (see `readExifTags`). The embedded thumbnail's own tags are under
|
|
||||||
// `Thumbnail`, so its `Orientation` or `ImageWidth` cannot hide the main
|
|
||||||
// image's.
|
|
||||||
export type ExifTags = Omit<NonNullable<ExpandedTags["exif"]>, "Thumbnail"> & {
|
|
||||||
Thumbnail?: Omit<
|
|
||||||
NonNullable<ExpandedTags["Thumbnail"]>,
|
|
||||||
"type" | "image" | "base64"
|
|
||||||
>;
|
|
||||||
};
|
|
||||||
|
|
||||||
// Every EXIF tag in `bytes`: `{}` when the file has no EXIF, exifreader cannot
|
|
||||||
// read its EXIF, or it is not an image exifreader reads.
|
|
||||||
export const readAllExifTags = (bytes: Uint8Array): ExifTags => {
|
|
||||||
const tags = readExifTags(bytes);
|
|
||||||
if (!tags?.Thumbnail) return tags?.exif ?? {};
|
|
||||||
// exifreader puts the thumbnail's JPEG image beside its tags, as `type`,
|
|
||||||
// `image` and `base64`. The image is not a tag, so it is left out.
|
|
||||||
const {
|
|
||||||
type: _type,
|
|
||||||
image: _image,
|
|
||||||
base64: _base64,
|
|
||||||
...thumbnail
|
|
||||||
} = tags.Thumbnail;
|
|
||||||
return { ...tags.exif, Thumbnail: thumbnail };
|
|
||||||
};
|
|
||||||
|
|
||||||
// The common EXIF fields of an original, one for each of `Photo`'s typed
|
|
||||||
// methods. Each is absent when the file lacks it.
|
|
||||||
export interface PhotoExif {
|
|
||||||
make?: string;
|
|
||||||
model?: string;
|
|
||||||
lensModel?: string;
|
|
||||||
// When the photo was taken, by the camera's clock. EXIF writes this as text
|
|
||||||
// with no time zone, and it is read as if it were UTC: the Date's UTC
|
|
||||||
// fields are the clock reading, which is the moment it was taken only when
|
|
||||||
// the clock was set to UTC.
|
|
||||||
dateTimeOriginal?: Date;
|
|
||||||
// The camera clock's offset from UTC, such as "+02:00".
|
|
||||||
offsetTimeOriginal?: string;
|
|
||||||
// Seconds.
|
|
||||||
exposureTime?: number;
|
|
||||||
fNumber?: number;
|
|
||||||
iso?: number;
|
|
||||||
// Millimetres.
|
|
||||||
focalLength?: number;
|
|
||||||
// The EXIF orientation code, 1 to 8.
|
|
||||||
orientation?: number;
|
|
||||||
// Decimal degrees, negative south of the equator and west of Greenwich.
|
|
||||||
gpsLatitude?: number;
|
|
||||||
gpsLongitude?: number;
|
|
||||||
// Metres, negative below sea level.
|
|
||||||
gpsAltitude?: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
// exifreader gives "<faulty value>" for a tag whose value lies outside the
|
|
||||||
// file; that tag is left out like one the file lacks.
|
|
||||||
const asString = (v: unknown): string | undefined =>
|
|
||||||
typeof v === "string" && v.length > 0 && v !== "<faulty value>"
|
|
||||||
? v
|
|
||||||
: undefined;
|
|
||||||
|
|
||||||
const asNumber = (v: unknown): number | undefined =>
|
|
||||||
typeof v === "number" && Number.isFinite(v) ? v : undefined;
|
|
||||||
|
|
||||||
// EXIF writes a date and time as "2021:07:15 14:30:00". This is that reading
|
|
||||||
// in a Date's UTC fields.
|
|
||||||
const asDate = (v: unknown): Date | undefined => {
|
|
||||||
const m =
|
|
||||||
typeof v === "string"
|
|
||||||
? /^(\d{4}):(\d{2}):(\d{2}) (\d{2}:\d{2}:\d{2})$/.exec(v)
|
|
||||||
: null;
|
|
||||||
if (!m) return undefined;
|
|
||||||
const date = new Date(`${m[1]}-${m[2]}-${m[3]}T${m[4]}Z`);
|
|
||||||
return Number.isNaN(date.getTime()) ? undefined : date;
|
|
||||||
};
|
|
||||||
|
|
||||||
// GPSLatitude and GPSLongitude hold degrees, minutes and seconds, computed as
|
|
||||||
// three numbers. This is them in decimal degrees, negative when `ref`, the
|
|
||||||
// GPSLatitudeRef or GPSLongitudeRef tag, is `negativeRef` ("S" or "W").
|
|
||||||
// Without that tag the hemisphere is unknown, so it is undefined.
|
|
||||||
const asDegrees = (
|
|
||||||
dms: unknown,
|
|
||||||
ref: unknown,
|
|
||||||
negativeRef: string,
|
|
||||||
): number | undefined => {
|
|
||||||
if (!Array.isArray(dms) || ref === undefined) return undefined;
|
|
||||||
const [d, m, s] = dms.map(asNumber);
|
|
||||||
if (d === undefined || m === undefined || s === undefined) return undefined;
|
|
||||||
const degrees = d + m / 60 + s / 3600;
|
|
||||||
return ref === negativeRef ? -degrees : degrees;
|
|
||||||
};
|
|
||||||
|
|
||||||
// The common fields picked from an original's EXIF tags, `readAllExifTags`'s
|
|
||||||
// result: `{}` when there are none.
|
|
||||||
export const readPhotoExif = (tags: ExifTags): PhotoExif => {
|
|
||||||
const altitude = asNumber(tags.GPSAltitude?.computed);
|
|
||||||
const fields: PhotoExif = {
|
|
||||||
make: asString(tags.Make?.computed),
|
|
||||||
model: asString(tags.Model?.computed),
|
|
||||||
lensModel: asString(tags.LensModel?.computed),
|
|
||||||
dateTimeOriginal: asDate(tags.DateTimeOriginal?.computed),
|
|
||||||
offsetTimeOriginal: asString(tags.OffsetTimeOriginal?.computed),
|
|
||||||
exposureTime: asNumber(tags.ExposureTime?.computed),
|
|
||||||
fNumber: asNumber(tags.FNumber?.computed),
|
|
||||||
// Only when the tag holds a single number, as most cameras write it.
|
|
||||||
iso: asNumber(tags.ISOSpeedRatings?.computed),
|
|
||||||
focalLength: asNumber(tags.FocalLength?.computed),
|
|
||||||
orientation: asNumber(tags.Orientation?.computed),
|
|
||||||
gpsLatitude: asDegrees(
|
|
||||||
tags.GPSLatitude?.computed,
|
|
||||||
tags.GPSLatitudeRef?.computed,
|
|
||||||
"S",
|
|
||||||
),
|
|
||||||
gpsLongitude: asDegrees(
|
|
||||||
tags.GPSLongitude?.computed,
|
|
||||||
tags.GPSLongitudeRef?.computed,
|
|
||||||
"W",
|
|
||||||
),
|
|
||||||
// A GPSAltitudeRef of 1 means the altitude is below sea level.
|
|
||||||
gpsAltitude:
|
|
||||||
altitude !== undefined && tags.GPSAltitudeRef?.value === 1
|
|
||||||
? -altitude
|
|
||||||
: altitude,
|
|
||||||
};
|
|
||||||
// Leave out what the file lacks, so a missing field is absent rather than
|
|
||||||
// present and undefined.
|
|
||||||
return Object.fromEntries(
|
|
||||||
Object.entries(fields).filter(([, v]) => v !== undefined),
|
|
||||||
) as PhotoExif;
|
|
||||||
};
|
|
||||||
+2
-8
@@ -1,7 +1,5 @@
|
|||||||
// A build reports the version script/build stamps into dist/package.json;
|
// package.json is the one place the version is written. tsc copies it to
|
||||||
// package.json's own version is reported only when running from source. tsc
|
// dist/package.json, so this path resolves from source and from dist/src/.
|
||||||
// copies package.json to dist/package.json, so this path resolves from source
|
|
||||||
// and from dist/src/.
|
|
||||||
import pkg from "../package.json" with { type: "json" };
|
import pkg from "../package.json" with { type: "json" };
|
||||||
|
|
||||||
export const VERSION: string = pkg.version;
|
export const VERSION: string = pkg.version;
|
||||||
@@ -27,7 +25,6 @@ export {
|
|||||||
isRetryable,
|
isRetryable,
|
||||||
isSafeToReplay,
|
isSafeToReplay,
|
||||||
resolveRetryOptions,
|
resolveRetryOptions,
|
||||||
UNATTENDED_RETRY_OPTIONS,
|
|
||||||
withRetry,
|
withRetry,
|
||||||
type ResolvedRetryOptions,
|
type ResolvedRetryOptions,
|
||||||
type RetryOptions,
|
type RetryOptions,
|
||||||
@@ -56,7 +53,6 @@ export {
|
|||||||
type PhotoFilter,
|
type PhotoFilter,
|
||||||
type TimelineGroup,
|
type TimelineGroup,
|
||||||
type GroupBy,
|
type GroupBy,
|
||||||
type SavePathLookup,
|
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
type ContentResult,
|
type ContentResult,
|
||||||
type ContentEvent,
|
type ContentEvent,
|
||||||
@@ -67,7 +63,6 @@ export {
|
|||||||
type EnsureOptions,
|
type EnsureOptions,
|
||||||
type EnsureResult,
|
type EnsureResult,
|
||||||
type EnsureEvent,
|
type EnsureEvent,
|
||||||
lockBackupDirectory,
|
|
||||||
runBackup,
|
runBackup,
|
||||||
type BackupOptions,
|
type BackupOptions,
|
||||||
type BackupResult,
|
type BackupResult,
|
||||||
@@ -89,7 +84,6 @@ export type {
|
|||||||
LibrarySnapshot,
|
LibrarySnapshot,
|
||||||
LibraryChange,
|
LibraryChange,
|
||||||
} from "./library/records.js";
|
} from "./library/records.js";
|
||||||
export type { ExifTags, PhotoExif } from "./exif.js";
|
|
||||||
export { decryptCollection, decryptFile } from "./model/index.js";
|
export { decryptCollection, decryptFile } from "./model/index.js";
|
||||||
export { downloadFile, downloadThumbnail } from "./download/index.js";
|
export { downloadFile, downloadThumbnail } from "./download/index.js";
|
||||||
export type {
|
export type {
|
||||||
|
|||||||
+79
-159
@@ -23,19 +23,19 @@
|
|||||||
// original with no recorded hash is stored unchecked, as the upstream client
|
// original with no recorded hash is stored unchecked, as the upstream client
|
||||||
// does; thumbnails have none. On top of that this module refuses to record a
|
// does; thumbnails have none. On top of that this module refuses to record a
|
||||||
// stored file that came out empty.
|
// stored file that came out empty.
|
||||||
//
|
|
||||||
// It also names each original's save path under the download directory
|
|
||||||
// (`savePath`), where `Photo.download()` and `lib.backup()` put it. A copy in
|
|
||||||
// the cache does not count as saved there, but is copied there rather than
|
|
||||||
// fetched again.
|
|
||||||
|
|
||||||
import { existsSync, readFileSync, statSync } from "node:fs";
|
import {
|
||||||
|
closeSync,
|
||||||
|
existsSync,
|
||||||
|
openSync,
|
||||||
|
readFileSync,
|
||||||
|
readSync,
|
||||||
|
statSync,
|
||||||
|
} from "node:fs";
|
||||||
import {
|
import {
|
||||||
chmod,
|
chmod,
|
||||||
copyFile,
|
|
||||||
mkdir,
|
mkdir,
|
||||||
readdir,
|
readdir,
|
||||||
rename,
|
|
||||||
rm,
|
rm,
|
||||||
stat,
|
stat,
|
||||||
statfs,
|
statfs,
|
||||||
@@ -47,15 +47,13 @@ import type { ApiClient } from "../api/client.js";
|
|||||||
import {
|
import {
|
||||||
downloadFile,
|
downloadFile,
|
||||||
downloadThumbnail,
|
downloadThumbnail,
|
||||||
fsyncPath,
|
|
||||||
type ProgressCallback,
|
type ProgressCallback,
|
||||||
removeLeftoverTempFiles,
|
removeLeftoverTempFiles,
|
||||||
writeAtomic,
|
writeAtomic,
|
||||||
} from "../download/index.js";
|
} from "../download/index.js";
|
||||||
import { safeExtension, withExtension } from "../filename.js";
|
import { safeExtension } from "../filename.js";
|
||||||
import type { EnteFile } from "../model/types.js";
|
import type { EnteFile } from "../model/types.js";
|
||||||
import type { Priority, RequestPools } from "./pools.js";
|
import type { Priority, RequestPools } from "./pools.js";
|
||||||
import { takenAtOf } from "./records.js";
|
|
||||||
|
|
||||||
const DIR_MODE = 0o700;
|
const DIR_MODE = 0o700;
|
||||||
const FILE_MODE = 0o600;
|
const FILE_MODE = 0o600;
|
||||||
@@ -110,9 +108,6 @@ export interface ContentOptions {
|
|||||||
export interface PhotoContent {
|
export interface PhotoContent {
|
||||||
original(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
original(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
||||||
thumbnail(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
thumbnail(fileID: number, opts?: ContentOptions): Promise<ContentResult>;
|
||||||
// Put the original at the save path of `file`, the copy the `Photo` holds,
|
|
||||||
// and return it there.
|
|
||||||
download(file: EnteFile): Promise<ContentResult>;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface EnsureResult {
|
export interface EnsureResult {
|
||||||
@@ -199,10 +194,10 @@ export interface ContentCacheOptions {
|
|||||||
pools: RequestPools;
|
pools: RequestPools;
|
||||||
source: ContentSource;
|
source: ContentSource;
|
||||||
cacheDirectory: string;
|
cacheDirectory: string;
|
||||||
// The root of the save paths. An original already stored at its save path
|
// The backup destination (issue-level `downloadDirectory`). An original
|
||||||
// counts as present, so the cache serves it rather than fetching a second
|
// already stored there by a backup counts as present, so the cache serves
|
||||||
// copy.
|
// it rather than fetching a second copy.
|
||||||
downloadDirectory: string;
|
downloadDirectory?: string;
|
||||||
// Resolve any membership of a file; every membership shares the underlying
|
// Resolve any membership of a file; every membership shares the underlying
|
||||||
// content key, so any one decrypts the same bytes.
|
// content key, so any one decrypts the same bytes.
|
||||||
getFile: (fileID: number) => EnteFile | undefined;
|
getFile: (fileID: number) => EnteFile | undefined;
|
||||||
@@ -229,27 +224,11 @@ class AbortDrop extends Error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The name of a file's original in the cache's originals/: `<fileID><ext>`, the
|
// The name of a file's original in originals/: `<fileID><ext>`, the extension
|
||||||
// extension taken from the title (or `.bin`).
|
// taken from the title (or `.bin`). A backup names its originals the same way.
|
||||||
export const nameInOriginals = (file: EnteFile): string =>
|
export const nameInOriginals = (file: EnteFile): string =>
|
||||||
`${file.id}${safeExtension(file.metadata.title)}`;
|
`${file.id}${safeExtension(file.metadata.title)}`;
|
||||||
|
|
||||||
const pad = (n: number): string => String(n).padStart(2, "0");
|
|
||||||
|
|
||||||
// Where the original of `file` is saved under `root`:
|
|
||||||
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`, dated by the photo's
|
|
||||||
// `takenAt` in this machine's time zone, with the extension taken from the
|
|
||||||
// title (or `.bin`). A live photo is stored as its image and its video beside
|
|
||||||
// this path, each with the extension found inside the live photo.
|
|
||||||
export const savePath = (root: string, file: EnteFile): string => {
|
|
||||||
const taken = new Date(takenAtOf(file));
|
|
||||||
const year = String(taken.getFullYear());
|
|
||||||
const month = `${year}-${pad(taken.getMonth() + 1)}`;
|
|
||||||
const day = `${month}-${pad(taken.getDate())}`;
|
|
||||||
const ext = safeExtension(file.metadata.title);
|
|
||||||
return join(root, year, month, day, `${day}.${file.id}${ext}`);
|
|
||||||
};
|
|
||||||
|
|
||||||
// The fileID a cache filename encodes, or undefined when the name is not one
|
// The fileID a cache filename encodes, or undefined when the name is not one
|
||||||
// the cache writes (`<digits><ext>`).
|
// the cache writes (`<digits><ext>`).
|
||||||
const fileIDFromName = (name: string): number | undefined => {
|
const fileIDFromName = (name: string): number | undefined => {
|
||||||
@@ -274,31 +253,49 @@ const fileSize = (path: string): number | undefined => {
|
|||||||
const hasContent = (path: string | undefined): boolean =>
|
const hasContent = (path: string | undefined): boolean =>
|
||||||
path !== undefined && (fileSize(path) ?? 0) > 0;
|
path !== undefined && (fileSize(path) ?? 0) > 0;
|
||||||
|
|
||||||
|
// Whether the file at `path` begins as a ZIP does, with `PK\x03\x04`. False
|
||||||
|
// when it cannot be read.
|
||||||
|
const isZip = (path: string): boolean => {
|
||||||
|
try {
|
||||||
|
const fd = openSync(path, "r");
|
||||||
|
try {
|
||||||
|
const head = Buffer.alloc(4);
|
||||||
|
return (
|
||||||
|
readSync(fd, head, 0, 4, 0) === 4 &&
|
||||||
|
head.toString("latin1") === "PK\x03\x04"
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
closeSync(fd);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
// A live photo's image and video are named with the extensions from inside its
|
// A live photo's image and video are named with the extensions from inside its
|
||||||
// ZIP, so their names alone do not say which is which. Wherever the cache or a
|
// ZIP, so their names alone do not say which is which. Wherever the cache or a
|
||||||
// save path stores one, a JSON file of this name beside them names both.
|
// backup stores one, a JSON file of this name beside them names both.
|
||||||
// `name` is the original's name without its extension: `<fileID>` in the
|
const livePhotoJSONName = (fileID: number): string =>
|
||||||
// cache, `YYYY-MM-DD.<fileID>` at the save path.
|
`${fileID}.livephoto.json`;
|
||||||
const livePhotoJSONName = (name: string): string => `${name}.livephoto.json`;
|
|
||||||
|
|
||||||
// The image and video that the live photo's JSON file in `dir` names, or
|
// The image and video that the live photo's JSON file in `dir` names, or
|
||||||
// undefined when there is none. Only `name` with an extension is taken, so the
|
// undefined when there is none. Only names of the form the cache writes are
|
||||||
// file cannot point outside `dir`.
|
// taken, so the file cannot point outside `dir`.
|
||||||
const readLivePhotoJSON = (
|
const readLivePhotoJSON = (
|
||||||
dir: string,
|
dir: string,
|
||||||
name: string,
|
fileID: number,
|
||||||
): { path: string; videoPath: string } | undefined => {
|
): { path: string; videoPath: string } | undefined => {
|
||||||
const valid = (part: unknown): part is string =>
|
const valid = (name: unknown): name is string =>
|
||||||
typeof part === "string" && part === `${name}${safeExtension(part)}`;
|
typeof name === "string" && name === `${fileID}${safeExtension(name)}`;
|
||||||
try {
|
try {
|
||||||
const { image, video } = JSON.parse(
|
const { image, video } = JSON.parse(
|
||||||
readFileSync(join(dir, livePhotoJSONName(name)), "utf-8"),
|
readFileSync(join(dir, livePhotoJSONName(fileID)), "utf-8"),
|
||||||
);
|
);
|
||||||
if (valid(image) && valid(video)) {
|
if (valid(image) && valid(video)) {
|
||||||
return { path: join(dir, image), videoPath: join(dir, video) };
|
return { path: join(dir, image), videoPath: join(dir, video) };
|
||||||
}
|
}
|
||||||
} catch {
|
} catch {
|
||||||
// No such file, or not one quak wrote.
|
// No such file, or not one the cache wrote.
|
||||||
}
|
}
|
||||||
return undefined;
|
return undefined;
|
||||||
};
|
};
|
||||||
@@ -306,11 +303,11 @@ const readLivePhotoJSON = (
|
|||||||
// Write the JSON file naming a live photo's image and video, both in `dir`.
|
// Write the JSON file naming a live photo's image and video, both in `dir`.
|
||||||
export const writeLivePhotoJSON = (
|
export const writeLivePhotoJSON = (
|
||||||
dir: string,
|
dir: string,
|
||||||
name: string,
|
fileID: number,
|
||||||
stored: { path: string; videoPath: string },
|
stored: { path: string; videoPath: string },
|
||||||
): Promise<void> =>
|
): Promise<void> =>
|
||||||
writeAtomic(
|
writeAtomic(
|
||||||
join(dir, livePhotoJSONName(name)),
|
join(dir, livePhotoJSONName(fileID)),
|
||||||
new TextEncoder().encode(
|
new TextEncoder().encode(
|
||||||
JSON.stringify({
|
JSON.stringify({
|
||||||
image: basename(stored.path),
|
image: basename(stored.path),
|
||||||
@@ -319,19 +316,17 @@ export const writeLivePhotoJSON = (
|
|||||||
),
|
),
|
||||||
);
|
);
|
||||||
|
|
||||||
// The original of `file` as stored in `dir` under `name` (without its
|
// The original of `file` as the cache or a backup stored it in `dir`, when all
|
||||||
// extension), when all of it is there: `<name><ext>`, or a live photo's image
|
// of it is there: `<fileID><ext>`, or a live photo's image and video.
|
||||||
// and video.
|
|
||||||
export const storedOriginal = (
|
export const storedOriginal = (
|
||||||
dir: string,
|
dir: string,
|
||||||
name: string,
|
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
): { path: string; videoPath?: string } | undefined => {
|
): { path: string; videoPath?: string } | undefined => {
|
||||||
if (file.metadata.fileType !== "livePhoto") {
|
if (file.metadata.fileType !== "livePhoto") {
|
||||||
const path = join(dir, `${name}${safeExtension(file.metadata.title)}`);
|
const path = join(dir, nameInOriginals(file));
|
||||||
return hasContent(path) ? { path } : undefined;
|
return hasContent(path) ? { path } : undefined;
|
||||||
}
|
}
|
||||||
const stored = readLivePhotoJSON(dir, name);
|
const stored = readLivePhotoJSON(dir, file.id);
|
||||||
return stored !== undefined &&
|
return stored !== undefined &&
|
||||||
hasContent(stored.path) &&
|
hasContent(stored.path) &&
|
||||||
hasContent(stored.videoPath)
|
hasContent(stored.videoPath)
|
||||||
@@ -339,76 +334,10 @@ export const storedOriginal = (
|
|||||||
: undefined;
|
: undefined;
|
||||||
};
|
};
|
||||||
|
|
||||||
// The original of `file` as stored at its save path under `root`, when all of
|
|
||||||
// it is there.
|
|
||||||
export const storedAtSavePath = (
|
|
||||||
root: string,
|
|
||||||
file: EnteFile,
|
|
||||||
): { path: string; videoPath?: string } | undefined => {
|
|
||||||
const path = savePath(root, file);
|
|
||||||
return storedOriginal(dirname(path), basename(path, extname(path)), file);
|
|
||||||
};
|
|
||||||
|
|
||||||
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
|
||||||
// `dest` appears only once it is whole ("present means complete"). As in the
|
|
||||||
// download writer, the temp file is fsynced before the rename and the directory
|
|
||||||
// after it, so a power cut cannot leave a correctly named but short original.
|
|
||||||
// The temp name carries this process's ID so a later run can tell a leftover
|
|
||||||
// from a copy still in progress (see `removeLeftoverTempFiles`).
|
|
||||||
export const copyAtomic = async (src: string, dest: string): Promise<void> => {
|
|
||||||
if (src === dest) return;
|
|
||||||
const tmp = join(
|
|
||||||
dirname(dest),
|
|
||||||
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
|
|
||||||
.toString(36)
|
|
||||||
.slice(2)}.tmp`,
|
|
||||||
);
|
|
||||||
try {
|
|
||||||
await copyFile(src, tmp);
|
|
||||||
await fsyncPath(tmp);
|
|
||||||
// `rename` replaces the destination's directory entry: an existing
|
|
||||||
// symlink at `dest` is replaced, not followed, and the new file has
|
|
||||||
// the temp file's permissions (copied from `src`).
|
|
||||||
await rename(tmp, dest);
|
|
||||||
await fsyncPath(dirname(dest));
|
|
||||||
} finally {
|
|
||||||
await rm(tmp, { force: true });
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// Put the original of `file` at its save path under `root`, creating its
|
|
||||||
// folders. `get` is given the save path and returns where the original is: a
|
|
||||||
// fetch writes it there, and a copy the cache holds is copied there. A live
|
|
||||||
// photo's image and video go beside the save path, each with its own
|
|
||||||
// extension: when they came from the cache they are copied. Then the JSON file
|
|
||||||
// naming them is written, which is what makes the live photo count as stored.
|
|
||||||
export const placeOriginal = async (
|
|
||||||
root: string,
|
|
||||||
file: EnteFile,
|
|
||||||
get: (dest: string) => Promise<{ path: string; videoPath?: string }>,
|
|
||||||
): Promise<{ path: string; videoPath?: string }> => {
|
|
||||||
const dest = savePath(root, file);
|
|
||||||
await mkdir(dirname(dest), { recursive: true });
|
|
||||||
const got = await get(dest);
|
|
||||||
if (got.videoPath === undefined) {
|
|
||||||
await copyAtomic(got.path, dest);
|
|
||||||
return { path: dest };
|
|
||||||
}
|
|
||||||
const path = withExtension(dest, extname(got.path));
|
|
||||||
const videoPath = withExtension(dest, extname(got.videoPath));
|
|
||||||
if (got.path !== path) {
|
|
||||||
await copyAtomic(got.path, path);
|
|
||||||
await copyAtomic(got.videoPath, videoPath);
|
|
||||||
}
|
|
||||||
const name = basename(dest, extname(dest));
|
|
||||||
await writeLivePhotoJSON(dirname(dest), name, { path, videoPath });
|
|
||||||
return { path, videoPath };
|
|
||||||
};
|
|
||||||
|
|
||||||
export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||||
private readonly pools: RequestPools;
|
private readonly pools: RequestPools;
|
||||||
private readonly source: ContentSource;
|
private readonly source: ContentSource;
|
||||||
private readonly downloadDirectory: string;
|
private readonly downloadDirectory?: string;
|
||||||
private readonly getFile: (fileID: number) => EnteFile | undefined;
|
private readonly getFile: (fileID: number) => EnteFile | undefined;
|
||||||
private readonly originalsDir: string;
|
private readonly originalsDir: string;
|
||||||
private readonly thumbnailsDir: string;
|
private readonly thumbnailsDir: string;
|
||||||
@@ -506,23 +435,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Put the original at the save path of `file` under the download directory
|
// Get an original for a backup. One not present anywhere is written
|
||||||
// and return it there. `file` is the copy the `Photo` holds, so the path is
|
|
||||||
// the one its `savePath` names, even after a refresh changed the date. One
|
|
||||||
// already stored there is returned as it is; one the cache holds is copied
|
|
||||||
// from it; any other is fetched straight to the save path, with no copy
|
|
||||||
// left in the cache.
|
|
||||||
async download(file: EnteFile): Promise<ContentResult> {
|
|
||||||
const root = this.downloadDirectory;
|
|
||||||
const saved =
|
|
||||||
storedAtSavePath(root, file) ??
|
|
||||||
(await placeOriginal(root, file, (dest) =>
|
|
||||||
this.backupOriginal(file.id, dest),
|
|
||||||
));
|
|
||||||
return { ...saved, bytes: fileSize(saved.path) ?? 0 };
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get an original for a save path. One not present anywhere is written
|
|
||||||
// straight to `destination` and recorded there, so no second copy lands
|
// straight to `destination` and recorded there, so no second copy lands
|
||||||
// in the cache; one already present is returned where it is.
|
// in the cache; one already present is returned where it is.
|
||||||
async backupOriginal(
|
async backupOriginal(
|
||||||
@@ -687,16 +600,17 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
await this.touch(cached.path);
|
await this.touch(cached.path);
|
||||||
return { ...cached, bytes: size, cached: true };
|
return { ...cached, bytes: size, cached: true };
|
||||||
}
|
}
|
||||||
// A recorded file that has since gone re-fetches below. So does a
|
// A recorded file that has since gone, or a live photo an earlier
|
||||||
// live photo recorded with no video: the cache opened before the
|
// version stored as one ZIP, re-fetches below.
|
||||||
// library's records said it is a live photo, while its image and
|
|
||||||
// video had no JSON file beside them yet.
|
|
||||||
known.delete(fileID);
|
known.delete(fileID);
|
||||||
}
|
}
|
||||||
|
|
||||||
// An original already stored at its save path counts as present.
|
// An original a backup already stored counts as present.
|
||||||
if (kind === "original") {
|
if (kind === "original" && this.downloadDirectory !== undefined) {
|
||||||
const stored = storedAtSavePath(this.downloadDirectory, file);
|
const stored = storedOriginal(
|
||||||
|
join(this.downloadDirectory, "originals"),
|
||||||
|
file,
|
||||||
|
);
|
||||||
if (stored !== undefined) {
|
if (stored !== undefined) {
|
||||||
this.originals.set(fileID, stored);
|
this.originals.set(fileID, stored);
|
||||||
return {
|
return {
|
||||||
@@ -732,7 +646,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
? this.beginOriginalWrite(fileID)
|
? this.beginOriginalWrite(fileID)
|
||||||
: null;
|
: null;
|
||||||
try {
|
try {
|
||||||
const stored = await this.fetchInto(
|
const stored = await this.download(
|
||||||
file,
|
file,
|
||||||
dest,
|
dest,
|
||||||
kind,
|
kind,
|
||||||
@@ -747,12 +661,12 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// `placeOriginal` records a live photo it saves.
|
// A backup records its own live photos.
|
||||||
if (
|
if (
|
||||||
stored.videoPath !== undefined &&
|
stored.videoPath !== undefined &&
|
||||||
opts?.destination === undefined
|
opts?.destination === undefined
|
||||||
) {
|
) {
|
||||||
await writeLivePhotoJSON(dir, String(fileID), {
|
await writeLivePhotoJSON(dir, fileID, {
|
||||||
path: stored.path,
|
path: stored.path,
|
||||||
videoPath: stored.videoPath,
|
videoPath: stored.videoPath,
|
||||||
});
|
});
|
||||||
@@ -775,7 +689,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
|
|
||||||
// Fetch into `destination`, returning where the bytes landed: there, or
|
// Fetch into `destination`, returning where the bytes landed: there, or
|
||||||
// for a live photo, its image and video beside it.
|
// for a live photo, its image and video beside it.
|
||||||
private async fetchInto(
|
private async download(
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
destination: string,
|
destination: string,
|
||||||
kind: Kind,
|
kind: Kind,
|
||||||
@@ -800,10 +714,10 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
await utimes(path, now, now).catch(() => undefined);
|
await utimes(path, now, now).catch(() => undefined);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Every stored original that lives under `originalsDir` (a save-path hit
|
// Every stored original that lives under `originalsDir` (a backup-directory
|
||||||
// recorded in the map is excluded), with its size and mtime; a live
|
// hit recorded in the map is excluded), with its size and mtime; a live
|
||||||
// photo's size includes its video. Entries whose file has vanished are
|
// photo's size includes its video. Entries whose file has vanished are
|
||||||
// dropped from the map. Save paths and thumbnails are never counted.
|
// dropped from the map. Backups and thumbnails are never counted.
|
||||||
private async measureOriginals(): Promise<{
|
private async measureOriginals(): Promise<{
|
||||||
entries: {
|
entries: {
|
||||||
fileID: number;
|
fileID: number;
|
||||||
@@ -909,7 +823,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
await rm(
|
await rm(
|
||||||
join(
|
join(
|
||||||
this.originalsDir,
|
this.originalsDir,
|
||||||
livePhotoJSONName(String(e.fileID)),
|
livePhotoJSONName(e.fileID),
|
||||||
),
|
),
|
||||||
{ force: true },
|
{ force: true },
|
||||||
);
|
);
|
||||||
@@ -959,14 +873,20 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
if (id === undefined || !existsSync(path)) continue;
|
if (id === undefined || !existsSync(path)) continue;
|
||||||
// A live photo's image and video are one entry, as the JSON file
|
// A live photo's image and video are one entry, as the JSON file
|
||||||
// beside them names them. A live photo's file with no such JSON
|
// beside them names them. A live photo's file with no such JSON
|
||||||
// file is not its original and is left alone: another process may
|
// file is not its original. If it is a ZIP, it is the one an
|
||||||
// have just stored it and not yet written the JSON file.
|
// earlier version stored under the image's name, and is removed.
|
||||||
const livePhoto = names.has(livePhotoJSONName(String(id)))
|
// Any other is left alone: another process may have just stored
|
||||||
? readLivePhotoJSON(dir, String(id))
|
// it and not yet written the JSON file.
|
||||||
|
const livePhoto = names.has(livePhotoJSONName(id))
|
||||||
|
? readLivePhotoJSON(dir, id)
|
||||||
: undefined;
|
: undefined;
|
||||||
if (livePhoto !== undefined) {
|
if (livePhoto !== undefined) {
|
||||||
into.set(id, livePhoto);
|
into.set(id, livePhoto);
|
||||||
} else if (!isLivePhoto(id)) {
|
} else if (isLivePhoto(id)) {
|
||||||
|
if (isZip(path)) {
|
||||||
|
await rm(path, { force: true }).catch(() => undefined);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
into.set(id, { path });
|
into.set(id, { path });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+36
-68
@@ -27,7 +27,7 @@
|
|||||||
// never masked by a subsequent empty refresh.
|
// never masked by a subsequent empty refresh.
|
||||||
|
|
||||||
import { rm } from "node:fs/promises";
|
import { rm } from "node:fs/promises";
|
||||||
import { join, resolve } from "node:path";
|
import { join } from "node:path";
|
||||||
import envPaths from "env-paths";
|
import envPaths from "env-paths";
|
||||||
|
|
||||||
import { MetadataStore } from "./store.js";
|
import { MetadataStore } from "./store.js";
|
||||||
@@ -49,12 +49,9 @@ import {
|
|||||||
type PhotosAPI,
|
type PhotosAPI,
|
||||||
type TimelineAPI,
|
type TimelineAPI,
|
||||||
type FreshReads,
|
type FreshReads,
|
||||||
type SavePathLookup,
|
|
||||||
} from "./read.js";
|
} from "./read.js";
|
||||||
import {
|
import {
|
||||||
ContentCache,
|
ContentCache,
|
||||||
savePath,
|
|
||||||
storedAtSavePath,
|
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
type ThumbnailsAPI,
|
type ThumbnailsAPI,
|
||||||
type EnsureOptions,
|
type EnsureOptions,
|
||||||
@@ -73,7 +70,6 @@ export {
|
|||||||
type PhotoFilter,
|
type PhotoFilter,
|
||||||
type TimelineGroup,
|
type TimelineGroup,
|
||||||
type GroupBy,
|
type GroupBy,
|
||||||
type SavePathLookup,
|
|
||||||
} from "./read.js";
|
} from "./read.js";
|
||||||
export {
|
export {
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
@@ -94,7 +90,6 @@ import type { Collection, EnteFile } from "../model/types.js";
|
|||||||
import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
|
import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
|
||||||
|
|
||||||
export {
|
export {
|
||||||
lockBackupDirectory,
|
|
||||||
runBackup,
|
runBackup,
|
||||||
type BackupOptions,
|
type BackupOptions,
|
||||||
type BackupResult,
|
type BackupResult,
|
||||||
@@ -174,10 +169,8 @@ export interface LibraryOptions {
|
|||||||
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
||||||
// plus the user id, so each account has its own cache.
|
// plus the user id, so each account has its own cache.
|
||||||
cacheDirectory?: string;
|
cacheDirectory?: string;
|
||||||
// The root of every photo's save path, where `Photo.download()` and
|
// Persistent backup destination. The refresh loop does not use it; the
|
||||||
// `lib.backup()` put originals. Defaults to `photos` in the working
|
// content cache treats an original already stored there as present.
|
||||||
// directory at open. The content cache treats an original already stored
|
|
||||||
// at its save path as present.
|
|
||||||
downloadDirectory?: string;
|
downloadDirectory?: string;
|
||||||
refreshIntervalSeconds?: number;
|
refreshIntervalSeconds?: number;
|
||||||
onProgress?: RefreshProgressCallback;
|
onProgress?: RefreshProgressCallback;
|
||||||
@@ -241,7 +234,7 @@ export interface LibraryStatus {
|
|||||||
|
|
||||||
export class Library {
|
export class Library {
|
||||||
readonly cacheDirectory: string;
|
readonly cacheDirectory: string;
|
||||||
readonly downloadDirectory: string;
|
readonly downloadDirectory?: string;
|
||||||
|
|
||||||
// The in-process read surface (issue #44). Each namespace answers
|
// The in-process read surface (issue #44). Each namespace answers
|
||||||
// synchronously from the live record projection; no read touches the
|
// synchronously from the live record projection; no read touches the
|
||||||
@@ -280,8 +273,7 @@ export class Library {
|
|||||||
private cycle?: Promise<void>;
|
private cycle?: Promise<void>;
|
||||||
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
||||||
// refresh whose pass is still running kicks nothing new. Holds the running
|
// refresh whose pass is still running kicks nothing new. Holds the running
|
||||||
// pass, so `close()` and `backup()` can wait for it. It rejects when the
|
// pass, so `close()` can wait for it.
|
||||||
// pass fails.
|
|
||||||
private mlFetch?: Promise<void>;
|
private mlFetch?: Promise<void>;
|
||||||
private closed = false;
|
private closed = false;
|
||||||
private lastRefreshAt?: number;
|
private lastRefreshAt?: number;
|
||||||
@@ -304,7 +296,7 @@ export class Library {
|
|||||||
store: MetadataStore;
|
store: MetadataStore;
|
||||||
userID: number;
|
userID: number;
|
||||||
cacheDirectory: string;
|
cacheDirectory: string;
|
||||||
downloadDirectory: string;
|
downloadDirectory?: string;
|
||||||
intervalMs: number;
|
intervalMs: number;
|
||||||
onProgress?: RefreshProgressCallback;
|
onProgress?: RefreshProgressCallback;
|
||||||
pools: RequestPools;
|
pools: RequestPools;
|
||||||
@@ -328,14 +320,8 @@ export class Library {
|
|||||||
// The read namespaces derive fresh from the store on each call, so they
|
// The read namespaces derive fresh from the store on each call, so they
|
||||||
// always reflect the latest refresh.
|
// always reflect the latest refresh.
|
||||||
const derive = (): DerivedRecords => this.deriveNow();
|
const derive = (): DerivedRecords => this.deriveNow();
|
||||||
const root = this.downloadDirectory;
|
this.albums = makeAlbumsAPI(derive, this.cache);
|
||||||
const saves: SavePathLookup = {
|
this.photos = makePhotosAPI(derive, this.cache);
|
||||||
savePath: (file) =>
|
|
||||||
storedAtSavePath(root, file)?.path ?? savePath(root, file),
|
|
||||||
isLocal: (file) => storedAtSavePath(root, file) !== undefined,
|
|
||||||
};
|
|
||||||
this.albums = makeAlbumsAPI(derive, saves, this.cache);
|
|
||||||
this.photos = makePhotosAPI(derive, saves, this.cache);
|
|
||||||
this.timeline = makeTimelineAPI(derive);
|
this.timeline = makeTimelineAPI(derive);
|
||||||
this.thumbnails = {
|
this.thumbnails = {
|
||||||
ensure: (opts: EnsureOptions): Promise<EnsureResult[]> => {
|
ensure: (opts: EnsureOptions): Promise<EnsureResult[]> => {
|
||||||
@@ -364,13 +350,6 @@ export class Library {
|
|||||||
const { userID } = opts.client.whoami();
|
const { userID } = opts.client.whoami();
|
||||||
const cacheDirectory =
|
const cacheDirectory =
|
||||||
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
||||||
if (opts.downloadDirectory === "") {
|
|
||||||
throw new Error(
|
|
||||||
"library: downloadDirectory is empty (leave it out to save " +
|
|
||||||
"under photos/ in the working directory)",
|
|
||||||
);
|
|
||||||
}
|
|
||||||
const downloadDirectory = opts.downloadDirectory ?? resolve("photos");
|
|
||||||
const metadataPath = join(cacheDirectory, "metadata.json");
|
const metadataPath = join(cacheDirectory, "metadata.json");
|
||||||
let store = await MetadataStore.load(metadataPath);
|
let store = await MetadataStore.load(metadataPath);
|
||||||
// A cache directory given explicitly can hold another account's cache.
|
// A cache directory given explicitly can hold another account's cache.
|
||||||
@@ -425,7 +404,7 @@ export class Library {
|
|||||||
pools,
|
pools,
|
||||||
source,
|
source,
|
||||||
cacheDirectory,
|
cacheDirectory,
|
||||||
downloadDirectory,
|
downloadDirectory: opts.downloadDirectory,
|
||||||
getFile: (fileID) => store.getFileByID(fileID),
|
getFile: (fileID) => store.getFileByID(fileID),
|
||||||
cacheOriginalsMaxBytes: opts.cacheOriginalsMaxBytes,
|
cacheOriginalsMaxBytes: opts.cacheOriginalsMaxBytes,
|
||||||
freeBelowBytes: opts.freeBelowBytes,
|
freeBelowBytes: opts.freeBelowBytes,
|
||||||
@@ -447,7 +426,7 @@ export class Library {
|
|||||||
store,
|
store,
|
||||||
userID,
|
userID,
|
||||||
cacheDirectory,
|
cacheDirectory,
|
||||||
downloadDirectory,
|
downloadDirectory: opts.downloadDirectory,
|
||||||
intervalMs,
|
intervalMs,
|
||||||
onProgress: opts.onProgress,
|
onProgress: opts.onProgress,
|
||||||
pools,
|
pools,
|
||||||
@@ -491,10 +470,9 @@ export class Library {
|
|||||||
return this.store.getFile(collectionID, fileID);
|
return this.store.getFile(collectionID, fileID);
|
||||||
}
|
}
|
||||||
|
|
||||||
// The membership of a file its record is read from, addressed by file id
|
// Any membership of a file, addressed by file id alone. A file's own
|
||||||
// alone. A file's own metadata (title, creationTime) is identical across
|
// metadata (title, creationTime) is identical across the collections it
|
||||||
// the collections it belongs to, so this serves the point commands that
|
// belongs to, so this serves the point commands that hold only a fileID.
|
||||||
// hold only a fileID.
|
|
||||||
getFileByID(fileID: number): EnteFile | undefined {
|
getFileByID(fileID: number): EnteFile | undefined {
|
||||||
return this.store.getFileByID(fileID);
|
return this.store.getFileByID(fileID);
|
||||||
}
|
}
|
||||||
@@ -565,25 +543,27 @@ export class Library {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
// Back up every in-scope file to `downloadDirectory` in the historical
|
||||||
// library's, each original at its save path, with a durable failure
|
// on-disk layout, with a durable failure ledger (issue #51). Waits for a
|
||||||
// ledger (issue #51). Takes the lock in that directory first, unless
|
// completed refresh first, as `fresh()` does, joining one already running,
|
||||||
// `opts.lockHeld` says the caller holds it, failing at once while another
|
// and rejects before touching any file when it fails. Then fetches pending
|
||||||
// backup of it runs (see `runBackup`). Waits for a
|
// originals (and optional thumbnails) through the content cache and pools,
|
||||||
// completed refresh, as `fresh()` does, joining one already running, and
|
// and rebuilds the derived symlink/JSON views from the model. Throws before
|
||||||
// rejects before touching any file but the lock when it fails. Then puts
|
// any network work when no download directory is available or no content
|
||||||
// pending originals at their save paths as `Photo.download()` does (and
|
// cache backs the originals it must fetch.
|
||||||
// optional thumbnails) through the content cache and pools, waits for an
|
|
||||||
// ML data fetch, and rebuilds the derived symlink/JSON views from the
|
|
||||||
// model, each file's JSON with its ML data, beside an `account.json` with
|
|
||||||
// the account's email and user ID.
|
|
||||||
// Throws before any network work when no content cache backs the
|
|
||||||
// originals it must fetch.
|
|
||||||
backup(opts?: BackupOptions): Promise<BackupResult> {
|
backup(opts?: BackupOptions): Promise<BackupResult> {
|
||||||
const downloadDirectory =
|
const downloadDirectory =
|
||||||
opts?.downloadDirectory ?? this.downloadDirectory;
|
opts?.downloadDirectory ?? this.downloadDirectory;
|
||||||
const includeOriginals = opts?.includeOriginals ?? true;
|
const includeOriginals = opts?.includeOriginals ?? true;
|
||||||
const includeThumbnails = opts?.includeThumbnails ?? false;
|
const includeThumbnails = opts?.includeThumbnails ?? false;
|
||||||
|
if (!downloadDirectory) {
|
||||||
|
return Promise.reject(
|
||||||
|
new Error(
|
||||||
|
"backup requires a downloadDirectory (pass one to " +
|
||||||
|
"backup() or open the library with one)",
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
if ((includeOriginals || includeThumbnails) && !this.cache) {
|
if ((includeOriginals || includeThumbnails) && !this.cache) {
|
||||||
return Promise.reject(
|
return Promise.reject(
|
||||||
new Error(
|
new Error(
|
||||||
@@ -594,15 +574,12 @@ export class Library {
|
|||||||
const cache = this.cache;
|
const cache = this.cache;
|
||||||
return runBackup(
|
return runBackup(
|
||||||
{
|
{
|
||||||
whoami: () => this.client.whoami(),
|
|
||||||
refresh: () => this.refreshNow(),
|
refresh: () => this.refreshNow(),
|
||||||
listCollections: () => this.store.listCollections(),
|
listCollections: () => this.store.listCollections(),
|
||||||
listFiles: (id) => this.store.listFiles(id),
|
listFiles: (id) => this.store.listFiles(id),
|
||||||
original: (fileID, destination) =>
|
original: (fileID, destination) =>
|
||||||
cache!.backupOriginal(fileID, destination),
|
cache!.backupOriginal(fileID, destination),
|
||||||
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
||||||
fetchMLData: () => this.fetchMLDataNow(),
|
|
||||||
mlData: (fileID) => this.mldata.forFile({ fileID }),
|
|
||||||
},
|
},
|
||||||
{ ...opts, downloadDirectory },
|
{ ...opts, downloadDirectory },
|
||||||
);
|
);
|
||||||
@@ -622,7 +599,7 @@ export class Library {
|
|||||||
this.timer = undefined;
|
this.timer = undefined;
|
||||||
}
|
}
|
||||||
await this.cycle?.catch(() => {});
|
await this.cycle?.catch(() => {});
|
||||||
await this.mlFetch?.catch(() => {});
|
await this.mlFetch;
|
||||||
await precacheClosed;
|
await precacheClosed;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -686,9 +663,10 @@ export class Library {
|
|||||||
// Backfill ML data for the files this refresh knows about. It runs
|
// Backfill ML data for the files this refresh knows about. It runs
|
||||||
// outside the refresh's success/failure so a fetch or disk problem
|
// outside the refresh's success/failure so a fetch or disk problem
|
||||||
// there never marks the metadata refresh failed, and it is not
|
// there never marks the metadata refresh failed, and it is not
|
||||||
// awaited so it never stalls the refresh interval. Its failure is
|
// awaited so it never stalls the refresh interval.
|
||||||
// reported through `status()` and `onProgress`.
|
this.mlFetch ??= this.runMLFetch().finally(() => {
|
||||||
void this.fetchMLDataNow().catch(() => {});
|
this.mlFetch = undefined;
|
||||||
|
});
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
const error = err instanceof Error ? err.message : String(err);
|
const error = err instanceof Error ? err.message : String(err);
|
||||||
this.lastError = error;
|
this.lastError = error;
|
||||||
@@ -793,19 +771,10 @@ export class Library {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Join the running ML fetch pass, or start one when none runs. Resolves at
|
|
||||||
// once when the client cannot fetch ML data; rejects when the pass fails.
|
|
||||||
private fetchMLDataNow(): Promise<void> {
|
|
||||||
this.mlFetch ??= this.runMLFetch().finally(() => {
|
|
||||||
this.mlFetch = undefined;
|
|
||||||
});
|
|
||||||
return this.mlFetch;
|
|
||||||
}
|
|
||||||
|
|
||||||
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
||||||
// the store knows about that is not cached (or whose `updationTime` has
|
// the store knows about that is not cached (or whose `updationTime` has
|
||||||
// advanced), through the metadata pool, and update the CLIP index. A
|
// advanced), through the metadata pool, and update the CLIP index. Guarded
|
||||||
// failure is reported through `status()` and `onProgress`, then thrown.
|
// so passes never overlap; a failure is reported, not thrown.
|
||||||
private async runMLFetch(): Promise<void> {
|
private async runMLFetch(): Promise<void> {
|
||||||
const mldata = this.mlStore;
|
const mldata = this.mlStore;
|
||||||
// Bind so the call keeps the client as its receiver when invoked
|
// Bind so the call keeps the client as its receiver when invoked
|
||||||
@@ -849,7 +818,6 @@ export class Library {
|
|||||||
const error = err instanceof Error ? err.message : String(err);
|
const error = err instanceof Error ? err.message : String(err);
|
||||||
this.lastMLError = error;
|
this.lastMLError = error;
|
||||||
this.emit({ operation: "fetchMLData", status: "failed", error });
|
this.emit({ operation: "fetchMLData", status: "failed", error });
|
||||||
throw err;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+21
-169
@@ -11,32 +11,15 @@
|
|||||||
// access and, for an album, its photos. They are not sent across IPC — the
|
// access and, for an album, its photos. They are not sent across IPC — the
|
||||||
// plain records are the serializable surface, and `record()` returns one.
|
// plain records are the serializable surface, and `record()` returns one.
|
||||||
//
|
//
|
||||||
// A `Photo` also fetches its own bytes: `original()`, `thumbnail()`,
|
// A `Photo` also fetches its own bytes: `original()` and `thumbnail()` go
|
||||||
// `download()`, `content()`, `exif()` and the methods that each return one
|
// through the on-disk content cache (issue #46), the one place in this module
|
||||||
// EXIF field go through the on-disk content cache (issue #46), and are
|
// that is not synchronous and RAM-only. A library opened without a content
|
||||||
// the one place in this module that may touch the network. A library opened
|
// source leaves that cache absent, and those two methods then throw.
|
||||||
// without a content source leaves that cache absent, and those methods then
|
|
||||||
// throw. `savePath` and `isLocal` look only at the disk and need no cache.
|
|
||||||
|
|
||||||
import { readFile } from "node:fs/promises";
|
import type { CollectionType, FileType } from "../model/types.js";
|
||||||
|
|
||||||
import {
|
|
||||||
readAllExifTags,
|
|
||||||
readPhotoExif,
|
|
||||||
type ExifTags,
|
|
||||||
type PhotoExif,
|
|
||||||
} from "../exif.js";
|
|
||||||
import type { CollectionType, EnteFile, FileType } from "../model/types.js";
|
|
||||||
import type { ContentOptions, ContentResult, PhotoContent } from "./content.js";
|
import type { ContentOptions, ContentResult, PhotoContent } from "./content.js";
|
||||||
import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js";
|
import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js";
|
||||||
|
|
||||||
// Where a photo's original is saved, and whether all of it is there. The
|
|
||||||
// library answers both from the disk, with or without a content cache.
|
|
||||||
export interface SavePathLookup {
|
|
||||||
savePath(file: EnteFile): string;
|
|
||||||
isLocal(file: EnteFile): boolean;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Newest first, with fileID as a stable tiebreak so equal-timed files order
|
// Newest first, with fileID as a stable tiebreak so equal-timed files order
|
||||||
// deterministically — the same order the record projection uses.
|
// deterministically — the same order the record projection uses.
|
||||||
const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
|
const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
|
||||||
@@ -47,23 +30,12 @@ const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
|
|||||||
const byNewestAlbum = (a: AlbumRecord, b: AlbumRecord): number =>
|
const byNewestAlbum = (a: AlbumRecord, b: AlbumRecord): number =>
|
||||||
b.updationTime - a.updationTime || b.collectionID - a.collectionID;
|
b.updationTime - a.updationTime || b.collectionID - a.collectionID;
|
||||||
|
|
||||||
// A method for each `PhotoExif` field, named after it, taking the options
|
|
||||||
// `exif()` takes and giving that field. `Photo` implements it, so the build's
|
|
||||||
// type check fails when `PhotoExif` has a field `Photo` has no method for.
|
|
||||||
type PhotoExifMethods = {
|
|
||||||
[K in keyof PhotoExif]-?: (opts?: ContentOptions) => Promise<PhotoExif[K]>;
|
|
||||||
};
|
|
||||||
|
|
||||||
// A single photo. Field access mirrors `PhotoRecord`; `record()` returns the
|
// A single photo. Field access mirrors `PhotoRecord`; `record()` returns the
|
||||||
// underlying plain record for callers that need the IPC-safe value. `file` is
|
// underlying plain record for callers that need the IPC-safe value.
|
||||||
// the membership the record is read from, so the save path carries the date of
|
export class Photo {
|
||||||
// `takenAt` and stays known after a refresh removes the file from the library.
|
|
||||||
export class Photo implements PhotoExifMethods {
|
|
||||||
constructor(
|
constructor(
|
||||||
private readonly rec: PhotoRecord,
|
private readonly rec: PhotoRecord,
|
||||||
private readonly file: EnteFile,
|
private readonly content?: PhotoContent,
|
||||||
private readonly saves: SavePathLookup,
|
|
||||||
private readonly cache?: PhotoContent,
|
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
get fileID(): number {
|
get fileID(): number {
|
||||||
@@ -78,13 +50,6 @@ export class Photo implements PhotoExifMethods {
|
|||||||
get takenAt(): number {
|
get takenAt(): number {
|
||||||
return this.rec.takenAt;
|
return this.rec.takenAt;
|
||||||
}
|
}
|
||||||
get modifiedAt(): number {
|
|
||||||
return this.rec.modifiedAt;
|
|
||||||
}
|
|
||||||
// The local-time year of `takenAt`.
|
|
||||||
get year(): number {
|
|
||||||
return new Date(this.rec.takenAt).getFullYear();
|
|
||||||
}
|
|
||||||
get fileType(): FileType {
|
get fileType(): FileType {
|
||||||
return this.rec.fileType;
|
return this.rec.fileType;
|
||||||
}
|
}
|
||||||
@@ -103,9 +68,6 @@ export class Photo implements PhotoExifMethods {
|
|||||||
get longitude(): number | undefined {
|
get longitude(): number | undefined {
|
||||||
return this.rec.longitude;
|
return this.rec.longitude;
|
||||||
}
|
}
|
||||||
get hash(): string | undefined {
|
|
||||||
return this.rec.hash;
|
|
||||||
}
|
|
||||||
get isArchived(): boolean {
|
get isArchived(): boolean {
|
||||||
return this.rec.isArchived;
|
return this.rec.isArchived;
|
||||||
}
|
}
|
||||||
@@ -113,132 +75,30 @@ export class Photo implements PhotoExifMethods {
|
|||||||
return this.rec.isHidden;
|
return this.rec.isHidden;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Where `download()` and `lib.backup()` put the original under the
|
|
||||||
// library's download directory, whether or not it is there yet:
|
|
||||||
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`. For a live photo
|
|
||||||
// already stored, its image. For a live photo not yet stored, it carries
|
|
||||||
// the title's extension, and the image may be stored under a different
|
|
||||||
// one.
|
|
||||||
get savePath(): string {
|
|
||||||
return this.saves.savePath(this.file);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Whether the whole original is at `savePath`. A copy only in the cache
|
|
||||||
// does not count.
|
|
||||||
get isLocal(): boolean {
|
|
||||||
return this.saves.isLocal(this.file);
|
|
||||||
}
|
|
||||||
|
|
||||||
record(): PhotoRecord {
|
record(): PhotoRecord {
|
||||||
return this.rec;
|
return this.rec;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fetch and cache the full-resolution original, returning its on-disk path
|
// Fetch and cache the full-resolution original, returning its on-disk path
|
||||||
// and byte length; for a live photo, its image's, and its video's path as
|
// and byte length; for a live photo, its image's, and its video's path as
|
||||||
// `videoPath`. Served from the cache (or the save path) when already
|
// `videoPath`. Served from the cache (or the backup download directory)
|
||||||
// present, otherwise fetched through the content pool.
|
// when already present, otherwise fetched through the content pool.
|
||||||
async original(opts?: ContentOptions): Promise<ContentResult> {
|
async original(opts?: ContentOptions): Promise<ContentResult> {
|
||||||
return this.cacheOrThrow().original(this.rec.fileID, opts);
|
return this.contentOrThrow().original(this.rec.fileID, opts);
|
||||||
}
|
|
||||||
|
|
||||||
// Put the original at `savePath` and return it there, as `original()`
|
|
||||||
// does. When it is already there, nothing is written. When the cache holds
|
|
||||||
// it, it is copied from there; otherwise it is fetched straight to
|
|
||||||
// `savePath`.
|
|
||||||
async download(): Promise<ContentResult> {
|
|
||||||
return this.cacheOrThrow().download(this.file);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// As `original`, for the thumbnail, through the thumbnail pool.
|
// As `original`, for the thumbnail, through the thumbnail pool.
|
||||||
async thumbnail(opts?: ContentOptions): Promise<ContentResult> {
|
async thumbnail(opts?: ContentOptions): Promise<ContentResult> {
|
||||||
return this.cacheOrThrow().thumbnail(this.rec.fileID, opts);
|
return this.contentOrThrow().thumbnail(this.rec.fileID, opts);
|
||||||
}
|
}
|
||||||
|
|
||||||
// The original's bytes, read from where `original()` puts it. For a live
|
private contentOrThrow(): PhotoContent {
|
||||||
// photo, its image's.
|
if (!this.content) {
|
||||||
async content(opts?: ContentOptions): Promise<Uint8Array> {
|
|
||||||
const { path } = await this.original(opts);
|
|
||||||
return readFile(path);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Every EXIF tag of the original, keyed by name (see `ExifTags`), read
|
|
||||||
// from `content()`, so this may download it. EXIF is read from any image
|
|
||||||
// format exifreader reads, JPEG and HEIC/HEIF among them; any other file
|
|
||||||
// gives `{}`, and a video gives it without fetching anything. Like the
|
|
||||||
// other content methods, it throws when there is no content cache, video
|
|
||||||
// or not.
|
|
||||||
async exif(opts?: ContentOptions): Promise<ExifTags> {
|
|
||||||
this.cacheOrThrow();
|
|
||||||
if (this.rec.fileType === "video") return {};
|
|
||||||
return readAllExifTags(await this.content(opts));
|
|
||||||
}
|
|
||||||
|
|
||||||
// One field each, named and typed as in `PhotoExif`, picked from the tags
|
|
||||||
// `exif()` returns, and undefined when the file lacks it. Each call runs
|
|
||||||
// `exif()`, which reads the original again.
|
|
||||||
async make(opts?: ContentOptions): Promise<PhotoExif["make"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).make;
|
|
||||||
}
|
|
||||||
async model(opts?: ContentOptions): Promise<PhotoExif["model"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).model;
|
|
||||||
}
|
|
||||||
async lensModel(opts?: ContentOptions): Promise<PhotoExif["lensModel"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).lensModel;
|
|
||||||
}
|
|
||||||
async dateTimeOriginal(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["dateTimeOriginal"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).dateTimeOriginal;
|
|
||||||
}
|
|
||||||
async offsetTimeOriginal(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["offsetTimeOriginal"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).offsetTimeOriginal;
|
|
||||||
}
|
|
||||||
async exposureTime(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["exposureTime"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).exposureTime;
|
|
||||||
}
|
|
||||||
async fNumber(opts?: ContentOptions): Promise<PhotoExif["fNumber"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).fNumber;
|
|
||||||
}
|
|
||||||
async iso(opts?: ContentOptions): Promise<PhotoExif["iso"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).iso;
|
|
||||||
}
|
|
||||||
async focalLength(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["focalLength"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).focalLength;
|
|
||||||
}
|
|
||||||
async orientation(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["orientation"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).orientation;
|
|
||||||
}
|
|
||||||
async gpsLatitude(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["gpsLatitude"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).gpsLatitude;
|
|
||||||
}
|
|
||||||
async gpsLongitude(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["gpsLongitude"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).gpsLongitude;
|
|
||||||
}
|
|
||||||
async gpsAltitude(
|
|
||||||
opts?: ContentOptions,
|
|
||||||
): Promise<PhotoExif["gpsAltitude"]> {
|
|
||||||
return readPhotoExif(await this.exif(opts)).gpsAltitude;
|
|
||||||
}
|
|
||||||
|
|
||||||
private cacheOrThrow(): PhotoContent {
|
|
||||||
if (!this.cache) {
|
|
||||||
throw new Error(
|
throw new Error(
|
||||||
"Photo content requires a library opened with a content cache",
|
"Photo content requires a library opened with a content cache",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
return this.cache;
|
return this.content;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -248,7 +108,6 @@ export class Album {
|
|||||||
constructor(
|
constructor(
|
||||||
private readonly rec: AlbumRecord,
|
private readonly rec: AlbumRecord,
|
||||||
private readonly records: DerivedRecords,
|
private readonly records: DerivedRecords,
|
||||||
private readonly saves: SavePathLookup,
|
|
||||||
private readonly content?: PhotoContent,
|
private readonly content?: PhotoContent,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
@@ -283,10 +142,7 @@ export class Album {
|
|||||||
const out: Photo[] = [];
|
const out: Photo[] = [];
|
||||||
for (const id of this.rec.fileIDs) {
|
for (const id of this.rec.fileIDs) {
|
||||||
const p = this.records.photos.get(id);
|
const p = this.records.photos.get(id);
|
||||||
const file = this.records.files.get(id);
|
if (p) out.push(new Photo(p, this.content));
|
||||||
if (p && file) {
|
|
||||||
out.push(new Photo(p, file, this.saves, this.content));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
@@ -349,19 +205,18 @@ export interface FreshReads {
|
|||||||
|
|
||||||
export const makeAlbumsAPI = (
|
export const makeAlbumsAPI = (
|
||||||
derive: () => DerivedRecords,
|
derive: () => DerivedRecords,
|
||||||
saves: SavePathLookup,
|
|
||||||
content?: PhotoContent,
|
content?: PhotoContent,
|
||||||
): AlbumsAPI => ({
|
): AlbumsAPI => ({
|
||||||
list: (): Album[] => {
|
list: (): Album[] => {
|
||||||
const records = derive();
|
const records = derive();
|
||||||
return [...records.albums.values()]
|
return [...records.albums.values()]
|
||||||
.sort(byNewestAlbum)
|
.sort(byNewestAlbum)
|
||||||
.map((rec) => new Album(rec, records, saves, content));
|
.map((rec) => new Album(rec, records, content));
|
||||||
},
|
},
|
||||||
byID: ({ collectionID }): Album | undefined => {
|
byID: ({ collectionID }): Album | undefined => {
|
||||||
const records = derive();
|
const records = derive();
|
||||||
const rec = records.albums.get(collectionID);
|
const rec = records.albums.get(collectionID);
|
||||||
return rec ? new Album(rec, records, saves, content) : undefined;
|
return rec ? new Album(rec, records, content) : undefined;
|
||||||
},
|
},
|
||||||
byName: ({ albumName }): Album | undefined => {
|
byName: ({ albumName }): Album | undefined => {
|
||||||
const records = derive();
|
const records = derive();
|
||||||
@@ -370,20 +225,17 @@ export const makeAlbumsAPI = (
|
|||||||
const match = [...records.albums.values()]
|
const match = [...records.albums.values()]
|
||||||
.sort(byNewestAlbum)
|
.sort(byNewestAlbum)
|
||||||
.find((rec) => rec.name === albumName);
|
.find((rec) => rec.name === albumName);
|
||||||
return match ? new Album(match, records, saves, content) : undefined;
|
return match ? new Album(match, records, content) : undefined;
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
export const makePhotosAPI = (
|
export const makePhotosAPI = (
|
||||||
derive: () => DerivedRecords,
|
derive: () => DerivedRecords,
|
||||||
saves: SavePathLookup,
|
|
||||||
content?: PhotoContent,
|
content?: PhotoContent,
|
||||||
): PhotosAPI => ({
|
): PhotosAPI => ({
|
||||||
byID: ({ fileID }): Photo | undefined => {
|
byID: ({ fileID }): Photo | undefined => {
|
||||||
const records = derive();
|
const rec = derive().photos.get(fileID);
|
||||||
const rec = records.photos.get(fileID);
|
return rec ? new Photo(rec, content) : undefined;
|
||||||
const file = records.files.get(fileID);
|
|
||||||
return rec && file ? new Photo(rec, file, saves, content) : undefined;
|
|
||||||
},
|
},
|
||||||
records: ({ fileIDs }): PhotoRecord[] => {
|
records: ({ fileIDs }): PhotoRecord[] => {
|
||||||
const { photos } = derive();
|
const { photos } = derive();
|
||||||
|
|||||||
+17
-43
@@ -5,11 +5,10 @@
|
|||||||
// owner ruling 5). The decrypted `Collection`/`EnteFile` objects stay in RAM in
|
// owner ruling 5). The decrypted `Collection`/`EnteFile` objects stay in RAM in
|
||||||
// the main process; the window only ever sees these records.
|
// the main process; the window only ever sees these records.
|
||||||
//
|
//
|
||||||
// Ente holds edited/basic times in microseconds; records expose `takenAt` and
|
// Ente holds edited/basic times in microseconds; records expose `takenAt` in
|
||||||
// `modifiedAt` in milliseconds. The magic-metadata field names below are the
|
// milliseconds. The magic-metadata field names below are the ones the Ente
|
||||||
// ones the Ente clients write, confirmed against the repo's own fixtures:
|
// clients write, confirmed against the repo's own fixtures: `w`/`h` in
|
||||||
// `w`/`h` in test/cli/metadata-backup.test.ts, `visibility` in
|
// test/cli/metadata-backup.test.ts, `visibility` in test/library/store.test.ts.
|
||||||
// test/library/store.test.ts.
|
|
||||||
|
|
||||||
import type {
|
import type {
|
||||||
Collection,
|
Collection,
|
||||||
@@ -34,17 +33,12 @@ export interface PhotoRecord {
|
|||||||
// Milliseconds. `pubMagicMetadata.editedTime` when the user edited the
|
// Milliseconds. `pubMagicMetadata.editedTime` when the user edited the
|
||||||
// date, else basic-metadata `creationTime`.
|
// date, else basic-metadata `creationTime`.
|
||||||
takenAt: number;
|
takenAt: number;
|
||||||
// Milliseconds. Basic-metadata `modificationTime`.
|
|
||||||
modifiedAt: number;
|
|
||||||
fileType: FileType;
|
fileType: FileType;
|
||||||
caption?: string;
|
caption?: string;
|
||||||
width?: number;
|
width?: number;
|
||||||
height?: number;
|
height?: number;
|
||||||
latitude?: number;
|
latitude?: number;
|
||||||
longitude?: number;
|
longitude?: number;
|
||||||
// The content hash the uploader recorded (`FileMetadata.hash`); files from
|
|
||||||
// very old clients have none.
|
|
||||||
hash?: string;
|
|
||||||
isArchived: boolean;
|
isArchived: boolean;
|
||||||
isHidden: boolean;
|
isHidden: boolean;
|
||||||
// Local cache paths, set once a later phase caches the bytes; unset here.
|
// Local cache paths, set once a later phase caches the bytes; unset here.
|
||||||
@@ -86,10 +80,6 @@ export interface LibraryChange {
|
|||||||
export interface DerivedRecords {
|
export interface DerivedRecords {
|
||||||
albums: Map<number, AlbumRecord>;
|
albums: Map<number, AlbumRecord>;
|
||||||
photos: Map<number, PhotoRecord>;
|
photos: Map<number, PhotoRecord>;
|
||||||
// The membership each photo's record is read from, for its `Photo`'s save
|
|
||||||
// path. It holds the file's key, so it stays in this process: no snapshot
|
|
||||||
// or change carries it.
|
|
||||||
files: Map<number, EnteFile>;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const asString = (v: unknown): string | undefined =>
|
const asString = (v: unknown): string | undefined =>
|
||||||
@@ -105,29 +95,10 @@ const microsToMillis = (micros: number): number => Math.floor(micros / 1000);
|
|||||||
const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number =>
|
const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number =>
|
||||||
b.takenAt - a.takenAt || b.fileID - a.fileID;
|
b.takenAt - a.takenAt || b.fileID - a.fileID;
|
||||||
|
|
||||||
// A photo's `takenAt` in milliseconds: `pubMagicMetadata.editedTime` when the
|
|
||||||
// user edited the date, else basic-metadata `creationTime`.
|
|
||||||
export const takenAtOf = (file: EnteFile): number =>
|
|
||||||
microsToMillis(
|
|
||||||
asNumber(file.pubMagicMetadata?.editedTime) ??
|
|
||||||
file.metadata.creationTime,
|
|
||||||
);
|
|
||||||
|
|
||||||
// The membership a file's record is read from: the most recently synced, lowest
|
|
||||||
// collection id to break ties. Whatever dates a file's save path takes this
|
|
||||||
// membership too, so the path always carries the record's `takenAt`.
|
|
||||||
export const representative = (memberships: EnteFile[]): EnteFile =>
|
|
||||||
memberships.reduce((best, m) =>
|
|
||||||
m.updationTime > best.updationTime ||
|
|
||||||
(m.updationTime === best.updationTime &&
|
|
||||||
m.collectionID < best.collectionID)
|
|
||||||
? m
|
|
||||||
: best,
|
|
||||||
);
|
|
||||||
|
|
||||||
// Build one PhotoRecord from every membership of a file. The memberships share
|
// Build one PhotoRecord from every membership of a file. The memberships share
|
||||||
// the same underlying file, so metadata is read from a single representative;
|
// the same underlying file, so metadata is read from a single representative
|
||||||
// `albumIDs` gathers them all.
|
// (the most recently synced, lowest collection id to break ties); `albumIDs`
|
||||||
|
// gathers them all.
|
||||||
const toPhotoRecord = (
|
const toPhotoRecord = (
|
||||||
fileID: number,
|
fileID: number,
|
||||||
memberships: EnteFile[],
|
memberships: EnteFile[],
|
||||||
@@ -135,19 +106,25 @@ const toPhotoRecord = (
|
|||||||
const albumIDs = memberships
|
const albumIDs = memberships
|
||||||
.map((m) => m.collectionID)
|
.map((m) => m.collectionID)
|
||||||
.sort((a, b) => a - b);
|
.sort((a, b) => a - b);
|
||||||
const rep = representative(memberships);
|
const rep = memberships.reduce((best, m) =>
|
||||||
|
m.updationTime > best.updationTime ||
|
||||||
|
(m.updationTime === best.updationTime &&
|
||||||
|
m.collectionID < best.collectionID)
|
||||||
|
? m
|
||||||
|
: best,
|
||||||
|
);
|
||||||
|
|
||||||
const pub = rep.pubMagicMetadata ?? {};
|
const pub = rep.pubMagicMetadata ?? {};
|
||||||
const priv = rep.magicMetadata ?? {};
|
const priv = rep.magicMetadata ?? {};
|
||||||
|
|
||||||
|
const takenAtMicros = asNumber(pub.editedTime) ?? rep.metadata.creationTime;
|
||||||
const visibility = asNumber(priv.visibility);
|
const visibility = asNumber(priv.visibility);
|
||||||
|
|
||||||
const record: PhotoRecord = {
|
const record: PhotoRecord = {
|
||||||
fileID,
|
fileID,
|
||||||
albumIDs,
|
albumIDs,
|
||||||
title: asString(pub.editedName) ?? rep.metadata.title,
|
title: asString(pub.editedName) ?? rep.metadata.title,
|
||||||
takenAt: takenAtOf(rep),
|
takenAt: microsToMillis(takenAtMicros),
|
||||||
modifiedAt: microsToMillis(rep.metadata.modificationTime),
|
|
||||||
fileType: rep.metadata.fileType,
|
fileType: rep.metadata.fileType,
|
||||||
isArchived: visibility === VISIBILITY_ARCHIVED,
|
isArchived: visibility === VISIBILITY_ARCHIVED,
|
||||||
isHidden: visibility === VISIBILITY_HIDDEN,
|
isHidden: visibility === VISIBILITY_HIDDEN,
|
||||||
@@ -163,7 +140,6 @@ const toPhotoRecord = (
|
|||||||
record.latitude = rep.metadata.latitude;
|
record.latitude = rep.metadata.latitude;
|
||||||
if (rep.metadata.longitude !== undefined)
|
if (rep.metadata.longitude !== undefined)
|
||||||
record.longitude = rep.metadata.longitude;
|
record.longitude = rep.metadata.longitude;
|
||||||
if (rep.metadata.hash !== undefined) record.hash = rep.metadata.hash;
|
|
||||||
|
|
||||||
return record;
|
return record;
|
||||||
};
|
};
|
||||||
@@ -214,7 +190,6 @@ export const deriveRecords = (
|
|||||||
}
|
}
|
||||||
|
|
||||||
const photos = new Map<number, PhotoRecord>();
|
const photos = new Map<number, PhotoRecord>();
|
||||||
const photoFiles = new Map<number, EnteFile>();
|
|
||||||
const takenAtByFile = new Map<number, number>();
|
const takenAtByFile = new Map<number, number>();
|
||||||
for (const [fileID, memberships] of byFileID) {
|
for (const [fileID, memberships] of byFileID) {
|
||||||
const record = toPhotoRecord(fileID, memberships);
|
const record = toPhotoRecord(fileID, memberships);
|
||||||
@@ -226,7 +201,6 @@ export const deriveRecords = (
|
|||||||
record.thumbnailPath = paths.thumbnailPath;
|
record.thumbnailPath = paths.thumbnailPath;
|
||||||
}
|
}
|
||||||
photos.set(fileID, record);
|
photos.set(fileID, record);
|
||||||
photoFiles.set(fileID, representative(memberships));
|
|
||||||
takenAtByFile.set(fileID, record.takenAt);
|
takenAtByFile.set(fileID, record.takenAt);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -235,7 +209,7 @@ export const deriveRecords = (
|
|||||||
albums.set(c.id, toAlbumRecord(c, files, takenAtByFile));
|
albums.set(c.id, toAlbumRecord(c, files, takenAtByFile));
|
||||||
}
|
}
|
||||||
|
|
||||||
return { albums, photos, files: photoFiles };
|
return { albums, photos };
|
||||||
};
|
};
|
||||||
|
|
||||||
// Sorted, GUI-ready arrays: albums newest updated first, photos newest first.
|
// Sorted, GUI-ready arrays: albums newest updated first, photos newest first.
|
||||||
|
|||||||
@@ -16,7 +16,6 @@ import { dirname } from "node:path";
|
|||||||
|
|
||||||
import { writeAtomic } from "../download/index.js";
|
import { writeAtomic } from "../download/index.js";
|
||||||
import type { Collection, EnteFile, Microseconds } from "../model/types.js";
|
import type { Collection, EnteFile, Microseconds } from "../model/types.js";
|
||||||
import { representative } from "./records.js";
|
|
||||||
|
|
||||||
// Bumped only when the on-disk shape changes incompatibly. A file written
|
// Bumped only when the on-disk shape changes incompatibly. A file written
|
||||||
// under a different version is discarded on load (see `load`): re-fetching
|
// under a different version is discarded on load (see `load`): re-fetching
|
||||||
@@ -180,16 +179,14 @@ export class MetadataStore {
|
|||||||
return this.files.get(fileKey(collectionID, fileID));
|
return this.files.get(fileKey(collectionID, fileID));
|
||||||
}
|
}
|
||||||
|
|
||||||
// The membership of a file its record is read from (`representative`), or
|
// Any membership of a file, or undefined. Every membership re-wraps the
|
||||||
// undefined. Any membership could fetch the bytes, but the content cache
|
// same underlying content key, so any one is enough to fetch the bytes;
|
||||||
// resolves a fileID to a file this way so that it dates the save path
|
// the content cache resolves a fileID to a file this way.
|
||||||
// from the same membership as `photo.savePath`.
|
|
||||||
getFileByID(fileID: number): EnteFile | undefined {
|
getFileByID(fileID: number): EnteFile | undefined {
|
||||||
const memberships: EnteFile[] = [];
|
|
||||||
for (const file of this.files.values()) {
|
for (const file of this.files.values()) {
|
||||||
if (file.id === fileID) memberships.push(file);
|
if (file.id === fileID) return file;
|
||||||
}
|
}
|
||||||
return memberships.length > 0 ? representative(memberships) : undefined;
|
return undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
listFiles(collectionID: number): EnteFile[] {
|
listFiles(collectionID: number): EnteFile[] {
|
||||||
|
|||||||
+67
-16
@@ -1,8 +1,8 @@
|
|||||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
import * as jpeg from "jpeg-js";
|
import * as jpeg from "jpeg-js";
|
||||||
|
import exifReader from "exif-reader";
|
||||||
import type { Client } from "./client.js";
|
import type { Client } from "./client.js";
|
||||||
import { readExifTags } from "./exif.js";
|
|
||||||
import type { Library, Photo } from "./library/index.js";
|
import type { Library, Photo } from "./library/index.js";
|
||||||
import { sanitizeFileName } from "./filename.js";
|
import { sanitizeFileName } from "./filename.js";
|
||||||
import {
|
import {
|
||||||
@@ -19,10 +19,63 @@ export interface MetadataBackupOptions {
|
|||||||
onProgress?: ProgressCallback;
|
onProgress?: ProgressCallback;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Extract dimensions, EXIF and XMP from a file's bytes. `exif` is the EXIF tags
|
// Find the raw EXIF APP1 segment in JPEG bytes. Returns `exif` (the segment
|
||||||
// exifreader returns, from any image format it reads. When it finds an EXIF
|
// data, starting at the "Exif\0\0" header) when there is one, nothing when the
|
||||||
// block but reads no tag from it, the record keeps the block's bytes, base64,
|
// bytes are not a JPEG or carry no EXIF, and `error` when the segment layout is
|
||||||
// in `exifRaw`, with the reason in `exifError`.
|
// malformed. Each segment length is checked against the bytes that remain and
|
||||||
|
// each step moves forward by at least 4 bytes, so the scan ends on any input.
|
||||||
|
export const extractExifFromJpeg = (
|
||||||
|
buf: Uint8Array,
|
||||||
|
): { exif?: Buffer; error?: string } => {
|
||||||
|
if (buf[0] !== 0xff || buf[1] !== 0xd8) return {};
|
||||||
|
let offset = 2;
|
||||||
|
while (offset < buf.length) {
|
||||||
|
if (offset + 2 > buf.length)
|
||||||
|
return { error: `truncated segment marker at byte ${offset}` };
|
||||||
|
if (buf[offset] !== 0xff)
|
||||||
|
return { error: `no segment marker at byte ${offset}` };
|
||||||
|
const marker = buf[offset + 1]!;
|
||||||
|
if (marker === 0xda) return {}; // start of scan, no more markers
|
||||||
|
if (offset + 4 > buf.length)
|
||||||
|
return { error: `truncated segment length at byte ${offset}` };
|
||||||
|
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
|
||||||
|
// The length counts its own two bytes, so anything under 2 is invalid.
|
||||||
|
if (len < 2)
|
||||||
|
return {
|
||||||
|
error: `segment length ${len} at byte ${offset} is too small`,
|
||||||
|
};
|
||||||
|
if (offset + 2 + len > buf.length)
|
||||||
|
return {
|
||||||
|
error: `segment length ${len} at byte ${offset} runs past the end of the file`,
|
||||||
|
};
|
||||||
|
if (marker === 0xe1) {
|
||||||
|
// APP1 — check for "Exif\0\0" header. A length under 8 cannot hold
|
||||||
|
// the six-byte header, so the segment is not EXIF; below 6 the
|
||||||
|
// bytes compared would also lie past the segment.
|
||||||
|
if (
|
||||||
|
len >= 8 &&
|
||||||
|
buf[offset + 4] === 0x45 &&
|
||||||
|
buf[offset + 5] === 0x78 &&
|
||||||
|
buf[offset + 6] === 0x69 &&
|
||||||
|
buf[offset + 7] === 0x66
|
||||||
|
) {
|
||||||
|
return {
|
||||||
|
exif: Buffer.from(
|
||||||
|
buf.buffer,
|
||||||
|
buf.byteOffset + offset + 4,
|
||||||
|
len - 2,
|
||||||
|
),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
offset += 2 + len;
|
||||||
|
}
|
||||||
|
return { error: "file ends before the image data" };
|
||||||
|
};
|
||||||
|
|
||||||
|
// Extract dimensions, EXIF and XMP from a file's bytes. When the EXIF segment
|
||||||
|
// is malformed or cannot be parsed, the record carries the reason in
|
||||||
|
// `exifError`.
|
||||||
export const extractImageMetadata = (
|
export const extractImageMetadata = (
|
||||||
fileBytes: Uint8Array,
|
fileBytes: Uint8Array,
|
||||||
): Record<string, unknown> | undefined => {
|
): Record<string, unknown> | undefined => {
|
||||||
@@ -39,21 +92,19 @@ export const extractImageMetadata = (
|
|||||||
result.height = decoded.height;
|
result.height = decoded.height;
|
||||||
} catch {
|
} catch {
|
||||||
// Not every original is a JPEG (PNG, HEIC, video), so a failed decode
|
// Not every original is a JPEG (PNG, HEIC, video), so a failed decode
|
||||||
// is expected and only means no dimensions; unreadable EXIF is still
|
// is expected and only means no dimensions; a malformed JPEG is still
|
||||||
// reported below through `exifError`.
|
// reported below through `exifError`.
|
||||||
}
|
}
|
||||||
|
|
||||||
const tags = readExifTags(fileBytes);
|
const { exif, error } = extractExifFromJpeg(fileBytes);
|
||||||
if (tags?.exif && Object.keys(tags.exif).length > 0) {
|
if (error) result.exifError = error;
|
||||||
result.exif = tags.exif;
|
if (exif) {
|
||||||
} else if (tags?.exif) {
|
try {
|
||||||
const block = tags.metadataRange?.blocks.find((b) => b.type === "exif");
|
result.exif = exifReader(exif);
|
||||||
if (block) {
|
} catch (err) {
|
||||||
result.exifRaw = Buffer.from(
|
result.exifRaw = exif.toString("base64");
|
||||||
fileBytes.subarray(block.start, block.end),
|
result.exifError = err instanceof Error ? err.message : String(err);
|
||||||
).toString("base64");
|
|
||||||
}
|
}
|
||||||
result.exifError = "no tag could be read from the EXIF block";
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
|
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
|
||||||
|
|||||||
@@ -37,16 +37,6 @@ export const DEFAULT_RETRY_OPTIONS: ResolvedRetryOptions = {
|
|||||||
random: Math.random,
|
random: Math.random,
|
||||||
};
|
};
|
||||||
|
|
||||||
// For a run nobody is watching, such as `quak backup` from cron. A request that
|
|
||||||
// keeps failing waits at most 243 s in all before it gives up, usually about
|
|
||||||
// half that, since each wait is drawn at random below its ceiling.
|
|
||||||
export const UNATTENDED_RETRY_OPTIONS: ResolvedRetryOptions = {
|
|
||||||
...DEFAULT_RETRY_OPTIONS,
|
|
||||||
attempts: 10,
|
|
||||||
baseDelayMs: 1_000,
|
|
||||||
maxDelayMs: 60_000,
|
|
||||||
};
|
|
||||||
|
|
||||||
export const resolveRetryOptions = (
|
export const resolveRetryOptions = (
|
||||||
opts?: RetryOptions,
|
opts?: RetryOptions,
|
||||||
): ResolvedRetryOptions => ({
|
): ResolvedRetryOptions => ({
|
||||||
|
|||||||
+147
-859
File diff suppressed because it is too large
Load Diff
@@ -1,71 +0,0 @@
|
|||||||
/**
|
|
||||||
* Tests for the retry options `bin/quak.ts` loads the saved session with:
|
|
||||||
* `quak backup` gets the unattended ones, every other command the default.
|
|
||||||
*
|
|
||||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
|
||||||
* replaced by one that records the options it is given and reports no session,
|
|
||||||
* so the command stops before it makes any request.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { mkdtempSync, rmSync } from "node:fs";
|
|
||||||
import { tmpdir } from "node:os";
|
|
||||||
import { join } from "node:path";
|
|
||||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
|
||||||
import type { ApiClientOptions } from "../../src/api/client.js";
|
|
||||||
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
|
|
||||||
|
|
||||||
const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
|
|
||||||
|
|
||||||
vi.mock("../../src/cli-session.js", () => ({
|
|
||||||
loadSession: (_path: string, apiOptions?: ApiClientOptions) => {
|
|
||||||
loaded.push(apiOptions);
|
|
||||||
return null;
|
|
||||||
},
|
|
||||||
}));
|
|
||||||
|
|
||||||
const argv = process.argv;
|
|
||||||
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
|
|
||||||
|
|
||||||
// Run `quak <args>` to completion. The "Not logged in" message and the exit
|
|
||||||
// are swallowed.
|
|
||||||
const quak = async (...args: string[]): Promise<void> => {
|
|
||||||
vi.spyOn(process.stderr, "write").mockImplementation(() => true);
|
|
||||||
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
|
|
||||||
process.argv = ["node", "quak", ...args];
|
|
||||||
vi.resetModules();
|
|
||||||
await import("../../bin/quak.js");
|
|
||||||
};
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
process.argv = argv;
|
|
||||||
loaded.length = 0;
|
|
||||||
vi.restoreAllMocks();
|
|
||||||
});
|
|
||||||
|
|
||||||
afterAll(() => {
|
|
||||||
rmSync(dir, { recursive: true, force: true });
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("bin/quak.ts session loading", () => {
|
|
||||||
it("loads the session for backup with the unattended retry options", async () => {
|
|
||||||
await quak("backup", dir);
|
|
||||||
// `vi.resetModules()` gave `bin/quak.ts` its own copy of
|
|
||||||
// `src/retry.ts`, whose `sleep` is a different function, so the
|
|
||||||
// options are compared by their numbers.
|
|
||||||
const { attempts, baseDelayMs, maxDelayMs } = UNATTENDED_RETRY_OPTIONS;
|
|
||||||
expect(loaded).toEqual([
|
|
||||||
{
|
|
||||||
retry: expect.objectContaining({
|
|
||||||
attempts,
|
|
||||||
baseDelayMs,
|
|
||||||
maxDelayMs,
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("loads the session for another command with the default options", async () => {
|
|
||||||
await quak("collections");
|
|
||||||
expect(loaded).toEqual([undefined]);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
+16
-197
@@ -18,7 +18,6 @@ import {
|
|||||||
readFileSync,
|
readFileSync,
|
||||||
rmSync,
|
rmSync,
|
||||||
statSync,
|
statSync,
|
||||||
utimesSync,
|
|
||||||
writeFileSync,
|
writeFileSync,
|
||||||
} from "node:fs";
|
} from "node:fs";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
@@ -53,14 +52,12 @@ import {
|
|||||||
import { run } from "../../src/cli-run.js";
|
import { run } from "../../src/cli-run.js";
|
||||||
import { loadSession } from "../../src/cli-session.js";
|
import { loadSession } from "../../src/cli-session.js";
|
||||||
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
|
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
|
||||||
import { savePath, type ContentSource } from "../../src/library/content.js";
|
import type { ContentSource } from "../../src/library/content.js";
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||||
import { defaultCacheDirectory } from "../../src/library/index.js";
|
import { defaultCacheDirectory } from "../../src/library/index.js";
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
|
||||||
import {
|
import {
|
||||||
asLivePhoto,
|
asLivePhoto,
|
||||||
blake2b,
|
|
||||||
cdnSource,
|
cdnSource,
|
||||||
IMAGE,
|
IMAGE,
|
||||||
livePhotoHash,
|
livePhotoHash,
|
||||||
@@ -232,30 +229,20 @@ describe("session file", () => {
|
|||||||
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
|
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("a missing session exits 3 with 'Not logged in' from every command that needs one", async () => {
|
it("a missing session exits 1 with 'Not logged in'", async () => {
|
||||||
const ctx = { ...context(), loadSession };
|
const ctx = { ...context(), loadSession };
|
||||||
const dir = join(root, "backup");
|
expect(await whoamiCommand(ctx)).toBe(1);
|
||||||
expect(await whoamiCommand(ctx)).toBe(3);
|
expect(stderr.text).toBe(
|
||||||
expect(await collectionsCommand(ctx, {})).toBe(3);
|
|
||||||
expect(await filesCommand(ctx, { collection: "1" })).toBe(3);
|
|
||||||
expect(await getCommand(ctx, "100", {})).toBe(3);
|
|
||||||
expect(await getThumbCommand(ctx, "100", {})).toBe(3);
|
|
||||||
expect(await backupMetadataCommand(ctx, dir, {})).toBe(3);
|
|
||||||
expect(await backupCommand(ctx, dir, {})).toBe(3);
|
|
||||||
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(3);
|
|
||||||
expect(await fixMissingThumbnailsCommand(ctx, {})).toBe(3);
|
|
||||||
const notLoggedIn =
|
|
||||||
`Not logged in. Run "quak login" first.\n` +
|
`Not logged in. Run "quak login" first.\n` +
|
||||||
`Session file: ${join(ctx.sessionDir, "session.json")}\n`;
|
`Session file: ${join(ctx.sessionDir, "session.json")}\n`,
|
||||||
expect(stderr.text).toBe(notLoggedIn.repeat(9));
|
);
|
||||||
expect(stdout.text).toBe("");
|
expect(stdout.text).toBe("");
|
||||||
expect(existsSync(dir)).toBe(false);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("a corrupt session exits 3 and says it is corrupt", async () => {
|
it("a corrupt session exits 1 and says it is corrupt", async () => {
|
||||||
const ctx = { ...context(), loadSession };
|
const ctx = { ...context(), loadSession };
|
||||||
saveSession(ctx.sessionDir, snapshot);
|
saveSession(ctx.sessionDir, snapshot);
|
||||||
expect(await collectionsCommand(ctx, {})).toBe(3);
|
expect(await collectionsCommand(ctx, {})).toBe(1);
|
||||||
expect(stderr.text).toContain("is corrupt");
|
expect(stderr.text).toContain("is corrupt");
|
||||||
expect(stderr.text).toContain(
|
expect(stderr.text).toContain(
|
||||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||||
@@ -666,31 +653,6 @@ describe("a live photo", () => {
|
|||||||
height: 4,
|
height: 4,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("backup-metadata --exif records the EXIF of a HEIC image", async () => {
|
|
||||||
const dir = join(root, "dump");
|
|
||||||
|
|
||||||
expect(
|
|
||||||
await backupMetadataCommand(
|
|
||||||
context(await livePhotoClient(HEIC_WITH_EXIF)),
|
|
||||||
dir,
|
|
||||||
{ exif: true },
|
|
||||||
),
|
|
||||||
).toBe(0);
|
|
||||||
|
|
||||||
const record = JSON.parse(
|
|
||||||
readFileSync(
|
|
||||||
join(dir, "collections", "1-Vacation", "300.json"),
|
|
||||||
"utf-8",
|
|
||||||
),
|
|
||||||
);
|
|
||||||
expect(record.imageMetadata.exifError).toBeUndefined();
|
|
||||||
expect(record.imageMetadata.exif).toMatchObject({
|
|
||||||
Make: { value: ["Canon"] },
|
|
||||||
Model: { value: ["EOS R5"] },
|
|
||||||
DateTimeOriginal: { value: ["2021:07:15 14:30:00"] },
|
|
||||||
});
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("backup", () => {
|
describe("backup", () => {
|
||||||
@@ -705,7 +667,6 @@ describe("backup", () => {
|
|||||||
" Failed: 0\n",
|
" Failed: 0\n",
|
||||||
);
|
);
|
||||||
expect(stdout.text).toBe("");
|
expect(stdout.text).toBe("");
|
||||||
expect(existsSync(join(dir, "backup.lock"))).toBe(false);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
// The backup opens its library with the precache off: it fetches the
|
// The backup opens its library with the precache off: it fetches the
|
||||||
@@ -745,83 +706,15 @@ describe("backup", () => {
|
|||||||
expect(stderr.text).toBe("Starting backup...\n");
|
expect(stderr.text).toBe("Starting backup...\n");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("--verify downloads again an original that does not match its hash, prints the counts, and exits 0", async () => {
|
it("exits 1 with the error on one line when the refresh fails", async () => {
|
||||||
// Each file records the hash of the original the fake writes for it.
|
|
||||||
const client = {
|
const client = {
|
||||||
...fakeClient(),
|
...fakeClient(),
|
||||||
filesSince: async (args: { collectionID: number }) => ({
|
collectionsSince: async () => {
|
||||||
files: (FILES[args.collectionID] ?? []).map((f) => ({
|
throw new Error("HTTP 401 from server");
|
||||||
...f,
|
|
||||||
metadata: {
|
|
||||||
...f.metadata,
|
|
||||||
hash: blake2b(Buffer.alloc(7, f.id & 0xff)),
|
|
||||||
},
|
|
||||||
})),
|
|
||||||
deleted: [],
|
|
||||||
cursor: 1,
|
|
||||||
}),
|
|
||||||
} as unknown as Client;
|
|
||||||
const ctx = context(client);
|
|
||||||
const dir = join(root, "backup");
|
|
||||||
expect(await backupCommand(ctx, dir, {})).toBe(0);
|
|
||||||
writeFileSync(savePath(dir, FILES[1]![0]!), "corrupt");
|
|
||||||
|
|
||||||
expect(await backupCommand(ctx, dir, { verify: true })).toBe(0);
|
|
||||||
|
|
||||||
expect(stderr.text).toContain(
|
|
||||||
"MISMATCH original beach.jpg (100): its bytes do not match its content hash\n",
|
|
||||||
);
|
|
||||||
expect(stderr.text).toContain(
|
|
||||||
" Downloaded: 1\n" +
|
|
||||||
" Skipped: 2\n" +
|
|
||||||
" Verified: 2\n" +
|
|
||||||
" Mismatched: 1\n" +
|
|
||||||
" Unchecked: 0\n" +
|
|
||||||
" Failed: 0\n",
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("--verify --json adds the verified, mismatched and unchecked counts", async () => {
|
|
||||||
const dir = join(root, "backup");
|
|
||||||
expect(await backupCommand(context(), dir, {})).toBe(0);
|
|
||||||
|
|
||||||
const code = await backupCommand(context(), dir, {
|
|
||||||
verify: true,
|
|
||||||
json: true,
|
|
||||||
});
|
|
||||||
|
|
||||||
expect(code).toBe(0);
|
|
||||||
expect(JSON.parse(stdout.text)).toMatchObject({
|
|
||||||
skipped: 3,
|
|
||||||
verified: 0,
|
|
||||||
mismatched: 0,
|
|
||||||
unchecked: 3,
|
|
||||||
failed: 0,
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
it("exits 1 and lists each file when the ML data fetch fails", async () => {
|
|
||||||
const client = {
|
|
||||||
...fakeClient(),
|
|
||||||
fetchMLData: async () => {
|
|
||||||
throw new Error("HTTP 503 from server");
|
|
||||||
},
|
},
|
||||||
} as unknown as Client;
|
} as unknown as Client;
|
||||||
expect(
|
const dir = join(root, "backup");
|
||||||
await backupCommand(context(client), join(root, "backup"), {}),
|
// Through `run`, as `bin/quak.ts` does, which prints a thrown error.
|
||||||
).toBe(1);
|
|
||||||
expect(stderr.text).toContain(" Failed: 3\n");
|
|
||||||
expect(stderr.text).toContain(
|
|
||||||
" [Vacation] beach.jpg (id 100): ML data: HTTP 503 from server\n",
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
// Runs `backup` through `run`, as `bin/quak.ts` does, which prints a thrown
|
|
||||||
// error; returns the exit code and what `run` printed.
|
|
||||||
const backupThroughRun = async (
|
|
||||||
ctx: CliContext,
|
|
||||||
dir: string,
|
|
||||||
): Promise<{ code: number; runText: string }> => {
|
|
||||||
const runStderr = new PassThrough();
|
const runStderr = new PassThrough();
|
||||||
let runText = "";
|
let runText = "";
|
||||||
runStderr.on("data", (chunk: Buffer) => {
|
runStderr.on("data", (chunk: Buffer) => {
|
||||||
@@ -829,90 +722,16 @@ describe("backup", () => {
|
|||||||
});
|
});
|
||||||
const code = await new Promise<number>((resolve) => {
|
const code = await new Promise<number>((resolve) => {
|
||||||
void run(
|
void run(
|
||||||
backupCommand(ctx, dir, {}),
|
backupCommand(context(client), dir, {}),
|
||||||
new PassThrough(),
|
new PassThrough(),
|
||||||
runStderr,
|
runStderr,
|
||||||
resolve,
|
resolve,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
return { code, runText };
|
|
||||||
};
|
|
||||||
|
|
||||||
it("exits 1 with the error on one line when the refresh fails", async () => {
|
|
||||||
const client = {
|
|
||||||
...fakeClient(),
|
|
||||||
collectionsSince: async () => {
|
|
||||||
throw new Error("HTTP 503 from server");
|
|
||||||
},
|
|
||||||
} as unknown as Client;
|
|
||||||
const dir = join(root, "backup");
|
|
||||||
const { code, runText } = await backupThroughRun(context(client), dir);
|
|
||||||
expect(code).toBe(1);
|
expect(code).toBe(1);
|
||||||
expect(runText).toBe("quak: HTTP 503 from server\n");
|
expect(runText).toBe("quak: HTTP 401 from server\n");
|
||||||
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
||||||
expect(readdirSync(dir)).toEqual([]);
|
expect(existsSync(join(dir, "originals"))).toBe(false);
|
||||||
});
|
|
||||||
|
|
||||||
// A real saved session, read back by `loadSession`, whose server answers
|
|
||||||
// every request with 401, as it does once it no longer accepts the token.
|
|
||||||
// The context's prompts throw, so a prompt would end the run with another
|
|
||||||
// line and exit 1.
|
|
||||||
it("exits 3 with one line saying to log in again when the server answers 401", async () => {
|
|
||||||
const key = toBase64(new Uint8Array(32));
|
|
||||||
saveSession(join(root, "session"), {
|
|
||||||
email: "cli@example.com",
|
|
||||||
userID: USER_ID,
|
|
||||||
token: "expired",
|
|
||||||
masterKey: key,
|
|
||||||
secretKey: key,
|
|
||||||
publicKey: key,
|
|
||||||
});
|
|
||||||
const unauthorized = async (): Promise<Response> =>
|
|
||||||
new Response(null, { status: 401 });
|
|
||||||
const ctx = {
|
|
||||||
...context(),
|
|
||||||
loadSession: (path: string) =>
|
|
||||||
loadSession(path, { fetch: unauthorized }),
|
|
||||||
};
|
|
||||||
const dir = join(root, "backup");
|
|
||||||
const { code, runText } = await backupThroughRun(ctx, dir);
|
|
||||||
expect(code).toBe(3);
|
|
||||||
expect(runText).toBe(
|
|
||||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
|
||||||
);
|
|
||||||
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
|
||||||
expect(readdirSync(dir)).toEqual([]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("exits 2 with one line naming the directory, sending no request, while another backup of it runs", async () => {
|
|
||||||
const dir = join(root, "backup");
|
|
||||||
// The lock another backup holds. Its modification time is set an hour
|
|
||||||
// ahead, so it stays current however long this test takes.
|
|
||||||
const lock = join(dir, "backup.lock");
|
|
||||||
mkdirSync(lock, { recursive: true });
|
|
||||||
const hourAhead = new Date(Date.now() + 3_600_000);
|
|
||||||
utimesSync(lock, hourAhead, hourAhead);
|
|
||||||
// A client whose refresh never finishes, so a run that started one
|
|
||||||
// before it exits would never return.
|
|
||||||
let requests = 0;
|
|
||||||
const never = (): Promise<never> => {
|
|
||||||
requests++;
|
|
||||||
return new Promise(() => {});
|
|
||||||
};
|
|
||||||
const client = {
|
|
||||||
...fakeClient(),
|
|
||||||
collectionsSince: never,
|
|
||||||
filesSince: never,
|
|
||||||
} as unknown as Client;
|
|
||||||
|
|
||||||
expect(await backupCommand(context(client), dir, {})).toBe(2);
|
|
||||||
expect(requests).toBe(0);
|
|
||||||
expect(stderr.text).toBe(
|
|
||||||
`Starting backup...\nquak: another backup of ${dir} is running\n`,
|
|
||||||
);
|
|
||||||
expect(readdirSync(dir)).toEqual(["backup.lock"]);
|
|
||||||
// Opening the library would have made its cache directory.
|
|
||||||
expect(existsSync(join(root, "cache"))).toBe(false);
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
+69
-301
@@ -1,28 +1,21 @@
|
|||||||
/**
|
/**
|
||||||
* Tests for reading EXIF (`src/exif.ts`) and the image metadata
|
* Tests for the JPEG EXIF scan behind `quak backup-metadata --exif`.
|
||||||
* `quak backup-metadata --exif` records.
|
|
||||||
*
|
*
|
||||||
* The originals come from users' libraries, so a truncated or corrupt file
|
* The originals come from users' libraries, so a truncated or corrupt JPEG
|
||||||
* must neither hang the read nor throw out of it: `readAllExifTags` gives `{}`,
|
* must neither hang the scan nor throw out of it, and a malformed file must be
|
||||||
* and `backup-metadata` tells an EXIF block it cannot read apart from a file
|
* told apart from one that simply has no EXIF: the record carries the reason in
|
||||||
* that simply has no EXIF, carrying the reason in `exifError`. Each JPEG below
|
* `exifError`. Each input below is a short hand-built byte array.
|
||||||
* is a short hand-built byte array; the HEIC is a real file.
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import {
|
import {
|
||||||
readAllExifTags,
|
extractExifFromJpeg,
|
||||||
readExifTags,
|
extractImageMetadata,
|
||||||
readPhotoExif,
|
} from "../../src/metadata-backup.js";
|
||||||
} from "../../src/exif.js";
|
|
||||||
import { extractImageMetadata } from "../../src/metadata-backup.js";
|
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
|
||||||
|
|
||||||
const SOI = [0xff, 0xd8]; // start of image
|
const SOI = [0xff, 0xd8]; // start of image
|
||||||
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan
|
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan, where the scan stops
|
||||||
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
|
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
|
||||||
const APP0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
|
|
||||||
const ZERO_LENGTH_APP0 = [0xff, 0xe0, 0x00, 0x00];
|
|
||||||
|
|
||||||
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
|
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
|
||||||
const TIFF_ORIENTATION_6 = [
|
const TIFF_ORIENTATION_6 = [
|
||||||
@@ -31,132 +24,6 @@ const TIFF_ORIENTATION_6 = [
|
|||||||
0x00, 0x00,
|
0x00, 0x00,
|
||||||
];
|
];
|
||||||
|
|
||||||
// A big-endian TIFF block holding Orientation 6 and DateTimeOriginal
|
|
||||||
// "0000:00:00 00:00:00", which a camera with an unset clock writes.
|
|
||||||
const TIFF_UNSET_DATE = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
|
||||||
// The first IFD, at 8: two entries, then no next IFD.
|
|
||||||
...[0x00, 0x02],
|
|
||||||
// Orientation (0x0112), SHORT, 6.
|
|
||||||
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
|
|
||||||
// The Exif IFD's offset (0x8769), LONG, 38.
|
|
||||||
...[0x87, 0x69, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x26],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
// The Exif IFD, at 38: one entry, then no next IFD.
|
|
||||||
...[0x00, 0x01],
|
|
||||||
// DateTimeOriginal (0x9003), 20 ASCII bytes at 56.
|
|
||||||
...[0x90, 0x03, 0x00, 0x02, 0x00, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00, 0x38],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
...new TextEncoder().encode("0000:00:00 00:00:00\0"), // at 56
|
|
||||||
];
|
|
||||||
|
|
||||||
// A big-endian TIFF block holding a GPSAltitude of 12.5 m and no
|
|
||||||
// GPSAltitudeRef.
|
|
||||||
const TIFF_ALTITUDE_WITHOUT_REF = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
|
||||||
// The first IFD, at 8: one entry, then no next IFD.
|
|
||||||
...[0x00, 0x01],
|
|
||||||
// The GPS IFD's offset (0x8825), LONG, 26.
|
|
||||||
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
// The GPS IFD, at 26: one entry, then no next IFD.
|
|
||||||
...[0x00, 0x01],
|
|
||||||
// GPSAltitude (0x0006), one RATIONAL at 44.
|
|
||||||
...[0x00, 0x06, 0x00, 0x05, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x2c],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
...[0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x02], // 25/2, at 44
|
|
||||||
];
|
|
||||||
|
|
||||||
// A big-endian TIFF block holding a GPSLatitude of 33° 30' 0" and a
|
|
||||||
// GPSLatitudeRef of "S".
|
|
||||||
const TIFF_SOUTHERN_LATITUDE = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
|
||||||
// The first IFD, at 8: one entry, then no next IFD.
|
|
||||||
...[0x00, 0x01],
|
|
||||||
// The GPS IFD's offset (0x8825), LONG, 26.
|
|
||||||
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
// The GPS IFD, at 26: two entries, then no next IFD.
|
|
||||||
...[0x00, 0x02],
|
|
||||||
// GPSLatitudeRef (0x0001), 2 ASCII bytes: "S".
|
|
||||||
...[0x00, 0x01, 0x00, 0x02, 0x00, 0x00, 0x00, 0x02, 0x53, 0x00, 0x00, 0x00],
|
|
||||||
// GPSLatitude (0x0002), three RATIONALs at 56.
|
|
||||||
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
...[0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, 0x01], // 33/1, at 56
|
|
||||||
...[0x00, 0x00, 0x00, 0x1e, 0x00, 0x00, 0x00, 0x01], // 30/1
|
|
||||||
...[0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01], // 0/1
|
|
||||||
];
|
|
||||||
|
|
||||||
// A big-endian TIFF block holding a GPSLatitude of 40° 26' 46" and a
|
|
||||||
// GPSLongitude of 79° 58' 56", and neither GPSLatitudeRef nor GPSLongitudeRef.
|
|
||||||
const TIFF_POSITION_WITHOUT_REFS = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
|
||||||
// The first IFD, at 8: one entry, then no next IFD.
|
|
||||||
...[0x00, 0x01],
|
|
||||||
// The GPS IFD's offset (0x8825), LONG, 26.
|
|
||||||
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
// The GPS IFD, at 26: two entries, then no next IFD.
|
|
||||||
...[0x00, 0x02],
|
|
||||||
// GPSLatitude (0x0002), three RATIONALs at 56.
|
|
||||||
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
|
|
||||||
// GPSLongitude (0x0004), three RATIONALs at 80.
|
|
||||||
...[0x00, 0x04, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x50],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
...[0x00, 0x00, 0x00, 0x28, 0x00, 0x00, 0x00, 0x01], // 40/1, at 56
|
|
||||||
...[0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x01], // 26/1
|
|
||||||
...[0x00, 0x00, 0x00, 0x2e, 0x00, 0x00, 0x00, 0x01], // 46/1
|
|
||||||
...[0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, 0x01], // 79/1, at 80
|
|
||||||
...[0x00, 0x00, 0x00, 0x3a, 0x00, 0x00, 0x00, 0x01], // 58/1
|
|
||||||
...[0x00, 0x00, 0x00, 0x38, 0x00, 0x00, 0x00, 0x01], // 56/1
|
|
||||||
];
|
|
||||||
|
|
||||||
// A big-endian TIFF block holding Orientation 6 and a Make whose value lies
|
|
||||||
// past the end of the file.
|
|
||||||
const TIFF_MAKE_PAST_END = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
|
||||||
// The first IFD, at 8: two entries, then no next IFD.
|
|
||||||
...[0x00, 0x02],
|
|
||||||
// Make (0x010f), 6 ASCII bytes at 4096.
|
|
||||||
...[0x01, 0x0f, 0x00, 0x02, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x10, 0x00],
|
|
||||||
// Orientation (0x0112), SHORT, 6.
|
|
||||||
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
];
|
|
||||||
|
|
||||||
// A big-endian TIFF block with one IFD entry that exifreader has no name for:
|
|
||||||
// tag 0xc000 (49152), SHORT, 7.
|
|
||||||
const TIFF_UNNAMED_TAG = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
|
||||||
// The first IFD, at 8: one entry, then no next IFD.
|
|
||||||
...[0x00, 0x01],
|
|
||||||
// Tag 0xc000, SHORT, 7.
|
|
||||||
...[0xc0, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x07, 0x00, 0x00],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
];
|
|
||||||
|
|
||||||
// A big-endian TIFF block holding Orientation 6, and a thumbnail IFD holding
|
|
||||||
// its own Orientation 1 and a 4-byte JPEG thumbnail.
|
|
||||||
const TIFF_WITH_THUMBNAIL = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
|
|
||||||
// The first IFD, at 8: one entry, then the thumbnail IFD at 26.
|
|
||||||
...[0x00, 0x01],
|
|
||||||
// Orientation (0x0112), SHORT, 6.
|
|
||||||
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
|
|
||||||
...[0x00, 0x00, 0x00, 0x1a],
|
|
||||||
// The thumbnail IFD, at 26: three entries, then no next IFD.
|
|
||||||
...[0x00, 0x03],
|
|
||||||
// Orientation (0x0112), SHORT, 1.
|
|
||||||
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00],
|
|
||||||
// JPEGInterchangeFormat (0x0201), LONG: the thumbnail is at 68.
|
|
||||||
...[0x02, 0x01, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x44],
|
|
||||||
// JPEGInterchangeFormatLength (0x0202), LONG: 4 bytes.
|
|
||||||
...[0x02, 0x02, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04],
|
|
||||||
...[0x00, 0x00, 0x00, 0x00],
|
|
||||||
...[0xff, 0xd8, 0xff, 0xd9], // the thumbnail, at 68: an empty JPEG
|
|
||||||
];
|
|
||||||
|
|
||||||
// An APP1 segment whose length field matches its data.
|
// An APP1 segment whose length field matches its data.
|
||||||
const app1 = (data: number[]): number[] => {
|
const app1 = (data: number[]): number[] => {
|
||||||
const len = data.length + 2;
|
const len = data.length + 2;
|
||||||
@@ -166,147 +33,74 @@ const app1 = (data: number[]): number[] => {
|
|||||||
const bytes = (...parts: number[][]): Uint8Array =>
|
const bytes = (...parts: number[][]): Uint8Array =>
|
||||||
new Uint8Array(parts.flat());
|
new Uint8Array(parts.flat());
|
||||||
|
|
||||||
describe("readAllExifTags", () => {
|
describe("extractExifFromJpeg", () => {
|
||||||
it("keys a tag exifreader has no name for by its number", () => {
|
it("returns the EXIF segment of a valid JPEG", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_UNNAMED_TAG];
|
|
||||||
expect(readAllExifTags(bytes(SOI, app1(data), SOS))).toStrictEqual({
|
|
||||||
"undefined-49152": {
|
|
||||||
id: 49152,
|
|
||||||
value: 7,
|
|
||||||
description: 7,
|
|
||||||
computed: 7,
|
|
||||||
},
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
it("puts the thumbnail's tags under Thumbnail, without its image", () => {
|
|
||||||
const input = bytes(
|
|
||||||
SOI,
|
|
||||||
app1([...EXIF_HEADER, ...TIFF_WITH_THUMBNAIL]),
|
|
||||||
SOS,
|
|
||||||
);
|
|
||||||
// exifreader finds the thumbnail's image.
|
|
||||||
expect(readExifTags(input)?.Thumbnail?.type).toBe("image/jpeg");
|
|
||||||
const tags = readAllExifTags(input);
|
|
||||||
expect(tags.Orientation?.value).toBe(6);
|
|
||||||
expect(Object.keys(tags.Thumbnail ?? {}).sort()).toEqual([
|
|
||||||
"JPEGInterchangeFormat",
|
|
||||||
"JPEGInterchangeFormatLength",
|
|
||||||
"Orientation",
|
|
||||||
]);
|
|
||||||
expect(tags.Thumbnail?.Orientation?.value).toBe(1);
|
|
||||||
expect(readPhotoExif(tags)).toStrictEqual({ orientation: 6 });
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("readPhotoExif", () => {
|
|
||||||
it("reads the common fields of a valid JPEG", () => {
|
|
||||||
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
|
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
|
||||||
expect(
|
const scan = extractExifFromJpeg(bytes(SOI, app1(data), SOS));
|
||||||
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
expect(scan.error).toBeUndefined();
|
||||||
).toStrictEqual({
|
expect([...scan.exif!]).toEqual(data);
|
||||||
orientation: 6,
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("gives no dateTimeOriginal for a DateTimeOriginal of 0000:00:00 00:00:00", () => {
|
it("returns nothing for a file that is not a JPEG", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_UNSET_DATE];
|
const png = bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||||
expect(
|
expect(extractExifFromJpeg(png)).toEqual({});
|
||||||
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
|
||||||
).toStrictEqual({
|
|
||||||
orientation: 6,
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("reads a GPSAltitude without GPSAltitudeRef as above sea level", () => {
|
it("returns nothing for a JPEG without EXIF", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_ALTITUDE_WITHOUT_REF];
|
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
|
||||||
expect(
|
expect(extractExifFromJpeg(bytes(SOI, app0, SOS))).toEqual({});
|
||||||
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
|
||||||
).toStrictEqual({
|
|
||||||
gpsAltitude: 12.5,
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("reads a GPSLatitude with GPSLatitudeRef S as south of the equator", () => {
|
it("ignores an APP1 segment too short to hold the Exif header", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_SOUTHERN_LATITUDE];
|
// A length under 8 cannot hold the six-byte "Exif\0\0" header, so the
|
||||||
expect(
|
// segment is not EXIF. This one has length 7 and holds only "Exif\0",
|
||||||
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
// which the old code, lacking the length check, returned as EXIF.
|
||||||
).toStrictEqual({
|
const short = app1(EXIF_HEADER.slice(0, 5));
|
||||||
gpsLatitude: -33.5,
|
expect(extractExifFromJpeg(bytes(SOI, short, SOS))).toEqual({});
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("gives no gpsLatitude or gpsLongitude without their reference tags", () => {
|
it("accepts an APP1 segment of length 8 holding just the Exif header", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_POSITION_WITHOUT_REFS];
|
const scan = extractExifFromJpeg(bytes(SOI, app1(EXIF_HEADER), SOS));
|
||||||
const tags = readAllExifTags(bytes(SOI, app1(data), SOS));
|
expect(scan.error).toBeUndefined();
|
||||||
// The position is read; only its hemisphere is unknown.
|
expect([...scan.exif!]).toEqual(EXIF_HEADER);
|
||||||
expect(tags.GPSLatitude?.computed).toStrictEqual([40, 26, 46]);
|
|
||||||
expect(tags.GPSLongitude?.computed).toStrictEqual([79, 58, 56]);
|
|
||||||
expect(readPhotoExif(tags)).toStrictEqual({});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("gives no make for a Make whose value lies past the end of the file", () => {
|
it("reports a JPEG truncated inside a segment header", () => {
|
||||||
const data = [...EXIF_HEADER, ...TIFF_MAKE_PAST_END];
|
const scan = extractExifFromJpeg(bytes(SOI, [0xff, 0xe1, 0x00]));
|
||||||
expect(
|
expect(scan.exif).toBeUndefined();
|
||||||
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
|
expect(scan.error).toMatch(/truncated segment length/);
|
||||||
).toStrictEqual({
|
|
||||||
orientation: 6,
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it.each([
|
it("reports a JPEG that ends before the image data", () => {
|
||||||
[
|
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
|
||||||
"a file that is not an image",
|
const scan = extractExifFromJpeg(bytes(SOI, app0));
|
||||||
new TextEncoder().encode("just some text, not an image"),
|
expect(scan.error).toMatch(/ends before the image data/);
|
||||||
],
|
});
|
||||||
[
|
|
||||||
"a PNG without EXIF",
|
it("stops on a zero-length segment instead of looping", () => {
|
||||||
bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
|
// A length of 0 would otherwise step the scan by 2 bytes at a time
|
||||||
],
|
// through the rest of the file, reading garbage as markers.
|
||||||
["a JPEG without EXIF", bytes(SOI, APP0, SOS)],
|
const zero = [0xff, 0xe0, 0x00, 0x00];
|
||||||
// A length under 8 cannot hold the six-byte "Exif\0\0" header. This one
|
const scan = extractExifFromJpeg(
|
||||||
// has length 7 and holds only "Exif\0", so a read past its end would
|
bytes(SOI, zero, zero, zero, zero, SOS),
|
||||||
// take the next segment's bytes as EXIF.
|
);
|
||||||
[
|
expect(scan.error).toMatch(/segment length 0 at byte 2 is too small/);
|
||||||
"an APP1 segment too short to hold the Exif header",
|
});
|
||||||
bytes(SOI, app1(EXIF_HEADER.slice(0, 5)), SOS),
|
|
||||||
],
|
it("stops on a segment length of 1", () => {
|
||||||
[
|
const scan = extractExifFromJpeg(
|
||||||
"an APP1 segment holding just the Exif header",
|
bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS),
|
||||||
bytes(SOI, app1(EXIF_HEADER), SOS),
|
);
|
||||||
],
|
expect(scan.error).toMatch(/segment length 1 at byte 2 is too small/);
|
||||||
[
|
});
|
||||||
"a JPEG truncated inside a segment header",
|
|
||||||
bytes(SOI, [0xff, 0xe1, 0x00]),
|
it("reports a segment length that runs past the end of the file", () => {
|
||||||
],
|
|
||||||
["a JPEG that ends before the image data", bytes(SOI, APP0)],
|
|
||||||
// A length of 0 would step a scan by 2 bytes at a time through the
|
|
||||||
// rest of the file, reading garbage as markers.
|
|
||||||
[
|
|
||||||
"a zero-length segment",
|
|
||||||
bytes(
|
|
||||||
SOI,
|
|
||||||
ZERO_LENGTH_APP0,
|
|
||||||
ZERO_LENGTH_APP0,
|
|
||||||
ZERO_LENGTH_APP0,
|
|
||||||
ZERO_LENGTH_APP0,
|
|
||||||
SOS,
|
|
||||||
),
|
|
||||||
],
|
|
||||||
["a segment length of 1", bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS)],
|
|
||||||
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
|
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
|
||||||
[
|
const scan = extractExifFromJpeg(
|
||||||
"a segment length that runs past the end of the file",
|
|
||||||
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
||||||
],
|
);
|
||||||
// "XX" where the TIFF byte order belongs.
|
expect(scan.exif).toBeUndefined();
|
||||||
[
|
expect(scan.error).toMatch(/runs past the end of the file/);
|
||||||
"an EXIF block that cannot be parsed",
|
|
||||||
bytes(SOI, app1([...EXIF_HEADER, 0x58, 0x58]), SOS),
|
|
||||||
],
|
|
||||||
])("returns no tags and no fields for %s", (_, input) => {
|
|
||||||
expect(readAllExifTags(input)).toStrictEqual({});
|
|
||||||
expect(readPhotoExif(readAllExifTags(input))).toStrictEqual({});
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -316,30 +110,10 @@ describe("extractImageMetadata", () => {
|
|||||||
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
|
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
|
||||||
);
|
);
|
||||||
expect(meta?.exifError).toBeUndefined();
|
expect(meta?.exifError).toBeUndefined();
|
||||||
expect(meta?.exif).toMatchObject({ Orientation: { value: 6 } });
|
expect(meta?.exif).toMatchObject({ Image: { Orientation: 6 } });
|
||||||
});
|
});
|
||||||
|
|
||||||
it("parses EXIF from a HEIC", () => {
|
it("returns nothing for a file that is not a JPEG", () => {
|
||||||
const meta = extractImageMetadata(HEIC_WITH_EXIF);
|
|
||||||
expect(meta?.exifError).toBeUndefined();
|
|
||||||
expect(meta?.exif).toMatchObject({
|
|
||||||
Make: { value: ["Canon"] },
|
|
||||||
Model: { value: ["EOS R5"] },
|
|
||||||
DateTimeOriginal: { value: ["2021:07:15 14:30:00"] },
|
|
||||||
Orientation: { value: 6 },
|
|
||||||
GPSLatitudeRef: { value: ["N"] },
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
it("keys a tag exifreader has no name for by its number", () => {
|
|
||||||
const meta = extractImageMetadata(
|
|
||||||
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_UNNAMED_TAG]), SOS),
|
|
||||||
);
|
|
||||||
expect(meta?.exifError).toBeUndefined();
|
|
||||||
expect(meta?.exif).toMatchObject({ "undefined-49152": { value: 7 } });
|
|
||||||
});
|
|
||||||
|
|
||||||
it("returns nothing for a file that is not an image", () => {
|
|
||||||
const text = new TextEncoder().encode("just some text, not an image");
|
const text = new TextEncoder().encode("just some text, not an image");
|
||||||
expect(extractImageMetadata(text)).toBeUndefined();
|
expect(extractImageMetadata(text)).toBeUndefined();
|
||||||
});
|
});
|
||||||
@@ -349,20 +123,14 @@ describe("extractImageMetadata", () => {
|
|||||||
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
||||||
);
|
);
|
||||||
expect(meta?.exif).toBeUndefined();
|
expect(meta?.exif).toBeUndefined();
|
||||||
expect(meta?.exifError).toBe(
|
expect(meta?.exifError).toMatch(/runs past the end of the file/);
|
||||||
"no tag could be read from the EXIF block",
|
|
||||||
);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
|
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
|
||||||
// The raw bytes are the whole EXIF block as exifreader finds it: for a
|
const data = [...EXIF_HEADER, 0x58, 0x58];
|
||||||
// JPEG, the APP1 segment, marker and length included.
|
const meta = extractImageMetadata(bytes(SOI, app1(data), SOS));
|
||||||
const segment = app1([...EXIF_HEADER, 0x58, 0x58]);
|
|
||||||
const meta = extractImageMetadata(bytes(SOI, segment, SOS));
|
|
||||||
expect(meta?.exif).toBeUndefined();
|
expect(meta?.exif).toBeUndefined();
|
||||||
expect(meta?.exifRaw).toBe(Buffer.from(segment).toString("base64"));
|
expect(meta?.exifRaw).toBe(Buffer.from(data).toString("base64"));
|
||||||
expect(meta?.exifError).toBe(
|
expect(meta?.exifError).toEqual(expect.any(String));
|
||||||
"no tag could be read from the EXIF block",
|
|
||||||
);
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -5,7 +5,6 @@
|
|||||||
import { PassThrough } from "node:stream";
|
import { PassThrough } from "node:stream";
|
||||||
import { describe, it, expect } from "vitest";
|
import { describe, it, expect } from "vitest";
|
||||||
|
|
||||||
import { ApiError } from "../../src/api/client.js";
|
|
||||||
import { run } from "../../src/cli-run.js";
|
import { run } from "../../src/cli-run.js";
|
||||||
|
|
||||||
// A stream whose written text is kept in `text`; writes finish at once, so
|
// A stream whose written text is kept in `text`; writes finish at once, so
|
||||||
@@ -56,26 +55,4 @@ describe("run", () => {
|
|||||||
stderr: "quak: offline\n",
|
stderr: "quak: offline\n",
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
|
|
||||||
const result = await runToExit(
|
|
||||||
Promise.reject(new ApiError("unauthorized", 401)),
|
|
||||||
);
|
|
||||||
expect(result).toEqual({
|
|
||||||
code: 3,
|
|
||||||
stdout: "",
|
|
||||||
stderr: `quak: the saved session is no longer valid; run "quak login"\n`,
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
it("prints another HTTP error as it is and exits 1", async () => {
|
|
||||||
const result = await runToExit(
|
|
||||||
Promise.reject(new ApiError("forbidden", 403)),
|
|
||||||
);
|
|
||||||
expect(result).toEqual({
|
|
||||||
code: 1,
|
|
||||||
stdout: "",
|
|
||||||
stderr: "quak: forbidden\n",
|
|
||||||
});
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -12,10 +12,6 @@ import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
|||||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||||
import { Client, type ClientSnapshot } from "../../src/client.js";
|
import { Client, type ClientSnapshot } from "../../src/client.js";
|
||||||
import { loadSession } from "../../src/cli-session.js";
|
import { loadSession } from "../../src/cli-session.js";
|
||||||
import {
|
|
||||||
DEFAULT_RETRY_OPTIONS,
|
|
||||||
UNATTENDED_RETRY_OPTIONS,
|
|
||||||
} from "../../src/retry.js";
|
|
||||||
|
|
||||||
const validSnapshot = (): ClientSnapshot => {
|
const validSnapshot = (): ClientSnapshot => {
|
||||||
const kp = sodium.crypto_box_keypair();
|
const kp = sodium.crypto_box_keypair();
|
||||||
@@ -180,20 +176,6 @@ describe("loadSession", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("gives the restored client the retry options it is passed", () => {
|
|
||||||
const path = join(dir, "retry.json");
|
|
||||||
writeFileSync(path, JSON.stringify(validSnapshot()));
|
|
||||||
const retryOptions = (client: Client | null) =>
|
|
||||||
client!.getApiClient().getRetryOptions();
|
|
||||||
|
|
||||||
expect(
|
|
||||||
retryOptions(
|
|
||||||
loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
|
|
||||||
),
|
|
||||||
).toEqual(UNATTENDED_RETRY_OPTIONS);
|
|
||||||
expect(retryOptions(loadSession(path))).toEqual(DEFAULT_RETRY_OPTIONS);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("says the file is corrupt when it is not JSON", () => {
|
it("says the file is corrupt when it is not JSON", () => {
|
||||||
const path = join(dir, "truncated.json");
|
const path = join(dir, "truncated.json");
|
||||||
writeFileSync(path, '{"email": "user@exa');
|
writeFileSync(path, '{"email": "user@exa');
|
||||||
|
|||||||
@@ -1883,6 +1883,15 @@ describe("downloadFile live photos", () => {
|
|||||||
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
|
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("replaces what was at the destination, such as an earlier ZIP of the two", async () => {
|
||||||
|
const t = setup(livePhotoZip(), livePhoto);
|
||||||
|
writeFileSync(t.outPath, livePhotoZip());
|
||||||
|
|
||||||
|
await t.run();
|
||||||
|
|
||||||
|
expect(readdirSync(t.dir).sort()).toEqual(["f.heic", "f.mov"]);
|
||||||
|
});
|
||||||
|
|
||||||
it("renames the image and then the video into place, each from its own temp file", async () => {
|
it("renames the image and then the video into place, each from its own temp file", async () => {
|
||||||
const t = setup(livePhotoZip(), livePhoto);
|
const t = setup(livePhotoZip(), livePhoto);
|
||||||
|
|
||||||
|
|||||||
@@ -1,306 +0,0 @@
|
|||||||
/**
|
|
||||||
* The example script `examples/download-albums.ts` (issue #144), run twice
|
|
||||||
* against a stand-in account, as a user would run it twice.
|
|
||||||
*
|
|
||||||
* The account has two albums sharing one photo, and one of its photos is a
|
|
||||||
* live photo whose image is a HEIC with EXIF. The first run puts every
|
|
||||||
* original at its save path, writes each photo's metadata beside it and each
|
|
||||||
* album's photos under `albums/`. Before the second run the account gains an
|
|
||||||
* album holding a new photo. The second run downloads that photo and writes
|
|
||||||
* its files and the new album's, and fetches nothing else and changes no
|
|
||||||
* other file.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
|
||||||
import {
|
|
||||||
existsSync,
|
|
||||||
mkdtempSync,
|
|
||||||
readdirSync,
|
|
||||||
readFileSync,
|
|
||||||
rmSync,
|
|
||||||
statSync,
|
|
||||||
utimesSync,
|
|
||||||
writeFileSync,
|
|
||||||
} from "node:fs";
|
|
||||||
import { tmpdir } from "node:os";
|
|
||||||
import { join } from "node:path";
|
|
||||||
|
|
||||||
import { downloadAlbums } from "../../examples/download-albums.js";
|
|
||||||
import { Library, type ContentSource } from "../../src/index.js";
|
|
||||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
|
||||||
import { readAllExifTags } from "../../src/exif.js";
|
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
|
||||||
import {
|
|
||||||
asLivePhoto,
|
|
||||||
cdnSource,
|
|
||||||
livePhotoHash,
|
|
||||||
livePhotoZip,
|
|
||||||
VIDEO,
|
|
||||||
} from "../live-photo.js";
|
|
||||||
|
|
||||||
const USER_ID = 7;
|
|
||||||
|
|
||||||
// Every photo is taken at noon local time on 2026-03-01, so the machine's time
|
|
||||||
// zone cannot move it to another day; it is saved in the folder `DAY`. Ente
|
|
||||||
// stores times in microseconds.
|
|
||||||
const TAKEN_MS = new Date(2026, 2, 1, 12).getTime();
|
|
||||||
const DAY = join("2026", "2026-03", "2026-03-01");
|
|
||||||
|
|
||||||
const collection = (id: number, name: string): Collection => ({
|
|
||||||
id,
|
|
||||||
ownerID: USER_ID,
|
|
||||||
key: new Uint8Array([id]),
|
|
||||||
name,
|
|
||||||
type: "album",
|
|
||||||
updationTime: 1,
|
|
||||||
isShared: false,
|
|
||||||
});
|
|
||||||
|
|
||||||
const file = (id: number, collectionID: number): EnteFile => ({
|
|
||||||
id,
|
|
||||||
collectionID,
|
|
||||||
ownerID: USER_ID,
|
|
||||||
key: new Uint8Array([id]),
|
|
||||||
metadata: {
|
|
||||||
title: `file-${id}.jpg`,
|
|
||||||
fileType: "image",
|
|
||||||
creationTime: TAKEN_MS * 1000,
|
|
||||||
modificationTime: TAKEN_MS * 1000,
|
|
||||||
},
|
|
||||||
file: { decryptionHeader: "aGVhZGVy" },
|
|
||||||
thumbnail: { decryptionHeader: "dGh1bWI=" },
|
|
||||||
updationTime: 1,
|
|
||||||
});
|
|
||||||
|
|
||||||
let root: string;
|
|
||||||
|
|
||||||
beforeEach(() => {
|
|
||||||
root = mkdtempSync(join(tmpdir(), "quak-download-albums-"));
|
|
||||||
});
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
if (root && existsSync(root))
|
|
||||||
rmSync(root, { recursive: true, force: true });
|
|
||||||
});
|
|
||||||
|
|
||||||
// Every file and directory under `dir`, by path.
|
|
||||||
const entries = (dir: string): string[] =>
|
|
||||||
readdirSync(dir, { recursive: true, encoding: "utf-8" });
|
|
||||||
|
|
||||||
// Set the modification time of everything under `dir` to the epoch, so that
|
|
||||||
// anything written there afterwards has a later one, however soon it comes.
|
|
||||||
const backdate = (dir: string): void => {
|
|
||||||
for (const name of entries(dir)) utimesSync(join(dir, name), 0, 0);
|
|
||||||
};
|
|
||||||
|
|
||||||
// The modification time of everything under `dir`, by path.
|
|
||||||
const mtimes = (dir: string): Map<string, number> =>
|
|
||||||
new Map(
|
|
||||||
entries(dir).map((name) => [name, statSync(join(dir, name)).mtimeMs]),
|
|
||||||
);
|
|
||||||
|
|
||||||
const readJSON = (path: string): unknown =>
|
|
||||||
JSON.parse(readFileSync(path, "utf-8"));
|
|
||||||
|
|
||||||
describe("examples/download-albums.ts", () => {
|
|
||||||
it("downloads every album's photos with their metadata, and on a second run only what the account gained", async () => {
|
|
||||||
// Album 1, "Trip", holds photos 1 and 2. Album 2, "Family", holds
|
|
||||||
// photo 2 and photo 3, a live photo.
|
|
||||||
const live = await asLivePhoto(
|
|
||||||
file(3, 2),
|
|
||||||
livePhotoZip({ "image.heic": HEIC_WITH_EXIF, "video.mov": VIDEO }),
|
|
||||||
livePhotoHash(HEIC_WITH_EXIF, VIDEO),
|
|
||||||
);
|
|
||||||
const collections = [collection(1, "Trip"), collection(2, "Family")];
|
|
||||||
const filesByAlbum = new Map<number, EnteFile[]>([
|
|
||||||
[1, [file(1, 1), file(2, 1)]],
|
|
||||||
[2, [file(2, 2), live.file]],
|
|
||||||
]);
|
|
||||||
const client = {
|
|
||||||
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
|
|
||||||
collectionsSince: async (): Promise<CollectionsPage> => ({
|
|
||||||
collections: [...collections],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 1,
|
|
||||||
}),
|
|
||||||
filesSince: async (args: {
|
|
||||||
collectionID: number;
|
|
||||||
}): Promise<FilesPage> => ({
|
|
||||||
files: filesByAlbum.get(args.collectionID) ?? [],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 1,
|
|
||||||
}),
|
|
||||||
};
|
|
||||||
|
|
||||||
// Photo 3 comes from a stand-in server, encrypted as Ente serves a
|
|
||||||
// live photo. Any other original is a few bytes naming its photo.
|
|
||||||
// `calls` counts every fetch, thumbnails included.
|
|
||||||
const server = cdnSource(new Map([[3, live.body]]));
|
|
||||||
let calls = 0;
|
|
||||||
const source: ContentSource = {
|
|
||||||
original: async (args) => {
|
|
||||||
calls++;
|
|
||||||
if (args.file.id === 3) return server.original(args);
|
|
||||||
const bytes = `original-${args.file.id}`;
|
|
||||||
writeFileSync(args.destination, bytes);
|
|
||||||
return { bytesWritten: bytes.length };
|
|
||||||
},
|
|
||||||
thumbnail: async (args) => {
|
|
||||||
calls++;
|
|
||||||
return server.thumbnail(args);
|
|
||||||
},
|
|
||||||
};
|
|
||||||
|
|
||||||
const dir = join(root, "photos");
|
|
||||||
const open = (): Promise<Library> =>
|
|
||||||
Library.open({
|
|
||||||
client,
|
|
||||||
cacheDirectory: join(root, "cache"),
|
|
||||||
downloadDirectory: dir,
|
|
||||||
contentSource: source,
|
|
||||||
refreshIntervalSeconds: 3600,
|
|
||||||
precacheThumbnails: false,
|
|
||||||
precacheOriginals: false,
|
|
||||||
// The test counts fetches, so the free space of the disk it
|
|
||||||
// runs on must not shrink the cache and evict photo 1.
|
|
||||||
freeBelowBytes: 0,
|
|
||||||
});
|
|
||||||
|
|
||||||
const first = await open();
|
|
||||||
// The cache already holds photo 1's original, so its record names a
|
|
||||||
// cache path, which the metadata leaves out. download() copies it
|
|
||||||
// from the cache rather than fetching it again.
|
|
||||||
await first.photos.byID({ fileID: 1 })!.original();
|
|
||||||
expect(await downloadAlbums(first, dir)).toEqual({
|
|
||||||
downloaded: 3,
|
|
||||||
alreadyLocal: 0,
|
|
||||||
});
|
|
||||||
await first.close();
|
|
||||||
expect(calls).toBe(3);
|
|
||||||
|
|
||||||
// Each original at its save path, a live photo as its image, its video
|
|
||||||
// and the file naming them, and each photo's metadata beside it.
|
|
||||||
const day = join(dir, DAY);
|
|
||||||
expect(readdirSync(day).sort()).toEqual([
|
|
||||||
"2026-03-01.1.jpg",
|
|
||||||
"2026-03-01.1.jpg.json",
|
|
||||||
"2026-03-01.2.jpg",
|
|
||||||
"2026-03-01.2.jpg.json",
|
|
||||||
"2026-03-01.3.heic",
|
|
||||||
"2026-03-01.3.heic.json",
|
|
||||||
"2026-03-01.3.livephoto.json",
|
|
||||||
"2026-03-01.3.mov",
|
|
||||||
]);
|
|
||||||
expect(readFileSync(join(day, "2026-03-01.1.jpg"), "utf-8")).toBe(
|
|
||||||
"original-1",
|
|
||||||
);
|
|
||||||
expect(readFileSync(join(day, "2026-03-01.2.jpg"), "utf-8")).toBe(
|
|
||||||
"original-2",
|
|
||||||
);
|
|
||||||
expect(readFileSync(join(day, "2026-03-01.3.heic"))).toEqual(
|
|
||||||
Buffer.from(HEIC_WITH_EXIF),
|
|
||||||
);
|
|
||||||
expect(readFileSync(join(day, "2026-03-01.3.mov"))).toEqual(
|
|
||||||
Buffer.from(VIDEO),
|
|
||||||
);
|
|
||||||
|
|
||||||
// The metadata is the photo's record and its EXIF tags. The originals
|
|
||||||
// of photos 1 and 2 are not image data, so they have no EXIF tags.
|
|
||||||
// Photo 3's are every tag of its image, as `photo.exif()` returns
|
|
||||||
// them. `record` holds the fields the three records share.
|
|
||||||
const record = {
|
|
||||||
takenAt: TAKEN_MS,
|
|
||||||
modifiedAt: TAKEN_MS,
|
|
||||||
fileType: "image",
|
|
||||||
isArchived: false,
|
|
||||||
isHidden: false,
|
|
||||||
};
|
|
||||||
expect(readJSON(join(day, "2026-03-01.1.jpg.json"))).toEqual({
|
|
||||||
...record,
|
|
||||||
fileID: 1,
|
|
||||||
albumIDs: [1],
|
|
||||||
title: "file-1.jpg",
|
|
||||||
exif: {},
|
|
||||||
});
|
|
||||||
expect(readJSON(join(day, "2026-03-01.2.jpg.json"))).toEqual({
|
|
||||||
...record,
|
|
||||||
fileID: 2,
|
|
||||||
albumIDs: [1, 2],
|
|
||||||
title: "file-2.jpg",
|
|
||||||
exif: {},
|
|
||||||
});
|
|
||||||
expect(readJSON(join(day, "2026-03-01.3.heic.json"))).toEqual({
|
|
||||||
...record,
|
|
||||||
fileID: 3,
|
|
||||||
albumIDs: [2],
|
|
||||||
title: "file-3.jpg",
|
|
||||||
fileType: "livePhoto",
|
|
||||||
hash: livePhotoHash(HEIC_WITH_EXIF, VIDEO),
|
|
||||||
exif: readAllExifTags(HEIC_WITH_EXIF),
|
|
||||||
});
|
|
||||||
|
|
||||||
// Each album's photos, newest first, by save path relative to `dir`.
|
|
||||||
// Photo 2 is in both.
|
|
||||||
expect(readdirSync(join(dir, "albums")).sort()).toEqual([
|
|
||||||
"1.json",
|
|
||||||
"2.json",
|
|
||||||
]);
|
|
||||||
expect(readJSON(join(dir, "albums", "1.json"))).toEqual({
|
|
||||||
collectionID: 1,
|
|
||||||
name: "Trip",
|
|
||||||
savePaths: [
|
|
||||||
join(DAY, "2026-03-01.2.jpg"),
|
|
||||||
join(DAY, "2026-03-01.1.jpg"),
|
|
||||||
],
|
|
||||||
});
|
|
||||||
expect(readJSON(join(dir, "albums", "2.json"))).toEqual({
|
|
||||||
collectionID: 2,
|
|
||||||
name: "Family",
|
|
||||||
savePaths: [
|
|
||||||
join(DAY, "2026-03-01.3.heic"),
|
|
||||||
join(DAY, "2026-03-01.2.jpg"),
|
|
||||||
],
|
|
||||||
});
|
|
||||||
|
|
||||||
// Before the second run the account gains album 3, "Garden", holding
|
|
||||||
// a new photo 4. The second run, with a newly opened library, opens
|
|
||||||
// the cache written by the first and still downloads photo 4. It finds
|
|
||||||
// the other photos already local and fetches nothing else.
|
|
||||||
collections.push(collection(3, "Garden"));
|
|
||||||
filesByAlbum.set(3, [file(4, 3)]);
|
|
||||||
backdate(dir);
|
|
||||||
const before = mtimes(dir);
|
|
||||||
const second = await open();
|
|
||||||
expect(await downloadAlbums(second, dir)).toEqual({
|
|
||||||
downloaded: 1,
|
|
||||||
alreadyLocal: 3,
|
|
||||||
});
|
|
||||||
await second.close();
|
|
||||||
expect(calls).toBe(4);
|
|
||||||
expect(readFileSync(join(day, "2026-03-01.4.jpg"), "utf-8")).toBe(
|
|
||||||
"original-4",
|
|
||||||
);
|
|
||||||
expect(readJSON(join(dir, "albums", "3.json"))).toEqual({
|
|
||||||
collectionID: 3,
|
|
||||||
name: "Garden",
|
|
||||||
savePaths: [join(DAY, "2026-03-01.4.jpg")],
|
|
||||||
});
|
|
||||||
|
|
||||||
// The second run adds only photo 4's files and album 3's, and
|
|
||||||
// rewrites, renames or removes no file from the first run. The two
|
|
||||||
// directories that gain a file are the only other changes.
|
|
||||||
const after = mtimes(dir);
|
|
||||||
expect(
|
|
||||||
[...after.keys()].filter((name) => !before.has(name)).sort(),
|
|
||||||
).toEqual([
|
|
||||||
join(DAY, "2026-03-01.4.jpg"),
|
|
||||||
join(DAY, "2026-03-01.4.jpg.json"),
|
|
||||||
join("albums", "3.json"),
|
|
||||||
]);
|
|
||||||
for (const [name, mtime] of before) {
|
|
||||||
if (name === DAY || name === "albums") continue;
|
|
||||||
expect(after.get(name), name).toBe(mtime);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
/**
|
|
||||||
* `exif.heic`, beside this file: a real 64x64 HEIC whose EXIF holds the same
|
|
||||||
* values as the hand-built JPEG in `exif-jpeg.ts`, for the tests of `exif()`,
|
|
||||||
* `backup-metadata --exif` and the image metadata `quak backup` records.
|
|
||||||
*
|
|
||||||
* It was made once, in a throwaway node:22-alpine container (Alpine 3.23.3),
|
|
||||||
* with libheif 1.23.0 and exiftool 13.55:
|
|
||||||
*
|
|
||||||
* apk add libheif-tools exiftool imagemagick
|
|
||||||
* magick -size 64x64 gradient:red-blue -depth 8 in.png
|
|
||||||
* heif-enc -q 30 -o exif.heic in.png
|
|
||||||
* exiftool -overwrite_original \
|
|
||||||
* -Make=Canon -Model="EOS R5" -LensModel="RF50mm F1.8 STM" \
|
|
||||||
* -DateTimeOriginal="2021:07:15 14:30:00" -OffsetTimeOriginal="+02:00" \
|
|
||||||
* -ExposureTime=1/250 -FNumber=2.8 -ISO=400 -FocalLength=50 \
|
|
||||||
* -Orientation#=6 \
|
|
||||||
* -GPSLatitude="40 26 46" -GPSLatitudeRef=N \
|
|
||||||
* -GPSLongitude="79 58 56" -GPSLongitudeRef=W \
|
|
||||||
* -GPSAltitude=12.5 -GPSAltitudeRef#=1 \
|
|
||||||
* exif.heic
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { readFileSync } from "node:fs";
|
|
||||||
|
|
||||||
export const HEIC_WITH_EXIF = new Uint8Array(
|
|
||||||
readFileSync(new URL("exif.heic", import.meta.url)),
|
|
||||||
);
|
|
||||||
@@ -1,89 +0,0 @@
|
|||||||
/**
|
|
||||||
* Hand-built JPEGs for the tests of `exif()` and of the image metadata
|
|
||||||
* `quak backup` records: one whose EXIF holds the same values as `exif.heic`
|
|
||||||
* (see `exif-heic.ts`), and one whose EXIF cannot be parsed.
|
|
||||||
*/
|
|
||||||
|
|
||||||
// Big-endian bytes for the hand-built JPEG below.
|
|
||||||
const u16 = (n: number): number[] => [n >> 8, n & 0xff];
|
|
||||||
const u32 = (n: number): number[] => [...u16(n >>> 16), ...u16(n & 0xffff)];
|
|
||||||
const ascii = (s: string): number[] => [...new TextEncoder().encode(s), 0];
|
|
||||||
const rational = (num: number, den: number): number[] => [
|
|
||||||
...u32(num),
|
|
||||||
...u32(den),
|
|
||||||
];
|
|
||||||
// One IFD entry: tag, type (1 BYTE, 2 ASCII, 3 SHORT, 4 LONG, 5 RATIONAL),
|
|
||||||
// count, then the value when it fits in 4 bytes, else its offset.
|
|
||||||
const entry = (
|
|
||||||
tag: number,
|
|
||||||
type: number,
|
|
||||||
count: number,
|
|
||||||
value: number[],
|
|
||||||
): number[] => [...u16(tag), ...u16(type), ...u32(count), ...value];
|
|
||||||
|
|
||||||
// The TIFF block of a JPEG's EXIF segment, holding every field `Photo`'s typed
|
|
||||||
// methods return: the camera in the first IFD, the exposure in the Exif IFD,
|
|
||||||
// and a GPS position of 40°26'46" N, 79°58'56" W, 12.5 m below sea level.
|
|
||||||
// Offsets count from the start of this block.
|
|
||||||
const TIFF = [
|
|
||||||
...[0x4d, 0x4d, 0x00, 0x2a], // big-endian TIFF
|
|
||||||
...u32(8), // the first IFD's offset
|
|
||||||
// The first IFD, at 8: five entries, then no next IFD.
|
|
||||||
...u16(5),
|
|
||||||
...entry(0x010f, 2, 6, u32(74)), // Make
|
|
||||||
...entry(0x0110, 2, 7, u32(80)), // Model
|
|
||||||
...entry(0x0112, 3, 1, [...u16(6), 0, 0]), // Orientation
|
|
||||||
...entry(0x8769, 4, 1, u32(88)), // the Exif IFD's offset
|
|
||||||
...entry(0x8825, 4, 1, u32(246)), // the GPS IFD's offset
|
|
||||||
...u32(0),
|
|
||||||
...ascii("Canon"), // at 74
|
|
||||||
...ascii("EOS R5"), // at 80
|
|
||||||
0, // a pad byte
|
|
||||||
// The Exif IFD, at 88: seven entries, then no next IFD.
|
|
||||||
...u16(7),
|
|
||||||
...entry(0x829a, 5, 1, u32(178)), // ExposureTime
|
|
||||||
...entry(0x829d, 5, 1, u32(186)), // FNumber
|
|
||||||
...entry(0x8827, 3, 1, [...u16(400), 0, 0]), // ISOSpeedRatings
|
|
||||||
...entry(0x9003, 2, 20, u32(194)), // DateTimeOriginal
|
|
||||||
...entry(0x9011, 2, 7, u32(214)), // OffsetTimeOriginal
|
|
||||||
...entry(0x920a, 5, 1, u32(222)), // FocalLength
|
|
||||||
...entry(0xa434, 2, 16, u32(230)), // LensModel
|
|
||||||
...u32(0),
|
|
||||||
...rational(1, 250), // at 178
|
|
||||||
...rational(28, 10), // at 186
|
|
||||||
...ascii("2021:07:15 14:30:00"), // at 194
|
|
||||||
...ascii("+02:00"), // at 214
|
|
||||||
0, // a pad byte
|
|
||||||
...rational(50, 1), // at 222
|
|
||||||
...ascii("RF50mm F1.8 STM"), // at 230
|
|
||||||
// The GPS IFD, at 246: six entries, then no next IFD.
|
|
||||||
...u16(6),
|
|
||||||
...entry(0x0001, 2, 2, [...ascii("N"), 0, 0]), // GPSLatitudeRef
|
|
||||||
...entry(0x0002, 5, 3, u32(324)), // GPSLatitude
|
|
||||||
...entry(0x0003, 2, 2, [...ascii("W"), 0, 0]), // GPSLongitudeRef
|
|
||||||
...entry(0x0004, 5, 3, u32(348)), // GPSLongitude
|
|
||||||
...entry(0x0005, 1, 1, [1, 0, 0, 0]), // GPSAltitudeRef: below sea level
|
|
||||||
...entry(0x0006, 5, 1, u32(372)), // GPSAltitude
|
|
||||||
...u32(0),
|
|
||||||
...[...rational(40, 1), ...rational(26, 1), ...rational(46, 1)], // at 324
|
|
||||||
...[...rational(79, 1), ...rational(58, 1), ...rational(56, 1)], // at 348
|
|
||||||
...rational(25, 2), // at 372
|
|
||||||
];
|
|
||||||
|
|
||||||
export const JPEG_WITH_EXIF = new Uint8Array([
|
|
||||||
...[0xff, 0xd8], // start of image
|
|
||||||
...[0xff, 0xe1, ...u16(2 + 6 + TIFF.length)], // APP1 and its length
|
|
||||||
...[...ascii("Exif"), 0], // "Exif\0\0"
|
|
||||||
...TIFF,
|
|
||||||
...[0xff, 0xda, 0x00, 0x02], // start of scan
|
|
||||||
]);
|
|
||||||
|
|
||||||
// A JPEG whose EXIF segment is laid out correctly but holds "XX" where the TIFF
|
|
||||||
// byte order belongs, so exifreader cannot parse it.
|
|
||||||
export const JPEG_WITH_BAD_EXIF = new Uint8Array([
|
|
||||||
...[0xff, 0xd8], // start of image
|
|
||||||
...[0xff, 0xe1, ...u16(2 + 6 + 2)], // APP1 and its length
|
|
||||||
...[...ascii("Exif"), 0], // "Exif\0\0"
|
|
||||||
...[0x58, 0x58], // "XX"
|
|
||||||
...[0xff, 0xda, 0x00, 0x02], // start of scan
|
|
||||||
]);
|
|
||||||
Binary file not shown.
@@ -142,7 +142,6 @@ const buildCache = (args: {
|
|||||||
pools: new RequestPools(),
|
pools: new RequestPools(),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
downloadDirectory: join(root, "photos"),
|
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
statfs: args.statfs,
|
statfs: args.statfs,
|
||||||
cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes,
|
cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes,
|
||||||
|
|||||||
@@ -6,18 +6,15 @@
|
|||||||
* `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and
|
* `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and
|
||||||
* a cached path shows up on the projected record. A library opened without a
|
* a cached path shows up on the projected record. A library opened without a
|
||||||
* content source leaves those methods throwing rather than silently doing
|
* content source leaves those methods throwing rather than silently doing
|
||||||
* nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `download()`,
|
* nothing.
|
||||||
* `content()`, `exif()` and the methods that each return one EXIF field.
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||||
import {
|
import {
|
||||||
mkdtempSync,
|
mkdtempSync,
|
||||||
readdirSync,
|
readdirSync,
|
||||||
readFileSync,
|
|
||||||
rmSync,
|
rmSync,
|
||||||
existsSync,
|
existsSync,
|
||||||
statSync,
|
|
||||||
writeFileSync,
|
writeFileSync,
|
||||||
} from "node:fs";
|
} from "node:fs";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
@@ -27,26 +24,10 @@ import { Library, type LibraryOptions } from "../../src/library/index.js";
|
|||||||
import type { ContentSource } from "../../src/library/content.js";
|
import type { ContentSource } from "../../src/library/content.js";
|
||||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
import { readPhotoExif, type PhotoExif } from "../../src/exif.js";
|
import { asLivePhoto, cdnSource, livePhotoZip } from "../live-photo.js";
|
||||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
|
||||||
import { JPEG_WITH_BAD_EXIF, JPEG_WITH_EXIF } from "../exif-jpeg.js";
|
|
||||||
import {
|
|
||||||
asLivePhoto,
|
|
||||||
cdnSource,
|
|
||||||
IMAGE,
|
|
||||||
livePhotoHash,
|
|
||||||
livePhotoZip,
|
|
||||||
VIDEO,
|
|
||||||
} from "../live-photo.js";
|
|
||||||
|
|
||||||
const USER_ID = 7;
|
const USER_ID = 7;
|
||||||
|
|
||||||
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
|
|
||||||
// stores times, so the machine's time zone cannot move it to another day; it
|
|
||||||
// is saved in the folder `DAY`.
|
|
||||||
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
|
|
||||||
const DAY = join("2026", "2026-03", "2026-03-01");
|
|
||||||
|
|
||||||
const collection = (id: number): Collection => ({
|
const collection = (id: number): Collection => ({
|
||||||
id,
|
id,
|
||||||
ownerID: USER_ID,
|
ownerID: USER_ID,
|
||||||
@@ -65,7 +46,7 @@ const file = (id: number, collectionID: number): EnteFile => ({
|
|||||||
metadata: {
|
metadata: {
|
||||||
title: `file-${id}.jpg`,
|
title: `file-${id}.jpg`,
|
||||||
fileType: "image",
|
fileType: "image",
|
||||||
creationTime: TAKEN,
|
creationTime: 1,
|
||||||
modificationTime: 1,
|
modificationTime: 1,
|
||||||
},
|
},
|
||||||
file: { decryptionHeader: "aGVhZGVy" },
|
file: { decryptionHeader: "aGVhZGVy" },
|
||||||
@@ -89,23 +70,14 @@ class MockClient {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A content source that writes `original` as every original and a marker file
|
// A content source that writes a marker file and counts thumbnail fetches.
|
||||||
// as every thumbnail, and counts the fetches of each.
|
const stubSource = (): ContentSource & { thumbCalls: () => number } => {
|
||||||
const stubSource = (
|
|
||||||
original: string | Uint8Array = "orig-bytes",
|
|
||||||
): ContentSource & {
|
|
||||||
originalCalls: () => number;
|
|
||||||
thumbCalls: () => number;
|
|
||||||
} => {
|
|
||||||
let originalCalls = 0;
|
|
||||||
let thumbCalls = 0;
|
let thumbCalls = 0;
|
||||||
return {
|
return {
|
||||||
originalCalls: () => originalCalls,
|
|
||||||
thumbCalls: () => thumbCalls,
|
thumbCalls: () => thumbCalls,
|
||||||
original: async ({ destination }) => {
|
original: async ({ destination }) => {
|
||||||
originalCalls++;
|
writeFileSync(destination, "orig-bytes");
|
||||||
writeFileSync(destination, original);
|
return { bytesWritten: 10 };
|
||||||
return { bytesWritten: original.length };
|
|
||||||
},
|
},
|
||||||
thumbnail: async ({ destination }) => {
|
thumbnail: async ({ destination }) => {
|
||||||
thumbCalls++;
|
thumbCalls++;
|
||||||
@@ -177,7 +149,7 @@ describe("Library content wiring", () => {
|
|||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not take a live photo's image or video with no JSON file as its original when it opens, and precaches both", async () => {
|
it("removes a live photo's ZIP an earlier version cached when it opens, and precaches its image and video", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(file(1, 1));
|
const { file: live, body } = await asLivePhoto(file(1, 1));
|
||||||
class LiveClient extends MockClient {
|
class LiveClient extends MockClient {
|
||||||
override async filesSince(): Promise<FilesPage> {
|
override async filesSince(): Promise<FilesPage> {
|
||||||
@@ -198,8 +170,7 @@ describe("Library content wiring", () => {
|
|||||||
// A first run records the library, so the next one knows that file 1
|
// A first run records the library, so the next one knows that file 1
|
||||||
// is a live photo when it opens the cache.
|
// is a live photo when it opens the cache.
|
||||||
await (await open({})).close();
|
await (await open({})).close();
|
||||||
writeFileSync(join(originals, "1.heic"), "an image");
|
writeFileSync(join(originals, "1.jpg"), livePhotoZip());
|
||||||
writeFileSync(join(originals, "1.mov"), "a video");
|
|
||||||
|
|
||||||
let precached!: () => void;
|
let precached!: () => void;
|
||||||
const done = new Promise<void>((r) => (precached = r));
|
const done = new Promise<void>((r) => (precached = r));
|
||||||
@@ -210,9 +181,7 @@ describe("Library content wiring", () => {
|
|||||||
precached();
|
precached();
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
expect(
|
expect(existsSync(join(originals, "1.jpg"))).toBe(false);
|
||||||
lib.photos.byID({ fileID: 1 })!.record().originalPath,
|
|
||||||
).toBeUndefined();
|
|
||||||
await done;
|
await done;
|
||||||
|
|
||||||
expect(readdirSync(originals).sort()).toEqual([
|
expect(readdirSync(originals).sort()).toEqual([
|
||||||
@@ -220,17 +189,6 @@ describe("Library content wiring", () => {
|
|||||||
"1.livephoto.json",
|
"1.livephoto.json",
|
||||||
"1.mov",
|
"1.mov",
|
||||||
]);
|
]);
|
||||||
expect(readFileSync(join(originals, "1.heic"))).toEqual(
|
|
||||||
Buffer.from(IMAGE),
|
|
||||||
);
|
|
||||||
expect(readFileSync(join(originals, "1.mov"))).toEqual(
|
|
||||||
Buffer.from(VIDEO),
|
|
||||||
);
|
|
||||||
expect(
|
|
||||||
JSON.parse(
|
|
||||||
readFileSync(join(originals, "1.livephoto.json"), "utf-8"),
|
|
||||||
),
|
|
||||||
).toEqual({ image: "1.heic", video: "1.mov" });
|
|
||||||
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
|
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
|
||||||
join(originals, "1.heic"),
|
join(originals, "1.heic"),
|
||||||
);
|
);
|
||||||
@@ -247,500 +205,9 @@ describe("Library content wiring", () => {
|
|||||||
await expect(
|
await expect(
|
||||||
lib.photos.byID({ fileID: 1 })!.thumbnail(),
|
lib.photos.byID({ fileID: 1 })!.thumbnail(),
|
||||||
).rejects.toThrow(/content cache/i);
|
).rejects.toThrow(/content cache/i);
|
||||||
await expect(
|
|
||||||
lib.photos.byID({ fileID: 1 })!.download(),
|
|
||||||
).rejects.toThrow(/content cache/i);
|
|
||||||
await expect(
|
await expect(
|
||||||
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
||||||
).rejects.toThrow(/content cache/i);
|
).rejects.toThrow(/content cache/i);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("Photo save path, local copy, content and EXIF", () => {
|
|
||||||
// The same account, with `files` in its album instead.
|
|
||||||
class FilesClient extends MockClient {
|
|
||||||
constructor(private readonly files: EnteFile[]) {
|
|
||||||
super();
|
|
||||||
}
|
|
||||||
override async filesSince(): Promise<FilesPage> {
|
|
||||||
return { files: this.files, deleted: [], cursor: 1 };
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const open = (opts: Partial<LibraryOptions> = {}): Promise<Library> =>
|
|
||||||
Library.open({
|
|
||||||
client: new MockClient(),
|
|
||||||
cacheDirectory: join(root, "cache"),
|
|
||||||
downloadDirectory: join(root, "backup"),
|
|
||||||
contentSource: stubSource(),
|
|
||||||
refreshIntervalSeconds: 3600,
|
|
||||||
precacheThumbnails: false,
|
|
||||||
precacheOriginals: false,
|
|
||||||
...opts,
|
|
||||||
});
|
|
||||||
|
|
||||||
it("names where a backup writes the original, which is local once the backup has written it", async () => {
|
|
||||||
const lib = await open();
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
const savePath = join(root, "backup", DAY, "2026-03-01.1.jpg");
|
|
||||||
expect(photo.savePath).toBe(savePath);
|
|
||||||
expect(photo.isLocal).toBe(false);
|
|
||||||
|
|
||||||
await lib.backup();
|
|
||||||
expect(photo.savePath).toBe(savePath);
|
|
||||||
expect(existsSync(savePath)).toBe(true);
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("returns the original's bytes, and a copy only in the cache is not local", async () => {
|
|
||||||
const lib = await open();
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
expect(await photo.content()).toEqual(Buffer.from("orig-bytes"));
|
|
||||||
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
|
|
||||||
true,
|
|
||||||
);
|
|
||||||
expect(existsSync(photo.savePath)).toBe(false);
|
|
||||||
expect(photo.isLocal).toBe(false);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("saves under photos/ in the working directory at open without a download directory", async () => {
|
|
||||||
const cwd = vi.spyOn(process, "cwd").mockReturnValue(root);
|
|
||||||
const lib = await open({ downloadDirectory: undefined });
|
|
||||||
cwd.mockRestore();
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
expect(lib.downloadDirectory).toBe(join(root, "photos"));
|
|
||||||
expect(photo.savePath).toBe(
|
|
||||||
join(root, "photos", DAY, "2026-03-01.1.jpg"),
|
|
||||||
);
|
|
||||||
expect(photo.isLocal).toBe(false);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("refuses an empty download directory", async () => {
|
|
||||||
await expect(open({ downloadDirectory: "" })).rejects.toThrow(
|
|
||||||
/downloadDirectory is empty/,
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("keeps a photo's save path after a refresh removes its file", async () => {
|
|
||||||
// The same account, whose album is deleted on the second refresh.
|
|
||||||
class AlbumDeletedClient extends MockClient {
|
|
||||||
override async collectionsSince(): Promise<CollectionsPage> {
|
|
||||||
if (!this.served) return super.collectionsSince();
|
|
||||||
return { collections: [], deleted: [1], cursor: 2 };
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const lib = await open({ client: new AlbumDeletedClient() });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
await photo.download();
|
|
||||||
|
|
||||||
await lib.fresh();
|
|
||||||
|
|
||||||
expect(lib.photos.byID({ fileID: 1 })).toBeUndefined();
|
|
||||||
expect(photo.savePath).toBe(
|
|
||||||
join(root, "backup", DAY, "2026-03-01.1.jpg"),
|
|
||||||
);
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("dates the save path and download() by the membership its takenAt comes from", async () => {
|
|
||||||
// One file in two albums. The date edited in Ente has reached album 2's
|
|
||||||
// copy, synced later, but album 1 still has an earlier edit.
|
|
||||||
const older = file(1, 1);
|
|
||||||
older.pubMagicMetadata = {
|
|
||||||
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
|
|
||||||
};
|
|
||||||
const newer = file(1, 2);
|
|
||||||
newer.updationTime = 2;
|
|
||||||
newer.pubMagicMetadata = {
|
|
||||||
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
|
|
||||||
};
|
|
||||||
const client = {
|
|
||||||
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
|
|
||||||
collectionsSince: async (): Promise<CollectionsPage> => ({
|
|
||||||
collections: [collection(1), collection(2)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 1,
|
|
||||||
}),
|
|
||||||
filesSince: async (args: {
|
|
||||||
collectionID: number;
|
|
||||||
}): Promise<FilesPage> => ({
|
|
||||||
files: [args.collectionID === 1 ? older : newer],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 1,
|
|
||||||
}),
|
|
||||||
};
|
|
||||||
const lib = await open({ client });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
|
|
||||||
expect(photo.takenAt).toBe(new Date(2026, 3, 15, 12).getTime());
|
|
||||||
expect(photo.savePath).toBe(
|
|
||||||
join(
|
|
||||||
root,
|
|
||||||
"backup",
|
|
||||||
"2026",
|
|
||||||
"2026-04",
|
|
||||||
"2026-04-15",
|
|
||||||
"2026-04-15.1.jpg",
|
|
||||||
),
|
|
||||||
);
|
|
||||||
|
|
||||||
// download() writes to that same path, so the photo is then local.
|
|
||||||
const saved = await photo.download();
|
|
||||||
expect(saved.path).toBe(photo.savePath);
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
expect(existsSync(join(root, "backup", "2026", "2026-02"))).toBe(false);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("downloads a photo held across a refresh that edits its date to the save path it names", async () => {
|
|
||||||
// The same account, whose second refresh brings a date edited in Ente.
|
|
||||||
const edited = file(1, 1);
|
|
||||||
edited.updationTime = 2;
|
|
||||||
edited.pubMagicMetadata = {
|
|
||||||
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
|
|
||||||
};
|
|
||||||
class DateEditedClient extends MockClient {
|
|
||||||
refreshes = 0;
|
|
||||||
override async collectionsSince(): Promise<CollectionsPage> {
|
|
||||||
this.refreshes++;
|
|
||||||
return {
|
|
||||||
collections: [
|
|
||||||
{ ...collection(1), updationTime: this.refreshes },
|
|
||||||
],
|
|
||||||
deleted: [],
|
|
||||||
cursor: this.refreshes,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
override async filesSince(): Promise<FilesPage> {
|
|
||||||
return {
|
|
||||||
files: [this.refreshes === 1 ? file(1, 1) : edited],
|
|
||||||
deleted: [],
|
|
||||||
cursor: this.refreshes,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const lib = await open({ client: new DateEditedClient() });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
|
|
||||||
await lib.fresh();
|
|
||||||
expect(lib.photos.byID({ fileID: 1 })!.takenAt).toBe(
|
|
||||||
new Date(2026, 3, 15, 12).getTime(),
|
|
||||||
);
|
|
||||||
|
|
||||||
const saved = await photo.download();
|
|
||||||
expect(saved.path).toBe(photo.savePath);
|
|
||||||
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("has a save path, and is not local, without a content source", async () => {
|
|
||||||
const lib = await open({ contentSource: undefined });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
expect(photo.savePath).toBe(
|
|
||||||
join(root, "backup", DAY, "2026-03-01.1.jpg"),
|
|
||||||
);
|
|
||||||
expect(photo.isLocal).toBe(false);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("gives a live photo's image as its save path once a backup has stored it", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(file(1, 1));
|
|
||||||
const lib = await open({
|
|
||||||
client: new FilesClient([live]),
|
|
||||||
contentSource: cdnSource(new Map([[1, body]])),
|
|
||||||
});
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
const day = join(root, "backup", DAY);
|
|
||||||
// Until then the name comes from the title, file-1.jpg; the backup
|
|
||||||
// stores the image with the extension found inside the live photo.
|
|
||||||
expect(photo.savePath).toBe(join(day, "2026-03-01.1.jpg"));
|
|
||||||
|
|
||||||
await lib.backup();
|
|
||||||
expect(photo.savePath).toBe(join(day, "2026-03-01.1.heic"));
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
expect(await photo.content()).toEqual(Buffer.from(IMAGE));
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("downloads an original the cache holds by copying it, without fetching it again", async () => {
|
|
||||||
const source = stubSource();
|
|
||||||
const lib = await open({ contentSource: source });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
await photo.original();
|
|
||||||
expect(source.originalCalls()).toBe(1);
|
|
||||||
expect(photo.isLocal).toBe(false);
|
|
||||||
|
|
||||||
const saved = await photo.download();
|
|
||||||
|
|
||||||
expect(source.originalCalls()).toBe(1);
|
|
||||||
expect(saved).toEqual({
|
|
||||||
path: photo.savePath,
|
|
||||||
bytes: "orig-bytes".length,
|
|
||||||
});
|
|
||||||
expect(readFileSync(photo.savePath, "utf-8")).toBe("orig-bytes");
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
// The cache keeps its own copy.
|
|
||||||
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
|
|
||||||
true,
|
|
||||||
);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("downloads an original the cache does not hold straight to its save path", async () => {
|
|
||||||
const source = stubSource();
|
|
||||||
const lib = await open({ contentSource: source });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
|
|
||||||
const saved = await photo.download();
|
|
||||||
|
|
||||||
expect(source.originalCalls()).toBe(1);
|
|
||||||
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
|
|
||||||
expect(readFileSync(saved.path, "utf-8")).toBe("orig-bytes");
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("does nothing when download() finds the original already at its save path", async () => {
|
|
||||||
const source = stubSource();
|
|
||||||
const lib = await open({ contentSource: source });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
const first = await photo.download();
|
|
||||||
const written = statSync(first.path).ino;
|
|
||||||
|
|
||||||
const second = await photo.download();
|
|
||||||
|
|
||||||
expect(second).toEqual(first);
|
|
||||||
expect(source.originalCalls()).toBe(1);
|
|
||||||
expect(statSync(second.path).ino).toBe(written);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("downloads a live photo the cache holds as its image, its video and the JSON file naming them", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(file(1, 1));
|
|
||||||
const lib = await open({
|
|
||||||
client: new FilesClient([live]),
|
|
||||||
contentSource: cdnSource(new Map([[1, body]])),
|
|
||||||
});
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
await photo.original();
|
|
||||||
const day = join(root, "backup", DAY);
|
|
||||||
|
|
||||||
const saved = await photo.download();
|
|
||||||
|
|
||||||
expect(saved).toEqual({
|
|
||||||
path: join(day, "2026-03-01.1.heic"),
|
|
||||||
videoPath: join(day, "2026-03-01.1.mov"),
|
|
||||||
bytes: IMAGE.length,
|
|
||||||
});
|
|
||||||
expect(readdirSync(day).sort()).toEqual([
|
|
||||||
"2026-03-01.1.heic",
|
|
||||||
"2026-03-01.1.livephoto.json",
|
|
||||||
"2026-03-01.1.mov",
|
|
||||||
]);
|
|
||||||
expect(
|
|
||||||
JSON.parse(
|
|
||||||
readFileSync(join(day, "2026-03-01.1.livephoto.json"), "utf-8"),
|
|
||||||
),
|
|
||||||
).toEqual({ image: "2026-03-01.1.heic", video: "2026-03-01.1.mov" });
|
|
||||||
expect(photo.savePath).toBe(saved.path);
|
|
||||||
expect(photo.isLocal).toBe(true);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
// Every tag in JPEG_WITH_EXIF, by name, in the order of its IFDs.
|
|
||||||
const jpegTags = [
|
|
||||||
"Make",
|
|
||||||
"Model",
|
|
||||||
"Orientation",
|
|
||||||
"Exif IFD Pointer",
|
|
||||||
"GPS Info IFD Pointer",
|
|
||||||
"ExposureTime",
|
|
||||||
"FNumber",
|
|
||||||
"ISOSpeedRatings",
|
|
||||||
"DateTimeOriginal",
|
|
||||||
"OffsetTimeOriginal",
|
|
||||||
"FocalLength",
|
|
||||||
"LensModel",
|
|
||||||
"GPSLatitudeRef",
|
|
||||||
"GPSLatitude",
|
|
||||||
"GPSLongitudeRef",
|
|
||||||
"GPSLongitude",
|
|
||||||
"GPSAltitudeRef",
|
|
||||||
"GPSAltitude",
|
|
||||||
];
|
|
||||||
// HEIC_WITH_EXIF holds those and the tags exiftool adds to every file.
|
|
||||||
const heicTags = [
|
|
||||||
...jpegTags,
|
|
||||||
"YCbCrPositioning",
|
|
||||||
"ExifVersion",
|
|
||||||
"ComponentsConfiguration",
|
|
||||||
"ColorSpace",
|
|
||||||
"GPSVersionID",
|
|
||||||
];
|
|
||||||
|
|
||||||
it.each([
|
|
||||||
[
|
|
||||||
"JPEG",
|
|
||||||
JPEG_WITH_EXIF,
|
|
||||||
jpegTags,
|
|
||||||
{
|
|
||||||
"Exif IFD Pointer": { value: 88 },
|
|
||||||
GPSLatitudeRef: { value: ["N"], description: "North latitude" },
|
|
||||||
},
|
|
||||||
],
|
|
||||||
[
|
|
||||||
"HEIC",
|
|
||||||
HEIC_WITH_EXIF,
|
|
||||||
heicTags,
|
|
||||||
{
|
|
||||||
ColorSpace: { value: 0xffff, description: "Uncalibrated" },
|
|
||||||
ExifVersion: { description: "0232" },
|
|
||||||
},
|
|
||||||
],
|
|
||||||
])(
|
|
||||||
"returns every EXIF tag of a %s original, by name",
|
|
||||||
async (_, bytes, names, others) => {
|
|
||||||
const lib = await open({ contentSource: stubSource(bytes) });
|
|
||||||
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
|
||||||
expect(Object.keys(exif).sort()).toEqual([...names].sort());
|
|
||||||
// Tags outside the thirteen fields, as exifreader decodes them.
|
|
||||||
expect(exif).toMatchObject(others);
|
|
||||||
await lib.close();
|
|
||||||
},
|
|
||||||
);
|
|
||||||
|
|
||||||
it("picks the common EXIF fields from a JPEG original's tags", async () => {
|
|
||||||
const lib = await open({ contentSource: stubSource(JPEG_WITH_EXIF) });
|
|
||||||
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
|
||||||
expect(readPhotoExif(exif)).toStrictEqual({
|
|
||||||
make: "Canon",
|
|
||||||
model: "EOS R5",
|
|
||||||
lensModel: "RF50mm F1.8 STM",
|
|
||||||
// The camera's clock reading, held in the Date's UTC fields.
|
|
||||||
dateTimeOriginal: new Date(Date.UTC(2021, 6, 15, 14, 30)),
|
|
||||||
offsetTimeOriginal: "+02:00",
|
|
||||||
exposureTime: 1 / 250,
|
|
||||||
fNumber: 2.8,
|
|
||||||
iso: 400,
|
|
||||||
focalLength: 50,
|
|
||||||
orientation: 6,
|
|
||||||
gpsLatitude: 40 + 26 / 60 + 46 / 3600,
|
|
||||||
gpsLongitude: -(79 + 58 / 60 + 56 / 3600),
|
|
||||||
gpsAltitude: -12.5,
|
|
||||||
});
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
// The fields picked from HEIC_WITH_EXIF's tags, and from JPEG_WITH_EXIF's,
|
|
||||||
// which hold the same values.
|
|
||||||
const heicFields: PhotoExif = {
|
|
||||||
make: "Canon",
|
|
||||||
model: "EOS R5",
|
|
||||||
lensModel: "RF50mm F1.8 STM",
|
|
||||||
dateTimeOriginal: new Date(Date.UTC(2021, 6, 15, 14, 30)),
|
|
||||||
offsetTimeOriginal: "+02:00",
|
|
||||||
exposureTime: 1 / 250,
|
|
||||||
fNumber: 2.8,
|
|
||||||
iso: 400,
|
|
||||||
focalLength: 50,
|
|
||||||
orientation: 6,
|
|
||||||
gpsLatitude: 40 + 26 / 60 + 46 / 3600,
|
|
||||||
gpsLongitude: -(79 + 58 / 60 + 56 / 3600),
|
|
||||||
gpsAltitude: -12.5,
|
|
||||||
};
|
|
||||||
|
|
||||||
it("picks the same common EXIF fields from a HEIC original's tags", async () => {
|
|
||||||
const lib = await open({ contentSource: stubSource(HEIC_WITH_EXIF) });
|
|
||||||
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
|
||||||
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("reads the EXIF of a live photo whose image is a HEIC", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(
|
|
||||||
file(1, 1),
|
|
||||||
livePhotoZip({ "image.heic": HEIC_WITH_EXIF, "video.mov": VIDEO }),
|
|
||||||
livePhotoHash(HEIC_WITH_EXIF, VIDEO),
|
|
||||||
);
|
|
||||||
const lib = await open({
|
|
||||||
client: new FilesClient([live]),
|
|
||||||
contentSource: cdnSource(new Map([[1, body]])),
|
|
||||||
});
|
|
||||||
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
|
|
||||||
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
// The build's type check, not this test, makes sure `Photo` has a method
|
|
||||||
// for every `PhotoExif` field, whatever the fixtures hold: `Photo`
|
|
||||||
// implements a type with one method per field. This test checks that each
|
|
||||||
// method gives the field picked from the tags exif() returns.
|
|
||||||
it.each([
|
|
||||||
["JPEG", JPEG_WITH_EXIF],
|
|
||||||
["HEIC", HEIC_WITH_EXIF],
|
|
||||||
])(
|
|
||||||
"has a method for each field, agreeing with the tags exif() returns, for a %s",
|
|
||||||
async (_, bytes) => {
|
|
||||||
const lib = await open({ contentSource: stubSource(bytes) });
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
const fields = readPhotoExif(await photo.exif());
|
|
||||||
// The file holds every field, so every method is checked.
|
|
||||||
expect(fields).toStrictEqual(heicFields);
|
|
||||||
for (const [field, value] of Object.entries(fields)) {
|
|
||||||
expect(await photo[field as keyof PhotoExif]()).toStrictEqual(
|
|
||||||
value,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
await lib.close();
|
|
||||||
},
|
|
||||||
);
|
|
||||||
|
|
||||||
it("returns no EXIF tags for an original that is not an image", async () => {
|
|
||||||
const lib = await open();
|
|
||||||
const photo = lib.photos.byID({ fileID: 1 })!;
|
|
||||||
expect(await photo.exif()).toStrictEqual({});
|
|
||||||
expect(await photo.dateTimeOriginal()).toBeUndefined();
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("returns no EXIF tags for a JPEG whose EXIF cannot be parsed", async () => {
|
|
||||||
const lib = await open({
|
|
||||||
contentSource: stubSource(JPEG_WITH_BAD_EXIF),
|
|
||||||
});
|
|
||||||
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({});
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("throws from exif() on a video without a content source, as the other content methods do", async () => {
|
|
||||||
const video = file(1, 1);
|
|
||||||
video.metadata.fileType = "video";
|
|
||||||
const lib = await open({
|
|
||||||
client: new FilesClient([video]),
|
|
||||||
contentSource: undefined,
|
|
||||||
});
|
|
||||||
await expect(lib.photos.byID({ fileID: 1 })!.exif()).rejects.toThrow(
|
|
||||||
/content cache/i,
|
|
||||||
);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("returns no EXIF tags for a video, without fetching it", async () => {
|
|
||||||
const video = file(1, 1);
|
|
||||||
video.metadata.fileType = "video";
|
|
||||||
const source = stubSource(JPEG_WITH_EXIF);
|
|
||||||
const lib = await open({
|
|
||||||
client: new FilesClient([video]),
|
|
||||||
contentSource: source,
|
|
||||||
});
|
|
||||||
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({});
|
|
||||||
expect(source.originalCalls()).toBe(0);
|
|
||||||
await lib.close();
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|||||||
@@ -7,8 +7,8 @@
|
|||||||
* 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail`
|
* 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail`
|
||||||
* fetches through the request pool and stores the bytes; the next finds the
|
* fetches through the request pool and stores the bytes; the next finds the
|
||||||
* file present and returns its path with a single `skipped` event and no
|
* file present and returns its path with a single `skipped` event and no
|
||||||
* network. A file already stored at its save path under the
|
* network. A file already sitting in the backup `downloadDirectory` counts
|
||||||
* `downloadDirectory` counts as present too.
|
* as present too.
|
||||||
* 2. **Present-means-complete.** Content appears only by the streaming atomic
|
* 2. **Present-means-complete.** Content appears only by the streaming atomic
|
||||||
* writer's rename, so a file that exists is whole. The directory listing
|
* writer's rename, so a file that exists is whole. The directory listing
|
||||||
* taken at `open()` is the record of what is cached, and the orphan temp
|
* taken at `open()` is the record of what is cached, and the orphan temp
|
||||||
@@ -41,7 +41,6 @@ import { join } from "node:path";
|
|||||||
|
|
||||||
import {
|
import {
|
||||||
ContentCache,
|
ContentCache,
|
||||||
savePath,
|
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
type EnsureEvent,
|
type EnsureEvent,
|
||||||
} from "../../src/library/content.js";
|
} from "../../src/library/content.js";
|
||||||
@@ -153,48 +152,12 @@ const buildCache = (
|
|||||||
pools: args.pools ?? new RequestPools(),
|
pools: args.pools ?? new RequestPools(),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
downloadDirectory: args.downloadDirectory ?? join(root, "photos"),
|
downloadDirectory: args.downloadDirectory,
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
});
|
});
|
||||||
return { cache, source };
|
return { cache, source };
|
||||||
};
|
};
|
||||||
|
|
||||||
// Microseconds, as Ente stores times, for noon local time on a day, so the
|
|
||||||
// machine's time zone cannot move the photo to another day.
|
|
||||||
const noon = (year: number, month: number, day: number): number =>
|
|
||||||
new Date(year, month - 1, day, 12).getTime() * 1000;
|
|
||||||
|
|
||||||
describe("savePath", () => {
|
|
||||||
it("files an original by year, month and day under the root", () => {
|
|
||||||
const f = file(12345, "IMG_0001.HEIC");
|
|
||||||
f.metadata.creationTime = noon(2026, 3, 1);
|
|
||||||
|
|
||||||
expect(savePath("/photos", f)).toBe(
|
|
||||||
"/photos/2026/2026-03/2026-03-01/2026-03-01.12345.HEIC",
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("dates an original by the date the user set, when there is one", () => {
|
|
||||||
const f = file(7, "a.jpg");
|
|
||||||
f.metadata.creationTime = noon(2026, 3, 1);
|
|
||||||
f.pubMagicMetadata = { editedTime: noon(1999, 12, 31) };
|
|
||||||
|
|
||||||
expect(savePath("/photos", f)).toBe(
|
|
||||||
"/photos/1999/1999-12/1999-12-31/1999-12-31.7.jpg",
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("takes the extension from the title it was uploaded with, not a new name", () => {
|
|
||||||
const f = file(8, "upload");
|
|
||||||
f.metadata.creationTime = noon(2026, 3, 1);
|
|
||||||
f.pubMagicMetadata = { editedName: "renamed.png" };
|
|
||||||
|
|
||||||
expect(savePath("/photos", f)).toBe(
|
|
||||||
"/photos/2026/2026-03/2026-03-01/2026-03-01.8.bin",
|
|
||||||
);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("ContentCache.open", () => {
|
describe("ContentCache.open", () => {
|
||||||
it("creates the cache directories with 0700 permissions", async () => {
|
it("creates the cache directories with 0700 permissions", async () => {
|
||||||
const { cache } = buildCache();
|
const { cache } = buildCache();
|
||||||
@@ -311,17 +274,11 @@ describe("ContentCache.original / thumbnail", () => {
|
|||||||
|
|
||||||
it("serves a file already present in the download directory without fetching", async () => {
|
it("serves a file already present in the download directory without fetching", async () => {
|
||||||
const downloadDirectory = join(root, "backup");
|
const downloadDirectory = join(root, "backup");
|
||||||
const day = join(downloadDirectory, "2026", "2026-03", "2026-03-01");
|
mkdirSync(join(downloadDirectory, "originals"), { recursive: true });
|
||||||
mkdirSync(day, { recursive: true });
|
const backupPath = join(downloadDirectory, "originals", "1.jpg");
|
||||||
const backupPath = join(day, "2026-03-01.1.jpg");
|
|
||||||
writeFileSync(backupPath, "from-backup");
|
writeFileSync(backupPath, "from-backup");
|
||||||
const f = file(1);
|
|
||||||
f.metadata.creationTime = noon(2026, 3, 1);
|
|
||||||
|
|
||||||
const { cache, source } = buildCache({
|
const { cache, source } = buildCache({ downloadDirectory });
|
||||||
downloadDirectory,
|
|
||||||
files: [f],
|
|
||||||
});
|
|
||||||
await cache.open();
|
await cache.open();
|
||||||
|
|
||||||
const events: EnsureEvent["status"][] = [];
|
const events: EnsureEvent["status"][] = [];
|
||||||
@@ -566,6 +523,43 @@ describe("ContentCache live photos", () => {
|
|||||||
expect(events).toEqual(["skipped"]);
|
expect(events).toEqual(["skipped"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("replaces a live photo an earlier version stored as a ZIP under the image's name", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
|
mkdirSync(originals(), { recursive: true });
|
||||||
|
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
|
||||||
|
const cache = cacheOf([live], new Map([[5, body]]));
|
||||||
|
// Opened without being told that file 5 is a live photo, the cache
|
||||||
|
// records the ZIP, and does not serve it.
|
||||||
|
await cache.open();
|
||||||
|
|
||||||
|
const result = await cache.original(5);
|
||||||
|
|
||||||
|
expect(result.videoPath).toBe(join(originals(), "5.mov"));
|
||||||
|
expect(readdirSync(originals()).sort()).toEqual([
|
||||||
|
"5.heic",
|
||||||
|
"5.livephoto.json",
|
||||||
|
"5.mov",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("removes a live photo's ZIP an earlier version stored when it opens, so the precache fetches the image and video", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
|
mkdirSync(originals(), { recursive: true });
|
||||||
|
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
|
||||||
|
const cache = cacheOf([live], new Map([[5, body]]));
|
||||||
|
|
||||||
|
await cache.open((fileID) => fileID === 5);
|
||||||
|
|
||||||
|
expect(readdirSync(originals())).toEqual([]);
|
||||||
|
expect(cache.pathsFor(5)).toEqual({});
|
||||||
|
const [fetched] = await cache.ensureOriginals({ fileIDs: [5] });
|
||||||
|
expect(fetched).toEqual({
|
||||||
|
fileID: 5,
|
||||||
|
path: join(originals(), "5.heic"),
|
||||||
|
});
|
||||||
|
expect(cache.pathsFor(5)).toEqual({ originalPath: fetched!.path });
|
||||||
|
});
|
||||||
|
|
||||||
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
|
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
const server = cdnSource(new Map([[5, body]]));
|
const server = cdnSource(new Map([[5, body]]));
|
||||||
@@ -596,36 +590,6 @@ describe("ContentCache live photos", () => {
|
|||||||
expect(second!.pathsFor(5)).toEqual({});
|
expect(second!.pathsFor(5)).toEqual({});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("fetches a live photo's image and video again when the cache opened before knowing it is a live photo and no JSON file names them", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
|
||||||
mkdirSync(originals(), { recursive: true });
|
|
||||||
writeFileSync(join(originals(), "5.heic"), "an image");
|
|
||||||
writeFileSync(join(originals(), "5.mov"), "a video");
|
|
||||||
const cache = cacheOf([live], new Map([[5, body]]));
|
|
||||||
// Opened without being told that file 5 is a live photo, the cache
|
|
||||||
// records one of the two files as its original, with no video.
|
|
||||||
await cache.open();
|
|
||||||
|
|
||||||
const events: string[] = [];
|
|
||||||
const result = await cache.original(5, {
|
|
||||||
onProgress: (e) => events.push(e.status),
|
|
||||||
});
|
|
||||||
|
|
||||||
expect(events.at(-1)).toBe("done");
|
|
||||||
expect(result).toEqual({
|
|
||||||
path: join(originals(), "5.heic"),
|
|
||||||
videoPath: join(originals(), "5.mov"),
|
|
||||||
bytes: IMAGE.length,
|
|
||||||
});
|
|
||||||
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
|
|
||||||
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
|
|
||||||
expect(
|
|
||||||
JSON.parse(
|
|
||||||
readFileSync(join(originals(), "5.livephoto.json"), "utf-8"),
|
|
||||||
),
|
|
||||||
).toEqual({ image: "5.heic", video: "5.mov" });
|
|
||||||
});
|
|
||||||
|
|
||||||
it.each(["missing", "empty"])(
|
it.each(["missing", "empty"])(
|
||||||
"fetches a live photo again when the video its JSON file names is %s",
|
"fetches a live photo again when the video its JSON file names is %s",
|
||||||
async (state) => {
|
async (state) => {
|
||||||
@@ -685,7 +649,6 @@ describe("ContentCache live photos", () => {
|
|||||||
]),
|
]),
|
||||||
),
|
),
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
downloadDirectory: join(root, "photos"),
|
|
||||||
getFile: (id) => [a.file, b.file].find((f) => f.id === id),
|
getFile: (id) => [a.file, b.file].find((f) => f.id === id),
|
||||||
// Room for one live photo, on a disk with plenty free.
|
// Room for one live photo, on a disk with plenty free.
|
||||||
cacheOriginalsMaxBytes: size,
|
cacheOriginalsMaxBytes: size,
|
||||||
|
|||||||
@@ -280,7 +280,6 @@ describe("Precache eviction integration", () => {
|
|||||||
pools: new RequestPools(),
|
pools: new RequestPools(),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: cacheDir,
|
cacheDirectory: cacheDir,
|
||||||
downloadDirectory: join(root, "photos"),
|
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
statfs,
|
statfs,
|
||||||
cacheOriginalsMaxBytes: 25, // holds two 10-byte originals
|
cacheOriginalsMaxBytes: 25, // holds two 10-byte originals
|
||||||
@@ -349,7 +348,6 @@ describe("Precache preemption", () => {
|
|||||||
pools: new RequestPools({ contentConcurrency: 1 }),
|
pools: new RequestPools({ contentConcurrency: 1 }),
|
||||||
source,
|
source,
|
||||||
cacheDirectory: join(root, "cache"),
|
cacheDirectory: join(root, "cache"),
|
||||||
downloadDirectory: join(root, "photos"),
|
|
||||||
getFile: (id) => byID.get(id),
|
getFile: (id) => byID.get(id),
|
||||||
statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }),
|
statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }),
|
||||||
freeBelowBytes: 0,
|
freeBelowBytes: 0,
|
||||||
|
|||||||
@@ -36,7 +36,6 @@ import {
|
|||||||
makeAlbumsAPI,
|
makeAlbumsAPI,
|
||||||
makePhotosAPI,
|
makePhotosAPI,
|
||||||
makeTimelineAPI,
|
makeTimelineAPI,
|
||||||
type SavePathLookup,
|
|
||||||
type TimelineGroup,
|
type TimelineGroup,
|
||||||
} from "../../src/library/read.js";
|
} from "../../src/library/read.js";
|
||||||
import { Library } from "../../src/library/index.js";
|
import { Library } from "../../src/library/index.js";
|
||||||
@@ -102,19 +101,12 @@ const file = (
|
|||||||
};
|
};
|
||||||
|
|
||||||
// Build the three API objects over one fixed projection, the way `Library`
|
// Build the three API objects over one fixed projection, the way `Library`
|
||||||
// wires them over its live store. Save paths are covered in
|
// wires them over its live store.
|
||||||
// content-library.test.ts; these tests never ask for one.
|
|
||||||
const apis = (records: DerivedRecords) => {
|
const apis = (records: DerivedRecords) => {
|
||||||
const derive = () => records;
|
const derive = () => records;
|
||||||
const saves: SavePathLookup = {
|
|
||||||
savePath: () => {
|
|
||||||
throw new Error("no save paths in these tests");
|
|
||||||
},
|
|
||||||
isLocal: () => false,
|
|
||||||
};
|
|
||||||
return {
|
return {
|
||||||
albums: makeAlbumsAPI(derive, saves),
|
albums: makeAlbumsAPI(derive),
|
||||||
photos: makePhotosAPI(derive, saves),
|
photos: makePhotosAPI(derive),
|
||||||
timeline: makeTimelineAPI(derive),
|
timeline: makeTimelineAPI(derive),
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
@@ -217,29 +209,6 @@ describe("lib.photos", () => {
|
|||||||
expect("key" in photo.record()).toBe(false);
|
expect("key" in photo.record()).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("byID exposes modifiedAt, hash and the year taken", () => {
|
|
||||||
// Mid-July, so the year is 2021 in every time zone.
|
|
||||||
const takenAt = Date.UTC(2021, 6, 15, 12);
|
|
||||||
const records = deriveRecords(
|
|
||||||
[collection(1)],
|
|
||||||
[
|
|
||||||
file(1001, 1, {
|
|
||||||
metadata: {
|
|
||||||
title: "IMG.jpg",
|
|
||||||
fileType: "image",
|
|
||||||
creationTime: micros(takenAt),
|
|
||||||
modificationTime: micros(1_700_000_123_456),
|
|
||||||
hash: "aGFzaA==",
|
|
||||||
},
|
|
||||||
}),
|
|
||||||
],
|
|
||||||
);
|
|
||||||
const photo = apis(records).photos.byID({ fileID: 1001 })!;
|
|
||||||
expect(photo.modifiedAt).toBe(ms(1_700_000_123_456));
|
|
||||||
expect(photo.hash).toBe("aGFzaA==");
|
|
||||||
expect(photo.year).toBe(2021);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("byID returns undefined for an unknown file id", () => {
|
it("byID returns undefined for an unknown file id", () => {
|
||||||
const records = deriveRecords([collection(1)], [file(1, 1)]);
|
const records = deriveRecords([collection(1)], [file(1, 1)]);
|
||||||
expect(apis(records).photos.byID({ fileID: 999 })).toBeUndefined();
|
expect(apis(records).photos.byID({ fileID: 999 })).toBeUndefined();
|
||||||
|
|||||||
@@ -159,7 +159,6 @@ describe("deriveRecords: photo mapping", () => {
|
|||||||
expect("width" in rec).toBe(false);
|
expect("width" in rec).toBe(false);
|
||||||
expect("height" in rec).toBe(false);
|
expect("height" in rec).toBe(false);
|
||||||
expect("latitude" in rec).toBe(false);
|
expect("latitude" in rec).toBe(false);
|
||||||
expect("hash" in rec).toBe(false);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("reads archived and hidden from private magicMetadata.visibility", () => {
|
it("reads archived and hidden from private magicMetadata.visibility", () => {
|
||||||
|
|||||||
+1
-2
@@ -30,8 +30,7 @@ export const livePhotoZip = (
|
|||||||
},
|
},
|
||||||
): Uint8Array => zipSync(entries);
|
): Uint8Array => zipSync(entries);
|
||||||
|
|
||||||
// The content hash Ente's clients record for an original's bytes.
|
const blake2b = (bytes: Uint8Array): string =>
|
||||||
export const blake2b = (bytes: Uint8Array): string =>
|
|
||||||
createHash("blake2b512").update(bytes).digest("base64");
|
createHash("blake2b512").update(bytes).digest("base64");
|
||||||
|
|
||||||
// The hash Ente's clients record for a live photo: the unkeyed BLAKE2b-512 of
|
// The hash Ente's clients record for a live photo: the unkeyed BLAKE2b-512 of
|
||||||
|
|||||||
@@ -9,10 +9,8 @@
|
|||||||
// Excluding too much: Prettier 3 reads `.gitignore` as a default ignore file,
|
// Excluding too much: Prettier 3 reads `.gitignore` as a default ignore file,
|
||||||
// so dropping it from the context silently changes which files the lint
|
// so dropping it from the context silently changes which files the lint
|
||||||
// phase's prettier check looks at compared to `make fmt-check` on the host.
|
// phase's prettier check looks at compared to `make fmt-check` on the host.
|
||||||
// And without `.git`, a `docker build .` given no `VERSION` build arg cannot
|
|
||||||
// derive the version (`script/version`) and stamps `package.json`'s instead.
|
|
||||||
//
|
//
|
||||||
// None of these shows up as a build failure, so they are asserted here.
|
// Neither shows up as a build failure, so they are asserted here.
|
||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import { existsSync, readFileSync } from "node:fs";
|
import { existsSync, readFileSync } from "node:fs";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
@@ -29,19 +27,18 @@ const patterns = (name: string): string[] =>
|
|||||||
const dockerignore = patterns(".dockerignore");
|
const dockerignore = patterns(".dockerignore");
|
||||||
|
|
||||||
describe(".dockerignore", () => {
|
describe(".dockerignore", () => {
|
||||||
// Everything here is either generated, enormous, or secret. `.claude` is
|
// Everything here is either generated, enormous, or secret. `.claude/` is
|
||||||
// the correctness one: see the header comment and issue #25. The leading
|
// the correctness one: see the header comment and issue #25.
|
||||||
// `/` anchors an entry at the root of the context.
|
|
||||||
it.each([
|
it.each([
|
||||||
".claude",
|
".claude/",
|
||||||
"/.quak",
|
".quak/",
|
||||||
"/bin/quak",
|
"bin/quak",
|
||||||
"**/node_modules",
|
"node_modules",
|
||||||
"/coverage",
|
"coverage",
|
||||||
"/dist",
|
"dist",
|
||||||
"/.vitest-cache",
|
".vitest-cache/",
|
||||||
"/.nyc_output",
|
".nyc_output/",
|
||||||
"/*.tsbuildinfo",
|
"*.tsbuildinfo",
|
||||||
])("keeps %s out of the build context", (pattern) => {
|
])("keeps %s out of the build context", (pattern) => {
|
||||||
expect(dockerignore).toContain(pattern);
|
expect(dockerignore).toContain(pattern);
|
||||||
});
|
});
|
||||||
@@ -50,17 +47,6 @@ describe(".dockerignore", () => {
|
|||||||
expect(dockerignore).not.toContain(".gitignore");
|
expect(dockerignore).not.toContain(".gitignore");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("leaves .git in the build context for the version", () => {
|
|
||||||
expect(dockerignore).not.toContain(".git");
|
|
||||||
expect(dockerignore).not.toContain(".git/");
|
|
||||||
});
|
|
||||||
|
|
||||||
// The build stage is the final image, so a .git/config sent in would
|
|
||||||
// ship the clone's remote URL and any credential in it.
|
|
||||||
it("sends .git without its config", () => {
|
|
||||||
expect(dockerignore).toContain("**/.git/config");
|
|
||||||
});
|
|
||||||
|
|
||||||
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
|
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
|
||||||
// file would silently give the build a different, unreviewed context —
|
// file would silently give the build a different, unreviewed context —
|
||||||
// and eslint's flat config does not ignore dot-directories, so a stray
|
// and eslint's flat config does not ignore dot-directories, so a stray
|
||||||
|
|||||||
@@ -1,137 +0,0 @@
|
|||||||
// `script/version` prints the version `script/build` stamps into
|
|
||||||
// `dist/package.json`, which is what `quak --version` reports from a build.
|
|
||||||
// A `docker build .` of a clone is given no `VERSION` build arg, so the
|
|
||||||
// version has to come from the `.git` in its context: the tag on a tagged
|
|
||||||
// commit; the tag, the commits since it and the short commit on a later commit;
|
|
||||||
// the short commit when no tag is reachable. A checkout with `.git` that still
|
|
||||||
// yields no usable version must fail the build, not ship a version nobody can
|
|
||||||
// trace back to its commit.
|
|
||||||
//
|
|
||||||
// Each test copies the script into a fresh directory, which the script then
|
|
||||||
// treats as the checkout, and executes it there.
|
|
||||||
import { afterEach, describe, expect, it } from "vitest";
|
|
||||||
import { execFileSync, spawnSync } from "node:child_process";
|
|
||||||
import {
|
|
||||||
chmodSync,
|
|
||||||
copyFileSync,
|
|
||||||
mkdirSync,
|
|
||||||
mkdtempSync,
|
|
||||||
rmSync,
|
|
||||||
writeFileSync,
|
|
||||||
} from "node:fs";
|
|
||||||
import { tmpdir } from "node:os";
|
|
||||||
import { join } from "node:path";
|
|
||||||
import { fileURLToPath } from "node:url";
|
|
||||||
|
|
||||||
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
|
|
||||||
|
|
||||||
let checkout = "";
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
rmSync(checkout, { recursive: true, force: true });
|
|
||||||
});
|
|
||||||
|
|
||||||
// A checkout holding the script and a package.json that declares 0.0.0, with
|
|
||||||
// no .git yet.
|
|
||||||
const makeCheckout = (): void => {
|
|
||||||
checkout = mkdtempSync(join(tmpdir(), "quak-version-"));
|
|
||||||
mkdirSync(join(checkout, "script"));
|
|
||||||
copyFileSync(
|
|
||||||
join(repoRoot, "script/version"),
|
|
||||||
join(checkout, "script/version"),
|
|
||||||
);
|
|
||||||
chmodSync(join(checkout, "script/version"), 0o755);
|
|
||||||
writeFileSync(join(checkout, "package.json"), '{ "version": "0.0.0" }\n');
|
|
||||||
};
|
|
||||||
|
|
||||||
// git in the checkout, with an identity and no commit signing, whatever the
|
|
||||||
// host's own git config says.
|
|
||||||
const git = (...args: string[]): string =>
|
|
||||||
execFileSync(
|
|
||||||
"git",
|
|
||||||
[
|
|
||||||
"-c",
|
|
||||||
"user.name=quak",
|
|
||||||
"-c",
|
|
||||||
"user.email=quak@example.invalid",
|
|
||||||
"-c",
|
|
||||||
"commit.gpgsign=false",
|
|
||||||
...args,
|
|
||||||
],
|
|
||||||
{ cwd: checkout, encoding: "utf-8", stdio: ["ignore", "pipe", "pipe"] },
|
|
||||||
).trim();
|
|
||||||
|
|
||||||
const makeCommittedCheckout = (): void => {
|
|
||||||
makeCheckout();
|
|
||||||
git("init", "-q");
|
|
||||||
git("add", "package.json");
|
|
||||||
git("commit", "-q", "-m", "first");
|
|
||||||
};
|
|
||||||
|
|
||||||
// Runs the script with nothing in its environment but PATH and, when given,
|
|
||||||
// VERSION.
|
|
||||||
const runVersion = (version?: string) =>
|
|
||||||
spawnSync(join(checkout, "script/version"), {
|
|
||||||
cwd: checkout,
|
|
||||||
encoding: "utf-8",
|
|
||||||
env: { PATH: process.env.PATH, VERSION: version },
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("script/version", () => {
|
|
||||||
it("prints the short commit of an untagged commit", () => {
|
|
||||||
makeCommittedCheckout();
|
|
||||||
expect(runVersion().stdout.trim()).toBe(
|
|
||||||
git("rev-parse", "--short", "HEAD"),
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("prints the tag of a tagged commit", () => {
|
|
||||||
makeCommittedCheckout();
|
|
||||||
git("tag", "v1.2.3");
|
|
||||||
expect(runVersion().stdout.trim()).toBe("v1.2.3");
|
|
||||||
});
|
|
||||||
|
|
||||||
// script/docker and script/cibuild pass the version they resolve on the
|
|
||||||
// host as the VERSION build arg.
|
|
||||||
it("prints the VERSION it is given over what git would derive", () => {
|
|
||||||
makeCommittedCheckout();
|
|
||||||
expect(runVersion("x").stdout.trim()).toBe("x");
|
|
||||||
});
|
|
||||||
|
|
||||||
// `--build-arg VERSION=` must not stamp an empty version.
|
|
||||||
it("treats an empty VERSION as unset", () => {
|
|
||||||
makeCommittedCheckout();
|
|
||||||
expect(runVersion("").stdout.trim()).toBe(
|
|
||||||
git("rev-parse", "--short", "HEAD"),
|
|
||||||
);
|
|
||||||
});
|
|
||||||
|
|
||||||
// A source tarball has no .git: it keeps the version package.json
|
|
||||||
// declares, and must still build.
|
|
||||||
it("prints package.json's version where there is no .git", () => {
|
|
||||||
makeCheckout();
|
|
||||||
const result = runVersion();
|
|
||||||
expect(result.status).toBe(0);
|
|
||||||
expect(result.stdout.trim()).toBe("0.0.0");
|
|
||||||
});
|
|
||||||
|
|
||||||
// A repository with no commits stands in for any .git that git cannot
|
|
||||||
// describe: git missing from the image, or refusing to read the checkout.
|
|
||||||
it("fails where .git yields no version", () => {
|
|
||||||
makeCheckout();
|
|
||||||
git("init", "-q");
|
|
||||||
const result = runVersion();
|
|
||||||
expect(result.status).not.toBe(0);
|
|
||||||
expect(result.stdout).toBe("");
|
|
||||||
});
|
|
||||||
|
|
||||||
it.each(["dev", "unknown"])(
|
|
||||||
"fails where there is .git and the version is %s",
|
|
||||||
(version) => {
|
|
||||||
makeCommittedCheckout();
|
|
||||||
const result = runVersion(version);
|
|
||||||
expect(result.status).not.toBe(0);
|
|
||||||
expect(result.stdout).toBe("");
|
|
||||||
},
|
|
||||||
);
|
|
||||||
});
|
|
||||||
@@ -43,7 +43,6 @@ import {
|
|||||||
isRetryable,
|
isRetryable,
|
||||||
isSafeToReplay,
|
isSafeToReplay,
|
||||||
resolveRetryOptions,
|
resolveRetryOptions,
|
||||||
UNATTENDED_RETRY_OPTIONS,
|
|
||||||
withRetry,
|
withRetry,
|
||||||
} from "../../src/retry.js";
|
} from "../../src/retry.js";
|
||||||
import { ApiError, TruncatedStreamError } from "../../src/errors.js";
|
import { ApiError, TruncatedStreamError } from "../../src/errors.js";
|
||||||
@@ -640,33 +639,3 @@ describe("retry defaults", () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("unattended retry options", () => {
|
|
||||||
it("allow ten attempts, a 1 s base delay and a 60 s cap", () => {
|
|
||||||
// The numbers the README documents for `quak backup`.
|
|
||||||
expect(UNATTENDED_RETRY_OPTIONS.attempts).toBe(10);
|
|
||||||
expect(UNATTENDED_RETRY_OPTIONS.baseDelayMs).toBe(1_000);
|
|
||||||
expect(UNATTENDED_RETRY_OPTIONS.maxDelayMs).toBe(60_000);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("give up on a request that keeps failing after at most 243 s of waiting", async () => {
|
|
||||||
// `random: () => 1` makes every wait its ceiling, the worst case.
|
|
||||||
const { sleep, delays } = recordingSleep();
|
|
||||||
let calls = 0;
|
|
||||||
await expect(
|
|
||||||
withRetry(
|
|
||||||
() => {
|
|
||||||
calls++;
|
|
||||||
return Promise.reject(new ApiError("HTTP 503", 503));
|
|
||||||
},
|
|
||||||
{ ...UNATTENDED_RETRY_OPTIONS, sleep, random: () => 1 },
|
|
||||||
),
|
|
||||||
).rejects.toThrow("HTTP 503");
|
|
||||||
|
|
||||||
expect(calls).toBe(10);
|
|
||||||
expect(delays).toEqual([
|
|
||||||
1_000, 2_000, 4_000, 8_000, 16_000, 32_000, 60_000, 60_000, 60_000,
|
|
||||||
]);
|
|
||||||
expect(delays.reduce((sum, ms) => sum + ms, 0)).toBe(243_000);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|||||||
+1
-1
@@ -18,5 +18,5 @@
|
|||||||
"sourceMap": true,
|
"sourceMap": true,
|
||||||
"resolveJsonModule": true
|
"resolveJsonModule": true
|
||||||
},
|
},
|
||||||
"include": ["src/**/*", "bin/**/*", "examples/**/*"]
|
"include": ["src/**/*", "bin/**/*"]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -535,18 +535,6 @@
|
|||||||
dependencies:
|
dependencies:
|
||||||
undici-types "~6.21.0"
|
undici-types "~6.21.0"
|
||||||
|
|
||||||
"@types/proper-lockfile@4.1.4":
|
|
||||||
version "4.1.4"
|
|
||||||
resolved "https://registry.yarnpkg.com/@types/proper-lockfile/-/proper-lockfile-4.1.4.tgz#cd9fab92bdb04730c1ada542c356f03620f84008"
|
|
||||||
integrity sha512-uo2ABllncSqg9F1D4nugVl9v93RmjxF6LJzQLMLDdPaXCUIDPeOJ21Gbqi43xNKzBi/WQ0Q0dICqufzQbMjipQ==
|
|
||||||
dependencies:
|
|
||||||
"@types/retry" "*"
|
|
||||||
|
|
||||||
"@types/retry@*":
|
|
||||||
version "0.12.5"
|
|
||||||
resolved "https://registry.yarnpkg.com/@types/retry/-/retry-0.12.5.tgz#f090ff4bd8d2e5b940ff270ab39fd5ca1834a07e"
|
|
||||||
integrity sha512-3xSjTp3v03X/lSQLkczaN9UIEwJMoMCA1+Nb5HfbJEQWogdeQIyVtTvxPXDQjZ5zws8rFQfVfRdz03ARihPJgw==
|
|
||||||
|
|
||||||
"@typescript-eslint/eslint-plugin@8.46.2":
|
"@typescript-eslint/eslint-plugin@8.46.2":
|
||||||
version "8.46.2"
|
version "8.46.2"
|
||||||
resolved "https://registry.yarnpkg.com/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.46.2.tgz#dc4ab93ee3d7e6c8e38820a0d6c7c93c7183e2dc"
|
resolved "https://registry.yarnpkg.com/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.46.2.tgz#dc4ab93ee3d7e6c8e38820a0d6c7c93c7183e2dc"
|
||||||
@@ -714,11 +702,6 @@
|
|||||||
loupe "^3.1.2"
|
loupe "^3.1.2"
|
||||||
tinyrainbow "^1.2.0"
|
tinyrainbow "^1.2.0"
|
||||||
|
|
||||||
"@xmldom/xmldom@^0.9.10":
|
|
||||||
version "0.9.12"
|
|
||||||
resolved "https://registry.yarnpkg.com/@xmldom/xmldom/-/xmldom-0.9.12.tgz#1f84c07cb95ccf28202299f77b5fd7fc257151e8"
|
|
||||||
integrity sha512-5AXjrcMClTryPe9LgZrygpB1lj7s0S9E0+W+AHaVKAVyHanafK86iPSvG5xHVSp/jC+VH1UXu0TAEmY279xH7A==
|
|
||||||
|
|
||||||
acorn-jsx@^5.3.2:
|
acorn-jsx@^5.3.2:
|
||||||
version "5.3.2"
|
version "5.3.2"
|
||||||
resolved "https://registry.yarnpkg.com/acorn-jsx/-/acorn-jsx-5.3.2.tgz#7ed5bb55908b3b2f1bc55c6af1653bada7f07937"
|
resolved "https://registry.yarnpkg.com/acorn-jsx/-/acorn-jsx-5.3.2.tgz#7ed5bb55908b3b2f1bc55c6af1653bada7f07937"
|
||||||
@@ -1019,12 +1002,10 @@ esutils@^2.0.2:
|
|||||||
resolved "https://registry.yarnpkg.com/esutils/-/esutils-2.0.3.tgz#74d2eb4de0b8da1293711910d50775b9b710ef64"
|
resolved "https://registry.yarnpkg.com/esutils/-/esutils-2.0.3.tgz#74d2eb4de0b8da1293711910d50775b9b710ef64"
|
||||||
integrity sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==
|
integrity sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==
|
||||||
|
|
||||||
exifreader@4.46.0:
|
exif-reader@2.0.3:
|
||||||
version "4.46.0"
|
version "2.0.3"
|
||||||
resolved "https://registry.yarnpkg.com/exifreader/-/exifreader-4.46.0.tgz#b6216eae512997587c45114f972cc14ca979205f"
|
resolved "https://registry.yarnpkg.com/exif-reader/-/exif-reader-2.0.3.tgz#259997735080bc6bb959c37b32c60f004ec4391d"
|
||||||
integrity sha512-ksHTpjXKWzbckY+bYlGaomG0EobHJkaMWLg5OPbzlTPda04F2dfrDfYSz8iAPp/kXYUSQdINBVI6nCAjQ8PQ/Q==
|
integrity sha512-zFbQvguwT9JkqyYhR7pjE1Yn8SagwaGLNRU0Oh14xFa1paSf5Gzxn4gxgk0XhnudI0UIqU+HgnBX93+nva592A==
|
||||||
optionalDependencies:
|
|
||||||
"@xmldom/xmldom" "^0.9.10"
|
|
||||||
|
|
||||||
expect-type@^1.1.0:
|
expect-type@^1.1.0:
|
||||||
version "1.3.0"
|
version "1.3.0"
|
||||||
@@ -1152,11 +1133,6 @@ globals@^14.0.0:
|
|||||||
resolved "https://registry.yarnpkg.com/globals/-/globals-14.0.0.tgz#898d7413c29babcf6bafe56fcadded858ada724e"
|
resolved "https://registry.yarnpkg.com/globals/-/globals-14.0.0.tgz#898d7413c29babcf6bafe56fcadded858ada724e"
|
||||||
integrity sha512-oahGvuMGQlPw/ivIYBjVSrWAfWLBeku5tpPE2fOPLi+WHffIWbuh2tCjhyQhTBPMf5E9jDEH4FOmTYgYwbKwtQ==
|
integrity sha512-oahGvuMGQlPw/ivIYBjVSrWAfWLBeku5tpPE2fOPLi+WHffIWbuh2tCjhyQhTBPMf5E9jDEH4FOmTYgYwbKwtQ==
|
||||||
|
|
||||||
graceful-fs@^4.2.4:
|
|
||||||
version "4.2.11"
|
|
||||||
resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.2.11.tgz#4183e4e8bf08bb6e05bbb2f7d2e0c8f712ca40e3"
|
|
||||||
integrity sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==
|
|
||||||
|
|
||||||
graphemer@^1.4.0:
|
graphemer@^1.4.0:
|
||||||
version "1.4.0"
|
version "1.4.0"
|
||||||
resolved "https://registry.yarnpkg.com/graphemer/-/graphemer-1.4.0.tgz#fb2f1d55e0e3a1849aeffc90c4fa0dd53a0e66c6"
|
resolved "https://registry.yarnpkg.com/graphemer/-/graphemer-1.4.0.tgz#fb2f1d55e0e3a1849aeffc90c4fa0dd53a0e66c6"
|
||||||
@@ -1431,15 +1407,6 @@ prettier@3.8.1:
|
|||||||
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
||||||
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
||||||
|
|
||||||
proper-lockfile@4.1.2:
|
|
||||||
version "4.1.2"
|
|
||||||
resolved "https://registry.yarnpkg.com/proper-lockfile/-/proper-lockfile-4.1.2.tgz#c8b9de2af6b2f1601067f98e01ac66baa223141f"
|
|
||||||
integrity sha512-TjNPblN4BwAWMXU8s9AEz4JmQxnD1NNL7bNOY/AKUzyamc379FWASUhc/K1pL2noVb+XmZKLL68cjzLsiOAMaA==
|
|
||||||
dependencies:
|
|
||||||
graceful-fs "^4.2.4"
|
|
||||||
retry "^0.12.0"
|
|
||||||
signal-exit "^3.0.2"
|
|
||||||
|
|
||||||
punycode@^2.1.0:
|
punycode@^2.1.0:
|
||||||
version "2.3.1"
|
version "2.3.1"
|
||||||
resolved "https://registry.yarnpkg.com/punycode/-/punycode-2.3.1.tgz#027422e2faec0b25e1549c3e1bd8309b9133b6e5"
|
resolved "https://registry.yarnpkg.com/punycode/-/punycode-2.3.1.tgz#027422e2faec0b25e1549c3e1bd8309b9133b6e5"
|
||||||
@@ -1455,11 +1422,6 @@ resolve-from@^4.0.0:
|
|||||||
resolved "https://registry.yarnpkg.com/resolve-from/-/resolve-from-4.0.0.tgz#4abcd852ad32dd7baabfe9b40e00a36db5f392e6"
|
resolved "https://registry.yarnpkg.com/resolve-from/-/resolve-from-4.0.0.tgz#4abcd852ad32dd7baabfe9b40e00a36db5f392e6"
|
||||||
integrity sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==
|
integrity sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==
|
||||||
|
|
||||||
retry@^0.12.0:
|
|
||||||
version "0.12.0"
|
|
||||||
resolved "https://registry.yarnpkg.com/retry/-/retry-0.12.0.tgz#1b42a6266a21f07421d1b0b54b7dc167b01c013b"
|
|
||||||
integrity sha512-9LkiTwjUh6rT555DtE9rTX+BKByPfrMzEAtnlEtdEwr3Nkffwiihqe2bWADg+OQRjt9gl6ICdmB/ZFDCGAtSow==
|
|
||||||
|
|
||||||
reusify@^1.0.4:
|
reusify@^1.0.4:
|
||||||
version "1.1.0"
|
version "1.1.0"
|
||||||
resolved "https://registry.yarnpkg.com/reusify/-/reusify-1.1.0.tgz#0fe13b9522e1473f51b558ee796e08f11f9b489f"
|
resolved "https://registry.yarnpkg.com/reusify/-/reusify-1.1.0.tgz#0fe13b9522e1473f51b558ee796e08f11f9b489f"
|
||||||
@@ -1533,11 +1495,6 @@ siginfo@^2.0.0:
|
|||||||
resolved "https://registry.yarnpkg.com/siginfo/-/siginfo-2.0.0.tgz#32e76c70b79724e3bb567cb9d543eb858ccfaf30"
|
resolved "https://registry.yarnpkg.com/siginfo/-/siginfo-2.0.0.tgz#32e76c70b79724e3bb567cb9d543eb858ccfaf30"
|
||||||
integrity sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==
|
integrity sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==
|
||||||
|
|
||||||
signal-exit@^3.0.2:
|
|
||||||
version "3.0.7"
|
|
||||||
resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-3.0.7.tgz#a9a1767f8af84155114eaabd73f99273c8f59ad9"
|
|
||||||
integrity sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==
|
|
||||||
|
|
||||||
signal-exit@^4.1.0:
|
signal-exit@^4.1.0:
|
||||||
version "4.1.0"
|
version "4.1.0"
|
||||||
resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-4.1.0.tgz#952188c1cbd546070e2dd20d0f41c0ae0530cb04"
|
resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-4.1.0.tgz#952188c1cbd546070e2dd20d0f41c0ae0530cb04"
|
||||||
|
|||||||
Reference in New Issue
Block a user