24 Commits
Author SHA1 Message Date
sneak 893cc95ab1 feat(library/store): on-disk JSON metadata store for the local cache
check / check (push) Successful in 24s
Add `MetadataStore` (src/library/store.ts): one metadata.json holding the
user id, schema version, collections cursor, and decrypted collection and
file records. Loaded whole into RAM with Maps for id lookups; rewritten
whole through the download layer's fsync atomic writer (temp, fsync,
rename, dir fsync). Binary keys are base64-encoded on disk. A missing,
unparseable, or wrong-schema file loads as an empty store, because the
file is only a cache the refresh unit repopulates. Directory 0700, file
0600, matching session.json. No lock file; no sync() beyond the writer.

This unit only stores; issue 42's refresh unit populates it. (closes #41)

Model: opus-4-8
2026-09-22 10:20:13 +00:00
sneak 4f2a3864b5 test(library/store): metadata store round-trip, empty-on-corrupt, perms
Failing first: the on-disk JSON metadata store does not exist yet. These
tests fix its contract before the implementation lands — round-trip
fidelity including the binary keys, missing/corrupt files loading empty, a
schema-version mismatch discarded as stale cache, atomic writes leaving no
temp sibling, and directory 0700 / file 0600 permissions.

Model: opus-4-8
2026-09-22 10:18:46 +00:00
clawbot 8f575550af Durable atomic writer with fsync and per-chunk download progress (closes #39)
check / check (push) Successful in 41s
The atomic writer fsyncs the staged temp file before rename and the directory after, and is exported for reuse. downloadFile/downloadThumbnail gain an optional per-chunk onProgress hook (non-decreasing, final equals bytesWritten; no-op when absent). Retry and TAG_FINAL checks unchanged.

Model: opus-4-8
2026-09-22 12:01:06 +02:00
clawbot ead083c1d6 Carry file size, thumbnail size, and deletion flag through decryptFile (closes #37)
check / check (push) Successful in 25s
decryptFile now sets file.size/thumbnail.size from raw.info (undefined when absent) and carries an optional isDeleted on EnteFile. Plain EnteFile return, no caller changes; listFiles keeps filtering tombstones. Tests cover the three fields and the size-absent case.

Model: opus-4-8
2026-09-22 12:01:03 +02:00
clawbot d1d6cdd4f0 Revert the 16 commits pushed to next on 2026-09-04 by an agent outside the managed fleet
check / check (push) Successful in 1m40s
sneak, 2026-09-05: "inference instance stopped. undo its rogue work." The
reverted commits stay in history; nothing else on next is touched.

Model: fable-5-1
2026-09-05 09:33:23 +00:00
user 48db9b438a test(lint-once): read docker image/builder build as builds too
check / check (push) Successful in 1m12s
`docker image build` and `docker builder build` are management-command
spellings of the same build, take the same -f, and were emitting no edge at
all — the same shape as the buildx miss.
2026-09-04 12:08:54 +00:00
user 81150f433c test(lint-once): disclose the build shapes read, correct the run: | claim
check / check (push) Successful in 13s
Header now states which docker invocation shapes are recognised as builds
(bare, buildx, and either through global flags) and which are not (compose),
and that the RUN keyword is read case-insensitively with any whitespace
separator. The workflow comment claimed a `run: |` block fails the count;
it does not — the count is unmoved by it and the pinned resolved list is
what turns it red.
2026-09-04 12:06:44 +00:00
user f67a1c4d92 test(lint-once): match RUN case-insensitively, follow buildx builds
check / check (push) Successful in 1m8s
WIP round 9: items 1-3. Verification matrix still to re-run.
2026-09-04 12:04:24 +00:00
user ffc817522e test(lint-once): pin the two-build and later-flag shapes
check / check (push) Successful in 15s
The false green: script/lint chaining a second docker build after the
lint image ran prettier twice and counted once. The misresolution: a
bare docker build followed by cp -f resolved to the cp's file. Pin
both, and each separator that bounds an invocation.
2026-09-04 11:51:17 +00:00
user 88a5fcaa87 lint-once: bound the docker file flag to its own invocation
check / check (push) Failing after 42s
edgesOf used a single test/exec for docker build, so a line with two
builds produced one edge and the file flag was searched across the whole
line. Follow every occurrence with matchAll and slice each one to the
next shell separator before looking for its flag.

Also correct the README's claim that script/fmt-check is the one
formatting path left on the host.
2026-09-04 11:49:38 +00:00
user 2fcb3e6ced test(lint-once): follow a short file flag with its value attached
check / check (push) Successful in 14s
`docker build -fDockerfile.lint .` is a plain `-f` naming a literal file —
the flag parser reads the attached value for any shorthand — but the
resolver's `[=\s]+` required a separator, so the invocation fell through to
the default `Dockerfile` edge. Mutating the `check` recipe to that spelling
left the suite 52/52 green while `-f Dockerfile.lint` was caught.

The header already promised this form was followed: it claimed the file
named by `-f` is resolved wherever the flag sits, and disclosed only a
bundled short cluster (`-qf <file>`) as unfollowed. Overclaiming is the
defect, so the resolver is taught the form rather than the claim narrowed.

The attached form is allowed only for the short flag, keeping `--force-rm`
out of it; `--file` still requires `=` or whitespace. The `-qf` cluster
limitation is untouched and still pinned.
2026-09-04 11:37:08 +00:00
user 197296edba lint-once: follow every spelling of docker build's file flag
check / check (push) Successful in 1m17s
The edge resolver matched `-f <file>` only. `docker build --file=X` fell
through to the default `Dockerfile` edge, so a second prettier pass wired in
that way was followed into the wrong file, counted nothing, and left the suite
green -- the exact false green this test exists to prevent, reachable by
writing the flag the long way.

Mutation, adding one line to the `check` recipe, before this commit:

    @docker build -f Dockerfile.lint .        1 failed / 47  caught
    @docker build --file=Dockerfile.lint .    48 passed / 48  MISSED

After, all four spellings fail with `expected 2 to be 1`:
`-f X`, `-f=X`, `--file X`, `--file=X`.

Docker takes the value either way for both the short and long flag, so the
resolver now reads `(?:-f|--file)[=\s]+`, still searched anywhere in the
invocation rather than at a fixed position. A leading \s keeps a longer flag
ending in the same letters (`--force-rm`) from supplying the match.

The header claimed `docker build -f <file>` coverage without qualification,
which a reader could take to include the long form it did not follow; it now
names all four forms and the fallback. The one limitation it asserts -- a
bundled cluster like `-qf X` resolving to the default -- is pinned by a test,
since an unpinned limitation is how the header drifts back into overclaiming.
2026-09-04 11:31:06 +00:00
user bd88eced84 README: state the container-lint claim as what is actually true
check / check (push) Successful in 1m14s
The Linting section this PR adds claimed "There is no host lint path" and
then, two paragraphs later, documented script/fmt-check as a host-side
formatting check. Both cannot be true, and the absolute one is the false one.

What is true is narrower: no lint path reachable from script/check or
script/precommit runs on the host, so every lint verdict those two produce
comes from the container. script/fmt-check stays as a standalone entrypoint,
now stated as the one host formatting path with nothing reaching it. The
"exactly one place" and "in the container only" phrasings elsewhere are
qualified the same way.
2026-09-04 11:27:49 +00:00
user 956870889f Update the make variable header to match the ?= and prefix rules
check / check (push) Successful in 15s
2026-09-04 11:19:45 +00:00
user 18b0f039b5 Pin the export/override prefix and ?= first-wins with tests
check / check (push) Failing after 7s
2026-09-04 11:18:19 +00:00
user 18226d345b WIP: fix export/override prefix and ?= first-wins in make variable parser
check / check (push) Successful in 14s
Two defects in the lint-once resolver's make variable handling:

- The assignment pattern anchored at the start of the name, so
  `export FMT := script/fmt-check` was never collected and `@$(FMT)`
  went unresolved. An optional `export `/`override ` prefix is now
  allowed.
- `?=` assigns only when the name is unset, so the first assignment
  wins. The parser called .set() unconditionally, letting a later
  `FMT ?= script/build` overwrite an earlier `FMT := script/fmt-check`
  and resolve to a command make never runs.

Tests pinning both to follow.
2026-09-04 11:17:32 +00:00
clawbot 075b1bb921 Expand single-line make variables in the lint-once resolver
check / check (push) Successful in 1m13s
A recipe of `@$(FMT)` with `FMT := script/fmt-check` gave `make check` two
prettier passes while the suite stayed green: the resolver read the line as
invoking nothing. The shipped Makefile already writes recipes that way
(`@$(YARN) tsc --watch`), so this was a gap in the repo's own house style.

Variables assigned a literal on one line (`:=`, `=`, `?=`) are collected in a
pass of their own and substituted into target and recipe lines. Values needing
evaluation -- another reference, a make function, a `define` body -- are left
verbatim, and the header's not-followed list now says so.
2026-09-04 11:05:41 +00:00
clawbot ff4cc63c8b Name the define/endef hole in the lint-once header
check / check (push) Successful in 1m8s
A `define EXTRA ... endef` body pulled into a recipe as $(EXTRA) gives
`make check` a second prettier pass that the walk still scores as one.
The parser expands no variables, so this is a name it cannot resolve,
not a body it declines to read. The header's exclusion list says so
rather than claiming coverage the code does not have.
2026-09-04 10:55:41 +00:00
clawbot f4ecef8820 Follow make conditionals and semicolon recipes in lint-once
check / check (push) Successful in 15s
makeRecipes() reset the current target on every non-tab line, so an
ifeq/endif block ended the recipe and every tab-indented line inside it
was discarded; and the target line's tail was read entirely as
prerequisites, so `check: ; @script/fmt-check` split to tokens that
named no target and vanished. Both gave `make check` a second prettier
pass with the suite green.

Conditional directives no longer end a recipe, and every branch is
treated as reachable rather than evaluating the condition. The target
line is split on the first `;`: what precedes it is the prerequisite
list, what follows is the first recipe line. Both pinned directly, and
the header's exclusion list now names what the parser actually skips.
2026-09-04 10:54:24 +00:00
clawbot 9ed92e1231 Format the lint-once resolver changes
check / check (push) Successful in 34s
Reflow the heredoc RUN fixture array in the BuildKit heredoc test to
satisfy prettier --check, which rejected the single-line form. No
behavioural change to the tests or the resolver.
2026-09-04 10:31:40 +00:00
clawbot 8bf5138582 WIP: follow Makefile prerequisites and $(MAKE) in lint-once
check / check (push) Failing after 37s
2026-09-04 10:05:38 +00:00
sneak 2bfa11c10c Walk the path CI runs in lint-once, and pin the branch the container installs (closes #33)
check / check (push) Successful in 35s
The header of test/packaging/lint-once.test.ts claimed a duplicate prettier
pass is caught wherever it is added. It was not: the walk started at
`make check`, which never reads `Dockerfile`, so appending
`RUN yarn run prettier --check .` to the image that `script/cibuild` builds
left the suite green — two prettier passes on the one path where it matters
most. The walk now also starts at `.gitea/workflows/check.yml` and follows
its `run:` steps into `script/cibuild` and from there into both images, so
the graph under test is the one CI executes rather than the one it was
assumed to execute. Reaching `script/cibuild` and `Dockerfile` is asserted,
and the test and build image is asserted to invoke prettier zero times.

The lockfile assertion was a substring check against the whole of
`script/bootstrap`. That script has two install sites, and the containers
take the second, because the pinned node image ships yarn; changing that
site to a bare `yarn install` kept the suite green while the container's
install stopped being pinned. `install_js_deps` is now resolved out of the
script and split at its `missing yarn` guard, and every `yarn install`
occurrence in each branch is required to carry `--frozen-lockfile`. That the
container runs `script/bootstrap` at all is asserted too, so the lockfile
assertions cannot end up describing a script the image never executes.

Prettier is counted per occurrence instead of per line:
`prettier --check . && prettier --check src` was one invocation by the old
count. The `continue` that followed a counted line also dropped every
script, make, yarn and docker edge sharing that line, so a subtree could be
hidden behind a single `&&`; edges are now extracted from every line.

Undercounting is what would make this file worthless, so every way of
reaching nothing is a thrown error rather than a quiet zero: an unknown
Makefile target, an unknown package.json script, a missing script file, a
node that resolves to no commands, and an unknown node kind. All five are
tested, as is a walk that legitimately counts zero, and the cycle guard.

Every assertion in the file was mutation-tested: changed to assert something
else, run, and confirmed to fail for its own named reason. The two mutations
above were reproduced and both now turn the suite red.

test/packaging/entrypoints.test.ts said `make check` runs test, lint and
fmt-check. Formatting has been part of the lint container since the
duplicate host pass was removed, so the comment now says what it does.
2026-08-10 13:41:12 +00:00
sneak a73f0abbe8 Check formatting once per make check, in the container (closes #29)
check / check (push) Successful in 59s
script/check ran script/test, script/lint and script/fmt-check. Since
linting moved into Docker, script/lint is a build of Dockerfile.lint,
which runs `prettier --check .` as a build step — so make check checked
formatting twice over the same tree: once in the container and once on
the host. script/precommit had the same pair.

Drop the script/fmt-check call from both. The container keeps the check,
because a successful Dockerfile.lint build is what CI treats as proof of
a clean tree, and it is the stronger of the two verdicts: its prettier is
digest-pinned and installed under --frozen-lockfile, while the host's is
whatever the working tree happens to have. The pre-commit hook is
unchanged in what it catches — script/lint still fails a badly formatted
tree, and therefore the commit.

script/fmt-check survives as a standalone entrypoint, as REPO_POLICIES.md
requires, for asking the formatting question by itself without docker.
Its verdict cannot drift from the container's: prettier is pinned to an
exact version, installed from yarn.lock in both places, and reads
.gitignore as its default ignore file, which is why .dockerignore keeps
.gitignore in the build context.

The count is asserted rather than promised. test/packaging/lint-once.test.ts
walks the invocation graph from each entrypoint — through the Makefile
shims, the script/ calls, the package.json scripts and the docker build
into Dockerfile.lint's RUN steps — and counts prettier invocations: one
per make check, one per script/precommit, and one each for make lint and
make fmt-check alone, so neither can become a no-op that satisfies the
count trivially. The walk also asserts which nodes it reached, so a
restructure that defeats the resolver fails the test instead of quietly
counting zero.

Observed: 2 prettier invocations per make check before, 1 after.
2026-08-10 13:05:03 +00:00
sneak fed39d19cf Run all linting in Docker via Dockerfile.lint (closes #30)
check / check (push) Successful in 1m2s
Linting now happens in one place only: a new root Dockerfile.lint copies
the repo into the digest-pinned node image already used by Dockerfile and
runs eslint and prettier as build steps, so a successful build is a clean
lint. script/lint is reduced to building it, which also works where the
docker daemon is remote and bind mounts are impossible. No host lint path
survives: the "lint" script is gone from package.json, so there is no
second, unpinned way to get a lint verdict.

Caching is waived for lint, because a lint build over an unchanged tree
returns success in well under a second having linted nothing. LINT_EPOCH
is the cache buster and it fails closed exactly as CHECK_EPOCH does: an
unset ARG is the empty string, which is a perfectly stable cache key, so
the guard rejects it and a bare `docker build -f Dockerfile.lint .` errors
out instead of serving a green it did not earn. Both linters sit below the
guard, so a fresh epoch forces them to execute while the bootstrap and
dependency layers above stay cached.

That makes script/lint a docker build, which nothing inside a container
may call. script/check calls script/lint, so the Dockerfile image can no
longer run make check: the lint stage and its COPY --from=lint ordering
hack are deleted, and the remaining stage runs make test and make build
under the existing CHECK_EPOCH guard. script/cibuild is now the composite
gate and builds the lint image first, so a lint failure is reported before
the slower suite runs.

The .dockerignore exclusions are unchanged and still apply to the lint
build, including the .claude/ exclusion (eslint's flat config does not
ignore dot-directories, so a nested worktree in the context would be
linted) and the deliberate exception that keeps .gitignore in the context
for prettier. A new test asserts no per-Dockerfile ignore file shadows the
root one for either image, and test/packaging/lint-docker.test.ts asserts
the whole shape: the docker-only lint path, the digest pin, manifests
copied before sources, the fail-closed guard with both linters below it,
the absence of a lint stage or make check in Dockerfile, and the build
order in script/cibuild.
2026-08-10 12:39:23 +00:00
21 changed files with 1622 additions and 84 deletions
+9 -18
View File
@@ -1,36 +1,27 @@
# Lint stage — fast feedback on formatting and lint issues # Test and build image: the suite, then the compile.
# node 22.22.0 on Alpine 3.23.3 (node:22-alpine), 2026-08-09 #
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS lint # Linting deliberately does not happen here. `script/lint` is a build of
WORKDIR /app # Dockerfile.lint, and `script/check` calls `script/lint`, so running
COPY script/ script/ # `make check` in this image would mean running `docker build` inside a
COPY package.json yarn.lock ./ # container. Lint runs exactly once, in Dockerfile.lint; script/cibuild
RUN script/bootstrap # builds that first and this second.
COPY . .
RUN make fmt-check
RUN make lint
# Check stage — the full suite and the build
# node 22.22.0 on Alpine 3.23.3 (node:22-alpine), 2026-08-09 # node 22.22.0 on Alpine 3.23.3 (node:22-alpine), 2026-08-09
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS check FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS check
WORKDIR /app WORKDIR /app
# Force BuildKit to run the lint stage before proceeding. Without this the
# two stages run in parallel and a lint failure can lose the race.
COPY --from=lint /app/yarn.lock /dev/null
COPY script/ script/ COPY script/ script/
COPY package.json yarn.lock ./ COPY package.json yarn.lock ./
RUN script/bootstrap RUN script/bootstrap
COPY . . COPY . .
# CHECK_EPOCH is a cache buster: without it Docker serves `make check` from # CHECK_EPOCH is a cache buster: without it Docker serves the test layer from
# cache on an unchanged tree, the suite never executes, and the build still # cache on an unchanged tree, the suite never executes, and the build still
# exits 0. The guard makes an absent argument a hard failure — an unset ARG # exits 0. The guard makes an absent argument a hard failure — an unset ARG
# is the empty string, which is a perfectly stable cache key, so a plain # is the empty string, which is a perfectly stable cache key, so a plain
# `docker build .` would otherwise still get the false green. Fail closed. # `docker build .` would otherwise still get the false green. Fail closed.
ARG CHECK_EPOCH ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1 RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN make check RUN make test
ARG CHECK_EPOCH ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1 RUN [ -n "$CHECK_EPOCH" ] || exit 1
+35
View File
@@ -0,0 +1,35 @@
# Lint image: every lint run happens here, and nowhere else. The repo is
# COPYed into a digest-pinned image and the linters run as build steps, so a
# successful build IS a clean lint. `script/lint` does nothing but build this
# file, which also works where the docker daemon is remote and bind mounts are
# impossible. Nothing that runs inside a container may call `script/lint`:
# that is why Dockerfile no longer runs `make check`.
# node 22.22.0 on Alpine 3.23.3 (node:22-alpine), 2026-08-09
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS lint
WORKDIR /app
# Manifests before sources, so the dependency install layer stays cached
# until package.json or yarn.lock changes. script/bootstrap ends in
# `yarn install --frozen-lockfile`; the lint steps below are deliberately
# not cached.
COPY script/ script/
COPY package.json yarn.lock ./
RUN script/bootstrap
COPY . .
# LINT_EPOCH is a cache buster, with the same fail-closed contract as
# CHECK_EPOCH in Dockerfile. No lint cache is wanted: on an unchanged tree
# Docker serves the linter layers in well under a second, having linted
# nothing, and the build still exits 0. The guard makes an absent argument a
# hard failure — an unset ARG is the empty string, which is a perfectly
# stable cache key, so a plain `docker build -f Dockerfile.lint .` would
# otherwise get exactly that false green. Every layer below this one is a
# child of the guard, so a fresh epoch forces all of them to execute.
ARG LINT_EPOCH
RUN [ -n "$LINT_EPOCH" ] || exit 1
# The linters are invoked directly rather than through `make lint`, because
# `make lint` is the build of this file.
RUN yarn run eslint .
RUN yarn run prettier --check .
+76 -29
View File
@@ -86,31 +86,70 @@ alpine. We provide:
files the compiler wrote, and make the CLI executable (our own extension) files the compiler wrote, and make the CLI executable (our own extension)
- `script/test` — run the test suite (vitest, hard-capped at 30s where `timeout` - `script/test` — run the test suite (vitest, hard-capped at 30s where `timeout`
is available, verbose rerun on failure) is available, verbose rerun on failure)
- `script/lint` — run eslint and a prettier check - `script/lint` — run eslint and a prettier check, by building
`Dockerfile.lint`; requires docker (see Linting below)
- `script/fmt` — format all files with prettier (writes) - `script/fmt` — format all files with prettier (writes)
- `script/fmt-check` — check formatting (read-only) - `script/fmt-check` — check formatting on the host (read-only); standalone, and
- `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own not called by `script/check` or `script/precommit`, because `script/lint`
extension) already checks formatting in the container (see Linting below)
- `script/docker` — build the Docker image, tagged via `script/projectname` - `script/check` — run all checks: `test`, `lint` (our own extension)
- `script/cibuild` — cd to the repo root and run the image build (what CI runs; - `script/docker` — build the test and build image, tagged via
the build runs `make fmt-check` and `make lint` in a first stage, then `script/projectname`
`make check` and `make build` in a second) - `script/cibuild` — cd to the repo root and build both images (what CI runs):
`script/lint` first, then the `Dockerfile` image, which runs `make test` and
`make build`
- `script/precommit` — run by the git pre-commit hook (our own extension); runs - `script/precommit` — run by the git pre-commit hook (our own extension); runs
`script/lint` and `script/fmt-check` but deliberately not the tests, so the `script/lint`, which checks both lint and formatting, but deliberately not the
TDD red-phase commit can land tests, so the TDD red-phase commit can land
- `script/install-precommit` — installs the git pre-commit hook (our own - `script/install-precommit` — installs the git pre-commit hook (our own
extension); `make hooks` shims to it extension); `make hooks` shims to it
`make hooks` installs the pre-commit hook that runs `script/precommit`. `make hooks` installs the pre-commit hook that runs `script/precommit`.
Both `script/docker` and `script/cibuild` pass ### Linting
`--build-arg CHECK_EPOCH="$(date +%s)"`. The Dockerfile refuses to build without
it. This is deliberate: on an unchanged tree Docker would otherwise serve the Linting runs in a container, one way, everywhere. `script/lint` builds
`make check` layer from cache, so the suite would never run and the build would `Dockerfile.lint`, which copies the repo into a digest-pinned node image and
still exit 0. A changing epoch invalidates the check and build layers on every runs eslint and prettier as build steps, so a successful build is a clean lint.
invocation while leaving the dependency layers below them cached, and the There is no host lint path: docker is required to lint, and that also works
missing-argument guard means a bare `docker build .` fails loudly instead of where the docker daemon is remote and bind mounts are impossible.
quietly reporting a green it did not earn.
The formatting check is part of that, not a step beside it. `script/check` and
`script/precommit` therefore call `script/lint` and stop; neither calls
`script/fmt-check` as well, which would run prettier a second time over the same
tree for the same verdict — and the weaker of the two, since the host's prettier
is whatever the working tree has installed. So `make check` and the pre-commit
hook both still fail on a badly formatted tree, and prettier runs exactly once
in each. `test/packaging/lint-once.test.ts` asserts that count by walking the
invocation graph, so a second pass cannot creep back in unnoticed.
`script/fmt-check` remains as a standalone entrypoint for asking the formatting
question on its own, without docker and without the rest of lint. Its verdict
cannot drift from the container's: prettier is pinned to an exact version,
installed from `yarn.lock` under `--frozen-lockfile` in both places, and reads
`.gitignore` as its default ignore file — which is why `.dockerignore`
deliberately keeps `.gitignore` in the build context.
Lint happens in exactly one place, which constrains the rest of the build.
`script/check` calls `script/lint`, so `make check` cannot run inside a
container without asking for docker inside docker. The image built from
`Dockerfile` therefore runs `make test` and `make build` and does not lint;
`script/cibuild` builds `Dockerfile.lint` first and that image second, so CI
gets both verdicts.
### Build epochs
`script/lint` passes `--build-arg LINT_EPOCH="$(date +%s)"`, and `script/docker`
and `script/cibuild` pass `--build-arg CHECK_EPOCH="$(date +%s)"`. Both
Dockerfiles refuse to build without their argument. This is deliberate: on an
unchanged tree Docker would otherwise serve the linter and test layers from
cache, so nothing would run and the build would still exit 0 — a lint build over
an untouched tree returns success in well under a second, having linted nothing.
A changing epoch invalidates every layer below the guard on every invocation
while leaving the dependency layers above them cached, and the missing-argument
guard means a bare `docker build .` fails loudly instead of quietly reporting a
green it did not earn: an unset build argument is the empty string, which is a
perfectly stable cache key.
## Rationale ## Rationale
@@ -145,8 +184,10 @@ All work on quak is test-driven. No exceptions.
3. Subsequent commits add the implementation and any refactors needed to make 3. Subsequent commits add the implementation and any refactors needed to make
the tests pass. the tests pass.
4. A feature branch can only be merged into `main` when `make check` is green. 4. A feature branch can only be merged into `main` when `make check` is green.
`main` is always green. The Dockerfile runs `make check` and `make build`, so `main` is always green. CI runs `script/cibuild`, which lints via
neither a red branch nor one that does not compile can pass CI. `Dockerfile.lint` and then runs `make test` and `make build` in the
`Dockerfile` image, so neither a red branch nor one that does not compile can
pass CI.
5. Tests are the canonical API documentation for this library. Every test file 5. Tests are the canonical API documentation for this library. Every test file
is commented thoroughly enough that a reader who has never seen quak can is commented thoroughly enough that a reader who has never seen quak can
learn how to use it from the tests alone. Comments explain why a behavior learn how to use it from the tests alone. Comments explain why a behavior
@@ -160,11 +201,11 @@ All work on quak is test-driven. No exceptions.
history must still show tests landing before (or with) the matching history must still show tests landing before (or with) the matching
implementation. implementation.
8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which 8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which
runs the lint and format checks but not the full `make check`. This is runs `script/lint` — eslint and the prettier check, in the container — but
deliberate so the TDD red-phase commit (failing tests, no implementation yet) not the tests, and so not the full `make check`. This is deliberate so the
can land. The full `make check` runs as part of the image build, which is TDD red-phase commit (failing tests, no implementation yet) can land. The
what CI executes via `script/cibuild`, so a red branch still cannot reach suite runs as part of the image build, which is what CI executes via
`main`. `script/cibuild`, so a red branch still cannot reach `main`.
## Design ## Design
@@ -191,7 +232,8 @@ quak/
quak.ts CLI entrypoint (commander.js) quak.ts CLI entrypoint (commander.js)
test/ unit + integration tests (vitest) test/ unit + integration tests (vitest)
Makefile Makefile
Dockerfile Dockerfile test suite and compile
Dockerfile.lint eslint and prettier, as build steps
package.json package.json
tsconfig.json tsconfig.json
``` ```
@@ -478,9 +520,14 @@ documents:
implementation. Tests are the canonical API documentation and must be implementation. Tests are the canonical API documentation and must be
commented thoroughly. `main` is always green. commented thoroughly. `main` is always green.
- **Required checks before every commit:** `make lint` (eslint + prettier check) - **Required checks before every commit:** `make lint` must pass — that is
and `make fmt-check` must pass. The pre-commit hook enforces this. eslint plus the prettier check, and it builds `Dockerfile.lint`, so it needs
`make check` (which also runs tests) must pass before merging to `main`. docker. The pre-commit hook enforces exactly that. `make check` (which also
runs the tests) must pass before merging to `main`. `make fmt-check` is
available for a host-side formatting check on its own, but it is not a
separate requirement: `make lint` already covers it, and running both would
check formatting twice. Never invoke eslint or prettier directly; linting runs
in the container only.
- **Formatting:** prettier with 4-space indents and `proseWrap: always` for - **Formatting:** prettier with 4-space indents and `proseWrap: always` for
markdown. Use `make fmt` to format. Use `yarn` not `npm`. markdown. Use `make fmt` to format. Use `yarn` not `npm`.
+54
View File
@@ -18,6 +18,60 @@ Update the README API reference section to match the current implementation.
# Completed Steps # Completed Steps
- 2026-09-22: Added the on-disk JSON metadata store (`src/library/store.ts`,
issue 41), phase 2 of the cache/API work. One `metadata.json` holds the user
id, schema version, collections cursor, and the decrypted collection and file
records, loaded whole into RAM with `Map`s for id lookups and rewritten whole
through the download layer's fsync atomic writer. Missing, corrupt, or
wrong-schema files load empty; the directory is 0700 and the file 0600. The
store only stores — the refresh unit (issue 42) populates it.
- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
`decryptFile` (issue 37, foundation for the cache/API design). Live files now
populate `file.size`/`thumbnail.size` from the server's `info` (left
`undefined` when the server omits it), and `isDeleted` is carried from the
diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
rows before decrypting. Surfacing a tombstone through decryption belongs to
the enumeration unit (issue 38).
- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
`make check` only, so it never read `Dockerfile` — the image CI builds through
`script/cibuild` — and a second `prettier --check .` could be added there with
the suite staying green. The walk now also starts at
`.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
test is the one CI executes rather than the one someone assumed it executes.
The lockfile assertion was a substring check against the whole of
`script/bootstrap`, which has two install sites and so reported the branch the
containers never take; the two branches are now resolved separately and every
`yarn install` in each is required to be `--frozen-lockfile`. Prettier is
counted per occurrence instead of per line, so two invocations chained with
`&&` no longer read as one, and edges are followed on counted lines instead of
being skipped. Every way for the walk to reach nothing — an unknown target, an
unknown script, a missing file, a node with no commands, an unknown node kind
— is a thrown error rather than a quiet zero. Every assertion in the file was
mutation-tested individually.
- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
linting moved into Docker, the duplicate was one container pass and one host
pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
prettier as a build step, and `script/check` then called `script/fmt-check` as
well. The host call is gone from `script/check` and from `script/precommit`;
the container keeps checking formatting, because a successful
`Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
also what still fails the pre-commit hook on a badly formatted tree.
`script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
drift from the container's. A test walks the invocation graph from each
entrypoint — through the Makefile shims, the `script/` calls and the
`docker build` — and asserts the prettier count, so the duplication cannot
come back unnoticed.
- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
`Dockerfile.lint`, which copies the repo into the digest-pinned node image and
runs eslint and prettier as build steps, so a successful build is a clean
lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
cache, which is how a lint build returns success in under a second having
linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
ordering hack are gone: that image now runs `make test` and `make build` only,
because `script/check` calls `script/lint` and running it in a container would
mean docker inside docker. `script/cibuild` builds the lint image first, then
the test and build image.
- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage - 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
stage takes a `COPY --from=lint` dependency on it before running `make check` stage takes a `COPY --from=lint` dependency on it before running `make check`
-1
View File
@@ -24,7 +24,6 @@
"build": "script/build", "build": "script/build",
"quak": "node ./dist/bin/quak.js", "quak": "node ./dist/bin/quak.js",
"test": "vitest run", "test": "vitest run",
"lint": "eslint .",
"fmt": "prettier --write .", "fmt": "prettier --write .",
"fmt-check": "prettier --check ." "fmt-check": "prettier --check ."
}, },
+15 -3
View File
@@ -1,6 +1,19 @@
#!/bin/sh #!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own # script/check: run all checks (test, lint). Our own extension to
# extension to scripts-to-rule-them-all. Must not modify any files. # scripts-to-rule-them-all. Must not modify any files.
#
# The formatting check is part of lint, not a step of its own:
# script/lint builds Dockerfile.lint, which runs eslint AND
# `prettier --check .` as build steps. Calling script/fmt-check here as
# well would run prettier a second time over the same tree for the same
# verdict — the weaker of the two, since the host toolchain is whatever
# the working tree happens to have installed while the container's is
# digest-pinned. script/fmt-check remains a standalone entrypoint for
# asking the formatting question by itself.
#
# script/lint builds Dockerfile.lint, so this script requires docker and
# must never be run from inside a container: that is why the Dockerfile
# image runs script/test and script/build rather than this.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -8,7 +21,6 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() { main() {
"$SCRIPT_DIR/test" "$SCRIPT_DIR/test"
"$SCRIPT_DIR/lint" "$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
} }
main "$@" main "$@"
+14 -7
View File
@@ -1,16 +1,23 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs script/check and # script/cibuild: run the CI build, which is both images in a defined order.
# script/build, and CHECK_EPOCH differs on every invocation, so those two #
# layers cannot be served from Docker's cache: a green build here means # First script/lint, which builds Dockerfile.lint and is the one and only
# the checks ran now, not that a previous run was remembered. The layers # place linting happens — it goes first so a lint failure is reported before
# below the epoch (bootstrap, yarn install) are unaffected and stay # the slower suite runs. Then the Dockerfile image, which runs script/test
# cached. A build that omits the argument fails by design. # and script/build. CHECK_EPOCH and LINT_EPOCH differ on every invocation, so
# neither the linters nor the suite can be served from Docker's cache: a
# green build here means the checks ran now, not that a previous run was
# remembered. The layers below the epochs (bootstrap, yarn install) are
# unaffected and stay cached. A build that omits the arguments fails by
# design.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
"$SCRIPT_DIR/lint"
docker build --build-arg CHECK_EPOCH="$(date +%s)" . docker build --build-arg CHECK_EPOCH="$(date +%s)" .
} }
+2 -1
View File
@@ -3,7 +3,8 @@
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# CHECK_EPOCH is passed for the same reason script/cibuild passes it: the # CHECK_EPOCH is passed for the same reason script/cibuild passes it: the
# Dockerfile refuses to build without it, so that no path to an image can # Dockerfile refuses to build without it, so that no path to an image can
# quietly serve the check and build layers from cache. # quietly serve the test and build layers from cache. This builds the test
# and build image only; linting is a separate image, built by script/lint.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+14 -3
View File
@@ -1,13 +1,24 @@
#!/bin/sh #!/bin/sh
# script/lint: run the linter (eslint plus a prettier check). # script/lint: run the linters. eslint and prettier are never run against
# the working tree from here: linting runs via docker only, one way,
# everywhere — script/lint builds Dockerfile.lint, which COPYs the repo into
# the pinned node image and runs the linters as build steps. That works even
# when the docker daemon is remote and bind mounts are impossible.
#
# LINT_EPOCH is passed on every invocation because no lint cache is wanted:
# on an unchanged tree Docker would otherwise serve the linter layers, having
# linted nothing, and still exit 0. Dockerfile.lint refuses to build without
# the argument, so no path to a lint result can quietly come from cache.
#
# Nothing that runs inside a container may call this script; see the header
# of Dockerfile.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
yarn run eslint . docker build --build-arg LINT_EPOCH="$(date +%s)" -f Dockerfile.lint .
yarn run prettier --check .
} }
main "$@" main "$@"
+13 -5
View File
@@ -2,17 +2,25 @@
# script/precommit: run by the git pre-commit hook; fails the commit if # script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all. # checks fail. Our own extension to scripts-to-rule-them-all.
# #
# Runs lint and fmt-check but deliberately NOT the tests, so the TDD # Runs lint but deliberately NOT the tests, so the TDD red-phase commit
# red-phase commit (failing tests, no implementation yet) can land. CI # (failing tests, no implementation yet) can land. CI runs
# runs make check via docker build, which catches any branch that # script/cibuild, which builds both images and so catches any branch
# ships red. # that ships red.
#
# The formatting check is still enforced here, because script/lint is a
# build of Dockerfile.lint and that runs `prettier --check .` as a build
# step: a badly formatted tree fails this hook, and therefore the
# commit. Calling script/fmt-check as well would only run prettier a
# second time over the same tree for the same verdict.
#
# script/lint is a docker build (Dockerfile.lint); docker is required to
# commit, which is the point of linting one way, everywhere.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() { main() {
"$SCRIPT_DIR/lint" "$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
} }
main "$@" main "$@"
+51 -10
View File
@@ -1,5 +1,5 @@
import { randomUUID } from "node:crypto"; import { randomUUID } from "node:crypto";
import { rename, rm, writeFile } from "node:fs/promises"; import { open, rename, rm } from "node:fs/promises";
import { dirname, join } from "node:path"; import { dirname, join } from "node:path";
import { import {
fromBase64, fromBase64,
@@ -19,12 +19,20 @@ export interface DownloadResult {
bytesWritten: number; bytesWritten: number;
} }
// Fired as decrypted plaintext accumulates, with the running total of
// plaintext bytes recovered so far. Within one download it is non-decreasing
// and its last value equals the final `bytesWritten`. A retry restarts the
// file from byte zero (see `fetchAndDecrypt`), so a fresh attempt begins its
// own count from zero.
export type ProgressCallback = (bytesDone: number) => void;
const ENC_CHUNK_SIZE = STREAM_CHUNK_SIZE + STREAM_CHUNK_OVERHEAD; const ENC_CHUNK_SIZE = STREAM_CHUNK_SIZE + STREAM_CHUNK_OVERHEAD;
const streamDecrypt = async ( const streamDecrypt = async (
stream: ReadableStream<Uint8Array>, stream: ReadableStream<Uint8Array>,
header: Uint8Array, header: Uint8Array,
key: Uint8Array, key: Uint8Array,
onProgress?: ProgressCallback,
): Promise<Uint8Array> => { ): Promise<Uint8Array> => {
const state = initStreamPull(header, key); const state = initStreamPull(header, key);
const reader = stream.getReader(); const reader = stream.getReader();
@@ -51,6 +59,7 @@ const streamDecrypt = async (
totalPlain += plaintext.length; totalPlain += plaintext.length;
chunksPulled++; chunksPulled++;
lastTag = tag; lastTag = tag;
onProgress?.(totalPlain);
} }
if (done) { if (done) {
@@ -76,6 +85,7 @@ const streamDecrypt = async (
totalPlain += pulled.plaintext.length; totalPlain += pulled.plaintext.length;
chunksPulled++; chunksPulled++;
lastTag = pulled.tag; lastTag = pulled.tag;
onProgress?.(totalPlain);
} }
break; break;
} }
@@ -107,21 +117,47 @@ const streamDecrypt = async (
return result; return result;
}; };
// Write `plaintext` to `destination` atomically: stage it in a temporary // Write `plaintext` to `destination` atomically and durably: stage it in a
// sibling file (same directory, so the rename cannot cross a filesystem // temporary sibling file (same directory, so the rename cannot cross a
// boundary) and rename it into place. Callers therefore never observe a // filesystem boundary) and rename it into place. Callers therefore never
// partially written destination, and a pre-existing file at that path is // observe a partially written destination, and a pre-existing file at that
// replaced only once the new contents are complete on disk. // path is replaced only once the new contents are complete on disk.
const writeAtomic = async ( //
// Durability against a power cut needs two fsyncs. Without them the write can
// return while the data or the rename is still only in the kernel's page
// cache, and a crash then resurrects an empty renamed file — exactly the
// corruption a later backup run treats as a complete download. So the temp
// file's contents are fsynced before the rename, and the containing directory
// is fsynced after it, so both the bytes and the new directory entry are on
// stable storage before this returns.
//
// Exported so the metadata store can reuse the same durable write.
export const writeAtomic = async (
destination: string, destination: string,
plaintext: Uint8Array, plaintext: Uint8Array,
): Promise<void> => { ): Promise<void> => {
const dir = dirname(destination);
// The random suffix keeps concurrent downloads of the same destination // The random suffix keeps concurrent downloads of the same destination
// from stepping on each other's temporary file. // from stepping on each other's temporary file.
const tmpPath = join(dirname(destination), `.quak-${randomUUID()}.tmp`); const tmpPath = join(dir, `.quak-${randomUUID()}.tmp`);
try { try {
await writeFile(tmpPath, plaintext); const handle = await open(tmpPath, "w");
try {
await handle.writeFile(plaintext);
await handle.sync();
} finally {
await handle.close();
}
await rename(tmpPath, destination); await rename(tmpPath, destination);
// Fsync the directory so the rename itself survives a crash: renaming
// over a synced temp file still leaves the new directory entry in the
// page cache until the directory is synced.
const dirHandle = await open(dir, "r");
try {
await dirHandle.sync();
} finally {
await dirHandle.close();
}
} catch (err) { } catch (err) {
// Best-effort cleanup. A failure to remove the temporary file must // Best-effort cleanup. A failure to remove the temporary file must
// never replace the error that actually explains what went wrong. // never replace the error that actually explains what went wrong.
@@ -150,16 +186,18 @@ const fetchAndDecrypt = async (
openStream: () => Promise<ReadableStream<Uint8Array>>, openStream: () => Promise<ReadableStream<Uint8Array>>,
header: Uint8Array, header: Uint8Array,
key: Uint8Array, key: Uint8Array,
onProgress?: ProgressCallback,
): Promise<Uint8Array> => ): Promise<Uint8Array> =>
withRetry(async () => { withRetry(async () => {
const stream = await openStream(); const stream = await openStream();
return streamDecrypt(stream, header, key); return streamDecrypt(stream, header, key, onProgress);
}, api.getRetryOptions()); }, api.getRetryOptions());
export const downloadFile = async ( export const downloadFile = async (
api: ApiClient, api: ApiClient,
file: EnteFile, file: EnteFile,
outPath?: string, outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
const resolvedPath = outPath ?? file.metadata.title; const resolvedPath = outPath ?? file.metadata.title;
const header = fromBase64(file.file.decryptionHeader); const header = fromBase64(file.file.decryptionHeader);
@@ -168,6 +206,7 @@ export const downloadFile = async (
() => api.getFileStream(file.id, { retry: false }), () => api.getFileStream(file.id, { retry: false }),
header, header,
file.key, file.key,
onProgress,
); );
// Outside the retry, deliberately: only the attempt that produced a // Outside the retry, deliberately: only the attempt that produced a
// complete, authenticated plaintext gets to stage a temporary file, so a // complete, authenticated plaintext gets to stage a temporary file, so a
@@ -181,6 +220,7 @@ export const downloadThumbnail = async (
api: ApiClient, api: ApiClient,
file: EnteFile, file: EnteFile,
outPath?: string, outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
const resolvedPath = outPath ?? `thumb_${file.metadata.title}`; const resolvedPath = outPath ?? `thumb_${file.metadata.title}`;
const header = fromBase64(file.thumbnail.decryptionHeader); const header = fromBase64(file.thumbnail.decryptionHeader);
@@ -189,6 +229,7 @@ export const downloadThumbnail = async (
() => api.getThumbnailStream(file.id, { retry: false }), () => api.getThumbnailStream(file.id, { retry: false }),
header, header,
file.key, file.key,
onProgress,
); );
await writeAtomic(resolvedPath, plaintext); await writeAtomic(resolvedPath, plaintext);
return { path: resolvedPath, bytesWritten: plaintext.length }; return { path: resolvedPath, bytesWritten: plaintext.length };
+188
View File
@@ -0,0 +1,188 @@
// On-disk JSON metadata store for the local cache.
//
// The store keeps one `metadata.json` file holding the account's server
// state: the user id, a schema version, the cursor for the incremental
// collections listing, and the decrypted collection and file records. The
// whole file is read into RAM on load and rewritten as a whole on save; there
// is no partial update and no lock file. A separate refresh unit populates the
// store from the server — this module only stores what it is given.
//
// The file is a cache, so it is never trusted to exist or to be intact: a
// missing or unreadable file loads as an empty store rather than an error, and
// the refresh unit then repopulates it.
import { mkdir, chmod, readFile } from "node:fs/promises";
import { dirname } from "node:path";
import { writeAtomic } from "../download/index.js";
import type { Collection, EnteFile, Microseconds } from "../model/types.js";
// Bumped only when the on-disk shape changes incompatibly. A file written
// under a different version is discarded on load (see `load`): re-fetching
// from the server is always safe and cheaper than migrating a cache.
export const METADATA_SCHEMA_VERSION = 1;
// Directory and file modes match `session.json`: the records hold decrypted
// key material, so on a shared machine only the owner may read them.
const DIR_MODE = 0o700;
const FILE_MODE = 0o600;
// On-disk shapes. They mirror the in-memory model exactly except for the
// binary `key`, which JSON cannot hold and which is stored as base64.
type StoredCollection = Omit<Collection, "key"> & { key: string };
type StoredFile = Omit<EnteFile, "key"> & { key: string };
interface StoredMetadata {
schemaVersion: number;
userID: number;
collectionsSinceTime: Microseconds;
collections: StoredCollection[];
files: StoredFile[];
}
const encodeKey = (key: Uint8Array): string =>
Buffer.from(key).toString("base64");
const decodeKey = (encoded: string): Uint8Array =>
new Uint8Array(Buffer.from(encoded, "base64"));
// A file membership is identified by the pair (collectionID, fileID): the same
// underlying file can belong to several collections, each a distinct record
// with its own key.
const fileKey = (collectionID: number, fileID: number): string =>
`${collectionID}:${fileID}`;
export class MetadataStore {
readonly path: string;
readonly schemaVersion = METADATA_SCHEMA_VERSION;
userID = 0;
collectionsSinceTime: Microseconds = 0;
private readonly collections = new Map<number, Collection>();
private readonly files = new Map<string, EnteFile>();
private constructor(path: string) {
this.path = path;
}
// Load the store at `path`. A missing file, an unreadable one, unparseable
// contents, or a mismatched schema version all yield an empty store bound
// to that path — never a thrown error, because the file is only a cache.
static async load(path: string): Promise<MetadataStore> {
const store = new MetadataStore(path);
let raw: string;
try {
raw = await readFile(path, "utf8");
} catch {
return store;
}
try {
const parsed = JSON.parse(raw) as StoredMetadata;
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
return store;
}
store.userID = parsed.userID ?? 0;
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
for (const stored of parsed.collections ?? []) {
const collection: Collection = {
...stored,
key: decodeKey(stored.key),
};
store.collections.set(collection.id, collection);
}
for (const stored of parsed.files ?? []) {
const file: EnteFile = {
...stored,
key: decodeKey(stored.key),
};
store.files.set(fileKey(file.collectionID, file.id), file);
}
} catch {
// Any corruption discards the partial result: a half-read cache is
// worse than an empty one, since the refresh unit will rebuild it.
return new MetadataStore(path);
}
return store;
}
// Rewrite the whole file. The directory is created 0700 and the file left
// 0600; the write itself is the download layer's durable atomic writer
// (temp file, fsync, rename, dir fsync), so a reader never sees a partial
// file and a crash cannot leave a truncated one. There is no lock file and
// no `sync()` beyond the writer's own fsyncs.
async save(): Promise<void> {
const model: StoredMetadata = {
schemaVersion: METADATA_SCHEMA_VERSION,
userID: this.userID,
collectionsSinceTime: this.collectionsSinceTime,
collections: [...this.collections.values()].map((c) => ({
...c,
key: encodeKey(c.key),
})),
files: [...this.files.values()].map((f) => ({
...f,
key: encodeKey(f.key),
})),
};
const dir = dirname(this.path);
// chmod after mkdir so the mode is 0700 even when the directory
// already existed with a looser mode; mkdir alone would not tighten
// an existing directory.
await mkdir(dir, { recursive: true, mode: DIR_MODE });
await chmod(dir, DIR_MODE);
const payload = new TextEncoder().encode(
JSON.stringify(model, null, 2),
);
await writeAtomic(this.path, payload);
// The atomic writer's temp file inherits the default mode; tighten the
// renamed file to 0600. The 0700 directory already keeps other users
// out during the brief window before this runs.
await chmod(this.path, FILE_MODE);
}
getCollection(id: number): Collection | undefined {
return this.collections.get(id);
}
listCollections(): Collection[] {
return [...this.collections.values()];
}
putCollection(collection: Collection): void {
this.collections.set(collection.id, collection);
}
// Removing a collection also drops its file memberships: a file record is
// only meaningful as part of a collection the cache still knows about.
deleteCollection(id: number): void {
this.collections.delete(id);
for (const [key, file] of this.files) {
if (file.collectionID === id) {
this.files.delete(key);
}
}
}
getFile(collectionID: number, fileID: number): EnteFile | undefined {
return this.files.get(fileKey(collectionID, fileID));
}
listFiles(collectionID: number): EnteFile[] {
return [...this.files.values()].filter(
(f) => f.collectionID === collectionID,
);
}
putFile(file: EnteFile): void {
this.files.set(fileKey(file.collectionID, file.id), file);
}
deleteFile(collectionID: number, fileID: number): void {
this.files.delete(fileKey(collectionID, fileID));
}
}
+9 -2
View File
@@ -120,9 +120,16 @@ export const decryptFile = (
metadata, metadata,
magicMetadata, magicMetadata,
pubMagicMetadata, pubMagicMetadata,
file: { decryptionHeader: raw.file.decryptionHeader }, file: {
thumbnail: { decryptionHeader: raw.thumbnail.decryptionHeader }, decryptionHeader: raw.file.decryptionHeader,
size: raw.info?.fileSize,
},
thumbnail: {
decryptionHeader: raw.thumbnail.decryptionHeader,
size: raw.info?.thumbSize,
},
updationTime: raw.updationTime, updationTime: raw.updationTime,
isDeleted: raw.isDeleted,
}; };
}; };
+3
View File
@@ -48,6 +48,9 @@ export interface EnteFile {
file: FileBlob; file: FileBlob;
thumbnail: FileBlob; thumbnail: FileBlob;
updationTime: Microseconds; updationTime: Microseconds;
// Set from the diff row's flag. Live files decode with it absent/false;
// deleted rows are filtered out before decryptFile, so it is not set here.
isDeleted?: boolean;
} }
// The key material a logged-in client holds, everything needed to decrypt // The key material a logged-in client holds, everything needed to decrypt
+136 -1
View File
@@ -72,7 +72,11 @@ import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js";
import { ApiClient } from "../../src/api/client.js"; import { ApiClient } from "../../src/api/client.js";
import { ApiError, TruncatedStreamError } from "../../src/errors.js"; import { ApiError, TruncatedStreamError } from "../../src/errors.js";
import type { RetryOptions } from "../../src/retry.js"; import type { RetryOptions } from "../../src/retry.js";
import { downloadFile, downloadThumbnail } from "../../src/download/index.js"; import {
downloadFile,
downloadThumbnail,
writeAtomic,
} from "../../src/download/index.js";
import type { EnteFile, FileMetadata } from "../../src/model/types.js"; import type { EnteFile, FileMetadata } from "../../src/model/types.js";
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -101,17 +105,48 @@ const renameHook = vi.hoisted(() => ({
failWith: null as Error | null, failWith: null as Error | null,
})); }));
/**
* `open` is wrapped so the tests can observe the durability fsyncs the atomic
* writer performs — which are otherwise invisible: an fsync leaves no trace in
* the file's contents. Each `FileHandle.sync()` is recorded, and rename and
* sync events are appended to a single ordered `events` log so a test can pin
* the sequence "fsync the temp file, rename, fsync the directory" that makes a
* write survive a power cut. The flag the handle was opened with distinguishes
* the temp file (`w`) from its containing directory (`r`).
*/
const durabilityHook = vi.hoisted(() => ({
events: [] as string[],
}));
vi.mock("node:fs/promises", async (importOriginal) => { vi.mock("node:fs/promises", async (importOriginal) => {
const actual = await importOriginal<typeof import("node:fs/promises")>(); const actual = await importOriginal<typeof import("node:fs/promises")>();
const { existsSync: sourceExists } = await import("node:fs"); const { existsSync: sourceExists } = await import("node:fs");
return { return {
...actual, ...actual,
open: async (
path: Parameters<typeof actual.open>[0],
flags?: Parameters<typeof actual.open>[1],
...rest: unknown[]
): Promise<Awaited<ReturnType<typeof actual.open>>> => {
const handle = await actual.open(
path,
flags as Parameters<typeof actual.open>[1],
...(rest as []),
);
const realSync = handle.sync.bind(handle);
handle.sync = async (): Promise<void> => {
durabilityHook.events.push(`sync:${String(flags)}:${path}`);
await realSync();
};
return handle;
},
rename: async (from: string, to: string): Promise<void> => { rename: async (from: string, to: string): Promise<void> => {
renameHook.calls.push({ renameHook.calls.push({
from, from,
to, to,
sourceExisted: sourceExists(from), sourceExisted: sourceExists(from),
}); });
durabilityHook.events.push(`rename:${to}`);
if (renameHook.failWith !== null) { if (renameHook.failWith !== null) {
throw renameHook.failWith; throw renameHook.failWith;
} }
@@ -123,6 +158,7 @@ vi.mock("node:fs/promises", async (importOriginal) => {
beforeEach(() => { beforeEach(() => {
renameHook.calls.length = 0; renameHook.calls.length = 0;
renameHook.failWith = null; renameHook.failWith = null;
durabilityHook.events.length = 0;
}); });
let testDir: string; let testDir: string;
@@ -1061,3 +1097,102 @@ describe("download retries: corruption is not retried", () => {
expect(requests()).toBe(1); expect(requests()).toBe(1);
}); });
}); });
// ---------------------------------------------------------------------------
// Durable atomic writes
//
// `writeAtomic` is exported so the metadata store can reuse the same
// power-cut-safe write. Its durability is the point: the bytes and the new
// directory entry must both be on stable storage before it returns, so a crash
// immediately afterwards cannot resurrect an empty renamed file (#22 area 1).
// ---------------------------------------------------------------------------
describe("writeAtomic", () => {
it("fsyncs the temp file before the rename and the directory after", async () => {
const dir = mkdtempSync(join(testDir, "atomic-"));
const dest = join(dir, "durable.bin");
const bytes = patternBytes(2048, 71);
await writeAtomic(dest, bytes);
expect(readFileSync(dest)).toEqual(Buffer.from(bytes));
// The order is the durability contract: fsync the staged temp file so
// its contents are on disk, rename it into place, then fsync the
// directory so that new entry is on disk too. Do the directory fsync
// before the rename, or skip it, and a crash can lose the rename.
expect(durabilityHook.events).toHaveLength(3);
expect(durabilityHook.events[0]).toMatch(/^sync:w:.*\.tmp$/);
expect(durabilityHook.events[1]).toBe(`rename:${dest}`);
expect(durabilityHook.events[2]).toBe(`sync:r:${dir}`);
});
it("leaves no temp file behind when the write cannot be renamed", async () => {
const dir = mkdtempSync(join(testDir, "atomic-fail-"));
const dest = join(dir, "unrenamable.bin");
renameHook.failWith = new Error("simulated rename failure");
await expect(writeAtomic(dest, patternBytes(64, 72))).rejects.toThrow(
"simulated rename failure",
);
// The staged temp file was fsynced, then the rename failed; the cleanup
// path must remove it so a repeatedly failing write cannot fill the disk.
expect(existsSync(dest)).toBe(false);
expect(readdirSync(dir)).toEqual([]);
});
});
// ---------------------------------------------------------------------------
// Per-chunk progress
//
// Callers streaming a large file want bytes-written as it lands, not only the
// final total. The hook fires as decrypted plaintext accumulates; its values
// are non-decreasing and its last value is exactly `bytesWritten`.
// ---------------------------------------------------------------------------
describe.each(entryPoints)("$name progress", ({ name, download }) => {
it("reports monotonic progress ending at bytesWritten", async () => {
// The multi-chunk fixture pulls one full 4 MiB chunk and then a small
// final chunk, so the callback fires more than once and monotonicity is
// actually observable rather than trivially true for a single fire.
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
const outPath = join(
mkdtempSync(join(testDir, `${name}-progress-`)),
"p.bin",
);
const seen: number[] = [];
const result = await download(api, file, outPath, (bytesDone) => {
seen.push(bytesDone);
});
expect(seen.length).toBeGreaterThan(1);
for (let i = 1; i < seen.length; i++) {
expect(seen[i]!).toBeGreaterThan(seen[i - 1]!);
}
expect(seen[seen.length - 1]).toBe(result.bytesWritten);
expect(result.bytesWritten).toBe(multiChunk.plaintext.length);
});
it("downloads normally when no progress callback is given", async () => {
// The callback is optional and its absence must be side-effect-free:
// the download succeeds exactly as it does elsewhere in this file.
const plaintext = patternBytes(300, 73);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const { api, file } = fixtureFor(key, header, ciphertext);
const outPath = join(
mkdtempSync(join(testDir, `${name}-noprog-`)),
"n.bin",
);
const result = await download(api, file, outPath);
expect(result.bytesWritten).toBe(plaintext.length);
expectSameBytes(readFileSync(outPath), plaintext);
});
});
+246
View File
@@ -0,0 +1,246 @@
/**
* Tests for the on-disk JSON metadata store (`MetadataStore`).
*
* The store is the local cache the library keeps of the account's server
* state: one `metadata.json` file holding the user id, a schema version, the
* cursor for the incremental collections listing, and the decrypted
* collection and file records. The whole file is read into RAM on load and
* rewritten as a whole on save. A separate refresh unit (issue #42) is what
* populates it; this unit only stores.
*
* Four contracts are load-bearing and each is exercised below:
*
* 1. **Round-trip fidelity.** Everything put into the store — including the
* binary decryption keys, which JSON cannot hold directly and which the
* store base64-encodes — comes back byte-for-byte after a save and a fresh
* load. A cache that quietly dropped or mangled a field would hand the
* caller wrong keys or stale metadata.
*
* 2. **A missing or corrupt file loads as an empty store, never an error.**
* The file is only a cache: if it is absent (first run) or unreadable
* (interrupted write on an older build, disk corruption, hand-editing),
* the right answer is to start empty and let the refresh unit repopulate,
* not to crash the whole library.
*
* 3. **Writes are atomic and durable.** The store reuses the same
* fsync-before-rename atomic writer the download layer uses, so a reader
* never sees a half-written file and a crash cannot leave a truncated one.
* The observable consequence tested here is that a save leaves exactly the
* destination file behind — no temporary sibling — and that overwriting an
* existing store preserves a complete, re-loadable file.
*
* 4. **Permissions match `session.json`.** The directory is `0700` and the
* file is `0600`, because the records contain decrypted key material and
* must not be readable by other users on a shared machine.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import {
mkdtempSync,
rmSync,
readdirSync,
statSync,
writeFileSync,
mkdirSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
MetadataStore,
METADATA_SCHEMA_VERSION,
} from "../../src/library/store.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
// A representative decrypted collection, including a binary key and all three
// magic-metadata layers, so the round-trip test proves every field survives.
const sampleCollection = (): Collection => ({
id: 12345,
ownerID: 42,
key: new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8]),
name: "Holiday 2026",
type: "album",
updationTime: 1_700_000_000_000_000,
isShared: true,
magicMetadata: { visibility: 0 },
pubMagicMetadata: { subType: 0, coverID: 999 },
sharedMagicMetadata: { note: "shared with a friend" },
});
// A representative decrypted file membership: metadata, both blob headers, a
// binary key, a content hash, and file/thumbnail sizes.
const sampleFile = (): EnteFile => ({
id: 67890,
collectionID: 12345,
ownerID: 42,
key: new Uint8Array([9, 8, 7, 6, 5, 4, 3, 2, 1]),
metadata: {
title: "IMG_0001.jpg",
fileType: "image",
creationTime: 1_699_000_000_000_000,
modificationTime: 1_699_000_500_000_000,
latitude: 52.52,
longitude: 13.405,
hash: "sha256:deadbeef",
},
magicMetadata: { editedName: "sunset" },
pubMagicMetadata: { editedTime: 1_699_000_600_000_000 },
file: { decryptionHeader: "ZmlsZUhlYWRlcg==", size: 4_194_304 },
thumbnail: { decryptionHeader: "dGh1bWJIZWFkZXI=", size: 8192 },
updationTime: 1_700_000_100_000_000,
});
describe("MetadataStore", () => {
let dir: string;
let path: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-store-"));
// Deliberately nest the store one level below the temp dir so save()
// has to create its own directory and set its mode.
path = join(dir, "cache", "metadata.json");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("round-trips the whole model, keys and all", async () => {
const store = await MetadataStore.load(path);
store.userID = 42;
store.collectionsSinceTime = 1_700_000_000_000_000;
store.putCollection(sampleCollection());
store.putFile(sampleFile());
await store.save();
const reloaded = await MetadataStore.load(path);
expect(reloaded.userID).toBe(42);
expect(reloaded.collectionsSinceTime).toBe(1_700_000_000_000_000);
// The binary key must come back as the exact bytes, not a base64
// string or a plain object of numbered keys.
const collection = reloaded.getCollection(12345);
expect(collection).toEqual(sampleCollection());
expect(collection?.key).toBeInstanceOf(Uint8Array);
const file = reloaded.getFile(12345, 67890);
expect(file).toEqual(sampleFile());
expect(file?.key).toBeInstanceOf(Uint8Array);
expect(reloaded.listCollections()).toHaveLength(1);
expect(reloaded.listFiles(12345)).toHaveLength(1);
});
it("writes the declared schema version", async () => {
const store = await MetadataStore.load(path);
await store.save();
const reloaded = await MetadataStore.load(path);
expect(reloaded.schemaVersion).toBe(METADATA_SCHEMA_VERSION);
});
it("loads an empty store when the file is missing", async () => {
const store = await MetadataStore.load(path);
expect(store.userID).toBe(0);
expect(store.listCollections()).toEqual([]);
expect(store.getCollection(1)).toBeUndefined();
});
it("loads an empty store when the file is corrupt", async () => {
mkdirSync(join(dir, "cache"), { recursive: true });
writeFileSync(path, "{ this is not valid json ][");
const store = await MetadataStore.load(path);
expect(store.listCollections()).toEqual([]);
expect(store.listFiles(12345)).toEqual([]);
});
it("loads an empty store when the schema version does not match", async () => {
// A cache written by a future build with an incompatible schema is
// discarded rather than misread; the refresh unit repopulates it.
mkdirSync(join(dir, "cache"), { recursive: true });
writeFileSync(
path,
JSON.stringify({
schemaVersion: METADATA_SCHEMA_VERSION + 1,
userID: 42,
collectionsSinceTime: 0,
collections: [],
files: [],
}),
);
const store = await MetadataStore.load(path);
expect(store.userID).toBe(0);
expect(store.listCollections()).toEqual([]);
});
it("creates the directory 0700 and the file 0600", async () => {
const store = await MetadataStore.load(path);
store.putCollection(sampleCollection());
await store.save();
// Directory 0700, file 0600: on a shared machine the decrypted keys
// in this file must be readable only by their owner. Mask to the
// permission bits; the file-type bits are not part of the assertion.
expect(statSync(join(dir, "cache")).mode & 0o777).toBe(0o700);
expect(statSync(path).mode & 0o777).toBe(0o600);
});
it("leaves exactly the destination behind, with no temp sibling", async () => {
const store = await MetadataStore.load(path);
store.putCollection(sampleCollection());
await store.save();
// The atomic writer stages a temporary file and renames it into
// place; on success nothing temporary is left in the directory.
expect(readdirSync(join(dir, "cache"))).toEqual(["metadata.json"]);
});
it("overwrites an existing store atomically and stays re-loadable", async () => {
const first = await MetadataStore.load(path);
first.userID = 1;
first.putCollection(sampleCollection());
await first.save();
const second = await MetadataStore.load(path);
second.userID = 2;
second.deleteCollection(12345);
await second.save();
const reloaded = await MetadataStore.load(path);
expect(reloaded.userID).toBe(2);
expect(reloaded.getCollection(12345)).toBeUndefined();
expect(readdirSync(join(dir, "cache"))).toEqual(["metadata.json"]);
});
it("deletes a collection together with its file memberships", async () => {
const store = await MetadataStore.load(path);
store.putCollection(sampleCollection());
store.putFile(sampleFile());
store.deleteCollection(12345);
expect(store.getCollection(12345)).toBeUndefined();
expect(store.getFile(12345, 67890)).toBeUndefined();
expect(store.listFiles(12345)).toEqual([]);
});
it("scopes file records to their collection membership", async () => {
// The same underlying file can be a member of two collections, each a
// separate record with its own key. Storing one must not touch the
// other, and lookups are per membership.
const store = await MetadataStore.load(path);
const inA = sampleFile();
const inB: EnteFile = {
...sampleFile(),
collectionID: 55555,
key: new Uint8Array([100, 101, 102]),
};
store.putFile(inA);
store.putFile(inB);
expect(store.getFile(12345, 67890)?.key).toEqual(inA.key);
expect(store.getFile(55555, 67890)?.key).toEqual(inB.key);
expect(store.listFiles(12345)).toHaveLength(1);
expect(store.listFiles(55555)).toHaveLength(1);
store.deleteFile(12345, 67890);
expect(store.getFile(12345, 67890)).toBeUndefined();
expect(store.getFile(55555, 67890)?.key).toEqual(inB.key);
});
});
+50 -1
View File
@@ -145,7 +145,12 @@ const buildSharedRawCollection = (
const buildRawFile = ( const buildRawFile = (
collectionKey: Uint8Array, collectionKey: Uint8Array,
opts?: { title?: string; fileType?: number; creationTime?: number }, opts?: {
title?: string;
fileType?: number;
creationTime?: number;
info?: { fileSize?: number; thumbSize?: number };
},
): RawEnteFile => { ): RawEnteFile => {
const fileKey = sodium.crypto_secretbox_keygen(); const fileKey = sodium.crypto_secretbox_keygen();
const { ciphertext: encFileKey, nonce: fileKeyNonce } = secretboxEncrypt( const { ciphertext: encFileKey, nonce: fileKeyNonce } = secretboxEncrypt(
@@ -186,6 +191,7 @@ const buildRawFile = (
}, },
file: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) }, file: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
thumbnail: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) }, thumbnail: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
info: opts?.info,
updationTime: 1700000000000000, updationTime: 1700000000000000,
}; };
}; };
@@ -357,4 +363,47 @@ describe("model.decryptFile", () => {
expect(() => decryptFile(raw, wrongKey)).toThrow(); expect(() => decryptFile(raw, wrongKey)).toThrow();
}); });
it("carries the file and thumbnail byte sizes from info", () => {
// The server reports the encrypted-blob sizes in `info`; the cache
// needs them without a HEAD request, so decryptFile must copy them
// onto the file and thumbnail blobs.
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const raw = buildRawFile(collectionKey, {
info: { fileSize: 4096, thumbSize: 512 },
});
const file = decryptFile(raw, collectionKey);
expect(file.file.size).toBe(4096);
expect(file.thumbnail.size).toBe(512);
});
it("leaves the sizes undefined when the server omits info", () => {
// Older files predate the info field; the sizes must stay undefined
// rather than become 0, so callers can tell "unknown" from "empty".
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const raw = buildRawFile(collectionKey);
expect(raw.info).toBeUndefined();
const file = decryptFile(raw, collectionKey);
expect(file.file.size).toBeUndefined();
expect(file.thumbnail.size).toBeUndefined();
});
it("carries the deletion flag from the diff row", () => {
// The diff marks a deleted row with isDeleted; decryptFile copies it
// onto the file so a caller can tell a deleted row from a live one.
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const raw = buildRawFile(collectionKey);
raw.isDeleted = true;
const file = decryptFile(raw, collectionKey);
expect(file.isDeleted).toBe(true);
});
}); });
+16 -1
View File
@@ -12,7 +12,7 @@
// //
// Neither shows up as a build failure, so they are asserted here. // Neither shows up as a build failure, so they are asserted here.
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs"; import { existsSync, readFileSync } from "node:fs";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { join } from "node:path"; import { join } from "node:path";
@@ -46,4 +46,19 @@ describe(".dockerignore", () => {
it("leaves .gitignore in the build context for prettier", () => { it("leaves .gitignore in the build context for prettier", () => {
expect(dockerignore).not.toContain(".gitignore"); expect(dockerignore).not.toContain(".gitignore");
}); });
// Both images are built from this same context, and the lint image runs
// eslint and prettier across it. BuildKit lets a `<dockerfile>.dockerignore`
// shadow the root one for a single build; such a file would silently give
// the lint build a different, unreviewed context — and eslint's flat config
// does not ignore dot-directories, so a stray `.claude/` worktree would be
// linted.
it.each(["Dockerfile", "Dockerfile.lint"])(
"is not shadowed by a per-Dockerfile ignore file for %s",
(name) => {
expect(existsSync(join(repoRoot, `${name}.dockerignore`))).toBe(
false,
);
},
);
}); });
+4 -2
View File
@@ -1,7 +1,9 @@
// The package manifest promises three files that only exist after a build: // The package manifest promises three files that only exist after a build:
// `main`, `types`, and the `quak` binary. Nothing in the test suite used to // `main`, `types`, and the `quak` binary. Nothing in the test suite used to
// look at them, and `make check` runs test, lint and fmt-check but never the // look at them, and `make check` runs the suite and the lint container but
// build, so `tsconfig.json` and `package.json` were free to drift apart. They // never the build, so `tsconfig.json` and `package.json` were free to drift
// apart. (The formatting check is part of the lint container, not a step of
// its own; `test/packaging/lint-once.test.ts` is what holds that shape.) They
// did: `rootDir` was `./src` while `include` also pulled in `bin/**/*`, which // did: `rootDir` was `./src` while `include` also pulled in `bin/**/*`, which
// is TS6059, and no build had succeeded for as long as that was true. // is TS6059, and no build had succeeded for as long as that was true.
// //
+184
View File
@@ -0,0 +1,184 @@
// Linting runs in Docker, one way, everywhere: `script/lint` builds
// `Dockerfile.lint`, which COPYs the repo into a digest-pinned image and runs
// eslint and prettier as build steps, so a successful build IS a clean lint.
//
// Three things can quietly undo that, and none of them shows up as a build
// failure, which is why they are asserted here:
//
// 1. Recursion. `script/check` calls `script/lint`, and `script/lint` is now a
// `docker build`. Anything that runs `make check` inside a container is
// therefore asking for Docker inside Docker, and CI breaks. The image built
// from `Dockerfile` runs the suite and the compile only; lint happens once,
// in `Dockerfile.lint`.
// 2. Cache. A lint build over an unchanged tree returns success in well under a
// second having linted nothing. The `LINT_EPOCH` guard is what forces the
// linter layers to execute, and it has to fail closed: an unset build
// argument is the empty string, which is a perfectly stable cache key, so an
// invocation that omits it must be rejected rather than served a cached
// green.
// 3. A host lint path surviving alongside the container one, which would let a
// lint result come from an unpinned local toolchain.
import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { join } from "node:path";
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
const read = (name: string): string =>
readFileSync(join(repoRoot, name), "utf-8");
// The executable lines of a shell script or Dockerfile: comments carry the
// reasoning and frequently name the very commands these tests forbid, so they
// would otherwise trigger every assertion below.
const instructions = (name: string): string[] =>
read(name)
.split("\n")
.map((line) => line.trim())
.filter((line) => line !== "" && !line.startsWith("#"));
const lintScript = instructions("script/lint");
const dockerfileLint = instructions("Dockerfile.lint");
const dockerfile = instructions("Dockerfile");
const cibuild = instructions("script/cibuild");
const has = (lines: string[], pattern: RegExp): boolean =>
lines.some((line) => pattern.test(line));
describe("script/lint", () => {
it("lints by building Dockerfile.lint", () => {
expect(has(lintScript, /docker build .*-f Dockerfile\.lint/)).toBe(
true,
);
});
// The whole point of the ruling: no invocation of a linter against the
// working tree survives, so a lint verdict can only come from the pinned
// image.
it("runs no linter on the host", () => {
expect(has(lintScript, /eslint|prettier/)).toBe(false);
});
// Without a fresh epoch the build is served from cache in under a second,
// having linted nothing, and still exits 0.
it("passes a fresh LINT_EPOCH on every run", () => {
expect(
has(lintScript, /--build-arg LINT_EPOCH="\$\(date \+%s\)"/),
).toBe(true);
});
});
describe("Dockerfile.lint", () => {
// Tag references are server-mutable, so they are remote code execution.
it("pins its base image by digest", () => {
expect(has(dockerfileLint, /^FROM \S+@sha256:[0-9a-f]{64}/)).toBe(true);
});
it("runs eslint as a build step", () => {
expect(has(dockerfileLint, /^RUN .*eslint \./)).toBe(true);
});
it("runs prettier as a build step", () => {
expect(has(dockerfileLint, /^RUN .*prettier --check \./)).toBe(true);
});
// An unset ARG is the empty string, and an empty string is a perfectly
// stable cache key. Rejecting it is what stops a bare
// `docker build -f Dockerfile.lint .` from reporting a green it did not
// earn.
it("refuses to build without LINT_EPOCH", () => {
expect(has(dockerfileLint, /^ARG LINT_EPOCH$/)).toBe(true);
expect(
has(dockerfileLint, /^RUN \[ -n "\$LINT_EPOCH" \] \|\| exit 1$/),
).toBe(true);
});
// The guard only forces execution of the layers below it, so both linters
// have to sit after it. Layer order is the mechanism, not a style choice.
it("puts both linters below the epoch guard", () => {
const guard = dockerfileLint.findIndex((line) =>
/^RUN \[ -n "\$LINT_EPOCH" \]/.test(line),
);
const linters = dockerfileLint
.map((line, index) => ({ line, index }))
.filter(({ line }) => /^RUN .*(eslint|prettier)/.test(line));
expect(linters.length).toBeGreaterThan(0);
for (const { line, index } of linters) {
expect(
index,
`${line} must run below the LINT_EPOCH guard`,
).toBeGreaterThan(guard);
}
});
// Dependency installation is the slow layer and has nothing to do with the
// sources, so it caches separately: manifests first, sources afterwards.
it("copies the manifests before the sources", () => {
const manifests = dockerfileLint.findIndex((line) =>
/^COPY package\.json yarn\.lock/.test(line),
);
const sources = dockerfileLint.findIndex((line) =>
/^COPY \. \.$/.test(line),
);
expect(manifests).toBeGreaterThanOrEqual(0);
expect(sources).toBeGreaterThan(manifests);
});
// script/lint is a docker build; a lint step that shelled out to it would
// recurse.
it("does not call script/lint or make lint", () => {
expect(has(dockerfileLint, /make lint|script\/lint/)).toBe(false);
});
});
describe("Dockerfile", () => {
// `make check` runs script/lint, which is a docker build, so an image that
// ran it would need a Docker daemon inside the container.
it("does not run make check, make lint or script/lint", () => {
expect(
has(dockerfile, /make check|make lint|script\/(check|lint)/),
).toBe(false);
});
// The replaced lint stage took a `COPY --from=lint` dependency to order
// itself before the check stage. Dockerfile.lint is that stage now, and
// two definitions of how to lint is one too many.
it("has no lint stage", () => {
expect(has(dockerfile, /AS lint\b|--from=lint\b/)).toBe(false);
});
it("still runs the suite and the build under the epoch guard", () => {
expect(has(dockerfile, /^RUN make test$/)).toBe(true);
expect(has(dockerfile, /^RUN make build$/)).toBe(true);
expect(
has(dockerfile, /^RUN \[ -n "\$CHECK_EPOCH" \] \|\| exit 1$/),
).toBe(true);
});
});
describe("script/cibuild", () => {
// CI has to get both verdicts. Lint goes first so the fast failure is
// reported before the suite runs.
it("builds the lint image before the test and build image", () => {
const lint = cibuild.findIndex((line) => /\/lint"/.test(line));
const check = cibuild.findIndex((line) =>
/docker build .*CHECK_EPOCH/.test(line),
);
expect(lint).toBeGreaterThanOrEqual(0);
expect(check).toBeGreaterThan(lint);
});
});
describe("package.json", () => {
// `yarn lint` was a second, unpinned way to get a lint verdict, from
// whatever eslint the working tree happened to have installed.
it("exposes no host lint script", () => {
const pkg = JSON.parse(read("package.json")) as {
scripts: Record<string, string>;
};
expect(pkg.scripts.lint).toBeUndefined();
});
});
+503
View File
@@ -0,0 +1,503 @@
// `make check` used to run `prettier --check .` twice: once inside the lint
// container (`script/lint` builds `Dockerfile.lint`, which runs eslint and
// prettier as build steps) and once again on the host, because `script/check`
// also called `script/fmt-check`. Two passes, one verdict, and the host one is
// the weaker of the two — its prettier is whatever the working tree happens to
// have installed, while the container's is digest-pinned and installed under
// `--frozen-lockfile`.
//
// The fix was to delete the host call from `script/check` and `script/precommit`.
// Nothing about that fix is self-enforcing: anyone can wire `script/fmt-check`
// back in, or add a prettier step to a Dockerfile, and every build stays green
// while quietly doing the work twice again. So the count is asserted here
// rather than promised in a comment.
//
// The assertion is a static walk of the invocation graph, not a string match
// against one file. Starting from an entrypoint, it follows every edge the repo
// actually uses to reach another command — `run:` steps in the CI workflow,
// `"$SCRIPT_DIR/<name>"` and `script/<name>` into other scripts, `make <target>`
// through the Makefile shims, `yarn run <name>` through the `package.json`
// scripts, and `docker build -f <file>` into that Dockerfile's `RUN` steps — and
// counts the prettier invocations it finds. A prettier call added anywhere in
// that graph is therefore caught, wherever it is added.
//
// Two entrypoints are walked, because they cover different graphs: `make check`
// is what a developer runs, and `.gitea/workflows/check.yml` is what CI runs.
// The CI walk starts at the workflow file rather than at a hand-picked script,
// so "the path CI executes" is read out of the repo instead of assumed; it
// reaches `script/cibuild`, and through it the `Dockerfile` image that `make
// check` never touches. Walking only `make check` is how a duplicate prettier
// pass in `Dockerfile` stayed invisible.
//
// Undercounting is the failure mode that would make this test worthless. Three
// things guard against it: the walk is asserted to have reached the nodes that
// matter, an unresolvable or empty node is a thrown error rather than a quiet
// zero, and prettier is counted per occurrence rather than per line, so two
// invocations chained with `&&` cannot read as one.
import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { join } from "node:path";
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
const read = (name: string): string =>
readFileSync(join(repoRoot, name), "utf-8");
// A backslash at end of line continues the command; the resolver has to see the
// whole invocation, since the interesting flags (`-f Dockerfile.lint`) can sit
// on the continuation.
const joinContinuations = (text: string): string[] => {
const joined: string[] = [];
for (const raw of text.split("\n")) {
const line = raw.trim();
const previous = joined[joined.length - 1];
if (previous !== undefined && previous.endsWith("\\")) {
joined[joined.length - 1] =
`${previous.slice(0, -1).trim()} ${line}`;
} else {
joined.push(line);
}
}
return joined;
};
// Comments are stripped everywhere. The headers of these scripts explain the
// duplication this test exists to prevent, and therefore name `prettier` and
// `script/fmt-check` repeatedly; counting them would make the test assert the
// prose instead of the behaviour.
const executable = (text: string): string[] =>
joinContinuations(text).filter(
(line) => line !== "" && !line.startsWith("#"),
);
// Every occurrence, not "does this line mention prettier": a line that reads
// `yarn run prettier --check . && yarn run prettier --check src` is two passes
// over the same tree, which is exactly the bug this file exists to catch, and
// counting it as one would hide it. `.prettierrc` and `.prettierignore` are not
// invocations and do not match, because `\b` requires a non-word character
// after the name.
const countPrettier = (line: string): number =>
(line.match(/\bprettier\b/g) ?? []).length;
// Makefile targets are thin shims (`check:` / tab / `@script/check`), so a
// `make <target>` edge has to resolve through them to keep "per `make check`"
// meaning what it says. Recipe lines are the tab-indented ones.
const makeRecipes = (): Map<string, string[]> => {
const recipes = new Map<string, string[]>();
let current: string | null = null;
for (const raw of read("Makefile").split("\n")) {
if (raw.startsWith("\t")) {
if (current !== null) {
recipes.get(current)?.push(raw.trim().replace(/^[@-]+/, ""));
}
continue;
}
const target = /^([a-z][a-z-]*)\s*:(?!=)/.exec(raw);
current = target === null ? null : target[1];
if (current !== null && !recipes.has(current)) {
recipes.set(current, []);
}
}
return recipes;
};
const recipes = makeRecipes();
const packageScripts = (): Record<string, string> => {
const pkg = JSON.parse(read("package.json")) as {
scripts?: Record<string, string>;
};
return pkg.scripts ?? {};
};
const scripts = packageScripts();
// Node keys: `script/<name>`, `docker:<Dockerfile>`, `make:<target>`,
// `yarn:<package.json script>`, `workflow:<CI workflow file>`.
const resolve = (node: string): string[] => {
if (node.startsWith("script/")) return executable(read(node));
if (node.startsWith("docker:")) {
return executable(read(node.slice("docker:".length)))
.filter((line) => line.startsWith("RUN "))
.map((line) => line.slice("RUN ".length));
}
// The `run:` steps of a workflow, in file order. `uses:` steps are actions,
// not commands, and have no edges into this repo's graph. A `run: |` block
// would resolve to the bare `|`, which reaches nothing and therefore fails
// the count rather than passing quietly.
if (node.startsWith("workflow:")) {
return executable(read(node.slice("workflow:".length)))
.filter((line) => /^-?\s*run:\s*\S/.test(line))
.map((line) => line.replace(/^-?\s*run:\s*/, ""));
}
if (node.startsWith("make:")) {
const target = node.slice("make:".length);
const recipe = recipes.get(target);
// A renamed or deleted target must be a loud failure: silently walking
// an empty recipe would report zero prettier invocations, which reads
// like the tidiest possible result.
if (recipe === undefined) {
throw new Error(`no such Makefile target: ${target}`);
}
return recipe;
}
if (node.startsWith("yarn:")) {
const name = node.slice("yarn:".length);
const script = scripts[name];
if (script === undefined) {
throw new Error(`no such package.json script: ${name}`);
}
return [script];
}
throw new Error(`unresolvable node: ${node}`);
};
// Same reasoning as the missing-target error, applied to every node kind: a
// node that resolves to no commands contributes zero prettier invocations and
// zero edges, which is indistinguishable from a clean result. Fail instead.
const commandsOf = (node: string): string[] => {
const commands = resolve(node);
if (commands.length === 0) {
throw new Error(`node resolved to no commands: ${node}`);
}
return commands;
};
const edgesOf = (line: string): string[] => {
const edges: string[] = [];
// `"$SCRIPT_DIR/lint"`, `"$ROOT/script/lint"` and a bare `script/lint` are
// all the same edge.
for (const match of line.matchAll(
/(?:\$SCRIPT_DIR|\$\{SCRIPT_DIR\}|script)\/([a-z][a-z-]*)/g,
)) {
edges.push(`script/${match[1]}`);
}
// Only real targets: `pkg_install gnumake make make make` in
// script/bootstrap is a package name, not an invocation of this Makefile.
for (const match of line.matchAll(/\bmake\s+([a-z][a-z-]*)/g)) {
if (recipes.has(match[1] ?? "")) edges.push(`make:${match[1]}`);
}
// Same rule for yarn: `yarn run prettier` is the linter itself (counted,
// not followed), `yarn run fmt-check` would be a package.json script that
// runs it indirectly.
for (const match of line.matchAll(/\byarn(?:\s+run)?\s+([a-z][a-z-]*)/g)) {
if ((match[1] ?? "") in scripts) edges.push(`yarn:${match[1]}`);
}
// The container lint pass lives behind a `docker build`; without following
// it the count would miss the one invocation that is supposed to survive.
if (/\bdocker\s+build\b/.test(line)) {
const file = /\s-f\s+(\S+)/.exec(line);
edges.push(`docker:${file === null ? "Dockerfile" : file[1]}`);
}
return edges;
};
interface Walk {
prettier: number;
reached: Set<string>;
}
// Repeated invocations must count repeatedly — running the same script twice is
// exactly the bug — so nodes are not deduplicated. The path stack is only there
// to turn a cycle into a loud failure instead of a hang.
//
// Counting and edge-following both happen for every line: a line that invokes
// prettier can also invoke something else, and skipping the edges of counted
// lines silently truncated the graph.
const walk = (node: string, path: string[] = [], into?: Walk): Walk => {
const result = into ?? { prettier: 0, reached: new Set<string>() };
if (path.includes(node)) {
throw new Error(`invocation cycle: ${[...path, node].join(" -> ")}`);
}
result.reached.add(node);
for (const line of commandsOf(node)) {
result.prettier += countPrettier(line);
for (const edge of edgesOf(line)) {
walk(edge, [...path, node], result);
}
}
return result;
};
describe("prettier runs exactly once per make check", () => {
const check = walk("make:check");
// The headline assertion, and the one the issue is about.
it("invokes prettier once for the whole of make check", () => {
expect(check.prettier).toBe(1);
});
// Guards against the count being 1 (or 0) because the walk never got
// anywhere. `make check` has to reach the suite, the lint script, and the
// Dockerfile whose build IS the lint verdict.
it.each(["script/check", "script/test", "script/lint", "Dockerfile.lint"])(
"reaches %s while counting",
(node) => {
const key = node.startsWith("script/") ? node : `docker:${node}`;
expect([...check.reached]).toContain(key);
},
);
// The one that survives is the container's, not the host's: that is the
// authoritative verdict, since a successful Dockerfile.lint build is what
// CI treats as proof of a clean tree.
it("keeps the surviving invocation inside the lint container", () => {
expect(walk("docker:Dockerfile.lint").prettier).toBe(1);
});
it("does not reach the host formatting check from make check", () => {
expect([...check.reached]).not.toContain("script/fmt-check");
});
});
describe("prettier runs exactly once per CI build", () => {
// Rooted at the workflow file, so this is the graph CI executes rather than
// the graph someone believed CI executes. `make check` cannot stand in for
// it: CI runs script/cibuild, which builds Dockerfile as well as
// Dockerfile.lint, and nothing under `make check` ever reads Dockerfile.
const ci = walk("workflow:.gitea/workflows/check.yml");
it("invokes prettier once for the whole CI build", () => {
expect(ci.prettier).toBe(1);
});
// script/cibuild is here because the workflow is asserted to run it;
// Dockerfile is here because it is the half of the CI graph that the
// `make check` walk cannot see.
it.each([
"script/cibuild",
"script/lint",
"docker:Dockerfile.lint",
"docker:Dockerfile",
])("reaches %s while counting", (node) => {
expect([...ci.reached]).toContain(node);
});
// The test and build image must not lint: linting is Dockerfile.lint's job,
// and a prettier step added here would be a second pass over the same tree
// for the same verdict — on the one path where it matters most.
it("keeps prettier out of the test and build image", () => {
expect(walk("docker:Dockerfile").prettier).toBe(0);
});
});
describe("the standalone entrypoints still do what their names say", () => {
// REPO_POLICIES.md requires both `make lint` and `make fmt-check` to exist
// and mean something. Dropping fmt-check from script/check must not turn it
// into a target nobody can use, and must not leave `make check` passing
// because both halves became no-ops.
it("still checks formatting under make fmt-check", () => {
expect(walk("make:fmt-check").prettier).toBe(1);
});
it("still checks formatting under make lint", () => {
expect(walk("make:lint").prettier).toBe(1);
});
});
describe("script/precommit", () => {
// Same duplication as script/check, same fix. The hook still catches a
// badly formatted tree before the commit lands, because script/lint is the
// container prettier run — that is the whole reason the host call could go.
it("checks formatting exactly once", () => {
expect(walk("script/precommit").prettier).toBe(1);
});
it("gets that check from the lint container", () => {
expect([...walk("script/precommit").reached]).toContain(
"docker:Dockerfile.lint",
);
});
});
// script/bootstrap installs the dependencies, and it has two install sites: one
// for the case where yarn has to be reached through nvm, and one for the case
// where yarn is already on PATH. A substring check against the whole file
// cannot tell them apart, so it reports the first and says nothing about the
// second — which is the one the containers take, because the pinned node image
// ships yarn. Both are resolved separately here.
const installBranches = (): { withoutYarn: string[]; withYarn: string[] } => {
const lines = executable(read("script/bootstrap"));
const open = lines.findIndex((line) =>
/^install_js_deps\s*\(\)/.test(line),
);
if (open === -1) {
throw new Error("script/bootstrap: no install_js_deps function");
}
const close = lines.indexOf("}", open);
const body = lines.slice(open + 1, close === -1 ? undefined : close);
const guard = body.findIndex((line) =>
/^if\b.*\bmissing yarn\b/.test(line),
);
const otherwise = body.indexOf("else", guard);
const end = body.indexOf("fi", otherwise);
if (guard === -1 || otherwise === -1 || end === -1) {
throw new Error(
"script/bootstrap: install_js_deps is not the expected " +
"if missing yarn / else / fi shape",
);
}
return {
withoutYarn: body.slice(guard + 1, otherwise),
withYarn: body.slice(otherwise + 1, end),
};
};
// Every `yarn install` in the given lines, with its flags, so an unpinned
// install cannot hide next to a pinned one.
const yarnInstalls = (lines: string[]): string[] =>
lines.flatMap((line) =>
[...line.matchAll(/\byarn install\b[^"'&|;]*/g)].map((match) =>
match[0].trim(),
),
);
describe("host and container prettier cannot disagree", () => {
// With the host pass gone from `make check`, `make fmt-check` is the only
// host-side formatting check left, and the container is the gate. The two
// must keep producing the same verdict on the same tree, or a developer
// running `make fmt-check` gets a green that CI then rejects.
//
// Three things make them agree, and all three are load-bearing:
it("pins the same prettier for both", () => {
const pkg = JSON.parse(read("package.json")) as {
devDependencies: Record<string, string>;
};
// An exact version, not a range: `^3.8.1` would let the container and
// the host resolve different builds with different formatting.
expect(pkg.devDependencies.prettier).toMatch(/^\d+\.\d+\.\d+$/);
});
it("installs from the lockfile on the branch the container takes", () => {
// Both images are FROM a node image, which ships yarn, so `missing
// yarn` is false and this is the branch that runs in the container.
const installs = yarnInstalls(installBranches().withYarn);
expect(installs).not.toHaveLength(0);
for (const install of installs) {
expect(install).toContain("--frozen-lockfile");
}
});
it("installs from the lockfile on the nvm branch too", () => {
// Not the container's branch, but it is the one a developer without
// yarn on PATH gets, and their prettier has to match the container's.
const installs = yarnInstalls(installBranches().withoutYarn);
expect(installs).not.toHaveLength(0);
for (const install of installs) {
expect(install).toContain("--frozen-lockfile");
}
});
it("runs script/bootstrap inside the lint container", () => {
// Without this the lockfile assertions above would be about a script
// the container never executes.
expect([...walk("docker:Dockerfile.lint").reached]).toContain(
"script/bootstrap",
);
});
it("keeps .gitignore in the build context", () => {
// Prettier 3 reads .gitignore as a default ignore file, so excluding it
// from the context would change which files the container checks.
const dockerignore = read(".dockerignore")
.split("\n")
.map((line) => line.trim());
expect(dockerignore).not.toContain(".gitignore");
});
});
describe("the walk cannot pass vacuously", () => {
// An earlier draft of this file computed a Makefile target as
// `node.slice("make:")` — a string where a number belongs, which coerces to
// NaN and made every target resolve to nothing. The count went to zero and
// an assertion of "not twice" would have been satisfied by a walk that had
// read nothing at all. Every way of reaching nothing is therefore an
// error here, and the ways are tested rather than assumed.
it("reports zero for a subgraph that does not run prettier", () => {
expect(walk("make:clean").prettier).toBe(0);
});
it("refuses a Makefile target that does not exist", () => {
expect(() => walk("make:no-such-target")).toThrow(
/no such Makefile target/,
);
});
it("refuses a package.json script that does not exist", () => {
expect(() => walk("yarn:no-such-script")).toThrow(
/no such package.json script/,
);
});
it("refuses a script that does not exist", () => {
expect(() => walk("script/no-such-script")).toThrow(/ENOENT/);
});
it("refuses a node that resolves to no commands", () => {
// .dockerignore has no RUN steps, standing in for a Dockerfile whose
// steps a restructure moved somewhere the resolver cannot see.
expect(() => walk("docker:.dockerignore")).toThrow(
/resolved to no commands/,
);
});
it("refuses a node kind it does not understand", () => {
expect(() => walk("nonsense")).toThrow(/unresolvable node/);
});
it("refuses to walk in circles", () => {
expect(() => walk("make:check", ["script/check"])).toThrow(
/invocation cycle/,
);
});
});
describe("the resolver reads what the shell would run", () => {
// Counting per line is how `yarn run prettier --check . && yarn run
// prettier --check src` read as a single invocation.
it("counts every prettier invocation on a line", () => {
expect(
countPrettier(
"yarn run prettier --check . && yarn run prettier --check src",
),
).toBe(2);
});
it("does not count the config files as invocations", () => {
expect(countPrettier("COPY .prettierrc .prettierignore ./")).toBe(0);
});
// The counting `continue` also dropped every edge that shared a line with a
// prettier call, so a whole subtree could be hidden behind one `&&`.
it("still follows the edges of a line that invokes prettier", () => {
expect(
edgesOf('yarn run prettier --check . && "$SCRIPT_DIR/lint"'),
).toContain("script/lint");
});
it("resolves every spelling of a script call to one node", () => {
expect(
edgesOf('"$SCRIPT_DIR/lint" "${SCRIPT_DIR}/test" script/fmt'),
).toEqual(["script/lint", "script/test", "script/fmt"]);
});
it("follows a bare docker build to Dockerfile and -f to its file", () => {
expect(edgesOf("docker build .")).toContain("docker:Dockerfile");
expect(edgesOf("docker build -f Dockerfile.lint .")).toContain(
"docker:Dockerfile.lint",
);
});
it("reads the run steps of the CI workflow and not its uses steps", () => {
expect(commandsOf("workflow:.gitea/workflows/check.yml")).toEqual([
"script/cibuild",
]);
});
});