Make lint and test phases of the Dockerfile (closes #96)

Follows the template: Dockerfile.lint is gone; the Dockerfile has a lint
phase (eslint, prettier --check .) and a test phase (vitest, run as the
node user so the not-writable-directory tests are not skipped), and its
last stage compiles and depends on both. script/lint and script/test
build one phase each with --no-cache; script/docker and script/cibuild
pass --no-cache, so CHECK_EPOCH and LINT_EPOCH are removed.
script/cibuild is the single image build, so CI runs lint and the tests
once each. The tests that checked the old layout are deleted,
REPO_POLICIES.md is re-copied and the README describes the new layout.

Model: opus-5-5

Dockerfile

@@ -33,7 +33,7 @@ COPY . .
 
 # Unlike the template, the suite runs as the image's non-root `node` user:
 # root ignores directory permissions, so the tests of a destination that is
-# not writable would otherwise fail. vitest writes into /app.
+# not writable would otherwise be skipped. vitest writes into /app.
 RUN chown -R node:node /app
 USER node
 

README.md

@@ -475,20 +475,6 @@ failure still exits non-zero.
         <name>.json             collection metadata + file list
 ```
 
-A collection's directory and JSON are named after the collection, and a symlink
-after the file's title, both with unsafe characters replaced. When two
-collections would get the same name, or two files in one collection the same
-title (ignoring case in both), each of them gets its ID added: two albums named
-`Trip` become `Trip (10)/` and `Trip (11)/`, and two files titled `IMG_0001.JPG`
-become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. IDs never change, so a
-name stays the same from run to run until such a clash appears or goes away.
-
-Each run removes the symlinks into `originals/` that no longer belong in their
-collection's directory, and the directories (and JSON) of collections that were
-deleted or renamed. Nothing else in `collections/` is touched: a file or a
-symlink you put there stays, and a directory that still holds one after its
-symlinks are removed stays too, with its JSON.
-
 Each file is downloaded exactly once regardless of how many collections it
 appears in. On subsequent runs, existing originals are skipped. If a download
 fails, the error is logged and the backup continues with the next file. The exit
@@ -679,13 +665,10 @@ originals and thumbnails are kept; they are reached only through the files the
 current account's records name.
 
 A stored file appears only via an atomic temp-then-rename, so its presence means
-it is complete. Every downloaded original (by `quak get`, the cache, or
+it is complete. The design also calls for a content-hash comparison against
-`backup`) whose metadata records a content hash (`FileMetadata.hash`) is hashed
+`FileMetadata.hash` on each fetched original; that check is deferred (issue
-as it is written: unkeyed BLAKE2b with a 64-byte output, standard base64. For a
+https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
-live photo, which is stored as a ZIP, the image and the video are hashed
+cannot yet be confirmed against the repo's fixtures.
-separately and joined as `<imageHash>:<videoHash>`. A mismatch stores nothing
-and fails the download with an error naming the file ID. An original with no
-recorded hash, from a very old client, is stored unchecked.
 
 ### Key types by source file
 

TODO.md

@@ -24,21 +24,6 @@ Tag v1.0.0.
   stage compiles and depends on both, so `script/cibuild` is one build.
   `Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
   are gone; `REPO_POLICIES.md` is re-copied.
-- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
-  in one collection with the same title, and two collections with the same name,
-  each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
-  none replaces another's symlink or JSON. Each run removes symlinks into
-  `originals/` for files no longer in the collection, and the folders of deleted
-  or renamed collections, leaving anything else in `collections/` alone. The
-  README backup layout states the naming rule.
-- 2026-09-23: Checked downloaded originals against their recorded content hash
-  (issue 68). `downloadFile`, which `quak get`, the content cache and backup all
-  use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
-  stores nothing on a mismatch, failing with an error naming the file ID. A live
-  photo ZIP is unpacked as it streams with `fflate` and its image and video
-  hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
-  clients' `imageHash` and `videoHash` fields for live photos. A file with no
-  recorded hash is stored unchecked.
 - 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
   When `metadata.json` in the cache directory was written for a different,
   non-zero user ID than the client's, `Library.open` deletes it and `mldata/`

package.json

@@ -42,7 +42,6 @@
         "env-paths": "4.0.0",
         "exif-reader": "2.0.3",
         "fast-srp-hap": "2.0.4",
-        "fflate": "0.8.3",
         "jpeg-js": "0.4.4",
         "libsodium-wrappers-sumo": "0.8.4"
     }

src/backup.ts

@@ -17,9 +17,7 @@
 // temp-then-rename, so a file that exists is whole and is never re-fetched — an
 // interrupted run resumes by listing the directory. The derived views hold no
 // unique state, so they are rebuilt every run; that repairs stale sidecars and
-// missing or broken symlinks left by an earlier crash. A rebuild also removes
+// missing or broken symlinks left by an earlier crash.
-// the symlinks into originals/ that no longer belong to an album, and the
-// directories of albums that no longer exist.
 //
 // Resilience (issue #8): no per-file condition aborts the run. A failed
 // download or a failed symlink is caught, recorded in `failures.json` with a
@@ -36,14 +34,13 @@ import {
     readdirSync,
     readFileSync,
     readlinkSync,
-    rmdirSync,
     rmSync,
     statSync,
     symlinkSync,
     writeFileSync,
 } from "node:fs";
 import { copyFile, rename, rm } from "node:fs/promises";
-import { basename, dirname, extname, join, relative } from "node:path";
+import { basename, dirname, join, relative } from "node:path";
 
 import { fsyncPath } from "./download/index.js";
 import { safeExtension, sanitizeFileName } from "./filename.js";
@@ -230,93 +227,6 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
     symlinkSync(target, linkPath);
 };
 
-// The on-disk names for the entries of one directory, keyed by ID. Each name
-// is used as is unless another entry would get the same name, ignoring case
-// (two names that differ only in case are one entry on a case-insensitive
-// file system); then every entry sharing it gets ` (<id>)`, before the
-// extension when `beforeExtension` is set. A name with an ID added can match
-// another entry's own name (`IMG (6).JPG`), so this repeats until no name is
-// shared. IDs are stable, so the names are too.
-const namesByID = (
-    entries: { id: number; name: string }[],
-    beforeExtension: boolean,
-): Map<number, string> => {
-    const withID = (id: number, name: string): string => {
-        const ext = beforeExtension ? extname(name) : "";
-        const stem = name.slice(0, name.length - ext.length);
-        return `${stem} (${id})${ext}`;
-    };
-    const names = new Map<number, string>();
-    for (const { id, name } of entries) names.set(id, name);
-    const suffixed = new Set<number>();
-    for (;;) {
-        const counts = new Map<string, number>();
-        for (const name of names.values()) {
-            const key = name.toLowerCase();
-            counts.set(key, (counts.get(key) ?? 0) + 1);
-        }
-        let changed = false;
-        for (const { id, name } of entries) {
-            if (suffixed.has(id)) continue;
-            if (counts.get(name.toLowerCase()) === 1) continue;
-            names.set(id, withID(id, name));
-            suffixed.add(id);
-            changed = true;
-        }
-        if (!changed) return names;
-    }
-};
-
-// Remove the symlinks in the album directory `dir` that point into
-// `originalsDir` and are not named in `keep`. Nothing else in the directory
-// is touched: anything else there was put there by the user.
-const removeStaleLinks = (
-    dir: string,
-    keep: Set<string>,
-    originalsDir: string,
-): void => {
-    const target = relative(dir, originalsDir);
-    for (const name of readdirSync(dir)) {
-        if (keep.has(name)) continue;
-        const path = join(dir, name);
-        if (
-            lstatSync(path).isSymbolicLink() &&
-            dirname(readlinkSync(path)) === target
-        ) {
-            rmSync(path);
-        }
-    }
-};
-
-// Remove the directories under `collectionsDir` that an earlier run wrote for
-// an album that is gone or renamed: a directory not named in `current` with a
-// `<name>.json` beside it holding an album ID, which is what a run writes. Its
-// symlinks into originals/ are removed; if that leaves it empty, it and its
-// JSON are deleted, otherwise both stay for what the user put there.
-const removeStaleAlbumDirs = (
-    collectionsDir: string,
-    current: Set<string>,
-    originalsDir: string,
-): void => {
-    for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
-        if (!entry.isDirectory() || current.has(entry.name)) continue;
-        const jsonPath = join(collectionsDir, `${entry.name}.json`);
-        try {
-            const album = JSON.parse(readFileSync(jsonPath, "utf-8")) as {
-                id?: unknown;
-            };
-            if (typeof album.id !== "number") continue;
-        } catch {
-            continue;
-        }
-        const dir = join(collectionsDir, entry.name);
-        removeStaleLinks(dir, new Set(), originalsDir);
-        if (readdirSync(dir).length > 0) continue;
-        rmdirSync(dir);
-        rmSync(jsonPath);
-    }
-};
-
 const loadLedger = (path: string): Map<number, FailureEntry> => {
     const ledger = new Map<number, FailureEntry>();
     try {
@@ -393,10 +303,9 @@ export const runBackup = async (
 
     // Collections in scope, and the distinct files across them (a file shared
     // by two albums is one original).
-    const allCollections = lib.listCollections();
+    const collections = lib
-    const collections = allCollections.filter((c) =>
+        .listCollections()
-        only ? only.has(c.name) : true,
+        .filter((c) => (only ? only.has(c.name) : true));
-    );
     const collectionName = new Map<number, string>();
     for (const c of collections) collectionName.set(c.id, c.name);
 
@@ -497,55 +406,23 @@ export const runBackup = async (
         }
     }
 
-    // Then the per-collection symlink trees and JSON. Directory names are
+    // Then the per-collection symlink trees and JSON.
-    // chosen across every album, not just those in scope, so a scoped run
-    // names an album the same as a full one and never takes the directory of
-    // an album it skipped. Stale entries are removed before anything is
-    // rebuilt, so on a case-insensitive file system removing an old name can
-    // never remove the new one.
-    const albumDirNames = namesByID(
-        allCollections.map((c) => ({
-            id: c.id,
-            name: sanitizeFileName(c.name, `collection-${c.id}`),
-        })),
-        false,
-    );
-    try {
-        removeStaleAlbumDirs(
-            collectionsDir,
-            new Set(albumDirNames.values()),
-            originalsDir,
-        );
-    } catch (err) {
-        log(`FAILED removing old album directories: ${errorMessage(err)}`);
-    }
-
     for (const c of collections) {
-        const colDirName = albumDirNames.get(c.id)!;
+        const colDirName = sanitizeFileName(c.name, `collection-${c.id}`);
         const colDir = join(collectionsDir, colDirName);
         mkdirSync(colDir, { recursive: true });
 
         const files = filesByCollection.get(c.id) ?? [];
-        const linkNames = namesByID(
-            files.map((f) => ({
-                id: f.id,
-                name: sanitizeFileName(f.metadata.title, `file-${f.id}`),
-            })),
-            true,
-        );
-        try {
-            removeStaleLinks(colDir, new Set(linkNames.values()), originalsDir);
-        } catch (err) {
-            log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
-        }
-
         const metaFiles: { id: number; metadata: EnteFile["metadata"] }[] = [];
         for (const file of files) {
             metaFiles.push({ id: file.id, metadata: file.metadata });
             if (!includeOriginals) continue;
             const orig = join(originalsDir, originalName(file));
             if (!isPresent(orig)) continue;
-            const linkName = linkNames.get(file.id)!;
+            const linkName = sanitizeFileName(
+                file.metadata.title,
+                `file-${file.id}`,
+            );
             const linkPath = join(colDir, linkName);
             try {
                 rebuildSymlink(linkPath, relative(colDir, orig));

src/crypto/hash.ts

@@ -1,22 +0,0 @@
-import sodium, { type StateAddress } from "libsodium-wrappers-sumo";
-import { toBase64 } from "./encoding.js";
-
-// The content hash an uploading client records in a file's metadata: unkeyed
-// BLAKE2b with a 64-byte output over the original's bytes, fed in chunks, as
-// standard base64 with padding. Named after the upstream client's functions.
-// The output length is read at call time for the same reason as
-// `streamTagFinal` in stream.ts: libsodium sets its constants only once ready.
-
-export const chunkHashInit = (): StateAddress =>
-    sodium.crypto_generichash_init(null, sodium.crypto_generichash_BYTES_MAX);
-
-export const chunkHashUpdate = (state: StateAddress, chunk: Uint8Array): void =>
-    sodium.crypto_generichash_update(state, chunk);
-
-export const chunkHashFinal = (state: StateAddress): string =>
-    toBase64(
-        sodium.crypto_generichash_final(
-            state,
-            sodium.crypto_generichash_BYTES_MAX,
-        ),
-    );

src/crypto/index.ts

@@ -7,7 +7,6 @@ export {
 } from "./encoding.js";
 export { deriveKEK, deriveLoginSubkey } from "./kdf.js";
 export { decryptBox, decryptSealed } from "./box.js";
-export { chunkHashFinal, chunkHashInit, chunkHashUpdate } from "./hash.js";
 export {
     decryptBlob,
     encryptBlob,

src/download/index.ts

@@ -2,11 +2,7 @@ import { randomUUID } from "node:crypto";
 import { open, rename, rm } from "node:fs/promises";
 import type { FileHandle } from "node:fs/promises";
 import { dirname, join } from "node:path";
-import { Unzip, UnzipInflate } from "fflate";
 import {
-    chunkHashFinal,
-    chunkHashInit,
-    chunkHashUpdate,
     fromBase64,
     initStreamPull,
     pullStreamChunk,
@@ -241,109 +237,19 @@ export const writeAtomic = async (
 ): Promise<void> =>
     stageAtomic(destination, (handle) => handle.writeFile(plaintext));
 
-// Hashes an original's bytes as they are decrypted, for comparison with the
-// hash its uploader recorded.
-interface ContentHasher {
-    update: (plaintext: Uint8Array) => void;
-    digest: () => string;
-}
-
-const fileHasher = (): ContentHasher => {
-    const state = chunkHashInit();
-    return {
-        update: (plaintext) => chunkHashUpdate(state, plaintext),
-        digest: () => chunkHashFinal(state),
-    };
-};
-
-// A live photo is stored as a ZIP of its image and its video, and its recorded
-// hash is `<imageHash>:<videoHash>`, each over that part's own bytes. Like the
-// upstream client's decoder, this takes the first entries whose names start
-// with `image` and `video`.
-//
-// The ZIP is chosen by its uploader and may expand enormously, so entries are
-// hashed as they decompress and never held. fflate's `Unzip` inflates each
-// push in one piece, and deflate expands at most about 1000-fold, so the ZIP
-// is pushed in 4 KiB slices to keep each decompressed piece near 4 MiB, one
-// plaintext chunk. Every entry is started, even one that is not hashed,
-// because fflate keeps an unstarted entry's data in memory.
-const livePhotoHasher = (fileID: number): ContentHasher => {
-    const sliceSize = 4096;
-    const fail = (message: string, cause?: unknown): Error =>
-        new Error(`download: file ${fileID}: ${message}`, { cause });
-    const claimed = new Set<string>();
-    const hashes = new Map<string, string>();
-    const unzip = new Unzip((entry) => {
-        const part = ["image", "video"].find((p) => entry.name.startsWith(p));
-        const target =
-            part === undefined || claimed.has(part)
-                ? undefined
-                : { part, state: chunkHashInit() };
-        if (target !== undefined) claimed.add(target.part);
-        entry.ondata = (err, data, final) => {
-            if (err) throw err;
-            if (target === undefined) return;
-            chunkHashUpdate(target.state, data);
-            if (final) hashes.set(target.part, chunkHashFinal(target.state));
-        };
-        entry.start();
-    });
-    unzip.register(UnzipInflate);
-    // fflate reports a bad ZIP by throwing, sometimes a TypeError, which the
-    // retry would take for a network failure; a bad ZIP is never retried.
-    const push = (data: Uint8Array, final: boolean): void => {
-        try {
-            unzip.push(data, final);
-        } catch (err) {
-            throw fail("live photo is not a readable ZIP", err);
-        }
-    };
-    return {
-        update: (plaintext) => {
-            for (let i = 0; i < plaintext.length; i += sliceSize) {
-                push(plaintext.subarray(i, i + sliceSize), false);
-            }
-        },
-        digest: () => {
-            push(new Uint8Array(0), true);
-            const image = hashes.get("image");
-            const video = hashes.get("video");
-            if (image === undefined || video === undefined) {
-                throw fail(
-                    "live photo ZIP does not hold both an image and a video",
-                );
-            }
-            return `${image}:${video}`;
-        },
-    };
-};
-
 // Decrypt `stream` straight to `destination`, one plaintext chunk at a time,
 // under the atomic writer's temp-then-rename discipline. Memory stays bounded
 // by the chunk size: each decrypted chunk is written to the temp file and
 // dropped. The rename happens only after the stream authenticates as terminated
 // on TAG_FINAL; a truncated stream throws and leaves the destination untouched.
 // Returns the plaintext length written.
-//
-// `original` is the file whose original this is (none for a thumbnail, which
-// has no recorded hash). When its metadata has a hash, the decrypted bytes
-// must match it or nothing is stored. Both a plain file and a live photo's
-// parts are hashed as they stream. The mismatch error is not retried.
 const decryptToTemp = async (
     destination: string,
     stream: ReadableStream<Uint8Array>,
     header: Uint8Array,
     key: Uint8Array,
     onProgress?: ProgressCallback,
-    original?: EnteFile,
 ): Promise<number> => {
-    const expected = original?.metadata.hash;
-    const hasher =
-        original === undefined || expected === undefined
-            ? undefined
-            : original.metadata.fileType === "livePhoto"
-              ? livePhotoHasher(original.id)
-              : fileHasher();
     let bytesWritten = 0;
     try {
         await stageAtomic(destination, async (handle) => {
@@ -352,18 +258,10 @@ const decryptToTemp = async (
                 header,
                 key,
                 async (plaintext) => {
-                    hasher?.update(plaintext);
                     await handle.write(plaintext);
                 },
                 onProgress,
             );
-            if (original === undefined || hasher === undefined) return;
-            const actual = hasher.digest();
-            if (actual !== expected) {
-                throw new Error(
-                    `download: file ${original.id}: content hash ${actual} does not match the hash its uploader recorded, ${expected}`,
-                );
-            }
         });
     } catch (err) {
         // Cancel the body so its connection is closed now rather than held
@@ -404,18 +302,10 @@ const fetchAndDecrypt = async (
     key: Uint8Array,
     destination: string,
     onProgress?: ProgressCallback,
-    original?: EnteFile,
 ): Promise<number> =>
     withRetry(async () => {
         const stream = await openStream();
-        return decryptToTemp(
+        return decryptToTemp(destination, stream, header, key, onProgress);
-            destination,
-            stream,
-            header,
-            key,
-            onProgress,
-            original,
-        );
     }, api.getRetryOptions());
 
 export const downloadFile = async (
@@ -436,7 +326,6 @@ export const downloadFile = async (
         file.key,
         resolvedPath,
         onProgress,
-        file,
     );
     return { path: resolvedPath, bytesWritten };
 };

src/library/content.ts

@@ -15,12 +15,12 @@
 // Integrity. The reused streaming decrypt is the enforced guarantee: every
 // chunk is authenticated and the writer renames the file into place only once
 // the stream ends on TAG_FINAL, so a truncated or corrupt fetch throws and
-// nothing is stored. For an original whose metadata records a content hash
+// nothing is stored. On top of that this module refuses to record a stored file
-// (`FileMetadata.hash`), the writer also hashes the decrypted bytes and stores
+// that came out empty. The design also asks for a content-hash comparison
-// nothing if they differ, failing the fetch with an error naming the file. An
+// against `FileMetadata.hash` (with a `fileSize` fallback); that is deferred —
-// original with no recorded hash is stored unchecked, as the upstream client
+// see the PR — because the exact hash construction cannot be confirmed against
-// does; thumbnails have none. On top of that this module refuses to record a
+// the repo's fixtures and `FileBlob.size` is the encrypted object size, not the
-// stored file that came out empty.
+// decrypted length this layer has.
 
 import { existsSync, statSync } from "node:fs";
 import {

src/model/decrypt.ts

@@ -34,28 +34,6 @@ const FILE_TYPE_MAP: Record<number, FileType> = {
 
 const parseFileType = (n: number): FileType => FILE_TYPE_MAP[n] ?? "unknown";
 
-// The hash the uploading client recorded for the original's bytes, read the
-// way the upstream client's `metadataHash` reads it: `hash` if present,
-// otherwise, for a live photo from an older client that wrote the two parts
-// separately, `<imageHash>:<videoHash>`. A field that is not a non-empty
-// string counts as absent, and a file with no hash at all is normal.
-const expectedHash = (json: Record<string, unknown>): string | undefined => {
-    const text = (v: unknown): string | undefined =>
-        typeof v === "string" && v !== "" ? v : undefined;
-    const hash = text(json.hash);
-    if (hash !== undefined) return hash;
-    const imageHash = text(json.imageHash);
-    const videoHash = text(json.videoHash);
-    if (
-        json.fileType === 2 &&
-        imageHash !== undefined &&
-        videoHash !== undefined
-    ) {
-        return `${imageHash}:${videoHash}`;
-    }
-    return undefined;
-};
-
 export const decryptCollection = (
     raw: RawCollection,
     keys: KeyMaterial,
@@ -137,7 +115,7 @@ export const decryptFile = (
         modificationTime: metadataJSON.modificationTime ?? 0,
         latitude: metadataJSON.latitude,
         longitude: metadataJSON.longitude,
-        hash: expectedHash(metadataJSON),
+        hash: metadataJSON.hash,
     };
 
     const magicMetadata = decryptMagicMetadata(raw.magicMetadata, key);

src/model/types.ts

@@ -29,9 +29,6 @@ export interface FileMetadata {
     modificationTime: Microseconds;
     latitude?: number;
     longitude?: number;
-    // The content hash the uploader recorded (see `expectedHash` in
-    // decrypt.ts); `downloadFile` refuses an original that does not match it.
-    // Absent for files from very old clients.
     hash?: string;
 }
 

test/cli/backup.test.ts

@@ -40,7 +40,6 @@ import {
     readFileSync,
     readlinkSync,
     rmSync,
-    symlinkSync,
     writeFileSync,
 } from "node:fs";
 import { spawnSync } from "node:child_process";
@@ -48,7 +47,6 @@ import { join } from "node:path";
 import { tmpdir } from "node:os";
 import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
 
-import { runBackup, type BackupLibrary } from "../../src/backup.js";
 import { Library } from "../../src/library/index.js";
 import type { ContentSource } from "../../src/library/content.js";
 import type { CollectionsPage, FilesPage } from "../../src/client.js";
@@ -624,318 +622,3 @@ describe("lib.backup", () => {
         lib.close();
     });
 });
-
-// The album folders under collections/, driven through `runBackup` with a
-// stand-in library whose albums a test changes between runs.
-describe("backup album folders", () => {
-    interface Album {
-        collection: Collection;
-        files: EnteFile[];
-    }
-
-    const libraryOf = (albums: Album[]): BackupLibrary => ({
-        refresh: async () => {},
-        listCollections: () => albums.map((a) => a.collection),
-        listFiles: (id) =>
-            albums.find((a) => a.collection.id === id)?.files ?? [],
-        original: async (fileID) => {
-            const path = join(root, `source-${fileID}`);
-            writeFileSync(path, `original ${fileID}`);
-            return { path };
-        },
-        thumbnail: async () => {
-            throw new Error("no thumbnails in this stand-in");
-        },
-    });
-
-    // Every entry under collections/, one level of directories deep, with each
-    // symlink's target.
-    const tree = (outDir: string): string[] => {
-        const lines: string[] = [];
-        const list = (dir: string, prefix: string): void => {
-            for (const name of readdirSync(dir).sort()) {
-                const path = join(dir, name);
-                const st = lstatSync(path);
-                if (st.isSymbolicLink()) {
-                    lines.push(`${prefix}${name} -> ${readlinkSync(path)}`);
-                } else if (st.isDirectory() && prefix === "") {
-                    lines.push(`${name}/`);
-                    list(path, `${name}/`);
-                } else {
-                    lines.push(`${prefix}${name}`);
-                }
-            }
-        };
-        list(join(outDir, "collections"), "");
-        return lines;
-    };
-
-    const albumID = (outDir: string, jsonName: string): number =>
-        JSON.parse(readFileSync(join(outDir, "collections", jsonName), "utf-8"))
-            .id;
-
-    it("gives every file and every album its own name when names repeat", async () => {
-        const outDir = join(root, "backup");
-        const lib = libraryOf([
-            {
-                collection: collection(10, "Trip"),
-                files: [
-                    file(1, 10, "IMG_0001.JPG"),
-                    file(2, 10, "IMG_0001.JPG"),
-                    file(4, 10, "img_0001.jpg"),
-                    file(3, 10, "other.jpg"),
-                ],
-            },
-            {
-                collection: collection(11, "Trip"),
-                files: [file(3, 11, "other.jpg")],
-            },
-        ]);
-
-        const result = await runBackup(lib, { downloadDirectory: outDir });
-
-        expect(result.failed).toBe(0);
-        expect(tree(outDir)).toEqual([
-            "Trip (10)/",
-            "Trip (10)/IMG_0001 (1).JPG -> ../../originals/1.JPG",
-            "Trip (10)/IMG_0001 (2).JPG -> ../../originals/2.JPG",
-            "Trip (10)/img_0001 (4).jpg -> ../../originals/4.jpg",
-            "Trip (10)/other.jpg -> ../../originals/3.jpg",
-            "Trip (10).json",
-            "Trip (11)/",
-            "Trip (11)/other.jpg -> ../../originals/3.jpg",
-            "Trip (11).json",
-        ]);
-        expect(albumID(outDir, "Trip (10).json")).toBe(10);
-        expect(albumID(outDir, "Trip (11).json")).toBe(11);
-    });
-
-    it("keeps names unique when a name with an ID added is another entry's own name", async () => {
-        const outDir = join(root, "backup");
-        const lib = libraryOf([
-            {
-                collection: collection(10, "Trip"),
-                files: [
-                    file(5, 10, "IMG (6).JPG"),
-                    file(6, 10, "IMG.JPG"),
-                    file(7, 10, "IMG.JPG"),
-                ],
-            },
-            {
-                collection: collection(11, "Trip"),
-                files: [file(8, 11, "a.jpg")],
-            },
-            {
-                collection: collection(12, "Trip (11)"),
-                files: [file(9, 12, "b.jpg")],
-            },
-        ]);
-
-        const result = await runBackup(lib, { downloadDirectory: outDir });
-
-        expect(result.failed).toBe(0);
-        expect(tree(outDir)).toEqual([
-            "Trip (10)/",
-            "Trip (10)/IMG (6) (5).JPG -> ../../originals/5.JPG",
-            "Trip (10)/IMG (6).JPG -> ../../originals/6.JPG",
-            "Trip (10)/IMG (7).JPG -> ../../originals/7.JPG",
-            "Trip (10).json",
-            "Trip (11)/",
-            "Trip (11)/a.jpg -> ../../originals/8.jpg",
-            "Trip (11) (12)/",
-            "Trip (11) (12)/b.jpg -> ../../originals/9.jpg",
-            "Trip (11) (12).json",
-            "Trip (11).json",
-        ]);
-        expect(albumID(outDir, "Trip (10).json")).toBe(10);
-        expect(albumID(outDir, "Trip (11).json")).toBe(11);
-        expect(albumID(outDir, "Trip (11) (12).json")).toBe(12);
-    });
-
-    it("changes nothing on a second run over an unchanged account", async () => {
-        const outDir = join(root, "backup");
-        const lib = libraryOf([
-            {
-                collection: collection(10, "Trip"),
-                files: [
-                    file(1, 10, "IMG_0001.JPG"),
-                    file(2, 10, "IMG_0001.JPG"),
-                ],
-            },
-            {
-                collection: collection(11, "Trip"),
-                files: [file(3, 11, "other.jpg")],
-            },
-        ]);
-
-        await runBackup(lib, { downloadDirectory: outDir });
-        const before = tree(outDir);
-        const second = await runBackup(lib, { downloadDirectory: outDir });
-
-        expect(second.downloaded).toBe(0);
-        expect(second.failed).toBe(0);
-        expect(tree(outDir)).toEqual(before);
-    });
-
-    it("leaves the albums an onlyAlbumNames run skips as they were", async () => {
-        const outDir = join(root, "backup");
-        // "trip" is skipped by the scoped run but its name clashes with the
-        // in-scope "Trip", so "Trip" must keep its ID suffix.
-        const lib = libraryOf([
-            {
-                collection: collection(10, "Trip"),
-                files: [file(1, 10, "a.jpg")],
-            },
-            {
-                collection: collection(11, "trip"),
-                files: [file(2, 11, "b.jpg")],
-            },
-            {
-                collection: collection(12, "Work"),
-                files: [file(3, 12, "c.jpg")],
-            },
-        ]);
-        const json = (name: string): string =>
-            readFileSync(join(outDir, "collections", name), "utf-8");
-
-        await runBackup(lib, { downloadDirectory: outDir });
-        const before = tree(outDir);
-        const skippedJSON = [json("trip (11).json"), json("Work.json")];
-        const scoped = await runBackup(lib, {
-            downloadDirectory: outDir,
-            onlyAlbumNames: ["Trip"],
-        });
-
-        expect(scoped.failed).toBe(0);
-        expect(before).toEqual([
-            "Trip (10)/",
-            "Trip (10)/a.jpg -> ../../originals/1.jpg",
-            "Trip (10).json",
-            "Work/",
-            "Work/c.jpg -> ../../originals/3.jpg",
-            "Work.json",
-            "trip (11)/",
-            "trip (11)/b.jpg -> ../../originals/2.jpg",
-            "trip (11).json",
-        ]);
-        expect(tree(outDir)).toEqual(before);
-        expect([json("trip (11).json"), json("Work.json")]).toEqual(
-            skippedJSON,
-        );
-    });
-
-    it("removes links and album folders that are gone, and nothing the user added", async () => {
-        const outDir = join(root, "backup");
-        const albums: Album[] = [
-            {
-                collection: collection(10, "Trip"),
-                files: [
-                    file(1, 10, "IMG_0001.JPG"),
-                    file(2, 10, "IMG_0001.JPG"),
-                    file(3, 10, "other.jpg"),
-                ],
-            },
-            {
-                collection: collection(12, "Work"),
-                files: [file(5, 12, "a.jpg")],
-            },
-            {
-                collection: collection(13, "Old"),
-                files: [file(5, 13, "a.jpg")],
-            },
-        ];
-        const lib = libraryOf(albums);
-        await runBackup(lib, { downloadDirectory: outDir });
-
-        // What the user put in the tree: a note and a symlink of their own in
-        // an album, a note in an album about to be renamed, and a folder quak
-        // did not create.
-        const collectionsDir = join(outDir, "collections");
-        writeFileSync(join(collectionsDir, "Trip", "notes.txt"), "mine");
-        symlinkSync("../elsewhere", join(collectionsDir, "Trip", "mine"));
-        writeFileSync(join(collectionsDir, "Work", "keep.txt"), "mine");
-        mkdirSync(join(collectionsDir, "Mine"));
-        writeFileSync(join(collectionsDir, "Mine", "keep.txt"), "mine");
-
-        // File 2 leaves Trip, Work is renamed Office, Old is deleted.
-        albums[0]!.files.splice(1, 1);
-        albums[1]!.collection = collection(12, "Office");
-        albums.splice(2, 1);
-        const result = await runBackup(lib, { downloadDirectory: outDir });
-
-        expect(result.failed).toBe(0);
-        expect(tree(outDir)).toEqual([
-            "Mine/",
-            "Mine/keep.txt",
-            "Office/",
-            "Office/a.jpg -> ../../originals/5.jpg",
-            "Office.json",
-            "Trip/",
-            "Trip/IMG_0001.JPG -> ../../originals/1.JPG",
-            "Trip/mine -> ../elsewhere",
-            "Trip/notes.txt",
-            "Trip/other.jpg -> ../../originals/3.jpg",
-            "Trip.json",
-            "Work/",
-            "Work/keep.txt",
-            "Work.json",
-        ]);
-    });
-
-    // One album backed up, then a folder the user made beside it holding a
-    // symlink into originals/, with `json` (if given) as its sibling JSON.
-    const backupWithUserFolder = async (
-        json: string | undefined,
-    ): Promise<{ outDir: string; failed: number }> => {
-        const outDir = join(root, "backup");
-        const lib = libraryOf([
-            {
-                collection: collection(10, "Trip"),
-                files: [file(1, 10, "a.jpg")],
-            },
-        ]);
-        await runBackup(lib, { downloadDirectory: outDir });
-        const collectionsDir = join(outDir, "collections");
-        mkdirSync(join(collectionsDir, "Mine"));
-        symlinkSync(
-            "../../originals/1.jpg",
-            join(collectionsDir, "Mine", "a.jpg"),
-        );
-        if (json !== undefined) {
-            writeFileSync(join(collectionsDir, "Mine.json"), json);
-        }
-        const result = await runBackup(lib, { downloadDirectory: outDir });
-        return { outDir, failed: result.failed };
-    };
-
-    it("leaves a user folder with no JSON beside it as it was", async () => {
-        const { outDir, failed } = await backupWithUserFolder(undefined);
-
-        expect(failed).toBe(0);
-        expect(tree(outDir)).toEqual([
-            "Mine/",
-            "Mine/a.jpg -> ../../originals/1.jpg",
-            "Trip/",
-            "Trip/a.jpg -> ../../originals/1.jpg",
-            "Trip.json",
-        ]);
-    });
-
-    it("leaves a user folder whose JSON has no album ID as it was", async () => {
-        const json = '{"name":"Mine"}';
-        const { outDir, failed } = await backupWithUserFolder(json);
-
-        expect(failed).toBe(0);
-        expect(tree(outDir)).toEqual([
-            "Mine/",
-            "Mine/a.jpg -> ../../originals/1.jpg",
-            "Mine.json",
-            "Trip/",
-            "Trip/a.jpg -> ../../originals/1.jpg",
-            "Trip.json",
-        ]);
-        expect(
-            readFileSync(join(outDir, "collections", "Mine.json"), "utf-8"),
-        ).toBe(json);
-    });
-});

test/crypto/hash.test.ts

@@ -1,33 +0,0 @@
-import { beforeAll, describe, expect, it } from "vitest";
-import {
-    chunkHashFinal,
-    chunkHashInit,
-    chunkHashUpdate,
-    init,
-} from "../../src/crypto/index.js";
-
-beforeAll(async () => {
-    await init();
-});
-
-describe("content hash", () => {
-    // RFC 7693 Appendix A: BLAKE2b-512 of "abc".
-    const abc = Buffer.from(
-        "ba80a53f981c4d0d6a2797b69f12f6e94c212f14685ac4b74b12bb6fdbffa2d1" +
-            "7d87c5392aab792dc252d5de4533cc9518d38aa8dbf1925ab92386edd4009923",
-        "hex",
-    ).toString("base64");
-
-    it("is unkeyed BLAKE2b-512 in standard base64", () => {
-        const state = chunkHashInit();
-        chunkHashUpdate(state, new TextEncoder().encode("abc"));
-        expect(chunkHashFinal(state)).toBe(abc);
-    });
-
-    it("gives the same hash when the input arrives in chunks", () => {
-        const state = chunkHashInit();
-        chunkHashUpdate(state, new TextEncoder().encode("a"));
-        chunkHashUpdate(state, new TextEncoder().encode("bc"));
-        expect(chunkHashFinal(state)).toBe(abc);
-    });
-});

test/crypto/kdf.test.ts

@@ -29,10 +29,10 @@ describe("crypto.deriveKEK (Argon2id)", () => {
     });
 
     /**
-     * Cheap parameters used so the test suite stays under the 90-second
+     * Cheap parameters used so the test suite stays under the 30-second
-     * `timeout` in the `test` phase of the `Dockerfile`. The real production
+     * budget. The real production parameters Ente uses are larger
-     * parameters Ente uses are larger (memLimit up to 1 GiB, opsLimit 3-16).
+     * (memLimit up to 1 GiB, opsLimit 3-16). The algorithm is the same
-     * The algorithm is the same regardless of parameters.
+     * regardless of parameters.
      */
     const TEST_OPS = 2;
     const TEST_MEM = 64 * 1024 * 1024; // 64 MiB

test/download/download.test.ts

@@ -61,7 +61,6 @@ import { dirname, join } from "node:path";
 import { tmpdir } from "node:os";
 import { createHash } from "node:crypto";
 import sodium from "libsodium-wrappers-sumo";
-import { zipSync } from "fflate";
 import {
     beforeAll,
     beforeEach,
@@ -189,31 +188,7 @@ vi.mock("node:fs/promises", async (importOriginal) => {
     };
 });
 
-/**
- * `chunkHashUpdate` is wrapped to record the length of every piece hashed, so
- * a test can show that a live photo entry reaches the hash in pieces far
- * smaller than the entry, rather than decompressed whole first.
- */
-const hashHook = vi.hoisted(() => ({
-    lengths: [] as number[],
-}));
-
-vi.mock("../../src/crypto/index.js", async (importOriginal) => {
-    const actual =
-        await importOriginal<typeof import("../../src/crypto/index.js")>();
-    return {
-        ...actual,
-        chunkHashUpdate: (
-            ...args: Parameters<typeof actual.chunkHashUpdate>
-        ): void => {
-            hashHook.lengths.push(args[1].length);
-            actual.chunkHashUpdate(...args);
-        },
-    };
-});
-
 beforeEach(() => {
-    hashHook.lengths.length = 0;
     renameHook.calls.length = 0;
     renameHook.failWith = null;
     durabilityHook.events.length = 0;
@@ -1035,28 +1010,33 @@ describe.each(entryPoints)(
             expect(readdirSync(dir)).toEqual([]);
         });
 
-        // Root ignores directory permissions, so this fails when run as root.
+        // Root ignores directory permissions, so this cannot fail as root.
-        // The `test` phase of the `Dockerfile` runs as the `node` user.
+        // The `test` phase of the `Dockerfile` runs as the `node` user so
-        it("fails without creating anything when the destination directory is not writable", async () => {
+        // that `make check` runs it.
-            const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        it.skipIf(process.getuid?.() === 0)(
-            const { header, ciphertext } = encryptFileBody(
+            "fails without creating anything when the destination directory is not writable",
-                patternBytes(64, 34),
+            async () => {
-                key,
+                const key =
-            );
+                    sodium.crypto_secretstream_xchacha20poly1305_keygen();
-            const { api, file } = fixtureFor(key, header, ciphertext);
+                const { header, ciphertext } = encryptFileBody(
-            const dir = freshDir();
+                    patternBytes(64, 34),
-            const outPath = join(dir, "never.bin");
+                    key,
-            chmodSync(dir, 0o500);
+                );
-            try {
+                const { api, file } = fixtureFor(key, header, ciphertext);
-                await expect(
+                const dir = freshDir();
-                    download(api, file, outPath),
+                const outPath = join(dir, "never.bin");
-                ).rejects.toMatchObject({ code: "EACCES" });
+                chmodSync(dir, 0o500);
-            } finally {
+                try {
-                chmodSync(dir, 0o700);
+                    await expect(
-            }
+                        download(api, file, outPath),
+                    ).rejects.toMatchObject({ code: "EACCES" });
+                } finally {
+                    chmodSync(dir, 0o700);
+                }
 
-            expect(readdirSync(dir)).toEqual([]);
+                expect(readdirSync(dir)).toEqual([]);
-        });
+            },
+        );
     },
 );
 
@@ -1635,113 +1615,3 @@ describe.each(entryPoints)("$name progress", ({ name, download }) => {
         expectSameBytes(readFileSync(outPath), plaintext);
     });
 });
-
-describe("downloadFile content hash", () => {
-    // Node's own BLAKE2b-512 is the reference, so these tests do not depend
-    // on the code under test to compute what they expect.
-    const blake2b = (bytes: Uint8Array): string =>
-        createHash("blake2b512").update(bytes).digest("base64");
-
-    // Serve `plaintext` encrypted as file 999 with the given metadata. Four
-    // responses are scripted so a retried mismatch would show in `requests`.
-    const setup = (plaintext: Uint8Array, metadata: Partial<FileMetadata>) => {
-        const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
-        const { header, ciphertext } = encryptFileBody(plaintext, key);
-        const file = buildMockEnteFile(key, header, header);
-        file.metadata = { ...file.metadata, ...metadata };
-        const body = { kind: "body", bytes: ciphertext } as const;
-        const { fetch, requests } = scriptedCdnFetch(body, body, body, body);
-        const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 4 } });
-        const dir = mkdtempSync(join(testDir, "hash-"));
-        const outPath = join(dir, "f.bin");
-        return {
-            run: () => downloadFile(api, file, outPath),
-            dir,
-            outPath,
-            requests,
-        };
-    };
-
-    const livePhotoZip = zipSync({
-        "image.heic": patternBytes(500, 81),
-        "video.mov": patternBytes(900, 82),
-    });
-    const livePhotoHash = `${blake2b(patternBytes(500, 81))}:${blake2b(patternBytes(900, 82))}`;
-
-    it("stores a file whose hash matches", async () => {
-        const plaintext = patternBytes(700, 80);
-        const t = setup(plaintext, { hash: blake2b(plaintext) });
-
-        await t.run();
-
-        expectSameBytes(readFileSync(t.outPath), plaintext);
-    });
-
-    it("rejects a mismatch, stores nothing, names the file and does not retry", async () => {
-        const t = setup(patternBytes(700, 80), {
-            hash: blake2b(patternBytes(700, 79)),
-        });
-
-        await expect(t.run()).rejects.toThrow(
-            /file 999: content hash .* does not match/,
-        );
-
-        expect(readdirSync(t.dir)).toEqual([]);
-        expect(t.requests()).toBe(1);
-    });
-
-    it("stores a file with no recorded hash unchecked", async () => {
-        const plaintext = patternBytes(700, 80);
-        const t = setup(plaintext, { hash: undefined });
-
-        await t.run();
-
-        expectSameBytes(readFileSync(t.outPath), plaintext);
-    });
-
-    it("stores a live photo whose image and video hashes match", async () => {
-        const t = setup(livePhotoZip, {
-            fileType: "livePhoto",
-            hash: livePhotoHash,
-        });
-
-        await t.run();
-
-        expectSameBytes(readFileSync(t.outPath), livePhotoZip);
-    });
-
-    it("hashes a large live photo entry as it decompresses, never whole", async () => {
-        // 64 MiB of zeros deflates to a few kilobytes, the shape of a ZIP
-        // that would exhaust memory if expanded whole.
-        const image = new Uint8Array(64 * 1024 * 1024);
-        const video = patternBytes(900, 83);
-        const zip = zipSync({ "image.heic": image, "video.mov": video });
-        const t = setup(zip, {
-            fileType: "livePhoto",
-            hash: `${blake2b(image)}:${blake2b(video)}`,
-        });
-
-        await t.run();
-
-        expectSameBytes(readFileSync(t.outPath), zip);
-        const hashed = hashHook.lengths.reduce((a, b) => a + b, 0);
-        expect(hashed).toBe(image.length + video.length);
-        expect(Math.max(...hashHook.lengths)).toBeLessThanOrEqual(
-            2 * STREAM_CHUNK_SIZE,
-        );
-    });
-
-    it("rejects a live photo whose hash does not match", async () => {
-        // The whole ZIP's hash is not the recorded one: each part is hashed.
-        const t = setup(livePhotoZip, {
-            fileType: "livePhoto",
-            hash: blake2b(livePhotoZip),
-        });
-
-        await expect(t.run()).rejects.toThrow(
-            /file 999: content hash .* does not match/,
-        );
-
-        expect(readdirSync(t.dir)).toEqual([]);
-    });
-});

test/model/decrypt.test.ts

@@ -351,46 +351,6 @@ describe("model.decryptFile", () => {
         }
     });
 
-    it("reads the recorded content hash, joining an older live photo's two parts", () => {
-        const masterKey = sodium.crypto_secretbox_keygen();
-        const { collectionKey } = buildRawCollection(masterKey);
-        const hashOf = (metadata: Record<string, unknown>) =>
-            decryptFile(
-                buildRawFile(collectionKey, {
-                    metadata: { title: "x", ...metadata },
-                }),
-                collectionKey,
-            ).metadata.hash;
-
-        expect(hashOf({ fileType: 0, hash: "H" })).toBe("H");
-        expect(
-            hashOf({ fileType: 2, hash: "H", imageHash: "I", videoHash: "V" }),
-        ).toBe("H");
-        expect(hashOf({ fileType: 2, imageHash: "I", videoHash: "V" })).toBe(
-            "I:V",
-        );
-        expect(hashOf({ fileType: 2, imageHash: "I" })).toBeUndefined();
-        expect(
-            hashOf({ fileType: 0, imageHash: "I", videoHash: "V" }),
-        ).toBeUndefined();
-        expect(hashOf({ fileType: 0 })).toBeUndefined();
-        expect(hashOf({ fileType: 0, hash: 42 })).toBeUndefined();
-        expect(
-            hashOf({ fileType: 2, imageHash: "I", videoHash: 7 }),
-        ).toBeUndefined();
-        // An empty string counts as absent, not as a hash to match.
-        expect(hashOf({ fileType: 0, hash: "" })).toBeUndefined();
-        expect(
-            hashOf({ fileType: 2, hash: "", imageHash: "I", videoHash: "V" }),
-        ).toBe("I:V");
-        expect(
-            hashOf({ fileType: 2, imageHash: "", videoHash: "V" }),
-        ).toBeUndefined();
-        expect(
-            hashOf({ fileType: 2, imageHash: "I", videoHash: "" }),
-        ).toBeUndefined();
-    });
-
     it("maps fileType numbers to FileType strings", () => {
         // Ente uses: 0=image, 1=video, 2=livePhoto
         const masterKey = sodium.crypto_secretbox_keygen();

yarn.lock

@@ -1069,11 +1069,6 @@ fastq@^1.6.0:
   dependencies:
     reusify "^1.0.4"
 
-fflate@0.8.3:
-  version "0.8.3"
-  resolved "https://registry.yarnpkg.com/fflate/-/fflate-0.8.3.tgz#bc27d8eb30343d4d512abb03480202ce65d825fc"
-  integrity sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==
-
 file-entry-cache@^8.0.0:
   version "8.0.0"
   resolved "https://registry.yarnpkg.com/file-entry-cache/-/file-entry-cache-8.0.0.tgz#7787bddcf1131bffb92636c69457bbc0edd6d81f"