Compare commits

...
2 Commits
Author SHA1 Message Date
sneak c34e05cfe0 quak files takes the collection ID as an optional argument (closes #211)
check / check (push) Canceled after 0s
Mandatory arguments are positional, so `quak files` takes the collection
ID as an optional argument and the `--collection` option is removed.
With a collection ID it lists that collection's files. Without one it
lists every file in the account, collection by collection, each file
once: the library keeps a record of a file for each collection that
holds it, and only the first collection's record is listed, so that
collection is its `collectionID` in `--json`. The `--help` text, the
README and `TODO.md` say so.

Model: opus-5-5
2026-10-18 18:07:57 +00:00
clawbot 66e3ac77e8 Names quak builds from server data fit in 255 bytes (closes #205)
check / check (push) Canceled after 0s
The default names of `quak get`, `quak get-thumb`, `downloadFile` and `downloadThumbnail`, and the album folders of `backup-metadata`, could be over 255 bytes, which a file system refuses. `defaultFileName` and `shortenKeepingExtension` in `src/filename.ts` now cut such a name short, keeping its extension when it fits; a live photo's image and video get their own extension before the cut. A cut name never starts with a dot and is never a Windows device name. `--out` and `outPath` are kept as given. A `backup-metadata` folder keeps its `<id>-`. `safeExtension` gives `.bin` for an extension of more than 16 letters and digits, so a save path's name always fits.

Model: opus-5-5
2026-10-18 20:01:34 +02:00
17 changed files with 706 additions and 144 deletions
+49 -25
View File
@@ -35,8 +35,11 @@ yarn quak login
# List the user's collections (albums).
yarn quak collections
# List files in a collection.
yarn quak files --collection 12345
# List every file in the account.
yarn quak files
# List the files in a collection.
yarn quak files 12345
# Download and decrypt a single file.
yarn quak get 67890 --out ./photo.jpg
@@ -647,7 +650,7 @@ quak login interactive or QUAK_EM
quak whoami print logged-in account as JSON
quak logout end the session, delete it
quak collections [--json] list all collections
quak files --collection <id> [--json] list files in a collection
quak files [collectionID] [--json] list files in a collection, or every file
quak get <fileID> [--out path] [--collection] download and decrypt a file
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
quak backup <dir> [--json] [--verify] full incremental backup
@@ -676,34 +679,53 @@ answers from what did decrypt (see "Default reads vs. fresh reads").
`--cache-dir` overrides where the cache lives; without it each account gets its
own directory under the per-user cache path.
An ID given to `files --collection`, `get`, `get-thumb` or
`files <collectionID>` lists the files in that collection. `files` with no
argument lists every file in the account, collection by collection, each file
once: a file in several collections is listed with the first of them, which is
its `collectionID` in `--json`.
An ID given to `files`, `get`, `get-thumb` or
`helper fix-missing-thumbnails --file` is written in decimal digits, no larger
than 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused
before any request with one line, `Invalid file ID: <arg>` or
`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses
the whole command, so nothing is fixed.
Without `--out`, `get` writes the file to the working directory under its title,
and `get-thumb` writes the thumbnail there under `thumb_` followed by the title,
each with unsafe characters replaced. A name over 255 bytes, which most file
systems refuse, is cut short between two characters before its extension, so the
extension is kept whenever it fits. An extension too long to leave room for the
name's first character is cut short too, so the name never starts with a dot.
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
accepted for backward compatibility but ignored. For a live photo, `get` writes
its image and its video, each named after the title with its own extension, as
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
`--out PATH`, the image is written to `PATH` and the video beside it, with
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
refused. `backup-metadata --exif` (alias `--all`) additionally fetches each
file's original through the cache and records, from it or a live photo's image,
its XMP metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is
read with [exifreader](https://github.com/mattiasw/ExifReader) from any image
format it reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's
`exif` field is exifreader's EXIF tag output: each tag by name, with its
`value`, `description` and `computed` value. An EXIF block exifreader finds but
reads no tag from is recorded, base64, as `exifRaw`, with the reason in
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is
cut short as above once its own extension is in it. With `--out PATH`, the image
is written to `PATH` and the video beside it, with `PATH`'s name and the video's
extension; a `PATH` with the video's extension is refused.
`backup-metadata --exif` (alias `--all`) additionally fetches each file's
original through the cache and records, from it or a live photo's image, its XMP
metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with
[exifreader](https://github.com/mattiasw/ExifReader) from any image format it
reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field
is exifreader's EXIF tag output: each tag by name, with its `value`,
`description` and `computed` value. An EXIF block exifreader finds but reads no
tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`.
`collections`, `files`, `backup`, `helper list-missing-thumbnails` and
`helper fix-missing-thumbnails` take `--json` for machine-readable output.
`backup --verify` also hashes the originals already in the backup and replaces
any that do not match the content hash Ente records; one it cannot replace stays
as it was and goes into `failures.json` (see "Backup layout").
`backup-metadata <dir>` writes `account.json` and, for each collection, a
directory `collections/<id>-<name>/` holding `_collection.json` and a
`<fileID>.json` for each of its files. The collection's name has unsafe
characters replaced and is cut short between two characters so that the
directory's name fits in 255 bytes; the ID keeps the directories apart.
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
fails, the error is logged, each of its files is written with the reason in an
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
@@ -756,9 +778,10 @@ Each original is saved at its save path, the same path `photo.savePath` gives
and `photo.download()` writes (see Read surface below). The date is the photo's
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
time zone of the machine running quak. The extension is the one in the file's
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
letters and digits. When the date or the time zone changes, the next run saves
the original at its new path and leaves the old copy where it is.
name as uploaded, case kept, or `.bin` when it has none, it holds anything but
letters and digits, or it is more than 16 of them long. When the date or the
time zone changes, the next run saves the original at its new path and leaves
the old copy where it is.
`account.json` holds the account's `email` and `userID`, as `backup-metadata`
writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`,
@@ -851,11 +874,12 @@ gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album
named `x.json` gets the directory `x.json (12)/`. A name too long for a file
name, which most file systems limit to 255 bytes, gets its ID added too. A name
with its ID added is cut short between two characters in the part before the ID,
just far enough that it fits. When a symlink's extension alone leaves no room
for the ID, the extension is cut short and the part before the ID keeps only the
bytes that cut leaves over. A collection's directory name also leaves room for
the `.json` its JSON adds. IDs never change, so a name stays the same from run
to run until such a clash appears or goes away.
just far enough that it fits. When a symlink's extension leaves no room for the
ID and the first character of the name, the extension is cut short, and the part
before the ID keeps that character and any bytes the cut leaves over. A
collection's directory name also leaves room for the `.json` its JSON adds. IDs
never change, so a name stays the same from run to run until such a clash
appears or goes away.
A collection whose directory or JSON cannot be written, such as one whose
directory name is taken by a file you put in `collections/`, does not stop the
+26 -3
View File
@@ -17,15 +17,38 @@ pre-1.0
# Next Step
One bug is open: https://git.eeqj.de/sneak/quak/issues/205. The check of quak
against a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an
account only sneak has, so it waits on him.
No bug is open. One change sneak asked for is open:
https://git.eeqj.de/sneak/quak/issues/212. The check of quak against a live Ente
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
has, so it waits on him.
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
# Completed Steps
- 2026-10-18: `quak files` takes the collection ID as an optional argument
instead of the `--collection` option, which is gone
(https://git.eeqj.de/sneak/quak/issues/211). With no argument it lists every
file in the account once, collection by collection; a file in several
collections is listed with the first of them, which is its `collectionID` in
`--json`. The fresh read in `src/cli-read.ts` does this.
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
`src/filename.ts` cuts a name short to a number of bytes, keeping its
extension when it fits and always its first character, and can put text right
before the extension; the backup's `withID` uses it for an ID.
`defaultFileName` there gives the name a file is saved under when the caller
gives no path: a prefix such as `thumb_` and the title with unsafe characters
replaced, cut short with it to 255 bytes, and, for a live photo's image or
video, with that part's own extension before the cut. `quak get`,
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
extension of more than 16 letters and digits, so a save path's name always
fits.
- 2026-10-18: `quak collections` no longer waits for the files of every
collection (issue 210). On an empty cache it opened its library with a refresh
that fetched and decrypted the files of each collection in turn, printing
+7 -6
View File
@@ -78,14 +78,15 @@ program
program
.command("files")
.description("List files in a collection")
.requiredOption(
"--collection <id>",
"Collection ID (from `quak collections`)",
.description("List the files in a collection, or every file in the account")
.argument(
"[collectionID]",
"Collection ID (from `quak collections`); without it, every file " +
"is listed once",
)
.option("--json", "Output as JSON array")
.action((opts: { collection: string; json?: boolean }) =>
run(filesCommand(context(), opts)),
.action((collectionID: string | undefined, opts: { json?: boolean }) =>
run(filesCommand(context(), collectionID, opts)),
);
program
+8 -8
View File
@@ -82,6 +82,7 @@ import { describeError } from "./errors.js";
import {
MAX_NAME_BYTES,
sanitizeFileName,
shortenKeepingExtension,
shortenToBytes,
withExtension,
} from "./filename.js";
@@ -289,9 +290,10 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
// case (two names that differ only in case are one entry on a case-insensitive
// file system), or it is too long for a file name; then it gets ` (<id>)`,
// right before the extension when `beforeExtension` is set. To fit, the part
// before the ID is cut short as far as it must be. When the extension alone
// leaves no room for the ID, the extension is cut short and the part before
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each
// before the ID is cut short as far as it must be. When the extension leaves
// no room for the ID and the first character of the name, the extension is
// cut short, and the part before the ID keeps that character and any bytes
// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each
// entry also has the JSON file `<name>.json` beside it, as an album does: a
// name then leaves room for `.json`, and one that is another entry's JSON name
// gets its ID added too.
@@ -305,11 +307,9 @@ const uniqueNames = (
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
const withID = (id: number, name: string): string => {
const tag = ` (${id})`;
const fullExt = opts.beforeExtension ? extname(name) : "";
const stem = name.slice(0, name.length - fullExt.length);
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag));
const room = maxBytes - Buffer.byteLength(tag + ext);
return shortenToBytes(stem, room) + tag + ext;
return opts.beforeExtension
? shortenKeepingExtension(name, maxBytes, tag)
: shortenToBytes(name, maxBytes - Buffer.byteLength(tag)) + tag;
};
const names = entries.map((e) => e.name);
const suffixed = new Set<number>();
+17 -7
View File
@@ -269,15 +269,21 @@ export const collectionsCommand = async (
}
};
// Lists the files of the collection `collectionIDStr` names, or without it every
// file in the account, each once (issue #211).
export const filesCommand = async (
ctx: CliContext,
opts: { collection: string; json?: boolean },
collectionIDStr: string | undefined,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const collectionID = parseID(ctx, "collection", opts.collection);
if (collectionID === undefined) return 1;
const collectionID =
collectionIDStr === undefined
? undefined
: parseID(ctx, "collection", collectionIDStr);
if (collectionIDStr !== undefined && collectionID === undefined) return 1;
const lib = await openReadLibrary(ctx, client, opts);
try {
@@ -332,8 +338,8 @@ export const getCommand = async (
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
if (result.videoPath === undefined) {
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
@@ -341,9 +347,13 @@ export const getCommand = async (
// A live photo is written as its image and its video, each named after
// the title with its own extension, as Ente's clients name them. With
// --out, the image goes there and the video beside it.
const imageOut =
opts.out ?? withExtension(outPath, extname(result.path));
const videoOut = withExtension(outPath, extname(result.videoPath));
const imageExt = extname(result.path);
const videoExt = extname(result.videoPath);
const imageOut = opts.out ?? originalName(file, imageExt);
const videoOut =
opts.out === undefined
? originalName(file, videoExt)
: withExtension(opts.out, videoExt);
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
ctx.stderr.write(
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
+9 -7
View File
@@ -9,7 +9,7 @@
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
// the commands shape their output from the raw `EnteFile` through here.
import { sanitizeFileName } from "./filename.js";
import { defaultFileName } from "./filename.js";
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
// One row of `quak files --json`.
@@ -33,11 +33,13 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
export const fileListLine = (file: EnteFile): string =>
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
// Default output path for `quak get` when `--out` is not given. The title comes
// from the server, so it is sanitized; `--out` is the user's and is used as is.
export const originalName = (file: EnteFile): string =>
sanitizeFileName(file.metadata.title, `file-${file.id}`);
// Default output path for `quak get` when `--out` is not given, made from the
// title (see `defaultFileName`), with `ext` in place of its extension for a
// live photo's image or video; `--out` is the user's and is used as is.
export const originalName = (file: EnteFile, ext?: string): string =>
defaultFileName(file.metadata.title, file.id, { ext });
// Default output path for `quak get-thumb` when `--out` is not given.
// Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
// and the title.
export const thumbnailName = (file: EnteFile): string =>
`thumb_${originalName(file)}`;
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
+25 -6
View File
@@ -1,6 +1,6 @@
// How the CLI's read commands obtain current data.
//
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
// `collections`, `files`, `get`, and `get-thumb` must answer for
// the account's state at the moment the command runs, not for whatever the
// local cache last happened to hold (owner amendment, issue #36). `files`,
// `get` and `get-thumb` therefore force a server round-trip through
@@ -14,7 +14,10 @@
// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
// rather than the `albums`/`photos` projection's newest-first order, which
// re-sorts the rows. The field values still come from each record's raw
// metadata via `cli-output.ts`.
// metadata via `cli-output.ts`. `files` without a collection ID lists every
// file in the account once (issue #211): the library keeps a record of a file
// for each collection that holds it, and only the first collection's record is
// listed.
import type { Collection, EnteFile } from "./model/types.js";
import type { Photo, PhotosAPI } from "./library/index.js";
@@ -27,6 +30,7 @@ export interface FreshReadLibrary {
fresh(): Promise<unknown>;
freshCollections(): Promise<Collection[]>;
getCollection(id: number): Collection | undefined;
listCollections(): Collection[];
listFiles(collectionID: number): EnteFile[];
getFileByID(fileID: number): EnteFile | undefined;
photos: Pick<PhotosAPI, "byID">;
@@ -40,14 +44,29 @@ export const freshCollections = (
// The files of one collection, current as of a forced refresh, in enumeration
// order. `undefined` (not an empty list) when the collection does not exist, so
// the caller can tell "no such collection" from "an empty collection".
// the caller can tell "no such collection" from "an empty collection". Without
// a collection ID, every file in the account, each once, as held by the first
// collection that holds it.
export const freshFiles = async (
lib: FreshReadLibrary,
collectionID: number,
collectionID?: number,
): Promise<EnteFile[] | undefined> => {
await lib.fresh();
if (!lib.getCollection(collectionID)) return undefined;
return lib.listFiles(collectionID);
if (collectionID !== undefined) {
return lib.getCollection(collectionID)
? lib.listFiles(collectionID)
: undefined;
}
const listed = new Set<number>();
const files: EnteFile[] = [];
for (const collection of lib.listCollections()) {
for (const file of lib.listFiles(collection.id)) {
if (listed.has(file.id)) continue;
listed.add(file.id);
files.push(file);
}
}
return files;
};
// One file, current as of a forced refresh, resolved to both its content
+28 -11
View File
@@ -16,7 +16,7 @@ import {
streamTagFinal,
} from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js";
import { safeExtension, sanitizeFileName, withExtension } from "../filename.js";
import { defaultFileName, safeExtension, withExtension } from "../filename.js";
import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js";
@@ -365,10 +365,14 @@ const openPart = async (
// name starts with `image` as the image and the one whose name starts with
// `video` as the video, and refuses a ZIP holding a second of either. It is
// written unpacked: each part is named `destination` with the extension
// replaced by its own entry's, and the two must differ ignoring case. When the
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
// part's own bytes. Only then are the image, then the video, renamed into
// place; on any failure neither is stored.
// replaced by its own entry's, and the two must differ ignoring case. When
// `defaultName` is set, `destination` is the name `defaultFileName` made from
// the title rather than a path the caller gave, and each part is named as
// `defaultFileName` names it with its own extension, so that the name is cut
// short to fit with that extension. When the file records a hash,
// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
// Only then are the image, then the video, renamed into place; on any failure
// neither is stored.
//
// The ZIP is chosen by its uploader and may expand enormously, so each part is
// written as it decompresses and never held, and the ZIP is refused once the
@@ -388,6 +392,7 @@ const decryptLivePhoto = async (
key: Uint8Array,
onProgress: ProgressCallback | undefined,
file: EnteFile,
defaultName: boolean,
): Promise<DownloadResult> => {
const sliceSize = 4096;
const dir = dirname(destination);
@@ -486,8 +491,12 @@ const decryptLivePhoto = async (
`live photo's image and video have the same extension, ${video.ext}`,
);
}
const path = withExtension(destination, image.ext);
const videoPath = withExtension(destination, video.ext);
const partPath = (ext: string): string =>
defaultName
? defaultFileName(file.metadata.title, file.id, { ext })
: withExtension(destination, ext);
const path = partPath(image.ext);
const videoPath = partPath(video.ext);
for (const part of parts) {
await part.handle.sync();
await part.handle.close();
@@ -531,6 +540,9 @@ const decryptLivePhoto = async (
// and then removes its own temp file, and only the attempt that reaches
// TAG_FINAL renames one into place, so a download that needed three tries still
// performs exactly one rename over the destination.
//
// `defaultName` says `destination` is the name `defaultFileName` made, not a
// path the caller gave (see `decryptLivePhoto`).
const fetchAndDecrypt = async (
api: ApiClient,
openStream: () => Promise<ReadableStream<Uint8Array>>,
@@ -539,6 +551,7 @@ const fetchAndDecrypt = async (
destination: string,
onProgress?: ProgressCallback,
original?: EnteFile,
defaultName = false,
): Promise<DownloadResult> =>
withRetry(async () => {
const stream = await openStream();
@@ -551,6 +564,7 @@ const fetchAndDecrypt = async (
key,
onProgress,
original,
defaultName,
);
}
const bytesWritten = await decryptToTemp(
@@ -582,10 +596,11 @@ export const downloadFile = async (
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
// `outPath` is the caller's and is used as is; the title is the server's
// and is sanitized so it can only name a file in the current directory.
// `outPath` is the caller's and is used as is. Without it, the file is
// written in the current directory under the name `defaultFileName` makes
// from the title, which is the server's.
const resolvedPath =
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
outPath ?? defaultFileName(file.metadata.title, file.id);
const header = fromBase64(file.file.decryptionHeader);
return fetchAndDecrypt(
api,
@@ -595,6 +610,7 @@ export const downloadFile = async (
resolvedPath,
onProgress,
file,
outPath === undefined,
);
};
@@ -604,9 +620,10 @@ export const downloadThumbnail = async (
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
// As in `downloadFile`, with `thumb_` before the title.
const resolvedPath =
outPath ??
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
const header = fromBase64(file.thumbnail.decryptionHeader);
return fetchAndDecrypt(
api,
+56 -4
View File
@@ -5,7 +5,9 @@
// used on disk goes through here, so it can only ever name one file inside the
// directory the caller chose: never a path, never `..`, never hidden, never a
// Windows device name. Such a name can still be too long for a file name, and
// `shortenToBytes` cuts one short.
// `shortenToBytes` or `shortenKeepingExtension` cuts one short.
// `defaultFileName` does both for the name a file is saved under when the
// caller gives no path.
//
// A path the user typed (`--out`, `outPath`) is not passed through here: the
// caller is trusted, the server is not.
@@ -30,11 +32,11 @@ export const sanitizeFileName = (name: string, fallback: string): string => {
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
};
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
// anything but letters and digits.
// The extension of `title` (".jpg"), or ".bin" when it has none, it holds
// anything but letters and digits, or it is more than 16 of them long.
export const safeExtension = (title: string): string => {
const ext = extname(title);
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin";
};
// `name` with its extension, if it has one, replaced by `ext` (".mov").
@@ -57,3 +59,53 @@ export const shortenToBytes = (name: string, maxBytes: number): string => {
}
return name.slice(0, end);
};
// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
// makes it. The part before the extension is cut first, so the extension is
// kept whenever it fits, but that part always keeps its first character:
// when the extension leaves no room for `insert` and that character, the
// extension is cut short too, and the part before it keeps the bytes that cut
// leaves over. So a name that does not start with a dot never comes back
// starting with one, which would hide it. `insert` is never cut. With no
// `insert`, a name that fits comes back unchanged.
export const shortenKeepingExtension = (
name: string,
maxBytes: number,
insert = "",
): string => {
const fullExt = extname(name);
const stem = name.slice(0, name.length - fullExt.length);
const first = Array.from(stem)[0] ?? "";
const ext = shortenToBytes(
fullExt,
maxBytes - Buffer.byteLength(insert + first),
);
const room = maxBytes - Buffer.byteLength(insert + ext);
return shortenToBytes(stem, room) + insert + ext;
};
// The name a file is saved under in the working directory when the caller
// gives no path (`quak get` without `--out`, `downloadFile` without
// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
// unsafe characters replaced, or `file-<id>` when the title is empty, cut
// short so that the whole fits in a file name, keeping the title's extension
// whenever it fits. `ext` (".mov"), when given, takes the place of the
// title's extension before the name is cut short; a live photo's image and
// video are each named with their own. The cut can leave a device name
// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
// leave room for it.
export const defaultFileName = (
title: string,
fileID: number,
opts: { prefix?: string; ext?: string } = {},
): string => {
const { prefix = "", ext } = opts;
const safe = sanitizeFileName(title, `file-${fileID}`);
const name = ext === undefined ? safe : withExtension(safe, ext);
const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
const cut = prefix + shortenKeepingExtension(name, room);
if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
};
+14 -2
View File
@@ -5,7 +5,11 @@ import type { Client } from "./client.js";
import { describeError } from "./errors.js";
import { readExifTags } from "./exif.js";
import type { Library, Photo } from "./library/index.js";
import { sanitizeFileName } from "./filename.js";
import {
MAX_NAME_BYTES,
sanitizeFileName,
shortenToBytes,
} from "./filename.js";
import {
fetchMLDataBatch,
MLDATA_BATCH_SIZE,
@@ -127,7 +131,15 @@ export const runMetadataBackup = async (
const col = lib.getCollection(album.collectionID);
if (!col) continue;
const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
// The album's ID keeps the folders apart, so the name after it can be
// cut short to fit in a file name.
const prefix = `${col.id}-`;
const dirName =
prefix +
shortenToBytes(
sanitizeFileName(col.name, "unnamed"),
MAX_NAME_BYTES - prefix.length,
);
const colDir = join(outDir, "collections", dirName);
mkdirSync(colDir, { recursive: true });
+1 -1
View File
@@ -548,7 +548,7 @@ describe("the read commands with records quak cannot decrypt", () => {
expect(await collectionsCommand(context(ente), {})).toBe(0);
expect(stdout.text).toBe("1\talbum\tTrip\n");
expect(await filesCommand(context(ente), { collection: "1" })).toBe(0);
expect(await filesCommand(context(ente), "1", {})).toBe(0);
expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
});
});
+10 -11
View File
@@ -1795,12 +1795,11 @@ describe("backup album folders", () => {
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
});
// The extensions of these titles leave no room for the ID, so each is cut
// short and the name before the ID keeps only the bytes that cut leaves
// over. File 1's extension is 303 bytes and is cut to 251, which leaves
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three
// bytes, which leaves one, for the "a". Neither extension is only letters
// and digits, so both originals are saved with `.bin`, a name that fits.
// The extensions of these titles leave no room for the ID and the "a"
// before them, so each is cut short to 250 bytes, which leaves one, for
// the "a". File 1's extension is 303 bytes and file 2's 271. Neither
// extension is only letters and digits, so both originals are saved with
// `.bin`, a name that fits.
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
const outDir = join(root, "backup");
const lib = libraryOf([
@@ -1815,7 +1814,7 @@ describe("backup album folders", () => {
const result = await runBackup(lib, { downloadDirectory: outDir });
const link = ` (1).${"L".repeat(250)}`;
const link = `a (1).${"L".repeat(249)}`;
const link2 = `a (2).${"€".repeat(83)}`;
expect(Buffer.byteLength(link)).toBe(255);
expect(Buffer.byteLength(link2)).toBe(255);
@@ -1830,9 +1829,9 @@ describe("backup album folders", () => {
// Files 1 and 3 share a title, so each gets its ID before the
// extension. File 2's title is too long, and its extension leaves no
// room for the ID. Were the ID put at the end of file 2's title cut
// short, its link would have file 1's name, and the link made second
// would replace the first.
// room for the ID and the "x" before it. Were the ID put at the end of
// file 2's title cut short, its link would have file 1's name, and the
// link made second would replace the first.
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
const outDir = join(root, "backup");
const shared = `x.${"Q".repeat(245)} (2)`;
@@ -1852,8 +1851,8 @@ describe("backup album folders", () => {
expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([
"Trip/",
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
`Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`,
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
"Trip.json",
]);
+24 -6
View File
@@ -1,6 +1,7 @@
/**
* Tests for the retry options `bin/quak.ts` loads the saved session with:
* `quak backup` gets the unattended ones, every other command the default.
* Tests for the retry options `bin/quak.ts` loads the saved session with
* (`quak backup` gets the unattended ones, every other command the default),
* and for `quak files` taking its collection ID as an optional argument.
*
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
* replaced by one that records the options it is given and reports no session,
@@ -26,14 +27,19 @@ vi.mock("../../src/cli-session.js", () => ({
const argv = process.argv;
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
// Run `quak <args>` to completion. The "Not logged in" message and the exit
// are swallowed.
const quak = async (...args: string[]): Promise<void> => {
vi.spyOn(process.stderr, "write").mockImplementation(() => true);
// Run `quak <args>` to completion and return what it wrote on stderr, which is
// kept off the terminal. The exit is swallowed.
const quak = async (...args: string[]): Promise<string> => {
let written = "";
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
written += String(text);
return true;
});
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
process.argv = ["node", "quak", ...args];
vi.resetModules();
await import("../../bin/quak.js");
return written;
};
afterEach(() => {
@@ -69,3 +75,15 @@ describe("bin/quak.ts session loading", () => {
expect(loaded).toEqual([undefined]);
});
});
// The command line takes `files` with or without a collection ID, and the
// command runs as far as its session check either way (issue #211).
describe("bin/quak.ts files", () => {
it("runs without a collection ID", async () => {
expect(await quak("files")).toMatch(/^Not logged in/);
});
it("runs with a collection ID", async () => {
expect(await quak("files", "7")).toMatch(/^Not logged in/);
});
});
+207 -47
View File
@@ -21,7 +21,7 @@ import {
utimesSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { basename, join } from "node:path";
import { tmpdir } from "node:os";
import { PassThrough } from "node:stream";
import * as jpegJs from "jpeg-js";
@@ -121,25 +121,31 @@ const FILES: Record<number, EnteFile[]> = {
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
// original fail; `emptyThumbID` makes the server report that file's
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
// the collection's updationTime as the server does, and `refreshError` makes
// listing collections fail with that message.
// the collection's updationTime as the server does. `beachInWork` adds
// beach.jpg (100) to Work as well, so that one file is in both albums.
// `refreshError` makes listing collections fail with that message.
const fakeClient = (
opts: {
failID?: number;
emptyThumbID?: number;
withNewFile?: boolean;
beachInWork?: boolean;
refreshError?: string;
} = {},
) => {
const collections = opts.withNewFile
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
: COLLECTIONS;
const files = opts.withNewFile
? {
...FILES,
1: [...FILES[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
}
: FILES;
let files = FILES;
if (opts.withNewFile) {
files = {
...files,
1: [...files[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
};
}
if (opts.beachInWork) {
files = { ...files, 2: [...files[2], file(100, 2, "beach.jpg")] };
}
const source: ContentSource = {
original: async ({ file: f, destination }) => {
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
@@ -187,6 +193,9 @@ const FETCHED_BOTH_ALBUMS =
"Fetched the files of 1 of 2 collections\n" +
"Fetched the files of 2 of 2 collections\n";
// The same for an account with one album.
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
// Collects everything written to it.
class Output {
text = "";
@@ -254,7 +263,7 @@ describe("session file", () => {
const dir = join(root, "backup");
expect(await whoamiCommand(ctx)).toBe(3);
expect(await collectionsCommand(ctx, {})).toBe(3);
expect(await filesCommand(ctx, { collection: "1" })).toBe(3);
expect(await filesCommand(ctx, "1", {})).toBe(3);
expect(await getCommand(ctx, "100", {})).toBe(3);
expect(await getThumbCommand(ctx, "100", {})).toBe(3);
expect(await backupMetadataCommand(ctx, dir, {})).toBe(3);
@@ -501,17 +510,14 @@ describe("collections", () => {
describe("files", () => {
it("prints one tab-separated line per file", async () => {
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
expect(await filesCommand(context(), "1", {})).toBe(0);
expect(stdout.text).toBe(
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
);
});
it("prints a JSON array with --json", async () => {
const code = await filesCommand(context(), {
collection: "2",
json: true,
});
const code = await filesCommand(context(), "2", { json: true });
expect(code).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
@@ -525,11 +531,43 @@ describe("files", () => {
});
it("exits 1 for an unknown collection", async () => {
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
expect(await filesCommand(context(), "9", {})).toBe(1);
expect(stderr.text).toBe(
FETCHED_BOTH_ALBUMS + "Collection 9 not found\n",
);
});
// Without a collection ID it lists every file in the account once (issue
// #211). beach.jpg is in Vacation and Work, and Vacation comes first, so it
// is listed once, as Vacation's.
it("without a collection ID prints every file once", async () => {
const ctx = context(fakeClient({ beachInWork: true }));
expect(await filesCommand(ctx, undefined, {})).toBe(0);
expect(stdout.text).toBe(
"100\timage\tbeach.jpg\n" +
"101\timage\tsunset.jpg\n" +
"200\timage\tdiagram.png\n",
);
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
});
it("without a collection ID gives each file the first album holding it with --json", async () => {
const ctx = context(fakeClient({ beachInWork: true }));
expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
expect(
JSON.parse(stdout.text).map(
(row: { id: number; collectionID: number }) => [
row.id,
row.collectionID,
],
),
).toEqual([
[100, 1],
[101, 1],
[200, 2],
]);
expect(stderr.text).toBe("");
});
});
describe("get and get-thumb", () => {
@@ -562,17 +600,109 @@ describe("get and get-thumb", () => {
});
});
// A file system refuses a file name over 255 bytes. The account here is one
// album, whose name is 300 bytes, holding one file, 700, with the title given.
describe("names from an album name or a title over 255 bytes", () => {
const longNames = (title: string): Client =>
({
...fakeClient(),
collectionsSince: async () => ({
collections: [collection(7, "A".repeat(300))],
deleted: [],
cursor: 1,
}),
filesSince: async () => ({
files: [file(700, 7, title)],
deleted: [],
cursor: 1,
}),
}) as unknown as Client;
const longTitle = `${"t".repeat(296)}.jpg`;
// Run `command` in a new directory, where `get` and `get-thumb` without
// `--out` write, and return its exit code and the directory.
const inNewDirectory = async (
command: () => Promise<number>,
): Promise<{ code: number; dir: string }> => {
const dir = mkdtempSync(join(root, "cwd-"));
const previous = process.cwd();
process.chdir(dir);
try {
return { code: await command(), dir };
} finally {
process.chdir(previous);
}
};
it("get cuts the title short to 255 bytes, keeping its extension, and exits 0", async () => {
const { code, dir } = await inNewDirectory(() =>
getCommand(context(longNames(longTitle)), "700", {}),
);
const name = `${"t".repeat(251)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(dir)).toEqual([name]);
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
});
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
const { code, dir } = await inNewDirectory(() =>
getThumbCommand(context(longNames(longTitle)), "700", {}),
);
const name = `thumb_${"t".repeat(245)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(dir)).toEqual([name]);
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
});
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
const dir = join(root, "dump");
const code = await backupMetadataCommand(
context(longNames("beach.jpg")),
dir,
{},
);
const folder = `7-${"A".repeat(253)}`;
expect(Buffer.byteLength(folder)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(join(dir, "collections"))).toEqual([folder]);
expect(readdirSync(join(dir, "collections", folder)).sort()).toEqual([
"700.json",
"_collection.json",
]);
});
// The title's extension is 300 letters, too long for the save path's
// name, so the original is saved with `.bin`.
it("backup saves a title with a 300-letter extension as .bin, and exits 0", async () => {
const dir = join(root, "backup");
const title = `b.${"e".repeat(300)}`;
const code = await backupCommand(context(longNames(title)), dir, {});
const path = savePath(dir, file(700, 7, title));
expect(code).toBe(0);
expect(basename(path)).toMatch(/^\d{4}-\d{2}-\d{2}\.700\.bin$/);
expect(readFileSync(path)).toEqual(Buffer.alloc(7, 700 & 0xff));
});
});
// A live photo, which Ente stores as one ZIP, is written as its image and its
// video, which a photo viewer can open. The account here is Vacation holding
// one live photo, 300, downloaded through the real download layer
// (test/live-photo.ts).
describe("a live photo", () => {
// What `get` prints on stderr while its refresh fetches Vacation's files.
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
const livePhotoClient = async (image = IMAGE): Promise<Client> => {
const livePhotoClient = async (
image = IMAGE,
title = "IMG_0300.HEIC",
): Promise<Client> => {
const { file: live, body } = await asLivePhoto(
file(300, 1, "IMG_0300.HEIC"),
file(300, 1, title),
livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
livePhotoHash(image, VIDEO),
);
@@ -620,6 +750,44 @@ describe("a live photo", () => {
);
});
// Run `get` without `--out` in a new directory for a live photo titled
// `title`, and return its exit code and the names it wrote.
const getInNewDirectory = async (
title: string,
): Promise<{ code: number; names: string[] }> => {
const ctx = context(await livePhotoClient(IMAGE, title));
const dir = mkdtempSync(join(root, "cwd-"));
const previous = process.cwd();
process.chdir(dir);
try {
const code = await getCommand(ctx, "300", {});
return { code, names: readdirSync(dir).sort() };
} finally {
process.chdir(previous);
}
};
const imageName = `${"t".repeat(250)}.heic`;
const videoName = `${"t".repeat(251)}.mov`;
// The title fits, but ".heic" is one byte longer than ".jpg", so the
// image's name is cut short once its extension is in it.
it("get cuts its image's name short for a 255-byte title ending in .jpg, and exits 0", async () => {
const title = `${"t".repeat(251)}.jpg`;
const result = await getInNewDirectory(title);
expect(Buffer.byteLength(title)).toBe(255);
expect(Buffer.byteLength(imageName)).toBe(255);
expect(Buffer.byteLength(videoName)).toBe(255);
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
});
it("get cuts the names of its image and video short for a 300-byte title with no extension, and exits 0", async () => {
const result = await getInNewDirectory("t".repeat(300));
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
});
it("get --out writes the image there and the video beside it", async () => {
const out = join(root, "photo.jpg");
const video = join(root, "photo.mov");
@@ -1132,7 +1300,7 @@ describe("backup-metadata --exif", () => {
// collection list alone and stores nothing.)
describe("backup-metadata and the thumbnail helpers refresh first", () => {
beforeEach(async () => {
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
expect(await filesCommand(context(), "1", {})).toBe(0);
stdout.text = "";
stderr.text = "";
});
@@ -1191,10 +1359,10 @@ describe("backup-metadata and the thumbnail helpers refresh first", () => {
});
});
// `files --collection`, `get`, `get-thumb` and `helper fix-missing-thumbnails
// --file` take an ID only when it is written in decimal digits and is a safe
// integer. `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty
// string as 0, so the command would act on a file the user never named.
// `files`, `get`, `get-thumb` and `helper fix-missing-thumbnails --file` take
// an ID only when it is written in decimal digits and is a safe integer.
// `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty string as 0,
// so the command would act on a file the user never named.
// Anything else is refused with one line naming it, and exit 1, before any
// request.
//
@@ -1248,21 +1416,18 @@ describe("an ID given on the command line", () => {
it("files looks up a decimal collection ID", async () => {
const ctx = emptyAccount();
expect(await filesCommand(ctx, { collection: "1" })).toBe(1);
expect(await filesCommand(ctx, "1", {})).toBe(1);
expect(stderr.text).toBe("Collection 1 not found\n");
expect(requests).not.toHaveLength(0);
});
it.each(NOT_IDS)(
"files refuses --collection %j before any request",
async (arg) => {
const ctx = emptyAccount();
expect(await filesCommand(ctx, { collection: arg })).toBe(1);
expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
expect(stdout.text).toBe("");
expect(requests).toHaveLength(0);
},
);
it.each(NOT_IDS)("files refuses %j before any request", async (arg) => {
const ctx = emptyAccount();
expect(await filesCommand(ctx, arg, {})).toBe(1);
expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
expect(stdout.text).toBe("");
expect(requests).toHaveLength(0);
});
it("get looks up a decimal file ID", async () => {
expect(await getCommand(emptyAccount(), "100", {})).toBe(1);
@@ -1444,14 +1609,12 @@ describe("an account with many albums", () => {
// It stored nothing that would make a refresh skip an album's files:
// `files` after it still finds the album's photo.
stdout.text = "";
expect(await filesCommand(ctx, { collection: "7" })).toBe(0);
expect(await filesCommand(ctx, "7", {})).toBe(0);
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
});
it("files counts on stderr each album whose files its refresh fetched", async () => {
expect(await filesCommand(accountContext(), { collection: "7" })).toBe(
0,
);
expect(await filesCommand(accountContext(), "7", {})).toBe(0);
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
let progress = "";
@@ -1464,12 +1627,9 @@ describe("an account with many albums", () => {
// With `--json` the refresh's progress is not printed, as `backup` and
// the thumbnail helpers print none of theirs.
it("files --json prints no progress", async () => {
expect(
await filesCommand(accountContext(), {
collection: "7",
json: true,
}),
).toBe(0);
expect(await filesCommand(accountContext(), "7", { json: true })).toBe(
0,
);
expect(JSON.parse(stdout.text)).toEqual([
{
+28
View File
@@ -93,6 +93,10 @@ class FakeLibrary implements FreshReadLibrary {
return this.collections.find((c) => c.id === id);
}
listCollections(): Collection[] {
return this.refreshed ? this.collections : [];
}
listFiles(collectionID: number): EnteFile[] {
return this.refreshed
? this.files.filter((f) => f.collectionID === collectionID)
@@ -175,6 +179,30 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
]);
});
// Without a collection ID, `files` lists every file in the account once
// (issue #211). File 11 is in collections 1 and 2; collection 2 is
// enumerated first, so 11 is listed once, with collection 2's record.
it("freshFiles without a collection ID lists every file once, from the first collection holding it", async () => {
const files = [
file(10, 1, 1_700_000_000_000_000),
file(11, 1, 1_700_000_000_000_001),
file(20, 2, 1_700_000_000_000_002),
file(11, 2, 1_700_000_000_000_001),
];
const lib = new FakeLibrary(
[collection(2, 100), collection(1, 200)],
files,
);
const rows = await freshFiles(lib);
expect(lib.freshCalls).toBe(1);
expect(rows?.map((f) => [f.id, f.collectionID])).toEqual([
[20, 2],
[11, 2],
[10, 1],
]);
});
it("freshFiles returns undefined for an unknown collection", async () => {
const lib = new FakeLibrary([collection(1, 100)], []);
const rows = await freshFiles(lib, 999);
+89
View File
@@ -658,6 +658,26 @@ describe("downloadFile", () => {
expect(readdirSync(parent)).toEqual(["cwd"]);
});
// A file system refuses a file name over 255 bytes.
it("cuts a title too long for a file name short, keeping its extension", async () => {
const plaintext = new Uint8Array([1, 2, 3]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const thumbPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const file = buildMockEnteFile(key, header, thumbPush.header);
file.metadata.title = `${"t".repeat(296)}.jpg`;
const dir = mkdtempSync(join(testDir, "long-title-"));
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
const result = await inDirectory(dir, () => downloadFile(api, file));
const name = `${"t".repeat(251)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(result.path).toBe(name);
expect(readdirSync(dir)).toEqual([name]);
});
it("uses an explicit outPath verbatim, even one with ..", async () => {
// The caller is trusted: its path is not sanitized.
const { api, file } = fixtureFor(
@@ -751,6 +771,28 @@ describe("downloadThumbnail", () => {
expect(result.path).toBe("thumb__etc_passwd");
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
});
it("leaves room for thumb_ when it cuts a title too long for a file name short", async () => {
const plaintext = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const filePush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const { header: thumbHeader, ciphertext: thumbCipher } =
encryptFileBody(plaintext, key);
const file = buildMockEnteFile(key, filePush.header, thumbHeader);
file.metadata.title = `${"t".repeat(296)}.jpg`;
const dir = mkdtempSync(join(testDir, "long-thumb-title-"));
const api = new ApiClient({ fetch: mockFetchForBody(thumbCipher) });
const result = await inDirectory(dir, () =>
downloadThumbnail(api, file),
);
const name = `thumb_${"t".repeat(245)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(result.path).toBe(name);
expect(readdirSync(dir)).toEqual([name]);
});
});
// ---------------------------------------------------------------------------
@@ -1881,6 +1923,53 @@ describe("downloadFile live photos", () => {
]);
});
// A file system refuses a file name over 255 bytes. Each name is cut
// short once its own extension is in it.
const imageName = `${"t".repeat(250)}.heic`;
const videoName = `${"t".repeat(251)}.mov`;
// The title fits, but ".heic" is one byte longer than ".jpg".
it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => {
const title = `${"t".repeat(251)}.jpg`;
const t = setup(livePhotoZip(), { ...livePhoto, title });
const result = await inDirectory(t.dir, () =>
downloadFile(t.api, t.file),
);
expect(Buffer.byteLength(title)).toBe(255);
expect(Buffer.byteLength(imageName)).toBe(255);
expect(Buffer.byteLength(videoName)).toBe(255);
expect(result.path).toBe(imageName);
expect(result.videoPath).toBe(videoName);
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
});
it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => {
const t = setup(livePhotoZip(), {
...livePhoto,
title: "t".repeat(300),
});
const result = await inDirectory(t.dir, () =>
downloadFile(t.api, t.file),
);
expect(result.path).toBe(imageName);
expect(result.videoPath).toBe(videoName);
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
});
// The caller's path is used as is; only the extension is each part's own.
it("keeps an outPath too long for a file name as given", async () => {
const t = setup(livePhotoZip(), livePhoto);
const outPath = join(t.dir, `${"t".repeat(251)}.jpg`);
await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow(
/ENAMETOOLONG/,
);
});
it("gives each part its own extension from the ZIP, letters and digits only", async () => {
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
const t = setup(zip, livePhoto);
+108
View File
@@ -15,8 +15,10 @@
import { describe, expect, it } from "vitest";
import {
defaultFileName,
safeExtension,
sanitizeFileName,
shortenKeepingExtension,
shortenToBytes,
} from "../../src/filename.js";
@@ -94,6 +96,12 @@ describe("safeExtension", () => {
expect(safeExtension("x.jp g")).toBe(".bin");
expect(safeExtension("x.jpg\0")).toBe(".bin");
});
it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => {
expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`);
expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin");
expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin");
});
});
describe("shortenToBytes", () => {
@@ -118,3 +126,103 @@ describe("shortenToBytes", () => {
);
});
});
describe("shortenKeepingExtension", () => {
it("returns a name that fits unchanged", () => {
expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg");
expect(shortenKeepingExtension("beach", 5)).toBe("beach");
});
it("cuts the part before the extension short and keeps the extension", () => {
expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(shortenKeepingExtension("é".repeat(150), 255)).toBe(
"é".repeat(127),
);
});
// A name cut down to its extension would start with a dot, and so be
// hidden.
it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
`b.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
`a.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
`é.${"e".repeat(252)}`,
);
// "€" takes three bytes, so the extension is cut to 253 and the
// part before it keeps the two bytes left over, for the "a".
expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe(
`a.${"€".repeat(84)}`,
);
});
it("puts `insert` right before the extension and leaves room for it", () => {
expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe(
"beach (7).jpg",
);
expect(
shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"),
).toBe(`${"t".repeat(247)} (7).jpg`);
expect(
shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
).toBe(`b (7).${"e".repeat(249)}`);
});
});
describe("defaultFileName", () => {
it("is the title when it is safe and fits", () => {
expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
});
it("replaces unsafe characters, and is file-<id> for an empty title", () => {
expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
expect(defaultFileName("", 42)).toBe("file-42");
});
it("cuts a title over 255 bytes short, keeping its extension", () => {
expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
`b.${"e".repeat(253)}`,
);
});
it("puts `prefix` before the title and leaves room for it", () => {
expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
"thumb_beach.jpg",
);
expect(
defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
).toBe(`thumb_${"t".repeat(245)}.jpg`);
});
// A live photo's image and video are each named with their own
// extension, and the name is cut short once that extension is in it.
it("puts `ext` in place of the title's extension before cutting the name short", () => {
expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
"IMG_0001.mov",
);
expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
expect(
defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
).toBe(`${"t".repeat(250)}.heic`);
expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
`${"t".repeat(250)}.heic`,
);
});
// Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
// or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
// name, and the part before the extension gives up a byte for it.
it("never returns a Windows device name, even once cut short", () => {
const ext = `.${"e".repeat(251)}`;
expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
});
});