Names quak builds from server data fit in 255 bytes (closes #205)
check / check (push) Waiting to run

The default names of `quak get`, `quak get-thumb`, `downloadFile` and `downloadThumbnail`, and the album folders of `backup-metadata`, could be over 255 bytes, which a file system refuses. `defaultFileName` and `shortenKeepingExtension` in `src/filename.ts` now cut such a name short, keeping its extension when it fits; a live photo's image and video get their own extension before the cut. A cut name never starts with a dot and is never a Windows device name. `--out` and `outPath` are kept as given. A `backup-metadata` folder keeps its `<id>-`. `safeExtension` gives `.bin` for an extension of more than 16 letters and digits, so a save path's name always fits.

Model: opus-5-5
This commit was merged in pull request #213.
This commit is contained in:
2026-10-18 20:01:34 +02:00
parent 982b878d69
commit 66e3ac77e8
12 changed files with 526 additions and 77 deletions
+37 -21
View File
@@ -683,27 +683,41 @@ before any request with one line, `Invalid file ID: <arg>` or
`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses
the whole command, so nothing is fixed.
Without `--out`, `get` writes the file to the working directory under its title,
and `get-thumb` writes the thumbnail there under `thumb_` followed by the title,
each with unsafe characters replaced. A name over 255 bytes, which most file
systems refuse, is cut short between two characters before its extension, so the
extension is kept whenever it fits. An extension too long to leave room for the
name's first character is cut short too, so the name never starts with a dot.
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
accepted for backward compatibility but ignored. For a live photo, `get` writes
its image and its video, each named after the title with its own extension, as
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
`--out PATH`, the image is written to `PATH` and the video beside it, with
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
refused. `backup-metadata --exif` (alias `--all`) additionally fetches each
file's original through the cache and records, from it or a live photo's image,
its XMP metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is
read with [exifreader](https://github.com/mattiasw/ExifReader) from any image
format it reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's
`exif` field is exifreader's EXIF tag output: each tag by name, with its
`value`, `description` and `computed` value. An EXIF block exifreader finds but
reads no tag from is recorded, base64, as `exifRaw`, with the reason in
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is
cut short as above once its own extension is in it. With `--out PATH`, the image
is written to `PATH` and the video beside it, with `PATH`'s name and the video's
extension; a `PATH` with the video's extension is refused.
`backup-metadata --exif` (alias `--all`) additionally fetches each file's
original through the cache and records, from it or a live photo's image, its XMP
metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with
[exifreader](https://github.com/mattiasw/ExifReader) from any image format it
reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field
is exifreader's EXIF tag output: each tag by name, with its `value`,
`description` and `computed` value. An EXIF block exifreader finds but reads no
tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`.
`collections`, `files`, `backup`, `helper list-missing-thumbnails` and
`helper fix-missing-thumbnails` take `--json` for machine-readable output.
`backup --verify` also hashes the originals already in the backup and replaces
any that do not match the content hash Ente records; one it cannot replace stays
as it was and goes into `failures.json` (see "Backup layout").
`backup-metadata <dir>` writes `account.json` and, for each collection, a
directory `collections/<id>-<name>/` holding `_collection.json` and a
`<fileID>.json` for each of its files. The collection's name has unsafe
characters replaced and is cut short between two characters so that the
directory's name fits in 255 bytes; the ID keeps the directories apart.
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
fails, the error is logged, each of its files is written with the reason in an
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
@@ -756,9 +770,10 @@ Each original is saved at its save path, the same path `photo.savePath` gives
and `photo.download()` writes (see Read surface below). The date is the photo's
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
time zone of the machine running quak. The extension is the one in the file's
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
letters and digits. When the date or the time zone changes, the next run saves
the original at its new path and leaves the old copy where it is.
name as uploaded, case kept, or `.bin` when it has none, it holds anything but
letters and digits, or it is more than 16 of them long. When the date or the
time zone changes, the next run saves the original at its new path and leaves
the old copy where it is.
`account.json` holds the account's `email` and `userID`, as `backup-metadata`
writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`,
@@ -851,11 +866,12 @@ gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album
named `x.json` gets the directory `x.json (12)/`. A name too long for a file
name, which most file systems limit to 255 bytes, gets its ID added too. A name
with its ID added is cut short between two characters in the part before the ID,
just far enough that it fits. When a symlink's extension alone leaves no room
for the ID, the extension is cut short and the part before the ID keeps only the
bytes that cut leaves over. A collection's directory name also leaves room for
the `.json` its JSON adds. IDs never change, so a name stays the same from run
to run until such a clash appears or goes away.
just far enough that it fits. When a symlink's extension leaves no room for the
ID and the first character of the name, the extension is cut short, and the part
before the ID keeps that character and any bytes the cut leaves over. A
collection's directory name also leaves room for the `.json` its JSON adds. IDs
never change, so a name stays the same from run to run until such a clash
appears or goes away.
A collection whose directory or JSON cannot be written, such as one whose
directory name is taken by a file you put in `collections/`, does not stop the
+20 -3
View File
@@ -17,15 +17,32 @@ pre-1.0
# Next Step
One bug is open: https://git.eeqj.de/sneak/quak/issues/205. The check of quak
against a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an
account only sneak has, so it waits on him.
No bug is open. Two changes sneak asked for are open:
https://git.eeqj.de/sneak/quak/issues/211 and
https://git.eeqj.de/sneak/quak/issues/212. The check of quak against a live Ente
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
has, so it waits on him.
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
# Completed Steps
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
`src/filename.ts` cuts a name short to a number of bytes, keeping its
extension when it fits and always its first character, and can put text right
before the extension; the backup's `withID` uses it for an ID.
`defaultFileName` there gives the name a file is saved under when the caller
gives no path: a prefix such as `thumb_` and the title with unsafe characters
replaced, cut short with it to 255 bytes, and, for a live photo's image or
video, with that part's own extension before the cut. `quak get`,
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
extension of more than 16 letters and digits, so a save path's name always
fits.
- 2026-10-18: `quak collections` no longer waits for the files of every
collection (issue 210). On an empty cache it opened its library with a refresh
that fetched and decrypted the files of each collection in turn, printing
+8 -8
View File
@@ -82,6 +82,7 @@ import { describeError } from "./errors.js";
import {
MAX_NAME_BYTES,
sanitizeFileName,
shortenKeepingExtension,
shortenToBytes,
withExtension,
} from "./filename.js";
@@ -289,9 +290,10 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
// case (two names that differ only in case are one entry on a case-insensitive
// file system), or it is too long for a file name; then it gets ` (<id>)`,
// right before the extension when `beforeExtension` is set. To fit, the part
// before the ID is cut short as far as it must be. When the extension alone
// leaves no room for the ID, the extension is cut short and the part before
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each
// before the ID is cut short as far as it must be. When the extension leaves
// no room for the ID and the first character of the name, the extension is
// cut short, and the part before the ID keeps that character and any bytes
// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each
// entry also has the JSON file `<name>.json` beside it, as an album does: a
// name then leaves room for `.json`, and one that is another entry's JSON name
// gets its ID added too.
@@ -305,11 +307,9 @@ const uniqueNames = (
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
const withID = (id: number, name: string): string => {
const tag = ` (${id})`;
const fullExt = opts.beforeExtension ? extname(name) : "";
const stem = name.slice(0, name.length - fullExt.length);
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag));
const room = maxBytes - Buffer.byteLength(tag + ext);
return shortenToBytes(stem, room) + tag + ext;
return opts.beforeExtension
? shortenKeepingExtension(name, maxBytes, tag)
: shortenToBytes(name, maxBytes - Buffer.byteLength(tag)) + tag;
};
const names = entries.map((e) => e.name);
const suffixed = new Set<number>();
+8 -4
View File
@@ -332,8 +332,8 @@ export const getCommand = async (
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
if (result.videoPath === undefined) {
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
@@ -341,9 +341,13 @@ export const getCommand = async (
// A live photo is written as its image and its video, each named after
// the title with its own extension, as Ente's clients name them. With
// --out, the image goes there and the video beside it.
const imageOut =
opts.out ?? withExtension(outPath, extname(result.path));
const videoOut = withExtension(outPath, extname(result.videoPath));
const imageExt = extname(result.path);
const videoExt = extname(result.videoPath);
const imageOut = opts.out ?? originalName(file, imageExt);
const videoOut =
opts.out === undefined
? originalName(file, videoExt)
: withExtension(opts.out, videoExt);
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
ctx.stderr.write(
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
+9 -7
View File
@@ -9,7 +9,7 @@
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
// the commands shape their output from the raw `EnteFile` through here.
import { sanitizeFileName } from "./filename.js";
import { defaultFileName } from "./filename.js";
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
// One row of `quak files --json`.
@@ -33,11 +33,13 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
export const fileListLine = (file: EnteFile): string =>
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
// Default output path for `quak get` when `--out` is not given. The title comes
// from the server, so it is sanitized; `--out` is the user's and is used as is.
export const originalName = (file: EnteFile): string =>
sanitizeFileName(file.metadata.title, `file-${file.id}`);
// Default output path for `quak get` when `--out` is not given, made from the
// title (see `defaultFileName`), with `ext` in place of its extension for a
// live photo's image or video; `--out` is the user's and is used as is.
export const originalName = (file: EnteFile, ext?: string): string =>
defaultFileName(file.metadata.title, file.id, { ext });
// Default output path for `quak get-thumb` when `--out` is not given.
// Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
// and the title.
export const thumbnailName = (file: EnteFile): string =>
`thumb_${originalName(file)}`;
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
+28 -11
View File
@@ -16,7 +16,7 @@ import {
streamTagFinal,
} from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js";
import { safeExtension, sanitizeFileName, withExtension } from "../filename.js";
import { defaultFileName, safeExtension, withExtension } from "../filename.js";
import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js";
@@ -365,10 +365,14 @@ const openPart = async (
// name starts with `image` as the image and the one whose name starts with
// `video` as the video, and refuses a ZIP holding a second of either. It is
// written unpacked: each part is named `destination` with the extension
// replaced by its own entry's, and the two must differ ignoring case. When the
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
// part's own bytes. Only then are the image, then the video, renamed into
// place; on any failure neither is stored.
// replaced by its own entry's, and the two must differ ignoring case. When
// `defaultName` is set, `destination` is the name `defaultFileName` made from
// the title rather than a path the caller gave, and each part is named as
// `defaultFileName` names it with its own extension, so that the name is cut
// short to fit with that extension. When the file records a hash,
// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
// Only then are the image, then the video, renamed into place; on any failure
// neither is stored.
//
// The ZIP is chosen by its uploader and may expand enormously, so each part is
// written as it decompresses and never held, and the ZIP is refused once the
@@ -388,6 +392,7 @@ const decryptLivePhoto = async (
key: Uint8Array,
onProgress: ProgressCallback | undefined,
file: EnteFile,
defaultName: boolean,
): Promise<DownloadResult> => {
const sliceSize = 4096;
const dir = dirname(destination);
@@ -486,8 +491,12 @@ const decryptLivePhoto = async (
`live photo's image and video have the same extension, ${video.ext}`,
);
}
const path = withExtension(destination, image.ext);
const videoPath = withExtension(destination, video.ext);
const partPath = (ext: string): string =>
defaultName
? defaultFileName(file.metadata.title, file.id, { ext })
: withExtension(destination, ext);
const path = partPath(image.ext);
const videoPath = partPath(video.ext);
for (const part of parts) {
await part.handle.sync();
await part.handle.close();
@@ -531,6 +540,9 @@ const decryptLivePhoto = async (
// and then removes its own temp file, and only the attempt that reaches
// TAG_FINAL renames one into place, so a download that needed three tries still
// performs exactly one rename over the destination.
//
// `defaultName` says `destination` is the name `defaultFileName` made, not a
// path the caller gave (see `decryptLivePhoto`).
const fetchAndDecrypt = async (
api: ApiClient,
openStream: () => Promise<ReadableStream<Uint8Array>>,
@@ -539,6 +551,7 @@ const fetchAndDecrypt = async (
destination: string,
onProgress?: ProgressCallback,
original?: EnteFile,
defaultName = false,
): Promise<DownloadResult> =>
withRetry(async () => {
const stream = await openStream();
@@ -551,6 +564,7 @@ const fetchAndDecrypt = async (
key,
onProgress,
original,
defaultName,
);
}
const bytesWritten = await decryptToTemp(
@@ -582,10 +596,11 @@ export const downloadFile = async (
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
// `outPath` is the caller's and is used as is; the title is the server's
// and is sanitized so it can only name a file in the current directory.
// `outPath` is the caller's and is used as is. Without it, the file is
// written in the current directory under the name `defaultFileName` makes
// from the title, which is the server's.
const resolvedPath =
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
outPath ?? defaultFileName(file.metadata.title, file.id);
const header = fromBase64(file.file.decryptionHeader);
return fetchAndDecrypt(
api,
@@ -595,6 +610,7 @@ export const downloadFile = async (
resolvedPath,
onProgress,
file,
outPath === undefined,
);
};
@@ -604,9 +620,10 @@ export const downloadThumbnail = async (
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
// As in `downloadFile`, with `thumb_` before the title.
const resolvedPath =
outPath ??
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
const header = fromBase64(file.thumbnail.decryptionHeader);
return fetchAndDecrypt(
api,
+56 -4
View File
@@ -5,7 +5,9 @@
// used on disk goes through here, so it can only ever name one file inside the
// directory the caller chose: never a path, never `..`, never hidden, never a
// Windows device name. Such a name can still be too long for a file name, and
// `shortenToBytes` cuts one short.
// `shortenToBytes` or `shortenKeepingExtension` cuts one short.
// `defaultFileName` does both for the name a file is saved under when the
// caller gives no path.
//
// A path the user typed (`--out`, `outPath`) is not passed through here: the
// caller is trusted, the server is not.
@@ -30,11 +32,11 @@ export const sanitizeFileName = (name: string, fallback: string): string => {
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
};
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
// anything but letters and digits.
// The extension of `title` (".jpg"), or ".bin" when it has none, it holds
// anything but letters and digits, or it is more than 16 of them long.
export const safeExtension = (title: string): string => {
const ext = extname(title);
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin";
};
// `name` with its extension, if it has one, replaced by `ext` (".mov").
@@ -57,3 +59,53 @@ export const shortenToBytes = (name: string, maxBytes: number): string => {
}
return name.slice(0, end);
};
// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
// makes it. The part before the extension is cut first, so the extension is
// kept whenever it fits, but that part always keeps its first character:
// when the extension leaves no room for `insert` and that character, the
// extension is cut short too, and the part before it keeps the bytes that cut
// leaves over. So a name that does not start with a dot never comes back
// starting with one, which would hide it. `insert` is never cut. With no
// `insert`, a name that fits comes back unchanged.
export const shortenKeepingExtension = (
name: string,
maxBytes: number,
insert = "",
): string => {
const fullExt = extname(name);
const stem = name.slice(0, name.length - fullExt.length);
const first = Array.from(stem)[0] ?? "";
const ext = shortenToBytes(
fullExt,
maxBytes - Buffer.byteLength(insert + first),
);
const room = maxBytes - Buffer.byteLength(insert + ext);
return shortenToBytes(stem, room) + insert + ext;
};
// The name a file is saved under in the working directory when the caller
// gives no path (`quak get` without `--out`, `downloadFile` without
// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
// unsafe characters replaced, or `file-<id>` when the title is empty, cut
// short so that the whole fits in a file name, keeping the title's extension
// whenever it fits. `ext` (".mov"), when given, takes the place of the
// title's extension before the name is cut short; a live photo's image and
// video are each named with their own. The cut can leave a device name
// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
// leave room for it.
export const defaultFileName = (
title: string,
fileID: number,
opts: { prefix?: string; ext?: string } = {},
): string => {
const { prefix = "", ext } = opts;
const safe = sanitizeFileName(title, `file-${fileID}`);
const name = ext === undefined ? safe : withExtension(safe, ext);
const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
const cut = prefix + shortenKeepingExtension(name, room);
if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
};
+14 -2
View File
@@ -5,7 +5,11 @@ import type { Client } from "./client.js";
import { describeError } from "./errors.js";
import { readExifTags } from "./exif.js";
import type { Library, Photo } from "./library/index.js";
import { sanitizeFileName } from "./filename.js";
import {
MAX_NAME_BYTES,
sanitizeFileName,
shortenToBytes,
} from "./filename.js";
import {
fetchMLDataBatch,
MLDATA_BATCH_SIZE,
@@ -127,7 +131,15 @@ export const runMetadataBackup = async (
const col = lib.getCollection(album.collectionID);
if (!col) continue;
const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
// The album's ID keeps the folders apart, so the name after it can be
// cut short to fit in a file name.
const prefix = `${col.id}-`;
const dirName =
prefix +
shortenToBytes(
sanitizeFileName(col.name, "unnamed"),
MAX_NAME_BYTES - prefix.length,
);
const colDir = join(outDir, "collections", dirName);
mkdirSync(colDir, { recursive: true });
+10 -11
View File
@@ -1795,12 +1795,11 @@ describe("backup album folders", () => {
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
});
// The extensions of these titles leave no room for the ID, so each is cut
// short and the name before the ID keeps only the bytes that cut leaves
// over. File 1's extension is 303 bytes and is cut to 251, which leaves
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three
// bytes, which leaves one, for the "a". Neither extension is only letters
// and digits, so both originals are saved with `.bin`, a name that fits.
// The extensions of these titles leave no room for the ID and the "a"
// before them, so each is cut short to 250 bytes, which leaves one, for
// the "a". File 1's extension is 303 bytes and file 2's 271. Neither
// extension is only letters and digits, so both originals are saved with
// `.bin`, a name that fits.
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
const outDir = join(root, "backup");
const lib = libraryOf([
@@ -1815,7 +1814,7 @@ describe("backup album folders", () => {
const result = await runBackup(lib, { downloadDirectory: outDir });
const link = ` (1).${"L".repeat(250)}`;
const link = `a (1).${"L".repeat(249)}`;
const link2 = `a (2).${"€".repeat(83)}`;
expect(Buffer.byteLength(link)).toBe(255);
expect(Buffer.byteLength(link2)).toBe(255);
@@ -1830,9 +1829,9 @@ describe("backup album folders", () => {
// Files 1 and 3 share a title, so each gets its ID before the
// extension. File 2's title is too long, and its extension leaves no
// room for the ID. Were the ID put at the end of file 2's title cut
// short, its link would have file 1's name, and the link made second
// would replace the first.
// room for the ID and the "x" before it. Were the ID put at the end of
// file 2's title cut short, its link would have file 1's name, and the
// link made second would replace the first.
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
const outDir = join(root, "backup");
const shared = `x.${"Q".repeat(245)} (2)`;
@@ -1852,8 +1851,8 @@ describe("backup album folders", () => {
expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([
"Trip/",
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
`Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`,
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
"Trip.json",
]);
+139 -6
View File
@@ -21,7 +21,7 @@ import {
utimesSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { basename, join } from "node:path";
import { tmpdir } from "node:os";
import { PassThrough } from "node:stream";
import * as jpegJs from "jpeg-js";
@@ -187,6 +187,9 @@ const FETCHED_BOTH_ALBUMS =
"Fetched the files of 1 of 2 collections\n" +
"Fetched the files of 2 of 2 collections\n";
// The same for an account with one album.
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
// Collects everything written to it.
class Output {
text = "";
@@ -562,17 +565,109 @@ describe("get and get-thumb", () => {
});
});
// A file system refuses a file name over 255 bytes. The account here is one
// album, whose name is 300 bytes, holding one file, 700, with the title given.
describe("names from an album name or a title over 255 bytes", () => {
const longNames = (title: string): Client =>
({
...fakeClient(),
collectionsSince: async () => ({
collections: [collection(7, "A".repeat(300))],
deleted: [],
cursor: 1,
}),
filesSince: async () => ({
files: [file(700, 7, title)],
deleted: [],
cursor: 1,
}),
}) as unknown as Client;
const longTitle = `${"t".repeat(296)}.jpg`;
// Run `command` in a new directory, where `get` and `get-thumb` without
// `--out` write, and return its exit code and the directory.
const inNewDirectory = async (
command: () => Promise<number>,
): Promise<{ code: number; dir: string }> => {
const dir = mkdtempSync(join(root, "cwd-"));
const previous = process.cwd();
process.chdir(dir);
try {
return { code: await command(), dir };
} finally {
process.chdir(previous);
}
};
it("get cuts the title short to 255 bytes, keeping its extension, and exits 0", async () => {
const { code, dir } = await inNewDirectory(() =>
getCommand(context(longNames(longTitle)), "700", {}),
);
const name = `${"t".repeat(251)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(dir)).toEqual([name]);
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
});
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
const { code, dir } = await inNewDirectory(() =>
getThumbCommand(context(longNames(longTitle)), "700", {}),
);
const name = `thumb_${"t".repeat(245)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(dir)).toEqual([name]);
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
});
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
const dir = join(root, "dump");
const code = await backupMetadataCommand(
context(longNames("beach.jpg")),
dir,
{},
);
const folder = `7-${"A".repeat(253)}`;
expect(Buffer.byteLength(folder)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(join(dir, "collections"))).toEqual([folder]);
expect(readdirSync(join(dir, "collections", folder)).sort()).toEqual([
"700.json",
"_collection.json",
]);
});
// The title's extension is 300 letters, too long for the save path's
// name, so the original is saved with `.bin`.
it("backup saves a title with a 300-letter extension as .bin, and exits 0", async () => {
const dir = join(root, "backup");
const title = `b.${"e".repeat(300)}`;
const code = await backupCommand(context(longNames(title)), dir, {});
const path = savePath(dir, file(700, 7, title));
expect(code).toBe(0);
expect(basename(path)).toMatch(/^\d{4}-\d{2}-\d{2}\.700\.bin$/);
expect(readFileSync(path)).toEqual(Buffer.alloc(7, 700 & 0xff));
});
});
// A live photo, which Ente stores as one ZIP, is written as its image and its
// video, which a photo viewer can open. The account here is Vacation holding
// one live photo, 300, downloaded through the real download layer
// (test/live-photo.ts).
describe("a live photo", () => {
// What `get` prints on stderr while its refresh fetches Vacation's files.
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
const livePhotoClient = async (image = IMAGE): Promise<Client> => {
const livePhotoClient = async (
image = IMAGE,
title = "IMG_0300.HEIC",
): Promise<Client> => {
const { file: live, body } = await asLivePhoto(
file(300, 1, "IMG_0300.HEIC"),
file(300, 1, title),
livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
livePhotoHash(image, VIDEO),
);
@@ -620,6 +715,44 @@ describe("a live photo", () => {
);
});
// Run `get` without `--out` in a new directory for a live photo titled
// `title`, and return its exit code and the names it wrote.
const getInNewDirectory = async (
title: string,
): Promise<{ code: number; names: string[] }> => {
const ctx = context(await livePhotoClient(IMAGE, title));
const dir = mkdtempSync(join(root, "cwd-"));
const previous = process.cwd();
process.chdir(dir);
try {
const code = await getCommand(ctx, "300", {});
return { code, names: readdirSync(dir).sort() };
} finally {
process.chdir(previous);
}
};
const imageName = `${"t".repeat(250)}.heic`;
const videoName = `${"t".repeat(251)}.mov`;
// The title fits, but ".heic" is one byte longer than ".jpg", so the
// image's name is cut short once its extension is in it.
it("get cuts its image's name short for a 255-byte title ending in .jpg, and exits 0", async () => {
const title = `${"t".repeat(251)}.jpg`;
const result = await getInNewDirectory(title);
expect(Buffer.byteLength(title)).toBe(255);
expect(Buffer.byteLength(imageName)).toBe(255);
expect(Buffer.byteLength(videoName)).toBe(255);
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
});
it("get cuts the names of its image and video short for a 300-byte title with no extension, and exits 0", async () => {
const result = await getInNewDirectory("t".repeat(300));
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
});
it("get --out writes the image there and the video beside it", async () => {
const out = join(root, "photo.jpg");
const video = join(root, "photo.mov");
+89
View File
@@ -658,6 +658,26 @@ describe("downloadFile", () => {
expect(readdirSync(parent)).toEqual(["cwd"]);
});
// A file system refuses a file name over 255 bytes.
it("cuts a title too long for a file name short, keeping its extension", async () => {
const plaintext = new Uint8Array([1, 2, 3]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const thumbPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const file = buildMockEnteFile(key, header, thumbPush.header);
file.metadata.title = `${"t".repeat(296)}.jpg`;
const dir = mkdtempSync(join(testDir, "long-title-"));
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
const result = await inDirectory(dir, () => downloadFile(api, file));
const name = `${"t".repeat(251)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(result.path).toBe(name);
expect(readdirSync(dir)).toEqual([name]);
});
it("uses an explicit outPath verbatim, even one with ..", async () => {
// The caller is trusted: its path is not sanitized.
const { api, file } = fixtureFor(
@@ -751,6 +771,28 @@ describe("downloadThumbnail", () => {
expect(result.path).toBe("thumb__etc_passwd");
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
});
it("leaves room for thumb_ when it cuts a title too long for a file name short", async () => {
const plaintext = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const filePush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const { header: thumbHeader, ciphertext: thumbCipher } =
encryptFileBody(plaintext, key);
const file = buildMockEnteFile(key, filePush.header, thumbHeader);
file.metadata.title = `${"t".repeat(296)}.jpg`;
const dir = mkdtempSync(join(testDir, "long-thumb-title-"));
const api = new ApiClient({ fetch: mockFetchForBody(thumbCipher) });
const result = await inDirectory(dir, () =>
downloadThumbnail(api, file),
);
const name = `thumb_${"t".repeat(245)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(result.path).toBe(name);
expect(readdirSync(dir)).toEqual([name]);
});
});
// ---------------------------------------------------------------------------
@@ -1881,6 +1923,53 @@ describe("downloadFile live photos", () => {
]);
});
// A file system refuses a file name over 255 bytes. Each name is cut
// short once its own extension is in it.
const imageName = `${"t".repeat(250)}.heic`;
const videoName = `${"t".repeat(251)}.mov`;
// The title fits, but ".heic" is one byte longer than ".jpg".
it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => {
const title = `${"t".repeat(251)}.jpg`;
const t = setup(livePhotoZip(), { ...livePhoto, title });
const result = await inDirectory(t.dir, () =>
downloadFile(t.api, t.file),
);
expect(Buffer.byteLength(title)).toBe(255);
expect(Buffer.byteLength(imageName)).toBe(255);
expect(Buffer.byteLength(videoName)).toBe(255);
expect(result.path).toBe(imageName);
expect(result.videoPath).toBe(videoName);
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
});
it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => {
const t = setup(livePhotoZip(), {
...livePhoto,
title: "t".repeat(300),
});
const result = await inDirectory(t.dir, () =>
downloadFile(t.api, t.file),
);
expect(result.path).toBe(imageName);
expect(result.videoPath).toBe(videoName);
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
});
// The caller's path is used as is; only the extension is each part's own.
it("keeps an outPath too long for a file name as given", async () => {
const t = setup(livePhotoZip(), livePhoto);
const outPath = join(t.dir, `${"t".repeat(251)}.jpg`);
await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow(
/ENAMETOOLONG/,
);
});
it("gives each part its own extension from the ZIP, letters and digits only", async () => {
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
const t = setup(zip, livePhoto);
+108
View File
@@ -15,8 +15,10 @@
import { describe, expect, it } from "vitest";
import {
defaultFileName,
safeExtension,
sanitizeFileName,
shortenKeepingExtension,
shortenToBytes,
} from "../../src/filename.js";
@@ -94,6 +96,12 @@ describe("safeExtension", () => {
expect(safeExtension("x.jp g")).toBe(".bin");
expect(safeExtension("x.jpg\0")).toBe(".bin");
});
it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => {
expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`);
expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin");
expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin");
});
});
describe("shortenToBytes", () => {
@@ -118,3 +126,103 @@ describe("shortenToBytes", () => {
);
});
});
describe("shortenKeepingExtension", () => {
it("returns a name that fits unchanged", () => {
expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg");
expect(shortenKeepingExtension("beach", 5)).toBe("beach");
});
it("cuts the part before the extension short and keeps the extension", () => {
expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(shortenKeepingExtension("é".repeat(150), 255)).toBe(
"é".repeat(127),
);
});
// A name cut down to its extension would start with a dot, and so be
// hidden.
it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
`b.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
`a.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
`é.${"e".repeat(252)}`,
);
// "€" takes three bytes, so the extension is cut to 253 and the
// part before it keeps the two bytes left over, for the "a".
expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe(
`a.${"€".repeat(84)}`,
);
});
it("puts `insert` right before the extension and leaves room for it", () => {
expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe(
"beach (7).jpg",
);
expect(
shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"),
).toBe(`${"t".repeat(247)} (7).jpg`);
expect(
shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
).toBe(`b (7).${"e".repeat(249)}`);
});
});
describe("defaultFileName", () => {
it("is the title when it is safe and fits", () => {
expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
});
it("replaces unsafe characters, and is file-<id> for an empty title", () => {
expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
expect(defaultFileName("", 42)).toBe("file-42");
});
it("cuts a title over 255 bytes short, keeping its extension", () => {
expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
`b.${"e".repeat(253)}`,
);
});
it("puts `prefix` before the title and leaves room for it", () => {
expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
"thumb_beach.jpg",
);
expect(
defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
).toBe(`thumb_${"t".repeat(245)}.jpg`);
});
// A live photo's image and video are each named with their own
// extension, and the name is cut short once that extension is in it.
it("puts `ext` in place of the title's extension before cutting the name short", () => {
expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
"IMG_0001.mov",
);
expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
expect(
defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
).toBe(`${"t".repeat(250)}.heic`);
expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
`${"t".repeat(250)}.heic`,
);
});
// Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
// or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
// name, and the part before the extension gives up a byte for it.
it("never returns a Windows device name, even once cut short", () => {
const ext = `.${"e".repeat(251)}`;
expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
});
});