Compare commits
2
Commits
35bd45a575
...
c34e05cfe0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c34e05cfe0 | ||
|
|
66e3ac77e8 |
No files matched your search
@@ -35,8 +35,11 @@ yarn quak login
|
|||||||
# List the user's collections (albums).
|
# List the user's collections (albums).
|
||||||
yarn quak collections
|
yarn quak collections
|
||||||
|
|
||||||
# List files in a collection.
|
# List every file in the account.
|
||||||
yarn quak files --collection 12345
|
yarn quak files
|
||||||
|
|
||||||
|
# List the files in a collection.
|
||||||
|
yarn quak files 12345
|
||||||
|
|
||||||
# Download and decrypt a single file.
|
# Download and decrypt a single file.
|
||||||
yarn quak get 67890 --out ./photo.jpg
|
yarn quak get 67890 --out ./photo.jpg
|
||||||
@@ -647,7 +650,7 @@ quak login interactive or QUAK_EM
|
|||||||
quak whoami print logged-in account as JSON
|
quak whoami print logged-in account as JSON
|
||||||
quak logout end the session, delete it
|
quak logout end the session, delete it
|
||||||
quak collections [--json] list all collections
|
quak collections [--json] list all collections
|
||||||
quak files --collection <id> [--json] list files in a collection
|
quak files [collectionID] [--json] list files in a collection, or every file
|
||||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||||
quak backup <dir> [--json] [--verify] full incremental backup
|
quak backup <dir> [--json] [--verify] full incremental backup
|
||||||
@@ -676,34 +679,53 @@ answers from what did decrypt (see "Default reads vs. fresh reads").
|
|||||||
`--cache-dir` overrides where the cache lives; without it each account gets its
|
`--cache-dir` overrides where the cache lives; without it each account gets its
|
||||||
own directory under the per-user cache path.
|
own directory under the per-user cache path.
|
||||||
|
|
||||||
An ID given to `files --collection`, `get`, `get-thumb` or
|
`files <collectionID>` lists the files in that collection. `files` with no
|
||||||
|
argument lists every file in the account, collection by collection, each file
|
||||||
|
once: a file in several collections is listed with the first of them, which is
|
||||||
|
its `collectionID` in `--json`.
|
||||||
|
|
||||||
|
An ID given to `files`, `get`, `get-thumb` or
|
||||||
`helper fix-missing-thumbnails --file` is written in decimal digits, no larger
|
`helper fix-missing-thumbnails --file` is written in decimal digits, no larger
|
||||||
than 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused
|
than 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused
|
||||||
before any request with one line, `Invalid file ID: <arg>` or
|
before any request with one line, `Invalid file ID: <arg>` or
|
||||||
`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses
|
`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses
|
||||||
the whole command, so nothing is fixed.
|
the whole command, so nothing is fixed.
|
||||||
|
|
||||||
|
Without `--out`, `get` writes the file to the working directory under its title,
|
||||||
|
and `get-thumb` writes the thumbnail there under `thumb_` followed by the title,
|
||||||
|
each with unsafe characters replaced. A name over 255 bytes, which most file
|
||||||
|
systems refuse, is cut short between two characters before its extension, so the
|
||||||
|
extension is kept whenever it fits. An extension too long to leave room for the
|
||||||
|
name's first character is cut short too, so the name never starts with a dot.
|
||||||
|
|
||||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||||
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
||||||
its image and its video, each named after the title with its own extension, as
|
its image and its video, each named after the title with its own extension, as
|
||||||
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
|
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is
|
||||||
`--out PATH`, the image is written to `PATH` and the video beside it, with
|
cut short as above once its own extension is in it. With `--out PATH`, the image
|
||||||
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
|
is written to `PATH` and the video beside it, with `PATH`'s name and the video's
|
||||||
refused. `backup-metadata --exif` (alias `--all`) additionally fetches each
|
extension; a `PATH` with the video's extension is refused.
|
||||||
file's original through the cache and records, from it or a live photo's image,
|
`backup-metadata --exif` (alias `--all`) additionally fetches each file's
|
||||||
its XMP metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is
|
original through the cache and records, from it or a live photo's image, its XMP
|
||||||
read with [exifreader](https://github.com/mattiasw/ExifReader) from any image
|
metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with
|
||||||
format it reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's
|
[exifreader](https://github.com/mattiasw/ExifReader) from any image format it
|
||||||
`exif` field is exifreader's EXIF tag output: each tag by name, with its
|
reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field
|
||||||
`value`, `description` and `computed` value. An EXIF block exifreader finds but
|
is exifreader's EXIF tag output: each tag by name, with its `value`,
|
||||||
reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
`description` and `computed` value. An EXIF block exifreader finds but reads no
|
||||||
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
|
tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`.
|
||||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
`collections`, `files`, `backup`, `helper list-missing-thumbnails` and
|
||||||
|
`helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||||
|
|
||||||
`backup --verify` also hashes the originals already in the backup and replaces
|
`backup --verify` also hashes the originals already in the backup and replaces
|
||||||
any that do not match the content hash Ente records; one it cannot replace stays
|
any that do not match the content hash Ente records; one it cannot replace stays
|
||||||
as it was and goes into `failures.json` (see "Backup layout").
|
as it was and goes into `failures.json` (see "Backup layout").
|
||||||
|
|
||||||
|
`backup-metadata <dir>` writes `account.json` and, for each collection, a
|
||||||
|
directory `collections/<id>-<name>/` holding `_collection.json` and a
|
||||||
|
`<fileID>.json` for each of its files. The collection's name has unsafe
|
||||||
|
characters replaced and is cut short between two characters so that the
|
||||||
|
directory's name fits in 255 bytes; the ID keeps the directories apart.
|
||||||
|
|
||||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||||
fails, the error is logged, each of its files is written with the reason in an
|
fails, the error is logged, each of its files is written with the reason in an
|
||||||
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
|
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
|
||||||
@@ -756,9 +778,10 @@ Each original is saved at its save path, the same path `photo.savePath` gives
|
|||||||
and `photo.download()` writes (see Read surface below). The date is the photo's
|
and `photo.download()` writes (see Read surface below). The date is the photo's
|
||||||
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
|
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
|
||||||
time zone of the machine running quak. The extension is the one in the file's
|
time zone of the machine running quak. The extension is the one in the file's
|
||||||
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
|
name as uploaded, case kept, or `.bin` when it has none, it holds anything but
|
||||||
letters and digits. When the date or the time zone changes, the next run saves
|
letters and digits, or it is more than 16 of them long. When the date or the
|
||||||
the original at its new path and leaves the old copy where it is.
|
time zone changes, the next run saves the original at its new path and leaves
|
||||||
|
the old copy where it is.
|
||||||
|
|
||||||
`account.json` holds the account's `email` and `userID`, as `backup-metadata`
|
`account.json` holds the account's `email` and `userID`, as `backup-metadata`
|
||||||
writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`,
|
writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`,
|
||||||
@@ -851,11 +874,12 @@ gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album
|
|||||||
named `x.json` gets the directory `x.json (12)/`. A name too long for a file
|
named `x.json` gets the directory `x.json (12)/`. A name too long for a file
|
||||||
name, which most file systems limit to 255 bytes, gets its ID added too. A name
|
name, which most file systems limit to 255 bytes, gets its ID added too. A name
|
||||||
with its ID added is cut short between two characters in the part before the ID,
|
with its ID added is cut short between two characters in the part before the ID,
|
||||||
just far enough that it fits. When a symlink's extension alone leaves no room
|
just far enough that it fits. When a symlink's extension leaves no room for the
|
||||||
for the ID, the extension is cut short and the part before the ID keeps only the
|
ID and the first character of the name, the extension is cut short, and the part
|
||||||
bytes that cut leaves over. A collection's directory name also leaves room for
|
before the ID keeps that character and any bytes the cut leaves over. A
|
||||||
the `.json` its JSON adds. IDs never change, so a name stays the same from run
|
collection's directory name also leaves room for the `.json` its JSON adds. IDs
|
||||||
to run until such a clash appears or goes away.
|
never change, so a name stays the same from run to run until such a clash
|
||||||
|
appears or goes away.
|
||||||
|
|
||||||
A collection whose directory or JSON cannot be written, such as one whose
|
A collection whose directory or JSON cannot be written, such as one whose
|
||||||
directory name is taken by a file you put in `collections/`, does not stop the
|
directory name is taken by a file you put in `collections/`, does not stop the
|
||||||
|
|||||||
@@ -17,15 +17,38 @@ pre-1.0
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
One bug is open: https://git.eeqj.de/sneak/quak/issues/205. The check of quak
|
No bug is open. One change sneak asked for is open:
|
||||||
against a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an
|
https://git.eeqj.de/sneak/quak/issues/212. The check of quak against a live Ente
|
||||||
account only sneak has, so it waits on him.
|
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
||||||
|
has, so it waits on him.
|
||||||
|
|
||||||
Tagging and releases are decided by sneak alone, and happen only when he
|
Tagging and releases are decided by sneak alone, and happen only when he
|
||||||
declares one.
|
declares one.
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-18: `quak files` takes the collection ID as an optional argument
|
||||||
|
instead of the `--collection` option, which is gone
|
||||||
|
(https://git.eeqj.de/sneak/quak/issues/211). With no argument it lists every
|
||||||
|
file in the account once, collection by collection; a file in several
|
||||||
|
collections is listed with the first of them, which is its `collectionID` in
|
||||||
|
`--json`. The fresh read in `src/cli-read.ts` does this.
|
||||||
|
|
||||||
|
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
|
||||||
|
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
|
||||||
|
`src/filename.ts` cuts a name short to a number of bytes, keeping its
|
||||||
|
extension when it fits and always its first character, and can put text right
|
||||||
|
before the extension; the backup's `withID` uses it for an ID.
|
||||||
|
`defaultFileName` there gives the name a file is saved under when the caller
|
||||||
|
gives no path: a prefix such as `thumb_` and the title with unsafe characters
|
||||||
|
replaced, cut short with it to 255 bytes, and, for a live photo's image or
|
||||||
|
video, with that part's own extension before the cut. `quak get`,
|
||||||
|
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
|
||||||
|
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
|
||||||
|
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
|
||||||
|
extension of more than 16 letters and digits, so a save path's name always
|
||||||
|
fits.
|
||||||
|
|
||||||
- 2026-10-18: `quak collections` no longer waits for the files of every
|
- 2026-10-18: `quak collections` no longer waits for the files of every
|
||||||
collection (issue 210). On an empty cache it opened its library with a refresh
|
collection (issue 210). On an empty cache it opened its library with a refresh
|
||||||
that fetched and decrypted the files of each collection in turn, printing
|
that fetched and decrypted the files of each collection in turn, printing
|
||||||
|
|||||||
+7
-6
@@ -78,14 +78,15 @@ program
|
|||||||
|
|
||||||
program
|
program
|
||||||
.command("files")
|
.command("files")
|
||||||
.description("List files in a collection")
|
.description("List the files in a collection, or every file in the account")
|
||||||
.requiredOption(
|
.argument(
|
||||||
"--collection <id>",
|
"[collectionID]",
|
||||||
"Collection ID (from `quak collections`)",
|
"Collection ID (from `quak collections`); without it, every file " +
|
||||||
|
"is listed once",
|
||||||
)
|
)
|
||||||
.option("--json", "Output as JSON array")
|
.option("--json", "Output as JSON array")
|
||||||
.action((opts: { collection: string; json?: boolean }) =>
|
.action((collectionID: string | undefined, opts: { json?: boolean }) =>
|
||||||
run(filesCommand(context(), opts)),
|
run(filesCommand(context(), collectionID, opts)),
|
||||||
);
|
);
|
||||||
|
|
||||||
program
|
program
|
||||||
|
|||||||
+8
-8
@@ -82,6 +82,7 @@ import { describeError } from "./errors.js";
|
|||||||
import {
|
import {
|
||||||
MAX_NAME_BYTES,
|
MAX_NAME_BYTES,
|
||||||
sanitizeFileName,
|
sanitizeFileName,
|
||||||
|
shortenKeepingExtension,
|
||||||
shortenToBytes,
|
shortenToBytes,
|
||||||
withExtension,
|
withExtension,
|
||||||
} from "./filename.js";
|
} from "./filename.js";
|
||||||
@@ -289,9 +290,10 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
|
|||||||
// case (two names that differ only in case are one entry on a case-insensitive
|
// case (two names that differ only in case are one entry on a case-insensitive
|
||||||
// file system), or it is too long for a file name; then it gets ` (<id>)`,
|
// file system), or it is too long for a file name; then it gets ` (<id>)`,
|
||||||
// right before the extension when `beforeExtension` is set. To fit, the part
|
// right before the extension when `beforeExtension` is set. To fit, the part
|
||||||
// before the ID is cut short as far as it must be. When the extension alone
|
// before the ID is cut short as far as it must be. When the extension leaves
|
||||||
// leaves no room for the ID, the extension is cut short and the part before
|
// no room for the ID and the first character of the name, the extension is
|
||||||
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each
|
// cut short, and the part before the ID keeps that character and any bytes
|
||||||
|
// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each
|
||||||
// entry also has the JSON file `<name>.json` beside it, as an album does: a
|
// entry also has the JSON file `<name>.json` beside it, as an album does: a
|
||||||
// name then leaves room for `.json`, and one that is another entry's JSON name
|
// name then leaves room for `.json`, and one that is another entry's JSON name
|
||||||
// gets its ID added too.
|
// gets its ID added too.
|
||||||
@@ -305,11 +307,9 @@ const uniqueNames = (
|
|||||||
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
|
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
|
||||||
const withID = (id: number, name: string): string => {
|
const withID = (id: number, name: string): string => {
|
||||||
const tag = ` (${id})`;
|
const tag = ` (${id})`;
|
||||||
const fullExt = opts.beforeExtension ? extname(name) : "";
|
return opts.beforeExtension
|
||||||
const stem = name.slice(0, name.length - fullExt.length);
|
? shortenKeepingExtension(name, maxBytes, tag)
|
||||||
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag));
|
: shortenToBytes(name, maxBytes - Buffer.byteLength(tag)) + tag;
|
||||||
const room = maxBytes - Buffer.byteLength(tag + ext);
|
|
||||||
return shortenToBytes(stem, room) + tag + ext;
|
|
||||||
};
|
};
|
||||||
const names = entries.map((e) => e.name);
|
const names = entries.map((e) => e.name);
|
||||||
const suffixed = new Set<number>();
|
const suffixed = new Set<number>();
|
||||||
|
|||||||
+17
-7
@@ -269,15 +269,21 @@ export const collectionsCommand = async (
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// Lists the files of the collection `collectionIDStr` names, or without it every
|
||||||
|
// file in the account, each once (issue #211).
|
||||||
export const filesCommand = async (
|
export const filesCommand = async (
|
||||||
ctx: CliContext,
|
ctx: CliContext,
|
||||||
opts: { collection: string; json?: boolean },
|
collectionIDStr: string | undefined,
|
||||||
|
opts: { json?: boolean },
|
||||||
): Promise<number> => {
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
const client = requireSession(ctx);
|
const client = requireSession(ctx);
|
||||||
if (!client) return 3;
|
if (!client) return 3;
|
||||||
const collectionID = parseID(ctx, "collection", opts.collection);
|
const collectionID =
|
||||||
if (collectionID === undefined) return 1;
|
collectionIDStr === undefined
|
||||||
|
? undefined
|
||||||
|
: parseID(ctx, "collection", collectionIDStr);
|
||||||
|
if (collectionIDStr !== undefined && collectionID === undefined) return 1;
|
||||||
|
|
||||||
const lib = await openReadLibrary(ctx, client, opts);
|
const lib = await openReadLibrary(ctx, client, opts);
|
||||||
try {
|
try {
|
||||||
@@ -332,8 +338,8 @@ export const getCommand = async (
|
|||||||
const result = await photo.original();
|
const result = await photo.original();
|
||||||
// Default name is the file's own title, as the pre-library CLI used
|
// Default name is the file's own title, as the pre-library CLI used
|
||||||
// (not the editedName-preferring projection title) (issue #52).
|
// (not the editedName-preferring projection title) (issue #52).
|
||||||
const outPath = opts.out ?? originalName(file);
|
|
||||||
if (result.videoPath === undefined) {
|
if (result.videoPath === undefined) {
|
||||||
|
const outPath = opts.out ?? originalName(file);
|
||||||
copyFileSync(result.path, outPath);
|
copyFileSync(result.path, outPath);
|
||||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||||
return 0;
|
return 0;
|
||||||
@@ -341,9 +347,13 @@ export const getCommand = async (
|
|||||||
// A live photo is written as its image and its video, each named after
|
// A live photo is written as its image and its video, each named after
|
||||||
// the title with its own extension, as Ente's clients name them. With
|
// the title with its own extension, as Ente's clients name them. With
|
||||||
// --out, the image goes there and the video beside it.
|
// --out, the image goes there and the video beside it.
|
||||||
const imageOut =
|
const imageExt = extname(result.path);
|
||||||
opts.out ?? withExtension(outPath, extname(result.path));
|
const videoExt = extname(result.videoPath);
|
||||||
const videoOut = withExtension(outPath, extname(result.videoPath));
|
const imageOut = opts.out ?? originalName(file, imageExt);
|
||||||
|
const videoOut =
|
||||||
|
opts.out === undefined
|
||||||
|
? originalName(file, videoExt)
|
||||||
|
: withExtension(opts.out, videoExt);
|
||||||
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
|
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
|
||||||
ctx.stderr.write(
|
ctx.stderr.write(
|
||||||
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
|
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
|
||||||
|
|||||||
+9
-7
@@ -9,7 +9,7 @@
|
|||||||
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
|
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
|
||||||
// the commands shape their output from the raw `EnteFile` through here.
|
// the commands shape their output from the raw `EnteFile` through here.
|
||||||
|
|
||||||
import { sanitizeFileName } from "./filename.js";
|
import { defaultFileName } from "./filename.js";
|
||||||
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
|
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
|
||||||
|
|
||||||
// One row of `quak files --json`.
|
// One row of `quak files --json`.
|
||||||
@@ -33,11 +33,13 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
|
|||||||
export const fileListLine = (file: EnteFile): string =>
|
export const fileListLine = (file: EnteFile): string =>
|
||||||
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
|
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
|
||||||
|
|
||||||
// Default output path for `quak get` when `--out` is not given. The title comes
|
// Default output path for `quak get` when `--out` is not given, made from the
|
||||||
// from the server, so it is sanitized; `--out` is the user's and is used as is.
|
// title (see `defaultFileName`), with `ext` in place of its extension for a
|
||||||
export const originalName = (file: EnteFile): string =>
|
// live photo's image or video; `--out` is the user's and is used as is.
|
||||||
sanitizeFileName(file.metadata.title, `file-${file.id}`);
|
export const originalName = (file: EnteFile, ext?: string): string =>
|
||||||
|
defaultFileName(file.metadata.title, file.id, { ext });
|
||||||
|
|
||||||
// Default output path for `quak get-thumb` when `--out` is not given.
|
// Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
|
||||||
|
// and the title.
|
||||||
export const thumbnailName = (file: EnteFile): string =>
|
export const thumbnailName = (file: EnteFile): string =>
|
||||||
`thumb_${originalName(file)}`;
|
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
|
||||||
+25
-6
@@ -1,6 +1,6 @@
|
|||||||
// How the CLI's read commands obtain current data.
|
// How the CLI's read commands obtain current data.
|
||||||
//
|
//
|
||||||
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
|
// `collections`, `files`, `get`, and `get-thumb` must answer for
|
||||||
// the account's state at the moment the command runs, not for whatever the
|
// the account's state at the moment the command runs, not for whatever the
|
||||||
// local cache last happened to hold (owner amendment, issue #36). `files`,
|
// local cache last happened to hold (owner amendment, issue #36). `files`,
|
||||||
// `get` and `get-thumb` therefore force a server round-trip through
|
// `get` and `get-thumb` therefore force a server round-trip through
|
||||||
@@ -14,7 +14,10 @@
|
|||||||
// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
|
// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
|
||||||
// rather than the `albums`/`photos` projection's newest-first order, which
|
// rather than the `albums`/`photos` projection's newest-first order, which
|
||||||
// re-sorts the rows. The field values still come from each record's raw
|
// re-sorts the rows. The field values still come from each record's raw
|
||||||
// metadata via `cli-output.ts`.
|
// metadata via `cli-output.ts`. `files` without a collection ID lists every
|
||||||
|
// file in the account once (issue #211): the library keeps a record of a file
|
||||||
|
// for each collection that holds it, and only the first collection's record is
|
||||||
|
// listed.
|
||||||
|
|
||||||
import type { Collection, EnteFile } from "./model/types.js";
|
import type { Collection, EnteFile } from "./model/types.js";
|
||||||
import type { Photo, PhotosAPI } from "./library/index.js";
|
import type { Photo, PhotosAPI } from "./library/index.js";
|
||||||
@@ -27,6 +30,7 @@ export interface FreshReadLibrary {
|
|||||||
fresh(): Promise<unknown>;
|
fresh(): Promise<unknown>;
|
||||||
freshCollections(): Promise<Collection[]>;
|
freshCollections(): Promise<Collection[]>;
|
||||||
getCollection(id: number): Collection | undefined;
|
getCollection(id: number): Collection | undefined;
|
||||||
|
listCollections(): Collection[];
|
||||||
listFiles(collectionID: number): EnteFile[];
|
listFiles(collectionID: number): EnteFile[];
|
||||||
getFileByID(fileID: number): EnteFile | undefined;
|
getFileByID(fileID: number): EnteFile | undefined;
|
||||||
photos: Pick<PhotosAPI, "byID">;
|
photos: Pick<PhotosAPI, "byID">;
|
||||||
@@ -40,14 +44,29 @@ export const freshCollections = (
|
|||||||
|
|
||||||
// The files of one collection, current as of a forced refresh, in enumeration
|
// The files of one collection, current as of a forced refresh, in enumeration
|
||||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||||
// the caller can tell "no such collection" from "an empty collection".
|
// the caller can tell "no such collection" from "an empty collection". Without
|
||||||
|
// a collection ID, every file in the account, each once, as held by the first
|
||||||
|
// collection that holds it.
|
||||||
export const freshFiles = async (
|
export const freshFiles = async (
|
||||||
lib: FreshReadLibrary,
|
lib: FreshReadLibrary,
|
||||||
collectionID: number,
|
collectionID?: number,
|
||||||
): Promise<EnteFile[] | undefined> => {
|
): Promise<EnteFile[] | undefined> => {
|
||||||
await lib.fresh();
|
await lib.fresh();
|
||||||
if (!lib.getCollection(collectionID)) return undefined;
|
if (collectionID !== undefined) {
|
||||||
return lib.listFiles(collectionID);
|
return lib.getCollection(collectionID)
|
||||||
|
? lib.listFiles(collectionID)
|
||||||
|
: undefined;
|
||||||
|
}
|
||||||
|
const listed = new Set<number>();
|
||||||
|
const files: EnteFile[] = [];
|
||||||
|
for (const collection of lib.listCollections()) {
|
||||||
|
for (const file of lib.listFiles(collection.id)) {
|
||||||
|
if (listed.has(file.id)) continue;
|
||||||
|
listed.add(file.id);
|
||||||
|
files.push(file);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return files;
|
||||||
};
|
};
|
||||||
|
|
||||||
// One file, current as of a forced refresh, resolved to both its content
|
// One file, current as of a forced refresh, resolved to both its content
|
||||||
|
|||||||
+28
-11
@@ -16,7 +16,7 @@ import {
|
|||||||
streamTagFinal,
|
streamTagFinal,
|
||||||
} from "../crypto/index.js";
|
} from "../crypto/index.js";
|
||||||
import { TruncatedStreamError } from "../errors.js";
|
import { TruncatedStreamError } from "../errors.js";
|
||||||
import { safeExtension, sanitizeFileName, withExtension } from "../filename.js";
|
import { defaultFileName, safeExtension, withExtension } from "../filename.js";
|
||||||
import { withRetry } from "../retry.js";
|
import { withRetry } from "../retry.js";
|
||||||
import type { ApiClient } from "../api/client.js";
|
import type { ApiClient } from "../api/client.js";
|
||||||
import type { EnteFile } from "../model/types.js";
|
import type { EnteFile } from "../model/types.js";
|
||||||
@@ -365,10 +365,14 @@ const openPart = async (
|
|||||||
// name starts with `image` as the image and the one whose name starts with
|
// name starts with `image` as the image and the one whose name starts with
|
||||||
// `video` as the video, and refuses a ZIP holding a second of either. It is
|
// `video` as the video, and refuses a ZIP holding a second of either. It is
|
||||||
// written unpacked: each part is named `destination` with the extension
|
// written unpacked: each part is named `destination` with the extension
|
||||||
// replaced by its own entry's, and the two must differ ignoring case. When the
|
// replaced by its own entry's, and the two must differ ignoring case. When
|
||||||
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
// `defaultName` is set, `destination` is the name `defaultFileName` made from
|
||||||
// part's own bytes. Only then are the image, then the video, renamed into
|
// the title rather than a path the caller gave, and each part is named as
|
||||||
// place; on any failure neither is stored.
|
// `defaultFileName` names it with its own extension, so that the name is cut
|
||||||
|
// short to fit with that extension. When the file records a hash,
|
||||||
|
// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
|
||||||
|
// Only then are the image, then the video, renamed into place; on any failure
|
||||||
|
// neither is stored.
|
||||||
//
|
//
|
||||||
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
||||||
// written as it decompresses and never held, and the ZIP is refused once the
|
// written as it decompresses and never held, and the ZIP is refused once the
|
||||||
@@ -388,6 +392,7 @@ const decryptLivePhoto = async (
|
|||||||
key: Uint8Array,
|
key: Uint8Array,
|
||||||
onProgress: ProgressCallback | undefined,
|
onProgress: ProgressCallback | undefined,
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
|
defaultName: boolean,
|
||||||
): Promise<DownloadResult> => {
|
): Promise<DownloadResult> => {
|
||||||
const sliceSize = 4096;
|
const sliceSize = 4096;
|
||||||
const dir = dirname(destination);
|
const dir = dirname(destination);
|
||||||
@@ -486,8 +491,12 @@ const decryptLivePhoto = async (
|
|||||||
`live photo's image and video have the same extension, ${video.ext}`,
|
`live photo's image and video have the same extension, ${video.ext}`,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
const path = withExtension(destination, image.ext);
|
const partPath = (ext: string): string =>
|
||||||
const videoPath = withExtension(destination, video.ext);
|
defaultName
|
||||||
|
? defaultFileName(file.metadata.title, file.id, { ext })
|
||||||
|
: withExtension(destination, ext);
|
||||||
|
const path = partPath(image.ext);
|
||||||
|
const videoPath = partPath(video.ext);
|
||||||
for (const part of parts) {
|
for (const part of parts) {
|
||||||
await part.handle.sync();
|
await part.handle.sync();
|
||||||
await part.handle.close();
|
await part.handle.close();
|
||||||
@@ -531,6 +540,9 @@ const decryptLivePhoto = async (
|
|||||||
// and then removes its own temp file, and only the attempt that reaches
|
// and then removes its own temp file, and only the attempt that reaches
|
||||||
// TAG_FINAL renames one into place, so a download that needed three tries still
|
// TAG_FINAL renames one into place, so a download that needed three tries still
|
||||||
// performs exactly one rename over the destination.
|
// performs exactly one rename over the destination.
|
||||||
|
//
|
||||||
|
// `defaultName` says `destination` is the name `defaultFileName` made, not a
|
||||||
|
// path the caller gave (see `decryptLivePhoto`).
|
||||||
const fetchAndDecrypt = async (
|
const fetchAndDecrypt = async (
|
||||||
api: ApiClient,
|
api: ApiClient,
|
||||||
openStream: () => Promise<ReadableStream<Uint8Array>>,
|
openStream: () => Promise<ReadableStream<Uint8Array>>,
|
||||||
@@ -539,6 +551,7 @@ const fetchAndDecrypt = async (
|
|||||||
destination: string,
|
destination: string,
|
||||||
onProgress?: ProgressCallback,
|
onProgress?: ProgressCallback,
|
||||||
original?: EnteFile,
|
original?: EnteFile,
|
||||||
|
defaultName = false,
|
||||||
): Promise<DownloadResult> =>
|
): Promise<DownloadResult> =>
|
||||||
withRetry(async () => {
|
withRetry(async () => {
|
||||||
const stream = await openStream();
|
const stream = await openStream();
|
||||||
@@ -551,6 +564,7 @@ const fetchAndDecrypt = async (
|
|||||||
key,
|
key,
|
||||||
onProgress,
|
onProgress,
|
||||||
original,
|
original,
|
||||||
|
defaultName,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
const bytesWritten = await decryptToTemp(
|
const bytesWritten = await decryptToTemp(
|
||||||
@@ -582,10 +596,11 @@ export const downloadFile = async (
|
|||||||
outPath?: string,
|
outPath?: string,
|
||||||
onProgress?: ProgressCallback,
|
onProgress?: ProgressCallback,
|
||||||
): Promise<DownloadResult> => {
|
): Promise<DownloadResult> => {
|
||||||
// `outPath` is the caller's and is used as is; the title is the server's
|
// `outPath` is the caller's and is used as is. Without it, the file is
|
||||||
// and is sanitized so it can only name a file in the current directory.
|
// written in the current directory under the name `defaultFileName` makes
|
||||||
|
// from the title, which is the server's.
|
||||||
const resolvedPath =
|
const resolvedPath =
|
||||||
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
|
outPath ?? defaultFileName(file.metadata.title, file.id);
|
||||||
const header = fromBase64(file.file.decryptionHeader);
|
const header = fromBase64(file.file.decryptionHeader);
|
||||||
return fetchAndDecrypt(
|
return fetchAndDecrypt(
|
||||||
api,
|
api,
|
||||||
@@ -595,6 +610,7 @@ export const downloadFile = async (
|
|||||||
resolvedPath,
|
resolvedPath,
|
||||||
onProgress,
|
onProgress,
|
||||||
file,
|
file,
|
||||||
|
outPath === undefined,
|
||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -604,9 +620,10 @@ export const downloadThumbnail = async (
|
|||||||
outPath?: string,
|
outPath?: string,
|
||||||
onProgress?: ProgressCallback,
|
onProgress?: ProgressCallback,
|
||||||
): Promise<DownloadResult> => {
|
): Promise<DownloadResult> => {
|
||||||
|
// As in `downloadFile`, with `thumb_` before the title.
|
||||||
const resolvedPath =
|
const resolvedPath =
|
||||||
outPath ??
|
outPath ??
|
||||||
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
|
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
|
||||||
const header = fromBase64(file.thumbnail.decryptionHeader);
|
const header = fromBase64(file.thumbnail.decryptionHeader);
|
||||||
return fetchAndDecrypt(
|
return fetchAndDecrypt(
|
||||||
api,
|
api,
|
||||||
|
|||||||
+56
-4
@@ -5,7 +5,9 @@
|
|||||||
// used on disk goes through here, so it can only ever name one file inside the
|
// used on disk goes through here, so it can only ever name one file inside the
|
||||||
// directory the caller chose: never a path, never `..`, never hidden, never a
|
// directory the caller chose: never a path, never `..`, never hidden, never a
|
||||||
// Windows device name. Such a name can still be too long for a file name, and
|
// Windows device name. Such a name can still be too long for a file name, and
|
||||||
// `shortenToBytes` cuts one short.
|
// `shortenToBytes` or `shortenKeepingExtension` cuts one short.
|
||||||
|
// `defaultFileName` does both for the name a file is saved under when the
|
||||||
|
// caller gives no path.
|
||||||
//
|
//
|
||||||
// A path the user typed (`--out`, `outPath`) is not passed through here: the
|
// A path the user typed (`--out`, `outPath`) is not passed through here: the
|
||||||
// caller is trusted, the server is not.
|
// caller is trusted, the server is not.
|
||||||
@@ -30,11 +32,11 @@ export const sanitizeFileName = (name: string, fallback: string): string => {
|
|||||||
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
|
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
|
||||||
};
|
};
|
||||||
|
|
||||||
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
|
// The extension of `title` (".jpg"), or ".bin" when it has none, it holds
|
||||||
// anything but letters and digits.
|
// anything but letters and digits, or it is more than 16 of them long.
|
||||||
export const safeExtension = (title: string): string => {
|
export const safeExtension = (title: string): string => {
|
||||||
const ext = extname(title);
|
const ext = extname(title);
|
||||||
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
|
return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin";
|
||||||
};
|
};
|
||||||
|
|
||||||
// `name` with its extension, if it has one, replaced by `ext` (".mov").
|
// `name` with its extension, if it has one, replaced by `ext` (".mov").
|
||||||
@@ -57,3 +59,53 @@ export const shortenToBytes = (name: string, maxBytes: number): string => {
|
|||||||
}
|
}
|
||||||
return name.slice(0, end);
|
return name.slice(0, end);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
|
||||||
|
// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
|
||||||
|
// makes it. The part before the extension is cut first, so the extension is
|
||||||
|
// kept whenever it fits, but that part always keeps its first character:
|
||||||
|
// when the extension leaves no room for `insert` and that character, the
|
||||||
|
// extension is cut short too, and the part before it keeps the bytes that cut
|
||||||
|
// leaves over. So a name that does not start with a dot never comes back
|
||||||
|
// starting with one, which would hide it. `insert` is never cut. With no
|
||||||
|
// `insert`, a name that fits comes back unchanged.
|
||||||
|
export const shortenKeepingExtension = (
|
||||||
|
name: string,
|
||||||
|
maxBytes: number,
|
||||||
|
insert = "",
|
||||||
|
): string => {
|
||||||
|
const fullExt = extname(name);
|
||||||
|
const stem = name.slice(0, name.length - fullExt.length);
|
||||||
|
const first = Array.from(stem)[0] ?? "";
|
||||||
|
const ext = shortenToBytes(
|
||||||
|
fullExt,
|
||||||
|
maxBytes - Buffer.byteLength(insert + first),
|
||||||
|
);
|
||||||
|
const room = maxBytes - Buffer.byteLength(insert + ext);
|
||||||
|
return shortenToBytes(stem, room) + insert + ext;
|
||||||
|
};
|
||||||
|
|
||||||
|
// The name a file is saved under in the working directory when the caller
|
||||||
|
// gives no path (`quak get` without `--out`, `downloadFile` without
|
||||||
|
// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
|
||||||
|
// unsafe characters replaced, or `file-<id>` when the title is empty, cut
|
||||||
|
// short so that the whole fits in a file name, keeping the title's extension
|
||||||
|
// whenever it fits. `ext` (".mov"), when given, takes the place of the
|
||||||
|
// title's extension before the name is cut short; a live photo's image and
|
||||||
|
// video are each named with their own. The cut can leave a device name
|
||||||
|
// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
|
||||||
|
// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
|
||||||
|
// leave room for it.
|
||||||
|
export const defaultFileName = (
|
||||||
|
title: string,
|
||||||
|
fileID: number,
|
||||||
|
opts: { prefix?: string; ext?: string } = {},
|
||||||
|
): string => {
|
||||||
|
const { prefix = "", ext } = opts;
|
||||||
|
const safe = sanitizeFileName(title, `file-${fileID}`);
|
||||||
|
const name = ext === undefined ? safe : withExtension(safe, ext);
|
||||||
|
const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
|
||||||
|
const cut = prefix + shortenKeepingExtension(name, room);
|
||||||
|
if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
|
||||||
|
return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
|
||||||
|
};
|
||||||
+14
-2
@@ -5,7 +5,11 @@ import type { Client } from "./client.js";
|
|||||||
import { describeError } from "./errors.js";
|
import { describeError } from "./errors.js";
|
||||||
import { readExifTags } from "./exif.js";
|
import { readExifTags } from "./exif.js";
|
||||||
import type { Library, Photo } from "./library/index.js";
|
import type { Library, Photo } from "./library/index.js";
|
||||||
import { sanitizeFileName } from "./filename.js";
|
import {
|
||||||
|
MAX_NAME_BYTES,
|
||||||
|
sanitizeFileName,
|
||||||
|
shortenToBytes,
|
||||||
|
} from "./filename.js";
|
||||||
import {
|
import {
|
||||||
fetchMLDataBatch,
|
fetchMLDataBatch,
|
||||||
MLDATA_BATCH_SIZE,
|
MLDATA_BATCH_SIZE,
|
||||||
@@ -127,7 +131,15 @@ export const runMetadataBackup = async (
|
|||||||
const col = lib.getCollection(album.collectionID);
|
const col = lib.getCollection(album.collectionID);
|
||||||
if (!col) continue;
|
if (!col) continue;
|
||||||
|
|
||||||
const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
|
// The album's ID keeps the folders apart, so the name after it can be
|
||||||
|
// cut short to fit in a file name.
|
||||||
|
const prefix = `${col.id}-`;
|
||||||
|
const dirName =
|
||||||
|
prefix +
|
||||||
|
shortenToBytes(
|
||||||
|
sanitizeFileName(col.name, "unnamed"),
|
||||||
|
MAX_NAME_BYTES - prefix.length,
|
||||||
|
);
|
||||||
const colDir = join(outDir, "collections", dirName);
|
const colDir = join(outDir, "collections", dirName);
|
||||||
mkdirSync(colDir, { recursive: true });
|
mkdirSync(colDir, { recursive: true });
|
||||||
|
|
||||||
|
|||||||
@@ -548,7 +548,7 @@ describe("the read commands with records quak cannot decrypt", () => {
|
|||||||
expect(await collectionsCommand(context(ente), {})).toBe(0);
|
expect(await collectionsCommand(context(ente), {})).toBe(0);
|
||||||
expect(stdout.text).toBe("1\talbum\tTrip\n");
|
expect(stdout.text).toBe("1\talbum\tTrip\n");
|
||||||
|
|
||||||
expect(await filesCommand(context(ente), { collection: "1" })).toBe(0);
|
expect(await filesCommand(context(ente), "1", {})).toBe(0);
|
||||||
expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
|
expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
+10
-11
@@ -1795,12 +1795,11 @@ describe("backup album folders", () => {
|
|||||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
|
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
|
||||||
});
|
});
|
||||||
|
|
||||||
// The extensions of these titles leave no room for the ID, so each is cut
|
// The extensions of these titles leave no room for the ID and the "a"
|
||||||
// short and the name before the ID keeps only the bytes that cut leaves
|
// before them, so each is cut short to 250 bytes, which leaves one, for
|
||||||
// over. File 1's extension is 303 bytes and is cut to 251, which leaves
|
// the "a". File 1's extension is 303 bytes and file 2's 271. Neither
|
||||||
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three
|
// extension is only letters and digits, so both originals are saved with
|
||||||
// bytes, which leaves one, for the "a". Neither extension is only letters
|
// `.bin`, a name that fits.
|
||||||
// and digits, so both originals are saved with `.bin`, a name that fits.
|
|
||||||
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
|
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
const lib = libraryOf([
|
const lib = libraryOf([
|
||||||
@@ -1815,7 +1814,7 @@ describe("backup album folders", () => {
|
|||||||
|
|
||||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||||
|
|
||||||
const link = ` (1).${"L".repeat(250)}`;
|
const link = `a (1).${"L".repeat(249)}`;
|
||||||
const link2 = `a (2).${"€".repeat(83)}`;
|
const link2 = `a (2).${"€".repeat(83)}`;
|
||||||
expect(Buffer.byteLength(link)).toBe(255);
|
expect(Buffer.byteLength(link)).toBe(255);
|
||||||
expect(Buffer.byteLength(link2)).toBe(255);
|
expect(Buffer.byteLength(link2)).toBe(255);
|
||||||
@@ -1830,9 +1829,9 @@ describe("backup album folders", () => {
|
|||||||
|
|
||||||
// Files 1 and 3 share a title, so each gets its ID before the
|
// Files 1 and 3 share a title, so each gets its ID before the
|
||||||
// extension. File 2's title is too long, and its extension leaves no
|
// extension. File 2's title is too long, and its extension leaves no
|
||||||
// room for the ID. Were the ID put at the end of file 2's title cut
|
// room for the ID and the "x" before it. Were the ID put at the end of
|
||||||
// short, its link would have file 1's name, and the link made second
|
// file 2's title cut short, its link would have file 1's name, and the
|
||||||
// would replace the first.
|
// link made second would replace the first.
|
||||||
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
|
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
const shared = `x.${"Q".repeat(245)} (2)`;
|
const shared = `x.${"Q".repeat(245)} (2)`;
|
||||||
@@ -1852,8 +1851,8 @@ describe("backup album folders", () => {
|
|||||||
expect(result.failed).toBe(0);
|
expect(result.failed).toBe(0);
|
||||||
expect(tree(outDir)).toEqual([
|
expect(tree(outDir)).toEqual([
|
||||||
"Trip/",
|
"Trip/",
|
||||||
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
|
|
||||||
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
|
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
|
||||||
|
`Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`,
|
||||||
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
|
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
|
||||||
"Trip.json",
|
"Trip.json",
|
||||||
]);
|
]);
|
||||||
|
|||||||
+24
-6
@@ -1,6 +1,7 @@
|
|||||||
/**
|
/**
|
||||||
* Tests for the retry options `bin/quak.ts` loads the saved session with:
|
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
||||||
* `quak backup` gets the unattended ones, every other command the default.
|
* (`quak backup` gets the unattended ones, every other command the default),
|
||||||
|
* and for `quak files` taking its collection ID as an optional argument.
|
||||||
*
|
*
|
||||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||||
* replaced by one that records the options it is given and reports no session,
|
* replaced by one that records the options it is given and reports no session,
|
||||||
@@ -26,14 +27,19 @@ vi.mock("../../src/cli-session.js", () => ({
|
|||||||
const argv = process.argv;
|
const argv = process.argv;
|
||||||
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
|
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
|
||||||
|
|
||||||
// Run `quak <args>` to completion. The "Not logged in" message and the exit
|
// Run `quak <args>` to completion and return what it wrote on stderr, which is
|
||||||
// are swallowed.
|
// kept off the terminal. The exit is swallowed.
|
||||||
const quak = async (...args: string[]): Promise<void> => {
|
const quak = async (...args: string[]): Promise<string> => {
|
||||||
vi.spyOn(process.stderr, "write").mockImplementation(() => true);
|
let written = "";
|
||||||
|
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
|
||||||
|
written += String(text);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
|
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
|
||||||
process.argv = ["node", "quak", ...args];
|
process.argv = ["node", "quak", ...args];
|
||||||
vi.resetModules();
|
vi.resetModules();
|
||||||
await import("../../bin/quak.js");
|
await import("../../bin/quak.js");
|
||||||
|
return written;
|
||||||
};
|
};
|
||||||
|
|
||||||
afterEach(() => {
|
afterEach(() => {
|
||||||
@@ -69,3 +75,15 @@ describe("bin/quak.ts session loading", () => {
|
|||||||
expect(loaded).toEqual([undefined]);
|
expect(loaded).toEqual([undefined]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// The command line takes `files` with or without a collection ID, and the
|
||||||
|
// command runs as far as its session check either way (issue #211).
|
||||||
|
describe("bin/quak.ts files", () => {
|
||||||
|
it("runs without a collection ID", async () => {
|
||||||
|
expect(await quak("files")).toMatch(/^Not logged in/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("runs with a collection ID", async () => {
|
||||||
|
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
||||||
|
});
|
||||||
|
});
|
||||||
+207
-47
@@ -21,7 +21,7 @@ import {
|
|||||||
utimesSync,
|
utimesSync,
|
||||||
writeFileSync,
|
writeFileSync,
|
||||||
} from "node:fs";
|
} from "node:fs";
|
||||||
import { join } from "node:path";
|
import { basename, join } from "node:path";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
import { PassThrough } from "node:stream";
|
import { PassThrough } from "node:stream";
|
||||||
import * as jpegJs from "jpeg-js";
|
import * as jpegJs from "jpeg-js";
|
||||||
@@ -121,25 +121,31 @@ const FILES: Record<number, EnteFile[]> = {
|
|||||||
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
|
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
|
||||||
// original fail; `emptyThumbID` makes the server report that file's
|
// original fail; `emptyThumbID` makes the server report that file's
|
||||||
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
|
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
|
||||||
// the collection's updationTime as the server does, and `refreshError` makes
|
// the collection's updationTime as the server does. `beachInWork` adds
|
||||||
// listing collections fail with that message.
|
// beach.jpg (100) to Work as well, so that one file is in both albums.
|
||||||
|
// `refreshError` makes listing collections fail with that message.
|
||||||
const fakeClient = (
|
const fakeClient = (
|
||||||
opts: {
|
opts: {
|
||||||
failID?: number;
|
failID?: number;
|
||||||
emptyThumbID?: number;
|
emptyThumbID?: number;
|
||||||
withNewFile?: boolean;
|
withNewFile?: boolean;
|
||||||
|
beachInWork?: boolean;
|
||||||
refreshError?: string;
|
refreshError?: string;
|
||||||
} = {},
|
} = {},
|
||||||
) => {
|
) => {
|
||||||
const collections = opts.withNewFile
|
const collections = opts.withNewFile
|
||||||
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
|
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
|
||||||
: COLLECTIONS;
|
: COLLECTIONS;
|
||||||
const files = opts.withNewFile
|
let files = FILES;
|
||||||
? {
|
if (opts.withNewFile) {
|
||||||
...FILES,
|
files = {
|
||||||
1: [...FILES[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
|
...files,
|
||||||
}
|
1: [...files[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
|
||||||
: FILES;
|
};
|
||||||
|
}
|
||||||
|
if (opts.beachInWork) {
|
||||||
|
files = { ...files, 2: [...files[2], file(100, 2, "beach.jpg")] };
|
||||||
|
}
|
||||||
const source: ContentSource = {
|
const source: ContentSource = {
|
||||||
original: async ({ file: f, destination }) => {
|
original: async ({ file: f, destination }) => {
|
||||||
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
|
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
|
||||||
@@ -187,6 +193,9 @@ const FETCHED_BOTH_ALBUMS =
|
|||||||
"Fetched the files of 1 of 2 collections\n" +
|
"Fetched the files of 1 of 2 collections\n" +
|
||||||
"Fetched the files of 2 of 2 collections\n";
|
"Fetched the files of 2 of 2 collections\n";
|
||||||
|
|
||||||
|
// The same for an account with one album.
|
||||||
|
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
|
||||||
|
|
||||||
// Collects everything written to it.
|
// Collects everything written to it.
|
||||||
class Output {
|
class Output {
|
||||||
text = "";
|
text = "";
|
||||||
@@ -254,7 +263,7 @@ describe("session file", () => {
|
|||||||
const dir = join(root, "backup");
|
const dir = join(root, "backup");
|
||||||
expect(await whoamiCommand(ctx)).toBe(3);
|
expect(await whoamiCommand(ctx)).toBe(3);
|
||||||
expect(await collectionsCommand(ctx, {})).toBe(3);
|
expect(await collectionsCommand(ctx, {})).toBe(3);
|
||||||
expect(await filesCommand(ctx, { collection: "1" })).toBe(3);
|
expect(await filesCommand(ctx, "1", {})).toBe(3);
|
||||||
expect(await getCommand(ctx, "100", {})).toBe(3);
|
expect(await getCommand(ctx, "100", {})).toBe(3);
|
||||||
expect(await getThumbCommand(ctx, "100", {})).toBe(3);
|
expect(await getThumbCommand(ctx, "100", {})).toBe(3);
|
||||||
expect(await backupMetadataCommand(ctx, dir, {})).toBe(3);
|
expect(await backupMetadataCommand(ctx, dir, {})).toBe(3);
|
||||||
@@ -501,17 +510,14 @@ describe("collections", () => {
|
|||||||
|
|
||||||
describe("files", () => {
|
describe("files", () => {
|
||||||
it("prints one tab-separated line per file", async () => {
|
it("prints one tab-separated line per file", async () => {
|
||||||
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
|
expect(await filesCommand(context(), "1", {})).toBe(0);
|
||||||
expect(stdout.text).toBe(
|
expect(stdout.text).toBe(
|
||||||
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
|
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("prints a JSON array with --json", async () => {
|
it("prints a JSON array with --json", async () => {
|
||||||
const code = await filesCommand(context(), {
|
const code = await filesCommand(context(), "2", { json: true });
|
||||||
collection: "2",
|
|
||||||
json: true,
|
|
||||||
});
|
|
||||||
expect(code).toBe(0);
|
expect(code).toBe(0);
|
||||||
expect(JSON.parse(stdout.text)).toEqual([
|
expect(JSON.parse(stdout.text)).toEqual([
|
||||||
{
|
{
|
||||||
@@ -525,11 +531,43 @@ describe("files", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it("exits 1 for an unknown collection", async () => {
|
it("exits 1 for an unknown collection", async () => {
|
||||||
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
|
expect(await filesCommand(context(), "9", {})).toBe(1);
|
||||||
expect(stderr.text).toBe(
|
expect(stderr.text).toBe(
|
||||||
FETCHED_BOTH_ALBUMS + "Collection 9 not found\n",
|
FETCHED_BOTH_ALBUMS + "Collection 9 not found\n",
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Without a collection ID it lists every file in the account once (issue
|
||||||
|
// #211). beach.jpg is in Vacation and Work, and Vacation comes first, so it
|
||||||
|
// is listed once, as Vacation's.
|
||||||
|
it("without a collection ID prints every file once", async () => {
|
||||||
|
const ctx = context(fakeClient({ beachInWork: true }));
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
expect(stdout.text).toBe(
|
||||||
|
"100\timage\tbeach.jpg\n" +
|
||||||
|
"101\timage\tsunset.jpg\n" +
|
||||||
|
"200\timage\tdiagram.png\n",
|
||||||
|
);
|
||||||
|
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("without a collection ID gives each file the first album holding it with --json", async () => {
|
||||||
|
const ctx = context(fakeClient({ beachInWork: true }));
|
||||||
|
expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
|
||||||
|
expect(
|
||||||
|
JSON.parse(stdout.text).map(
|
||||||
|
(row: { id: number; collectionID: number }) => [
|
||||||
|
row.id,
|
||||||
|
row.collectionID,
|
||||||
|
],
|
||||||
|
),
|
||||||
|
).toEqual([
|
||||||
|
[100, 1],
|
||||||
|
[101, 1],
|
||||||
|
[200, 2],
|
||||||
|
]);
|
||||||
|
expect(stderr.text).toBe("");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("get and get-thumb", () => {
|
describe("get and get-thumb", () => {
|
||||||
@@ -562,17 +600,109 @@ describe("get and get-thumb", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// A file system refuses a file name over 255 bytes. The account here is one
|
||||||
|
// album, whose name is 300 bytes, holding one file, 700, with the title given.
|
||||||
|
describe("names from an album name or a title over 255 bytes", () => {
|
||||||
|
const longNames = (title: string): Client =>
|
||||||
|
({
|
||||||
|
...fakeClient(),
|
||||||
|
collectionsSince: async () => ({
|
||||||
|
collections: [collection(7, "A".repeat(300))],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
}),
|
||||||
|
filesSince: async () => ({
|
||||||
|
files: [file(700, 7, title)],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 1,
|
||||||
|
}),
|
||||||
|
}) as unknown as Client;
|
||||||
|
const longTitle = `${"t".repeat(296)}.jpg`;
|
||||||
|
|
||||||
|
// Run `command` in a new directory, where `get` and `get-thumb` without
|
||||||
|
// `--out` write, and return its exit code and the directory.
|
||||||
|
const inNewDirectory = async (
|
||||||
|
command: () => Promise<number>,
|
||||||
|
): Promise<{ code: number; dir: string }> => {
|
||||||
|
const dir = mkdtempSync(join(root, "cwd-"));
|
||||||
|
const previous = process.cwd();
|
||||||
|
process.chdir(dir);
|
||||||
|
try {
|
||||||
|
return { code: await command(), dir };
|
||||||
|
} finally {
|
||||||
|
process.chdir(previous);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
it("get cuts the title short to 255 bytes, keeping its extension, and exits 0", async () => {
|
||||||
|
const { code, dir } = await inNewDirectory(() =>
|
||||||
|
getCommand(context(longNames(longTitle)), "700", {}),
|
||||||
|
);
|
||||||
|
|
||||||
|
const name = `${"t".repeat(251)}.jpg`;
|
||||||
|
expect(Buffer.byteLength(name)).toBe(255);
|
||||||
|
expect(code).toBe(0);
|
||||||
|
expect(readdirSync(dir)).toEqual([name]);
|
||||||
|
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
|
||||||
|
const { code, dir } = await inNewDirectory(() =>
|
||||||
|
getThumbCommand(context(longNames(longTitle)), "700", {}),
|
||||||
|
);
|
||||||
|
|
||||||
|
const name = `thumb_${"t".repeat(245)}.jpg`;
|
||||||
|
expect(Buffer.byteLength(name)).toBe(255);
|
||||||
|
expect(code).toBe(0);
|
||||||
|
expect(readdirSync(dir)).toEqual([name]);
|
||||||
|
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
|
||||||
|
const dir = join(root, "dump");
|
||||||
|
|
||||||
|
const code = await backupMetadataCommand(
|
||||||
|
context(longNames("beach.jpg")),
|
||||||
|
dir,
|
||||||
|
{},
|
||||||
|
);
|
||||||
|
|
||||||
|
const folder = `7-${"A".repeat(253)}`;
|
||||||
|
expect(Buffer.byteLength(folder)).toBe(255);
|
||||||
|
expect(code).toBe(0);
|
||||||
|
expect(readdirSync(join(dir, "collections"))).toEqual([folder]);
|
||||||
|
expect(readdirSync(join(dir, "collections", folder)).sort()).toEqual([
|
||||||
|
"700.json",
|
||||||
|
"_collection.json",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
// The title's extension is 300 letters, too long for the save path's
|
||||||
|
// name, so the original is saved with `.bin`.
|
||||||
|
it("backup saves a title with a 300-letter extension as .bin, and exits 0", async () => {
|
||||||
|
const dir = join(root, "backup");
|
||||||
|
const title = `b.${"e".repeat(300)}`;
|
||||||
|
|
||||||
|
const code = await backupCommand(context(longNames(title)), dir, {});
|
||||||
|
|
||||||
|
const path = savePath(dir, file(700, 7, title));
|
||||||
|
expect(code).toBe(0);
|
||||||
|
expect(basename(path)).toMatch(/^\d{4}-\d{2}-\d{2}\.700\.bin$/);
|
||||||
|
expect(readFileSync(path)).toEqual(Buffer.alloc(7, 700 & 0xff));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
// A live photo, which Ente stores as one ZIP, is written as its image and its
|
// A live photo, which Ente stores as one ZIP, is written as its image and its
|
||||||
// video, which a photo viewer can open. The account here is Vacation holding
|
// video, which a photo viewer can open. The account here is Vacation holding
|
||||||
// one live photo, 300, downloaded through the real download layer
|
// one live photo, 300, downloaded through the real download layer
|
||||||
// (test/live-photo.ts).
|
// (test/live-photo.ts).
|
||||||
describe("a live photo", () => {
|
describe("a live photo", () => {
|
||||||
// What `get` prints on stderr while its refresh fetches Vacation's files.
|
const livePhotoClient = async (
|
||||||
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
|
image = IMAGE,
|
||||||
|
title = "IMG_0300.HEIC",
|
||||||
const livePhotoClient = async (image = IMAGE): Promise<Client> => {
|
): Promise<Client> => {
|
||||||
const { file: live, body } = await asLivePhoto(
|
const { file: live, body } = await asLivePhoto(
|
||||||
file(300, 1, "IMG_0300.HEIC"),
|
file(300, 1, title),
|
||||||
livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
|
livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
|
||||||
livePhotoHash(image, VIDEO),
|
livePhotoHash(image, VIDEO),
|
||||||
);
|
);
|
||||||
@@ -620,6 +750,44 @@ describe("a live photo", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Run `get` without `--out` in a new directory for a live photo titled
|
||||||
|
// `title`, and return its exit code and the names it wrote.
|
||||||
|
const getInNewDirectory = async (
|
||||||
|
title: string,
|
||||||
|
): Promise<{ code: number; names: string[] }> => {
|
||||||
|
const ctx = context(await livePhotoClient(IMAGE, title));
|
||||||
|
const dir = mkdtempSync(join(root, "cwd-"));
|
||||||
|
const previous = process.cwd();
|
||||||
|
process.chdir(dir);
|
||||||
|
try {
|
||||||
|
const code = await getCommand(ctx, "300", {});
|
||||||
|
return { code, names: readdirSync(dir).sort() };
|
||||||
|
} finally {
|
||||||
|
process.chdir(previous);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
const imageName = `${"t".repeat(250)}.heic`;
|
||||||
|
const videoName = `${"t".repeat(251)}.mov`;
|
||||||
|
|
||||||
|
// The title fits, but ".heic" is one byte longer than ".jpg", so the
|
||||||
|
// image's name is cut short once its extension is in it.
|
||||||
|
it("get cuts its image's name short for a 255-byte title ending in .jpg, and exits 0", async () => {
|
||||||
|
const title = `${"t".repeat(251)}.jpg`;
|
||||||
|
|
||||||
|
const result = await getInNewDirectory(title);
|
||||||
|
|
||||||
|
expect(Buffer.byteLength(title)).toBe(255);
|
||||||
|
expect(Buffer.byteLength(imageName)).toBe(255);
|
||||||
|
expect(Buffer.byteLength(videoName)).toBe(255);
|
||||||
|
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("get cuts the names of its image and video short for a 300-byte title with no extension, and exits 0", async () => {
|
||||||
|
const result = await getInNewDirectory("t".repeat(300));
|
||||||
|
|
||||||
|
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
|
||||||
|
});
|
||||||
|
|
||||||
it("get --out writes the image there and the video beside it", async () => {
|
it("get --out writes the image there and the video beside it", async () => {
|
||||||
const out = join(root, "photo.jpg");
|
const out = join(root, "photo.jpg");
|
||||||
const video = join(root, "photo.mov");
|
const video = join(root, "photo.mov");
|
||||||
@@ -1132,7 +1300,7 @@ describe("backup-metadata --exif", () => {
|
|||||||
// collection list alone and stores nothing.)
|
// collection list alone and stores nothing.)
|
||||||
describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
||||||
beforeEach(async () => {
|
beforeEach(async () => {
|
||||||
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
|
expect(await filesCommand(context(), "1", {})).toBe(0);
|
||||||
stdout.text = "";
|
stdout.text = "";
|
||||||
stderr.text = "";
|
stderr.text = "";
|
||||||
});
|
});
|
||||||
@@ -1191,10 +1359,10 @@ describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// `files --collection`, `get`, `get-thumb` and `helper fix-missing-thumbnails
|
// `files`, `get`, `get-thumb` and `helper fix-missing-thumbnails --file` take
|
||||||
// --file` take an ID only when it is written in decimal digits and is a safe
|
// an ID only when it is written in decimal digits and is a safe integer.
|
||||||
// integer. `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty
|
// `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty string as 0,
|
||||||
// string as 0, so the command would act on a file the user never named.
|
// so the command would act on a file the user never named.
|
||||||
// Anything else is refused with one line naming it, and exit 1, before any
|
// Anything else is refused with one line naming it, and exit 1, before any
|
||||||
// request.
|
// request.
|
||||||
//
|
//
|
||||||
@@ -1248,21 +1416,18 @@ describe("an ID given on the command line", () => {
|
|||||||
|
|
||||||
it("files looks up a decimal collection ID", async () => {
|
it("files looks up a decimal collection ID", async () => {
|
||||||
const ctx = emptyAccount();
|
const ctx = emptyAccount();
|
||||||
expect(await filesCommand(ctx, { collection: "1" })).toBe(1);
|
expect(await filesCommand(ctx, "1", {})).toBe(1);
|
||||||
expect(stderr.text).toBe("Collection 1 not found\n");
|
expect(stderr.text).toBe("Collection 1 not found\n");
|
||||||
expect(requests).not.toHaveLength(0);
|
expect(requests).not.toHaveLength(0);
|
||||||
});
|
});
|
||||||
|
|
||||||
it.each(NOT_IDS)(
|
it.each(NOT_IDS)("files refuses %j before any request", async (arg) => {
|
||||||
"files refuses --collection %j before any request",
|
const ctx = emptyAccount();
|
||||||
async (arg) => {
|
expect(await filesCommand(ctx, arg, {})).toBe(1);
|
||||||
const ctx = emptyAccount();
|
expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
|
||||||
expect(await filesCommand(ctx, { collection: arg })).toBe(1);
|
expect(stdout.text).toBe("");
|
||||||
expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
|
expect(requests).toHaveLength(0);
|
||||||
expect(stdout.text).toBe("");
|
});
|
||||||
expect(requests).toHaveLength(0);
|
|
||||||
},
|
|
||||||
);
|
|
||||||
|
|
||||||
it("get looks up a decimal file ID", async () => {
|
it("get looks up a decimal file ID", async () => {
|
||||||
expect(await getCommand(emptyAccount(), "100", {})).toBe(1);
|
expect(await getCommand(emptyAccount(), "100", {})).toBe(1);
|
||||||
@@ -1444,14 +1609,12 @@ describe("an account with many albums", () => {
|
|||||||
// It stored nothing that would make a refresh skip an album's files:
|
// It stored nothing that would make a refresh skip an album's files:
|
||||||
// `files` after it still finds the album's photo.
|
// `files` after it still finds the album's photo.
|
||||||
stdout.text = "";
|
stdout.text = "";
|
||||||
expect(await filesCommand(ctx, { collection: "7" })).toBe(0);
|
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
||||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("files counts on stderr each album whose files its refresh fetched", async () => {
|
it("files counts on stderr each album whose files its refresh fetched", async () => {
|
||||||
expect(await filesCommand(accountContext(), { collection: "7" })).toBe(
|
expect(await filesCommand(accountContext(), "7", {})).toBe(0);
|
||||||
0,
|
|
||||||
);
|
|
||||||
|
|
||||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||||
let progress = "";
|
let progress = "";
|
||||||
@@ -1464,12 +1627,9 @@ describe("an account with many albums", () => {
|
|||||||
// With `--json` the refresh's progress is not printed, as `backup` and
|
// With `--json` the refresh's progress is not printed, as `backup` and
|
||||||
// the thumbnail helpers print none of theirs.
|
// the thumbnail helpers print none of theirs.
|
||||||
it("files --json prints no progress", async () => {
|
it("files --json prints no progress", async () => {
|
||||||
expect(
|
expect(await filesCommand(accountContext(), "7", { json: true })).toBe(
|
||||||
await filesCommand(accountContext(), {
|
0,
|
||||||
collection: "7",
|
);
|
||||||
json: true,
|
|
||||||
}),
|
|
||||||
).toBe(0);
|
|
||||||
|
|
||||||
expect(JSON.parse(stdout.text)).toEqual([
|
expect(JSON.parse(stdout.text)).toEqual([
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -93,6 +93,10 @@ class FakeLibrary implements FreshReadLibrary {
|
|||||||
return this.collections.find((c) => c.id === id);
|
return this.collections.find((c) => c.id === id);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
listCollections(): Collection[] {
|
||||||
|
return this.refreshed ? this.collections : [];
|
||||||
|
}
|
||||||
|
|
||||||
listFiles(collectionID: number): EnteFile[] {
|
listFiles(collectionID: number): EnteFile[] {
|
||||||
return this.refreshed
|
return this.refreshed
|
||||||
? this.files.filter((f) => f.collectionID === collectionID)
|
? this.files.filter((f) => f.collectionID === collectionID)
|
||||||
@@ -175,6 +179,30 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
|||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Without a collection ID, `files` lists every file in the account once
|
||||||
|
// (issue #211). File 11 is in collections 1 and 2; collection 2 is
|
||||||
|
// enumerated first, so 11 is listed once, with collection 2's record.
|
||||||
|
it("freshFiles without a collection ID lists every file once, from the first collection holding it", async () => {
|
||||||
|
const files = [
|
||||||
|
file(10, 1, 1_700_000_000_000_000),
|
||||||
|
file(11, 1, 1_700_000_000_000_001),
|
||||||
|
file(20, 2, 1_700_000_000_000_002),
|
||||||
|
file(11, 2, 1_700_000_000_000_001),
|
||||||
|
];
|
||||||
|
const lib = new FakeLibrary(
|
||||||
|
[collection(2, 100), collection(1, 200)],
|
||||||
|
files,
|
||||||
|
);
|
||||||
|
|
||||||
|
const rows = await freshFiles(lib);
|
||||||
|
expect(lib.freshCalls).toBe(1);
|
||||||
|
expect(rows?.map((f) => [f.id, f.collectionID])).toEqual([
|
||||||
|
[20, 2],
|
||||||
|
[11, 2],
|
||||||
|
[10, 1],
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
it("freshFiles returns undefined for an unknown collection", async () => {
|
it("freshFiles returns undefined for an unknown collection", async () => {
|
||||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||||
const rows = await freshFiles(lib, 999);
|
const rows = await freshFiles(lib, 999);
|
||||||
|
|||||||
@@ -658,6 +658,26 @@ describe("downloadFile", () => {
|
|||||||
expect(readdirSync(parent)).toEqual(["cwd"]);
|
expect(readdirSync(parent)).toEqual(["cwd"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// A file system refuses a file name over 255 bytes.
|
||||||
|
it("cuts a title too long for a file name short, keeping its extension", async () => {
|
||||||
|
const plaintext = new Uint8Array([1, 2, 3]);
|
||||||
|
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||||
|
const { header, ciphertext } = encryptFileBody(plaintext, key);
|
||||||
|
const thumbPush =
|
||||||
|
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
|
||||||
|
const file = buildMockEnteFile(key, header, thumbPush.header);
|
||||||
|
file.metadata.title = `${"t".repeat(296)}.jpg`;
|
||||||
|
const dir = mkdtempSync(join(testDir, "long-title-"));
|
||||||
|
|
||||||
|
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
|
||||||
|
const result = await inDirectory(dir, () => downloadFile(api, file));
|
||||||
|
|
||||||
|
const name = `${"t".repeat(251)}.jpg`;
|
||||||
|
expect(Buffer.byteLength(name)).toBe(255);
|
||||||
|
expect(result.path).toBe(name);
|
||||||
|
expect(readdirSync(dir)).toEqual([name]);
|
||||||
|
});
|
||||||
|
|
||||||
it("uses an explicit outPath verbatim, even one with ..", async () => {
|
it("uses an explicit outPath verbatim, even one with ..", async () => {
|
||||||
// The caller is trusted: its path is not sanitized.
|
// The caller is trusted: its path is not sanitized.
|
||||||
const { api, file } = fixtureFor(
|
const { api, file } = fixtureFor(
|
||||||
@@ -751,6 +771,28 @@ describe("downloadThumbnail", () => {
|
|||||||
expect(result.path).toBe("thumb__etc_passwd");
|
expect(result.path).toBe("thumb__etc_passwd");
|
||||||
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
|
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("leaves room for thumb_ when it cuts a title too long for a file name short", async () => {
|
||||||
|
const plaintext = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]);
|
||||||
|
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||||
|
const filePush =
|
||||||
|
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
|
||||||
|
const { header: thumbHeader, ciphertext: thumbCipher } =
|
||||||
|
encryptFileBody(plaintext, key);
|
||||||
|
const file = buildMockEnteFile(key, filePush.header, thumbHeader);
|
||||||
|
file.metadata.title = `${"t".repeat(296)}.jpg`;
|
||||||
|
const dir = mkdtempSync(join(testDir, "long-thumb-title-"));
|
||||||
|
|
||||||
|
const api = new ApiClient({ fetch: mockFetchForBody(thumbCipher) });
|
||||||
|
const result = await inDirectory(dir, () =>
|
||||||
|
downloadThumbnail(api, file),
|
||||||
|
);
|
||||||
|
|
||||||
|
const name = `thumb_${"t".repeat(245)}.jpg`;
|
||||||
|
expect(Buffer.byteLength(name)).toBe(255);
|
||||||
|
expect(result.path).toBe(name);
|
||||||
|
expect(readdirSync(dir)).toEqual([name]);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -1881,6 +1923,53 @@ describe("downloadFile live photos", () => {
|
|||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// A file system refuses a file name over 255 bytes. Each name is cut
|
||||||
|
// short once its own extension is in it.
|
||||||
|
const imageName = `${"t".repeat(250)}.heic`;
|
||||||
|
const videoName = `${"t".repeat(251)}.mov`;
|
||||||
|
|
||||||
|
// The title fits, but ".heic" is one byte longer than ".jpg".
|
||||||
|
it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => {
|
||||||
|
const title = `${"t".repeat(251)}.jpg`;
|
||||||
|
const t = setup(livePhotoZip(), { ...livePhoto, title });
|
||||||
|
|
||||||
|
const result = await inDirectory(t.dir, () =>
|
||||||
|
downloadFile(t.api, t.file),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(Buffer.byteLength(title)).toBe(255);
|
||||||
|
expect(Buffer.byteLength(imageName)).toBe(255);
|
||||||
|
expect(Buffer.byteLength(videoName)).toBe(255);
|
||||||
|
expect(result.path).toBe(imageName);
|
||||||
|
expect(result.videoPath).toBe(videoName);
|
||||||
|
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => {
|
||||||
|
const t = setup(livePhotoZip(), {
|
||||||
|
...livePhoto,
|
||||||
|
title: "t".repeat(300),
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await inDirectory(t.dir, () =>
|
||||||
|
downloadFile(t.api, t.file),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result.path).toBe(imageName);
|
||||||
|
expect(result.videoPath).toBe(videoName);
|
||||||
|
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
|
||||||
|
});
|
||||||
|
|
||||||
|
// The caller's path is used as is; only the extension is each part's own.
|
||||||
|
it("keeps an outPath too long for a file name as given", async () => {
|
||||||
|
const t = setup(livePhotoZip(), livePhoto);
|
||||||
|
const outPath = join(t.dir, `${"t".repeat(251)}.jpg`);
|
||||||
|
|
||||||
|
await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow(
|
||||||
|
/ENAMETOOLONG/,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it("gives each part its own extension from the ZIP, letters and digits only", async () => {
|
it("gives each part its own extension from the ZIP, letters and digits only", async () => {
|
||||||
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
|
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
|
||||||
const t = setup(zip, livePhoto);
|
const t = setup(zip, livePhoto);
|
||||||
|
|||||||
@@ -15,8 +15,10 @@
|
|||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
|
|
||||||
import {
|
import {
|
||||||
|
defaultFileName,
|
||||||
safeExtension,
|
safeExtension,
|
||||||
sanitizeFileName,
|
sanitizeFileName,
|
||||||
|
shortenKeepingExtension,
|
||||||
shortenToBytes,
|
shortenToBytes,
|
||||||
} from "../../src/filename.js";
|
} from "../../src/filename.js";
|
||||||
|
|
||||||
@@ -94,6 +96,12 @@ describe("safeExtension", () => {
|
|||||||
expect(safeExtension("x.jp g")).toBe(".bin");
|
expect(safeExtension("x.jp g")).toBe(".bin");
|
||||||
expect(safeExtension("x.jpg\0")).toBe(".bin");
|
expect(safeExtension("x.jpg\0")).toBe(".bin");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => {
|
||||||
|
expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`);
|
||||||
|
expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin");
|
||||||
|
expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("shortenToBytes", () => {
|
describe("shortenToBytes", () => {
|
||||||
@@ -118,3 +126,103 @@ describe("shortenToBytes", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("shortenKeepingExtension", () => {
|
||||||
|
it("returns a name that fits unchanged", () => {
|
||||||
|
expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg");
|
||||||
|
expect(shortenKeepingExtension("beach", 5)).toBe("beach");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("cuts the part before the extension short and keeps the extension", () => {
|
||||||
|
expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe(
|
||||||
|
`${"t".repeat(251)}.jpg`,
|
||||||
|
);
|
||||||
|
expect(shortenKeepingExtension("é".repeat(150), 255)).toBe(
|
||||||
|
"é".repeat(127),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// A name cut down to its extension would start with a dot, and so be
|
||||||
|
// hidden.
|
||||||
|
it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
|
||||||
|
expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
|
||||||
|
`b.${"e".repeat(253)}`,
|
||||||
|
);
|
||||||
|
expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
|
||||||
|
`a.${"e".repeat(253)}`,
|
||||||
|
);
|
||||||
|
expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
|
||||||
|
`é.${"e".repeat(252)}`,
|
||||||
|
);
|
||||||
|
// "€" takes three bytes, so the extension is cut to 253 and the
|
||||||
|
// part before it keeps the two bytes left over, for the "a".
|
||||||
|
expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe(
|
||||||
|
`a.${"€".repeat(84)}`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("puts `insert` right before the extension and leaves room for it", () => {
|
||||||
|
expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe(
|
||||||
|
"beach (7).jpg",
|
||||||
|
);
|
||||||
|
expect(
|
||||||
|
shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"),
|
||||||
|
).toBe(`${"t".repeat(247)} (7).jpg`);
|
||||||
|
expect(
|
||||||
|
shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
|
||||||
|
).toBe(`b (7).${"e".repeat(249)}`);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("defaultFileName", () => {
|
||||||
|
it("is the title when it is safe and fits", () => {
|
||||||
|
expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("replaces unsafe characters, and is file-<id> for an empty title", () => {
|
||||||
|
expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
|
||||||
|
expect(defaultFileName("", 42)).toBe("file-42");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("cuts a title over 255 bytes short, keeping its extension", () => {
|
||||||
|
expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
|
||||||
|
`${"t".repeat(251)}.jpg`,
|
||||||
|
);
|
||||||
|
expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
|
||||||
|
`b.${"e".repeat(253)}`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("puts `prefix` before the title and leaves room for it", () => {
|
||||||
|
expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
|
||||||
|
"thumb_beach.jpg",
|
||||||
|
);
|
||||||
|
expect(
|
||||||
|
defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
|
||||||
|
).toBe(`thumb_${"t".repeat(245)}.jpg`);
|
||||||
|
});
|
||||||
|
|
||||||
|
// A live photo's image and video are each named with their own
|
||||||
|
// extension, and the name is cut short once that extension is in it.
|
||||||
|
it("puts `ext` in place of the title's extension before cutting the name short", () => {
|
||||||
|
expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
|
||||||
|
"IMG_0001.mov",
|
||||||
|
);
|
||||||
|
expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
|
||||||
|
expect(
|
||||||
|
defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
|
||||||
|
).toBe(`${"t".repeat(250)}.heic`);
|
||||||
|
expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
|
||||||
|
`${"t".repeat(250)}.heic`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
|
||||||
|
// or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
|
||||||
|
// name, and the part before the extension gives up a byte for it.
|
||||||
|
it("never returns a Windows device name, even once cut short", () => {
|
||||||
|
const ext = `.${"e".repeat(251)}`;
|
||||||
|
expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
|
||||||
|
expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in new issue
Block a user