Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
aeb275e137 | ||
|
|
67794bf5f8 | ||
|
|
b5fcbf4400 |
@@ -158,11 +158,11 @@ output to `~/quak-backup.log`. Sunday's run adds `--verify`:
|
||||
|
||||
A run with `--verify` does all a plain run does. It also hashes each original
|
||||
already in the backup and compares the result with the content hash. The content
|
||||
hash is the hash of the file's bytes that Ente recorded at upload. It fetches a
|
||||
new copy of each original that does not match, and replaces the original once
|
||||
the copy matches. Sunday's run is the `--verify` run, rather than a second job
|
||||
that night, because two backups of one directory cannot run at once. The second
|
||||
one exits 2 without doing anything.
|
||||
hash is the hash of the file's bytes that Ente recorded at upload. The run
|
||||
fetches a new copy of each original that does not match, and replaces the
|
||||
original once the copy matches. Sunday's run is the `--verify` run, rather than
|
||||
a second job that night, because two backups of one directory cannot run at
|
||||
once. The second one exits 2 without doing anything.
|
||||
|
||||
Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab
|
||||
names quak by its full path. quak finds the saved session through `HOME`, which
|
||||
@@ -320,10 +320,10 @@ grid. It writes one file, also for a live photo. Without `--out`, the name is
|
||||
#### `quak backup <dir> [--json] [--verify]`
|
||||
|
||||
Backs up every file in the account to `<dir>`. The layout is described under
|
||||
"The backup directory". It downloads only the originals that are not yet in the
|
||||
backup, and it goes on past a file that fails. A file is downloaded once,
|
||||
however many collections hold it. An original the cache already holds is copied
|
||||
from there instead.
|
||||
"The backup directory". The command downloads only the originals that are not
|
||||
yet in the backup, and it goes on past a file that fails. A file is downloaded
|
||||
once, however many collections hold it. An original the cache already holds is
|
||||
copied from there instead.
|
||||
|
||||
Progress goes to stderr: one line per original fetched, and a `FAILED` line for
|
||||
each original or album that failed. The run ends with a summary of counts and,
|
||||
@@ -484,15 +484,6 @@ levels. From the quietest, the levels are fatal, error, warning, info, debug and
|
||||
trace. A run prints the lines at its own level and at every quieter one. The
|
||||
level is info unless `-v`, `-vv` or `-q` says otherwise.
|
||||
|
||||
Two things on stderr are not log lines. A command line quak cannot parse, such
|
||||
as a missing argument, an unknown option or `-q` with `-v`, is refused with an
|
||||
`error: …` message, and quak exits 1. `-q` does not hide the message. `quak`
|
||||
with no command prints the usage on stderr and exits 1.
|
||||
|
||||
`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's
|
||||
output, run `yarn --silent quak files --json` or
|
||||
`node dist/bin/quak.js files --json` instead.
|
||||
|
||||
- Fatal: why a command exits non-zero. A command that runs and exits non-zero
|
||||
logs at least one fatal line. A command that exits 1 because files or albums
|
||||
failed logs it last, saying how many failed and where to look.
|
||||
@@ -517,6 +508,15 @@ output, run `yarn --silent quak files --json` or
|
||||
one that exits non-zero prints only its fatal lines. The prompts of `login` are
|
||||
not log lines, and `-q` leaves them alone.
|
||||
|
||||
Two things on stderr are not log lines. A command line that quak cannot parse,
|
||||
such as a missing argument, an unknown option or `-q` with `-v`, is refused with
|
||||
an `error: …` message, and quak exits 1. `-q` does not hide the message. `quak`
|
||||
with no command prints the usage on stderr and exits 1.
|
||||
|
||||
`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's
|
||||
output, run `yarn --silent quak files --json` or
|
||||
`node dist/bin/quak.js files --json` instead.
|
||||
|
||||
With `--json`, a command's JSON on stdout takes the place of its summary on
|
||||
stderr. For `backup`, that is the `--- Backup complete ---` line, the counts and
|
||||
the `Failed files:` list. For `list-missing-thumbnails`, it is
|
||||
@@ -689,13 +689,13 @@ The JSON file beside an original holds:
|
||||
|
||||
Update times are in microseconds, as Ente records them.
|
||||
|
||||
The file also holds the machine-learning data Ente computed for the file, as
|
||||
The JSON also holds the machine-learning data Ente computed for the photo, as
|
||||
`mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding
|
||||
for it, a list of 512 numbers that describes what the photo shows. `mlData`
|
||||
holds everything but that embedding. In its place the file holds `keywords`,
|
||||
derived from the embedding as described under "Keywords". A file with no
|
||||
machine-learning data has neither field. A file whose data holds no usable
|
||||
embedding has no `keywords`.
|
||||
holds everything but the `clip` block, which holds that embedding. In its place
|
||||
the JSON holds `keywords`, derived from the embedding as described under
|
||||
"Keywords". A file with no machine-learning data has neither field. A file whose
|
||||
data holds no usable embedding has no `keywords`.
|
||||
|
||||
The backup waits for the library's fetch of machine-learning data to finish
|
||||
before it writes the JSON files. When a file's data cannot be fetched,
|
||||
@@ -748,11 +748,11 @@ derives a file's keywords the way the app sorts it. The categories are:
|
||||
|
||||
Each category has a query, and a text embedding of that query from Ente's CLIP
|
||||
text model. A file's score for a category is the cosine similarity of the file's
|
||||
embedding and the category's: a number from -1 to 1, higher the more alike the
|
||||
two embeddings are. The file falls in the category when its score reaches the
|
||||
category's minimum score. The keywords are the titles of the categories the file
|
||||
falls in, highest score first. A video never falls in a category marked "no
|
||||
videos", as in the app.
|
||||
embedding and the category's. That is a number from -1 to 1; the more alike the
|
||||
two embeddings are, the higher it is. The file falls in the category when its
|
||||
score reaches the category's minimum score. The keywords are the titles of the
|
||||
categories the file falls in, highest score first. A video never falls in a
|
||||
category marked "no videos", as in the app.
|
||||
|
||||
#### Collection directories
|
||||
|
||||
@@ -775,9 +775,10 @@ ignoring case, each gets its ID added: two albums named `Trip` become
|
||||
`Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match,
|
||||
ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG`
|
||||
and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON
|
||||
file, such as `Trip.json` beside `Trip`, also gets its ID added. A name too long
|
||||
for a file name, which most file systems limit to 255 bytes, gets its ID added
|
||||
too. It is then cut short before the ID, just far enough to fit. IDs never
|
||||
file, such as `Trip.json` beside `Trip`, also gets its ID added. A file name
|
||||
longer than 255 bytes, the limit on most file systems, gets its ID added too. So
|
||||
does a collection name longer than 250 bytes, which leaves room for `.json`.
|
||||
Either name is then cut short before the ID, just far enough to fit. IDs never
|
||||
change, so a name stays the same from run to run until such a clash appears or
|
||||
goes away.
|
||||
|
||||
@@ -834,7 +835,8 @@ Each entry holds:
|
||||
- `fileID` and `title`;
|
||||
- `classification`: the kind of failure;
|
||||
- `attempts`: how many runs in a row have failed the file;
|
||||
- `lastTriedAt`: when it was last tried, as a Unix time in milliseconds;
|
||||
- `lastTriedAt`: when the run that last tried it finished its refresh, as a Unix
|
||||
time in milliseconds;
|
||||
- `error`: the error, as quak printed it.
|
||||
|
||||
The kinds are:
|
||||
@@ -857,8 +859,8 @@ ends, whether it succeeded or failed.
|
||||
|
||||
The lock is a directory that
|
||||
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates.
|
||||
While the backup runs, the backup updates the directory's modification time
|
||||
every 5 seconds. A second backup of the directory fails at once: it prints
|
||||
While the backup runs, it updates the modification time of `backup.lock` every 5
|
||||
seconds. A second backup of `<dir>` fails at once: it prints
|
||||
`quak: another backup of <dir> is running` and exits 2. A run that is killed, or
|
||||
cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds
|
||||
without an update, the next run takes it over, so nobody has to remove it by
|
||||
@@ -962,9 +964,9 @@ of the same name apart.
|
||||
`files`. Each `<fileID>.json` holds the fields a backup's JSON beside an
|
||||
original holds, with these differences:
|
||||
|
||||
- `imageMetadata` is written only with `--exif`. The original of every file is
|
||||
then downloaded, videos included, and read as a backup reads it (see "The JSON
|
||||
beside each original").
|
||||
- `imageMetadata` is written only with `--exif`. The original of every file,
|
||||
videos included, is then downloaded and read for its EXIF, XMP and JPEG
|
||||
dimensions.
|
||||
- A file whose original holds no image metadata gets no `imageMetadata` field.
|
||||
- An original that cannot be downloaded or read gives `imageMetadataError`
|
||||
instead.
|
||||
@@ -1055,14 +1057,14 @@ still in flight.
|
||||
`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and
|
||||
`"precacheThumbnails"` or `"precacheOriginals"` for the two background
|
||||
downloads. Its `status` is `"started"` as the work starts and `"done"` as it
|
||||
ends. A refresh or a machine-learning fetch that fails ends with `"failed"` and
|
||||
the error message instead. In between, a refresh sends `"progress"` after each
|
||||
collection whose files it fetched: `fetched` is the number of collections
|
||||
fetched so far, and `total` the number it will fetch. A refresh fetches the
|
||||
files of each collection that is new or changed since the cache's copy, so on an
|
||||
empty cache it fetches those of every collection. The `"done"` event of a
|
||||
machine-learning fetch, and of each background download, also carries `fetched`:
|
||||
how many items it newly stored.
|
||||
ends. A refresh or a machine-learning fetch that fails ends with `"failed"`
|
||||
instead, with the message in `error`. In between, a refresh sends `"progress"`
|
||||
after each collection whose files it fetched: `fetched` is the number of
|
||||
collections fetched so far, and `total` the number it will fetch. A refresh
|
||||
fetches the files of each collection that is new or changed since the cache's
|
||||
copy, so on an empty cache it fetches those of every collection. The `"done"`
|
||||
event of a machine-learning fetch, and of each background download, also carries
|
||||
`fetched`: how many items it newly stored.
|
||||
|
||||
`lib.status()` returns a `LibraryStatus`. It holds the collection and file
|
||||
counts. `lastRefreshAt` is the time of the last refresh that succeeded, and
|
||||
@@ -1077,10 +1079,10 @@ cached, so the next fetch asks for it again. The reasons are kept in memory
|
||||
only.
|
||||
|
||||
`lib.close()` stops the background timer. It can be called more than once. A
|
||||
refresh in flight is left to finish, with its write to the cache. The promise
|
||||
`close()` returns resolves once that refresh, the machine-learning fetch and the
|
||||
background downloads already running have all finished. The cache directory can
|
||||
then be removed. After `close()`, `fresh()`, `freshCollections()`,
|
||||
refresh in flight is left to finish, with its write to the cache. `close()`
|
||||
returns a promise. It resolves once that refresh, the machine-learning fetch and
|
||||
the background downloads already running have all finished. The cache directory
|
||||
can then be removed. After `close()`, `fresh()`, `freshCollections()`,
|
||||
`freshFiles()` and `lib.backup()` reject with `the library is closed`.
|
||||
|
||||
### Default reads and fresh reads
|
||||
@@ -1381,9 +1383,9 @@ They differ because a download's length depends on the file and the link. A
|
||||
whole-transfer deadline short enough to catch a hung connection would cancel a
|
||||
large video on a slow link that is still making progress. The download deadline
|
||||
restarts each time bytes arrive, so a slow download runs as long as it keeps
|
||||
moving, and one that stalls is aborted after 60 seconds of silence. It covers
|
||||
the wait for the headers as well as the body. There is no limit on the total
|
||||
length of a download.
|
||||
moving, and one that stalls is aborted after 60 seconds of silence. The download
|
||||
deadline covers the wait for the headers as well as the body. There is no limit
|
||||
on the total length of a download.
|
||||
|
||||
A request that changes state is not blindly replayed. Every `POST` and `PUT` in
|
||||
the endpoint list under "HTTP API" goes through this rule. Verifying a second
|
||||
@@ -1456,8 +1458,8 @@ exported `beginLogin`, `unwrapAuth`, `decryptCollection`, `decryptFile`,
|
||||
`downloadFile` and `downloadThumbnail` need the same.
|
||||
|
||||
`client.logout()` clears the token and zeroes the key buffers in place. After
|
||||
it, every other method on that client throws. It does not contact the server, so
|
||||
the token stays valid there and in any saved snapshot. Call
|
||||
it, every other method on that client throws. `logout()` does not contact the
|
||||
server, so the token stays valid there and in any saved snapshot. Call
|
||||
`await client.logoutOnServer()` before `logout()` to end the session on the
|
||||
server.
|
||||
|
||||
|
||||
+10
-7
@@ -1,15 +1,18 @@
|
||||
// The CLI's log level, and the one function the commands write stderr lines
|
||||
// through.
|
||||
//
|
||||
// Every line quak prints on stderr is a log line at one of six levels, from
|
||||
// the quietest: fatal, error, warning, info, debug, trace. A run prints the
|
||||
// lines at its own level and at every quieter one. Its level is info unless
|
||||
// `-v` or `-q` asks for another (see "What quak prints" in the README). Every
|
||||
// command that exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||
// prints only fatal lines, never leaves a non-zero exit unexplained. An error
|
||||
// Every line quak logs goes to stderr, at one of six levels, from the
|
||||
// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
|
||||
// at its own level and at every quieter one. Its level is info unless `-v` or
|
||||
// `-q` asks for another (see "What quak prints" in the README). A command that
|
||||
// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||
// prints only fatal lines, never leaves such an exit unexplained. An error
|
||||
// line reports something that failed, such as a file or an album, and a
|
||||
// warning something that did not fail but needs the user's attention. stdout
|
||||
// carries only a command's output and is not a log.
|
||||
// carries a command's output and the prompts of `login`, and is not a log.
|
||||
// Two things on stderr are not log lines: the `error:` message for a
|
||||
// command line that quak cannot parse, and the usage printed when no command
|
||||
// is given.
|
||||
//
|
||||
// The library itself writes no lines: its layers report through callbacks, and
|
||||
// the CLI logs what they report. `logRequests` gives the API client options
|
||||
|
||||
@@ -328,7 +328,7 @@ describe("what the library reports", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// Each operation is named in plain words, as the README names it.
|
||||
// Each operation is named in plain words.
|
||||
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
||||
const { log, lines } = collect();
|
||||
const onProgress = logRefresh(log);
|
||||
|
||||
Reference in New Issue
Block a user