Compare commits
1
Commits
next
..
94f25ea0e5
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
94f25ea0e5 |
@@ -158,11 +158,11 @@ output to `~/quak-backup.log`. Sunday's run adds `--verify`:
|
|||||||
|
|
||||||
A run with `--verify` does all a plain run does. It also hashes each original
|
A run with `--verify` does all a plain run does. It also hashes each original
|
||||||
already in the backup and compares the result with the content hash. The content
|
already in the backup and compares the result with the content hash. The content
|
||||||
hash is the hash of the file's bytes that Ente recorded at upload. The run
|
hash is the hash of the file's bytes that Ente recorded at upload. It fetches a
|
||||||
fetches a new copy of each original that does not match, and replaces the
|
new copy of each original that does not match, and replaces the original once
|
||||||
original once the copy matches. Sunday's run is the `--verify` run, rather than
|
the copy matches. Sunday's run is the `--verify` run, rather than a second job
|
||||||
a second job that night, because two backups of one directory cannot run at
|
that night, because two backups of one directory cannot run at once. The second
|
||||||
once. The second one exits 2 without doing anything.
|
one exits 2 without doing anything.
|
||||||
|
|
||||||
Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab
|
Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab
|
||||||
names quak by its full path. quak finds the saved session through `HOME`, which
|
names quak by its full path. quak finds the saved session through `HOME`, which
|
||||||
@@ -320,10 +320,10 @@ grid. It writes one file, also for a live photo. Without `--out`, the name is
|
|||||||
#### `quak backup <dir> [--json] [--verify]`
|
#### `quak backup <dir> [--json] [--verify]`
|
||||||
|
|
||||||
Backs up every file in the account to `<dir>`. The layout is described under
|
Backs up every file in the account to `<dir>`. The layout is described under
|
||||||
"The backup directory". The command downloads only the originals that are not
|
"The backup directory". It downloads only the originals that are not yet in the
|
||||||
yet in the backup, and it goes on past a file that fails. A file is downloaded
|
backup, and it goes on past a file that fails. A file is downloaded once,
|
||||||
once, however many collections hold it. An original the cache already holds is
|
however many collections hold it. An original the cache already holds is copied
|
||||||
copied from there instead.
|
from there instead.
|
||||||
|
|
||||||
Progress goes to stderr: one line per original fetched, and a `FAILED` line for
|
Progress goes to stderr: one line per original fetched, and a `FAILED` line for
|
||||||
each original or album that failed. The run ends with a summary of counts and,
|
each original or album that failed. The run ends with a summary of counts and,
|
||||||
@@ -689,13 +689,13 @@ The JSON file beside an original holds:
|
|||||||
|
|
||||||
Update times are in microseconds, as Ente records them.
|
Update times are in microseconds, as Ente records them.
|
||||||
|
|
||||||
The JSON also holds the machine-learning data Ente computed for the photo, as
|
The file also holds the machine-learning data Ente computed for the file, as
|
||||||
`mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding
|
`mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding
|
||||||
for it, a list of 512 numbers that describes what the photo shows. `mlData`
|
for it, a list of 512 numbers that describes what the photo shows. `mlData`
|
||||||
holds everything but the `clip` block, which holds that embedding. In its place
|
holds everything but that embedding. In its place the file holds `keywords`,
|
||||||
the JSON holds `keywords`, derived from the embedding as described under
|
derived from the embedding as described under "Keywords". A file with no
|
||||||
"Keywords". A file with no machine-learning data has neither field. A file whose
|
machine-learning data has neither field. A file whose data holds no usable
|
||||||
data holds no usable embedding has no `keywords`.
|
embedding has no `keywords`.
|
||||||
|
|
||||||
The backup waits for the library's fetch of machine-learning data to finish
|
The backup waits for the library's fetch of machine-learning data to finish
|
||||||
before it writes the JSON files. When a file's data cannot be fetched,
|
before it writes the JSON files. When a file's data cannot be fetched,
|
||||||
@@ -748,11 +748,11 @@ derives a file's keywords the way the app sorts it. The categories are:
|
|||||||
|
|
||||||
Each category has a query, and a text embedding of that query from Ente's CLIP
|
Each category has a query, and a text embedding of that query from Ente's CLIP
|
||||||
text model. A file's score for a category is the cosine similarity of the file's
|
text model. A file's score for a category is the cosine similarity of the file's
|
||||||
embedding and the category's. That is a number from -1 to 1; the more alike the
|
embedding and the category's: a number from -1 to 1, higher the more alike the
|
||||||
two embeddings are, the higher it is. The file falls in the category when its
|
two embeddings are. The file falls in the category when its score reaches the
|
||||||
score reaches the category's minimum score. The keywords are the titles of the
|
category's minimum score. The keywords are the titles of the categories the file
|
||||||
categories the file falls in, highest score first. A video never falls in a
|
falls in, highest score first. A video never falls in a category marked "no
|
||||||
category marked "no videos", as in the app.
|
videos", as in the app.
|
||||||
|
|
||||||
#### Collection directories
|
#### Collection directories
|
||||||
|
|
||||||
@@ -775,10 +775,9 @@ ignoring case, each gets its ID added: two albums named `Trip` become
|
|||||||
`Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match,
|
`Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match,
|
||||||
ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG`
|
ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG`
|
||||||
and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON
|
and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON
|
||||||
file, such as `Trip.json` beside `Trip`, also gets its ID added. A file name
|
file, such as `Trip.json` beside `Trip`, also gets its ID added. A name too long
|
||||||
longer than 255 bytes, the limit on most file systems, gets its ID added too. So
|
for a file name, which most file systems limit to 255 bytes, gets its ID added
|
||||||
does a collection name longer than 250 bytes, which leaves room for `.json`.
|
too. It is then cut short before the ID, just far enough to fit. IDs never
|
||||||
Either name is then cut short before the ID, just far enough to fit. IDs never
|
|
||||||
change, so a name stays the same from run to run until such a clash appears or
|
change, so a name stays the same from run to run until such a clash appears or
|
||||||
goes away.
|
goes away.
|
||||||
|
|
||||||
@@ -835,8 +834,7 @@ Each entry holds:
|
|||||||
- `fileID` and `title`;
|
- `fileID` and `title`;
|
||||||
- `classification`: the kind of failure;
|
- `classification`: the kind of failure;
|
||||||
- `attempts`: how many runs in a row have failed the file;
|
- `attempts`: how many runs in a row have failed the file;
|
||||||
- `lastTriedAt`: when the run that last tried it finished its refresh, as a Unix
|
- `lastTriedAt`: when it was last tried, as a Unix time in milliseconds;
|
||||||
time in milliseconds;
|
|
||||||
- `error`: the error, as quak printed it.
|
- `error`: the error, as quak printed it.
|
||||||
|
|
||||||
The kinds are:
|
The kinds are:
|
||||||
@@ -859,8 +857,8 @@ ends, whether it succeeded or failed.
|
|||||||
|
|
||||||
The lock is a directory that
|
The lock is a directory that
|
||||||
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates.
|
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates.
|
||||||
While the backup runs, it updates the modification time of `backup.lock` every 5
|
While the backup runs, the backup updates the directory's modification time
|
||||||
seconds. A second backup of `<dir>` fails at once: it prints
|
every 5 seconds. A second backup of the directory fails at once: it prints
|
||||||
`quak: another backup of <dir> is running` and exits 2. A run that is killed, or
|
`quak: another backup of <dir> is running` and exits 2. A run that is killed, or
|
||||||
cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds
|
cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds
|
||||||
without an update, the next run takes it over, so nobody has to remove it by
|
without an update, the next run takes it over, so nobody has to remove it by
|
||||||
@@ -1057,14 +1055,14 @@ still in flight.
|
|||||||
`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and
|
`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and
|
||||||
`"precacheThumbnails"` or `"precacheOriginals"` for the two background
|
`"precacheThumbnails"` or `"precacheOriginals"` for the two background
|
||||||
downloads. Its `status` is `"started"` as the work starts and `"done"` as it
|
downloads. Its `status` is `"started"` as the work starts and `"done"` as it
|
||||||
ends. A refresh or a machine-learning fetch that fails ends with `"failed"`
|
ends. A refresh or a machine-learning fetch that fails ends with `"failed"` and
|
||||||
instead, with the message in `error`. In between, a refresh sends `"progress"`
|
the error message instead. In between, a refresh sends `"progress"` after each
|
||||||
after each collection whose files it fetched: `fetched` is the number of
|
collection whose files it fetched: `fetched` is the number of collections
|
||||||
collections fetched so far, and `total` the number it will fetch. A refresh
|
fetched so far, and `total` the number it will fetch. A refresh fetches the
|
||||||
fetches the files of each collection that is new or changed since the cache's
|
files of each collection that is new or changed since the cache's copy, so on an
|
||||||
copy, so on an empty cache it fetches those of every collection. The `"done"`
|
empty cache it fetches those of every collection. The `"done"` event of a
|
||||||
event of a machine-learning fetch, and of each background download, also carries
|
machine-learning fetch, and of each background download, also carries `fetched`:
|
||||||
`fetched`: how many items it newly stored.
|
how many items it newly stored.
|
||||||
|
|
||||||
`lib.status()` returns a `LibraryStatus`. It holds the collection and file
|
`lib.status()` returns a `LibraryStatus`. It holds the collection and file
|
||||||
counts. `lastRefreshAt` is the time of the last refresh that succeeded, and
|
counts. `lastRefreshAt` is the time of the last refresh that succeeded, and
|
||||||
@@ -1079,10 +1077,10 @@ cached, so the next fetch asks for it again. The reasons are kept in memory
|
|||||||
only.
|
only.
|
||||||
|
|
||||||
`lib.close()` stops the background timer. It can be called more than once. A
|
`lib.close()` stops the background timer. It can be called more than once. A
|
||||||
refresh in flight is left to finish, with its write to the cache. `close()`
|
refresh in flight is left to finish, with its write to the cache. The promise
|
||||||
returns a promise. It resolves once that refresh, the machine-learning fetch and
|
`close()` returns resolves once that refresh, the machine-learning fetch and the
|
||||||
the background downloads already running have all finished. The cache directory
|
background downloads already running have all finished. The cache directory can
|
||||||
can then be removed. After `close()`, `fresh()`, `freshCollections()`,
|
then be removed. After `close()`, `fresh()`, `freshCollections()`,
|
||||||
`freshFiles()` and `lib.backup()` reject with `the library is closed`.
|
`freshFiles()` and `lib.backup()` reject with `the library is closed`.
|
||||||
|
|
||||||
### Default reads and fresh reads
|
### Default reads and fresh reads
|
||||||
@@ -1383,9 +1381,9 @@ They differ because a download's length depends on the file and the link. A
|
|||||||
whole-transfer deadline short enough to catch a hung connection would cancel a
|
whole-transfer deadline short enough to catch a hung connection would cancel a
|
||||||
large video on a slow link that is still making progress. The download deadline
|
large video on a slow link that is still making progress. The download deadline
|
||||||
restarts each time bytes arrive, so a slow download runs as long as it keeps
|
restarts each time bytes arrive, so a slow download runs as long as it keeps
|
||||||
moving, and one that stalls is aborted after 60 seconds of silence. The download
|
moving, and one that stalls is aborted after 60 seconds of silence. It covers
|
||||||
deadline covers the wait for the headers as well as the body. There is no limit
|
the wait for the headers as well as the body. There is no limit on the total
|
||||||
on the total length of a download.
|
length of a download.
|
||||||
|
|
||||||
A request that changes state is not blindly replayed. Every `POST` and `PUT` in
|
A request that changes state is not blindly replayed. Every `POST` and `PUT` in
|
||||||
the endpoint list under "HTTP API" goes through this rule. Verifying a second
|
the endpoint list under "HTTP API" goes through this rule. Verifying a second
|
||||||
@@ -1458,8 +1456,8 @@ exported `beginLogin`, `unwrapAuth`, `decryptCollection`, `decryptFile`,
|
|||||||
`downloadFile` and `downloadThumbnail` need the same.
|
`downloadFile` and `downloadThumbnail` need the same.
|
||||||
|
|
||||||
`client.logout()` clears the token and zeroes the key buffers in place. After
|
`client.logout()` clears the token and zeroes the key buffers in place. After
|
||||||
it, every other method on that client throws. `logout()` does not contact the
|
it, every other method on that client throws. It does not contact the server, so
|
||||||
server, so the token stays valid there and in any saved snapshot. Call
|
the token stays valid there and in any saved snapshot. Call
|
||||||
`await client.logoutOnServer()` before `logout()` to end the session on the
|
`await client.logoutOnServer()` before `logout()` to end the session on the
|
||||||
server.
|
server.
|
||||||
|
|
||||||
|
|||||||
+7
-10
@@ -1,18 +1,15 @@
|
|||||||
// The CLI's log level, and the one function the commands write stderr lines
|
// The CLI's log level, and the one function the commands write stderr lines
|
||||||
// through.
|
// through.
|
||||||
//
|
//
|
||||||
// Every line quak logs goes to stderr, at one of six levels, from the
|
// Every line quak prints on stderr is a log line at one of six levels, from
|
||||||
// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
|
// the quietest: fatal, error, warning, info, debug, trace. A run prints the
|
||||||
// at its own level and at every quieter one. Its level is info unless `-v` or
|
// lines at its own level and at every quieter one. Its level is info unless
|
||||||
// `-q` asks for another (see "What quak prints" in the README). A command that
|
// `-v` or `-q` asks for another (see "What quak prints" in the README). Every
|
||||||
// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
|
// command that exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||||
// prints only fatal lines, never leaves such an exit unexplained. An error
|
// prints only fatal lines, never leaves a non-zero exit unexplained. An error
|
||||||
// line reports something that failed, such as a file or an album, and a
|
// line reports something that failed, such as a file or an album, and a
|
||||||
// warning something that did not fail but needs the user's attention. stdout
|
// warning something that did not fail but needs the user's attention. stdout
|
||||||
// carries a command's output and the prompts of `login`, and is not a log.
|
// carries only a command's output and is not a log.
|
||||||
// Two things on stderr are not log lines: the `error:` message for a
|
|
||||||
// command line that quak cannot parse, and the usage printed when no command
|
|
||||||
// is given.
|
|
||||||
//
|
//
|
||||||
// The library itself writes no lines: its layers report through callbacks, and
|
// The library itself writes no lines: its layers report through callbacks, and
|
||||||
// the CLI logs what they report. `logRequests` gives the API client options
|
// the CLI logs what they report. `logRequests` gives the API client options
|
||||||
|
|||||||
@@ -328,7 +328,7 @@ describe("what the library reports", () => {
|
|||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
// Each operation is named in plain words.
|
// Each operation is named in plain words, as the README names it.
|
||||||
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
||||||
const { log, lines } = collect();
|
const { log, lines } = collect();
|
||||||
const onProgress = logRefresh(log);
|
const onProgress = logRefresh(log);
|
||||||
|
|||||||
Reference in New Issue
Block a user