sneak/quak
1
0
Fork 0
Code Issues 8 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Wire content-integrity hash comparison once the hash construction is confirmed #68

New Issue
Closed
opened 2026-09-22 18:05:32 +02:00 by clawbot · 3 comments
clawbot commented 2026-09-22 18:05:32 +02:00
Collaborator
Copy Link
Copy Source

Follow-up from the content/thumbnail cache (#46).

The content cache currently guarantees integrity through the authenticated streaming
decrypt — every chunk is authenticated (poly1305) and the file is renamed into place only
when the secretstream ends cleanly on TAG_FINAL — plus a stored-non-empty check. The
hash comparison the design sketched is deferred, for two grounded reasons:

  • The exact content-hash construction is unconfirmed; the design
    (#36) itself preconditions relying on it on
    confirming the construction (algorithm; the live-photo combined case), and the only hash
    fixture in the repo is a placeholder.
  • FileMetadata.size is the server's ENCRYPTED object size, not the decrypted content
    length this layer produces, so the design's "fall back to fileSize" comparison would
    always mismatch as written.

Done when

  • The exact content-hash construction is confirmed against a real fixture (algorithm; the
    live-photo combined case).
  • The content cache compares the decrypted content hash to FileMetadata.hash when
    present and stores it; the fallback (if any) uses a correct decrypted-length measure,
    not the encrypted size field.
  • Tests cover a hash match, a mismatch (rejected), and the hash-absent path.

Low priority: the authenticated decrypt already provides strong integrity; this adds a
second, content-level check once its inputs are trustworthy.

Model: opus-4-8

Follow-up from the content/thumbnail cache (https://git.eeqj.de/sneak/quak/issues/46). The content cache currently guarantees integrity through the authenticated streaming decrypt — every chunk is authenticated (poly1305) and the file is renamed into place only when the secretstream ends cleanly on `TAG_FINAL` — plus a stored-non-empty check. The hash comparison the design sketched is deferred, for two grounded reasons: - The exact content-hash construction is unconfirmed; the design (https://git.eeqj.de/sneak/quak/issues/36) itself preconditions relying on it on confirming the construction (algorithm; the live-photo combined case), and the only hash fixture in the repo is a placeholder. - `FileMetadata.size` is the server's ENCRYPTED object size, not the decrypted content length this layer produces, so the design's "fall back to fileSize" comparison would always mismatch as written. ## Done when - The exact content-hash construction is confirmed against a real fixture (algorithm; the live-photo combined case). - The content cache compares the decrypted content hash to `FileMetadata.hash` when present and stores it; the fallback (if any) uses a correct decrypted-length measure, not the encrypted `size` field. - Tests cover a hash match, a mismatch (rejected), and the hash-absent path. Low priority: the authenticated decrypt already provides strong integrity; this adds a second, content-level check once its inputs are trustworthy. Model: opus-4-8
clawbot commented 2026-09-23 01:54:05 +02:00
Author
Collaborator
Copy Link
Copy Source

Finding: how the content hash is built

The server does not build or check this hash. hash is a field inside the file's end-to-end encrypted metadata, which the server stores without being able to read. The uploading client defines it. Source is the upstream ente monorepo at commit af32f237295536a3aef6b57698d9d5ad204576b4 (2026-09-22):

  • Algorithm: unkeyed BLAKE2b with a 64-byte output (crypto_generichash, BYTES_MAX), fed with the original file's plaintext bytes in upload chunks, then standard base64. See web/packages/base/crypto/libsodium.ts (chunkHashInit/Update/Final) and computeHash in web/packages/gallery/services/upload/upload-service.ts.
  • Live photos: hash is <imageHash>:<videoHash>, each hashed separately over its own original bytes. The stored object is a ZIP of the two parts, so hashing the decrypted object never matches. The cache would have to unpack the ZIP and hash each part. Older clients wrote imageHash and videoHash as separate fields with no hash; metadataHash in web/packages/media/file-metadata.ts joins them the same way.
  • Missing hash: files from very old clients may have none. Upstream treats that as normal, not as an error.

What this means for the issue: the construction is confirmed from source, but "against a real fixture" is still open. The check also only confirms that the bytes match what the uploading client declared. It adds nothing against a dishonest uploader beyond what the authenticated decrypt already gives. This stays low priority. I recommend a unit once a real image and live-photo fixture with known hashes is in the repo.

Model: opus-5-5

## Finding: how the content hash is built The server does not build or check this hash. `hash` is a field inside the file's end-to-end encrypted metadata, which the server stores without being able to read. The uploading client defines it. Source is the upstream ente monorepo at commit `af32f237295536a3aef6b57698d9d5ad204576b4` (2026-09-22): - Algorithm: unkeyed BLAKE2b with a 64-byte output (`crypto_generichash`, `BYTES_MAX`), fed with the original file's plaintext bytes in upload chunks, then standard base64. See `web/packages/base/crypto/libsodium.ts` (`chunkHashInit`/`Update`/`Final`) and `computeHash` in `web/packages/gallery/services/upload/upload-service.ts`. - Live photos: `hash` is `<imageHash>:<videoHash>`, each hashed separately over its own original bytes. The stored object is a ZIP of the two parts, so hashing the decrypted object never matches. The cache would have to unpack the ZIP and hash each part. Older clients wrote `imageHash` and `videoHash` as separate fields with no `hash`; `metadataHash` in `web/packages/media/file-metadata.ts` joins them the same way. - Missing hash: files from very old clients may have none. Upstream treats that as normal, not as an error. What this means for the issue: the construction is confirmed from source, but "against a real fixture" is still open. The check also only confirms that the bytes match what the uploading client declared. It adds nothing against a dishonest uploader beyond what the authenticated decrypt already gives. This stays low priority. I recommend a unit once a real image and live-photo fixture with known hashes is in the repo. Model: opus-5-5
clawbot commented 2026-09-23 05:27:25 +02:00
Author
Collaborator
Copy Link
Copy Source

Definition of done (branch next2)

The construction is confirmed from the upstream client source (#68 (comment)), so this is no longer blocked.

Decision: a mismatch is a failure, never a silent accept. A backup tool that stores a file whose content differs from what the uploader recorded is unfit for purpose. So when the hash does not match, the file is not stored, the download fails with an error naming the file ID, and the backup records it in its failure record like any other failed file.

  1. decryptFile (src/model/decrypt.ts:118) also reads the older imageHash and videoHash fields. It produces the expected hash the way upstream metadataHash does: hash if present; otherwise, for a live photo with both fields, <imageHash>:<videoHash>; otherwise none. A field that is not a string counts as absent.
  2. The plain file path (image, video) hashes the decrypted bytes as they are written: unkeyed BLAKE2b with a 64-byte output through libsodium, standard base64 with padding. The file is renamed into place only if the hash matches. This happens in the download writer shared by quak get, the content cache (src/library/content.ts:147) and backup, so all three check it.
  3. Live photos: the stored object is a ZIP of the image and the video. Hash each entry's bytes and compare to <imageHash>:<videoHash>. Use one widely used, maintained ZIP-reading library, name it in the PR body, and prefer the smallest well-known one. No hand-written ZIP parser.
  4. No expected hash: store the file unchecked, as upstream does for files from very old clients. This is not an error.
  5. The integrity comment at the top of src/library/content.ts (lines 15-23) says what is now checked.
  6. Tests:
    • A known-answer test pins the algorithm and encoding, for example the BLAKE2b-512 test vector for "abc" from RFC 7693, in base64.
    • A match is stored.
    • A mismatch is rejected: nothing is stored, and the error names the file.
    • A missing hash is stored.
    • A live photo ZIP built in the test is checked both ways.
    • Legacy imageHash/videoHash metadata produces the joined expected hash.
  7. make check green; TODO.md updated in the same commit.

Out of scope: thumbnails, which carry no hash, and any check against a real account's files.

Model: opus-5-5

## Definition of done (branch `next2`) The construction is confirmed from the upstream client source (https://git.eeqj.de/sneak/quak/issues/68#issuecomment-100442), so this is no longer blocked. **Decision: a mismatch is a failure, never a silent accept.** A backup tool that stores a file whose content differs from what the uploader recorded is unfit for purpose. So when the hash does not match, the file is not stored, the download fails with an error naming the file ID, and the backup records it in its failure record like any other failed file. 1. `decryptFile` (`src/model/decrypt.ts:118`) also reads the older `imageHash` and `videoHash` fields. It produces the expected hash the way upstream `metadataHash` does: `hash` if present; otherwise, for a live photo with both fields, `<imageHash>:<videoHash>`; otherwise none. A field that is not a string counts as absent. 2. The plain file path (image, video) hashes the decrypted bytes as they are written: unkeyed BLAKE2b with a 64-byte output through libsodium, standard base64 with padding. The file is renamed into place only if the hash matches. This happens in the download writer shared by `quak get`, the content cache (`src/library/content.ts:147`) and backup, so all three check it. 3. Live photos: the stored object is a ZIP of the image and the video. Hash each entry's bytes and compare to `<imageHash>:<videoHash>`. Use one widely used, maintained ZIP-reading library, name it in the PR body, and prefer the smallest well-known one. No hand-written ZIP parser. 4. No expected hash: store the file unchecked, as upstream does for files from very old clients. This is not an error. 5. The integrity comment at the top of `src/library/content.ts` (lines 15-23) says what is now checked. 6. Tests: - A known-answer test pins the algorithm and encoding, for example the BLAKE2b-512 test vector for `"abc"` from RFC 7693, in base64. - A match is stored. - A mismatch is rejected: nothing is stored, and the error names the file. - A missing hash is stored. - A live photo ZIP built in the test is checked both ways. - Legacy `imageHash`/`videoHash` metadata produces the joined expected hash. 7. `make check` green; `TODO.md` updated in the same commit. Out of scope: thumbnails, which carry no hash, and any check against a real account's files. Model: opus-5-5
clawbot commented 2026-09-23 05:32:20 +02:00
Author
Collaborator
Copy Link
Copy Source

Implemented in #98: every downloaded original with a recorded hash is checked (BLAKE2b-512, base64; live photos per part via fflate), a mismatch stores nothing and fails naming the file ID, and a missing hash is stored unchecked.

Model: opus-5-5

Implemented in https://git.eeqj.de/sneak/quak/pulls/98: every downloaded original with a recorded hash is checked (BLAKE2b-512, base64; live photos per part via `fflate`), a mismatch stores nothing and fails naming the file ID, and a missing hash is stored unchecked. Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
merge-ready
Review passed; ready to merge
 needs-checks
CI / make check is not green on this PR
 needs-rebase
PR conflicts with main and must be rebased
 needs-review
PR is awaiting adversarial review
 needs-rework
Review failed; PR needs changes
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/quak#68