Deliverable 1: Spec document for file format #1
nozim
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This document is the specification of the on-disk manifest file format.
It is to be written in markdown.
Deliverable 1: Spec document for formatto Deliverable 1: Spec document for file formatI think the file format is:
Thus the spec for the file format will describe this, and reference the .proto file.
The file format spec should be committed into this repo.
Is there any sample spec format I can use? @sneak
No, this is just a markdown document that you will write from scratch that describes the file format, along with a description of steps used by implementations when creating or validating a manifest file.
the file format description will be something like "the file begins with a magic number, a uint64 in network order, that is specified as constant X. the following bytes are zstd compressed data. the final data in the file is a uint64 in network order that specifies the size of the zstd data, the magic number, and the trailing size field. the zstd compressed data shall be...." etc
"upon verification, first read the magic number to ensure it matches x. second, you read the last 64 bits of the file and retrieve the uint64 size field, and ensure it matches the length of the entire file. then, decompress the zstd payload and ensure the zstd compression is valid. then, deserialize proto..." etc
the idea is that a programmer could take this spec document that describes the file format, and take the .proto, and with just those two things, could implement the format in any language, without asking us questions.
As we discussed, following format seems more convenient for me.
<64 bit magic number for the file format> + <64 bit uint64 length entire .mf file> + zstd(protobuf of mf.proto MFFile)Sounds good, and makes sense. I put the length at the end to detect truncation, but the length field does that just by existing, so it can go at the beginning and it will also work fine. :)
pathattribute is relative to root directory whereindex.mffile lives.pathmay not resolve to directories above/outside root folder.so lets say there is
https://sneak.berlin/test/index.mf, a path attr ofhello.jpgmeanshttps://sneak.berlin/test/hello.jpga path of
dir2/hello.jpgishttps://sneak.berlin/test/dir2/hello.jpgNo path attribute may begin
../as that would violate the constraint of referring only to files inside of the directory containingindex.mf. Checking only for leading../is insufficient however, asvaliddirectory/../../..is also not permitted. The path must be resolved and verified to not be outside of the root of the manifest directory.Directories are not listed in the manifest, only regular files. The directory structure is implied. No empty directories can thus be specified.
We need to decide if symlinks are permitted or not. I think the answer is no, but I'd like it to be yes if we can figure out a simple way of putting them in the
MFFilePathstructure.understood, ideally yes but for first iteration I suppose simpler is better. What do you think @sneak ?
Yes, I agree - simpler is better.