Manifest file generator and checker.
Given a plain URL, there is no standard way to safely and programmatically download everything “under” that URL path.
wget -r can traverse directory listings if they’re enabled, but every server has a different format, and this does not verify cryptographic integrity of the files, or enable them to be fetched using a different protocol other than HTTP/s.
Currently, the solution that people are using are sidecar files in the format of
SHASUMS checksum files, as well as a
SHASUMS.asc PGP detached signature. This is not checksum-algorithm-agnostic and the sidecar file is not always consistently named.
Real issues I face:
A standard, a manifest file format, and a tool for generating same.
The manifest file would be called
index.mf, and the tool for generating such would be called
The manifest file would do several important things:
https://example.com/downloadpackage/one could fetch
https://example.com/downloadpackage/index.mfto enumerate the full directory listing.
index.mfdoes not imply publishing on ipfs at that time
mfer gen .
mfer check .
mfer fetch https://example.com/stuff/
/stuff/index.mfand downloads all files listed in manifest, optionally resuming any that already exist locally, and assures cryptographic integrity of downloaded files.
https://example.com/manifestdirectory/index.mf, downloads and checksums all files, resumes any that exist locally already)
mfer fetch https://example.com/manifestdirectory/
mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2to assert in the URL which PGP signing key should be used in the manifest, so that shared URLs have a cryptographic trust root
berlin.sneak.app.slideshowkey that maps to a json slideshow config listing what image paths to show, and for how long, and in what order
I’d like to be able to put a bunch of images into a directory, generate a manifest, and then point a slideshow client (such as an ambient display, or a react app with the target directory in a query string arg) at that statically hosted directory, and have it discover the full list of images available at that URL.
I’d like to be able to download a whole tree of files available via HTTP resumably by either HTTP or IPFS/BitTorrent without a .torrent file.
I use filesystems that don’t include data checksums, and I would like a cryptographically signed checksum file so that I can later verify that a set of archive files have not been modified, none are missing, and that the checksums have not been altered in storage by a second party.
I would like to be able to plug in a hard drive or flash drive and, if there is an
index.mf in the root, automatically detect missing/corrupted files, regardless of filesystem format.
email@example.com with your desired username for an account on this Gitea instance.
I am currently interested in hiring a contractor skilled with the Go standard library interfaces to specify this tool in full and develop a prototype implementation.