fix missing functionality in afero upstream #459

Open
opened 2026-11-19 07:30:13 +01:00 by sneak · 1 comment
Owner

investigate creating a branch off of https://github.com/spf13/afero master branch adding symlink support to the in-memory fs so that it's more useful in our tests.

i want:

  • a report on any other missing basic posix filesystem functionality from the in-memory fs
  • an estimate of how much time it would take to make this change in a way suitable for upstreaming
investigate creating a branch off of https://github.com/spf13/afero master branch adding symlink support to the in-memory fs so that it's more useful in our tests. i want: - a report on any other missing basic posix filesystem functionality from the in-memory fs - an estimate of how much time it would take to make this change in a way suitable for upstreaming
clawbot added the needs-decision label 2026-11-19 07:40:55 +01:00
Collaborator

No PR: this issue asks for a report and an estimate, and whether to build the afero branch is sneak's call. Labelled needs-decision.

Checked against afero master (d2bffea); vaultik pins v1.15.0. Each gap below was confirmed by a probe program against MemMapFs.

Symlinks

  • MemMapFs has no SymlinkIfPossible or ReadlinkIfPossible, so it does not implement Symlinker.
  • LstatIfPossible is plain Stat and reports that Lstat was not used. Nothing ever carries os.ModeSymlink.
  • afero has no interface at all for Lchown or for setting times on a link itself. vaultik's restore calls os.Lchown and unix.Lutimes directly, and its scanner calls os.Readlink directly. Symlinks in MemMapFs alone would not remove those calls.

Other missing POSIX behaviour in MemMapFs

  • Missing parent directories are created silently by Mkdir, Create and Rename. POSIX returns "no such file or directory".
  • A file can be created under a regular file (/reg/child). POSIX returns "not a directory".
  • Create on an existing directory replaces it with an empty file. The directory's children stay reachable by path.
  • Remove on a non-empty directory succeeds and leaves its children reachable. POSIX returns "directory not empty".
  • Rename succeeds when moving a directory into its own subdirectory, and when moving a file over a directory.
  • OpenFile with O_RDWR on a directory succeeds. POSIX returns "is a directory".
  • A handle opened O_WRONLY can be read.
  • Permissions are checked only on the file's own mode bits, ignoring uid and gid. Directory modes are never checked: creating a file inside a 0555 directory succeeds.
  • Seek to a negative offset succeeds.
  • There are no hard links, link counts or inode numbers. Sys() returns a syscall.Stat_t holding only uid, gid and size; Dev, Ino, Nlink, Mode and all times are zero. os.SameFile is always false.
  • Only the modification time is stored. Chtimes drops the access time, and there is no change time.
  • A directory's modification time does not change when entries are added or removed.
  • No umask is applied, and there are no FIFOs, sockets or device nodes.

Estimate for upstream-quality symlink support

  • Work: a path-resolution step that follows links in every path component, relative targets, a hop limit returning "too many levels of symbolic links", and the three link methods. Every existing method then goes through that step, following or not following the final link as os does. Tests run one table against OsFs in a temp directory and against MemMapFs, so the two must match. Windows paths need care, since upstream tests on Windows.
  • Size: about 400 to 600 lines of code and a similar amount of tests. About two to three working days.
  • Upstream acceptance is unpredictable. The upstream symlink request (https://github.com/spf13/afero/issues/258) has been open since 2020 with no reply. 67 pull requests are open, some since 2022. vaultik would carry a replace pointing at a fork until a merge, possibly for months.
  • The other gaps above are each small to fix. Upstream would want each as its own pull request, and several change behaviour that existing users' tests may rely on, such as parents being created automatically.

Question for sneak

Should vaultik build the afero symlink branch?

  • Fork and use it: fork afero (its home and account are yours to choose), add symlinks to MemMapFs, offer it upstream, and point vaultik's go.mod at the fork with replace. vaultik's tests then cover symlink restore without the real disk.
  • Upstream only: build and offer the branch upstream, but keep vaultik on released afero. Its symlink tests keep using a temp directory on disk until a release includes the change.
  • Not now: keep the current temp-directory tests.

Recommendation: Upstream only. vaultik's symlink paths already go through os, and it would need the missing Lchown and link-time interfaces too before in-memory tests could cover them.

Aside, for a separate issue if wanted: vaultik's deviceNumberFromStat in internal/snapshot/scanner.go says it returns false for the in-memory filesystem. On Linux it returns device 0 and true, because MemMapFs returns a syscall.Stat_t.

Model: opus-5-5

No PR: this issue asks for a report and an estimate, and whether to build the `afero` branch is sneak's call. Labelled `needs-decision`. Checked against `afero` `master` (`d2bffea`); vaultik pins `v1.15.0`. Each gap below was confirmed by a probe program against `MemMapFs`. **Symlinks** - `MemMapFs` has no `SymlinkIfPossible` or `ReadlinkIfPossible`, so it does not implement `Symlinker`. - `LstatIfPossible` is plain `Stat` and reports that `Lstat` was not used. Nothing ever carries `os.ModeSymlink`. - `afero` has no interface at all for `Lchown` or for setting times on a link itself. vaultik's restore calls `os.Lchown` and `unix.Lutimes` directly, and its scanner calls `os.Readlink` directly. Symlinks in `MemMapFs` alone would not remove those calls. **Other missing POSIX behaviour in `MemMapFs`** - Missing parent directories are created silently by `Mkdir`, `Create` and `Rename`. POSIX returns "no such file or directory". - A file can be created under a regular file (`/reg/child`). POSIX returns "not a directory". - `Create` on an existing directory replaces it with an empty file. The directory's children stay reachable by path. - `Remove` on a non-empty directory succeeds and leaves its children reachable. POSIX returns "directory not empty". - `Rename` succeeds when moving a directory into its own subdirectory, and when moving a file over a directory. - `OpenFile` with `O_RDWR` on a directory succeeds. POSIX returns "is a directory". - A handle opened `O_WRONLY` can be read. - Permissions are checked only on the file's own mode bits, ignoring uid and gid. Directory modes are never checked: creating a file inside a `0555` directory succeeds. - `Seek` to a negative offset succeeds. - There are no hard links, link counts or inode numbers. `Sys()` returns a `syscall.Stat_t` holding only uid, gid and size; `Dev`, `Ino`, `Nlink`, `Mode` and all times are zero. `os.SameFile` is always false. - Only the modification time is stored. `Chtimes` drops the access time, and there is no change time. - A directory's modification time does not change when entries are added or removed. - No umask is applied, and there are no FIFOs, sockets or device nodes. **Estimate for upstream-quality symlink support** - Work: a path-resolution step that follows links in every path component, relative targets, a hop limit returning "too many levels of symbolic links", and the three link methods. Every existing method then goes through that step, following or not following the final link as `os` does. Tests run one table against `OsFs` in a temp directory and against `MemMapFs`, so the two must match. Windows paths need care, since upstream tests on Windows. - Size: about 400 to 600 lines of code and a similar amount of tests. About two to three working days. - Upstream acceptance is unpredictable. The upstream symlink request (https://github.com/spf13/afero/issues/258) has been open since 2020 with no reply. 67 pull requests are open, some since 2022. vaultik would carry a `replace` pointing at a fork until a merge, possibly for months. - The other gaps above are each small to fix. Upstream would want each as its own pull request, and several change behaviour that existing users' tests may rely on, such as parents being created automatically. **Question for sneak** Should vaultik build the `afero` symlink branch? - **Fork and use it**: fork `afero` (its home and account are yours to choose), add symlinks to `MemMapFs`, offer it upstream, and point vaultik's `go.mod` at the fork with `replace`. vaultik's tests then cover symlink restore without the real disk. - **Upstream only**: build and offer the branch upstream, but keep vaultik on released `afero`. Its symlink tests keep using a temp directory on disk until a release includes the change. - **Not now**: keep the current temp-directory tests. Recommendation: **Upstream only**. vaultik's symlink paths already go through `os`, and it would need the missing `Lchown` and link-time interfaces too before in-memory tests could cover them. Aside, for a separate issue if wanted: vaultik's `deviceNumberFromStat` in `internal/snapshot/scanner.go` says it returns false for the in-memory filesystem. On Linux it returns device 0 and true, because `MemMapFs` returns a `syscall.Stat_t`. Model: opus-5-5
sneak was assigned by clawbot 2026-11-19 07:54:20 +01:00
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/vaultik#459