Good evening Ted, You bring up good points, and I'll clarify. The original prompt for me was that there are tons of network queue, distributed lock, and network file sharing software. On a single system, all of these have been solved through simple file idioms like named pipes, file locking, and basic file operations. Wouldn't it be nice to have all those idioms transparently work for single systems as in a distributed manner? And I'd like to avoid a single point of failure while better understanding and addressing concerns system administrators have with NFS. You're right, I don't have many details about this prompt. There's not much I can do about my ignorance without trying. There are a set of characteristics which I would like to preserve. I would like connected mount points using a consensus algorithm to preserve user facing guarantees specified by POSIX.1 such that an operation applied to one mount point is applied synchronously to all connected mount points. Separate operations are asynchronous and it is the responsibility of the user or program to handle these cases appropriately, as would be the case if operating strictly locally with similar concurrency conditions. It should be accessible in some way without special permissions and require secure operation via key exchange and encrypted messaging. This description is still malformed. I want to do a PhD on the subject, but I haven't done that yet. Before I step into that thicket, it looks like my trivial case is a "pass-through" file system. So I need to learn and implement that. It seems like a pass-through would just forward all function calls. That's non-trivial and I need to investigate security hooks because they might meet my needs, overlayfs because it may allow locking the underlying directory in a serviceable way, and much of the work referenced by Amir. On Mon, Jan 9, 2023 at 7:48 PM Theodore Ts'o <tytso@xxxxxxx> wrote: > > On Sun, Jan 08, 2023 at 11:46:38PM -0500, Anadon wrote: > > I am looking into implementing a distributed RAFT filesystem for > > reasons. Before this, I want what is in effect a simple pass-through > > filesystem. Something which just takes in calls to open, read, close, > > etc and forwards them to a specified mounted* filesystem. Hopefully > > through FUSE before jumping straight into kernel development. > > > > Doing this and having files appear in two places by calling `mount()` > > then calling the (potentially) userland functions to the mapped file > > by changing the file path is a way to technically accomplish > > something. This has the effect of the files being accessible in two > > locations. > > I fon't quite understand *why* you want to implement a passthrough > filesystem in terms of how it relates to creating a distributed RAFT > file system. > > Files (and indeed, entire directory hierarchies) being accessible in > two locations is not a big deal; this can be done using a bind mount > without needing any kernel code. > > And if the question is how to deal with files that can be modified > externally, from a system elsewhere in the local network (or > cluster/cell), well, all networked file systems such as NFS, et.al., > do this. If a networked file system knows that a particular file has > been modified, it can easily invalidate the local page cache copies of > the file. Now, if that file is currently being used, and is being > mmap'ed into some process's address space, perhaps as the read-only > text (code) segment, what the semantics should be if it is modified > remotely out from under that process --- or whether the file should be > allowed to be modified at all if it is being used in certain ways, is > a semantic/design/policy question you need to answer before we can > talk about how this might be implemented. > > > The problems start where the underlying filesystem won't > > notify my passthrough layer if there are changes made. > > Now, how an underlying file system might notify your passthrough layer > will no doubt be comletely different from how a distributed/networked > file system will notify the node-level implementation of that file > system. So I'm not at all sure how this is relevant for your use > case. (And if you want a file to appear in two places at the same > time, just make that file *be* the same file in two places via a bind > mount.) > > > What would be better is to have some struct with all > > relevant function pointers and data accessible. That sounds like > > adding syscalls `int mount2(const char* device, ..., struct > > return_fs_interface)` and `int umuont3(struct return_fs_interface)`. > > I don't understand what you want to do here. What is going to be in > this "struct return_fs_interface"? Function pointers? Do you really > want to have kernel code calling userspace functions? Uh, that's a > really bad idea. You should take a closer look at how the FUSE > kernel/usersace interface works, if the goal is to do something like > this via a userspace FUSE-like scheme. > > > I have looked at `fsopen(...)` as an alternative, but it still does > > not meet my use case. Another way would be to compile in every > > filesystem driver but this just seems downright mad. Is there a good > > option I have overlooked? Am I even asking in the right place? > > I'm not sure what "compiling in efvery filesystem driver" is trying to > accomplish. Compiling into *what*? For what purpose? > > I'm not sure you are asking the right questions. It might be better > to say in more detail what are the functional requirements what it is > you are trying to achieve, before asking people to evaluate potential > implementation approaches. > > Best regards, > > - Ted
