> On Feb 1, 2018, at 8:51 AM, Boaz Harrosh <boazh@xxxxxxxxxx> wrote: > > Sorry I'm resending because this mail never appeared on linux-fsdevel@ML. I'm assuming because it had an attachment. So here is a URL on the web of the slides attached: > http://linuxplumbersconf.org/2017/ocw//system/presentations/4703/original/ZUFS_for_LinuxPlumbers_LPC.pptx > ~~~~~ > > Hi Fellow coders > > Last plumbers I have introduced a new project ZUFS, > (See attached slides of what was presented then. Specially the POC benchmarks) > > ZUFS - stands for Zero-copy User-mode FS > - It is geared towards true zero copy end to end of both data and meta data. > - It is geared towards very *low latency*, very high CPU locality, lock-less parallelism. > - Synchronous operations > - Numa awareness > > Short description: > ZUFS is a from scratch implementation of a filesystem-in-user-space, which tries to address the above goals. from the get go it is aimed for pmem based FSs. But can easily support other type of FSs that can utilize x10 latency and parallelism improvements. The novelty of this project is that the interface is designed with a modern multi-core NUMA machine in mind down to the ABI, so to reach these goals. > > Not only FSs need apply, also any kind of user-mode Server can set up a pseudo filesystem and communicate with application via virtual files. These can then benefit from zero copy low-latency communication directly to/from application buffers. Or Application mmap direct Server resources. As long as it looks like a file system to the Kernel. > > During Plumbers a few people and companies showed deep interest in this project. Mainly because it introduces a new kind of synchronous, low-latency communication between many application and a user-mode Server, as the FS example above. (Perhaps this pattern could be extended to other areas but this is out of scope for now.) > > Since then I have been banging on some real implementation and very soon (Once Netapp legal approves the code, should be done next week) will release a first draft RFC for review. (I will be sending some design notes soon). > > Current status is that we have couple trivial filesystem implementations and together with the Kernel module the UM-Server and the FSs User-mode pluggin can actually pass a good bunch of xfstests quick run. (Still working on Stability) > > I would like to present current (to LSF time) implementation and status. But mainly to consult with the Kernel Gurus at LSF/MM how to HARDEN and secure this very ambitious project. Specially as there are couple of mm and scheduler patches that will need to be submitted along with this project. > > CC'ed are people that stated interest in the past about ZUFS. Sorry if I forgot some. Please comment if you are interested to talk about this on LSF/MM > > Just to get some points across as I said this project is all about performance and low latency. Here below are a POC results I have run (Repeated from the attached slides) > > In Kernel FS ZUFS FUSE > Threads Op/s Lat (us) Op/s Lat [us] Op/s Lat [us] > 1 388361 2.271589 200799 4.6 71820 13.5 > 2 635115 2.604376 314321 5.9 148083 13.1 > 4 1260307 2.626361 565574 6.6 212133 18.3 > 8 2744963 2.485292 1113138 6.6 209799 37.6 > 12 2126945 5.020506 1598451 6.8 201689 58.7 > 18 4350995 3.386433 1648689 7.8 174823 101.8 > 24 4211180 4.784997 1702285 8 149413 159 > 36 3057166 9.291997 1783346 13.4 148276 240.7 > 48 3148972 10.382461 1741873 17.4 145296 327.3 > > I have used an average server machine in our lab with two NUMA nodes and total of 40 cores (Can't remember all the details). > Running fio with 4k random writes. The IO is then just memcpy_nt() to a pmem simulated DRAM. The fio was run with more and > more threads (see threads column) (See the nice graphs inside) > > We can see that we are still > x2 slower than in-Kernel FS. But I believe I can shave off another 1 us by optimizing the app-to-server thread switch by utilizing perhaps the "Binder" scheduler object or devising another way to not be going through the scheduler (and its locks) when switching VMs This work was also presented at the SNIA Persistent Memory Summit last week. The use case of course is providing a user space platform for the development and deployment of memory-based file systems. The value-add of this kind of file system is ultra-low latency, which is a challenge for the current most popular such framework, FUSE. To start, I can think of three areas where specific questions might be entertained by LSF/MM attendees: - Spectre mitigations make this whole "user space filesystem" arrangement even slower, thanks to additional context switches between user space and the kernel. What can be done for FUSE and ZUFS to reduce the impact of Spectre mitigations? - The fundamental innovation of ZUFS is porting Solaris "doors" to Linux. A "door" is a local RPC mechanism that stays on the same thread to reduce scheduling overhead during calls to services provided by daemons on the local system. Is there interest in building a generic "doors" facility in Linux? - ZUFS currently supports only synchronous calls to the file system, as the assumption is the services (filesystems) will be I/O-less, typically. Is there a need to support asynchronicity in the ZUFS API? -- Chuck Lever
