Hi Xiaoguang and others, You have identified an issue that we are also running into and had also planned to address with the community. We currently solve this by explicitly telling the virtio-fs device which architecture is running on the host through a side-channel so that it can correctly interpret the flags (e.g. O_DIRECT). As others are also running into this with the increased popularity of hardware virtio-fs devices, let's get into this issue. I have included the FUSE maintainer Milkos Szeredi and the Virtio maintainer+specification chair Michael S. Tsirkin in this thread. The core issue here lies in the fact that these fcntl.h definitions are different per architecture (as of now there are 9 fcntl.h headers). More specifically, the real numbers defined in the header are different, on top of possible endianness differences. FUSE has a mechanism to detect endianness differences with the 32bit in.opcode value (via FUSE_INIT_BSWAP_RESERVED). However, there is no mechanism in FUSE or virtio-fs to tell the FUSE file system or the virtio-fs device which architecture is running on the host. Thus, the virtio-fs device currently cannot know how to correctly interpret the fcntl.h flags. For example, we are dealing with systems that have ARM64 host and ARM64 virtio-fs device, or x86 host and ARM64 virtio-fs device. As FUSE already contains the mechanism for endianness, it would make sense to also include a mechanism for the architecture in FUSE to solve this issue. We would like to make a proposal regarding our idea for solving this issue before sending in a patch: Use a uint32_t from the unused array in FUSE_INIT to encode an `uint32_t arch_indicator` that contains one of the architecture IDs specified in a new enum (is there an existing enum like such?): enum fuse_arch_indicator { FUSE_ARCH_NONE = 0, FUSE_ARCH_X86 = 1, FUSE_ARCH_ARM64 = 2, ... } Through this the host tells the FUSE file system which version of fcntl.h it will use. The FUSE file system should keep a copy of all the possible fcntl headers and use the one indicated by the `fuse_init_in.arch_indicator`. For backwards compatibility, a minor version bump is needed. A new file system implementation connected to an old driver will see the FUSE_ARCH_NONE or the old minor version, and it will know that it cannot read the `arch_indicator` and that it cannot make better assumptions than previously possible. This would be a minimal, backwards compatible change that extends the current FUSE portability scheme and doesn't require any specification changes. When the time comes that a new architecture is introduced with its own fcntl.h we must simply add another enumerator and an ifdef to the code setting the `arch_indicator`. - Peter-Jan On Thu, 2024-05-30 at 09:31 +0000, Lege Wang wrote: > External email: Use caution opening links or attachments > > > Hello, > > I see that you have added multi-queue support for virtio-fs, thanks > for this work. > From your patch's commit log, your host is x86-64, dpu is arm64, but > there're > differences about O_DIRECT and O_DIRECTORY between these two > architectures. > > Test program: > #define _GNU_SOURCE > > #include <stdio.h> > #include <fcntl.h> > > int main(void) > { > printf("O_DIRECT:%o\n", O_DIRECT); > printf("O_DIRECTORY:%o\n", O_DIRECTORY); > return 0; > } > > In x86-64, this test program outputs: > O_DIRECT:40000 > O_DIRECTORY:200000 > > But in arm64, it outpus: > O_DIRECT:200000 > O_DIRECTORY:40000 > > In kernel fuse module, fuse_create_in->flags will used to hold > open(2)'s flags, then > a O_DIRECT flag from host(x86) would be treated as O_DIRECTORY in > dpu(arm64), which > seems a serious bug. > > From your fio job, you use libaio engine, so it's assumed that direct- > io is > enabled, so I wonder why you don't get errors. Could you please try > below > command in your virtio-fs mount point: > dd if=/dev/zero of=tst_file bs=4096 count=1 oflag=direct > to see whether it occurs any error. > > Regards, > Xiaoguang Wang