Hi Xiaoguang and others,
You have identified an issue that we are also running into and had also
planned to address with the community.
We currently solve this by explicitly telling the virtio-fs device which
architecture is running on the host through a side-channel so that it
can correctly interpret the flags (e.g. O_DIRECT).
As others are also running into this with the increased popularity of
hardware virtio-fs devices, let's get into this issue.
I have included the FUSE maintainer Milkos Szeredi and the Virtio
maintainer+specification chair Michael S. Tsirkin in this thread.
The core issue here lies in the fact that these fcntl.h definitions are
different per architecture (as of now there are 9 fcntl.h headers). More
specifically, the real numbers defined in the header are different, on
top of possible endianness differences.
FUSE has a mechanism to detect endianness differences with the 32bit
in.opcode value (via FUSE_INIT_BSWAP_RESERVED).
However, there is no mechanism in FUSE or virtio-fs to tell the FUSE
file system or the virtio-fs device which architecture is running on the
host. Thus, the virtio-fs device currently cannot know how to correctly
interpret the fcntl.h flags.
For example, we are dealing with systems that have ARM64 host and ARM64
virtio-fs device, or x86 host and ARM64 virtio-fs device.
As FUSE already contains the mechanism for endianness, it would make
sense to also include a mechanism for the architecture in FUSE to solve
this issue.
We would like to make a proposal regarding our idea for solving this
issue before sending in a patch:
Use a uint32_t from the unused array in FUSE_INIT to encode an `uint32_t
arch_indicator` that contains one of the architecture IDs specified in a
new enum (is there an existing enum like such?):
enum fuse_arch_indicator {
FUSE_ARCH_NONE = 0,
FUSE_ARCH_X86 = 1,
FUSE_ARCH_ARM64 = 2,
...
}
Through this the host tells the FUSE file system which version of
fcntl.h it will use.
The FUSE file system should keep a copy of all the possible fcntl
headers and use the one indicated by the `fuse_init_in.arch_indicator`.
For backwards compatibility, a minor version bump is needed. A new file
system implementation connected to an old driver will see the
FUSE_ARCH_NONE or the old minor version, and it will know that it cannot
read the `arch_indicator` and that it cannot make better assumptions
than previously possible.
This would be a minimal, backwards compatible change that extends the
current FUSE portability scheme and doesn't require any specification
changes. When the time comes that a new architecture is introduced with
its own fcntl.h we must simply add another enumerator and an ifdef to
the code setting the `arch_indicator`.
- Peter-Jan
On Thu, 2024-05-30 at 09:31 +0000, Lege Wang wrote:
> External email: Use caution opening links or attachments
>
>
> Hello,
>
> I see that you have added multi-queue support for virtio-fs, thanks
> for this work.
> From your patch's commit log, your host is x86-64, dpu is arm64, but
> there're
> differences about O_DIRECT and O_DIRECTORY between these two
> architectures.
>
> Test program:
> #define _GNU_SOURCE
>
> #include <stdio.h>
> #include <fcntl.h>
>
> int main(void)
> {
> printf("O_DIRECT:%o\n", O_DIRECT);
> printf("O_DIRECTORY:%o\n", O_DIRECTORY);
> return 0;
> }
>
> In x86-64, this test program outputs:
> O_DIRECT:40000
> O_DIRECTORY:200000
>
> But in arm64, it outpus:
> O_DIRECT:200000
> O_DIRECTORY:40000
>
> In kernel fuse module, fuse_create_in->flags will used to hold
> open(2)'s flags, then
> a O_DIRECT flag from host(x86) would be treated as O_DIRECTORY in
> dpu(arm64), which
> seems a serious bug.
>
> From your fio job, you use libaio engine, so it's assumed that direct-
> io is
> enabled, so I wonder why you don't get errors. Could you please try
> below
> command in your virtio-fs mount point:
> dd if=/dev/zero of=tst_file bs=4096 count=1 oflag=direct
> to see whether it occurs any error.
>
> Regards,
> Xiaoguang Wang
