On Thu, Nov 30, 2023 at 12:01 AM Bernd Schubert <bernd.schubert@xxxxxxxxxxx> wrote: > > > > On 11/29/23 22:39, Antonio SJ Musumeci wrote: > > On 11/29/23 14:46, Bernd Schubert wrote: > >> > >> > >> On 11/29/23 18:39, Amir Goldstein wrote: > >>> On Wed, Nov 29, 2023 at 6:55 PM Miklos Szeredi <miklos@xxxxxxxxxx> wrote: > >>>> > >>>> On Wed, 29 Nov 2023 at 16:52, Amir Goldstein <amir73il@xxxxxxxxx> wrote: > >>>> > >>>>> direct I/O read()/write() is never a problem. > >>>>> > >>>>> The question is whether mmap() on a file opened with FOPEN_DIRECT_IO > >>>>> when the inode is in passthrough mode, also uses fuse_passthrough_mmap()? > >>>> > >>>> I think it should. > >>>> > >>>>> or denied, similar to how mmap with ff->open_flags & FOPEN_DIRECT_IO && > >>>>> vma->vm_flags & VM_MAYSHARE) && !fc->direct_io_relax > >>>>> is denied? > >>>> > >>>> What would be the use case for FOPEN_DIRECT_IO with passthrough mmap? > >>>> > >>> > >>> I don't have a use case. That's why I was wondering if we should > >>> support it at all, but will try to make it work. > >> > >> What is actually the use case for FOPEN_DIRECT_IO and passthrough? > >> Avoiding double page cache? > >> > >>> > >>>>> A bit more challenging, because we will need to track unmounts, or at > >>>>> least track > >>>>> "was_cached_mmaped" state per file, but doable. > >>>> > >>>> Tracking unmaps via fuse_vma_close() should not be difficult. > >>>> > >>> > >>> OK. so any existing mmap, whether on FOPEN_DIRECT_IO or not > >>> always prevents an inode from being "neutral". > >>> > >> > >> > >> Thanks, > >> Bernd > >> > > > > > Avoiding double page cache? > > > > Currently my users enable direct_io because 1) it is typically > > materially faster than not 2) avoiding double page caching (it is a > > union filesystem). > > 3) You want coherency for network file systems (our use case). > > So performance kind of means it is preferred to have it enabled for > passthrough. And with that MAP_SHARED gets rather important, imho. I > don't know if recent gcc versions still do it, but gcc used to write > files using MAP_SHARED. In the HPC AI world python tools also tend to do > IO with MAP_SHARED. > I see. good to know. That means that an inode in passthrough mode should always imply direct_io_relax behavior. The way that passthrough mode and network file systems use cases can overlap is by per-inode basis, which is the typical case for HSM (out use case): - some inodes are local and some are remote - local inodes are open passthrough mode - remote inodes are open in direct or cached mode - inode can change state local => remote (a.k.a local storage evict) when there are no files open on the inode - inode can change state remote => local if there are no files open in cached mode and no mmaps (*) - specifically, remote files open in direct mode, do not block the transition remote => local, where later opens will be in passthrough mode, but the files already open in direct mode will not enjoy the performance boost * If we relax the last condition we will end up with cache incoherency that exists in overlayfs after copy up, when lower file is still mmaped Thanks, Amir.
