Writing to FUSE via mmap extremely slow (sometimes) on some machines?

[Michael Stapelberg <michael+lkml@xxxxxxxxxxxxx>]

Hey,

I’m running into an issue where writes via mmap are extremely slow.
The mmap program I’m using to test is attached.

The symptom is that the program usually completes in 0.x seconds, but
then sometimes takes minutes to complete! E.g.:

% dd if=/dev/urandom of=/tmp/was bs=1M count=99

% ./fusexmp_fh /tmp/mnt

% time ~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1
Mapped src: 0x10000  and dst: 0x21b8b000
memcpy done
~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1  0.06s user 0.20s system 48%
cpu 0.544 total

% time   ~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1
Mapped src: 0x10000  and dst: 0x471fb000
memcpy done
~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1  0.05s user 0.22s system 0%
cpu 2:03.39 total

This affects both an in-house FUSE file system and also FUSE’s
fusexmp_fh from 2.9.7 (matching what our in-house FS uses).

While this is happening, the machine is otherwise idle. E.g. dstat shows:

--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  1   0  98   1   0|   0     0 |  19k   23k|   0     0 |  14k   27k
  1   0  98   1   0|   0     0 |  33k   53k|   0     0 |  14k   29k
  0   0  98   1   0|   0   176k|  27k   26k|   0     0 |  13k   25k
[…]

While this is happening, using cp(1) to copy the same file is fast (1
second). It’s only mmap-based writing that’s slow.

This is with Linux 5.2.17, but has been going on for years apparently.

I haven’t quite figured out what the pattern is with regards to the
machines that are affected. One wild guess I have is that it might be
related to RAM? The machine on which I can most frequently reproduce
the issue has 192GB of RAM, whereas I haven’t been able to reproduce
the issue on my workstation with 64GB of RAM.

Any ideas what I could check to further narrow down this issue?

Thanks,

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h> 
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

/*
 * An implementation of copy ("cp") that uses memory maps.  Various
 * error checking has been removed to promote readability
 */

// Where we want the source file's memory map to live in virtual memory
// The destination file resides immediately after the source file
#define MAP_LOCATION 0x6100

int main (int argc, char *argv[]) {
 int fdin, fdout;
 char *src, *dst;
 struct stat statbuf;
 off_t fileSize = 0;

 if (argc != 3) {
   printf ("usage: a.out <fromfile> <tofile>\n");
   exit(0);
 }

 /* open the input file */
 if ((fdin = open (argv[1], O_RDONLY)) < 0) {
   printf ("can't open %s for reading\n", argv[1]);
   exit(0);
 }

 /* open/create the output file */
 if ((fdout = open (argv[2], O_RDWR | O_CREAT | O_TRUNC, 0600)) < 0) {
   printf ("can't create %s for writing\n", argv[2]);
   exit(0);
 }
 
 /* find size of input file */
 fstat (fdin,&statbuf) ;
 fileSize = statbuf.st_size;
 
 /* go to the location corresponding to the last byte */
 if (lseek (fdout, fileSize - 1, SEEK_SET) == -1) {
   printf ("lseek error\n");
   exit(0);
 }
 
 /* write a dummy byte at the last location */
 write (fdout, "", 1);
 
 /* 
  * memory map the input file.  Only the first two arguments are
  * interesting: 1) the location and 2) the size of the memory map 
  * in virtual memory space. Note that the location is only a "hint";
  * the OS can choose to return a different virtual memory address.
  * This is illustrated by the printf command below.
 */

 src = mmap ((void*) MAP_LOCATION, fileSize, 
	     PROT_READ, MAP_SHARED | MAP_POPULATE, fdin, 0);

 /* memory map the output file after the input file */
 dst = mmap ((void*) MAP_LOCATION + fileSize , fileSize , 
	     PROT_READ | PROT_WRITE, MAP_SHARED, fdout, 0);


 printf("Mapped src: 0x%x  and dst: 0x%x\n",src,dst);

 /* Copy the input file to the output file */
 memcpy (dst, src, fileSize);

 printf("memcpy done\n");

 // we should probably unmap memory and close the files
} /* main */