> On Aug 2, 2024, at 2:11 PM, Damir Chanyshev <conflict342@xxxxxxxxx> wrote: > > Hello! > Recently I bumped into an interesting situation, lab environment > consists of two identical servers, interconnected via infiniband cx6 > nics. > Server with 6.9.12 release and client with 6.10.2. > During simple fio testing, throughput capped to 90K 4k IOPS without > nconnect and 160k 4k IOPS with nconnect=16. For block 1M almost the > same story 6GB/s vs 9GB/s, for one fio job. > Howether client was not maxed out. > What could we do in the current situation? I would be very grateful > for any input. > > Previous discussion: > 1) https://marc.info/?l=linux-nfs&m=170964724830582&w=4 > 2) https://marc.info/?l=linux-nfs&m=171966720614369&w=2 > 3) https://marc.info/?l=linux-nfs&m=172196920511337&w=2 > > Lab env details: > > # ======== > # captured on : Fri Aug 2 11:18:39 2024 > # header version : 1 > # data offset : 2456 > # data size : 22161720 > # feat offset : 22164176 > # hostname : nfsd-test.local > # os release : 6.9.12-060912-generic > # perf version : 6.9.12 > # arch : x86_64 > # nrcpus online : 128 > # nrcpus avail : 128 > # cpudesc : AMD EPYC 9354 32-Core Processor > # cpuid : AuthenticAMD,25,17,1 > # total memory : 1584976412 kB > > mount config: > 198.51.100.10:/tmp/ram_export /mnt/nfs_ram nfs4 > rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=rdma,nconnect=16,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=198.51.100.33,local_lock=none,addr=198.51.100.10 > 0 0 > > exports configs: > tmpfs /tmp/ram_export tmpfs > rw,relatime,size=536870912k,nr_inodes=262144,inode64,huge=always,noswap > 0 0 > /tmp/ram_export *(rw,no_root_squash,sync,insecure,no_wdelay) > > > nfsd config: > [exportd] > threads = 32 > > [general] > pipefs-directory = /run/rpc_pipefs > > [mountd] > manage-gids = y > threads = 32 > > [nfsd] > rdma = y > rdma-port = 20049 > threads = 32 > > fio config: > > [globall] > #bs=4k > bs=1M > #ioengine=libaio > ioengine=io_uring > rw=write > direct=1 > group_reporting > time_based > runtime=900 > size=8G > iodepth=32 > exitall > > [bench_rdma_cpu] > directory=/mnt/nfs_ram The flame graph shows that the inode_lock in shmem_file_write_iter is very contented. My first guess would be that your fio test is configured to use only one file. -- Chuck Lever
