Hello! Recently I bumped into an interesting situation, lab environment consists of two identical servers, interconnected via infiniband cx6 nics. Server with 6.9.12 release and client with 6.10.2. During simple fio testing, throughput capped to 90K 4k IOPS without nconnect and 160k 4k IOPS with nconnect=16. For block 1M almost the same story 6GB/s vs 9GB/s, for one fio job. Howether client was not maxed out. What could we do in the current situation? I would be very grateful for any input. Previous discussion: 1) https://marc.info/?l=linux-nfs&m=170964724830582&w=4 2) https://marc.info/?l=linux-nfs&m=171966720614369&w=2 3) https://marc.info/?l=linux-nfs&m=172196920511337&w=2 Lab env details: # ======== # captured on : Fri Aug 2 11:18:39 2024 # header version : 1 # data offset : 2456 # data size : 22161720 # feat offset : 22164176 # hostname : nfsd-test.local # os release : 6.9.12-060912-generic # perf version : 6.9.12 # arch : x86_64 # nrcpus online : 128 # nrcpus avail : 128 # cpudesc : AMD EPYC 9354 32-Core Processor # cpuid : AuthenticAMD,25,17,1 # total memory : 1584976412 kB mount config: 198.51.100.10:/tmp/ram_export /mnt/nfs_ram nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=rdma,nconnect=16,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=198.51.100.33,local_lock=none,addr=198.51.100.10 0 0 exports configs: tmpfs /tmp/ram_export tmpfs rw,relatime,size=536870912k,nr_inodes=262144,inode64,huge=always,noswap 0 0 /tmp/ram_export *(rw,no_root_squash,sync,insecure,no_wdelay) nfsd config: [exportd] threads = 32 [general] pipefs-directory = /run/rpc_pipefs [mountd] manage-gids = y threads = 32 [nfsd] rdma = y rdma-port = 20049 threads = 32 fio config: [globall] #bs=4k bs=1M #ioengine=libaio ioengine=io_uring rw=write direct=1 group_reporting time_based runtime=900 size=8G iodepth=32 exitall [bench_rdma_cpu] directory=/mnt/nfs_ram -- Thanks, Damir Chanyshev
<<< text/html; charset="UTF-8"; name="flamegraph_test.html": Unrecognized >>>
Attachment:
test_nfs.svg
Description: image/svg
