On Wed, Apr 17, 2013 at 10:32 AM, Atchley, Scott <atchleyes@xxxxxxxx> wrote: > On Apr 17, 2013, at 1:15 PM, Wendy Cheng <s.wendy.cheng@xxxxxxxxx> wrote: > >> On Wed, Apr 17, 2013 at 7:36 AM, Yan Burman <yanb@xxxxxxxxxxxx> wrote: >>> Hi. >>> >>> I've been trying to do some benchmarks for NFS over RDMA and I seem to only get about half of the bandwidth that the HW can give me. >>> My setup consists of 2 servers each with 16 cores, 32Gb of memory, and Mellanox ConnectX3 QDR card over PCI-e gen3. >>> These servers are connected to a QDR IB switch. The backing storage on the server is tmpfs mounted with noatime. >>> I am running kernel 3.5.7. >>> >>> When running ib_send_bw, I get 4.3-4.5 GB/sec for block sizes 4-512K. >>> When I run fio over rdma mounted nfs, I get 260-2200MB/sec for the same block sizes (4-512K). running over IPoIB-CM, I get 200-980MB/sec. > > Yan, > > Are you trying to optimize single client performance or server performance with multiple clients? > > >> Remember there are always gaps between wire speed (that ib_send_bw >> measures) and real world applications. >> >> That being said, does your server use default export (sync) option ? >> Export the share with "async" option can bring you closer to wire >> speed. However, the practice (async) is generally not recommended in a >> real production system - as it can cause data integrity issues, e.g. >> you have more chances to lose data when the boxes crash. >> >> -- Wendy > > > Wendy, > > It has a been a few years since I looked at RPCRDMA, but I seem to remember that RPCs were limited to 32KB which means that you have to pipeline them to get linerate. In addition to requiring pipelining, the argument from the authors was that the goal was to maximize server performance and not single client performance. > > Scott > That (client count) brings up a good point ... FIO is really not a good benchmark for NFS. Does anyone have SPECsfs numbers on NFS over RDMA to share ? -- Wendy -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
