Re: [PATCH 0/7] Misc NFS + pNFS performance enhancements

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> On Sep 5, 2018, at 4:36 PM, Trond Myklebust <trondmy@xxxxxxxxxxxxxxx> wrote:
> 
> On Wed, 2018-09-05 at 15:33 -0400, Chuck Lever wrote:
>>> On Sep 5, 2018, at 3:23 PM, Trond Myklebust <trondmy@xxxxxxxxx>
>>> wrote:
>>> 
>>> Fallout from a bunch of flame graphs...
>> 
>> Hey, are these in a public git repo I can pull from?
>> 
> 
> I've pushed out my 'testing' branch to git.linux-nfs.org. The usual
> caveats apply: please do not treat that branch as being stable, assume
> it won't be rebased or assume that I won't change the contents. It is
> just there for testing purposes.
> 
> Cheers
> Trond
> 
>> 
>>> Trond Myklebust (7):
>>> pNFS: Don't zero out the array in nfs4_alloc_pages()
>>> pNFS: Don't allocate more pages than we need to fit a layoutget
>>>   response
>>> NFS: Convert lookups of the lock context to RCU
>>> NFS: Simplify internal check for whether file is open for write
>>> NFS: Convert lookups of the open context to RCU
>>> NFSv4: Convert open state lookup to use RCU
>>> NFSv4: Convert struct nfs4_state to use refcount_t
>>> 
>>> fs/nfs/delegation.c                    | 11 ++--
>>> fs/nfs/filelayout/filelayout.c         |  1 +
>>> fs/nfs/flexfilelayout/flexfilelayout.c |  1 +
>>> fs/nfs/inode.c                         | 70 +++++++++++----------
>>> -----
>>> fs/nfs/nfs4_fs.h                       |  3 +-
>>> fs/nfs/nfs4proc.c                      | 38 ++++++++++----
>>> fs/nfs/nfs4state.c                     | 32 ++++++------
>>> fs/nfs/pnfs.c                          | 16 ++++--
>>> fs/nfs/pnfs.h                          |  1 +
>>> include/linux/nfs_fs.h                 |  2 +
>>> 10 files changed, 98 insertions(+), 77 deletions(-)
>>> 
>>> -- 
>>> 2.17.1

Some performance testing results for the full "testing" series.
The fio tests are designed to push the IOPS rate, and the third
is a QD=1 test to measure the latency of 512KB NFS WRITE
operations. All three tests use direct I/O.

The "without fair queuing" kernels have this commit reverted:

commit ae03d238e8a11ddc76668c64ad405cd8412446a6
Author:     Trond Myklebust <trond.myklebust@xxxxxxxxxxxxxxx>
AuthorDate: Tue Sep 4 11:47:51 2018 -0400
Commit:     Trond Myklebust <trond.myklebust@xxxxxxxxxxxxxxx>
CommitDate: Wed Sep 5 14:37:07 2018 -0400

    SUNRPC: Queue fairness for all.


Client: 12-core, two-socket, 56Gb InfiniBand
Server: 4-core, one-socket, 56Gb InfiniBand, tmpfs export

Test: /usr/bin/fio --size=1G --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=240 --group_reporting

NFSv3 on RDMA:
Stock v4.19-rc2:
	• read: IOPS=109k, BW=849MiB/s (890MB/s)(11.2GiB/13506msec)
	• write: IOPS=46.6k, BW=364MiB/s (382MB/s)(4915MiB/13506msec)
Trond's kernel (with fair queuing):
	• read: IOPS=83.0k, BW=649MiB/s (680MB/s)(11.2GiB/17676msec)
	• write: IOPS=35.6k, BW=278MiB/s (292MB/s)(4921MiB/17676msec)
Trond's kernel (without fair queuing):
	• read: IOPS=90.5k, BW=707MiB/s (742MB/s)(11.2GiB/16216msec)
	• write: IOPS=38.8k, BW=303MiB/s (318MB/s)(4917MiB/16216msec)

NFSv3 on TCP (IPoIB):
Stock v4.19-rc2:
	• read: IOPS=23.8k, BW=186MiB/s (195MB/s)(11.2GiB/61635msec)
	• write: IOPS=10.2k, BW=79.9MiB/s (83.8MB/s)(4923MiB/61635msec)
Trond's kernel (with fair queuing):
	• read: IOPS=25.9k, BW=202MiB/s (212MB/s)(11.2GiB/56710msec)
	• write: IOPS=11.1k, BW=86.7MiB/s (90.9MB/s)(4916MiB/56710msec)
Trond's kernel (without fair queuing):
	• read: IOPS=25.0k, BW=203MiB/s (213MB/s)(11.2GiB/56492msec)
	• write: IOPS=11.1k, BW=86.0MiB/s (91.2MB/s)(4915MiB/56492msec)


Test: /usr/bin/fio --size=1G --direct=1 --rw=randread --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=1024 --numjobs=16 --runtime=240 --group_reporting

NFSv3 on RDMA:
Stock v4.19-rc2:
	• read: IOPS=149k, BW=580MiB/s (608MB/s)(16.0GiB/28241msec)
Trond's kernel (with fair queuing):
	• read: IOPS=81.5k, BW=318MiB/s (334MB/s)(16.0GiB/51450msec)
Trond's kernel (without fair queuing):
	• read: IOPS=82.4k, BW=322MiB/s (337MB/s)(16.0GiB/50918msec)

NFSv3 on TCP (IPoIB):
Stock v4.19-rc2:
	• read: IOPS=37.2k, BW=145MiB/s (153MB/s)(16.0GiB/112630msec)
Trond's kernel (with fair queuing):
	• read: IOPS=2715, BW=10.6MiB/s (11.1MB/s)(2573MiB/242594msec)
Trond's kernel (without fair queuing):
	• read: IOPS=2869, BW=11.2MiB/s (11.8MB/s)(2724MiB/242979msec)


Test: /home/cel/bin/iozone -M -i0 -s8g -r512k -az -I -N

My kernel: 4.19.0-rc2-00026-g50d68a4
system call latencies in microseconds, N=5:
	• write:    mean=602, std=13.0
	• rewrite:  mean=541, std=17.3
server round trip latency in microseconds, N=5:
	• RTT:      mean=354, std=3.0

Trond's kernel (with fair queuing):
system call latencies in microseconds, N=5:
	• write:    mean=572, std=10.6
	• rewrite:  mean=533, std=7.9
server round trip latency in microseconds, N=5:
	• RTT:      mean=352, std=2.7


--
Chuck Lever







[Index of Archives]     [Linux Filesystem Development]     [Linux USB Development]     [Linux Media Development]     [Video for Linux]     [Linux NILFS]     [Linux Audio Users]     [Yosemite Info]     [Linux SCSI]

  Powered by Linux