On Mon, 2017-07-17 at 13:26 +0000, Trond Myklebust wrote:
> On Mon, 2017-07-17 at 11:22 +0200, Chuck Lever wrote:
> > FYI, I found this in my notes. It gives a flavor of the
> > kind of lock contention we were discussing last week.
> >
> > My client is a 12-core dual socket system. Networking
> > is 56Gbps IB, NFS/RDMA. I'm running some multi-threaded
> > iozone tests. Here I was mixing buffered and direct I/O
> > tests, so I can't say for certain if one is worse than
> > the other.
> >
> > The output below is the first entry in /proc/lock_stat.
> >
> > "sb->s_type->i_lock_key" looked like it was related to
> > the superblock, but looking at the actual code in the
> > functions named below, this is probably the inode lock.
> >
> > The max lock holdtime is 82 milliseconds, if I read this
> > correctly. It's not clear how often that happens, but
> > that looks pathological.
> >
> >
> > lock_stat version 0.4
> > -------------------------------------------------------------------
> > -------------------------------------------------------------------
> > -------------------------------------------------------------------
> > --------------------
> > class name con-
> > bounces contentions waittime-min waittime-max waittime-
> > total waittime-avg acq-bounces acquisitions holdtime-
> > min holdtime-max holdtime-total holdtime-avg
> > -------------------------------------------------------------------
> > -------------------------------------------------------------------
> > -------------------------------------------------------------------
> > --------------------
> >
> > &sb->s_type-
> > > i_lock_key#2: 23185746 23275952 0.10 8
> > > 177
> >
> > 1.03 27980172.31 1.20 81467297 238209945
> >
> > 0.09 81842.12 80513710.68 0.34
> > -------------------------
> > &sb->s_type-
> > > i_lock_key#2 1453378 [<ffffffffa06c6e49>]
> >
> > nfs_flush_incompatible+0x75/0x1ad [nfs]
> > &sb->s_type-
> > > i_lock_key#2 4777000 [<ffffffffa06c4ed6>]
> >
> > nfs_lock_and_join_requests+0x68/0x2a3 [nfs]
> > &sb->s_type-
> > > i_lock_key#2 157315 [<ffffffffa06c72f5>]
> >
> > nfs_updatepage+0x374/0x912 [nfs]
> > &sb->s_type-
> > > i_lock_key#2 2085447 [<ffffffffa06c74d7>]
> >
> > nfs_updatepage+0x556/0x912 [nfs]
> > -------------------------
> > &sb->s_type-
> > > i_lock_key#2 132 [<ffffffff8123cc2c>]
> >
> > writeback_sb_inodes+0xfb/0x50c
> > &sb->s_type-
> > > i_lock_key#2 7858928 [<ffffffffa06c4ed6>]
> >
> > nfs_lock_and_join_requests+0x68/0x2a3 [nfs]
> > &sb->s_type-
> > > i_lock_key#2 1225835 [<ffffffffa06c6e49>]
> >
> > nfs_flush_incompatible+0x75/0x1ad [nfs]
> > &sb->s_type-
> > > i_lock_key#2 111821 [<ffffffffa06c72f5>]
> >
> > nfs_updatepage+0x374/0x912 [nfs]
> >
> >
>
> Interesting... There are definitely some optimisations we can do
> there
> in order to get rid of some of this contention (for instance simple
> things like doing a lockless check for
> PagePrivate(page)||PageSwapCache(page) before we grab the lock in
> nfs_page_find_head_request().
>
> Thanks Chuck! I'll take a deeper look at this today.
>
So, the other thing to note is that the inode->i_lock protection in
nfs_lock_and_join_requests() is pretty much worthless once we've found
the page head. After that, we can and should rely on the page and page
group locks. I'm running a test on a set of patches that clean up the
locking there and should hopefully get rid of the contention.
--
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@xxxxxxxxxxxxxxx
��.n��������+%������w��{.n�����{��w��� jg��� �����ݢj����G���� ���j:+v���w�m������w�������h�����٥
