On Sun, 2012-06-24 at 14:34 -0700, Andi Kleen wrote:
> On Sat, Jun 23, 2012 at 08:26:04PM +0800, Fengguang Wu wrote:
> > Hi,
> >
> > When doing parallel kernel builds on an NFSROOT system with 32 logical
> > CPUs, I see half CPU time spent in kernel:
> >
> > top - 20:08:28 up 5:01, 8 users, load average: 62.97, 55.82, 54.56
> > Tasks: 751 total, 80 running, 671 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 0.1%us, 53.5%sy, 46.4%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
> > Mem: 65401132k total, 36455604k used, 28945528k free, 0k buffers
> > Swap: 0k total, 0k used, 0k free, 31858244k cached
> >
> > The attached lock_stat shows that rpcauth_lookup_credcache() has the
> > outstanding contentions.
>
> Hmm, the cache is already using RCU for the read case, just takes
> a lock for writing. Or when checking an entry.
>
> So you either have a lot of lookups on the same entry, or you thrash
> the cache.
>
> When you look up the look address, is it the one in the first loop?
> If yes then it's the first.
>
> I'm not fully sure why it takes the lock in the read case anyways. Neither
> test_bit nor atomic_inc under RCU should need a lock, so I suppose
> it can be just removed.
>
> 367 spin_lock(&cache->lock);
> 368 if (test_bit(RPCAUTH_CRED_HASHED, &entry->cr_flags) == 0) {
> 369 spin_unlock(&cache->lock);
> 370 continue;
> 371 }
> 372 cred = get_rpccred(entry);
> 373 spin_unlock(&cache->lock);
Hi Andi,
I believe that the spinlock is there in order to ensure that the
test_bit() and get_rpccred() are atomic w.r.t. garbage collection.
What we could perhaps do is create a lock-less fast path. For instance
use atomic_inc_unless_zero to grab a reference in cases where we know
that the garbage collector can't act on this cred. We'd still need to
test for RPCAUTH_CRED_HASHED, but since we'd be holding a reference we
could do that outside the lock...
Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@xxxxxxxxxx
www.netapp.com
��.n��������+%������w��{.n�����{��w��� jg��� �����ݢj����G���� ���j:+v���w�m������w�������h�����٥
