Re: [PATCH] md/raid5: reduce lock contention in read_one_chunk()

["NeilBrown" <neilb@xxxxxxx>]

On Thu, 03 Jun 2021, gal.ofri@xxxxxxxxxx wrote:
> * Note: I tried to use a simple spinlock rather than a rwlock, but contention
> remains this way.

This surprises me.  I only expect rwlocks to be a benefit when the
readlock is held for significantly longer than the time it takes to get
an uncontended lock, and I don't think that is happening here.
However, you have done the measurements (thanks for that!) and I cannot
argue with numbers.

However it does suggest that the lock is still heavily contented.  I
wonder if we can do better.  Can we make the common case lockless?
e.g. change the read_one_chunk code to something like
  
  if (!conf->quiesce) {
         atomic_inc(&conf->active_aligned_reads);
         did_inc = true;
  }
  if (smp_load_acquire(&conf->quiesce)) {
            if (did_inc && atomic_dec_and_test(&cnf->active_aligned_reads))
                 wakeup(conf->wait_for_quiescent);
            old code goes here
  }

and probably change the "conf->quiesce = 2" in raid5_quiesce() to
    smp_store_release(&conf->quiesce, 2);


Could you try that and report results?  Thanks.

NeilBrown