Re: [RFC] fix parallelism for rpc tasks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




> On Feb 14, 2018, at 6:13 PM, Mora, Jorge <Jorge.Mora@xxxxxxxxxx> wrote:
> 
> Hello,
> 
> The patch gives some performance improvement on Kerberos read.
> The following results show performance comparisons between unpatched
> and patched systems. The html files included as attachments show the
> results as line charts.
> 
> - Best read performance improvement when testing with a single dd transfer.
>  The patched system gives 70% better performance than the unpatched system.
>  (first set of results)
> 
> - The patched system gives 18% better performance than the unpatched system
>  when testing with multiple dd transfers.
>  (second set of results)
> 
> - The write test shows there is no performance hit by the patch.
>  (third set of results)
> 
> - When testing on a different client having less RAM and fewer number of CPU cores,
>  there is no performance degradation for Kerberos in the unpatched system.
>  In this case, the patch does not provide any performance improvement.
>  (fourth set of results)
> 
> ================================================================================
> Test environment:
> 
> NFS client:  CPU: 16 cores, RAM: 32GB (E5620 @ 2.40GHz)
> NFS servers: CPU: 16 cores, RAM: 32GB (E5620 @ 2.40GHz)
> NFS mount:   NFSv3 with sec=(sys or krb5p)
> 
> For tests with a single dd transfer there is of course one NFS server used
> and one file being read -- only one transfer was needed to fill up the
> network connection.
> 
> For tests with multiple dd transfers, three different NFS server were used
> and four different files were used per NFS server for a total of 12 different
> files being read (12 different transfers in parallel).
> 
> The patch was applied on top of 4.14.0-rc3 kernel and the NFS servers were
> running RHEL 7.4.
> 
> The fourth set of results below show an unpatched system with no Kerberos
> degradation (same kernel 4.14.0-rc3) but in contrast with the main client
> used for testing this client has only 4 CPU cores and 8GB of RAM.
> I believe that even though this system has less CPU cores and less RAM,
> the CPU is faster (E31220 @ 3.10GHz vs E5620 @ 2.40GHz) so it is able
> to handle the Kerberos load better and fill up the network connection
> with a single thread than the main client with more CPU cores and more
> memory.

Jorge, thanks for publishing these results.

Can you do a "numactl -H" on your clients and post the output? I suspect
the throughput improvement on the big client is because WQ_UNBOUND
behaves differently on NUMA systems. (Even so, I agree that the proposed
change is valuable).


> ================================================================================
> 
> Kerberos Read Performance: 170.15% (patched system over unpatched system)
> 
> Client CPU:        Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
> CPU cores:         16
> RAM:               32 GB
> NFS version:       3
> Mount points:      1
> dd's per mount:    1
> Total dd's:        1
> Data transferred:  7.81 GB (per run)
> Number of runs:    10
> 
> Kerberos Read Performance (unpatched system vs patched system)
> Transfer rate (unpatched system)  avg:  65.88 MB/s,   var:  20.28,   stddev:   4.50
> Transfer rate (patched system)    avg: 112.10 MB/s,   var:   0.00,   stddev:   0.01
> Performance (patched over unpatched):  170.15%
> 
> Unpatched System Read Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 111.96 MB/s,   var:   0.02,   stddev:   0.13
> Transfer rate (sec=krb5p)  avg:  65.88 MB/s,   var:  20.28,   stddev:   4.50
> Performance (krb5p over sys):   58.84%
> 
> Patched System Read Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 111.94 MB/s,   var:   0.02,   stddev:   0.14
> Transfer rate (sec=krb5p)  avg: 112.10 MB/s,   var:   0.00,   stddev:   0.01
> Performance (krb5p over sys):  100.14%
> 
> ================================================================================
> 
> Kerberos Read Performance: 118.02% (patched system over unpatched system)
> 
> Client CPU:        Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
> CPU cores:         16
> RAM:               32 GB
> NFS version:       3
> Mount points:      3
> dd's per mount:    4
> Total dd's:        12
> Data transferred:  93.75 GB (per run)
> Number of runs:    10
> 
> Kerberos Read Performance (unpatched system vs patched system)
> Transfer rate (unpatched system)  avg:  94.99 MB/s,   var:  68.96,   stddev:   8.30
> Transfer rate (patched system)    avg: 112.11 MB/s,   var:   0.00,   stddev:   0.03
> Performance (patched over unpatched):  118.02%
> 
> Unpatched System Read Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 112.21 MB/s,   var:   0.00,   stddev:   0.00
> Transfer rate (sec=krb5p)  avg:  94.99 MB/s,   var:  68.96,   stddev:   8.30
> Performance (krb5p over sys):   84.66%
> 
> Patched System Read Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 112.20 MB/s,   var:   0.00,   stddev:   0.00
> Transfer rate (sec=krb5p)  avg: 112.11 MB/s,   var:   0.00,   stddev:   0.03
> Performance (krb5p over sys):   99.92%
> 
> ================================================================================
> 
> Kerberos Write Performance: 101.55% (patched system over unpatched system)
> 
> Client CPU:        Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
> CPU cores:         16
> RAM:               32 GB
> NFS version:       3
> Mount points:      3
> dd's per mount:    4
> Total dd's:        12
> Data transferred:  93.75 GB (per run)
> Number of runs:    10
> 
> Kerberos Write Performance (unpatched system vs patched system)
> Transfer rate (unpatched system)  avg: 103.70 MB/s,   var: 110.51,   stddev:  10.51
> Transfer rate (patched system)    avg: 105.31 MB/s,   var:  35.04,   stddev:   5.92
> Performance (patched over unpatched):  101.55%
> 
> Unpatched System Write Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 109.87 MB/s,   var:  10.27,   stddev:   3.20
> Transfer rate (sec=krb5p)  avg: 103.70 MB/s,   var: 110.51,   stddev:  10.51
> Performance (krb5p over sys):   94.39%
> 
> Patched System Write Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 111.03 MB/s,   var:   0.58,   stddev:   0.76
> Transfer rate (sec=krb5p)  avg: 105.31 MB/s,   var:  35.04,   stddev:   5.92
> Performance (krb5p over sys):   94.85%
> 
> ================================================================================
> 
> Kerberos Read Performance: 99.99% (patched system over unpatched system)
> 
> Client CPU:        Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
> CPU cores:         4
> RAM:               8 GB
> NFS version:       3
> Mount points:      1
> dd's per mount:    1
> Total dd's:        1
> Data transferred:  7.81 GB (per run)
> Number of runs:    10
> 
> Kerberos Read Performance (unpatched system vs patched system)
> Transfer rate (unpatched system)  avg: 112.02 MB/s,   var:   0.04,   stddev:   0.21
> Transfer rate (patched system)    avg: 112.01 MB/s,   var:   0.06,   stddev:   0.25
> Performance (patched over unpatched):   99.99%
> 
> Unpatched System Read Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 111.86 MB/s,   var:   0.06,   stddev:   0.24
> Transfer rate (sec=krb5p)  avg: 112.02 MB/s,   var:   0.04,   stddev:   0.21
> Performance (krb5p over sys):  100.14%
> 
> Patched System Read Performance (sys vs krb5p)
> Transfer rate (sec=sys)    avg: 111.76 MB/s,   var:   0.12,   stddev:   0.34
> Transfer rate (sec=krb5p)  avg: 112.01 MB/s,   var:   0.06,   stddev:   0.25
> Performance (krb5p over sys):  100.22%
> 
> 
> --Jorge
> 
> ________________________________________
> From: linux-nfs-owner@xxxxxxxxxxxxxxx <linux-nfs-owner@xxxxxxxxxxxxxxx> on behalf of Olga Kornievskaia <aglo@xxxxxxxxx>
> Sent: Wednesday, July 19, 2017 11:59 AM
> To: Trond Myklebust
> Cc: linux-nfs@xxxxxxxxxxxxxxx; chuck.lever@xxxxxxxxxx
> Subject: Re: [RFC] fix parallelism for rpc tasks
> 
> On Wed, Jul 5, 2017 at 1:33 PM, Olga Kornievskaia <aglo@xxxxxxxxx> wrote:
>> On Wed, Jul 5, 2017 at 12:14 PM, Trond Myklebust
>> <trondmy@xxxxxxxxxxxxxxx> wrote:
>>> On Wed, 2017-07-05 at 12:09 -0400, Olga Kornievskaia wrote:
>>>> On Wed, Jul 5, 2017 at 11:46 AM, Trond Myklebust
>>>> <trondmy@xxxxxxxxxxxxxxx> wrote:
>>>>> On Wed, 2017-07-05 at 11:11 -0400, Chuck Lever wrote:
>>>>>>> On Jul 5, 2017, at 10:44 AM, Olga Kornievskaia <aglo@xxxxxxxxx>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> On Mon, Jul 3, 2017 at 10:58 AM, Trond Myklebust
>>>>>>> <trondmy@xxxxxxxxxxxxxxx> wrote:
>>>>>>>> On Thu, 2017-06-29 at 09:25 -0400, Olga Kornievskaia wrote:
>>>>>>>>> Hi folks,
>>>>>>>>> 
>>>>>>>>> On a multi-core machine, is it expected that we can have
>>>>>>>>> parallel
>>>>>>>>> RPCs
>>>>>>>>> handled by each of the per-core workqueue?
>>>>>>>>> 
>>>>>>>>> In testing a read workload, observing via "top" command
>>>>>>>>> that a
>>>>>>>>> single
>>>>>>>>> "kworker" thread is running servicing the requests (no
>>>>>>>>> parallelism).
>>>>>>>>> It's more prominent while doing these operations over krb5p
>>>>>>>>> mount.
>>>>>>>>> 
>>>>>>>>> What has been suggested by Bruce is to try this and in my
>>>>>>>>> testing I
>>>>>>>>> see then the read workload spread among all the kworker
>>>>>>>>> threads.
>>>>>>>>> 
>>>>>>>>> Signed-off-by: Olga Kornievskaia <kolga@xxxxxxxxxx>
>>>>>>>>> 
>>>>>>>>> diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
>>>>>>>>> index 0cc8383..f80e688 100644
>>>>>>>>> --- a/net/sunrpc/sched.c
>>>>>>>>> +++ b/net/sunrpc/sched.c
>>>>>>>>> @@ -1095,7 +1095,7 @@ static int rpciod_start(void)
>>>>>>>>> * Create the rpciod thread and wait for it to start.
>>>>>>>>> */
>>>>>>>>> dprintk("RPC:       creating workqueue rpciod\n");
>>>>>>>>> - wq = alloc_workqueue("rpciod", WQ_MEM_RECLAIM, 0);
>>>>>>>>> + wq = alloc_workqueue("rpciod", WQ_MEM_RECLAIM |
>>>>>>>>> WQ_UNBOUND,
>>>>>>>>> 0);
>>>>>>>>> if (!wq)
>>>>>>>>> goto out_failed;
>>>>>>>>> rpciod_workqueue = wq;
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> WQ_UNBOUND turns off concurrency management on the thread
>>>>>>>> pool
>>>>>>>> (See
>>>>>>>> Documentation/core-api/workqueue.rst. It also means we
>>>>>>>> contend
>>>>>>>> for work
>>>>>>>> item queuing/dequeuing locks, since the threads which run the
>>>>>>>> work
>>>>>>>> items are not bound to a CPU.
>>>>>>>> 
>>>>>>>> IOW: This is not a slam-dunk obvious gain.
>>>>>>> 
>>>>>>> I agree but I think it's worth consideration. I'm waiting to
>>>>>>> get
>>>>>>> (real) performance numbers of improvement (instead of my VM
>>>>>>> setup)
>>>>>>> to
>>>>>>> help my case. However, it was reported 90% degradation for the
>>>>>>> read
>>>>>>> performance over krb5p when 1CPU is executing all ops.
>>>>>>> 
>>>>>>> Is there a different way to make sure that on a multi-processor
>>>>>>> machine we can take advantage of all available CPUs? Simple
>>>>>>> kernel
>>>>>>> threads instead of a work queue?
>>>>>> 
>>>>>> There is a trade-off between spreading the work, and ensuring it
>>>>>> is executed on a CPU close to the I/O and application. IMO
>>>>>> UNBOUND
>>>>>> is a good way to do that. UNBOUND will attempt to schedule the
>>>>>> work on the preferred CPU, but allow it to be migrated if that
>>>>>> CPU is busy.
>>>>>> 
>>>>>> The advantage of this is that when the client workload is CPU
>>>>>> intensive (say, a software build), RPC client work can be
>>>>>> scheduled
>>>>>> and run more quickly, which reduces latency.
>>>>>> 
>>>>> 
>>>>> That should no longer be a huge issue, since queue_work() will now
>>>>> default to the WORK_CPU_UNBOUND flag, which prefers the local CPU,
>>>>> but
>>>>> will schedule elsewhere if the local CPU is congested.
>>>> 
>>>> I don't believe NFS use workqueue_congested() to somehow schedule the
>>>> work elsewhere. Unless the queue is marked UNBOUNDED I don't believe
>>>> there is any intention of balancing the CPU load.
>>>> 
>>> 
>>> I shouldn't have to test the queue when scheduling with
>>> WORK_CPU_UNBOUND.
>>> 
>> 
>> Comments in the code says that "if CPU dies" it'll be re-scheduled on
>> another. I think the code requires to mark the queue UNBOUND to really
>> be scheduled on a different CPU. Just my reading of the code and it
>> matches what is seen with the krb5 workload.
> 
> Trond, what's the path forward here? What about a run-time
> configuration that starts rpciod with the UNBOUND options instead?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> <dd_read_single.html><dd_read_mult.html><dd_write_mult.html><dd_read_single1.html>

--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Filesystem Development]     [Linux USB Development]     [Linux Media Development]     [Video for Linux]     [Linux NILFS]     [Linux Audio Users]     [Yosemite Info]     [Linux SCSI]

  Powered by Linux