On 4/13/17, 11:45 AM, "J. Bruce Fields" <bfields@xxxxxxxxxxxx> wrote:
> On Wed, Apr 12, 2017 at 02:47:30PM +0000, Mora, Jorge wrote:
> > The following shows a comparison between inter server-side copy and a traditional copy.
>
> Thanks for doing this! Adding the linux-nfs list, I hope that's OK.
>
> > Setup:
> > Client: 16 CPUs, 32GB
> > SRC server: 4 CPUs, 8GB
> > DST server: 4 CPUs, 8GB
>
> Could you also tell us about the network? How much bandwidth is
> available between the two servers, and between the each server and the
> client? Disk bandwidth might be useful to know too.
Client: pair (1000baseT/Full)
SRC Server: ricoh (1000baseT/Full)
DST Server: haddock (1000baseT/Full)
Transfer from source server to client:
[mora@pair test]$ scp ricoh:/home/exports/nfstest_ssc_source_file /dev/null
nfstest_ssc_source_file 100% 8192MB 110.7MB/s 01:14
Transfer from destination server to client:
[mora@pair test]$ scp haddock:/home/exports/nfstest_ssc_source_file /dev/null
nfstest_ssc_source_file 100% 8192MB 112.2MB/s 01:13
Transfer from source to destination server:
[mora@haddock ~]$ scp ricoh:/home/exports/nfstest_ssc_source_file /dev/null
nfstest_ssc_source_file 100% 8192MB 110.7MB/s 01:14
Disk I/O on source server:
[mora@ricoh ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@ricoh ~]$ dd if=/home/exports/nfstest_ssc_source_file of=/dev/null bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 56.8953 s, 151 MB/s
[mora@ricoh ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@ricoh ~]$ dd if=/dev/zero of=/home/exports/test bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 46.9064 s, 183 MB/s
Disk I/O on destination server:
[mora@haddock ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@haddock ~]$ dd if=/home/exports/nfstest_ssc_source_file of=/dev/null bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 57.3356 s, 150 MB/s
[mora@haddock ~]$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
[mora@haddock ~]$ dd if=/dev/zero of=/home/exports/test bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 47.3809 s, 181 MB/s
> > All machines are running 4.10.0-ssc-02242017.1345,
> > 4.10.0 kernel on top RHEL 7.3 with Olga’s SSC patches 02242017.tgz (asynchronous SSC)
> > Test:
> > The test runs 10 times for each copy size and the average is compared.
> > The servers are mounted and un-mounted for each copy (using default mount options)
> > Testing copy size: 1KB – 4GB, NOTE: the last copy is 4GB-1 bytes, the maximum copy_file_range() accepts.
>
> If the client unmounts the servers, but if the servers keep their
> exports mounted then the performance may depend on whether data is
> already cached on the source server, since all these copies fit in RAM.
Good point, I will clear the VM caches using "echo 3 > /proc/sys/vm/drop_caches"
on both servers before each copy.
> Are you timing just the copy_file_range() call, or do you include a
> following sync?
I am timing right before calling copy_file_range() up to doing an fsync() and close() of the destination file.
For the traditional copy is the same, I am timing right before the first read on the source file up to the
fsync() and close() of the destination file.
> > Results:
> > For a copy size below 16MB, traditional copy runs faster than server-side copy
> > For a copy size of 32MB and above, server-side copy is at least 30% faster.
> > For a copy size of 128MB and above, server-side copy is about 50% faster.
> > For the 1GB and 2GB copy sizes, the performance improvement is only about 30-40%, investigating why this is happening.
>
> Might also be interesting to look at performance when copying a larger
> file with multiple copy_file_range() calls.
I will do this as well.
--Jorge
> --b.
>
> > *** Inter-SSC performance test
> > TEST: Running test 'perf02'
> >
> > Performance degradation:
> > INFO: 00:08:50.035165 - perf02 copy with size 1KB
> > INFO: 00:09:01.302004 - Server-side COPY: 0.104049992561 seconds
> > INFO: 00:09:01.302206 - Traditional COPY: 0.0319610118866 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 1KB file: 225%
> > INFO: 00:09:01.302378 - perf02 copy with size 2KB
> > INFO: 00:09:12.332144 - Server-side COPY: 0.0923588514328 seconds
> > INFO: 00:09:12.332335 - Traditional COPY: 0.033596777916 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 2KB file: 174%
> > INFO: 00:09:12.332510 - perf02 copy with size 4KB
> > INFO: 00:09:23.320714 - Server-side COPY: 0.100715565681 seconds
> > INFO: 00:09:23.320915 - Traditional COPY: 0.0336240530014 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 4KB file: 199%
> > INFO: 00:09:23.321086 - perf02 copy with size 8KB
> > INFO: 00:09:34.542515 - Server-side COPY: 0.0881641149521 seconds
> > INFO: 00:09:34.542742 - Traditional COPY: 0.0336608886719 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 8KB file: 161%
> > INFO: 00:09:34.542924 - perf02 copy with size 16KB
> > INFO: 00:09:45.439424 - Server-side COPY: 0.0939898014069 seconds
> > INFO: 00:09:45.439630 - Traditional COPY: 0.0277063846588 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 16KB file: 239%
> > INFO: 00:09:45.439795 - perf02 copy with size 32KB
> > INFO: 00:09:56.103480 - Server-side COPY: 0.0798309087753 seconds
> > INFO: 00:09:56.103691 - Traditional COPY: 0.0335722208023 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 32KB file: 137%
> > INFO: 00:09:56.103848 - perf02 copy with size 64KB
> > INFO: 00:10:07.217026 - Server-side COPY: 0.106491327286 seconds
> > INFO: 00:10:07.217228 - Traditional COPY: 0.034387588501 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 64KB file: 209%
> > INFO: 00:10:07.217400 - perf02 copy with size 128KB
> > INFO: 00:10:18.022320 - Server-side COPY: 0.105627346039 seconds
> > INFO: 00:10:18.022516 - Traditional COPY: 0.0511318922043 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 128KB file: 106%
> > INFO: 00:10:18.022686 - perf02 copy with size 256KB
> > INFO: 00:10:29.452353 - Server-side COPY: 0.124787020683 seconds
> > INFO: 00:10:29.452538 - Traditional COPY: 0.0552408218384 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 256KB file: 125%
> > INFO: 00:10:29.452713 - perf02 copy with size 512KB
> > INFO: 00:10:41.357315 - Server-side COPY: 0.110602092743 seconds
> > INFO: 00:10:41.357489 - Traditional COPY: 0.0643961668015 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 512KB file: 71%
> > INFO: 00:10:41.357656 - perf02 copy with size 1MB
> > INFO: 00:10:52.704159 - Server-side COPY: 0.118950200081 seconds
> > INFO: 00:10:52.704341 - Traditional COPY: 0.0703304767609 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 1MB file: 69%
> > INFO: 00:10:52.704499 - perf02 copy with size 2MB
> > INFO: 00:11:05.367265 - Server-side COPY: 0.13633646965 seconds
> > INFO: 00:11:05.367451 - Traditional COPY: 0.103643512726 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 2MB file: 31%
> > INFO: 00:11:05.367614 - perf02 copy with size 4MB
> > INFO: 00:11:18.872004 - Server-side COPY: 0.181165075302 seconds
> > INFO: 00:11:18.872204 - Traditional COPY: 0.164586114883 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 4MB file: 10%
> > INFO: 00:11:18.872368 - perf02 copy with size 8MB
> > INFO: 00:11:35.093071 - Server-side COPY: 0.29020409584 seconds
> > INFO: 00:11:35.093297 - Traditional COPY: 0.283315181732 seconds
> > FAIL: SSC should outperform traditional copy, performance degradation for a 8MB file: 2%
> >
> > Performance gain:
> > INFO: 00:11:35.093446 - perf02 copy with size 16MB
> > INFO: 00:11:54.779844 - Server-side COPY: 0.455569577217 seconds
> > INFO: 00:11:54.780038 - Traditional COPY: 0.506252598763 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 16MB file: 11%
> > INFO: 00:11:54.780185 - perf02 copy with size 32MB
> > INFO: 00:12:22.415131 - Server-side COPY: 0.71369125843 seconds
> > INFO: 00:12:22.415319 - Traditional COPY: 0.93473637104 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 32MB file: 30%
> > INFO: 00:12:22.415466 - perf02 copy with size 64MB
> > INFO: 00:13:03.498260 - Server-side COPY: 1.23845338821 seconds
> > INFO: 00:13:03.498456 - Traditional COPY: 1.73098192215 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 64MB file: 39%
> > INFO: 00:13:03.498637 - perf02 copy with size 128MB
> > INFO: 00:14:11.502475 - Server-side COPY: 2.22639911175 seconds
> > INFO: 00:14:11.502896 - Traditional COPY: 3.34778087139 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 128MB file: 50%
> > INFO: 00:14:11.503102 - perf02 copy with size 256MB
> > INFO: 00:16:12.323350 - Server-side COPY: 4.29401872158 seconds
> > INFO: 00:16:12.323537 - Traditional COPY: 6.54622249603 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 256MB file: 52%
> > INFO: 00:16:12.323703 - perf02 copy with size 512MB
> > INFO: 00:19:58.311598 - Server-side COPY: 8.3841770649 seconds
> > INFO: 00:19:58.311793 - Traditional COPY: 12.9200412273 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 512MB file: 54%
> > INFO: 00:19:58.311935 - perf02 copy with size 1GB
> > INFO: 00:26:01.275809 - Server-side COPY: 14.5967838049 seconds
> > INFO: 00:26:01.276020 - Traditional COPY: 20.1957453728 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 1GB file: 38%
> > INFO: 00:26:01.276175 - perf02 copy with size 2GB
> > INFO: 00:35:29.990220 - Server-side COPY: 23.606795764 seconds
> > INFO: 00:35:29.990432 - Traditional COPY: 31.4261539459 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 2GB file: 33%
> > INFO: 00:35:29.990580 - perf02 copy with size 4GB
> > INFO: 00:53:17.198827 - Server-side COPY: 41.2261408091 seconds
> > INFO: 00:53:17.199033 - Traditional COPY: 62.9210350513 seconds
> > PASS: SSC should outperform traditional copy, performance improvement for a 4GB file: 52%
> > TIME: 44m27.164286s
> >
> > Logfile: /home/mora/logs/nfstest_ssc_20170412000849.log
> >
> > 23 tests (9 passed, 14 failed)
> >
> > Total time: 44m28.334177s
> >
> >
> >
> > --Jorge
> >
��.n��������+%������w��{.n�����{��w��� jg��� �����ݢj����G���� ���j:+v���w�m������w�������h�����٥
