On Sat, 5 Feb 2011, Stan Hoeppner wrote:
Justin Piszcz put forth on 2/5/2011 4:56 PM:
Not sure how to copy/paste from an IPMI window, but I made my own
kernel for 2.6.37 in CentOS 5.5 (a pain) and now it is doing ~482-500MiB/s
sustained with a single copy (netcat from A->B). Poor performance with the
default 2.6.18 kernel. Seems to also slow down over time, down to 434MiB/s now,
but it started very quick and it remains between 420-500MiB/s sustained. Now
435-445MiB/s..
I forgot you mentioned CentOS. Their kernel and apps are always very old.
2.6.18 was release in Sept 2006 IIRC--4+ years ago. It was the "pirate themed"
release.
With 2.6.37 what do you get with 4 concurrent nfs copy ops? If the aggregate
nfs throughput doesn't increase, you need to tweak your nfs server (and possibly
client). With that hardware and a recent kernel you should be able to fill that
10 GbE pipe, or come really close.
Hi,
I installed Debian Testing & my own kernel again (2.6.37):
One thread:
# get bigfile.1
`bigfile.1' at 1168441344 (5%) 493.10M/s eta:38s [Receiving data]
`bigfile.1' at 2226847744 (10%) 557.16M/s eta:32s [Receiving data]
`bigfile.1' at 3274768384 (15%) 578.57M/s eta:29s [Receiving data]
`bigfile.1' at 4781113344 (22%) 585.02M/s eta:26s [Receiving data]
`bigfile.1' at 6294077440 (30%) 588.96M/s eta:24s [Receiving data]
`bigfile.1' at 9318563840 (44%) 592.87M/s eta:19s [Receiving data]
`bigfile.1' at 12805996544 (61%) 592.46M/s eta:13s [Receiving data]
20971520000 bytes transferred in 34 seconds (592.72M/s)
Two threads:
[0] get bigfile.1
`bigfile.1' at 3225878528 (15%) 306.51M/s eta:55s [Receiving data]
[1] get bigfile.2
`bigfile.2' at 3200516096 (15%) 306.49M/s eta:55s [Receiving data]
Seems like some problem achieving > 600MiB/s now.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 34184 112 3875676 0 0 622592 0 49060 9798 0 5 95 0
0 0 0 32820 112 3876976 0 0 622592 0 49043 9745 0 5 95 0
0 0 0 34184 112 3876188 0 0 622592 0 48997 9714 0 5 95 0
1 0 0 32572 112 3877780 0 0 606208 0 48815 9537 0 5 95 0
0 0 0 32820 112 3878104 0 0 622592 0 48968 9741 0 5 95 0
0 0 0 31456 112 3878024 0 0 622592 0 49051 9773 0 5 95 0
0 0 0 31952 112 3877448 0 0 622592 4 49076 9804 0 5 95 0
3 0 0 30836 112 3871364 0 0 606208 0 48901 9650 0 5 95 0
But much better than 250MiB/s.
[0] Done (get bigfile.1)
20971520000 bytes transferred in 64 seconds (312.95M/s)
[1] Done (get bigfile.2)
20971520000 bytes transferred in 64 seconds (312.78M/s)
So approx 624MiB/s..
With three threads:
[0] get bigfile.1
`bigfile.1' at 6659241132 (31%) 222.41M/s eta:61s [Receiving data]
[1] get bigfile.2
`bigfile.2' at 6620446720 (31%) 222.00M/s eta:62s [Receiving data]
[2] get bigfile.3
`bigfile.3' at 6601441280 (31%) 222.17M/s eta:62s [Receiving data]
vmstat:
0 0 0 33028 112 3877856 0 0 704512 0 53859 13108 0 6 94 0
1 0 0 32532 112 3877476 0 0 688128 0 54063 12484 0 6 94 0
1 0 0 33276 112 3877128 0 0 704512 0 54143 12548 0 6 94 0
1 0 0 33284 112 3868860 0 0 700672 0 54200 12398 0 6 94 0
[0] Done (get bigfile.1)
20971520000 bytes transferred in 90 seconds (221.70M/s)
[1] Done (get bigfile.2)
20971520000 bytes transferred in 90 seconds (221.50M/s)
[2] Done (get bigfile.3)
20971520000 bytes transferred in 90 seconds (221.42M/s)
A little better, 663MiB/s.
Four threads:
[0] get bigfile.1
`bigfile.1' at 2641887232 (12%) 162.77M/s eta:2m [Receiving data]
[1] get bigfile.2
`bigfile.2' at 2601254912 (12%) 161.97M/s eta:2m [Receiving data]
[2] get bigfile.3
`bigfile.3' at 2592931840 (12%) 162.05M/s eta:2m [Receiving data]
[3] get bigfile.4
`bigfile.4' at 2546794496 (12%) 159.92M/s eta:2m [Receiving data]
vmstat:
1 0 0 34492 112 3859748 0 0 678144 0 58816 11481 0 7 93 0
0 0 0 35360 112 3871060 0 0 681728 0 58889 11344 0 6 94 0
1 0 0 33872 112 3874252 0 0 688128 0 59052 11560 0 6 94 0
0 0 0 34492 112 3871400 0 0 688128 0 59079 11564 0 7 93 0
1 0 0 32136 112 3874888 0 0 688128 0 59089 11432 0 6 93 0
2 0 0 35360 112 3872436 0 0 655360 0 56672 10943 0 7 93 0
[0] Done (get bigfile.1)
20971520000 bytes transferred in 120 seconds (166.72M/s)
[1] Done (get bigfile.2)
20971520000 bytes transferred in 120 seconds (166.42M/s)
[2] Done (get bigfile.3)
20971520000 bytes transferred in 120 seconds (166.40M/s)
[3] Done (get bigfile.4)
20971520000 bytes transferred in 120 seconds (166.30M/s)
664MiB/s, so it appears this is the best it can do without any major tweaking
other than setting the blockdev readahead to 16384.
Overall performance is good, but now I need to figure out how to tweak the
network and/or disk paths so I can achieve > 1Gbyte/sec, as the RAID-0s can
read and write > 1.2Gbyte/sec.
Has anyone with 10GbE <-> 10GbE been able to transfer at or near line rate
with a single connection, or > 700MiB/s with multiple connections? Problem
I worry about with creating multiple connections between two machines it will
create contention within the RAID that is being read from..
Justin.
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
