|
[Not sure if this is the right forum - apologies in advance if it's
not...] I'm running some LVM/DM scaling performance runs on a 4-way IA64 box (HP RX2600), that is equipped with four dual-U320 SCSI host bus adapters. Each bus has 7 U320 disk drives (15K) attached for a total of 56 drives. [Each dual-U320 HBA is capable of doing on the order of 512MB/sec sustained throughput, I've measured the system using direct IO (no LVM) at being able to read data at up to 1.8GB/sec.] The system has RHEL4 Update 1 Beta bits installed (2.6.9 based kernel). The test program is the SPEW benchmark, and was run with the DIRECT IO flag set. There were ten (10) LVM volumes used, they were all constructed with a 16KB stripe size. The volumes were - Volume 1 - 1 disk Volume 2 - 2 disks, each disk on a separate U320 bus Volume 3 - 3 disks, each disk on a separate U320 bus ... Volume 8 - 8 disks, each disk on a separate U320 bus Volume 9 - 9 disks, 1 disk on 7 buses, 2 disks each on 1 U320 bus Volume 10 - 10 disks, 1 disk on 6 buses, 2 disks each on 2 U320 buses I then ran 4 groups of tests, with each group consisting of a separate run against each of the 10 volumes. The groups had a block size associated with it based upon the number of disks in a volume. Thus the first group of runs had a block size of 16KB times the number of disks in the volume. [The idea being to have a single transfer split into one 16KB against each of the disks in the volume.] The second group had a block size of 32KB times the number of disks in the volume. [The idea here was to see the effects of the wrapping - for example, for Volume 2 with 2 disks (striped at 16KB) we would expect each 64KB initial transfer to be split into four 16KB-transfers, with the first and third delivered to the first drive in the volume, and the second and fourth transfer going to the second drive in the volume.] The third group had a block size of 48KB times the number of disks, and the fourth had a block size of 64KB times the number of disks in the volume. The expectation was to see that the overall performance would be at least as good or hopefully even better as the overall transfer size increased. [If for no other reasons except that (a) the amount of time spent doing IO transfers at the device level should dwarf any overhead at the application/LVM(DM) layers, and (b) the targetted devices could be given multiple commands and thus reduce idle time.] However, that is *not* what was seen. The accompanying xmgrace-generated chart illustrates this (I've also attached the xmgrace file itself, in case you are having problems with the .png file included...):  Here we see the black line showing scaling along the first group (16KB per segmenet/disk per transfer) - in fact we see good scaling up to 4 disks (pretty linear), with a slight hiccup, and then it continues onwards at less of an increase. Something strange at 5 disks happens, and again at 9/10 disks - the latter perhaps due to the fact that we're wrapping on the disks now (some adapters have two disks busy at a time). I'm going to look into the actual bus/target mappings for the volumes to see if they play a role. But note, when we go to group 2 (32KB per segment per transfer), the performance is always slightly lower up to 3 disks, and then starting at 4 disks we see almost a flattening out (with a big dip at volume 6). We do see some (slight) scaling again from 32KB to 48KB and then 64KB, but we never again attain the rate of simply issuing the 16KB per segment initial requests. It's interesting to see the dip at 6-disks per volume for all the transfer sizes that wrap. I've been investigating this, but was wondering if anyone had noticed something like this - or perhaps had a ready explanation for this... Alan D. Brunelle HP |
# Grace project file # @version 50117 @page size 792, 612 @page scroll 5% @page inout 5% @link page off @map font 0 to "Times-Roman", "Times-Roman" @map font 1 to "Times-Italic", "Times-Italic" @map font 2 to "Times-Bold", "Times-Bold" @map font 3 to "Times-BoldItalic", "Times-BoldItalic" @map font 4 to "Helvetica", "Helvetica" @map font 5 to "Helvetica-Oblique", "Helvetica-Oblique" @map font 6 to "Helvetica-Bold", "Helvetica-Bold" @map font 7 to "Helvetica-BoldOblique", "Helvetica-BoldOblique" @map font 8 to "Courier", "Courier" @map font 9 to "Courier-Oblique", "Courier-Oblique" @map font 10 to "Courier-Bold", "Courier-Bold" @map font 11 to "Courier-BoldOblique", "Courier-BoldOblique" @map font 12 to "Symbol", "Symbol" @map font 13 to "ZapfDingbats", "ZapfDingbats" @map color 0 to (255, 255, 255), "white" @map color 1 to (0, 0, 0), "black" @map color 2 to (255, 0, 0), "red" @map color 3 to (0, 255, 0), "green" @map color 4 to (0, 0, 255), "blue" @map color 5 to (255, 255, 0), "yellow" @map color 6 to (188, 143, 143), "brown" @map color 7 to (220, 220, 220), "grey" @map color 8 to (148, 0, 211), "violet" @map color 9 to (0, 255, 255), "cyan" @map color 10 to (255, 0, 255), "magenta" @map color 11 to (255, 165, 0), "orange" @map color 12 to (114, 33, 188), "indigo" @map color 13 to (103, 7, 72), "maroon" @map color 14 to (64, 224, 208), "turquoise" @map color 15 to (0, 139, 0), "green4" @reference date 0 @date wrap off @date wrap year 1950 @default linewidth 1.0 @default linestyle 1 @default color 1 @default pattern 1 @default font 0 @default char size 1.000000 @default symbol size 1.000000 @default sformat "%.8g" @background color 0 @page background fill on @timestamp off @timestamp 0.03, 0.03 @timestamp color 1 @timestamp rot 0 @timestamp font 0 @timestamp char size 1.000000 @timestamp def "Wed Jun 29 16:48:16 2005" @r0 off @link r0 to g0 @r0 type above @r0 linestyle 1 @r0 linewidth 1.0 @r0 color 1 @r0 line 0, 0, 0, 0 @r1 off @link r1 to g0 @r1 type above @r1 linestyle 1 @r1 linewidth 1.0 @r1 color 1 @r1 line 0, 0, 0, 0 @r2 off @link r2 to g0 @r2 type above @r2 linestyle 1 @r2 linewidth 1.0 @r2 color 1 @r2 line 0, 0, 0, 0 @r3 off @link r3 to g0 @r3 type above @r3 linestyle 1 @r3 linewidth 1.0 @r3 color 1 @r3 line 0, 0, 0, 0 @r4 off @link r4 to g0 @r4 type above @r4 linestyle 1 @r4 linewidth 1.0 @r4 color 1 @r4 line 0, 0, 0, 0 @g0 on @g0 hidden false @g0 type XY @g0 stacked false @g0 bar hgap 0.000000 @g0 fixedpoint off @g0 fixedpoint type 0 @g0 fixedpoint xy 0.000000, 0.000000 @g0 fixedpoint format general general @g0 fixedpoint prec 6, 6 @with g0 @ world 0.5, 0, 10.5, 400 @ stack world 0, 0, 0, 0 @ znorm 1 @ view 0.150000, 0.150000, 1.150000, 0.850000 @ title "IA64 4-way w/ U320 SPEW LVM" @ title font 0 @ title size 1.500000 @ title color 1 @ subtitle "" @ subtitle font 0 @ subtitle size 1.000000 @ subtitle color 1 @ xaxes scale Normal @ yaxes scale Normal @ xaxes invert off @ yaxes invert off @ xaxis on @ xaxis type zero false @ xaxis offset 0.000000 , 0.000000 @ xaxis bar on @ xaxis bar color 1 @ xaxis bar linestyle 1 @ xaxis bar linewidth 1.0 @ xaxis label "Segments per volume" @ xaxis label layout para @ xaxis label place auto @ xaxis label char size 1.000000 @ xaxis label font 0 @ xaxis label color 1 @ xaxis label place normal @ xaxis tick on @ xaxis tick major 1 @ xaxis tick minor ticks 0 @ xaxis tick default 6 @ xaxis tick place rounded true @ xaxis tick in @ xaxis tick major size 1.000000 @ xaxis tick major color 1 @ xaxis tick major linewidth 1.0 @ xaxis tick major linestyle 1 @ xaxis tick major grid off @ xaxis tick minor color 1 @ xaxis tick minor linewidth 1.0 @ xaxis tick minor linestyle 1 @ xaxis tick minor grid off @ xaxis tick minor size 0.500000 @ xaxis ticklabel on @ xaxis ticklabel format decimal @ xaxis ticklabel prec 0 @ xaxis ticklabel formula "" @ xaxis ticklabel append "" @ xaxis ticklabel prepend "" @ xaxis ticklabel angle 0 @ xaxis ticklabel skip 0 @ xaxis ticklabel stagger 0 @ xaxis ticklabel place normal @ xaxis ticklabel offset auto @ xaxis ticklabel offset 0.000000 , 0.010000 @ xaxis ticklabel start type auto @ xaxis ticklabel start 0.000000 @ xaxis ticklabel stop type auto @ xaxis ticklabel stop 0.000000 @ xaxis ticklabel char size 1.000000 @ xaxis ticklabel font 0 @ xaxis ticklabel color 1 @ xaxis tick place both @ xaxis tick spec type none @ yaxis on @ yaxis type zero false @ yaxis offset 0.000000 , 0.000000 @ yaxis bar on @ yaxis bar color 1 @ yaxis bar linestyle 1 @ yaxis bar linewidth 1.0 @ yaxis label "MBytes per second" @ yaxis label layout para @ yaxis label place auto @ yaxis label char size 1.000000 @ yaxis label font 0 @ yaxis label color 1 @ yaxis label place normal @ yaxis tick on @ yaxis tick major 100 @ yaxis tick minor ticks 3 @ yaxis tick default 6 @ yaxis tick place rounded true @ yaxis tick in @ yaxis tick major size 1.000000 @ yaxis tick major color 1 @ yaxis tick major linewidth 1.0 @ yaxis tick major linestyle 3 @ yaxis tick major grid on @ yaxis tick minor color 1 @ yaxis tick minor linewidth 1.0 @ yaxis tick minor linestyle 2 @ yaxis tick minor grid on @ yaxis tick minor size 0.500000 @ yaxis ticklabel on @ yaxis ticklabel format general @ yaxis ticklabel prec 5 @ yaxis ticklabel formula "" @ yaxis ticklabel append "" @ yaxis ticklabel prepend "" @ yaxis ticklabel angle 0 @ yaxis ticklabel skip 0 @ yaxis ticklabel stagger 0 @ yaxis ticklabel place both @ yaxis ticklabel offset auto @ yaxis ticklabel offset 0.000000 , 0.010000 @ yaxis ticklabel start type auto @ yaxis ticklabel start 0.000000 @ yaxis ticklabel stop type auto @ yaxis ticklabel stop 0.000000 @ yaxis ticklabel char size 1.000000 @ yaxis ticklabel font 0 @ yaxis ticklabel color 1 @ yaxis tick place both @ yaxis tick spec type none @ altxaxis off @ altyaxis off @ legend on @ legend loctype view @ legend 0.22, 0.8 @ legend box color 1 @ legend box pattern 1 @ legend box linewidth 1.0 @ legend box linestyle 1 @ legend box fill color 0 @ legend box fill pattern 1 @ legend font 0 @ legend char size 0.800000 @ legend color 1 @ legend length 4 @ legend vgap 1 @ legend hgap 1 @ legend invert false @ frame type 0 @ frame linestyle 1 @ frame linewidth 1.0 @ frame color 1 @ frame pattern 1 @ frame background color 0 @ frame background pattern 0 @ s0 hidden false @ s0 type xy @ s0 symbol 1 @ s0 symbol size 1.000000 @ s0 symbol color 1 @ s0 symbol pattern 1 @ s0 symbol fill color 1 @ s0 symbol fill pattern 0 @ s0 symbol linewidth 1.0 @ s0 symbol linestyle 1 @ s0 symbol char 65 @ s0 symbol char font 0 @ s0 symbol skip 0 @ s0 line type 1 @ s0 line linestyle 1 @ s0 line linewidth 1.0 @ s0 line color 1 @ s0 line pattern 1 @ s0 baseline type 0 @ s0 baseline off @ s0 dropline off @ s0 fill type 0 @ s0 fill rule 0 @ s0 fill color 1 @ s0 fill pattern 1 @ s0 avalue off @ s0 avalue type 2 @ s0 avalue char size 1.000000 @ s0 avalue font 0 @ s0 avalue color 1 @ s0 avalue rot 0 @ s0 avalue format general @ s0 avalue prec 3 @ s0 avalue prepend "" @ s0 avalue append "" @ s0 avalue offset 0.000000 , 0.000000 @ s0 errorbar on @ s0 errorbar place both @ s0 errorbar color 1 @ s0 errorbar pattern 1 @ s0 errorbar size 1.000000 @ s0 errorbar linewidth 1.0 @ s0 errorbar linestyle 1 @ s0 errorbar riser linewidth 1.0 @ s0 errorbar riser linestyle 1 @ s0 errorbar riser clip off @ s0 errorbar riser clip length 0.100000 @ s0 comment "/work/common/data/lvm/2005_06_29/results.dat" @ s0 legend "16KB per segment transfers" @ s1 hidden false @ s1 type xy @ s1 symbol 1 @ s1 symbol size 1.000000 @ s1 symbol color 2 @ s1 symbol pattern 1 @ s1 symbol fill color 2 @ s1 symbol fill pattern 0 @ s1 symbol linewidth 1.0 @ s1 symbol linestyle 1 @ s1 symbol char 65 @ s1 symbol char font 0 @ s1 symbol skip 0 @ s1 line type 1 @ s1 line linestyle 1 @ s1 line linewidth 1.0 @ s1 line color 2 @ s1 line pattern 1 @ s1 baseline type 0 @ s1 baseline off @ s1 dropline off @ s1 fill type 0 @ s1 fill rule 0 @ s1 fill color 1 @ s1 fill pattern 1 @ s1 avalue off @ s1 avalue type 2 @ s1 avalue char size 1.000000 @ s1 avalue font 0 @ s1 avalue color 1 @ s1 avalue rot 0 @ s1 avalue format general @ s1 avalue prec 3 @ s1 avalue prepend "" @ s1 avalue append "" @ s1 avalue offset 0.000000 , 0.000000 @ s1 errorbar on @ s1 errorbar place both @ s1 errorbar color 2 @ s1 errorbar pattern 1 @ s1 errorbar size 1.000000 @ s1 errorbar linewidth 1.0 @ s1 errorbar linestyle 1 @ s1 errorbar riser linewidth 1.0 @ s1 errorbar riser linestyle 1 @ s1 errorbar riser clip off @ s1 errorbar riser clip length 0.100000 @ s1 comment "/work/common/data/lvm/2005_06_29/results.dat" @ s1 legend "32KB" @ s2 hidden false @ s2 type xy @ s2 symbol 1 @ s2 symbol size 1.000000 @ s2 symbol color 3 @ s2 symbol pattern 1 @ s2 symbol fill color 3 @ s2 symbol fill pattern 0 @ s2 symbol linewidth 1.0 @ s2 symbol linestyle 1 @ s2 symbol char 65 @ s2 symbol char font 0 @ s2 symbol skip 0 @ s2 line type 1 @ s2 line linestyle 1 @ s2 line linewidth 1.0 @ s2 line color 3 @ s2 line pattern 1 @ s2 baseline type 0 @ s2 baseline off @ s2 dropline off @ s2 fill type 0 @ s2 fill rule 0 @ s2 fill color 1 @ s2 fill pattern 1 @ s2 avalue off @ s2 avalue type 2 @ s2 avalue char size 1.000000 @ s2 avalue font 0 @ s2 avalue color 1 @ s2 avalue rot 0 @ s2 avalue format general @ s2 avalue prec 3 @ s2 avalue prepend "" @ s2 avalue append "" @ s2 avalue offset 0.000000 , 0.000000 @ s2 errorbar on @ s2 errorbar place both @ s2 errorbar color 3 @ s2 errorbar pattern 1 @ s2 errorbar size 1.000000 @ s2 errorbar linewidth 1.0 @ s2 errorbar linestyle 1 @ s2 errorbar riser linewidth 1.0 @ s2 errorbar riser linestyle 1 @ s2 errorbar riser clip off @ s2 errorbar riser clip length 0.100000 @ s2 comment "/work/common/data/lvm/2005_06_29/results.dat" @ s2 legend "48KB" @ s3 hidden false @ s3 type xy @ s3 symbol 1 @ s3 symbol size 1.000000 @ s3 symbol color 4 @ s3 symbol pattern 1 @ s3 symbol fill color 4 @ s3 symbol fill pattern 0 @ s3 symbol linewidth 1.0 @ s3 symbol linestyle 1 @ s3 symbol char 65 @ s3 symbol char font 0 @ s3 symbol skip 0 @ s3 line type 1 @ s3 line linestyle 1 @ s3 line linewidth 1.0 @ s3 line color 4 @ s3 line pattern 1 @ s3 baseline type 0 @ s3 baseline off @ s3 dropline off @ s3 fill type 0 @ s3 fill rule 0 @ s3 fill color 1 @ s3 fill pattern 1 @ s3 avalue off @ s3 avalue type 2 @ s3 avalue char size 1.000000 @ s3 avalue font 0 @ s3 avalue color 1 @ s3 avalue rot 0 @ s3 avalue format general @ s3 avalue prec 3 @ s3 avalue prepend "" @ s3 avalue append "" @ s3 avalue offset 0.000000 , 0.000000 @ s3 errorbar on @ s3 errorbar place both @ s3 errorbar color 4 @ s3 errorbar pattern 1 @ s3 errorbar size 1.000000 @ s3 errorbar linewidth 1.0 @ s3 errorbar linestyle 1 @ s3 errorbar riser linewidth 1.0 @ s3 errorbar riser linestyle 1 @ s3 errorbar riser clip off @ s3 errorbar riser clip length 0.100000 @ s3 comment "/work/common/data/lvm/2005_06_29/results.dat" @ s3 legend "64KB" @target G0.S0 @type xy 1 70.52 2 138.21 3 201.13 4 271.44 5 275.3 6 319.66 7 356 8 366.55 9 327.93 10 362.51 & @target G0.S1 @type xy 1 64.62 2 103.68 3 190.05 4 202.64 5 210.48 6 160.71 7 171.36 8 170.12 9 188.88 10 206.91 & @target G0.S2 @type xy 1 70.32 2 106.69 3 190.09 4 209.55 5 251.58 6 185.25 7 210.11 8 224.61 9 218.05 10 219.72 & @target G0.S3 @type xy 1 70.41 2 109.8 3 190.6 4 216.79 5 264.03 6 202.39 7 225.53 8 240.77 9 252.18 10 245.6 &
