On 3/2/2011 2:59 PM, NeilBrown wrote:
On Wed, 02 Mar 2011 14:26:02 -0800 Bart Kus<me@xxxxxxxx> wrote:
Hello,
This seems contradictory:
jo ~ # blockdev --getiomin /dev/md5
524288
jo ~ # blockdev --getioopt /dev/md5
4194304
jo ~ # blockdev --getmaxsect /dev/md5
255
jo ~ # blockdev --getbsz /dev/md5
4096
jo ~ # blockdev --getss /dev/md5
512
jo ~ #
Optimal IO size is reported as 4MB (and that is indeed the stripe size),
but maximum sectors per request is only 128kB? How can software do
optimal 4MB IOs with the maxsect limit? Does XFS care about this limit?
md/raid doesn't use 'requests' the any maximum is meaningless.
raid4/5/6 does have a 'stripe cache' which is a vaguely similar thing. There
can sometimes be value in changing that.
The devices that the array are built from may have a 'maximum sectors per
request', but that probably isn't particularly related to chunk size.
BTW, where do you find "maximum sectors per request in only 128kB" in the
details you quoted - I don't see it.
I got the notion from seeing blockdev's help text:
--getmaxsect get max sectors per request
Seeing 255 sectors there, and multiplying by sector size (512B), you get
128kB (ok, 127.5kB).
The reason for my curiosity here is that I have a large linear write
load to this RAID6 md array, and I'm seeing about 25% of the IOs being
reads! I've configured the stripe cache size to 6553: 4096*6553 is
about 26MB/device, and there's 10 devices, so let's say 256MB total
stripe cache size. During this large linear write I mostly see "128" in
stripe_cache_active, which is 128*4096 = 512kB, which is the same as
chunk size. So given all this, here are the reads I'm seeing in "sar
-pd 1":
14:06:20 DEV tps rd_sec/s wr_sec/s avgrq-sz
avgqu-sz await svctm %util
[...snip!...]
14:06:21 sde 219.00 11304.00 30640.00 191.53
1.15 5.16 2.10 46.00
14:06:21 sdf 209.00 11016.00 29904.00 195.79
1.06 5.02 2.01 42.00
14:06:21 sdg 178.00 11512.00 28568.00 225.17
0.74 3.99 2.08 37.00
14:06:21 sdh 175.00 10736.00 26832.00 214.67
0.89 4.91 2.00 35.00
14:06:21 sdi 206.00 11512.00 29112.00 197.20
0.83 3.98 1.80 37.00
14:06:21 sdj 209.00 11264.00 30264.00 198.70
0.79 3.78 1.96 41.00
14:06:21 sds 214.00 10984.00 28552.00 184.75
0.78 3.60 1.78 38.00
14:06:21 sdt 194.00 13352.00 27808.00 212.16
0.83 4.23 1.91 37.00
14:06:21 sdu 183.00 12856.00 28872.00 228.02
0.60 3.22 2.13 39.00
14:06:21 sdv 189.00 11984.00 31696.00 231.11
0.57 2.96 1.69 32.00
14:06:21 md5 754.00 0.00 153848.00 204.04
0.00 0.00 0.00 0.00
14:06:21 DayTar-DayTar 753.00 0.00 153600.00 203.98
15.73 20.58 1.33 100.00
14:06:21 data 760.00 0.00 155800.00 205.00
4670.84 6070.91 1.32 100.00
The setup is md5 -> /dev/DayTar/DayTar (LVM2 VG / LV) ->
/dev/mapper/data (cryptsetup) -> XFS.
The avgrq-sz column shows about 205 (in sectors), which is about 105kB.
You can see there are NO reads in the layers leading up to the hard
drives. The reads look to be getting generated inside md, and look to
represent about 25% of IO load to the HDs. For large linear writes,
should there really be these reads? Shouldn't avgrq-sz be showing about
8192 sectors (4MB)? This data is what prompted my question about
maxsect since it seems to indicate 128kB is a limit.
Thanks for any insight,
--Bart
PS: I tried to force XFS to write big with the following mount options:
/dev/mapper/data on /data type xfs
(rw,noatime,nodiratime,allocsize=256m,nobarrier,noikeep,inode64,logbufs=8,logbsize=256k,sunit=1024,swidth=8192)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
