On 02/03/11 20:09, Bart Kus wrote:
Hello,
I have the following setup:
md_RAID6(10x2TB) -> LVM2 -> cryptsetup -> XFS
When copying data onto the target XFS, I notice a large number of
READs occurring on the physical hard drives. Is there any way of
monitoring what might be causing these read ops?
I have setup the system to minimize read-modify-write cycles as best I
can, but I fear I've missed some possible options in LVM2 or
cryptsetup. Here are the specifics:
11:43:54 sde 162.00 12040.00 34344.00 286.32 0.40 2.47 1.67 27.00
11:43:54 sdf 170.00 12008.00 36832.00 287.29 0.62 3.65 2.12 36.00
11:43:54 sdg 185.00 10552.00 37920.00 262.01 0.49 2.65 1.84 34.00
11:43:54 sdh 152.00 11824.00 37304.00 323.21 0.29 1.78 1.71 26.00
11:43:54 sdi 140.00 13016.00 35216.00 344.51 0.68 4.71 3.21 45.00
11:43:54 sdj 181.00 11784.00 36240.00 265.33 0.43 2.38 1.55 28.00
11:43:54 sds 162.00 11824.00 34040.00 283.11 0.46 2.84 1.67 27.00
11:43:54 sdt 157.00 11264.00 35192.00 295.90 0.65 4.14 2.29 36.00
11:43:54 sdu 154.00 12584.00 35424.00 311.74 0.46 2.79 1.69 26.00
11:43:54 sdv 131.00 12800.00 33264.00 351.63 0.39 2.75 1.98 26.00
11:43:54 md5 752.00 0.00 153688.00 204.37 0.00 0.00 0.00 0.00
11:43:54 DayTar-DayTar 752.00 0.00 153688.00 204.37 12.42 16.76 1.33
100.00
11:43:54 data 0.00 0.00 0.00 0.00 7238.71 0.00 0.00 100.00
Where md5 is the RAID6 holding the drives right above it,
DayTar-DayTar are the VG and LV respectively, and data is the
cryptsetup device derived from the LV. Hard drives are set to
"blockdev --setra 1024", md5 is set for stripe_cache_size of 6553 and
preread_bypass_threshold of 0. XFS is mounted with the following options:
/dev/mapper/data on /data type xfs
(rw,noatime,nodiratime,allocsize=256m,nobarrier,noikeep,inode64,logbufs=8,logbsize=256k)
And here are the format options of XFS:
meta-data=/dev/mapper/data isize=256 agcount=15, agsize=268435455 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=3906993152, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
I wasn't sure how to do any kind of stripe alignment with the md RAID6
given the layers in between. Here are the LVM2 properties:
--- Physical volume ---
PV Name /dev/md5
VG Name DayTar
PV Size 14.55 TiB / not usable 116.00 MiB
Allocatable yes (but full)
PE Size 256.00 MiB
Total PE 59616
Free PE 0
Allocated PE 59616
PV UUID jwcRz9-Yl0k-OHRQ-p5yR-AbAP-j09z-PCgSFo
--- Volume group ---
VG Name DayTar
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 14.55 TiB
PE Size 256.00 MiB
Total PE 59616
Alloc PE / Size 59616 / 14.55 TiB
Free PE / Size 0 / 0
VG UUID X8gbkZ-BOMq-D6x2-xx6y-r2wF-cePQ-JTKZQs
--- Logical volume ---
LV Name /dev/DayTar/DayTar
VG Name DayTar
LV UUID cdebg4-EcCR-6QR7-sAhT-EN1h-20Lv-qIFSH8
LV Write Access read/write
LV Status available
# open 1
LV Size 14.55 TiB
Current LE 59616
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 16384
Block device 253:0
And finally the cryptsetup properties:
/dev/mapper/data is active:
cipher: aes-cbc-essiv:sha256
keysize: 256 bits
device: /dev/mapper/DayTar-DayTar
offset: 8192 sectors
size: 31255945216 sectors
mode: read/write
Anyone have any suggestions on how to tune this to do better at pure
writing by eliminating needless reading?
Thanks,
--Bart
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
I once used a tool called dstat. dstat has modules which can tell you
which processes are using disk IO. I haven’t used dstat in a while so
maybe someone else can chime in
Cheers
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
