On 29 January 2011 23:26, CoolCold <coolthecold@xxxxxxxxx> wrote: > You may need to increase stripe cache size > http://peterkieser.com/2009/11/29/raid-mdraid-stripe_cache_size-vs-write-transfer/ > > On Sun, Jan 30, 2011 at 1:48 AM, Mathias BurÃn <mathias.buren@xxxxxxxxx> wrote: >> Hi, >> >> I'm wondering if the performance I'm getting is OK or if there's >> something I can do about it. Also, where the potential bottlenecks >> are. >> >> Setup: 6x2TB HDDs, their performance: >> >> /dev/sdb: >> ÂTiming cached reads:  1322 MB in Â2.00 seconds = 661.51 MB/sec >> ÂTiming buffered disk reads: 362 MB in Â3.02 seconds = 120.06 MB/sec >> >> /dev/sdc: >> ÂTiming cached reads:  1282 MB in Â2.00 seconds = 641.20 MB/sec >> ÂTiming buffered disk reads: 342 MB in Â3.01 seconds = 113.53 MB/sec >> >> /dev/sdd: >> ÂTiming cached reads:  1282 MB in Â2.00 seconds = 640.55 MB/sec >> ÂTiming buffered disk reads: 344 MB in Â3.00 seconds = 114.58 MB/sec >> >> /dev/sde: >> ÂTiming cached reads:  1328 MB in Â2.00 seconds = 664.46 MB/sec >> ÂTiming buffered disk reads: 350 MB in Â3.01 seconds = 116.37 MB/sec >> >> /dev/sdf: >> ÂTiming cached reads:  1304 MB in Â2.00 seconds = 651.55 MB/sec >> ÂTiming buffered disk reads: 378 MB in Â3.01 seconds = 125.62 MB/sec >> >> /dev/sdg: >> ÂTiming cached reads:  1324 MB in Â2.00 seconds = 661.91 MB/sec >> ÂTiming buffered disk reads: 400 MB in Â3.00 seconds = 133.15 MB/sec >> >> These are used in a RAID5 setup: >> >> Personalities : [raid6] [raid5] [raid4] >> md0 : active raid5 sdf1[0] sdg1[6] sde1[5] sdc1[3] sdd1[4] sdb1[1] >>   Â9751756800 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] >> >> unused devices: <none> >> >> /dev/md0: >>    ÂVersion : 1.2 >> ÂCreation Time : Tue Oct 19 08:58:41 2010 >>   Raid Level : raid5 >>   Array Size : 9751756800 (9300.00 GiB 9985.80 GB) >> ÂUsed Dev Size : 1950351360 (1860.00 GiB 1997.16 GB) >>  Raid Devices : 6 >> ÂTotal Devices : 6 >>  ÂPersistence : Superblock is persistent >> >>  ÂUpdate Time : Fri Jan 28 14:55:48 2011 >>     ÂState : clean >> ÂActive Devices : 6 >> Working Devices : 6 >> ÂFailed Devices : 0 >> ÂSpare Devices : 0 >> >>     Layout : left-symmetric >>   Chunk Size : 64K >> >>      Name : ion:0 Â(local to host ion) >>      UUID : e6595c64:b3ae90b3:f01133ac:3f402d20 >>     Events : 3035769 >> >>  ÂNumber  Major  Minor  RaidDevice State >>    0    8    81    Â0   Âactive sync  /dev/sdf1 >>    1    8    17    Â1   Âactive sync  /dev/sdb1 >>    4    8    49    Â2   Âactive sync  /dev/sdd1 >>    3    8    33    Â3   Âactive sync  /dev/sdc1 >>    5    8    65    Â4   Âactive sync  /dev/sde1 >>    6    8    97    Â5   Âactive sync  /dev/sdg1 >> >> As you can see they are partitioned. They are all identical like this: >> >> Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes >> 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 512 bytes >> I/O size (minimum/optimal): 512 bytes / 512 bytes >> Disk identifier: 0x0e5b3a7a >> >>  Device Boot   ÂStart     End   ÂBlocks  Id ÂSystem >> /dev/sdb1      Â2048 Â3907029167 Â1953513560  fd ÂLinux raid autodetect >> >> On this I run LVM: >> >> Â--- Physical volume --- >> ÂPV Name        /dev/md0 >> ÂVG Name        lvstorage >> ÂPV Size        9.08 TiB / not usable 1.00 MiB >> ÂAllocatable      yes (but full) >> ÂPE Size        1.00 MiB >> ÂTotal PE       Â9523199 >> ÂFree PE        0 >> ÂAllocated PE     Â9523199 >> ÂPV UUID        YLEUKB-klxF-X3gF-6dG3-DL4R-xebv-6gKQc2 >> >> On top of the LVM I have: >> >> Â--- Volume group --- >> ÂVG Name        lvstorage >> ÂSystem ID >> ÂFormat        Âlvm2 >> ÂMetadata Areas    Â1 >> ÂMetadata Sequence No Â6 >> ÂVG Access       read/write >> ÂVG Status       resizable >> ÂMAX LV        Â0 >> ÂCur LV        Â1 >> ÂOpen LV        1 >> ÂMax PV        Â0 >> ÂCur PV        Â1 >> ÂAct PV        Â1 >> ÂVG Size        9.08 TiB >> ÂPE Size        1.00 MiB >> ÂTotal PE       Â9523199 >> ÂAlloc PE / Size    9523199 / 9.08 TiB >> ÂFree ÂPE / Size    0 / 0 >> ÂVG UUID        Xd0HTM-azdN-v9kJ-C7vD-COcU-Cnn8-6AJ6hI >> >> And in turn: >> >> Â--- Logical volume --- >> ÂLV Name        Â/dev/lvstorage/storage >> ÂVG Name        Âlvstorage >> ÂLV UUID        Â9wsJ0u-0QMs-lL5h-E2UA-7QJa-l46j-oWkSr3 >> ÂLV Write Access    Âread/write >> ÂLV Status       Âavailable >> Â# open         1 >> ÂLV Size        Â9.08 TiB >> ÂCurrent LE       9523199 >> ÂSegments        1 >> ÂAllocation       inherit >> ÂRead ahead sectors   auto >> Â- currently set to   1280 >> ÂBlock device      254:1 >> >> And on that (sorry) there's the ext4 partition: >> >> /dev/mapper/lvstorage-storage on /raid5volume type ext4 >> (rw,noatime,barrier=1,nouser_xattr) >> >> Here are the numbers: >> >> /raid5volume $ time dd if=/dev/zero of=./bigfile.tmp bs=1M count=8192 >> 8192+0 records in >> 8192+0 records out >> 8589934592 bytes (8.6 GB) copied, 94.0967 s, 91.3 MB/s >> >> real  Â1m34.102s >> user  Â0m0.107s >> sys   0m54.693s >> >> /raid5volume $ time dd if=./bigfile.tmp of=/dev/null bs=1M >> 8192+0 records in >> 8192+0 records out >> 8589934592 bytes (8.6 GB) copied, 37.8557 s, 227 MB/s >> >> real  Â0m37.861s >> user  Â0m0.053s >> sys   0m23.608s >> >> I saw that the process md0_raid5 spike sometimes on CPU usage. This is >> an Atom @ 1.6GHz, is that what is limiting the results? Here's >> bonnie++: >> >> /raid5volume/temp $ time bonnie++ -d ./ -m ion >> Writing with putc()...done >> Writing intelligently...done >> Rewriting...done >> Reading with getc()...done >> Reading intelligently...done >> start 'em...done...done...done... >> Create files in sequential order...done. >> Stat files in sequential order...done. >> Delete files in sequential order...done. >> Create files in random order...done. >> Stat files in random order...done. >> Delete files in random order...done. >> Version 1.03e    ------Sequential Output------ --Sequential Input- --Random- >>          Â-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Machine    ÂSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP Â/sec %CP >> ion       Â7G 13726 Â98 148051 Â87 68020 Â41 14547 Â99 286647 >> 61 404.1  2 >>          Â------Sequential Create------ --------Random Create-------- >>          Â-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >>       Âfiles Â/sec %CP Â/sec %CP Â/sec %CP Â/sec %CP Â/sec %CP Â/sec %CP >>         16 20707 Â99 +++++ +++ 25870 Â99 21242 Â98 +++++ +++ 25630 100 >> ion,7G,13726,98,148051,87,68020,41,14547,99,286647,61,404.1,2,16,20707,99,+++++,+++,25870,99,21242,98,+++++,+++,25630,100 >> >> real  Â20m54.320s >> user  Â16m10.447s >> sys   2m45.543s >> >> >> Thanks in advance, >> // Mathias >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at Âhttp://vger.kernel.org/majordomo-info.html >> > > > > -- > Best regards, > [COOLCOLD-RIPN] > I ran the benchmark found on the page (except for writes); results: tripe_cache_size: 256 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.4351 s, 271 MB/s stripe_cache_size: 256 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.8224 s, 287 MB/s stripe_cache_size: 256 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.1066 s, 277 MB/s stripe_cache_size: 512 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.6833 s, 288 MB/s stripe_cache_size: 512 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.3497 s, 285 MB/s stripe_cache_size: 512 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.7853 s, 287 MB/s stripe_cache_size: 768 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.5398 s, 288 MB/s stripe_cache_size: 768 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.1769 s, 285 MB/s stripe_cache_size: 768 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.5354 s, 284 MB/s stripe_cache_size: 1024 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.1814 s, 285 MB/s stripe_cache_size: 1024 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.6288 s, 279 MB/s stripe_cache_size: 1024 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.9942 s, 277 MB/s stripe_cache_size: 2048 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.177 s, 281 MB/s stripe_cache_size: 2048 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.3905 s, 280 MB/s stripe_cache_size: 2048 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.0274 s, 281 MB/s stripe_cache_size: 4096 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.607 s, 274 MB/s stripe_cache_size: 4096 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.1505 s, 272 MB/s stripe_cache_size: 4096 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.4747 s, 279 MB/s stripe_cache_size: 8192 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.0839 s, 277 MB/s stripe_cache_size: 8192 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.7944 s, 274 MB/s stripe_cache_size: 8192 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.4443 s, 280 MB/s stripe_cache_size: 16834 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.9554 s, 277 MB/s stripe_cache_size: 16834 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.8002 s, 269 MB/s stripe_cache_size: 16834 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.2772 s, 276 MB/s stripe_cache_size: 32768 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.4692 s, 275 MB/s stripe_cache_size: 32768 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.6707 s, 279 MB/s stripe_cache_size: 32768 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.4744 s, 271 MB/s It looks like a small stripe cache is favoured here. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
