On Sun, Jan 31, 2010 at 12:21 PM, Aryeh Gregor <Simetrical+list@xxxxxxxxx> wrote: > On Fri, Jan 22, 2010 at 11:37 AM, Goswin von Brederlow > <goswin-v-b@xxxxxx> wrote: >> I think his initial idea of raid1 is verry good. With write-mostly all >> reads should come from the fast SSD. I would also add write-behind. >> That way writes will not wait for the rotating disk and get the full >> SSD speed as well. > > Recently I got an Intel X25-M 80G, and have been trying this out. I > figured some people here might be interested in the results. I've > mirrored the SSD to a slow 80G partition on an old spinning-rust 160G > disk, which is set to write-mostly, with write-behind of 16384 (the > max). The bitmap is on a separate unmirrored partition on the SSD. > My root filesystem is on it, ext4 on top of LVM, using 60G out of > 73.77G allocated to LVM, with a separate 500M-ish mirrored boot > partition at the start, and a gap in between for storing unmirrored > things (like all my RAID bitmaps). > > I did six runs of bonnie++ overnight: > > 1) 128 files, both disks working. > 2) 1024 files, both disks working. > 3) 128 files, conventional disk failed and removed. > 4) 1024 files, conventional disk failed and removed. > 5) Same as (2). > 6) Same as (4). (Note that in this case, the disk ended up being > failed but not removed, but this should make no difference.) > > At the end of this e-mail, I've attached the full script I ran and its > full output. These are the bonnie++ numbers for the six tests, best > viewed in a fixed-width font on a pretty wide screen (126 chars): > > | Sequential out | Sequential in | Random | > | Sequential create | Random create | > |per char| block |rewrite |per char| block | Seeks | > | Create | Read | Delete | Create | Read |Delete | > monoid,6G,47571,90,55343,30,41390,24,55803,96,233594,54,4923.0,36, > 128,25973,94,+++++,+++,15910,41,27179,85,+++++,+++,11922,38 > monoid,6G,48358,90,55378,25,41338,25,56883,95,233689,52,4233.0,29,1024,19819,70,64728, > 75, 848, 4,14928,60,55484, 79, 463, 3 > monoid,6G,48751,90,77485,27,45263,25,55660,94,238757,49,5107.7,33, > 128,28027,86,+++++,+++,20161,49,29618,88,+++++,+++,14780,43 > monoid,6G,44503,83,73894,29,42975,26,56463,95,237173,53,5052.9,36,1024,18073,73,66151, > 81, 5348,28,18102,65,60213, 77, 3441,23 > monoid,6G,48580,91,55567,24,40748,22,56803,95,237006,49,4828.2,29,1024,20332,67,73175, > 80, 783, 4,13615,45,58066, 74, 489, 3 > monoid,6G,48966,92,77517,29,43999,24,56863,94,198557,38,3553.8,16,1024,19273,68,72414, > 86, 6326,31,16499,60,56438, 75, 2784,15 > > (Where two numbers are given, the first is KB out per second and the > second is CPU usage, as far as I can figure.) Note that there was lots > of memory free here, about 2.5G (-/+ buffers/cache as reported by > free). My conclusions: > > 1) The file creation tests with only 128 files should just be ignored, > they look like they're in-memory. The 1024-file tests look more > plausible. > > 2) All reads perform comparably whether the extra disk is present or > not, as expected. There's some variation, but then, I wasn't being > very scientific. > > 3) Tests 3, 4, and 6 (only SSD) are about 35% faster on the sequential > out rewrite test. The per-char sequential out test was CPU-bound, and > everything performed the same. Everything also performed the same on > the rewrite test; I'm not sure why. > > 4) Sequential file creation is about the same between tests 4 and 6 > (only SSD) and tests 2 and 5 (both disks). Random file creation gives > the disk+SSD about a 20% lead. But both sequential and random > deletion have an order-of-magnitude difference between the two. I > don't know why this might be -- lots of buffering for some operations > but not others? > > Overall, this seems like a very feasible setup, and I'll certainly be > sticking with it, even though it will obviously slow down some writes. > Hope this data will be useful (or at least interesting) to someone. > > The script that I ran from the at job follows, with a couple of > comments added. I ran it using ionice -c1 to reduce the effects of > any concurrent operations. > > > > #!/bin/bash > cd /tmp > echo Test 1; echo > free -m > bonnie -u 0 -n 128 > echo; echo Test 2; echo > free -m > echo > bonnie -u 0 -n 1024 > echo; echo Test 3; echo > free -m > echo > mdadm --fail /dev/md1 /dev/sda2 > sleep 5 > mdadm --remove /dev/md1 /dev/sda2 > echo > bonnie -u 0 -n 128 > echo; echo Test 4; echo > free -m > echo > bonnie -u 0 -n 1024 > echo > free -m > echo; echo 'Re-adding sda2 and waiting for sync'; echo > sudo mdadm --add /dev/md1 --write-mostly /dev/sda2 > sleep 1800 > cat /proc/mdstat > echo; echo Test 5; echo > free -m > echo > bonnie -u 0 -n 1024 > echo > mdadm --fail /dev/md1 /dev/sda2 > # The second command was run too fast and failed, I forgot the sleep 5 > from above. > mdadm --remove /dev/md1 /dev/sda2 > echo; echo Test 6; echo > free -m > echo > bonnie -u 0 -n 1024 > echo > free -m > echo > # Also failed, since the device hadn't been removed. > sudo mdadm --add /dev/md1 --write-mostly /dev/sda2 > > > > > Output of script: > > > > > Test 1 > > total used free shared buffers cached > Mem: 3024 2085 938 0 589 973 > -/+ buffers/cache: 522 2501 > Swap: 1023 526 497 > Using uid:0, gid:0. > Writing with putc()...done > Writing intelligently...done > Rewriting...done > Reading with getc()...done > Reading intelligently...done > start 'em...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.03c ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > monoid 6G 47571 90 55343 30 41390 24 55803 96 233594 54 4923 36 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 128 25973 94 +++++ +++ 15910 41 27179 85 +++++ +++ 11922 38 > monoid,6G,47571,90,55343,30,41390,24,55803,96,233594,54,4923.0,36,128,25973,94,+++++,+++,15910,41,27179,85,+++++,+++,11922,38 > > Test 2 > > total used free shared buffers cached > Mem: 3024 544 2480 0 51 30 > -/+ buffers/cache: 462 2562 > Swap: 1023 526 497 > > Using uid:0, gid:0. > Writing with putc()...done > Writing intelligently...done > Rewriting...done > Reading with getc()...done > Reading intelligently...done > start 'em...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.03c ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > monoid 6G 48358 90 55378 25 41338 25 56883 95 233689 52 4233 29 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 1024 19819 70 64728 75 848 4 14928 60 55484 79 463 3 > monoid,6G,48358,90,55378,25,41338,25,56883,95,233689,52,4233.0,29,1024,19819,70,64728,75,848,4,14928,60,55484,79,463,3 > > Test 3 > > total used free shared buffers cached > Mem: 3024 788 2235 0 332 50 > -/+ buffers/cache: 405 2619 > Swap: 1023 565 458 > > mdadm: set /dev/sda2 faulty in /dev/md1 > mdadm: hot removed /dev/sda2 > > Using uid:0, gid:0. > Writing with putc()...done > Writing intelligently...done > Rewriting...done > Reading with getc()...done > Reading intelligently...done > start 'em...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.03c ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > monoid 6G 48751 90 77485 27 45263 25 55660 94 238757 49 5108 33 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 128 28027 86 +++++ +++ 20161 49 29618 88 +++++ +++ 14780 43 > monoid,6G,48751,90,77485,27,45263,25,55660,94,238757,49,5107.7,33,128,28027,86,+++++,+++,20161,49,29618,88,+++++,+++,14780,43 > > Test 4 > > total used free shared buffers cached > Mem: 3024 441 2582 0 48 25 > -/+ buffers/cache: 367 2656 > Swap: 1023 594 429 > > Using uid:0, gid:0. > Writing with putc()...done > Writing intelligently...done > Rewriting...done > Reading with getc()...done > Reading intelligently...done > start 'em...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.03c ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > monoid 6G 44503 83 73894 29 42975 26 56463 95 237173 53 5053 36 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 1024 18073 73 66151 81 5348 28 18102 65 60213 77 3441 23 > monoid,6G,44503,83,73894,29,42975,26,56463,95,237173,53,5052.9,36,1024,18073,73,66151,81,5348,28,18102,65,60213,77,3441,23 > > total used free shared buffers cached > Mem: 3024 734 2289 0 329 42 > -/+ buffers/cache: 362 2661 > Swap: 1023 609 414 > > Re-adding sda2 and waiting for sync > > mdadm: re-added /dev/sda2 > Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] > [raid4] [raid10] > md2 : active raid1 sdb1[0] sda1[1](W) > 521984 blocks [2/2] [UU] > bitmap: 0/64 pages [0KB], 4KB chunk, file: /mnt/ssd-junk/boot-bitmap > > md3 : active raid10 sdc2[2] sdd2[1] > 488102656 blocks super 1.2 64K chunks 2 far-copies [2/2] [UU] > bitmap: 1/466 pages [4KB], 512KB chunk, file: /mnt/ssd-junk/extra-bitmap > > md1 : active raid1 sda2[1](W) sdb2[0] > 77352896 blocks [2/2] [UU] > bitmap: 29/296 pages [116KB], 128KB chunk, file: /mnt/ssd-junk/root-bitmap > > unused devices: <none> > > Test 5 > > total used free shared buffers cached > Mem: 3024 1374 1649 0 357 621 > -/+ buffers/cache: 396 2627 > Swap: 1023 602 421 > > Using uid:0, gid:0. > Writing with putc()...done > Writing intelligently...done > Rewriting...done > Reading with getc()...done > Reading intelligently...done > start 'em...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.03c ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > monoid 6G 48580 91 55567 24 40748 22 56803 95 237006 49 4828 29 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 1024 20332 67 73175 80 783 4 13615 45 58066 74 489 3 > monoid,6G,48580,91,55567,24,40748,22,56803,95,237006,49,4828.2,29,1024,20332,67,73175,80,783,4,13615,45,58066,74,489,3 > > mdadm: set /dev/sda2 faulty in /dev/md1 > mdadm: hot remove failed for /dev/sda2: Device or resource busy > > Test 6 > > total used free shared buffers cached > Mem: 3024 720 2303 0 335 40 > -/+ buffers/cache: 345 2678 > Swap: 1023 653 370 > > Using uid:0, gid:0. > Writing with putc()...done > Writing intelligently...done > Rewriting...done > Reading with getc()...done > Reading intelligently...done > start 'em...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.03c ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > monoid 6G 48966 92 77517 29 43999 24 56863 94 198557 38 3554 16 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 1024 19273 68 72414 86 6326 31 16499 60 56438 75 2784 15 > monoid,6G,48966,92,77517,29,43999,24,56863,94,198557,38,3553.8,16,1024,19273,68,72414,86,6326,31,16499,60,56438,75,2784,15 > > total used free shared buffers cached > Mem: 3024 2174 849 0 545 1261 > -/+ buffers/cache: 366 2657 > Swap: 1023 674 349 > > mdadm: Cannot open /dev/sda2: Device or resource busy > Thanks for providing this information, its quite useful. I suspect I'll be trying something like this myself in the near future. Wil -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
