On Mon, 8 Sep 2003 at 11:18pm, Aaron Lehmann wrote
> Well, I'm really sorry I didn't run these benchmarks. In the process
> of installing these new drives, something happened to my old SCSI disk
> such that it isn't detected on the bus. This is why I wanted RAID in
> the first place! In addition, one of the new drives appears to be
> defective, which will delay the process.
>
> It hasn't been my day.
While these aren't exactly what you had in mind, I recently did some
benchmarking on a big 3ware based system (see attached). The system
has dual 2.4GHz Xeons (with HT on), 2GB RAM, 2 7500-8 boards, and 16
Hitachi 180GB drives. For the software RAID, I did a RAID5 across 15 of
the disks (leaving one for a hot spare) with a 64k chunk-size (same as the
3wares do in hardware). For the hardware RAID, I did a RAID5 with hot
spare on each card, and a software RAID0 stripe (512k chunk size) across
the cards.
Unfortunately, the tests *are* with two different kernels (well, three
really -- I tested the HW RAID config with two different kernels). All
were patched with XFS, and all the tests were done on an XFS partition.
--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University
Version 1.02c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
buckbeak 8G 25517 99 141407 73 98245 59 27676 99 360682 87 478.0 2
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 1112 10 +++++ +++ 993 9 1112 11 +++++ +++ 695 14
buckbeak,8G,25517,99,141407,73,98245,59,27676,99,360682,87,478.0,2,16,1112,10,+++++,+++,993,9,1112,11,+++++,+++,695,14
[jlb@buckbeak tiobench-0.3.3]$ ./tiobench.pl --size 8192
Run #1: ./tiotest -t 8 -f 1024 -r 500 -b 4096 -d . -TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 1 267.57 86.00% 0.014 94.80 0.00000 0.00000 311
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 2 294.90 121.6% 0.025 48.80 0.00000 0.00000 242
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 4 172.77 120.6% 0.089 209.19 0.00000 0.00000 143
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 8 166.20 110.4% 0.169 6843.29 0.00081 0.00000 151
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 1 0.79 0.952% 4.970 18.40 0.00000 0.00000 82
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 2 1.47 1.490% 5.207 26.07 0.00000 0.00000 99
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 4 2.44 2.500% 6.123 36.94 0.00000 0.00000 98
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 8 3.63 3.902% 7.549 43.57 0.00000 0.00000 93
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 1 145.55 85.55% 0.025 718.62 0.00000 0.00000 170
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 2 131.19 143.2% 0.054 2557.99 0.00010 0.00000 92
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 4 93.70 134.3% 0.151 5748.75 0.00043 0.00000 70
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 8 67.09 103.6% 0.418 7592.24 0.00372 0.00000 65
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 1 2.57 1.447% 0.011 0.10 0.00000 0.00000 178
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 2 1.35 1.127% 0.017 0.11 0.00000 0.00000 119
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 4 2.41 3.914% 0.029 0.15 0.00000 0.00000 62
2.4.18-18SGI_XFS_1.2.0smp 8192 4096 8 2.14 3.565% 0.029 5.15 0.00000 0.00000 60
HW RAID, 2.4.21-xfs (1.3 release)
[jlb@buckbeak jlb]$ bonnie++ -s 8192
Version 1.02c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
buckbeak 8G 25283 96 131142 39 83869 32 28038 99 342849 68 444.1 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2644 21 +++++ +++ 2227 17 2548 20 +++++ +++ 2118 21
buckbeak,8G,25283,96,131142,39,83869,32,28038,99,342849,68,444.1,1,16,2644,21,+++++,+++,2227,17,2548,20,+++++,+++,2118,21
[jlb@buckbeak tiobench-0.3.3]$ ./tiobench.pl --size 8192
Run #1: ./tiotest -t 8 -f 1024 -r 500 -b 4096 -d . -TT
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.21-xfs 8192 4096 1 340.95 87.56% 0.011 22.33 0.00000 0.00000 389
2.4.21-xfs 8192 4096 2 294.01 91.05% 0.026 76.29 0.00000 0.00000 323
2.4.21-xfs 8192 4096 4 251.19 80.82% 0.060 95.17 0.00000 0.00000 311
2.4.21-xfs 8192 4096 8 255.43 84.03% 0.119 266.16 0.00000 0.00000 304
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.21-xfs 8192 4096 1 0.76 0.340% 5.132 25.90 0.00000 0.00000 223
2.4.21-xfs 8192 4096 2 1.37 0.437% 5.487 21.35 0.00000 0.00000 312
2.4.21-xfs 8192 4096 4 2.32 1.190% 6.452 42.38 0.00000 0.00000 195
2.4.21-xfs 8192 4096 8 3.50 1.568% 8.275 41.85 0.00000 0.00000 223
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.21-xfs 8192 4096 1 116.36 44.09% 0.031 737.00 0.00000 0.00000 264
2.4.21-xfs 8192 4096 2 63.30 28.41% 0.114 4194.28 0.00057 0.00000 223
2.4.21-xfs 8192 4096 4 43.49 24.81% 0.331 11465.55 0.00310 0.00005 175
2.4.21-xfs 8192 4096 8 38.26 25.88% 0.749 14738.81 0.01159 0.00038 148
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.21-xfs 8192 4096 1 1.64 0.838% 0.011 0.08 0.00000 0.00000 195
2.4.21-xfs 8192 4096 2 1.68 1.183% 0.017 0.16 0.00000 0.00000 142
2.4.21-xfs 8192 4096 4 1.63 1.776% 0.028 2.80 0.00000 0.00000 92
2.4.21-xfs 8192 4096 8 1.58 2.330% 0.034 18.03 0.00000 0.00000 68
HW RAID, 2.4.20-19.7.XFS1.3.0smp
[jlb@buckbeak tiobench-0.3.3]$ ./tiobench.pl --size 4096
Run #1: ./tiotest -t 8 -f 512 -r 500 -b 4096 -d . -T-T
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.20-19.7.XFS1.3.0smp 4096 4096 1 276.27 81.68% 0.013 99.82 0.00000 0.00000 338
2.4.20-19.7.XFS1.3.0smp 4096 4096 2 275.17 102.6% 0.027 106.56 0.00000 0.00000 268
2.4.20-19.7.XFS1.3.0smp 4096 4096 4 225.11 108.5% 0.067 256.05 0.00000 0.00000 207
2.4.20-19.7.XFS1.3.0smp 4096 4096 8 221.65 111.8% 0.132 217.67 0.00000 0.00000 198
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.20-19.7.XFS1.3.0smp 4096 4096 1 1.13 2.676% 3.454 54.82 0.00000 0.00000 42
2.4.20-19.7.XFS1.3.0smp 4096 4096 2 1.93 16.33% 4.002 53.27 0.00000 0.00000 12
2.4.20-19.7.XFS1.3.0smp 4096 4096 4 2.94 20.87% 4.971 78.67 0.00000 0.00000 14
2.4.20-19.7.XFS1.3.0smp 4096 4096 8 4.61 26.52% 5.989 103.03 0.00000 0.00000 17
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.20-19.7.XFS1.3.0smp 4096 4096 1 58.92 23.21% 0.054 2981.43 0.00038 0.00000 254
2.4.20-19.7.XFS1.3.0smp 4096 4096 2 44.03 22.14% 0.150 4731.74 0.00114 0.00000 199
2.4.20-19.7.XFS1.3.0smp 4096 4096 4 33.53 26.27% 0.382 6946.62 0.00458 0.00000 128
2.4.20-19.7.XFS1.3.0smp 4096 4096 8 29.41 24.99% 0.889 13357.19 0.01450 0.00010 118
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.20-19.7.XFS1.3.0smp 4096 4096 1 1.66 0.531% 0.011 0.11 0.00000 0.00000 313
2.4.20-19.7.XFS1.3.0smp 4096 4096 2 1.67 1.071% 0.016 0.13 0.00000 0.00000 156
2.4.20-19.7.XFS1.3.0smp 4096 4096 4 1.63 1.873% 0.029 4.73 0.00000 0.00000 87
2.4.20-19.7.XFS1.3.0smp 4096 4096 8 1.50 2.117% 0.025 1.25 0.00000 0.00000 71
[jlb@buckbeak tmp]$ bonnie++ -s 8192
Version 1.02c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
buckbeak 8G 20485 76 55244 17 27481 10 27383 97 365660 81 446.0 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2168 19 +++++ +++ 2710 23 1196 11 +++++ +++ 3178 32
buckbeak,8G,20485,76,55244,17,27481,10,27383,97,365660,81,446.0,1,16,2168,19,+++++,+++,2710,23,1196,11,+++++,+++,3178,32
