Good morning,
Here are the promised benchmarks for the ddraid cluster raid 3.5.
Test configuration:
- 1 GHz PIII, 1 GB memory
- 5 scsi disks in a hotplug scsi backplane, 39 GB/sec each
- ddraid server running on same machine
- AF_UNIX network connection to server from dm target
Two tests from opposite ends of the disk load spectrum:
1) untar linux-2.6.11.3 to root dir
- Many small, nonlinear transfers
2) cp linux-2.6.11.3 to root dir
- A few large, linear transfers
Unmount after each test. Unmounting flushes the cache, which
constitutes some or all of the actual disk IO. So for each test, two
timings are shown, one for the test and one for the unmount.
To see where the ddraid overheads lie, each ddraid test was run in four
combinations of parity calculation enabled or disabled, and network
synchronization enabled or disabled, as indicated. Some tests were
done both with a 3 disk (ddraid order 1) and 5 disk (ddraid order 2)
array. Some of the 5 disk tests failed to complete (a timer BUG, not
investigated yet).
DDraid synchronization includes not only network message traffic, but
recording the region-by-region dirty state persistently to disk before
completing each write IO. Parity calculations are performed both on
write and read (the later to double-check for errors).
First, the fragmented load:
untar linux-2.6.11.3
====================
raw scsi disk
process: real 48.994s user 45.526s sys 3.063s
umount: real 3.084s user 0.002s sys 0.429s
raw scsi disk (again)
process: real 48.603s user 45.380s sys 2.976s
umount: real 3.145s user 0.005s sys 0.421s
ddraid order 1, no calc, no sync
process: real 49.942s user 46.328s sys 3.028s
umount: real 2.034s user 0.005s sys 0.626s
ddraid order 1, calc, no sync
process: real 50.864s user 46.221s sys 3.195s
umount: real 1.839s user 0.006s sys 1.099s
ddraid order 1, calc, sync
process: real 50.979s user 46.382s sys 3.222s
umount: real 1.895s user 0.002s sys 0.531s
ddraid order 2, no calc, no sync
process: real 49.532s user 45.837s sys 3.145s
umount: real 1.318s user 0.004s sys 0.718s
ddraid order 2, calc, no sync
process: real 49.742s user 45.527s sys 3.135s
umount: real 1.625s user 0.004s sys 1.054s
ddraid order 2, calc, sync: <oops>
Interpretation: Fragmented IO to the ddraid array runs in about the same
time as IO to a single, raw disk, regardless of whether the persistent
dirty is running or not, and regardless of parity calculations or not.
Unmount time is consistently faster for ddraid than for the raw disk,
giving ddraid a slight edge over the raw disk overall.
I had hoped to show considerably more improvement over the raw disk
here, but there are per-transfer overheads that prevent the array from
running at full speed. I suspect these overheads are not in the ddraid
code, but we shall see. The main point is: ddraid is not slower than a
draw disk for this demanding load.
Next, the linear load:
cp /zoo/linux-2.6.11.3.tar.bz2 /x
raw scsi disk
process: real 0.258s user 0.008s sys 0.236s
umount: real 1.019s user 0.003s sys 0.032s
raw scsi disk (again)
process: real 0.264s user 0.013s sys 0.237s
umount: real 1.053s user 0.005s sys 0.029s
raw scsi disk (again)
process: real 0.267s user 0.018s sys 0.233s
umount: real 1.019s user 0.006s sys 0.028s
ddraid order 1, calc, no sync
process: real 0.267s user 0.007s sys 0.243s
umount: real 0.568s user 0.006s sys 0.250s
ddraid order 1, no calc, sync
process: real 0.267s user 0.011s sys 0.240s
umount: real 0.608s user 0.002s sys 0.032s
ddraid order 1, calc, sync
process: real 0.265s user 0.008s sys 0.239s
umount: real 0.596s user 0.004s sys 0.042s
ddraid order 2, no calc, no sync
process: real 0.266s user 0.013s sys 0.234s
umount: real 0.381s user 0.004s sys 0.049s
ddraid order 2, calc, no sync
process: real 0.269s user 0.010s sys 0.239s
umount: real 0.392s user 0.004s sys 0.201s
ddraid order 2, no calc, sync: <oops>
ddraid order 2, calc, sync: <oops>
Interpretation: DDraid really shines on the linear load, beating the raw
disk by 41% with a 3 disk array and 62% with a 5 disk array.
Bottom line: Even with the overhead of running a persistent dirty log
and doing parity calculations, ddraid is not slower than a single raw
disk on worst case fragmented loads, and is dramatically faster on best
case linear loads. In my opinion, this performance is good enough for
production use. Besides the performance advantage, there is of course
the redundancy, which is actually the main point. (Note however, that
there are some bugs and miscellaneous deficiencies remaining to be
ironed out.)
There is still a lot room for further optimization. It is likely that
ddraid performance can eventually approach the aggregate throughput of
the array, less the parity disk.
Regards,
Daniel
