On Wed October 14 2009, mark delfman wrote:
> Hi Chris... we tried the direct DD as requested and the problem is
> still there...
> 1.3GBsec > 325MBsec (even more dromatic)... hopefully this helps
> narrow it down?
>
>
> Write > MD
> linux-poly:~ # dd if=/dev/zero of=/dev/md0 oflag=direct bs=1M count=20000
> 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 15.7671 s, 1.3 GB/s
>
>
> Write > XFS > MD
> linux-poly:~ # dd if=/dev/zero of=/mnt/md0/test oflag=direct bs=1M
> count=20000 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 64.616 s, 325 MB/s
If it helps, I'm seeing the same sort of thing.
The most I can seemingly tweak out of my new 5x1TB array is 170MB/s write.
Using dd with oflags=direct drops it down to 31MB/s.
Oddly, I see spikes of over 200MB/s write when not using oflags=direct,
but it slows down in between to 11MB/s so over all,
it averages a max of 170MB/s. the device itself is capable of over 500MB/s.
(66% drop?)
small test:
$ dd if=/dev/zero of=/mnt/test-data/test.file bs=512KiB count=4096 oflag=direct
4096+0 records in
4096+0 records out
2147483648 bytes (2.1 GB) copied, 71.8088 s, 29.9 MB/s
$ dd if=/dev/zero of=/mnt/test-data/test.file bs=512KiB count=4096
4096+0 records in
4096+0 records out
2147483648 bytes (2.1 GB) copied, 19.7101 s, 109 MB/s
$ sudo dd if=/dev/md0 of=/tmp/test-data.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.39796 s, 448 MB/s
$ sudo dd if=/tmp/test-data.img of=/dev/md0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.05666 s, 522 MB/s
$ cd /mnt/test-data/test
$ iozone -A -s4G -y512k -q512k
...
KB reclen write rewrite read reread
4194304 512 161732 333316 382361 388726
[snip]
>
info, if it helps:
# mdadm -D /dev/md0
/dev/md0:
Version : 1.01
Creation Time : Wed Oct 14 08:55:25 2009
Raid Level : raid5
Array Size : 3907049472 (3726.05 GiB 4000.82 GB)
Used Dev Size : 976762368 (931.51 GiB 1000.20 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Tue Oct 27 04:18:50 2009
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : natasha:0 (local to host natasha)
UUID : 7d0e9847:ec3a4a46:32b60a80:06d0ee1c
Events : 4952
Number Major Minor RaidDevice State
0 8 64 0 active sync /dev/sde
1 8 80 1 active sync /dev/sdf
2 8 32 2 active sync /dev/sdc
3 8 48 3 active sync /dev/sdd
5 8 96 4 active sync /dev/sdg
# xfs_info /dev/md0
meta-data=/dev/md0 isize=256 agcount=32, agsize=30523776 blks
= sectsz=4096 attr=2
data = bsize=4096 blocks=976760832, imaxpct=5
= sunit=128 swidth=512 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=476934, version=2
= sectsz=4096 sunit=1 blks, lazy-count=0
realtime =none extsz=2097152 blocks=0, rtextents=0
--
Thomas Fjellstrom
tfjellstrom@xxxxxxx
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
