On Tue October 27 2009, Thomas Fjellstrom wrote: > On Wed October 14 2009, mark delfman wrote: > > Hi Chris... we tried the direct DD as requested and the problem is > > still there... > > 1.3GBsec > 325MBsec (even more dromatic)... hopefully this helps > > narrow it down? > > > > > > Write > MD > > linux-poly:~ # dd if=/dev/zero of=/dev/md0 oflag=direct bs=1M > > count=20000 20000+0 records in > > 20000+0 records out > > 20971520000 bytes (21 GB) copied, 15.7671 s, 1.3 GB/s > > > > > > Write > XFS > MD > > linux-poly:~ # dd if=/dev/zero of=/mnt/md0/test oflag=direct bs=1M > > count=20000 20000+0 records in > > 20000+0 records out > > 20971520000 bytes (21 GB) copied, 64.616 s, 325 MB/s > > If it helps, I'm seeing the same sort of thing. > The most I can seemingly tweak out of my new 5x1TB array is 170MB/s > write. Using dd with oflags=direct drops it down to 31MB/s. > > Oddly, I see spikes of over 200MB/s write when not using oflags=direct, > but it slows down in between to 11MB/s so over all, > it averages a max of 170MB/s. the device itself is capable of over > 500MB/s. (66% drop?) > > small test: > > $ dd if=/dev/zero of=/mnt/test-data/test.file bs=512KiB count=4096 > oflag=direct 4096+0 records in > 4096+0 records out > 2147483648 bytes (2.1 GB) copied, 71.8088 s, 29.9 MB/s > > $ dd if=/dev/zero of=/mnt/test-data/test.file bs=512KiB count=4096 > 4096+0 records in > 4096+0 records out > 2147483648 bytes (2.1 GB) copied, 19.7101 s, 109 MB/s > > $ sudo dd if=/dev/md0 of=/tmp/test-data.img bs=1M count=1024 > 1024+0 records in > 1024+0 records out > 1073741824 bytes (1.1 GB) copied, 2.39796 s, 448 MB/s > > $ sudo dd if=/tmp/test-data.img of=/dev/md0 bs=1M count=1024 > 1024+0 records in > 1024+0 records out > 1073741824 bytes (1.1 GB) copied, 2.05666 s, 522 MB/s > > $ cd /mnt/test-data/test > $ iozone -A -s4G -y512k -q512k > ... > KB reclen write rewrite read reread > 4194304 512 161732 333316 382361 388726 > > > [snip] > > > > info, if it helps: > > # mdadm -D /dev/md0 > /dev/md0: > Version : 1.01 > Creation Time : Wed Oct 14 08:55:25 2009 > Raid Level : raid5 > Array Size : 3907049472 (3726.05 GiB 4000.82 GB) > Used Dev Size : 976762368 (931.51 GiB 1000.20 GB) > Raid Devices : 5 > Total Devices : 5 > Persistence : Superblock is persistent > > Update Time : Tue Oct 27 04:18:50 2009 > State : clean > Active Devices : 5 > Working Devices : 5 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 512K > > Name : natasha:0 (local to host natasha) > UUID : 7d0e9847:ec3a4a46:32b60a80:06d0ee1c > Events : 4952 > > Number Major Minor RaidDevice State > 0 8 64 0 active sync /dev/sde > 1 8 80 1 active sync /dev/sdf > 2 8 32 2 active sync /dev/sdc > 3 8 48 3 active sync /dev/sdd > 5 8 96 4 active sync /dev/sdg > > # xfs_info /dev/md0 > meta-data=/dev/md0 isize=256 agcount=32, agsize=30523776 > blks = sectsz=4096 attr=2 > data = bsize=4096 blocks=976760832, imaxpct=5 > = sunit=128 swidth=512 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal bsize=4096 blocks=476934, version=2 > = sectsz=4096 sunit=1 blks, lazy-count=0 > realtime =none extsz=2097152 blocks=0, rtextents=0 > ran 4 dd's in parallel all writing to a different file on the array: 4096+0 records in 4096+0 records out 2147483648 bytes (2.1 GB) copied, 33.4193 s, 64.3 MB/s 4096+0 records in 4096+0 records out 2147483648 bytes (2.1 GB) copied, 35.5599 s, 60.4 MB/s 4096+0 records in 4096+0 records out 2147483648 bytes (2.1 GB) copied, 36.4677 s, 58.9 MB/s 4096+0 records in 4096+0 records out 2147483648 bytes (2.1 GB) copied, 37.912 s, 56.6 MB/s iostat showed spikes of up to 300MB/s and it usually hovered over 200MB/s. I tried bumping it to 8 at a time, but it seems to max out at just over 200MB/s. was hoping that with enough jobs, it might scale up to the devices actual max throughput. -- Thomas Fjellstrom tfjellstrom@xxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
