On 8/11/2011 10:58 AM, mark delfman wrote: > I seem to have hit a significant hard stop in MD RAID1/10 performance > which seems to be linked to a single CPU thread. What is the name of the kernel thread that is peaking your cores? Could the device driver be eating the CPU and not the md kernel threads? Is it both? Is it a different thread? How much CPU is the IO generator app eating? What Linux kernel version are you running? Which Linux distribution? What application are you using to generate the IO load? Does it work at the raw device/partition level or at the file level? > I am using extremely high speed (IOPS) internal block devices – 8 in > total. They are capable of achieving > 1million iops. 8 solid state drives of one model or another, probably occupying 8 PCIe slots. IBIS, VeloDrive, the LSI SSD, or other PCIe based SSD? Or are these plain SATA II SSDs that *claim* to have 125K 4KB random IOPS performance? > However if I use RAID1 / 10 then MD seems to use a single thread which > will reach 100% CPU utilisation (single core) at around 200K IOPS. > Limiting the entire performance to around 200K. CPU frequency? How many sockets? Total cores? Whose box? HP, Dell, IBM, whitebox, self built? If the latter two, whose motherboard? How many PCIe slots are occupied by the SSD cards? > If I use say 4 x RAID1 / 10’s and a RAID0 on top – I see not much > greater results. (although the theory seems to say I should and there > are now 4 CPU threads running, it still seems to hit 4 x 100% at maybe > 350K). Assuming you have 4 processors (cores), then yes, you should see better scaling. If you have less cores than threads, then no. Do you see more IOPS before running out of CPU when writing vs reading? You should as you're doing half the IOs when reading. > Is there any way to increase the number of threads per RAID set? Or > any other suggestions on configurations? (I have tried every > permutation of R0+R1/10’s) The answer to the first question AFAIK is no. Do you have the same problem with a single --linear array? What is the result when putting a filesystem on each individual drive? Do you get your 1 million IOPS? Is MSI enabled and verified to be working for each PCIe SSD device? See: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/PCI/MSI-HOWTO.txt;hb=HEAD -- Stan -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
