On Fri, 22 Nov 2013 21:34:41 -0800 John Williams <jwilliams4200@xxxxxxxxx> wrote: > On Fri, Nov 22, 2013 at 9:04 PM, NeilBrown <neilb@xxxxxxx> wrote: > > > I guess with that many drives you could hit PCI bus throughput limits. > > > > A 16-lane PCIe 4.0 could just about give 100MB/s to each of 16 devices. So > > you would really need top-end hardware to keep all of 16 drives busy in a > > recovery. > > So yes: rebuilding a drive in a 16-drive RAID6+ would be slower than in e.g. > > a 20 drive RAID10. > > Not really. A single 8x PCIe 2.0 card has 8 x 500MB/s = 4000MB/s of > potential bandwidth. That would be 250MB/s per drive for 16 drives. > > But quite a few people running software RAID with many drives have > multiple PCIe cards. For example, in one machine I have three IBM > M1015 cards (which I got for $75/ea) that are 8x PCIe 2.0. That comes > to 3 x 500MB/s x 8 = 12GB/s of IO bandwidth. > > Also, your math is wrong. PCIe 3.0 is 985 MB/s per lane. If we assume > PCIe 4.0 would double that, we would have 1970MB/s per lane. So one > lane of the hypothetical PCIe 4.0 would have enough IO bandwidth to > give about 120MB/s to each of 16 drives. A single 8x PCIe 4.0 card > would have 8 times that capability which is more than 15GB/s. It wasn't my math, it was my reading :-( 16-lane PCIe 4.0 is 31 GB/sec so 2GB/sec per drive. I was reading the "1-lane" number... > > Even a single 8x PCIe 3.0 card has potentially over 7GB/s of bandwidth. > > Bottom line is that IO bandwidth is not a problem for a system with > prudently chosen hardware. > > More likely is that you would be CPU limited (rather than bus limited) > in a high-parity rebuild where more than one drive failed. But even > that is not likely to be too bad, since Andrea's single-threaded > recovery code can recover two drives at nearly 1GB/s on one of my > machines. I think the code could probably be threaded to achieve a > multiple of that running on multiple cores. Indeed. It seems likely that with modern hardware, the linear write speed would be the limiting factor for spinning-rust drives. For SSDs the limit might end up being somewhere else ... Thanks, NeilBrown
Attachment:
signature.asc
Description: PGP signature
