hum, maybe with linear will have less cpu use instead stripe? i never tested a array with more than 8 disks with linear, and with stripe hehehe anyone could help here? 2011/3/20 Keld Jørn Simonsen <keld@xxxxxxxxxx>: > On Sun, Mar 20, 2011 at 06:22:30PM -0500, Stan Hoeppner wrote: >> Roberto Spadim put forth on 3/20/2011 12:32 AM: >> >> > i think it's better contact ibm/dell/hp/compaq/texas/anyother and talk >> > about the problem, post results here, this is a nice hardware question >> > :) >> >> I don't need vendor assistance to design a hardware system capable of >> the 10GB/s NFS throughput target. That's relatively easy. I've already >> specified one possible hardware combination capable of this level of >> performance (see below). The configuration will handle 10GB/s using the >> RAID function of the LSI SAS HBAs. The only question is if it has >> enough individual and aggregate CPU horsepower, memory, and HT >> interconnect bandwidth to do the same using mdraid. This is the reason >> for my questions directed at Neil. >> >> > don't tell about software raid, just the hardware to allow this >> > bandwidth (10gb/s) and share files >> >> I already posted some of the minimum hardware specs earlier in this >> thread for the given workload I described. Following is a description >> of the workload and a complete hardware specification. >> >> Target workload: >> >> 10GB/s continuous parallel NFS throughput serving 50+ NFS clients whose >> application performs large streaming reads. At the storage array level >> the 50+ parallel streaming reads become a random IO pattern workload >> requiring a huge number of spindles due to the high seek rates. >> >> Minimum hardware requirements, based on performance and cost. Ballpark >> guess on total cost of the hardware below is $150-250k USD. We can't >> get the data to the clients without a network, so the specification >> starts with the switching hardware needed. >> >> Ethernet switches: >> One HP A5820X-24XG-SFP+ (JC102A) 24 10 GbE SFP ports >> 488 Gb/s backplane switching capacity >> Five HP A5800-24G Switch (JC100A) 24 GbE ports, 4 10GbE SFP >> 208 Gb/s backplane switching capacity >> Maximum common MTU enabled (jumbo frame) globally >> Connect 12 server 10 GbE ports to A5820X >> Uplink 2 10 GbE ports from each A5800 to A5820X >> 2 open 10 GbE ports left on A5820X for cluster expansion >> or off cluster data transfers to the main network >> Link aggregate 12 server 10 GbE ports to A5820X >> Link aggregate each client's 2 GbE ports to A5800s >> Aggregate client->switch bandwidth = 12.5 GB/s >> Aggregate server->switch bandwidth = 15.0 GB/s >> The excess server b/w of 2.5GB/s is a result of the following: >> Allowing headroom for an additional 10 clients or out of cluster >> data transfers >> Balancing the packet load over the 3 quad port 10 GbE server NICs >> regardless of how many clients are active to prevent hot spots >> in the server memory and interconnect subsystems >> >> Server chassis >> HP Proliant DL585 G7 with the following specifications >> Dual AMD Opteron 6136, 16 cores @2.4GHz >> 20GB/s node-node HT b/w, 160GB/s aggregate >> 128GB DDR3 1333, 16x8GB RDIMMS in 8 channels >> 20GB/s/node memory bandwidth, 80GB/s aggregate >> 7 PCIe x8 slots and 4 PCIe x16 >> 8GB/s/slot, 56 GB/s aggregate PCIe x8 bandwidth >> >> IO controllers >> 4 x LSI SAS 9285-8e 8 port SAS, 800MHz dual core ROC, 1GB cache >> 3 x NIAGARA 32714 PCIe x8 Quad Port Fiber 10 Gigabit Server Adapter >> >> JBOD enclosures >> 16 x LSI 620J 2U 24 x 2.5" bay SAS 6Gb/s, w/SAS expander >> 2 SFF 8088 host and 1 expansion port per enclosure >> 384 total SAS 6GB/s 2.5" drive bays >> Two units are daisy chained with one in each pair >> connecting to one of 8 HBA SFF8088 ports, for a total of >> 32 6Gb/s SAS host connections, yielding 38.4 GB/s full duplex b/w >> >> Disks drives >> 384 HITACHI Ultrastar C15K147 147GB 15000 RPM 64MB Cache 2.5" SAS >> 6Gb/s Internal Enterprise Hard Drive >> >> >> Note that the HBA to disk bandwidths of 19.2GB/s one way and 38.4GB/s >> full duplex are in excess of the HBA to PCIe bandwidths, 16 and 32GB/s >> respectively, by approximately 20%. Also note that each drive can >> stream reads at 160MB/s peak, yielding 61GB/s aggregate streaming read >> capacity for the 384 disks. This is almost 4 times the aggregate one >> way transfer rate of the 4 PCIe x8 slots, and is 6 times our target host >> to parallel client data rate of 10GB/s. There are a few reasons why >> this excess of capacity is built into the system: >> >> 1. RAID10 is the only suitable RAID level for this type of system with >> this many disks, for many reasons that have been discussed before. >> RAID10 instantly cuts the number of stripe spindles in two, dropping the >> data rate by a factor of 2, giving us 30.5GB/s potential aggregate >> throughput. Now we're only at 3 times out target data rate. >> >> 2. As a single disk drive's seek rate increases, its transfer rate >> decreases in relation to its single streaming read performance. >> Parallel streaming reads will increase seek rates as the disk head must >> move between different regions of the disk platter. >> >> 3. In relation to 2, if we assume we'll lose no more than 66% of our >> single streaming performance with a multi stream workload, we're down to >> 10.1GB/s throughput, right at our target. >> >> By using relatively small arrays of 24 drives each (12 stripe spindles), >> concatenating (--linear) the 16 resulting arrays, and using a filesystem >> such as XFS across the entire array with its intelligent load balancing >> of streams using allocation groups, we minimize disk head seeking. >> Doing this can in essence divide our 50 client streams across 16 arrays, >> with each array seeing approximately 3 of the streaming client reads. >> Each disk should be able to easily maintain 33% of its max read rate >> while servicing 3 streaming reads. >> >> I hope you found this informative or interesting. I enjoyed the >> exercise. I'd been working on this system specification for quite a few >> days now but have been hesitant to post it due to its length, and the >> fact that AFAIK hardware discussion is a bit OT on this list. >> >> I hope it may be valuable to someone Google'ing for this type of >> information in the future. >> >> -- >> Stan >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > Are you then building the system yourself, and running Linux MD RAID? > > Anyway, with 384 spindles and only 50 users, each user will have in > average 7 spindles for himself. I think much of the time this would mean > no random IO, as most users are doing large sequential reading. > Thus on average you can expect quite close to striping speed if you > are running RAID capable of striping. > > I am puzzled about the --linear concatenating. I think this may cause > the disks in the --linear array to be considered as one spindle, and thus > no concurrent IO will be made. I may be wrong there. > > best regards > Keld > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
