with hardware raid we don't think about this problem, but with software we should consider since we run others app with software raid running too 2011/3/21 Roberto Spadim <roberto@xxxxxxxxxxxxx>: > hum, maybe with linear will have less cpu use instead stripe? > i never tested a array with more than 8 disks with linear, and with > stripe hehehe > anyone could help here? > > 2011/3/20 Keld Jørn Simonsen <keld@xxxxxxxxxx>: >> On Sun, Mar 20, 2011 at 06:22:30PM -0500, Stan Hoeppner wrote: >>> Roberto Spadim put forth on 3/20/2011 12:32 AM: >>> >>> > i think it's better contact ibm/dell/hp/compaq/texas/anyother and talk >>> > about the problem, post results here, this is a nice hardware question >>> > :) >>> >>> I don't need vendor assistance to design a hardware system capable of >>> the 10GB/s NFS throughput target. That's relatively easy. I've already >>> specified one possible hardware combination capable of this level of >>> performance (see below). The configuration will handle 10GB/s using the >>> RAID function of the LSI SAS HBAs. The only question is if it has >>> enough individual and aggregate CPU horsepower, memory, and HT >>> interconnect bandwidth to do the same using mdraid. This is the reason >>> for my questions directed at Neil. >>> >>> > don't tell about software raid, just the hardware to allow this >>> > bandwidth (10gb/s) and share files >>> >>> I already posted some of the minimum hardware specs earlier in this >>> thread for the given workload I described. Following is a description >>> of the workload and a complete hardware specification. >>> >>> Target workload: >>> >>> 10GB/s continuous parallel NFS throughput serving 50+ NFS clients whose >>> application performs large streaming reads. At the storage array level >>> the 50+ parallel streaming reads become a random IO pattern workload >>> requiring a huge number of spindles due to the high seek rates. >>> >>> Minimum hardware requirements, based on performance and cost. Ballpark >>> guess on total cost of the hardware below is $150-250k USD. We can't >>> get the data to the clients without a network, so the specification >>> starts with the switching hardware needed. >>> >>> Ethernet switches: >>> One HP A5820X-24XG-SFP+ (JC102A) 24 10 GbE SFP ports >>> 488 Gb/s backplane switching capacity >>> Five HP A5800-24G Switch (JC100A) 24 GbE ports, 4 10GbE SFP >>> 208 Gb/s backplane switching capacity >>> Maximum common MTU enabled (jumbo frame) globally >>> Connect 12 server 10 GbE ports to A5820X >>> Uplink 2 10 GbE ports from each A5800 to A5820X >>> 2 open 10 GbE ports left on A5820X for cluster expansion >>> or off cluster data transfers to the main network >>> Link aggregate 12 server 10 GbE ports to A5820X >>> Link aggregate each client's 2 GbE ports to A5800s >>> Aggregate client->switch bandwidth = 12.5 GB/s >>> Aggregate server->switch bandwidth = 15.0 GB/s >>> The excess server b/w of 2.5GB/s is a result of the following: >>> Allowing headroom for an additional 10 clients or out of cluster >>> data transfers >>> Balancing the packet load over the 3 quad port 10 GbE server NICs >>> regardless of how many clients are active to prevent hot spots >>> in the server memory and interconnect subsystems >>> >>> Server chassis >>> HP Proliant DL585 G7 with the following specifications >>> Dual AMD Opteron 6136, 16 cores @2.4GHz >>> 20GB/s node-node HT b/w, 160GB/s aggregate >>> 128GB DDR3 1333, 16x8GB RDIMMS in 8 channels >>> 20GB/s/node memory bandwidth, 80GB/s aggregate >>> 7 PCIe x8 slots and 4 PCIe x16 >>> 8GB/s/slot, 56 GB/s aggregate PCIe x8 bandwidth >>> >>> IO controllers >>> 4 x LSI SAS 9285-8e 8 port SAS, 800MHz dual core ROC, 1GB cache >>> 3 x NIAGARA 32714 PCIe x8 Quad Port Fiber 10 Gigabit Server Adapter >>> >>> JBOD enclosures >>> 16 x LSI 620J 2U 24 x 2.5" bay SAS 6Gb/s, w/SAS expander >>> 2 SFF 8088 host and 1 expansion port per enclosure >>> 384 total SAS 6GB/s 2.5" drive bays >>> Two units are daisy chained with one in each pair >>> connecting to one of 8 HBA SFF8088 ports, for a total of >>> 32 6Gb/s SAS host connections, yielding 38.4 GB/s full duplex b/w >>> >>> Disks drives >>> 384 HITACHI Ultrastar C15K147 147GB 15000 RPM 64MB Cache 2.5" SAS >>> 6Gb/s Internal Enterprise Hard Drive >>> >>> >>> Note that the HBA to disk bandwidths of 19.2GB/s one way and 38.4GB/s >>> full duplex are in excess of the HBA to PCIe bandwidths, 16 and 32GB/s >>> respectively, by approximately 20%. Also note that each drive can >>> stream reads at 160MB/s peak, yielding 61GB/s aggregate streaming read >>> capacity for the 384 disks. This is almost 4 times the aggregate one >>> way transfer rate of the 4 PCIe x8 slots, and is 6 times our target host >>> to parallel client data rate of 10GB/s. There are a few reasons why >>> this excess of capacity is built into the system: >>> >>> 1. RAID10 is the only suitable RAID level for this type of system with >>> this many disks, for many reasons that have been discussed before. >>> RAID10 instantly cuts the number of stripe spindles in two, dropping the >>> data rate by a factor of 2, giving us 30.5GB/s potential aggregate >>> throughput. Now we're only at 3 times out target data rate. >>> >>> 2. As a single disk drive's seek rate increases, its transfer rate >>> decreases in relation to its single streaming read performance. >>> Parallel streaming reads will increase seek rates as the disk head must >>> move between different regions of the disk platter. >>> >>> 3. In relation to 2, if we assume we'll lose no more than 66% of our >>> single streaming performance with a multi stream workload, we're down to >>> 10.1GB/s throughput, right at our target. >>> >>> By using relatively small arrays of 24 drives each (12 stripe spindles), >>> concatenating (--linear) the 16 resulting arrays, and using a filesystem >>> such as XFS across the entire array with its intelligent load balancing >>> of streams using allocation groups, we minimize disk head seeking. >>> Doing this can in essence divide our 50 client streams across 16 arrays, >>> with each array seeing approximately 3 of the streaming client reads. >>> Each disk should be able to easily maintain 33% of its max read rate >>> while servicing 3 streaming reads. >>> >>> I hope you found this informative or interesting. I enjoyed the >>> exercise. I'd been working on this system specification for quite a few >>> days now but have been hesitant to post it due to its length, and the >>> fact that AFAIK hardware discussion is a bit OT on this list. >>> >>> I hope it may be valuable to someone Google'ing for this type of >>> information in the future. >>> >>> -- >>> Stan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> Are you then building the system yourself, and running Linux MD RAID? >> >> Anyway, with 384 spindles and only 50 users, each user will have in >> average 7 spindles for himself. I think much of the time this would mean >> no random IO, as most users are doing large sequential reading. >> Thus on average you can expect quite close to striping speed if you >> are running RAID capable of striping. >> >> I am puzzled about the --linear concatenating. I think this may cause >> the disks in the --linear array to be considered as one spindle, and thus >> no concurrent IO will be made. I may be wrong there. >> >> best regards >> Keld >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > > > -- > Roberto Spadim > Spadim Technology / SPAEmpresarial > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
