Eric Comeau wrote:
Ideally 1 large file, but it may have to be multiple. We find that if we send multiple files it just causes the disk to thrash more so we get better throughput by sending one large file. If it's really one disk, sure. The problem you're facing is that your typical drive controller is going to top out at somewhere between 300 - 500MB/s of sequential writes before it becomes the bottleneck. Above somewhere between 6 and 10 drives attached to one controller on current hardware, adding more to a RAID-0 volume only increases the ability to handle seeks quickly. If you want to try and do this with traditional hard drives, I'd guess you'd need 3 controllers with at least 4 short-stroked drives attached to each to have any hope of hitting 1.25GB/s. Once you do that, you'll run into CPU time as the next bottleneck. At that point, you'll probably need one CPU per controller, all writing out at once, to keep up with your target. The only popular hardware design that comes to mind aimed at this sort of thing was Sun's "Thumper" design, most recently seen in the Sun Fire X4540. That put 8 controllers with 6 disks attached to each, claiming "demonstrated up to 2 GB/sec from disk to network". It will take a design like that, running across multiple controllers, to get what you're looking for on the disk side--presuming everything else keeps up. One of the big SSD-on-PCI-e designs mentioned here already may very well end up being a better choice for you here though, as those aren't going to require quite as much hardware all get wired up. -- Greg Smith 2ndQuadrant US greg@xxxxxxxxxxxxxxx Baltimore, MD PostgreSQL Training, Services and Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books |
