On Fri, Nov 30, 2012 at 07:21:54PM +0000, Mike Hanby wrote:
> We have the following hardware that we are going to use for a GlusterFS
> cluster.
>
> 6 x Dell R720xd's (16 cores, 96G)
Heavily over-specified, especially the RAM. Having such large amounts of RAM
can even cause problems if you're not careful. You probably want to use
sysctl and /etc/sysctl.conf to set
vm.dirty_background_ratio=1
vm.dirty_ratio=5 (or less)
so that dirty disk blocks are written to disk sooner, otherwise you may find
the system locks up for several minutes at a time as it flushes the enormous
disk cache.
I use 4 cores + 8GB for bricks with 24 disks (and they are never CPU-bound)
> I now need to decide how to configure the 12 x 3TB disks in each
> server, followed by partitioning / formatting them in the OS.
>
> The PERC H710 supports RAID 0,1,5,6,10,50,60. Ideally we'd like to get
> good performance, maximize storage capacity and still have parity :-)
For performance: RAID10
For maximum storage capacity: RAID5 or RAID6
> * Stripe Element Size: 64, 128, 256, 512KB, 1MB
Depends on workload. With RAID10 and lots of concurrent clients, I'd tend to
use a 1MB stripe size. Then R/W by one client is likely to be on a different
disk to R/W by another client, and although throughput to individual clients
will be similar to a single disk, the total throughput is maximised.
If your accesses are mostly by a single client, then you may not get enough
readahead to saturate the disks with such a large stripe size; with RAID5/6
your writes may be slow if you can't write a stripe at a time (which may be
possible if you have a battery-backed card). So for these scenarios
something like 256K may work better. But you do need to test it.
Finally: you don't mention your network setup.
With 12 SATA disks, you can expect to get 25-100MB/sec *per disk* depending
on how sequential and how large the transfers are. So your total disk
throughput is potentially 12 times that, i.e. 300-1200MB/sec. The bottom
end of this range is easily achievable, and is already 2.5 times a 1G link.
At the top end you could saturate a 10G link.
So if you have only 1G networking it's very likely going to be the
bottleneck.
Regards,
Brian.
