On 09/08/2009 01:01 PM, Alan Ivey wrote: > This is for running some "cloud" servers where we want all files available on each machine locally so all servers have the same files but can still get local performance. I don't think I'll need to run a cron like that but it's not my network so I'm trying to figure out how to get all of the gears working together. > Note that unless you are doing mostly read - or unless you have GigE or better connections to each of these servers - you are not going to "still get local performance". As stat() calls are distributed to multiple machines, and write() operations would require replication to multiple machines, these will be significantly slower than local disk, and significantly slower than NFS (which does not do replication). The more machines you have, the worse it will get. For example, in a test I just did, where we only have 100 Mbit/s between the nodes right now, with a 3-node replication cluster, I was only getting 5 Mbyte/s writes, but 70 - 130 Mbyte/s reads. Why? Because my write to the one server needed to be replicated to 2 other nodes, and if we divide 100 Mbit/s transmit speed by 2, we get 50 Mbit/s or 5 Mbyte/s to each. If you are going to have a "cloud" of 10 servers instead of 3 - this means that every write needs to be sent to all 10, or 9 other nodes (depending on if the client is on the server), which would divide your network upload capacity by 10, or 9. Go up to 100, and it gets even worse. The machines themselves - every time I do a write, every node in the replication cluster would do a write. They're all working in unison. So, if 10 machines in the cluster are all issuing writes, then 10X as many writes are all happening to local disk, which means 10X as many seeks, and 10X fewer I/O throughput to the disk. I want to make sure you understand that clustering in this way is not really giving you the ability to have each machine with local disk speeds. For most uses, clustering is really providing fail over / redundancy capabilities. However, if your workload is entirely dominated by reads, and very few writes, than this would also provide effective load balancing capabilities. > I did just discover that changing permissions does not seem to heal. I had two servers up, killed glusterfsd on server2, on client I changed the permissions of a file, brought server2 back online, ran ls on client, and server2 still had the old permissions. It's not really a big issue since I don't see myself changing permissions often, esp with any servers down, but interesting nonetheless. I think those settings are all configurable - I haven't played with them myself. More checks at run time = more expensive at run time. Cheers, mark -- Mark Mielke<mark at mielke.cc>
