Hans K. Rosbach wrote:
On Wed, 2011-06-08 at 12:34 +0100, Gordan Bobic wrote:
Hans K. Rosbach wrote:
-SCTP support, this might not be a silver bullet but it feels
[...]
Features that might need glusterfs code changes:
[...]
-Multihoming (failover when one nic dies)
How is this different to what can be achieved (probably much more
cleanly) with NIC bonding?
NIC bonding is nice for a small network, but routed networks might
have advantages from this. This is not something I feel that I need,
but I am sure it would be an advantage for some other users. This could
possibly be of help in geo-replication setups for example.
Not sure what routedness has to do with this. If you need route failover
this is probably something best done by having a HA/cluster service
change the routing table accordingly.
-Ability to have the storage nodes autosync themselves.
In our setup the normal nodes have 2x1Gbit connections while the
storage boxes have 2x10Gbit connections, so having the storage
boxes use their own bandwidth and resources to sync would be nice.
Sounds like you want server-side rather than client-side replication.
You could do this by using afr/replicate on the servers, and export via
NFS to the clients. Have failover handled as for any normal NFS server.
We have considered this, and might decide to go down this route
eventually, however it seems strange that this can not also be done
using the native client.
Is the current NFS wheel not quite round enough for you? ;)
The fact that each client writes to both servers is fine, but the
fact that the clients needs to do the re-sync work whenever the
storage nodes are out of sync (one of them rebooted for example)
seems strange and feels very unreliable especially since this is
a manual operation.
There is a plan C, though. You can make the servers also clients. You
can then have a process that does "ls -laR" periodically or upon failure.
-An ability for the clients to subscribe to metadata updates for
a specific directory would also be nice, so that it can cache that
folders stats while working there and still know that it will not
miss any changes. This would perhaps increase overhead in large
clusters but could improve performance by a lot in clusters where
several nodes work in the same folder (mail spool folder for example).
You have a shared mail spool on your nodes? How do you avoid race
conditions on deferred mail?
Several nodes can deliver mails to the spool folder, and dedicated queue
runners will pick them up and deliver them to local and/or remote hosts.
I am not certain what race conditions you are referring to, but locking
should make sure no more than one queue runner touches the file at one
time. Am I missing something?
Are you sure your MTA applies locks suitably? I wouldn't bet on it. I
would expect that most of them assume unshared spools. Also remember
that locking is a _major_ performance bottleneck when it comes to
cluster file systems. Multiple nodes doing locking and r/w in the same
directory will have an inverse scaling impact on performance, especially
on small I/O such as you are likely to experience on a mail spool.
If there is no file locking you will likely see non-deterministic
multiple sending of mail, especially deferred mail. Depending on how
your MTA produces mail spool file names, you may see non-deterministic
silent clobbering, too, if it doesn't do parent directory locking on
file creation/deletion.
If there is locking, you will likely see that the performance starts to
reduce as you add more servers due to lock contention.
Gordan
