Hi! 2008/9/21 Paolo Supino <paolo.supino at gmail.com>: > Hi Krishna > > I'm running glusterfs servers and clients on all the nodes that are > connected to the private network and I want to run glusterfs clients (only) > on researcher computers that have only access to the head node from an > external network. > I've attached an image of the final setup I want to reach ... > What I'm missing now: Head node doesn't iSCSI mount the toaster and > doesn't isn't a glusterfs server. The researcher's computer doesn't act as a > glusterfs client ... Thank you for posting the diagram. If I'd try to create a setup like this my first attempt at a configuration would look something like this: cluster nodes except head node: >>> # file: /etc/glusterfs/glusterfs-server.vol volume brick type storage/posix option directory /data/export end-volume volume brick-ns type storage/posix option directory /data/export-ns end-volume volume server type protocol/server option transport-type tcp/server option auth.ip.brick.allow <private subnet> option auth.ip.brick-ns.allow <private subnet> subvolumes brick brick-ns end-volume >>>EOF from http://www.gluster.org/docs/index.php/Aggregating_Three_Storage_Servers_with_Unify Following the mailing list thread, IIRC you already have something like that. cluster head node: >>> # file: /etc/glusterfs/glusterfs-client.vol # "unify"ing the cluster nodes and the "toaster" volume remote1 type protocol/client option transport-type tcp/client option remote-host storage1.example.com option remote-subvolume brick end-volume volume remote2 type protocol/client option transport-type tcp/client option remote-host storage2.example.com option remote-subvolume brick end-volume <add the other cluster nodes here> volume remote35 type protocol/client option transport-type tcp/client option remote-host storage35.example.com option remote-subvolume brick end-volume volume remote-ns type protocol/client option transport-type tcp/client option remote-host storage1.example.com option remote-subvolume brick-ns end-volume # local data volume remote36 type storage/posix option directory /data/export end-volume # storage on "toaster" volume toaster1 type storage/posix option directory /mnt/toaster1 # mounted from iSCSI end-volume volume toaster2 type storage/posix option directory /mnt/toaster2 # mounted from iSCSI end-volume # unify everything together volume unify0 type cluster/unify option scheduler rr # round robin option namespace remote-ns subvolumes remote1 remote2 <add the other 32 nodes here> remote35 remote36 toaster1 toaster2 end-volume # and now export that to the cluster nodes and faculty systems volume all-data type protocol/server option transport-type tcp/server option auth.ip.unify0.allow <faculty subnet> # should be limited to faculty systems option auth.i.remote36.allow <private subnet> # export to cluster nodes option auth.i.toaster1.allow <private subnet> # export to cluster nodes option auth.i.toaster2.allow <private subnet> # export to cluster nodes subvolumes unify0 remote36 toaster1 toaster2 end-volume >>>EOF bits and pieces copied from http://www.gluster.org/docs/index.php/Aggregating_Three_Storage_Servers_with_Unify faculty systems acting as clients: >>> # file: /etc/glusterfs/glusterfs-client.vol volume remote type protocol/client option transport-type tcp/client option remote-host <(external) IP of cluster head node> option remote-subvolume all-data end-volume >>> EOF adapted from http://www.gluster.org/docs/index.php/NFS_Like_Standalone_Storage_Server cluster nodes acting as clients, including head node: >>> # file: /etc/glusterfs/glusterfs-client.vol volume remote1 type protocol/client option transport-type tcp/client option remote-host storage1.example.com option remote-subvolume brick end-volume volume remote2 type protocol/client option transport-type tcp/client option remote-host storage2.example.com option remote-subvolume brick end-volume <add other 32 nodes here> volume remote35 type protocol/client option transport-type tcp/client option remote-host storage35.example.com option remote-subvolume brick end-volume volume remote36 type protocol/client option transport-type tcp/client option remote-host headnode.example.com option remote-subvolume brick end-volume volume toaster1 type protocol/client option transport-type tcp/client option remote-host headnode.example.com option remote-subvolume brick end-volume volume toaster2 type protocol/client option transport-type tcp/client option remote-host headnode.example.com option remote-subvolume brick end-volume volume remote-ns type protocol/client option transport-type tcp/client option remote-host storage1.example.com option remote-subvolume brick-ns end-volume volume unify0 # the same as used by the head node type cluster/unify option scheduler rr # round robin option namespace remote-ns subvolumes remote1 remote2 <> remote35 remote36 toaster1 toaster2 end-volume >>>EOF adapted from http://www.gluster.org/docs/index.php/Aggregating_Three_Storage_Servers_with_Unify There are a some things that could be done to improve performance, e.g. - adding performance translators at the "right places" ;-) - use NUFA scheduler for unify, as all cluster nodes act as clients and servers. I believe that adding the faculty systems as servers might be done - but I would not touch that topic as long as i would call the setup described above "complicated". <loud thinking> Some extensions to the volume description language might be nice: - having remote(1..36) automatically expand to "remote1 remote2 remote3 .. remote36" - some "if <regex done to hostname matches something> replace <this volume> with <that volume> - some loop structure to create volume descriptions, using copy/paste usually leads to error when editing at a later time and forgetting one copy - imagine a setup unifying AFR'ed volumes in a 500+ nodes cluster </loud thinking> Well, I should post that to the GlusterFS wishlist. Harald St?rzebecher
