Udo - We are tracking these sort of client side configuration issues as bug #2014 , however in this particular case the hosts file/DNS workaround appears to be working well at several sites, is there any reason that process won't work for you? Thanks, Craig --> Craig Carl Gluster, Inc. Cell - (408) 829-9953 (California, USA) Gtalk - craig.carl at gmail.com From: "Udo Waechter" <udo.waechter at uni-osnabrueck.de> To: gluster-users at gluster.org Cc: "Craig Carl" <craig at gluster.com> Sent: Tuesday, November 9, 2010 11:33:56 PM Subject: Re: 3.1: Multiple networks and Client access Hi, any news on this? Thanks for the effort, udo. On 01.11.2010, at 15:02, Udo Waechter wrote: > Hi and thanks for the answer. > I think there is a bug/problem with gluster. > First some pre-knowledge > > - our internal network does not resolve vie DNS. > - the internal hosts are only resolvable via /etc/hosts. > > Now, my idea would be this: > > 1. on the cluster nodes, use the internal (bonding) interfaces for communication > 2. the rest of the network should use the external interfaces to communicate with the storage cloud. > > To achieve this, the idea was now to: > > a) create the volume with the internal names > $ gluster volume create ... hostname1.internal:/exp hostname2.internal:/exp hostname3.internal:/exp > > b) On all the (external) nodes, that should reach the gluster-cluster, add hostnam1...X.internal to /etc/hosts resolving to their external ip-address. > > > Now to all the problems: > > Regarding point 1 above: > - when I add a peer with its internal name it works, except for the peer that I add the other peers from. > ---hostname2.internal: $ gluster peer probe hostname1.internal > ---hostname1.internal $ gluster peer status > Number of Peers: 1 > > Hostname: 10.10.33.142 > Uuid: a19fc9d3-d00f-4440-b096-c974db1cd8c7 > State: Peer in Cluster (Connected) > > This should be hostname2.internal > > When I do gluster peer probe hostname1.internal (on the host hostname1.internal) I get: > > "hostname1.internal is already part of another cluster" > so here, ip/name resolution works... > > this works in all permutations. The peer from which I do "gluster peer probe ..." always does not resolve to its internal name, but its ip-adress > > As a result from all this, point a) can not succeed, since: > > gluster volume create .... hostname... hostname... hostname... results in: > > "Host hostnameX is not a friend", where hostnameX ist the host where the volume creation was attempted. > > > I have tried and installed pdnsd for the internal network, but this does not solve the problems either. > > As a last resort, I edited /etc/glusterd/peers/* and replaced the ip-adresses by hand. Now, "gluster peer status" gives me the names instead of the ip-adress. > but "volume create" still tells me about the host (where I create the volume from) not being a friend. > > Any help, solution is highly apreciated. > Thanks, > udo. > > On 18.10.2010, at 07:10, Craig Carl wrote: > >> Udo - >> With 3.1 when you mount/create/change a volume those changes are propagated via RPC to all of the other Gluster servers and clients. When you created the volume using 10.10.x.x IP addresses those IPs where what got sent to the client. In previous versions you could have just edited the client side configuration file and change or added the 192. addresses but not in this version, due to DVM. There should be a way to make multiple networks work so I will file a bug. >> In the meantime I think I have a workaround. If you use names instead of IP addresses and then make sure DNS or host files are setup properly is should work as Gluster does export via all interfaces. For example if the servers have these IPs - >> >> server1 - 10.10.1.1, 192.168.1.1 >> server2 - 10.10.1.2, 192.168.1.2 >> server3 - 10.10.1.3, 192.168.1.3 >> >> #gluster volume create test-ext stripe 3 server1:/ext server2:/ext server3:/ext >> >> You would just need to make sure that hosts on the 10.10.x.x resolve the servername to its 10. IP, and clients on the 192.x resolve to the 192 IP. Should be a simple change to the /etc/host files. >> >> Please let me know if this works so I can include that information in my bug report. >> >> Thanks, >> >> Craig >> >> -- >> Craig Carl >> Senior Systems Engineer; Gluster, Inc. >> Cell - (408) 829-9953 (California, USA) >> Office - (408) 770-1884 >> Gtalk - craig.carl at gmail.com >> Twitter - @gluster >> Installing Gluster Storage Platform, the movie! >> http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ >> >> >> From: "Udo Waechter" <udo.waechter at uni-osnabrueck.de> >> To: gluster-users at gluster.org >> Sent: Sunday, October 17, 2010 12:57:18 AM >> Subject: Re: 3.1: Multiple networks and Client access >> >> Hi, >> although I totally forgot about the firewall, this problem is not related to it. >> >> The ports you mentioned are open. >> >> I have created another volume using the ip-adresses from the external interfaces >> >> $ gluster volume create test-ext stripe 3 192.168.x.x1:/ext 192.168.x.x2:/ext 192.168.x.x3:/ext >> >> and this can not only be mounted, it also works perfectly. >> >> How could this work >> $ gluster volume create test-ext stripe 3 10.10.x.x1:/ext 10.10.x.x2:/ext 10.10.x.x3:/ext >> if the ip-addresses of the 10.10.x.x network are not reachable from the 192.168.x.x network? >> >> I have read somewhere in the docs that by default, glusterd listens on all interfaces, but does it also by default export everything to all interfaces? >> >> Thanks again, >> udo. >> >> >> On 16.10.2010, at 16:37, Jacob Shucart wrote: >> >>> Udo, >>> >>> It sounds to me like a firewall is blocking access to the Gluster system preventing some of the traffic from happening. Please see: >>> >>> http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Installing_GlusterFS_on_Red_Hat_Package_Manager_%28RPM%29_Distributions >>> >>> This lists the ports that need to be open both between Gluster storage nodes and between the client. This also needs to be open on all storage nodes to all clients. >>> >>> Please try this and let us know if this helps. >>> >>> -Jacob >>> >>> ----- Original Message ----- >>> From: "Udo Waechter" <udo.waechter at uni-osnabrueck.de> >>> To: gluster-users at gluster.org >>> Sent: Saturday, October 16, 2010 5:07:15 AM >>> Subject: 3.1: Multiple networks and Client access >>> >>> Hello all, >>> I just ran over Gluster 3.1 and got it up and running in notime. Thanks for the great FS. >>> >>> I have a question regarding multiple network. >>> >>> We have all our servers connected via an non-routed internal switch (a private subnet through which only the servers shall communicate with each other), additionally the official subnet to enable their reachability from the outside world. >>> >>> The internal IPnetwork is 10.10.x.x and the external shall be 192.168.x.x (for the sake of the question). >>> >>> I have now set up 3 servers as a GlusterFS. With a Striped Volume over the 10.10.x.x network. >>> All clients that are within this network can mount and use the volume. >>> >>> The problem arises with the clients from the other 192.168.x network. They can mount the volume via: >>> >>> mount -t glusterfs 192.168.x.x:volname /mnt >>> >>> but they fail to use the mountpoint (ls and so) >>> >>> I have read another post (http://bit.ly/a3Tm6n) but the solution mentioned there does not seem to work with version 3.1 any more. >>> The server(s) complain about "transport.socket..." not being allowed in the volume file. >>> >>> The client from the 192.168.x.x network tries to contact the bricks via the 10.10.x network and not via 192.168.x.x (which would work) >>> >>> Is there a solution to this problem? >>> >>> As I understand it, I would need to define the bricks with their 192.168.x.x ip-adresses but this would mean that the internal network won't be used. >>> >>> Can I define one brick with both ip-adresses (or 0.0.0.0). >>> Setting the "auth.allow" option to both subnets did not solve the problem (except for the mount being possible after setting it) >>> >>> Any help is highly appreciated. >>> Thanks, >>> udo. >>> -- >>> ---[ Institute of Cognitive Science @ University of Osnabrueck >>> ---[ Albrechtstrasse 28, D-49076 Osnabrueck, 969-3362 >>> ---[ Documentation: https://doc.ikw.uni-osnabrueck.de >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >> >> -- >> :: udo waechter - root at zoide.net :: N 52?16'30.5" E 8?3'10.1" >> :: genuine input for your ears: http://auriculabovinari.de >> :: your eyes: http://ezag.zoide.net >> :: your brain: http://zoide.net >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > -- > ---[ Institute of Cognitive Science @ University of Osnabrueck > ---[ Albrechtstrasse 28, D-49076 Osnabrueck, 969-3362 > ---[ Documentation: https://doc.ikw.uni-osnabrueck.de > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > -- :: udo waechter - root at zoide.net :: N 52?16'30.5" E 8?3'10.1" :: genuine input for your ears: http://auriculabovinari.de :: your eyes: http://ezag.zoide.net :: your brain: http://zoide.net
