For instance, if I ssh over to 2-ib, and ask for a peer status, it shows the peers to to be 1-ib-r 2-ib-r and *155.52.48.1* (the last is the ethernet side of, not the IB side) I guess my question is this: How do I get the first node to use the ib side when connecting to the other nodes. (both of these networks are public.) Do I have to do something interesting with routes? I need to fix this. Don't most people with IB also have an internet side to things? How do their networks and any local routing info get set up? Am I ,missing something really obvious? Matt ------ Matt Temple Director, Research Computing Dana-Farber Cancer Institute. We have two --f46d042f9326318ea804d12b703c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>We have two networks and I have 4 servers I'm try= ing to turn into a distributed/replicated Gluster storage cluster.<br><br><= /div><div>Each of the servers has two addresses, one ethernet, one IB.<br> </div><div>The externet addresses are 155.42.48.1-4 and <br></div><div>the = Infiniband addresses are 172.24.215.1-4<br></div><div><br>I login to the fi= rst server ("1") which is 155.52.48.1.<br><br></div><div>Let'= s say the servers are named 1,2,1R and 2R (on the ethernet side), but <br> =A01-IB, 2-IB, 1-IB-R and 2-IB-R on he Infiniband side.<br></div><div>The V= olume is to be mounted by other servers in the IB network by native Gluster= clients.<br><br></div><div>so, I login to "1" over the ethernet = network and do a peer probe.<br> </div><div>All looks good from "1" with peer probe and peer statu= s.<br><br></div><div>I probe the peers like this, successfully<br><br></div= ><div><font face=3D"courier new,monospace">gluster peer probe 2-ib<br></fon= t></div> <div><font face=3D"courier new,monospace">gluster peer probe 1-ib-r<br></fo= nt></div><div><font face=3D"courier new,monospace">gluster peer prove 2-ib-= r<br><br></font></div><div><font face=3D"courier new,monospace"><font face= =3D"arial,helvetica,sans-serif">Peer status looks right.<br> </font></font>I issue the following command which fails, of course:<br><br>= <span style=3D"font-size:15px;font-family:'Courier New';color:rgb(0= ,0,0);background-color:transparent;font-weight:normal;font-style:normal;fon= t-variant:normal;text-decoration:none;vertical-align:baseline">gluster volu= me create gf2 replica 2 transport tcp,rdma \</span><br> <span style=3D"font-size:15px;font-family:'Courier New';color:rgb(0= ,0,0);background-color:transparent;font-weight:normal;font-style:normal;fon= t-variant:normal;text-decoration:none;vertical-align:baseline">1-ib:/mnt/d0= -0 1-ib-r:/mnt/d0-0 </span><span style=3D"font-size:15px;font-family:'C= ourier New';color:rgb(0,0,0);background-color:transparent;font-weight:n= ormal;font-style:normal;font-variant:normal;text-decoration:none;vertical-a= lign:baseline">2-ib:/mnt/d0-0 2-ib-r:/mnt/d0-0</span></div> <div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,san= s-serif"><br>From any of those nodes, peer status is wrong:<br>For instance= , if I ssh over to 2-ib, <br>and ask for a peer status, it shows the peers = to to be <br> <br></font></font></div><div><font face=3D"courier new,monospace"><font fac= e=3D"arial,helvetica,sans-serif"><font face=3D"courier new,monospace">1-ib-= r<br>2-ib-r <br>and <b>155.52.48.1</b></font> (the last is the ethernet sid= e of, not the IB side)<br> <br></font></font></div><div><font face=3D"courier new,monospace"><font fac= e=3D"arial,helvetica,sans-serif">I guess my question is this:=A0=A0 How do = I get the first node to use the ib side when<br></font></font></div><div><f= ont face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif= ">connecting to the other nodes. (both of these networks are public.)=A0=A0= Do I have to<br> </font></font></div><div><font face=3D"courier new,monospace"><font face=3D= "arial,helvetica,sans-serif">do something interesting with routes?<br></fon= t></font></div><div><font face=3D"courier new,monospace"><font face=3D"aria= l,helvetica,sans-serif"><br> </font></font></div><div>I need to fix this.=A0=A0 Don't most people wi= th IB also have an internet side to things?<br></div><div>How do their netw= orks and any local routing info get set up?<br></div><div>Am I ,missing som= ething really obvious?<br> <br></div><div>Matt<br></div><div><br></div><br><div><div>------<br>Matt Te= mple<br>Director, Research Computing<br>Dana-Farber Cancer Institute.<br></= div><br></div>We have two=A0 </div> --f46d042f9326318ea804d12b703c--
