That looks OK (but your 2 MTUs are mismatched - should fix that).
> UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 <- 1st
> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 <- 2nd
^^^^^
IBDEV=ibX
modprobe ib_umad
modprobe ib_ipoib
echo connected > /sys/class/net/${IBDEV}/mode
echo 65520 > /sys/class/net/${IBDEV}/mtu
how did you set up the peering? By name? by IP#?
(I assume pinging by hostname also works both ways?)
If you can't get the peers to ack, then what do the logs say on failure to:
peer probe <host>
or create the volume
gluster volume create <volname> host1ib:/gl_part host2ib:/gl_part
hjm
On Monday, February 25, 2013 09:50:02 PM Tony Saenz wrote:
> Trying to first get this working with IPoIB
>
> [root at fpsgluster ~]# ibhosts
> Ca : 0x00117500007937b2 ports 1 "fpsgluster2 qib0"
> Ca : 0x0011750000792af2 ports 1 "fpsgluster qib0"
>
> I'm able to ping the other box from Infiniband to Infiniband card
>
> Ifconfig uses the ioctl access method to get the full address information,
> which limits hardware addresses to 8 bytes. Because Infiniband address has
> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
> obsolete! For replacement check ip.
> ib0 Link encap:InfiniBand HWaddr
> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
> addr:10.0.4.35 Bcast:10.0.4.255 Mask:255.255.255.0 inet6 addr:
> fe80::211:7500:79:2af2/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
> RX packets:1567 errors:0 dropped:0 overruns:0 frame:0
> TX packets:587 errors:0 dropped:24 overruns:0 carrier:0
> collisions:0 txqueuelen:256
> RX bytes:342622 (334.5 KiB) TX bytes:96554 (94.2 KiB)
>
> [root at fpsgluster2 ~]# ifconfig ib0
> Ifconfig uses the ioctl access method to get the full address information,
> which limits hardware addresses to 8 bytes. Because Infiniband address has
> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
> obsolete! For replacement check ip.
> ib0 Link encap:InfiniBand HWaddr
> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
> addr:10.0.4.34 Bcast:10.0.4.255 Mask:255.255.255.0 inet6 addr:
> fe80::211:7500:79:37b2/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
> RX packets:599 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1558 errors:0 dropped:8 overruns:0 carrier:0
> collisions:0 txqueuelen:256
> RX bytes:95180 (92.9 KiB) TX bytes:346728 (338.6 KiB)
>
> [root at fpsgluster ~]# ping -I ib0 10.0.4.34
> PING 10.0.4.34 (10.0.4.34) from 10.0.4.35 ib0: 56(84) bytes of data.
> 64 bytes from 10.0.4.34: icmp_seq=1 ttl=64 time=12.6 ms
> 64 bytes from 10.0.4.34: icmp_seq=2 ttl=64 time=0.184 ms
>
> /etc/hosts looks correct
>
> [root at fpsgluster2 ~]# cat /etc/hosts | grep ib
> 10.0.4.35 fpsglusterib
> 10.0.4.34 fpsgluster2ib
>
> [root at fpsgluster ~]# cat /etc/hosts| grep ib
> 10.0.4.35 fpsglusterib
> 10.0.4.34 fpsgluster2ib
>
> I haven't created the new volume yet as I can't get the peer probe to work
> off the Infiniband card. It's only seeing the NIC cards I currently have it
> hooked in to.
>
>
> On Feb 25, 2013, at 11:57 AM, harry mangalam <harry.mangalam at uci.edu>
>
> wrote:
> > It /might be/probably is/ DNS-related.
> >
> > Are you trying to do this with RDMA or IPoIB?
> >
> > If IPoIB, are ALL your /etc/hosts files in sync (IB names separate and
> > distinct from the ethernet interfaces) and responsive to the appropriate
> > interfaces?
> >
> > Do the IB interfaces show up as distinct (and connected) on an 'ifconfig
> > -a' and 'ibstat' dump?
> >
> > Do all the peers show up on an 'ibhosts' query?
> >
> > What is the output of:
> > gluster volume status <your_volume>
> > and
> > gluster volume status <your_volume> detail
> >
> >
> > hjm
> >
> > On Monday, February 25, 2013 07:46:00 PM Tony Saenz wrote:
> >> It shows this but it's still going through my NIC cards and not the
> >> Infiniband. (Checked the traffic on the cards themselves)
> >>
> >> [root at fpsgluster ~]# gluster peer status
> >> Number of Peers: 1
> >>
> >> Hostname: fpsgluster2
> >> Uuid: 9b7e7c2d-f05b-4cc8-b55a-571e383328d0
> >> State: Peer in Cluster (Connected)
> >>
> >> On Feb 25, 2013, at 10:51 AM, Torbj?rn Thorsen <torbjorn at trollweb.no>
wrote:
> >>> Your error message seems to indicate that the peer is already in the
> >>> storage pool ?
> >>> What is the output of "gluster peer status" ?
> >>>
> >>> On Mon, Feb 25, 2013 at 7:28 PM, Tony Saenz <tony at filmsolutions.com>
> >
> > wrote:
> >>>> Any help please? The regular NICs are fine which is what it currently
> >>>> sees but I'd like to move them over to the Infiniband cards.>>
> >>>> On Feb 22, 2013, at 1:50 PM, Anthony Saenz <tony at filmsolutions.com>
> >
> > wrote:
> >>>>> Hey,
> >>>>>
> >>>>> I was wondering if I could get a bit of help.. I installed a new
> >>>>> Infiniband card into my servers but I'm unable to get it to come up as
> >>>>> a peer. Is there something I'm missing?
> >>>>>
> >>>>> [root at fpsgluster testvault]# gluster peer probe fpsgluster2ib
> >>>>> Probe on host fpsgluster2ib port 0 already in peer list
> >>>>>
> >>>>> [root at fpsgluster testvault]# yum list installed | grep gluster
> >>>>> glusterfs.x86_64 3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-devel.x86_64 3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-fuse.x86_64 3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-geo-replication.x86_64 3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-rdma.x86_64 3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-server.x86_64 3.3.1-1.el6
> >>>>> installed
> >>>>>
> >>>>> Thanks.
> >>>>
> >>>> _______________________________________________
> >>>> Gluster-users mailing list
> >>>> Gluster-users at gluster.org
> >>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>> --
> >>> Vennlig hilsen
> >>> Torbj?rn Thorsen
> >>> Utvikler / driftstekniker
> >>>
> >>> Trollweb Solutions AS
> >>> - Professional Magento Partner
> >>> www.trollweb.no
> >>>
> >>> Telefon dagtid: +47 51215300
> >>> Telefon kveld/helg: For kunder med Serviceavtale
> >>>
> >>> Bes?ksadresse: Luramyrveien 40, 4313 Sandnes
> >>> Postadresse: Maurholen 57, 4316 Sandnes
> >>>
> >>> Husk at alle v?re standard-vilk?r alltid er gjeldende
> >>
> >> _______________________________________________
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org
> >> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >
> > ---
> > Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
> > [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
> > 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
> > MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
> > ---
> > "Something must be done. [X] is something. Therefore, we must do it."
> > Bruce Schneier, on American response to just about anything.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
"Something must be done. [X] is something. Therefore, we must do it."
Bruce Schneier, on American response to just about anything.
