[root at fpsgluster ~]# gluster volume create testvault replica 2 transport rdma,tcp fpsglusterib:/mnt/testbrick1 fpsgluster2ib:/mnt/testbrick1 fpsglusterib:/mnt/testbrick2 fpsgluster2ib:/mnt/testbrick2 fpsglusterib:/mnt/testbrick3 fpsgluster2ib:/mnt/testbrick3
Host fpsgluster2ib not a friend
Not a friend error. I'm at a loss.
On Feb 25, 2013, at 2:40 PM, harry mangalam <harry.mangalam at uci.edu>
wrote:
> That looks OK (but your 2 MTUs are mismatched - should fix that).
>
>> UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 <- 1st
>> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 <- 2nd
> ^^^^^
> IBDEV=ibX
> modprobe ib_umad
> modprobe ib_ipoib
> echo connected > /sys/class/net/${IBDEV}/mode
> echo 65520 > /sys/class/net/${IBDEV}/mtu
>
> how did you set up the peering? By name? by IP#?
> (I assume pinging by hostname also works both ways?)
>
> If you can't get the peers to ack, then what do the logs say on failure to:
>
> peer probe <host>
> or create the volume
> gluster volume create <volname> host1ib:/gl_part host2ib:/gl_part
>
> hjm
>
> On Monday, February 25, 2013 09:50:02 PM Tony Saenz wrote:
>> Trying to first get this working with IPoIB
>>
>> [root at fpsgluster ~]# ibhosts
>> Ca : 0x00117500007937b2 ports 1 "fpsgluster2 qib0"
>> Ca : 0x0011750000792af2 ports 1 "fpsgluster qib0"
>>
>> I'm able to ping the other box from Infiniband to Infiniband card
>>
>> Ifconfig uses the ioctl access method to get the full address information,
>> which limits hardware addresses to 8 bytes. Because Infiniband address has
>> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
>> obsolete! For replacement check ip.
>> ib0 Link encap:InfiniBand HWaddr
>> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
>> addr:10.0.4.35 Bcast:10.0.4.255 Mask:255.255.255.0 inet6 addr:
>> fe80::211:7500:79:2af2/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
>> RX packets:1567 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:587 errors:0 dropped:24 overruns:0 carrier:0
>> collisions:0 txqueuelen:256
>> RX bytes:342622 (334.5 KiB) TX bytes:96554 (94.2 KiB)
>>
>> [root at fpsgluster2 ~]# ifconfig ib0
>> Ifconfig uses the ioctl access method to get the full address information,
>> which limits hardware addresses to 8 bytes. Because Infiniband address has
>> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
>> obsolete! For replacement check ip.
>> ib0 Link encap:InfiniBand HWaddr
>> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
>> addr:10.0.4.34 Bcast:10.0.4.255 Mask:255.255.255.0 inet6 addr:
>> fe80::211:7500:79:37b2/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
>> RX packets:599 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:1558 errors:0 dropped:8 overruns:0 carrier:0
>> collisions:0 txqueuelen:256
>> RX bytes:95180 (92.9 KiB) TX bytes:346728 (338.6 KiB)
>>
>> [root at fpsgluster ~]# ping -I ib0 10.0.4.34
>> PING 10.0.4.34 (10.0.4.34) from 10.0.4.35 ib0: 56(84) bytes of data.
>> 64 bytes from 10.0.4.34: icmp_seq=1 ttl=64 time=12.6 ms
>> 64 bytes from 10.0.4.34: icmp_seq=2 ttl=64 time=0.184 ms
>>
>> /etc/hosts looks correct
>>
>> [root at fpsgluster2 ~]# cat /etc/hosts | grep ib
>> 10.0.4.35 fpsglusterib
>> 10.0.4.34 fpsgluster2ib
>>
>> [root at fpsgluster ~]# cat /etc/hosts| grep ib
>> 10.0.4.35 fpsglusterib
>> 10.0.4.34 fpsgluster2ib
>>
>> I haven't created the new volume yet as I can't get the peer probe to work
>> off the Infiniband card. It's only seeing the NIC cards I currently have it
>> hooked in to.
>>
>>
>> On Feb 25, 2013, at 11:57 AM, harry mangalam <harry.mangalam at uci.edu>
>>
>> wrote:
>>> It /might be/probably is/ DNS-related.
>>>
>>> Are you trying to do this with RDMA or IPoIB?
>>>
>>> If IPoIB, are ALL your /etc/hosts files in sync (IB names separate and
>>> distinct from the ethernet interfaces) and responsive to the appropriate
>>> interfaces?
>>>
>>> Do the IB interfaces show up as distinct (and connected) on an 'ifconfig
>>> -a' and 'ibstat' dump?
>>>
>>> Do all the peers show up on an 'ibhosts' query?
>>>
>>> What is the output of:
>>> gluster volume status <your_volume>
>>> and
>>> gluster volume status <your_volume> detail
>>>
>>>
>>> hjm
>>>
>>> On Monday, February 25, 2013 07:46:00 PM Tony Saenz wrote:
>>>> It shows this but it's still going through my NIC cards and not the
>>>> Infiniband. (Checked the traffic on the cards themselves)
>>>>
>>>> [root at fpsgluster ~]# gluster peer status
>>>> Number of Peers: 1
>>>>
>>>> Hostname: fpsgluster2
>>>> Uuid: 9b7e7c2d-f05b-4cc8-b55a-571e383328d0
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> On Feb 25, 2013, at 10:51 AM, Torbj?rn Thorsen <torbjorn at trollweb.no>
> wrote:
>>>>> Your error message seems to indicate that the peer is already in the
>>>>> storage pool ?
>>>>> What is the output of "gluster peer status" ?
>>>>>
>>>>> On Mon, Feb 25, 2013 at 7:28 PM, Tony Saenz <tony at filmsolutions.com>
>>>
>>> wrote:
>>>>>> Any help please? The regular NICs are fine which is what it currently
>>>>>> sees but I'd like to move them over to the Infiniband cards.>>
>>>>>> On Feb 22, 2013, at 1:50 PM, Anthony Saenz <tony at filmsolutions.com>
>>>
>>> wrote:
>>>>>>> Hey,
>>>>>>>
>>>>>>> I was wondering if I could get a bit of help.. I installed a new
>>>>>>> Infiniband card into my servers but I'm unable to get it to come up as
>>>>>>> a peer. Is there something I'm missing?
>>>>>>>
>>>>>>> [root at fpsgluster testvault]# gluster peer probe fpsgluster2ib
>>>>>>> Probe on host fpsgluster2ib port 0 already in peer list
>>>>>>>
>>>>>>> [root at fpsgluster testvault]# yum list installed | grep gluster
>>>>>>> glusterfs.x86_64 3.3.1-1.el6
>>>>>>> installed
>>>>>>> glusterfs-devel.x86_64 3.3.1-1.el6
>>>>>>> installed
>>>>>>> glusterfs-fuse.x86_64 3.3.1-1.el6
>>>>>>> installed
>>>>>>> glusterfs-geo-replication.x86_64 3.3.1-1.el6
>>>>>>> installed
>>>>>>> glusterfs-rdma.x86_64 3.3.1-1.el6
>>>>>>> installed
>>>>>>> glusterfs-server.x86_64 3.3.1-1.el6
>>>>>>> installed
>>>>>>>
>>>>>>> Thanks.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>> --
>>>>> Vennlig hilsen
>>>>> Torbj?rn Thorsen
>>>>> Utvikler / driftstekniker
>>>>>
>>>>> Trollweb Solutions AS
>>>>> - Professional Magento Partner
>>>>> www.trollweb.no
>>>>>
>>>>> Telefon dagtid: +47 51215300
>>>>> Telefon kveld/helg: For kunder med Serviceavtale
>>>>>
>>>>> Bes?ksadresse: Luramyrveien 40, 4313 Sandnes
>>>>> Postadresse: Maurholen 57, 4316 Sandnes
>>>>>
>>>>> Husk at alle v?re standard-vilk?r alltid er gjeldende
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>> ---
>>> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
>>> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
>>> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
>>> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
>>> ---
>>> "Something must be done. [X] is something. Therefore, we must do it."
>>> Bruce Schneier, on American response to just about anything.
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
> ---
> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
> ---
> "Something must be done. [X] is something. Therefore, we must do it."
> Bruce Schneier, on American response to just about anything.
