On Tue, 22 Sep 2009, Anand Avati wrote: >>> What does the server log have to say? Can you also check if port-1 is >>> the active port in ibv_devinfo? Looks like ib-verbs messaging is not >>> happening. Does ibv_srq_pingpong give sane results? >>> >> >> [ 3.890311] ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, >> 2008) >> [ 3.890315] ib_mthca: Initializing 0000:08:00.0 >> [ 3.890354] ib_mthca 0000:08:00.0: PCI INT A -> GSI 28 (level, low) -> >> IRQ 28 >> [ 7.899804] ADDRCONF(NETDEV_UP): ib0: link is not ready >> [ 7.902722] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready >> [ 17.935013] ib0: no IPv6 routers present >> >> >> [root at xen1 ~]# ibv_devinfo >> hca_id: mthca0 >> fw_ver: 3.5.0 >> node_guid: 0005:ad00:0003:27e8 >> sys_image_guid: 0005:ad00:0100:d050 >> vendor_id: 0x02c9 >> vendor_part_id: 23108 >> hw_ver: 0xA1 >> board_id: MT_0270110001 >> phys_port_cnt: 2 >> port: 1 >> state: active (4) >> max_mtu: 2048 (4) >> active_mtu: 2048 (4) >> sm_lid: 2 >> port_lid: 11 >> port_lmc: 0x00 >> >> port: 2 >> state: down (1) >> max_mtu: 2048 (4) >> active_mtu: 512 (2) >> sm_lid: 0 >> port_lid: 0 >> port_lmc: 0x00 >> >> >> >> [root at xen1 ~]# ibv_srq_pingpong 10.13.0.220 >> local address: LID 0x000b, QPN 0x300406, PSN 0x1ace41 >> local address: LID 0x000b, QPN 0x300407, PSN 0x6ba197 >> local address: LID 0x000b, QPN 0x300408, PSN 0xa6f895 >> local address: LID 0x000b, QPN 0x300409, PSN 0xf054c0 >> local address: LID 0x000b, QPN 0x30040a, PSN 0xea4bd3 >> local address: LID 0x000b, QPN 0x30040b, PSN 0xfe3039 >> local address: LID 0x000b, QPN 0x30040c, PSN 0x037fa4 >> local address: LID 0x000b, QPN 0x30040d, PSN 0x1feccd >> local address: LID 0x000b, QPN 0x30040e, PSN 0x22daed >> local address: LID 0x000b, QPN 0x30040f, PSN 0xcaa26b >> local address: LID 0x000b, QPN 0x300410, PSN 0xe87f33 >> local address: LID 0x000b, QPN 0x300411, PSN 0x84bb4a >> local address: LID 0x000b, QPN 0x300412, PSN 0x09286e >> local address: LID 0x000b, QPN 0x300413, PSN 0xecf483 >> local address: LID 0x000b, QPN 0x300414, PSN 0xd55285 >> local address: LID 0x000b, QPN 0x300415, PSN 0xdd7065 >> remote address: LID 0x0004, QPN 0x460406, PSN 0x198bcb >> remote address: LID 0x0004, QPN 0x460407, PSN 0x645159 >> remote address: LID 0x0004, QPN 0x460408, PSN 0x4a1a2f >> remote address: LID 0x0004, QPN 0x460409, PSN 0x8dff52 >> remote address: LID 0x0004, QPN 0x46040a, PSN 0xe317fd >> remote address: LID 0x0004, QPN 0x46040b, PSN 0x12da1b >> remote address: LID 0x0004, QPN 0x460418, PSN 0xc8e0de >> remote address: LID 0x0004, QPN 0x460419, PSN 0xfc6e7f >> remote address: LID 0x0004, QPN 0x46041a, PSN 0xa3ffb7 >> remote address: LID 0x0004, QPN 0x46041b, PSN 0x0cc86d >> remote address: LID 0x0004, QPN 0x46041c, PSN 0x107a0d >> remote address: LID 0x0004, QPN 0x46041d, PSN 0xe2661c >> remote address: LID 0x0004, QPN 0x46041e, PSN 0xfb8fd8 >> remote address: LID 0x0004, QPN 0x46041f, PSN 0xc438a5 >> remote address: LID 0x0004, QPN 0x460420, PSN 0x0be0ff >> remote address: LID 0x0004, QPN 0x460421, PSN 0x91b657 >> >> This one ends odd, I think it should tell me more info, but it just sits >> there. > > Just starting ibv_srq_pingpong makes it the "server". You should run > "ibv_srq_pingpong <HOST>" from a second server and then the two will > ping-pong each other. Can you please post that output as well? The above was from the client using the IP address of the server. The server showed: [root at xen0 ~]# ibv_srq_pingpong local address: LID 0x0004, QPN 0x460406, PSN 0x198bcb local address: LID 0x0004, QPN 0x460407, PSN 0x645159 local address: LID 0x0004, QPN 0x460408, PSN 0x4a1a2f local address: LID 0x0004, QPN 0x460409, PSN 0x8dff52 local address: LID 0x0004, QPN 0x46040a, PSN 0xe317fd local address: LID 0x0004, QPN 0x46040b, PSN 0x12da1b local address: LID 0x0004, QPN 0x460418, PSN 0xc8e0de local address: LID 0x0004, QPN 0x460419, PSN 0xfc6e7f local address: LID 0x0004, QPN 0x46041a, PSN 0xa3ffb7 local address: LID 0x0004, QPN 0x46041b, PSN 0x0cc86d local address: LID 0x0004, QPN 0x46041c, PSN 0x107a0d local address: LID 0x0004, QPN 0x46041d, PSN 0xe2661c local address: LID 0x0004, QPN 0x46041e, PSN 0xfb8fd8 local address: LID 0x0004, QPN 0x46041f, PSN 0xc438a5 local address: LID 0x0004, QPN 0x460420, PSN 0x0be0ff local address: LID 0x0004, QPN 0x460421, PSN 0x91b657 remote address: LID 0x000b, QPN 0x300406, PSN 0x1ace41 remote address: LID 0x000b, QPN 0x300407, PSN 0x6ba197 remote address: LID 0x000b, QPN 0x300408, PSN 0xa6f895 remote address: LID 0x000b, QPN 0x300409, PSN 0xf054c0 remote address: LID 0x000b, QPN 0x30040a, PSN 0xea4bd3 remote address: LID 0x000b, QPN 0x30040b, PSN 0xfe3039 remote address: LID 0x000b, QPN 0x30040c, PSN 0x037fa4 remote address: LID 0x000b, QPN 0x30040d, PSN 0x1feccd remote address: LID 0x000b, QPN 0x30040e, PSN 0x22daed remote address: LID 0x000b, QPN 0x30040f, PSN 0xcaa26b remote address: LID 0x000b, QPN 0x300410, PSN 0xe87f33 remote address: LID 0x000b, QPN 0x300411, PSN 0x84bb4a remote address: LID 0x000b, QPN 0x300412, PSN 0x09286e remote address: LID 0x000b, QPN 0x300413, PSN 0xecf483 remote address: LID 0x000b, QPN 0x300414, PSN 0xd55285 remote address: LID 0x000b, QPN 0x300415, PSN 0xdd7065 -Nathan
