I've run into a problem with Gluster stability with the RDMA transport. Below is a description of the environment, a simple script that can replicate the problem, and log files from my test system.
I can work around the problem by using the TCP transport over IPoIB but would like some input onto what may be making the RDMA transport fail in this case.
=====
Symptoms
=====
- Error from test script (included below): "Could not open write: out.node02.15: Invalid argument"
- Client write failure when multiple clients are reading and writing while
using the RDMA transport.
- High memory usage on one Gluster server. In this case node06. Output is from 'top' command.
- node06: 14850 root 16 0 23.1g 17g 1956 S 120.9 56.6 8:38.78 glusterfsd
- node05: 12633 root 16 0 418m 157m 1852 S 0.0 0.5 2:56.02 glusterfsd
- node04: 21066 root 15 0 355m 151m 1852 S 0.0 0.6 1:07.71 glusterfsd
- Temporary work around by using IPoIB instead of RDMA
- May take 10 - 15 minutes for first failure.
=====
Version Information
=====
- CentOS 5.6 kernel 2.6.18-238.9.1.el5
- OFED 1.5.3.1
- Gluster 3.2.1 RPMs
- Ext3 filesystem
=====
Roles of nodes
=====
node04, node05, node06 - Gluster servers.
node01, node02, node03 - Clients. Mount node04:/gluster-vol01 on /gluster and run the test script in /gluster/test
=====
Gluster Volume Info.
=====
Volume Name: gluster-vol01
Type: Distribute
Status: Started
Number of Bricks: 3
Transport-type: rdma
Bricks:
Brick1: node04:/gluster-raw-storage
Brick2: node05:/gluster-raw-storage
Brick3: node06:/gluster-raw-storage
=====
Gluster peer status.
=====
Number of Peers: 2
Hostname: node06
Uuid: 2c5f66b3-ddc8-4811-bd45-f12c60a22891
State: Peer in Cluster (Connected)
Hostname: node05
Uuid: 00b8d063-8d74-4ffe-9a44-c50e46eca78c
State: Peer in Cluster (Connected)
=====
Simple test script. Run in /gluster/test. The files read.1-4 are 2 GB files
created with dd if=/dev/zero of=read.4 bs=1024k count=2048
=====
#!/usr/bin/perl
$| = 0;
$/ = undef;
$hostname = $ENV{HOSTNAME};
my $i = 1;
my $x = 1;
while ($i) {
if ($i >= 5) { $i = 1 }
print "Read: read.$i\n";
open(FILE, "read.$i") || die "Could not open read.$i: $!\n";
my $string = <FILE>;
close(FILE);
open(OUT, ">out.$hostname.$x") || die "Could not open write: out.$hostname.$x: $!\n";
print OUT "This was read.$i\n";
close(OUT);
$i++;
$x++;
}
=====
Log from client that failed.
=====
[2011-07-11 10:19:51.498271] W [write-behind.c:3029:init] 0-gluster-vol01-write-behind: disabling write-behind for first 0 bytes
[2011-07-11 10:19:51.857980] I [client.c:1935:notify] 0-gluster-vol01-client-0: parent translators are ready, attempting connect on transport
[2011-07-11 10:19:51.861556] I [client.c:1935:notify] 0-gluster-vol01-client-1: parent translators are ready, attempting connect on transport
[2011-07-11 10:19:51.865051] I [client.c:1935:notify] 0-gluster-vol01-client-2: parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
1: volume gluster-vol01-client-0
2: type protocol/client
3: option remote-host node04
4: option remote-subvolume /gluster-raw-storage
5: option transport-type rdma
6: end-volume
7:
8: volume gluster-vol01-client-1
9: type protocol/client
10: option remote-host node05
11: option remote-subvolume /gluster-raw-storage
12: option transport-type rdma
13: end-volume
14:
15: volume gluster-vol01-client-2
16: type protocol/client
17: option remote-host node06
18: option remote-subvolume /gluster-raw-storage
19: option transport-type rdma
20: end-volume
21:
22: volume gluster-vol01-dht
23: type cluster/distribute
24: subvolumes gluster-vol01-client-0 gluster-vol01-client-1 gluster-vol01-client-2
25: end-volume
26:
27: volume gluster-vol01-write-behind
28: type performance/write-behind
29: subvolumes gluster-vol01-dht
30: end-volume
31:
32: volume gluster-vol01-read-ahead
33: type performance/read-ahead
34: subvolumes gluster-vol01-write-behind
35: end-volume
36:
37: volume gluster-vol01-io-cache
38: type performance/io-cache
39: subvolumes gluster-vol01-read-ahead
40: end-volume
41:
42: volume gluster-vol01-quick-read
43: type performance/quick-read
44: subvolumes gluster-vol01-io-cache
45: end-volume
46:
47: volume gluster-vol01-stat-prefetch
48: type performance/stat-prefetch
49: subvolumes gluster-vol01-quick-read
50: end-volume
51:
52: volume gluster-vol01
53: type debug/io-stats
54: option latency-measurement off
55: option count-fop-hits off
56: subvolumes gluster-vol01-stat-prefetch
57: end-volume
+------------------------------------------------------------------------------+
[2011-07-11 10:19:51.870501] E [client-handshake.c:1163:client_query_portmap_cbk] 0-gluster-vol01-client-0: failed to get the port number for remote subvolume
[2011-07-11 10:19:51.871344] I [client.c:1883:client_rpc_notify] 0-gluster-vol01-client-0: disconnected
[2011-07-11 10:19:51.871722] E [client-handshake.c:1163:client_query_portmap_cbk] 0-gluster-vol01-client-2: failed to get the port number for remote subvolume
[2011-07-11 10:19:51.872036] I [client.c:1883:client_rpc_notify] 0-gluster-vol01-client-2: disconnected
[2011-07-11 10:19:52.253304] E [client-handshake.c:1163:client_query_portmap_cbk] 0-gluster-vol01-client-1: failed to get the port number for remote subvolume
[2011-07-11 10:19:52.253626] I [client.c:1883:client_rpc_notify] 0-gluster-vol01-client-1: disconnected
[2011-07-11 10:19:52.256681] I [fuse-bridge.c:3316:fuse_graph_setup] 0-fuse: switched to graph 0
[2011-07-11 10:19:52.256774] I [fuse-bridge.c:2897:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
[2011-07-11 10:19:55.503338] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-gluster-vol01-client-0: changing port to 24009 (from 0)
[2011-07-11 10:19:55.507930] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-gluster-vol01-client-2: changing port to 24009 (from 0)
[2011-07-11 10:19:55.511529] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-gluster-vol01-client-1: changing port to 24009 (from 0)
[2011-07-11 10:19:58.517058] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster-vol01-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-07-11 10:19:58.517230] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster-vol01-client-0: Connected to 10.1.1.4:24009, attached to remote volume '/gluster-raw-storage'.
[2011-07-11 10:19:58.521564] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster-vol01-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-07-11 10:19:58.521873] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster-vol01-client-2: Connected to 10.1.1.6:24009, attached to remote volume '/gluster-raw-storage'.
[2011-07-11 10:19:58.525207] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster-vol01-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-07-11 10:19:58.525506] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster-vol01-client-1: Connected to 10.1.1.5:24009, attached to remote volume '/gluster-raw-storage'.
[2011-07-11 10:21:20.834771] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.835168] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.835567] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.835671] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.836450] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.836750] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.837067] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.837181] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.838179] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.841091] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:20.841394] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:24.173646] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:24.173954] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:24.174258] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:24.174356] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:24.176734] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:24.177331] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:30.942090] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:30.942411] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:30.942650] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:30.942746] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:30.944956] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:30.945634] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:37.796164] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:37.796464] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:37.796794] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:37.796882] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:37.798055] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:37.798662] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:44.555551] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:44.555867] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:44.556189] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:44.557230] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:44.559343] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:47.931251] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:47.931596] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:47.931889] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:47.931988] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:47.933972] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:55.74281] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:55.74599] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:55.74984] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:55.75074] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:55.77049] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:58.738706] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: gluster-vol01-client-2: pollin received on tcp socket (peer: 10.1.1.6:24009) after handshake is complete
[2011-07-11 10:21:58.739226] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab7b9d8f6b9] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab7b9d8ee5e] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab7b9d8edce]))) 0-gluster-vol01-client-2: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2011-07-11 10:21:58.737035
[2011-07-11 10:21:58.739262] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-glusterfs: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:21:58.742823] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x57276x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:21:58.742850] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:21:58.746332] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x57277x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:21:58.746358] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:21:58.749833] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x57278x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:21:58.749859] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:21:58.753352] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x57279x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:21:58.753378] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:21:58.755034] W [page.c:922:ioc_page_error] 0-gluster-vol01-io-cache: page error for page = 0x2aaab0005800 & waitq = 0x2aaab00087b0
[2011-07-11 10:21:58.755164] I [client.c:1883:client_rpc_notify] 0-gluster-vol01-client-2: disconnected
[2011-07-11 10:21:58.755337] W [client3_1-fops.c:3597:client3_1_flush] 0-gluster-vol01-client-2: (8183861): failed to get fd ctx. EBADFD
[2011-07-11 10:21:58.755367] W [client3_1-fops.c:3627:client3_1_flush] 0-gluster-vol01-client-2: failed to send the fop: File descriptor in bad state
[2011-07-11 10:21:58.755460] W [client3_1-fops.c:2658:client3_1_stat] 0-gluster-vol01-client-2: failed to send the fop Transport endpoint is not connected
[2011-07-11 10:21:58.756083] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:58.756377] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:58.756739] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:59.153423] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster-vol01-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-07-11 10:21:59.153813] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:21:59.153925] W [rpc-common.c:64:xdr_to_generic] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) [0x2ab7b9d8f68d] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) [0x2ab7b9d8f492] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/xlator/protocol/client.so(client3_1_lookup_cbk+0xc3) [0x2aaaab1beec3]))) 0-xdr: XDR decoding failed
[2011-07-11 10:21:59.153951] E [client3_1-fops.c:2174:client3_1_lookup_cbk] 0-gluster-vol01-client-2: error
[2011-07-11 10:21:59.153973] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-gluster-vol01-client-2: remote operation failed: Invalid argument
[2011-07-11 10:21:59.154019] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster-vol01-client-2: Connected to 10.1.1.6:24009, attached to remote volume '/gluster-raw-storage'.
[2011-07-11 10:21:59.154038] I [client-handshake.c:779:client_post_handshake] 0-gluster-vol01-client-2: 1 fds open - Delaying child_up until they are re-opened
[2011-07-11 10:21:59.154430] I [client-lk.c:617:decrement_reopen_fd_count] 0-gluster-vol01-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2011-07-11 10:21:59.154477] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-gluster-vol01-client-2: sending release on fd
[2011-07-11 10:21:59.155858] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:05.532223] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:05.532542] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:05.532910] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:05.533013] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:05.534769] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:08.921201] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:08.921518] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:08.921799] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:08.922430] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:08.924503] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:15.829708] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:15.830215] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:15.830367] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:15.830640] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:15.833650] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:19.709940] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: gluster-vol01-client-2: pollin received on tcp socket (peer: 10.1.1.6:24009) after handshake is complete
[2011-07-11 10:22:19.710375] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab7b9d8f6b9] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab7b9d8ee5e] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab7b9d8edce]))) 0-gluster-vol01-client-2: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2011-07-11 10:22:19.708647
[2011-07-11 10:22:19.710406] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-glusterfs: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:19.712082] W [page.c:922:ioc_page_error] 0-gluster-vol01-io-cache: page error for page = 0x2aaab800d800 & waitq = 0x2aaab800b120
[2011-07-11 10:22:19.712189] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab7b9d8f6b9] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab7b9d8ee5e] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab7b9d8edce]))) 0-gluster-vol01-client-2: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2011-07-11 10:22:19.708691
[2011-07-11 10:22:19.712211] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-glusterfs: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:19.715785] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x82130x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:19.715831] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:19.719313] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x82131x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:19.719339] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:19.722808] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x82132x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:19.722834] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:19.726307] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x82133x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:19.726334] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:19.729801] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x82134x Program: GlusterFS 3.1, ProgVers: 310, Proc: 15) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:19.729827] I [client3_1-fops.c:734:client3_1_flush_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:19.729859] W [page.c:922:ioc_page_error] 0-gluster-vol01-io-cache: page error for page = 0x2aaab800c2a0 & waitq = 0x2aaab8000d50
[2011-07-11 10:22:19.729917] I [client.c:1883:client_rpc_notify] 0-gluster-vol01-client-2: disconnected
[2011-07-11 10:22:19.732616] W [rpc-common.c:64:xdr_to_generic] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) [0x2ab7b9d8f68d] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) [0x2ab7b9d8f492] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/xlator/protocol/client.so(client3_1_stat_cbk+0x91) [0x2aaaab1bce41]))) 0-xdr: XDR decoding failed
[2011-07-11 10:22:19.732660] E [client3_1-fops.c:398:client3_1_stat_cbk] 0-gluster-vol01-client-2: error
[2011-07-11 10:22:19.732678] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-gluster-vol01-client-2: remote operation failed: Invalid argument
[2011-07-11 10:22:19.732796] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster-vol01-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-07-11 10:22:19.733199] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:19.733418] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster-vol01-client-2: Connected to 10.1.1.6:24009, attached to remote volume '/gluster-raw-storage'.
[2011-07-11 10:22:19.733436] I [client-handshake.c:779:client_post_handshake] 0-gluster-vol01-client-2: 1 fds open - Delaying child_up until they are re-opened
[2011-07-11 10:22:19.733500] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:19.733789] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:19.733861] I [client-lk.c:617:decrement_reopen_fd_count] 0-gluster-vol01-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2011-07-11 10:22:19.734449] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:19.736692] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:25.348136] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:25.348478] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:25.348884] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:25.348985] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:25.351089] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:28.823864] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:28.824218] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:28.824528] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:28.824650] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:28.826931] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:34.435906] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:34.436234] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:34.436580] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:34.436677] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:34.439657] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:38.928523] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: gluster-vol01-client-2: pollin received on tcp socket (peer: 10.1.1.6:24009) after handshake is complete
[2011-07-11 10:22:38.928948] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab7b9d8f6b9] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab7b9d8ee5e] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab7b9d8edce]))) 0-gluster-vol01-client-2: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2011-07-11 10:22:38.927421
[2011-07-11 10:22:38.928979] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-glusterfs: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:38.932581] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x108279x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:38.932610] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:38.936092] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x108280x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:38.936118] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:38.939587] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x108281x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:38.939614] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:38.943089] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster-vol01-client-2: failed to submit rpc-request (XID: 0x108282x Program: GlusterFS 3.1, ProgVers: 310, Proc: 12) to rpc-transport (gluster-vol01-client-2)
[2011-07-11 10:22:38.943115] I [client3_1-fops.c:2302:client3_1_readv_cbk] 0-gluster-vol01-client-2: remote operation failed: Transport endpoint is not connected
[2011-07-11 10:22:38.944339] W [page.c:922:ioc_page_error] 0-gluster-vol01-io-cache: page error for page = 0x2aaab0013c00 & waitq = 0x2aaab0012e10
[2011-07-11 10:22:38.944465] I [client.c:1883:client_rpc_notify] 0-gluster-vol01-client-2: disconnected
[2011-07-11 10:22:38.944651] W [client3_1-fops.c:3597:client3_1_flush] 0-gluster-vol01-client-2: (8183861): failed to get fd ctx. EBADFD
[2011-07-11 10:22:38.944685] W [client3_1-fops.c:3627:client3_1_flush] 0-gluster-vol01-client-2: failed to send the fop: File descriptor in bad state
[2011-07-11 10:22:38.944777] W [client3_1-fops.c:2658:client3_1_stat] 0-gluster-vol01-client-2: failed to send the fop Transport endpoint is not connected
[2011-07-11 10:22:38.945113] W [client3_1-fops.c:2599:client3_1_lookup] 0-gluster-vol01-client-2: failed to send the fop: Transport endpoint is not connected
[2011-07-11 10:22:38.945274] W [client3_1-fops.c:2599:client3_1_lookup] 0-gluster-vol01-client-2: failed to send the fop: Transport endpoint is not connected
[2011-07-11 10:22:38.945398] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:38.945720] E [iobuf.c:526:iobuf_ref] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_recv_reply+0xdc) [0x2aaaac53234c] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/rpc-transport/rdma.so(rdma_pollin_notify+0x77) [0x2aaaac531c87] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_transport_pollin_alloc+0x82) [0x2ab7b9d8ab02]))) 0-iobuf: invalid argument: iobuf
[2011-07-11 10:22:39.409136] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster-vol01-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-07-11 10:22:39.409512] W [rpc-common.c:64:xdr_to_generic] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) [0x2ab7b9d8f68d] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) [0x2ab7b9d8f492] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/xlator/protocol/client.so(client3_1_statfs_cbk+0x7e) [0x2aaaab1b835e]))) 0-xdr: XDR decoding failed
[2011-07-11 10:22:39.409534] E [client3_1-fops.c:624:client3_1_statfs_cbk] 0-gluster-vol01-client-2: error
[2011-07-11 10:22:39.409566] I [client3_1-fops.c:637:client3_1_statfs_cbk] 0-gluster-vol01-client-2: remote operation failed: Invalid argument
[2011-07-11 10:22:39.409716] W [rpc-common.c:64:xdr_to_generic] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) [0x2ab7b9d8f68d] (-->/opt/glusterfs/3.2.1/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) [0x2ab7b9d8f492] (-->/opt/glusterfs/3.2.1/lib64/glusterfs/3.2.1/xlator/protocol/client.so(client3_1_create_cbk+0xe7) [0x2aaaab1c1047]))) 0-xdr: XDR decoding failed
[2011-07-11 10:22:39.409736] E [client3_1-fops.c:1682:client3_1_create_cbk] 0-gluster-vol01-client-2: error
[2011-07-11 10:22:39.409750] I [client3_1-fops.c:1722:client3_1_create_cbk] 0-gluster-vol01-client-2: remote operation failed: Invalid argument
[2011-07-11 10:22:39.409770] W [fuse-bridge.c:1567:fuse_create_cbk] 0-glusterfs-fuse: 222979: /test/out.node02.15 => -1 (Invalid argument)
[2011-07-11 10:22:39.409892] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster-vol01-client-2: Connected to 10.1.1.6:24009, attached to remote volume '/gluster-raw-storage'.
[2011-07-11 10:22:39.409913] I [client-handshake.c:779:client_post_handshake] 0-gluster-vol01-client-2: 2 fds open - Delaying child_up until they are re-opened
[2011-07-11 10:22:39.410538] I [client-lk.c:617:decrement_reopen_fd_count] 0-gluster-vol01-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2011-07-11 10:22:39.410572] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-gluster-vol01-client-2: sending release on fd
=====
Log from node04
=====
[2011-07-11 10:16:48.352084] I [glusterd.c:564:init] 0-management: Using /etc/glusterd as working directory
[2011-07-11 10:16:48.694351] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: d3f2a368-9640-40b8-a949-d4a61ac861c1
[2011-07-11 10:16:48.700503] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2011-07-11 10:16:48.700533] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2011-07-11 10:16:48.700548] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2011-07-11 10:16:48.873912] I [glusterd-handler.c:3399:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-07-11 10:16:48.876710] I [glusterd-handler.c:3399:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-07-11 10:16:48.876887] I [glusterd-utils.c:1092:glusterd_volume_start_glusterfs] 0-: About to start glusterfs for brick node04:/gluster-raw-storage
Given volfile:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option working-directory /etc/glusterd
4: option transport-type socket,rdma
5: option transport.socket.keepalive-time 10
6: option transport.socket.keepalive-interval 2
7: end-volume
8:
+------------------------------------------------------------------------------+
[2011-07-11 10:16:49.384291] I [glusterd-pmap.c:237:pmap_registry_bind] 0-pmap: adding brick /gluster-raw-storage on port 24009
[2011-07-11 10:16:49.391589] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.4:1020) after handshake is complete
[2011-07-11 10:16:49.730879] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-07-11 10:16:50.360200] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.5:1020) after handshake is complete
[2011-07-11 10:16:50.363632] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-07-11 10:16:51.20756] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.6:1020) after handshake is complete
[2011-07-11 10:16:53.398561] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.4:1021) after handshake is complete
[2011-07-11 10:16:54.372135] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.5:1017) after handshake is complete
[2011-07-11 10:16:55.28691] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.6:1021) after handshake is complete
[2011-07-11 10:19:25.616803] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.1:1022) after handshake is complete
[2011-07-11 10:19:29.240816] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.1:1019) after handshake is complete
[2011-07-11 10:19:52.165596] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.2:1022) after handshake is complete
[2011-07-11 10:19:55.798443] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.2:1019) after handshake is complete
[2011-07-11 10:19:58.860826] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.3:1022) after handshake is complete
[2011-07-11 10:20:02.504854] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.3:1019) after handshake is complete
[2011-07-11 10:27:47.193747] I [glusterd-handler.c:750:glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2011-07-11 10:27:47.195164] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:1020)
=====
Log from node05
=====
[2011-07-11 10:16:47.572132] I [glusterd.c:564:init] 0-management: Using /etc/glusterd as working directory
[2011-07-11 10:16:48.158382] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: 00b8d063-8d74-4ffe-9a44-c50e46eca78c
[2011-07-11 10:16:48.176241] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2011-07-11 10:16:48.176307] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2011-07-11 10:16:48.176333] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2011-07-11 10:16:48.536987] I [glusterd-handler.c:3399:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-07-11 10:16:48.542493] I [glusterd-handler.c:3399:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-07-11 10:16:48.542927] I [glusterd-utils.c:1092:glusterd_volume_start_glusterfs] 0-: About to start glusterfs for brick node05:/gluster-raw-storage
Given volfile:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option working-directory /etc/glusterd
4: option transport-type socket,rdma
5: option transport.socket.keepalive-time 10
6: option transport.socket.keepalive-interval 2
7: end-volume
8:
+------------------------------------------------------------------------------+
[2011-07-11 10:16:48.679495] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-07-11 10:16:48.679925] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.4:1019) after handshake is complete
[2011-07-11 10:16:49.307394] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.5:1019) after handshake is complete
[2011-07-11 10:16:49.312613] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-07-11 10:16:49.337559] I [glusterd-pmap.c:237:pmap_registry_bind] 0-pmap: adding brick /gluster-raw-storage on port 24009
[2011-07-11 10:16:50.245768] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.6:1019) after handshake is complete
[2011-07-11 10:16:52.351761] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.4:1017) after handshake is complete
[2011-07-11 10:16:53.313000] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.5:1021) after handshake is complete
[2011-07-11 10:16:53.993035] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.6:1016) after handshake is complete
[2011-07-11 10:19:25.48677] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.1:1021) after handshake is complete
[2011-07-11 10:19:28.197733] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.1:1017) after handshake is complete
[2011-07-11 10:19:51.496699] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.2:1021) after handshake is complete
[2011-07-11 10:19:54.754924] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.2:1017) after handshake is complete
[2011-07-11 10:19:58.108904] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.3:1021) after handshake is complete
[2011-07-11 10:20:01.461911] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.3:1017) after handshake is complete
[2011-07-11 10:26:41.143053] I [glusterd-handler.c:796:glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2011-07-11 10:26:41.144014] I [glusterd-handler.c:796:glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2011-07-11 10:27:25.284637] I [glusterd-handler.c:750:glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2011-07-11 10:27:36.231388] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:1020)
=====
Log from node06
=====
[2011-07-11 10:16:47.652401] I [glusterd.c:564:init] 0-management: Using /etc/glusterd as working directory
[2011-07-11 10:16:48.563424] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: 2c5f66b3-ddc8-4811-bd45-f12c60a22891
[2011-07-11 10:16:48.591609] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2011-07-11 10:16:48.591702] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2011-07-11 10:16:48.591736] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2011-07-11 10:16:49.129061] I [glusterd-handler.c:3399:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-07-11 10:16:49.140666] I [glusterd-handler.c:3399:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-07-11 10:16:49.141535] I [glusterd-utils.c:1092:glusterd_volume_start_glusterfs] 0-: About to start glusterfs for brick node06:/gluster-raw-storage
[2011-07-11 10:16:49.221893] E [socket.c:2080:socket_connect] 0-management: connection attempt failed (Connection refused)
Given volfile:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option working-directory /etc/glusterd
4: option transport-type socket,rdma
5: option transport.socket.keepalive-time 10
6: option transport.socket.keepalive-interval 2
7: end-volume
8:
+------------------------------------------------------------------------------+
[2011-07-11 10:16:49.312106] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-07-11 10:16:49.312360] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-07-11 10:16:49.313000] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.5:1018) after handshake is complete
[2011-07-11 10:16:49.969032] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.6:1018) after handshake is complete
[2011-07-11 10:16:49.975043] I [glusterd-pmap.c:237:pmap_registry_bind] 0-pmap: adding brick /gluster-raw-storage on port 24009
[2011-07-11 10:16:52.344630] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.4:1018) after handshake is complete
[2011-07-11 10:16:53.328225] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.5:1016) after handshake is complete
[2011-07-11 10:16:53.985138] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.6:1017) after handshake is complete
[2011-07-11 10:16:55.358356] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.4:1016) after handshake is complete
[2011-07-11 10:19:24.566389] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.1:1020) after handshake is complete
[2011-07-11 10:19:28.193613] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.1:1018) after handshake is complete
[2011-07-11 10:19:51.114971] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.2:1020) after handshake is complete
[2011-07-11 10:19:54.751192] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.2:1018) after handshake is complete
[2011-07-11 10:19:57.809996] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.3:1020) after handshake is complete
[2011-07-11 10:20:01.457953] E [rdma.c:4466:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 10.1.1.3:1018) after handshake is complete
--
justin penney @ advanced clustering technologies
sr. systems integrator
email + xmpp: jpenney at advancedclustering.com
phone: 913.643.0309
