I have lots of clients hitting my bricks and I noticed that the bricks thought that they were being syn flooded so they were sending out syn cookies. I was also having these sorts of disconnects so I turned off syn cookies, not sure yet if it helped but I haven't had a disconnect since then. On Fri, Apr 13, 2012 at 4:59 AM, anthony garnier <sokar6012 at hotmail.com> wrote: > Amar, > > > I'm facing an other issue, don't know if there is a link with the quota > issue. > > I got random disconnections during worload : > > I use the same scripts as previoulsy. > (https://bugzilla.redhat.com/show_bug.cgi?id=812230 ) > > Config info : > I'm accessing the mount point with nfs. I'm connected to ylal3510 > > > 10.80.49.10 ylal3510 > 10.80.49.11 ylal3520 > 10.80.49.12 ylal3530 > 10.80.49.13 ylal3540 > > Volume Name: poolsave > Type: Distributed-Replicate > Status: Started > Number of Bricks: 2 x 2 = 4 > Transport-type: tcp > Bricks: > Brick1: ylal3510:/users3/poolsave > Brick2: ylal3530:/users3/poolsave > Brick3: ylal3520:/users3/poolsave > Brick4: ylal3540:/users3/poolsave > Options Reconfigured: > features.limit-usage: /yval9000:1TB > diagnostics.count-fop-hits: on > diagnostics.latency-measurement: on > performance.io-thread-count: 64 > nfs.port: 2049 > performance.cache-refresh-timeout: 2 > performance.cache-max-file-size: 4GB > performance.cache-min-file-size: 1KB > network.ping-timeout: 60 > performance.cache-size: 6GB > features.quota: on > features.quota-timeout: 30 > nfs.enable-ino32: off > > > > > Here is the log on the first storage node (ylal3510): nfs.log > > First disconnection : > > [2012-04-12 03:01:40.700893] W [quota.c:2165:quota_fstat_cbk] > 0-poolsave-quota: quota context not set in inode (ino:-2126980194, > gfid:ec0c049e-78b9-4972-b14c-2fa60f503e29) > [2012-04-12 03:01:40.708029] W [quota.c:2165:quota_fstat_cbk] > 0-poolsave-quota: quota context not set in inode (ino:-2126980194, > gfid:ec0c049e-78b9-4972-b14c-2fa60f503e29) > > [2012-04-12 03:09:17.454283] C > [client-handshake.c:121:rpc_client_ping_timer_expired] 0-poolsave-client-1: > server 10.80.49.12:24010 has not responded in the last 30 seconds, > disconnecting. > [2012-04-12 03:09:17.455307] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FSYNC(16)) called at 2012-04-12 03:08:17.264052 > [2012-04-12 03:09:17.455413] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.480558] I [socket.c:2275:socket_submit_request] > 0-poolsave-client-1: not connected (priv->connected = 0) > [2012-04-12 03:09:17.480659] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709557x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.480710] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.480875] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709558x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.480918] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.481006] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709559x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.481117] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.481198] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709560x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.481237] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.481326] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709561x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.481364] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.481448] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709562x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.481487] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.481565] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709563x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.481604] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.481820] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 03:08:37.235705 > [2012-04-12 03:09:17.481872] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.482093] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709564x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.482198] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 03:08:37.235895 > [2012-04-12 03:09:17.482243] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-poolsave-client-1: remote operation failed: Transport endpoint is not > connected > [2012-04-12 03:09:17.482417] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709565x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-1) > [2012-04-12 03:09:17.482513] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236009 > [2012-04-12 03:09:17.482562] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.482650] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236370 > [2012-04-12 03:09:17.482722] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.482856] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236455 > [2012-04-12 03:09:17.482901] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.482986] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236631 > [2012-04-12 03:09:17.483028] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.483147] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237248 > [2012-04-12 03:09:17.483191] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.483357] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237537 > [2012-04-12 03:09:17.483402] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.483523] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237664 > [2012-04-12 03:09:17.483568] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.483689] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237960 > [2012-04-12 03:09:17.483734] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.483855] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238258 > [2012-04-12 03:09:17.483924] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.484045] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238554 > [2012-04-12 03:09:17.484090] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.484207] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238848 > [2012-04-12 03:09:17.484252] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.484373] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.239070 > [2012-04-12 03:09:17.484417] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote > operation failed: Transport endpoint is not connected > [2012-04-12 03:09:17.484536] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame > type(GlusterFS Handshake) op(PING(3)) called at 2012-04-12 03:08:47.450735 > [2012-04-12 03:09:17.484587] W [client-handshake.c:264:client_ping_cbk] > 0-poolsave-client-1: timer must have expired > [2012-04-12 03:09:17.484622] I [client.c:1883:client_rpc_notify] > 0-poolsave-client-1: disconnected > [2012-04-12 03:09:17.485639] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.485717] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.485834] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.485873] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.486432] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.486471] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.486620] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.486659] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.486820] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.486933] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.487075] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.487120] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.487245] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.487283] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.487429] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.487489] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.487628] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.487671] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.487827] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.487868] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.488009] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.488046] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:17.488174] W [client3_1-fops.c:4868:client3_1_finodelk] > 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD > [2012-04-12 03:09:17.488211] W [client3_1-fops.c:4915:client3_1_finodelk] > 0-poolsave-client-1: failed to send the fop: File descriptor in bad state > [2012-04-12 03:09:23.347623] I > [client-handshake.c:1090:select_server_supported_programs] > 0-poolsave-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version > (310) > [2012-04-12 03:09:23.348507] I [client-handshake.c:913:client_setvolume_cbk] > 0-poolsave-client-1: Connected to 10.80.49.12:24010, attached to remote > volume '/users3/poolsave'. > [2012-04-12 03:09:23.348540] I > [client-handshake.c:779:client_post_handshake] 0-poolsave-client-1: 3 fds > open - Delaying child_up until they are re-opened > [2012-04-12 03:09:23.348632] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_setvolume_cbk+0x1b2) > [0x7fc2a205aee2] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_post_handshake+0x193) > [0x7fc2a205acf3] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(protocol_client_reopen+0x75) > [0x7fc2a205a1b5]))) 0-poolsave/inode: no dentry for non-root inode > -6906554876: d0344eb6-6f41-4d14-8855-e94fa9d541b3 > [2012-04-12 03:09:23.348665] W > [client-handshake.c:683:protocol_client_reopen] 0-poolsave-client-1: > couldn't build path from inode d0344eb6-6f41-4d14-8855-e94fa9d541b3 > [2012-04-12 03:09:23.348680] E > [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the > re-open request > [2012-04-12 03:09:23.348724] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_setvolume_cbk+0x1b2) > [0x7fc2a205aee2] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_post_handshake+0x193) > [0x7fc2a205acf3] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(protocol_client_reopen+0x75) > [0x7fc2a205a1b5]))) 0-poolsave/inode: no dentry for non-root inode > 800038244: 70c18c81-cc2d-4698-b28d-2fba654a9ef8 > [2012-04-12 03:09:23.348749] W > [client-handshake.c:683:protocol_client_reopen] 0-poolsave-client-1: > couldn't build path from inode 70c18c81-cc2d-4698-b28d-2fba654a9ef8 > [2012-04-12 03:09:23.348802] E > [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the > re-open request > [2012-04-12 03:09:23.349450] I [client-lk.c:617:decrement_reopen_fd_count] > 0-poolsave-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP > [2012-04-12 03:09:23.352548] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:23.354603] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:23.356460] W [dict.c:418:dict_unref] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) > [0x7fc2a3fa6712] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client3_1_fstat_cbk+0x2c9) > [0x7fc2a20551a9] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0x1d0) > [0x7fc2a1efa590]))) 0-dict: dict is NULL > [2012-04-12 03:09:41.442063] I > [afr-self-heal-algorithm.c:520:sh_diff_loop_driver_done] > 0-poolsave-replicate-0: diff self-heal on > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar: completed. (5249 > blocks of 73670 were different (7.13%)) > [2012-04-12 03:09:41.443242] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.443660] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.444944] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.446530] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.446960] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.448516] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.450345] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.450774] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.452061] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.453921] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.454448] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.455766] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.457581] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.457959] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.459287] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.460707] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.460955] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.462187] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.463562] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.463885] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.465196] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.466648] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.466945] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.468313] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.469879] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.470142] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.471485] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.472874] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.473142] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.474379] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.475981] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.476246] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.477659] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.479086] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.479400] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.480587] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.482176] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.482444] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.483829] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.485441] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.485728] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.487107] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.488585] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:41.488858] I [afr-open.c:432:afr_openfd_sh] > 0-poolsave-replicate-0:? data missing-entry gfid self-heal triggered. path: > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up > down flush, data lock is held > [2012-04-12 03:09:41.490300] I > [afr-self-heal-common.c:1203:sh_missing_entries_create] > 0-poolsave-replicate-0: no missing files - > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to > metadata check > [2012-04-12 03:09:41.491699] I > [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] > 0-poolsave-replicate-0: background? data missing-entry gfid self-heal > completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:13:57.87121] W [quota.c:2165:quota_fstat_cbk] > 0-poolsave-quota: quota context not set in inode (ino:-2969972712, > gfid:9f59f832-ee26-455f-8230-a42e254c43de) > [2012-04-12 03:13:57.90601] W [quota.c:2165:quota_fstat_cbk] > 0-poolsave-quota: quota context not set in inode (ino:-2969972712, > gfid:9f59f832-ee26-455f-8230-a42e254c43de) > [2012-04-12 03:30:01.295850] W [quota.c:2165:quota_fstat_cbk] > 0-poolsave-quota: quota context not set in inode (ino:-1737201406, > gfid:a85246c1-e3b2-4dd4-afff-0e4d1c3b2634) > > A lot of those warning "quota context not set in inode" and then an another > disconnection : > > [2012-04-12 05:09:43.310624] C > [client-handshake.c:121:rpc_client_ping_timer_expired] 0-poolsave-client-3: > server 10.80.49.13:24010 has not responded in the last 30 seconds, > disconnecting. > [2012-04-12 05:09:43.311087] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame > type(GlusterFS 3.1) op(FSYNC(16)) called at 2012-04-12 05:08:50.912592 > [2012-04-12 05:09:43.311243] I [client3_1-fops.c:779:client3_1_fsync_cbk] > 0-poolsave-client-3: remote operation failed: Transport endpoint is not > connected > [2012-04-12 05:09:43.311341] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame > type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303333 > [2012-04-12 05:09:43.311387] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-poolsave-client-3: remote operation failed: Transport endpoint is not > connected > [2012-04-12 05:09:43.332543] I [socket.c:2275:socket_submit_request] > 0-poolsave-client-3: not connected (priv->connected = 0) > [2012-04-12 05:09:43.332650] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367995x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-3) > [2012-04-12 05:09:43.332763] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367996x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (poolsave-client-3) > [2012-04-12 05:09:43.332821] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame > type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303419 > [2012-04-12 05:09:43.332891] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote > operation failed: Transport endpoint is not connected > [2012-04-12 05:09:43.333319] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-poolsave-client-3: remote operation failed: Transport endpoint is not > connected > [2012-04-12 05:09:43.333730] W [rpc-clnt.c:1417:rpc_clnt_submit] > 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367997x Program: > GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-3) > [2012-04-12 05:09:43.333852] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 05:09:05.303477 > [2012-04-12 05:09:43.333900] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote > operation failed: Transport endpoint is not connected > [2012-04-12 05:09:43.334017] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame > type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 05:09:05.303508 > [2012-04-12 05:09:43.334062] E > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote > operation failed: Transport endpoint is not connected > [2012-04-12 05:09:43.334154] E [rpc-clnt.c:341:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) > [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) > [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame > type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303758 > > > Log of ylal3530 for fisrt disconnection of 3:09am : > (log/glusterfs/bricks/users3-poolsave.log) > > [2012-04-12 03:09:23.346130] W [socket.c:204:__socket_rwv] > 0-tcp.poolsave-server: writev failed (Broken pipe) > [2012-04-12 03:09:23.346328] I [socket.c:1768:socket_event_handler] > 0-transport: disconnecting now > [2012-04-12 03:09:23.346452] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:23.346530] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:23.346551] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:23.346604] I [socket.c:2338:socket_submit_reply] > 0-tcp.poolsave-server: not connected (priv->connected = 255) > [2012-04-12 03:09:23.346625] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) > [0x7f5ec426d106] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) > [0x7f5ec426ce09] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) > [0x7f5ec426bb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode > 400019122: 70c18c81-cc2d-4698-b28d-2fba654a9ef8 > [2012-04-12 03:09:23.346714] E [rpcsvc.c:1033:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x10709543x, Program: > GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport > (tcp.poolsave-server) > [2012-04-12 03:09:23.346780] I [server-helpers.c:492:do_fd_cleanup] > 0-poolsave-server: fd cleanup on ino 400019122 with gfid > 70c18c81-cc2d-4698-b28d-2fba654a9ef8 > [2012-04-12 03:09:23.347047] E [server.c:136:server_submit_reply] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) > [0x7f5ec44ae109] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) > [0x7f5ec439be73] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) > [0x7f5ec42760e8]))) 0-: Reply submission failed > [2012-04-12 03:09:23.347074] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) > [0x7f5ec426d106] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) > [0x7f5ec426ce09] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) > [0x7f5ec426bb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode > -3453277438: d0344eb6-6f41-4d14-8855-e94fa9d541b3 > [2012-04-12 03:09:23.347162] I [server-helpers.c:492:do_fd_cleanup] > 0-poolsave-server: fd cleanup on ino -3453277438 with gfid > d0344eb6-6f41-4d14-8855-e94fa9d541b3 > [2012-04-12 03:09:23.347187] I [server-helpers.c:485:do_fd_cleanup] > 0-poolsave-server: fd cleanup on > /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar > [2012-04-12 03:09:23.347204] I [server.c:438:server_rpc_notify] > 0-poolsave-server: disconnected connection from 10.80.49.10:1015 > [2012-04-12 03:09:23.347226] I > [server-helpers.c:783:server_connection_destroy] 0-poolsave-server: > destroyed connection of > ylal3510-25300-2012/04/11-09:11:19:688358-poolsave-client-1 > [2012-04-12 03:09:23.348129] I [server-handshake.c:542:server_setvolume] > 0-poolsave-server: accepted client from 10.80.49.10:1008 (version: 3.2.5) > > > log of ylal3540 for disconnection of 5:09 am : > > [2012-04-12 05:09:47.540357] W [socket.c:1494:__socket_proto_state_machine] > 0-tcp.poolsave-server: reading from socket failed. Error (Transport endpoint > is not connected), peer (10.80.49.10:1012) > [2012-04-12 05:09:47.540473] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540507] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540525] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540543] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540564] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540582] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540583] I [socket.c:2338:socket_submit_reply] > 0-tcp.poolsave-server: not connected (priv->connected = 255) > [2012-04-12 05:09:47.540602] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540807] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540705] E [rpcsvc.c:1033:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x13367977x, Program: > GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport > (tcp.poolsave-server) > [2012-04-12 05:09:47.540831] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.540739] E [rpcsvc.c:1033:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x13367984x, Program: > GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport > (tcp.poolsave-server) > [2012-04-12 05:09:47.540935] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.541141] E [server.c:136:server_submit_reply] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) > [0x7f9952011109] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) > [0x7f9951efee73] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) > [0x7f9951dd90e8]))) 0-: Reply submission failed > [2012-04-12 05:09:47.541059] E [server.c:136:server_submit_reply] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) > [0x7f9952011109] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) > [0x7f9951efee73] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) > [0x7f9951dd90e8]))) 0-: Reply submission failed > [2012-04-12 05:09:47.541252] E [server.c:136:server_submit_reply] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) > [0x7f9952011109] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) > [0x7f9951efee73] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) > [0x7f9951dd90e8]))) 0-: Reply submission failed > [2012-04-12 05:09:47.541210] E [rpcsvc.c:1033:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x13367987x, Program: > GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport > (tcp.poolsave-server) > [2012-04-12 05:09:47.541164] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.541552] E [server.c:136:server_submit_reply] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) > [0x7f9952011109] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) > [0x7f9951efee73] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) > [0x7f9951dd90e8]))) 0-: Reply submission failed > [2012-04-12 05:09:47.541593] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.541618] E [rpcsvc.c:1033:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x13367988x, Program: > GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport > (tcp.poolsave-server) > [2012-04-12 05:09:47.541665] I [server-helpers.c:343:do_lock_table_cleanup] > 0-poolsave-server: finodelk released on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.541788] E [server.c:136:server_submit_reply] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) > [0x7f9952011109] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) > [0x7f9951efee73] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) > [0x7f9951dd90e8]))) 0-: Reply submission failed > [2012-04-12 05:09:47.541837] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) > [0x7f9951dd0106] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) > [0x7f9951dcfe09] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) > [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode > -450063628: bfdedf4b-671e-4d2f-ba7a-27362ab26bbe > [2012-04-12 05:09:47.541929] I [server-helpers.c:492:do_fd_cleanup] > 0-poolsave-server: fd cleanup on ino -450063628 with gfid > bfdedf4b-671e-4d2f-ba7a-27362ab26bbe > [2012-04-12 05:09:47.541983] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) > [0x7f9951dd0106] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) > [0x7f9951dcfe09] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) > [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode > 393879387: 157da663-ff30-4ba9-bd24-0dc45a561197 > [2012-04-12 05:09:47.542007] I [server-helpers.c:492:do_fd_cleanup] > 0-poolsave-server: fd cleanup on ino 393879387 with gfid > 157da663-ff30-4ba9-bd24-0dc45a561197 > [2012-04-12 05:09:47.542106] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) > [0x7f9951dd0106] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) > [0x7f9951dcfe09] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) > [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode > -1147302459: e88b34c2-9453-4e6b-b4e0-066706bd875e > [2012-04-12 05:09:47.542129] I [server-helpers.c:492:do_fd_cleanup] > 0-poolsave-server: fd cleanup on ino -1147302459 with gfid > e88b34c2-9453-4e6b-b4e0-066706bd875e > [2012-04-12 05:09:47.542173] W [inode.c:1044:inode_path] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) > [0x7f9951dd0106] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) > [0x7f9951dcfe09] > (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) > [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode > -462949261: dbec18c6-4f55-442d-ba76-267929f1cdfa > [2012-04-12 05:09:47.542194] I [server-helpers.c:492:do_fd_cleanup] > 0-poolsave-server: fd cleanup on ino -462949261 with gfid > dbec18c6-4f55-442d-ba76-267929f1cdfa > [2012-04-12 05:09:47.542231] I [server-helpers.c:485:do_fd_cleanup] > 0-poolsave-server: fd cleanup on > /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar > [2012-04-12 05:09:47.542247] I [server.c:438:server_rpc_notify] > 0-poolsave-server: disconnected connection from 10.80.49.10:1012 > [2012-04-12 05:09:47.542282] I > [server-helpers.c:783:server_connection_destroy] 0-poolsave-server: > destroyed connection of > ylal3510-25300-2012/04/11-09:11:19:688358-poolsave-client-3 > [2012-04-12 05:09:47.542349] I [server-handshake.c:542:server_setvolume] > 0-poolsave-server: accepted client from 10.80.49.10:1015 (version: 3.2.5) > > > > Regards, > > Anthony > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
