Hi everyone,
I create a replicate volume using two nodes,A board and B board.
A board ip:10.32.1.144
B board ip:10.32.0.48
One brick and mount point is on A board
Another brick is on B board
I found that I can't access the mount point because the disconnection happen between client and two server.
The client log is below.(Please notice the red line)
[2016-10-31 04:04:08.063543] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
...
[2016-10-31 04:06:03.626047] W [socket.c:588:__socket_rwv] 0-c_glusterfs-client-9: readv on 10.32.1.144:49391 failed (Connection reset by peer)
[2016-10-31 04:06:03.627345] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] )))))
0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2016-10-31 04:06:03.626033 (xid=0x7f5e)
[2016-10-31 04:06:03.627395] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 0)
[2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (c_glusterfs-client-9)
[2016-10-31 04:06:03.628466] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.628475] I [MSGID: 108019] [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: unable to lock on even one child
[2016-10-31 04:06:03.628539] I [MSGID: 108019] [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] 0-c_glusterfs-replicate-0: Blocking inodelks failed.
[2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] )))))
0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a)
[2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0)
[2016-10-31 04:06:03.629210] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]
[2016-10-31 04:06:03.629266] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 255)
[2016-10-31 04:06:03.629277] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (gfid = b0e5a94e-a432-4dce-b86f-a551555780a2). Holes=1 overlaps=0
[2016-10-31 04:06:03.629293] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f62 Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (c_glusterfs-client-9)
[2016-10-31 04:06:03.629333] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: b0e5a94e-a432-4dce-b86f-a551555780a2: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.629363] W [fuse-bridge.c:3385:fuse_getxattr_resume] 0-glusterfs-fuse: 20784: GETXATTR b0e5a94e-a432-4dce-b86f-a551555780a2/70366685109140 (system.posix_acl_access) resolution failed
[2016-10-31 04:06:03.629729] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 04:06:03.624568 (xid=0x7f5b)
[2016-10-31 04:06:03.629779] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /pmd/2 (fa866805-ab22-4afe-8acb-528d15ce4117) [Transport endpoint is not connected]
[2016-10-31 04:06:03.629784] E [MSGID: 114031] [client-rpc-fops.c:2883:client3_3_opendir_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]
[2016-10-31 04:06:03.629836] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/2 (gfid = fa866805-ab22-4afe-8acb-528d15ce4117). Holes=1 overlaps=0
[2016-10-31 04:06:03.629857] W [fuse-bridge.c:980:fuse_fd_cbk] 0-glusterfs-fuse: 20794: OPENDIR() /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.629896] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: fa866805-ab22-4afe-8acb-528d15ce4117: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.629990] E [fuse-bridge.c:787:fuse_getattr_resume] 0-glusterfs-fuse: 20785: GETATTR 70366685017568 (fa866805-ab22-4afe-8acb-528d15ce4117) resolution failed
[2016-10-31 04:06:03.630164] W [defaults.c:2212:default_releasedir] (-->/usr/lib64/glusterfs/3.7.6/xlator/protocol/client.so(client_local_wipe-0x42d64) [0x3fff86c9bc5c] -->/usr/lib64/libglusterfs.so.0(fd_unref-0x7ba98) [0x3fff8abb7ad0] -->/usr/lib64/libglusterfs.so.0(default_releasedir-0x9d6d8) [0x3fff8ab93c58] ) 0-fuse: xlator does not implement releasedir_cbk
[2016-10-31 04:06:03.630367] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 04:06:03.625675 (xid=0x7f5c)
[2016-10-31 04:06:03.630368] W [MSGID: 114031] [client-rpc-fops.c:1569:client3_3_fstat_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.630497] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) [Transport endpoint is not connected]
[2016-10-31 04:06:03.630871] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:06:03.631488] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:06:03.631523] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:06:03.631555] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-c_glusterfs-client-9: 6 fds open - Delaying child_up until they are re-opened
[2016-10-31 04:06:03.631475] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/ethmac.cfg (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2016-10-31 04:06:03.632256] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 1)
[2016-10-31 04:06:03.632728] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:257add5a-9b87-4014-8a3a-09dd0a699eec> succeeded (fd = 2)
[2016-10-31 04:06:03.632790] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:6fd47fe9-a566-47c7-8df7-e7a5e379101f> succeeded (fd = 3)
[2016-10-31 04:06:03.632847] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:fa866805-ab22-4afe-8acb-528d15ce4117> succeeded (fd = 4)
[2016-10-31 04:06:03.632905] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:139f5034-5b42-4fb8-9372-6dd7cac1e674> succeeded (fd = 5)
[2016-10-31 04:06:03.632935] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-c_glusterfs-client-9: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-10-31 04:06:03.633028] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-c_glusterfs-replicate-0: Subvolume 'c_glusterfs-client-9' came back up; going online.
[2016-10-31 04:06:03.633386] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-9: Server lk version = 1
[2016-10-31 04:06:03.634579] W [fuse-bridge.c:758:fuse_attr_cbk] 0-glusterfs-fuse: 20795: FSTAT() /configuration/oamrdncy.cfg => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.634889] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] )))))
0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3)
op(XATTROP(33)) called at 2016-10-31 04:06:03.625716 (xid=0x7f5d)
[2016-10-31 04:06:03.634950] W [MSGID: 114031] [client-rpc-fops.c:1845:client3_3_xattrop_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f)
[2016-10-31 04:06:03.635134] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-c_glusterfs-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid 200b0bb2-d903-4924-b025-262d805c0f2f. (Possible split-brain)
[2016-10-31 04:06:03.635364] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-c_glusterfs-client-9: disconnected from c_glusterfs-client-9. Client process will keep trying to connect to glusterd until brick's port is available
[2016-10-31 04:06:03.635420] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-c_glusterfs-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
analyze:
When 04:04:08 the client connect successtion with server.
When 04:06:03.626047 the disconnection happened because the readv error.
When 04:06:03.631488 the client connect successtion again.
The server log on A board is following.(Please notice the red line)
[2016-10-31 04:04:08.063317] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:08.063359] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:10.032591] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:04:10.032672] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:05:04.530186] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:04.530267] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 60a17763-95fd-4162-8402-9651dfbe9ec4 held by {client=0x3fff7c002990, pid=0 lk-owner=a87b0a7cff3f0000}
[2016-10-31 04:05:04.530306] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:05:04.530375] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760618] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760693] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:57.341892] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:05:57.341928] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-1604-2016/10/31-04:05:53:637848-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.291380] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.291441] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.403916] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.403963] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-3292-2016/10/31-04:05:57:685417-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:03.631296] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:03.631332] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:06:03.634762] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.634945] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 200b0bb2-d903-4924-b025-262d805c0f2f held by {client=0x3fff7c026520, pid=6232 lk-owner=6cd0c189ff3f0000}
[2016-10-31 04:06:03.634987] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /pmd/2
[2016-10-31 04:06:03.635035] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /configuration/oamrdncy.cfg
[2016-10-31 04:06:03.635349] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /
[2016-10-31 04:06:03.635414] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:06:03.635468] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001
[2016-10-31 04:06:03.635554] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl
[2016-10-31 04:06:03.635889] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001/cello
[2016-10-31 04:06:03.640329] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.641143] E [inodelk.c:309:__inode_unlock_lock] 0-c_glusterfs-locks: Matching lock not found for unlock 0-9223372036854775807, by 6cd0c189ff3f0000 on 0x3fff840fd9a0
[2016-10-31 04:06:03.641200] E [MSGID: 115053] [server-rpc-fops.c:275:server_inodelk_cbk] 0-c_glusterfs-server: 32623: INODELK /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) ==> (Invalid argument) [Invalid argument]
[2016-10-31 04:06:06.270314] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0
[2016-10-31 04:06:06.270449] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0
analyze：
When 04:04:08 the server accept the client（002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0 ）.
When 04:06:03 the server accept the client (002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-1 ) again.
Question:
1.I don't understand why connection immediately happen at 04:06:03.631488 after disconnect happen at 04:06:03.626047.
I think it should be 10 second interval before reconnect.
From server the second time connection even happen before disconnection.
So, why connect happen twice before disconnection?
2.After the disconnection happen at 04:06:03.626047, why the connection will not success again?
Thanks,
Xin
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
