There, re-created the VM from scratch, and still got the same errors. Attached are the logs, I created the VM on node 50, worked fine. I tried to reboot it and start my import again, still worked fine. I powered off the VM, then started it again on node 2, rebooted it a bunch and just got the error as usual, just attached a screen of the VM's console, might help. I can see that everytime the VM powers down, glusterFS complains about an inode still active, might it be the problem ? Thanks for the help ! On Wed, May 25, 2016 at 04:10:02PM +0200, Kevin Lemonnier wrote: > Just did that, below is the output. > Didn't seem to move after the boot, and no new lines when the I/O errors appeared. > Also, as mentionned I tried moving the disk on NFS and had the exact same errors, > so it doesn't look like it's a libgfapi problem .. > I should probably re-create the VM, maybe the errors from this night corrupted > the disk and I now get errors unrelated to the original issue. > > Let me re-create the VM from scratch and try to reproduce the problem with > the logs enabled, maybe it'll be more informative than this ! > > > [2016-05-25 13:56:30.851493] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up > [2016-05-25 13:56:30.851553] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport > [2016-05-25 13:56:30.852130] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport > [2016-05-25 13:56:30.852650] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport > [2016-05-25 13:56:30.852909] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0) > [2016-05-25 13:56:30.853434] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0) > [2016-05-25 13:56:30.853484] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0) > [2016-05-25 13:56:30.854182] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) > [2016-05-25 13:56:30.854398] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) > [2016-05-25 13:56:30.854441] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) > [2016-05-25 13:56:30.861931] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'. > [2016-05-25 13:56:30.861965] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds > [2016-05-25 13:56:30.862073] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-2' came back up; going online. > [2016-05-25 13:56:30.862139] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1 > [2016-05-25 13:56:30.865451] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'. > [2016-05-25 13:56:30.865485] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds > [2016-05-25 13:56:30.865757] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1 > [2016-05-25 13:56:30.865826] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'. > [2016-05-25 13:56:30.865841] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds > [2016-05-25 13:56:30.888604] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1 > [2016-05-25 13:56:30.890388] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2 > [2016-05-25 13:56:30.890731] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0) > > > > On Wed, May 25, 2016 at 02:48:27PM +0530, Krutika Dhananjay wrote: > > Also, it seems Lindsay knows a way to get the gluster client logs when > > using proxmox and libgfapi. > > Would it be possible for you to get that sorted with Lindsay's help before > > recreating this issue next time > > and share the glusterfs client logs from all the nodes when you do hit the > > issue? > > It is critical for some of the debugging we do. :) > > > > -Krutika > > On Wed, May 25, 2016 at 2:38 PM, Krutika Dhananjay <kdhananj@xxxxxxxxxx> > > wrote: > > > > Hi Kevin, > > > > If you actually ran into a 'read-only filesystem' issue, then it could > > possibly because of a bug in AFR > > that Pranith recently fixed. > > To confirm if that is indeed the case, could you tell meA if you saw > > the pause after a brick (single brick) was > > down while IO was going on? > > > > -Krutika > > On Wed, May 25, 2016 at 1:28 PM, Kevin Lemonnier <lemonnierk@xxxxxxxxx> > > wrote: > > > > >A A Whats the underlying filesystem under the bricks? > > > > I use XFS, I read that was recommended. What are you using ? > > Since yours seems to work, I'm not opposed to changing ! > > -- > > Kevin Lemonnier > > PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users@xxxxxxxxxxx > > http://www.gluster.org/mailman/listinfo/gluster-users > > -- > Kevin Lemonnier > PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
[2016-05-25 14:15:51.986216] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up [2016-05-25 14:15:51.986267] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport [2016-05-25 14:15:51.986794] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport [2016-05-25 14:15:51.987164] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport [2016-05-25 14:15:51.987476] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0) [2016-05-25 14:15:51.987779] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0) [2016-05-25 14:15:51.987846] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0) [2016-05-25 14:15:51.988602] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 14:15:51.988751] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 14:15:51.988897] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 14:15:51.989164] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 14:15:51.989190] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 14:15:51.989248] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 14:15:51.989265] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 14:15:51.989267] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online. [2016-05-25 14:15:51.989310] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 14:15:51.989323] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 14:15:51.989380] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1 [2016-05-25 14:15:51.989488] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1 [2016-05-25 14:15:52.008578] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1 [2016-05-25 14:15:52.009952] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2 [2016-05-25 14:15:52.010405] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0) [2016-05-25 14:53:58.342467] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-0: current graph is no longer active, destroying rpc_client [2016-05-25 14:53:58.342546] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-1: current graph is no longer active, destroying rpc_client [2016-05-25 14:53:58.342577] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-2: current graph is no longer active, destroying rpc_client [2016-05-25 14:53:58.342605] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-0: disconnected from gluster-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 14:53:58.342623] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-1: disconnected from gluster-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 14:53:58.342681] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-2: disconnected from gluster-client-2. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 14:53:58.342683] W [MSGID: 108001] [afr-common.c:4090:afr_notify] 0-gluster-replicate-0: Client-quorum is not met [2016-05-25 14:53:58.342729] E [MSGID: 108006] [afr-common.c:4046:afr_notify] 0-gluster-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2016-05-25 14:53:58.343159] W [inode.c:1792:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3f9) [0x79401d003239] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x4e) [0x79401cd7484e] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xe3) [0x79401cd74753] ) 0-gfapi: Active inode(0x7940020002cc) with refcount(1) found during cleanup [2016-05-25 14:53:58.344168] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=84 max=439 total=439 [2016-05-25 14:53:58.345653] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=156 max=444 total=1051 [2016-05-25 14:53:58.348407] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=108 max=33 total=17888 [2016-05-25 14:53:58.348468] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=588 max=111 total=401948 [2016-05-25 14:53:58.348488] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=124 max=111 total=401948 [2016-05-25 14:53:58.348508] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=1300 max=64 total=173435 [2016-05-25 14:53:58.348525] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=588 max=111 total=402594 [2016-05-25 14:53:58.348670] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=124 max=111 total=402594 [2016-05-25 14:53:58.348691] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=1300 max=64 total=174080 [2016-05-25 14:53:58.348709] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=588 max=62 total=426158 [2016-05-25 14:53:58.348722] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=124 max=62 total=426158 [2016-05-25 14:53:58.348737] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=1300 max=62 total=197641 [2016-05-25 14:53:58.348753] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-replicate-0: size=10524 max=243 total=453393 [2016-05-25 14:53:58.349278] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=1148 max=0 total=0 [2016-05-25 14:53:58.349438] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=2284 max=111 total=343736 [2016-05-25 14:53:58.349672] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-shard: size=1396 max=110 total=115585 [2016-05-25 14:53:58.349693] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-readdir-ahead: size=52 max=0 total=0 [2016-05-25 14:53:58.349713] I [io-stats.c:2951:fini] 0-gluster: io-stats translator unloaded [2016-05-25 14:53:58.350395] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2 [2016-05-25 14:53:58.350407] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1 [2016-05-25 14:57:19.258341] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up [2016-05-25 14:57:19.258393] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport [2016-05-25 14:57:19.258937] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport [2016-05-25 14:57:19.259336] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport [2016-05-25 14:57:19.259691] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0) [2016-05-25 14:57:19.259944] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0) [2016-05-25 14:57:19.260495] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0) [2016-05-25 14:57:19.260854] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 14:57:19.260954] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 14:57:19.261137] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 14:57:19.261371] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 14:57:19.261394] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 14:57:19.261415] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 14:57:19.261432] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 14:57:19.261472] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online. [2016-05-25 14:57:19.261575] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 14:57:19.261597] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 14:57:19.261602] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1 [2016-05-25 14:57:19.261647] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1 [2016-05-25 14:57:19.272598] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1 [2016-05-25 14:57:19.274117] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2 [2016-05-25 14:57:19.274445] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0) [2016-05-25 15:29:42.577939] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-0: current graph is no longer active, destroying rpc_client [2016-05-25 15:29:42.577977] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-1: current graph is no longer active, destroying rpc_client [2016-05-25 15:29:42.577994] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-2: current graph is no longer active, destroying rpc_client [2016-05-25 15:29:42.577991] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-0: disconnected from gluster-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 15:29:42.578032] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-1: disconnected from gluster-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 15:29:42.578043] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-2: disconnected from gluster-client-2. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 15:29:42.578079] W [MSGID: 108001] [afr-common.c:4090:afr_notify] 0-gluster-replicate-0: Client-quorum is not met [2016-05-25 15:29:42.578112] E [MSGID: 108006] [afr-common.c:4046:afr_notify] 0-gluster-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2016-05-25 15:29:42.578342] W [inode.c:1792:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3f9) [0x71a50b99e239] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x4e) [0x71a50b70f84e] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xe3) [0x71a50b70f753] ) 0-gfapi: Active inode(0x71a4f0c002cc) with refcount(1) found during cleanup [2016-05-25 15:29:42.579064] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=84 max=464 total=464 [2016-05-25 15:29:42.580114] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=156 max=466 total=1362 [2016-05-25 15:29:42.582010] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=108 max=32 total=28137 [2016-05-25 15:29:42.582026] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=588 max=59 total=394992 [2016-05-25 15:29:42.582035] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=124 max=59 total=394992 [2016-05-25 15:29:42.582045] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=1300 max=59 total=148576 [2016-05-25 15:29:42.582132] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=588 max=60 total=395060 [2016-05-25 15:29:42.582141] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=124 max=60 total=395060 [2016-05-25 15:29:42.582149] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=1300 max=59 total=148641 [2016-05-25 15:29:42.582159] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=588 max=36 total=432612 [2016-05-25 15:29:42.582185] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=124 max=36 total=432612 [2016-05-25 15:29:42.582216] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=1300 max=36 total=186181 [2016-05-25 15:29:42.582296] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-replicate-0: size=10524 max=125 total=513483 [2016-05-25 15:29:42.582586] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=1148 max=0 total=0 [2016-05-25 15:29:42.582664] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=2284 max=53 total=397149 [2016-05-25 15:29:42.582787] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-shard: size=1396 max=53 total=151134 [2016-05-25 15:29:42.582800] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-readdir-ahead: size=52 max=0 total=0 [2016-05-25 15:29:42.582819] I [io-stats.c:2951:fini] 0-gluster: io-stats translator unloaded [2016-05-25 15:29:42.583446] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1 [2016-05-25 15:29:42.583441] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2
[2016-05-25 15:30:28.098039] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 73322e70-6172-6170-6861-726d617a656e (0) coming up [2016-05-25 15:30:28.098100] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport [2016-05-25 15:30:28.098744] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport [2016-05-25 15:30:28.099261] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport [2016-05-25 15:30:28.100732] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0) [2016-05-25 15:30:28.100808] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0) [2016-05-25 15:30:28.100880] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0) [2016-05-25 15:30:28.101713] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 15:30:28.101959] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 15:30:28.102139] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 15:30:28.102150] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 15:30:28.102175] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 15:30:28.102248] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online. [2016-05-25 15:30:28.102287] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1 [2016-05-25 15:30:28.102400] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 15:30:28.102413] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 15:30:28.102605] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1 [2016-05-25 15:30:28.102654] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 15:30:28.102671] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 15:30:28.118768] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1 [2016-05-25 15:30:28.120203] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-0 [2016-05-25 15:30:28.120644] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 73322e70-6172-6170-6861-726d617a656e (0) [2016-05-25 15:41:56.201463] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-0: current graph is no longer active, destroying rpc_client [2016-05-25 15:41:56.201533] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-1: current graph is no longer active, destroying rpc_client [2016-05-25 15:41:56.201566] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-2: current graph is no longer active, destroying rpc_client [2016-05-25 15:41:56.201576] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-0: disconnected from gluster-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 15:41:56.201587] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-1: disconnected from gluster-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 15:41:56.201632] W [MSGID: 108001] [afr-common.c:4090:afr_notify] 0-gluster-replicate-0: Client-quorum is not met [2016-05-25 15:41:56.201638] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-2: disconnected from gluster-client-2. Client process will keep trying to connect to glusterd until brick's port is available [2016-05-25 15:41:56.201670] E [MSGID: 108006] [afr-common.c:4046:afr_notify] 0-gluster-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2016-05-25 15:41:56.207335] W [inode.c:1792:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3f9) [0x64a06250d239] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x4e) [0x64a06227e84e] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xe3) [0x64a06227e753] ) 0-gfapi: Active inode(0x64a0478002cc) with refcount(1) found during cleanup [2016-05-25 15:41:56.207969] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=84 max=338 total=338 [2016-05-25 15:41:56.209033] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=156 max=340 total=877 [2016-05-25 15:41:56.210896] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=108 max=38 total=14869 [2016-05-25 15:41:56.210922] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=588 max=25 total=173570 [2016-05-25 15:41:56.210932] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=124 max=25 total=173570 [2016-05-25 15:41:56.210945] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=1300 max=24 total=77362 [2016-05-25 15:41:56.211045] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=588 max=28 total=156927 [2016-05-25 15:41:56.211057] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=124 max=28 total=156927 [2016-05-25 15:41:56.211068] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=1300 max=27 total=60719 [2016-05-25 15:41:56.211079] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=588 max=26 total=156954 [2016-05-25 15:41:56.211117] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=124 max=26 total=156954 [2016-05-25 15:41:56.211151] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=1300 max=25 total=60745 [2016-05-25 15:41:56.211257] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-replicate-0: size=10524 max=60 total=207019 [2016-05-25 15:41:56.211549] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=1148 max=0 total=0 [2016-05-25 15:41:56.211630] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=2284 max=21 total=159841 [2016-05-25 15:41:56.211771] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-shard: size=1396 max=20 total=63784 [2016-05-25 15:41:56.211788] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-readdir-ahead: size=52 max=0 total=0 [2016-05-25 15:41:56.211801] I [io-stats.c:2951:fini] 0-gluster: io-stats translator unloaded [2016-05-25 15:41:56.211967] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1 [2016-05-25 15:41:56.211963] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2 [2016-05-25 15:42:58.096640] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 73322e70-6172-6170-6861-726d617a656e (0) coming up [2016-05-25 15:42:58.096687] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport [2016-05-25 15:42:58.097266] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport [2016-05-25 15:42:58.097518] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0) [2016-05-25 15:42:58.097815] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport [2016-05-25 15:42:58.098087] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 15:42:58.098299] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0) [2016-05-25 15:42:58.098554] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 15:42:58.098573] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 15:42:58.098633] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online. [2016-05-25 15:42:58.098673] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1 [2016-05-25 15:42:58.098790] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0) [2016-05-25 15:42:58.098898] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 15:42:58.099345] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 15:42:58.099360] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 15:42:58.099459] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 15:42:58.099533] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1 [2016-05-25 15:42:58.101238] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 15:42:58.101269] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 15:42:58.123883] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1 [2016-05-25 15:42:58.125043] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-0 [2016-05-25 15:42:58.125633] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 73322e70-6172-6170-6861-726d617a656e (0)
Attachment:
VM.png
Description: PNG image
Attachment:
signature.asc
Description: Digital signature
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
