Just did that, below is the output. Didn't seem to move after the boot, and no new lines when the I/O errors appeared. Also, as mentionned I tried moving the disk on NFS and had the exact same errors, so it doesn't look like it's a libgfapi problem .. I should probably re-create the VM, maybe the errors from this night corrupted the disk and I now get errors unrelated to the original issue. Let me re-create the VM from scratch and try to reproduce the problem with the logs enabled, maybe it'll be more informative than this ! [2016-05-25 13:56:30.851493] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up [2016-05-25 13:56:30.851553] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport [2016-05-25 13:56:30.852130] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport [2016-05-25 13:56:30.852650] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport [2016-05-25 13:56:30.852909] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0) [2016-05-25 13:56:30.853434] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0) [2016-05-25 13:56:30.853484] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0) [2016-05-25 13:56:30.854182] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 13:56:30.854398] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 13:56:30.854441] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-05-25 13:56:30.861931] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 13:56:30.861965] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 13:56:30.862073] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-2' came back up; going online. [2016-05-25 13:56:30.862139] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1 [2016-05-25 13:56:30.865451] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 13:56:30.865485] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 13:56:30.865757] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1 [2016-05-25 13:56:30.865826] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'. [2016-05-25 13:56:30.865841] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-05-25 13:56:30.888604] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1 [2016-05-25 13:56:30.890388] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2 [2016-05-25 13:56:30.890731] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0) On Wed, May 25, 2016 at 02:48:27PM +0530, Krutika Dhananjay wrote: > Also, it seems Lindsay knows a way to get the gluster client logs when > using proxmox and libgfapi. > Would it be possible for you to get that sorted with Lindsay's help before > recreating this issue next time > and share the glusterfs client logs from all the nodes when you do hit the > issue? > It is critical for some of the debugging we do. :) > > -Krutika > On Wed, May 25, 2016 at 2:38 PM, Krutika Dhananjay <kdhananj@xxxxxxxxxx> > wrote: > > Hi Kevin, > > If you actually ran into a 'read-only filesystem' issue, then it could > possibly because of a bug in AFR > that Pranith recently fixed. > To confirm if that is indeed the case, could you tell meA if you saw > the pause after a brick (single brick) was > down while IO was going on? > > -Krutika > On Wed, May 25, 2016 at 1:28 PM, Kevin Lemonnier <lemonnierk@xxxxxxxxx> > wrote: > > >A A Whats the underlying filesystem under the bricks? > > I use XFS, I read that was recommended. What are you using ? > Since yours seems to work, I'm not opposed to changing ! > -- > Kevin Lemonnier > PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
Attachment:
signature.asc
Description: Digital signature
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
