That's unfortunately not possible in my case as the clients are K8s worker nodes. The gluster client version is tied to the Kuberntes release/deployer - Rancher in my case.
Disabling ctime "gluster volume set $volname ctime off" fixed the mounting issue.
On Thu, Jan 30, 2020 at 4:16 AM Mahdi Adnan <mahdi@xxxxxxxxx> wrote:
Hello,We had a similar issue when we upgraded one of our clusters to 6.5 and clients were running 4.1.5 and 4.1.9, both crashed after few seconds of mounting, we did not dig into the issue instead, we upgraded the clients to 6.5 and it worked fine.On Tue, Jan 28, 2020 at 1:35 AM Laurent Dumont <laurentfdumont@xxxxxxxxx> wrote:After some more digging, it seems I'm hitting the following bug - https://github.com/gluster/glusterfs/issues/658________On Mon, Jan 27, 2020 at 4:23 PM Laurent Dumont <laurentfdumont@xxxxxxxxx> wrote:Hi everyone,Small question. I'm trying to mount a Gluster volume (server is at 6.7) and the client is at 4.1.5. I'm seeing the mount start on the client but it looks like the client crashes and is left in a strange state. Is there any inherent compatibility issues between the two versions? Is 4.1.5 too old when talking to a 6.7 server?Client :sh-4.4# glusterfs --version
glusterfs 4.1.5
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.Server :root@gluster01:~# glusterd --version
glusterfs 6.7
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.Steps :sh-4.4# mount -t glusterfs 10.10.99.29:kube_vol /test
Mount failed. Please check the log file for more details.sh-4.4# cat /var/log/glusterfs/test.log
[2020-01-27 21:03:26.770751] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=10.10.99.29 --volfile-id=kube_vol /test)
[2020-01-27 21:03:26.775717] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2020-01-27 21:03:26
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 4.1.5
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x237aa)[0x7f2e1061d7aa]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2f7)[0x7f2e10627517]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7f2e0ec6f060]
/lib/x86_64-linux-gnu/libc.so.6(+0xbf944)[0x7f2e0ecfb944]
/lib/x86_64-linux-gnu/libc.so.6(fnmatch+0x61)[0x7f2e0ecfce41]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_volume_option_get_list+0x35)[0x7f2e10670905]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7698e)[0x7f2e1067098e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach_match+0x87)[0x7f2e10614b97]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach+0x18)[0x7f2e10614d78]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate_list+0x3f)[0x7f2e10670b4f]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate+0x39)[0x7f2e10670bc9]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5ab99)[0x7f2e10654b99]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_activate+0x24)[0x7f2e106554d4]
/usr/sbin/glusterfs(glusterfs_process_volfp+0x100)[0x5643b3bc1f40]
/usr/sbin/glusterfs(mgmt_getspec_cbk+0x6bd)[0x5643b3bc889d]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe110)[0x7f2e103ec110]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe44f)[0x7f2e103ec44f]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f2e103e8863]
/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x6cfb)[0x7f2e0b4e9cfb]
/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x9605)[0x7f2e0b4ec605]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x8052e)[0x7f2e1067a52e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f2e0f47c4a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f2e0ed24d0f]
---------
[2020-01-27 21:22:35.074154] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=10.10.99.29 --volfile-id=kube_vol /test)
[2020-01-27 21:22:35.078542] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2020-01-27 21:22:35
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 4.1.5
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x237aa)[0x7f6ef309a7aa]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2f7)[0x7f6ef30a4517]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7f6ef16ec060]
/lib/x86_64-linux-gnu/libc.so.6(+0xbf944)[0x7f6ef1778944]
/lib/x86_64-linux-gnu/libc.so.6(fnmatch+0x61)[0x7f6ef1779e41]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_volume_option_get_list+0x35)[0x7f6ef30ed905]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7698e)[0x7f6ef30ed98e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach_match+0x87)[0x7f6ef3091b97]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach+0x18)[0x7f6ef3091d78]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate_list+0x3f)[0x7f6ef30edb4f]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate+0x39)[0x7f6ef30edbc9]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5ab99)[0x7f6ef30d1b99]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_activate+0x24)[0x7f6ef30d24d4]
/usr/sbin/glusterfs(glusterfs_process_volfp+0x100)[0x55b5e83a7f40]
/usr/sbin/glusterfs(mgmt_getspec_cbk+0x6bd)[0x55b5e83ae89d]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe110)[0x7f6ef2e69110]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe44f)[0x7f6ef2e6944f]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f6ef2e65863]
/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x6cfb)[0x7f6eedf66cfb]
/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x9605)[0x7f6eedf69605]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x8052e)[0x7f6ef30f752e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f6ef1ef94a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f6ef17a1d0f]
---------
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
--RespectfullyMahdi
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
