Re: glusterfs 4.1.7 + nfs-ganesha 2.7.1 freeze during write

[Soumya Koduri <skoduri@xxxxxxxxxx>]

On 2/8/19 11:53 AM, Soumya Koduri wrote:
> On 2/8/19 3:20 AM, Maurits Lamers wrote:
> > Hi,
> >
> > > > [2019-02-07 10:11:24.812606] E [MSGID: 104055] 
> > > > [glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
> > > > event_type(1) and gfid(yøêÙ
> > > >          Mz„–îSL4_@) failed
> > > > [2019-02-07 10:11:24.819376] E [MSGID: 104055] 
> > > > [glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
> > > > event_type(1) and gfid(eTnôEU«H.<Y4failed
> > > > [2019-02-07 10:11:24.833299] E [MSGID: 104055] 
> > > > [glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
> > > > event_type(1) and gfid(gÇLÁèFà»0bЯk) failed
> > > > [2019-02-07 10:25:01.642509] C 
> > > > [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-2: 
> > > > server [node1]:49152 has not responded in the last 42 seconds, 
> > > > disconnecting.
> > > > [2019-02-07 10:25:01.642805] C 
> > > > [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-1: 
> > > > server [node2]:49152 has not responded in the last 42 seconds, 
> > > > disconnecting.
> > > > [2019-02-07 10:25:01.642946] C 
> > > > [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-4: 
> > > > server [node3]:49152 has not responded in the last 42 seconds, 
> > > > disconnecting.
> > > > [2019-02-07 10:25:02.643120] C 
> > > > [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-3: 
> > > > server 127.0.1.1:49152 has not responded in the last 42 seconds, 
> > > > disconnecting.
> > > > [2019-02-07 10:25:02.643314] C 
> > > > [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-0: 
> > > > server [node4]:49152 has not responded in the last 42 seconds, 
> > > > disconnecting.
> > >
> > > Strange that synctask failed. Could you please turn off 
> > > features.cache-invalidation volume option and check if the issue 
> > > still persists.
> > > >
> >
> > Turning the cache invalidation option off seems to have solved the 
> > freeze. Still testing, but it looks promising.
>
> If thats the case, please turn on cache invalidation option back and 
> collect couple of stack traces (using gstack) when the system freezes 
> again.

FYI - Have got a chance to reproduce and RCA the issue [1]. Posted fix 
for review in the upstream [2]

Thanks,
Soumya

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1693575
[2] https://review.gluster.org/22436

> Thanks,
> Soumya
> > cheers
> >
> > Maurits
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users