Amar, Ravi - > This thread has been one of the largest effort to stabilize the systems in > recent times. Well thanks to you guys too. It would have been easy to stop replying when things got hard. I understand best effort community support and appreciate you sticking with us. The test system I had is disappearing on Monday. However, a larger test system will be less booked after a release finalizes. So I have a test platform through early next week, and will again have something in a couple weeks. I also may have a window at 1k nodes at a customer site during a maintenance window... And we should have a couple big ones going through the factory in the comings weeks in the 1k size. At 1k nodes, we have 3 gluster servers. THANKS AGAIN. Wow what a relief. Let me get these changes checked in so I can get it to some customers and then look at getting a new thread going on the thread hangs. Erik > > Thanks for patience and number of retries you did, Erik! > > Thanks indeed! Once https://review.gluster.org/#/c/glusterfs/+/24316/ gets > merged on master, I will back port it to the release branches. > > > We surely need to get to the glitch you found with the 7.4 version, as with > every higher version, we expect more stability! > > True, maybe we should start a separate thread... > > Regards, > Ravi > > Regards, > Amar > > On Fri, Apr 17, 2020 at 2:46 AM Erik Jacobson <erik.jacobson@xxxxxxx> > wrote: > > I have some news. > > After many many many trials, reboots of gluster servers, reboots of > nodes... > in what should have reproduced the issue several times. I think we're > stable. > > It appears this glusterfs nfs daemon hang only happens in glusterfs74 > and not 72. > > So.... > 1) Your split-brain patch > 2) performance.parallel-readdir off > 3) glusterfs72 > > I declare it stable. I can't make it fail: split-brain, hang, noor seg > fault > with one leader down. > > I'm working on putting this in to a SW update. > > We are going to test if performance.parallel-readdir off impacts > booting > at scale but we don't have a system to try it on at this time. > > THAK YOU! > > I may have access to the 57 node test system if there is something > you'd > like me to try with regards to why glusterfs74 is unstable in this > situation. Just let me know. > > Erik > > On Thu, Apr 16, 2020 at 12:03:33PM -0500, Erik Jacobson wrote: > > So in my test runs since making that change, we have a different odd > > behavior now. As you recall, this is with your patch -- still not > > split-brain -- and now with performance.parallel-readdir off > > > > The NFS server grinds to a hault after a few test runs. It does not > core > > dump. > > > > All that shows up in the log is: > > > > "pending frames:" with nothing after it and no date stamp. > > > > I will start looking for interesting break points I guess. > > > > > > The glusterfs for nfs is still alive: > > > > root 30541 1 42 09:57 ? 00:51:06 /usr/sbin/glusterfs > -s localhost --volfile-id gluster/nfs -p /var/run/gluster/nfs/nfs.pid > -l /var/log/glusterfs/nfs.log -S /var/run/gluster/ > 9ddb5561058ff543.socket > > > > > > > > [root@leader3 ~]# strace -f -p 30541 > > strace: Process 30541 attached with 40 threads > > [pid 30580] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30579] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30578] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30577] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30576] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30575] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30574] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30573] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30572] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30571] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30570] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30569] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30568] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30567] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30566] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30565] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30564] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30563] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30562] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30561] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30560] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30559] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30558] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30557] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30556] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30555] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30554] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30553] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30552] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30551] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30550] restart_syscall(<... resuming interrupted restart_syscall > ...> <unfinished ...> > > [pid 30549] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30548] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=243775} > <unfinished ...> > > [pid 30546] restart_syscall(<... resuming interrupted restart_syscall > ...> <unfinished ...> > > [pid 30545] restart_syscall(<... resuming interrupted restart_syscall > ...> <unfinished ...> > > [pid 30544] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30543] rt_sigtimedwait([HUP INT USR1 USR2 TERM], <unfinished > ...> > > [pid 30542] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL > <unfinished ...> > > [pid 30541] futex(0x7f890c3a39d0, FUTEX_WAIT, 30548, NULL <unfinished > ...> > > [pid 30547] <... select resumed> ) = 0 (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 > (Timeout) > > [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}^ > Cstrace: Process 30541 detached > > strace: Process 30542 detached > > strace: Process 30543 detached > > strace: Process 30544 detached > > strace: Process 30545 detached > > strace: Process 30546 detached > > strace: Process 30547 detached > > <detached ...> > > strace: Process 30548 detached > > strace: Process 30549 detached > > strace: Process 30550 detached > > strace: Process 30551 detached > > strace: Process 30552 detached > > strace: Process 30553 detached > > strace: Process 30554 detached > > strace: Process 30555 detached > > strace: Process 30556 detached > > strace: Process 30557 detached > > strace: Process 30558 detached > > strace: Process 30559 detached > > strace: Process 30560 detached > > strace: Process 30561 detached > > strace: Process 30562 detached > > strace: Process 30563 detached > > strace: Process 30564 detached > > strace: Process 30565 detached > > strace: Process 30566 detached > > strace: Process 30567 detached > > strace: Process 30568 detached > > strace: Process 30569 detached > > strace: Process 30570 detached > > strace: Process 30571 detached > > strace: Process 30572 detached > > strace: Process 30573 detached > > strace: Process 30574 detached > > strace: Process 30575 detached > > strace: Process 30576 detached > > strace: Process 30577 detached > > strace: Process 30578 detached > > strace: Process 30579 detached > > strace: Process 30580 detached > > > > > > > > > > > On 16/04/20 8:04 pm, Erik Jacobson wrote: > > > > Quick update just on how this got set. > > > > > > > > gluster volume set cm_shared performance.parallel-readdir on > > > > > > > > Is something we did turn on, thinking it might make our NFS > services > > > > faster and not knowing about it possibly being negative. > > > > > > > > Below is a diff of the nfs volume file ON vs OFF. So I will > simply turn > > > > this OFF and do a test run. > > > Yes,that should do it. I am not sure if > performance.parallel-readdir was > > > intentionally made to have an effect on gnfs volfiles. Usually, for > other > > > performance xlators, `gluster volume set` only changes the fuse > volfile. > > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > -- > https://kadalu.io > Container Storage made easy! > > Erik Jacobson Software Engineer erik.jacobson@xxxxxxx +1 612 851 0550 Office Eagan, MN hpe.com ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
