This is good news, so something was definitely going on. Could you compress the file using bzip or something and see how much size it would be. I will check and see if there is a way for you to upload the logs. Pranith ----- Original Message ----- > From: "Franco Broi" <franco.broi@xxxxxxxxxx> > To: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > Cc: gluster-users@xxxxxxxxxxx > Sent: Monday, June 2, 2014 2:48:56 PM > Subject: Re: glusterfsd process spinning > > Hi Pranith > > Here's a listing of the brick logs, looks very odd especially the size > of the log for data10. > > [root@nas3 bricks]# ls -ltrh > total 2.6G > -rw------- 1 root root 381K May 13 12:15 data12-gvol.log-20140511 > -rw------- 1 root root 430M May 13 12:15 data11-gvol.log-20140511 > -rw------- 1 root root 328K May 13 12:15 data9-gvol.log-20140511 > -rw------- 1 root root 2.0M May 13 12:15 data10-gvol.log-20140511 > -rw------- 1 root root 0 May 18 03:43 data10-gvol.log-20140525 > -rw------- 1 root root 0 May 18 03:43 data11-gvol.log-20140525 > -rw------- 1 root root 0 May 18 03:43 data12-gvol.log-20140525 > -rw------- 1 root root 0 May 18 03:43 data9-gvol.log-20140525 > -rw------- 1 root root 0 May 25 03:19 data10-gvol.log-20140601 > -rw------- 1 root root 0 May 25 03:19 data11-gvol.log-20140601 > -rw------- 1 root root 0 May 25 03:19 data9-gvol.log-20140601 > -rw------- 1 root root 98M May 26 03:04 data12-gvol.log-20140518 > -rw------- 1 root root 0 Jun 1 03:37 data10-gvol.log > -rw------- 1 root root 0 Jun 1 03:37 data11-gvol.log > -rw------- 1 root root 0 Jun 1 03:37 data12-gvol.log > -rw------- 1 root root 0 Jun 1 03:37 data9-gvol.log > -rw------- 1 root root 1.8G Jun 2 16:35 data10-gvol.log-20140518 > -rw------- 1 root root 279M Jun 2 16:35 data9-gvol.log-20140518 > -rw------- 1 root root 328K Jun 2 16:35 data12-gvol.log-20140601 > -rw------- 1 root root 8.3M Jun 2 16:35 data11-gvol.log-20140518 > > Too big to post everything. > > Cheers, > > On Sun, 2014-06-01 at 22:00 -0400, Pranith Kumar Karampuri wrote: > > > > ----- Original Message ----- > > > From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > > > To: "Franco Broi" <franco.broi@xxxxxxxxxx> > > > Cc: gluster-users@xxxxxxxxxxx > > > Sent: Monday, June 2, 2014 7:01:34 AM > > > Subject: Re: glusterfsd process spinning > > > > > > > > > > > > ----- Original Message ----- > > > > From: "Franco Broi" <franco.broi@xxxxxxxxxx> > > > > To: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > > > > Cc: gluster-users@xxxxxxxxxxx > > > > Sent: Sunday, June 1, 2014 10:53:51 AM > > > > Subject: Re: glusterfsd process spinning > > > > > > > > > > > > The volume is almost completely idle now and the CPU for the brick > > > > process has returned to normal. I've included the profile and I think > > > > it > > > > shows the latency for the bad brick (data12) is unusually high, > > > > probably > > > > indicating the filesystem is at fault after all?? > > > > > > I am not sure if we can believe the outputs now that you say the brick > > > returned to normal. Next time it is acting up, do the same procedure and > > > post the result. > > > > On second thought may be its not a bad idea to inspect the log files of the > > bricks in nas3. Could you post them. > > > > Pranith > > > > > > > > Pranith > > > > > > > > On Sun, 2014-06-01 at 01:01 -0400, Pranith Kumar Karampuri wrote: > > > > > Franco, > > > > > Could you do the following to get more information: > > > > > > > > > > "gluster volume profile <volname> start" > > > > > > > > > > Wait for some time, this will start gathering what operations are > > > > > coming > > > > > to > > > > > all the bricks" > > > > > Now execute "gluster volume profile <volname> info" > > > > > > /file/you/should/reply/to/this/mail/with > > > > > > > > > > Then execute: > > > > > gluster volume profile <volname> stop > > > > > > > > > > Lets see if this throws any light on the problem at hand > > > > > > > > > > Pranith > > > > > ----- Original Message ----- > > > > > > From: "Franco Broi" <franco.broi@xxxxxxxxxx> > > > > > > To: gluster-users@xxxxxxxxxxx > > > > > > Sent: Sunday, June 1, 2014 9:02:48 AM > > > > > > Subject: glusterfsd process spinning > > > > > > > > > > > > Hi > > > > > > > > > > > > I've been suffering from continual problems with my gluster > > > > > > filesystem > > > > > > slowing down due to what I thought was congestion on a single brick > > > > > > being caused by a problem with the underlying filesystem running > > > > > > slow > > > > > > but I've just noticed that the glusterfsd process for that > > > > > > particular > > > > > > brick is running at 100%+, even when the filesystem is almost idle. > > > > > > > > > > > > I've done a couple of straces of the brick and another on the same > > > > > > server, does the high number of futex errors give any clues as to > > > > > > what > > > > > > might be wrong? > > > > > > > > > > > > % time seconds usecs/call calls errors syscall > > > > > > ------ ----------- ----------- --------- --------- ---------------- > > > > > > 45.58 0.027554 0 191665 20772 futex > > > > > > 28.26 0.017084 0 137133 readv > > > > > > 26.04 0.015743 0 66259 epoll_wait > > > > > > 0.13 0.000077 3 23 writev > > > > > > 0.00 0.000000 0 1 epoll_ctl > > > > > > ------ ----------- ----------- --------- --------- ---------------- > > > > > > 100.00 0.060458 395081 20772 total > > > > > > > > > > > > % time seconds usecs/call calls errors syscall > > > > > > ------ ----------- ----------- --------- --------- ---------------- > > > > > > 99.25 0.334020 133 2516 epoll_wait > > > > > > 0.40 0.001347 0 4090 26 futex > > > > > > 0.35 0.001192 0 5064 readv > > > > > > 0.00 0.000000 0 20 writev > > > > > > ------ ----------- ----------- --------- --------- ---------------- > > > > > > 100.00 0.336559 11690 26 total > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > _______________________________________________ > > > > > > Gluster-users mailing list > > > > > > Gluster-users@xxxxxxxxxxx > > > > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > > > > > > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
