Re: OSD Restarts cause excessively high load average and "requests are blocked > 32 sec"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



How long does it take for the OSDs to restart? Are you just issuing a
restart command via upstart/sysvinit/whatever? How many OSDMaps are
generated from the time you issue that command to the time the cluster
is healthy again?

This sounds like an issue we had for a while where OSDs would start
peering before they had processed the maps they needed to look at; the
fix might not have been backported to Emperor. But I'd like to be sure
this isn't some other issue you're seeing.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Sat, Mar 22, 2014 at 8:16 PM, Quenten Grasso <qgrasso@xxxxxxxxxx> wrote:
> Hi Kyle,
>
> Thanks, I turned on debug ms = 1 and debug osd = 10 and restarted osd.54 heres here's log for that one.
>
> ceph-osd.54.log.bz2
> http://www67.zippyshare.com/v/99704627/file.html
>
>
> Strace osd 53,
> strace.zip
> http://www43.zippyshare.com/v/17581165/file.html
>
>
> Thanks,
> Quenten
> -----Original Message-----
> From: Kyle Bader [mailto:kyle.bader@xxxxxxxxx]
> Sent: Sunday, 23 March 2014 12:10 PM
> To: Quenten Grasso
> Subject: Re:  OSD Restarts cause excessively high load average and "requests are blocked > 32 sec"
>
>> Any ideas on why the load average goes so crazy & starts to block IO?
>
> Could you turn on "debug ms = 1" and "debug osd = 10" prior to restarting the OSDs on one of your hosts and sharing the logs so we can take a look?
>
> It also might be worth while to strace one of the OSDs to try to determine what it's working so hard on, maybe:
>
> strace -fc -p <osd pid>  > strace.osd1.log
>
> Thanks!
>
> --
>
> Kyle
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux