On Thu, Apr 09, 2015 at 08:46:07AM -0700, Gregory Farnum wrote:
> On Thu, Apr 9, 2015 at 8:14 AM, Jacob Reid <lists-ceph@xxxxxxxxxxxxxxxx> wrote:
> > On Thu, Apr 09, 2015 at 06:43:45AM -0700, Gregory Farnum wrote:
> >> You can turn up debugging ("debug osd = 10" and "debug filestore = 10"
> >> are probably enough, or maybe 20 each) and see what comes out to get
> >> more information about why the threads are stuck.
> >>
> >> But just from the log my answer is the same as before, and now I don't
> >> trust that controller (or maybe its disks), regardless of what it's
> >> admitting to. ;)
> >> -Greg
> >>
> >
> > Ran with osd and filestore debug both at 20; still nothing jumping out at me. Logfile attached as it got huge fairly quickly, but mostly seems to be the same extra lines. I tried running some test I/O on the drives in question to try and provoke some kind of problem, but they seem fine now...
>
> Okay, this is strange. Something very wonky is happening with your
> scheduler — it looks like these threads are all idle, and they're
> scheduling wakeups that handle an appreciable amount of time after
> they're supposed to. For instance:
> 2015-04-09 15:56:55.953116 7f70a7963700 20
> filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.416704
> 2015-04-09 15:56:55.953153 7f70a7963700 20
> filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for
> max_interval 5.000000
>
> This is the thread that syncs your backing store, and it always sets
> itself to get woken up at 5-second intervals — but here it took >5.4
> seconds, and later on in your log it takes more than 6 seconds.
> It looks like all the threads which are getting timed out are also
> idle, but are taking so much longer to wake up than they're set for
> that they get a timeout warning.
>
> There might be some bugs in here where we're expecting wakeups to be
> more precise than they can be, but these sorts of misses are
> definitely not normal. Is this server overloaded on the CPU? Have you
> done something to make the scheduler or wakeups wonky?
> -Greg
CPU load is minimal - the host does nothing but run OSDs and has 8 cores that are all sitting idle with a load average of 0.1. I haven't done anything to scheduling. That was with the debug logging on, if that could be the cause of any delays. A scheduler issue seems possible - I haven't done anything to it, but `time sleep 5` run a few times returns anything spread randomly from 5.002 to 7.1(!) seconds but mostly in the 5.5-6.0 region where it managed fairly consistently <5.2 on the other servers in the cluster and <5.02 on my desktop. I have disabled the CPU power saving mode as the only thing I could think of that might be having an effect on this, and running the same test again gives more sane results... we'll see if this reflects in the OSD logs or not, I guess. If this is the cause, it's probably something that the next version might want to make a specific warning case of detecting. I will keep you updated as to their behaviour now...
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
