On 04/03/2013 11:49 AM, Gregory Farnum wrote: > On Wed, Apr 3, 2013 at 10:14 AM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >> On Wed, Apr 3, 2013 at 10:09 AM, Jim Schutt <jaschut@xxxxxxxxxx> wrote: >>> Hi Sage, >>> >>> On 04/03/2013 09:58 AM, Sage Weil wrote: >>>> Hi Jim, >>>> >>>> What happens if you change 'osd mon ack timeout = 300' (from the >>>> default of 30)? I suspect part of the problem is that the mons are just >>>> slow enough that the osd's resend the same thing again and it snowballs >>>> into more work for the monitor. >>> >>> Thanks, that helped. My OSDs aren't reconnecting to the mon any more, >>> and the new filesystem started up as expected. >>> >>> Hmmm, it occurs to me that I upgraded my mon hosts to 10 GbE NICs at >>> about the same time I started testing v0.59. Perhaps before the upgrade >>> I was running right at the edge of that timeout. After the NIC upgrade >>> the PGStat messages come flooding in at startup, and they bunch up >>> enough that working through the backlog pushed me over the timeout cliff? >>> >>> Is there any downside to using a large 'osd mon ack timeout', assuming I >>> run more than one mon? If so, I expect I'll work my way back from >>> 'osd mon ack timeout = 300' to see how big it needs to be to stay reliable >>> for my configuration. >> >> It's a timeout, so the generic downsides to larger timeouts — if the >> monitor actually has gone away it's going to take the OSDs more time >> to connect to somebody else for their updates and reports. This will >> probably be most apparent if they're trying to peer and can't make >> progress until they get acks from the monitors, but the one they're >> connected to has died. >> >> >>> Sorry for the noise about paxos. At least it was useful >>> to help Joao find that debug log message that was more expensive >>> than expected.... >> >> It's not noise — the reason this timeout is causing problems now is >> that the monitor disk commits are taking so long that it looks like >> they've failed. Which is bad. :/ So thanks for reporting it! > > Sorry, guess I forgot some of the history since this piece at least is > resolved now. I'm surprised if 30-second timeouts are causing issues > without those overloads you were seeing; have you seen this issue > without your high debugging levels and without the bad PG commits (due > to debugging)? I think so, because that's why I started with higher debugging levels. But, as it turns out, I'm just in the process of returning to my testing of next, with all my debugging back to 0. So, I'll try the default timeout of 30 seconds first. If I have trouble starting up a new file system, I'll turn up the timeout and try again, without any extra debugging. Either way, I'll let you know what happens. -- Jim > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
