On Fri, 2014-07-11 at 15:53 -0600, Tregaron Bayly wrote: > I have a four node ceph cluster for testing. As I'm watching the > relatively idle cluster I'm seeing quite a bit of traffic from one of > the OSD nodes to the monitor. This node has 8 OSDs and each of them are > involved in this behavior, but none of the other 24 OSDs located on the > other nodes are. > > Turning up debugging on the monitor shows this: > > 014-07-11 15:28:34.415164 7f947cf52700 10 mon.a at 1(peon).auth v420 prep_auth() blob_size=2 > 2014-07-11 15:28:34.415206 7f947cf52700 20 mon.a at 1(peon) e2 have connection > 2014-07-11 15:28:34.415210 7f947cf52700 20 mon.a at 1(peon) e2 ms_dispatch existing session MonSession: osd.14 10.8.6.230:6817/23666 is openallow rwx for osd.14 10.8.6.230:6817/23666 > 2014-07-11 15:28:34.415217 7f947cf52700 20 mon.a at 1(peon) e2 caps allow rwx > > over and over again for each of the osds on host 10.8.6.230. There's > never a message for any of the IP addresses of the other three nodes, so > this must be what is responsible for the chatter. It's not so much > traffic that I'm worried it will starve important traffic and the > cluster is HEALTH_OK; it just makes me wonder whether something is > misconfigured on my end. Following up for the list. Turning monc debug on the osd way up surfaced the corresponding client-side query repeating over and over: 2037-03-08 10:37:11.687257 7efef234f700 10 monclient: _send_mon_message to mon.nodea at 10.8.6.227:6789/0 2037-03-08 10:37:11.687292 7efef234f700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2037-03-08 10:36:41.687292) That was of course a wealth of information. First of all, it told me exactly what messages were being dispatched (renewing rotating keys). It also told me that somehow this OSD decided that it was 2037 already. O_o Fixing the system clock with ntp cleaned this right up.
 |