"ceph --admin-daemon <path/to/admin/socket> mon_status" "ceph --admin-daemon <path/to/admin/socket> quorum_status" from each monitor. These are good ones to know for debugging any sort of monitor issue as they let you see each monitor's perspective on its status and the total cluster status. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Jul 23, 2013 at 5:02 PM, Mandell Degerness <mandell@xxxxxxxxxxxxxxx> wrote: > Unfortunately, the test cluster was blown away and I only have the > logs here. What commands, precisely, would you like to see the output > from if we recreate this problem? > > On Tue, Jul 23, 2013 at 4:57 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >> Can you get the quorum and related dumps out of the admin socket for >> each running monitor and see what they say? >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Tue, Jul 23, 2013 at 4:51 PM, Mandell Degerness >> <mandell@xxxxxxxxxxxxxxx> wrote: >>> One of our tests last night failed in a weird way. We started with a >>> three node cluster, with three monitors, expanded to a 5 node cluster >>> with 5 monitors and dropped back to a 4 node cluster with three >>> monitors. >>> >>> The sequence of events was: >>> >>> start 3 monitors (monitors 0, 1, 2) - monmap e1 >>> add one node >>> restart the 3 monitors >>> add another node >>> add monitor 4 - monmap e2 >>> restart monitor 0 >>> add monitor 3 - monmap e3 >>> restart monitor 1 >>> restart monitor 2 >>> shutdown server with monitor 4 on it >>> remove monitor 4 - monmap e4 >>> restart monitor 0 >>> mon.0 had an odd time sync problem and respawned >>> stop monitor 3 >>> remove monitor 3 >>> >>> At that point (08:23:52 in the log), ceph stopped responding (as if >>> quorum was lost). Note that we do not see a new monmap (e5) created >>> by the removal of monitor 3. >>> >>> See the (sort of) full log at: >>> https://gist.github.com/mdegerne/06fa38243bd462c46d39 >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |