Thanks Dan, I've just managed to fixed it. It looks like the upgrade process required some extra ram, the mon node was heavily swapping, so I think it was just stalled rather than broken. Once it came back up, ram dropped down by a lot. Nick > -----Original Message----- > From: Dan van der Ster [mailto:dan@xxxxxxxxxxxxxx] > Sent: 12 April 2017 10:53 > To: Nick Fisk <nick@xxxxxxxxxx> > Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx> > Subject: Re: Mon not starting after upgrading to 10.2.7 > > Can't help, but just wanted to say that the upgrade worked for us: > > # ceph health > HEALTH_OK > # ceph tell mon.* version > mon.p01001532077488: ceph version 10.2.7 > (50e863e0f4bc8f4b9e31156de690d765af245185) > mon.p01001532149022: ceph version 10.2.7 > (50e863e0f4bc8f4b9e31156de690d765af245185) > mon.p01001532184554: ceph version 10.2.7 > (50e863e0f4bc8f4b9e31156de690d765af245185) > > -- dan > > On Wed, Apr 12, 2017 at 11:50 AM, Nick Fisk <nick@xxxxxxxxxx> wrote: > > Hi, > > > > I just upgraded one of my mons to 10.2.7 and it is now failing to > > start properly. What's really odd is all the mon specific commands are > > now missing from the admin socket. > > > > ceph --admin-daemon /var/run/ceph/ceph-mon.gp-ceph-mon2.asok help { > > "config diff": "dump diff of current config and default config", > > "config get": "config get <field>: get the config value", > > "config set": "config set <field> <val> [<val> ...]: set a config > > variable", > > "config show": "dump current config settings", > > "get_command_descriptions": "list available commands", > > "git_version": "get git sha1", > > "help": "list available commands", > > "log dump": "dump recent log entries to log file", > > "log flush": "flush log entries to log file", > > "log reopen": "reopen log file", > > "perf dump": "dump perfcounters value", > > "perf reset": "perf reset <name>: perf reset all or one > > perfcounter name", > > "perf schema": "dump perfcounters schema", > > "version": "get ceph version" > > } > > > > And from the log, with logging set to 20/20 > > 2017-04-12 10:47:35.667631 7fba2cf4d700 0 set uid:gid to 64045:64045 > > (ceph:ceph) > > 2017-04-12 10:47:35.667681 7fba2cf4d700 0 ceph version 10.2.7 > > (50e863e0f4bc8f4b9e31156de690d765af245185), process ceph-mon, pid > 7187 > > 2017-04-12 10:47:35.668335 7fba2cf4d700 0 pidfile_write: ignore empty > > --pid-file > > 2017-04-12 10:47:35.721480 7fba2cf4d700 5 asok(0x55a03144a2c0) init > > /var/run/ceph/ceph-mon.gp-ceph-mon2.asok > > 2017-04-12 10:47:35.721540 7fba2cf4d700 5 asok(0x55a03144a2c0) > > bind_and_listen /var/run/ceph/ceph-mon.gp-ceph-mon2.asok > > 2017-04-12 10:47:35.721692 7fba2cf4d700 20 asok(0x55a03144a2c0) unlink > > stale file /var/run/ceph/ceph-mon.gp-ceph-mon2.asok > > 2017-04-12 10:47:35.721729 7fba2cf4d700 5 asok(0x55a03144a2c0) > > register_command 0 hook 0x55a03142a0a8 > > 2017-04-12 10:47:35.721742 7fba2cf4d700 5 asok(0x55a03144a2c0) > > register_command version hook 0x55a03142a0a8 > > 2017-04-12 10:47:35.721752 7fba2cf4d700 5 asok(0x55a03144a2c0) > > register_command git_version hook 0x55a03142a0a8 > > 2017-04-12 10:47:35.721762 7fba2cf4d700 5 asok(0x55a03144a2c0) > > register_command help hook 0x55a03142e1d0 > > 2017-04-12 10:47:35.721771 7fba2cf4d700 5 asok(0x55a03144a2c0) > > register_command get_command_descriptions hook 0x55a03142e1e0 > > 2017-04-12 10:47:35.721902 7fba2778e700 5 asok(0x55a03144a2c0) entry > > start > > 2017-04-12 10:47:35.764691 7fba2cf4d700 10 load: jerasure load: lrc load: > > isa > > > > > > Any ideas? > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |