On 8/17/17, 9:28 AM, "Gregory Farnum" <gfarnum@xxxxxxxxxx> wrote: On Thu, Aug 17, 2017 at 9:18 AM, Kamble, Nitin A <Nitin.Kamble@xxxxxxxxxxxx> wrote: > > > On 8/16/17, 6:42 PM, "Dan Mick" <dmick@xxxxxxxxxx> wrote: > > > > > "overall": { > > > "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50, > > > "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3 > > > } > > > } > > > > > > > > > # ceph-osd --version > > > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc) > > > > > > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4? > > > > You didn't say you restarted the osds. Did you? > > > > Yes, I did restart them, and that too more than once. > > Dunno, then. That seems impossible. > > It is showing the version of OSD before the upgrade. > Is it possible the old version gets cached somewhere, and that is not getting updated after upgrade? I looked at the source for this briefly yesterday and it did not seem possible; the OSDs report their version as part of the "daemon metadata" and that is committed to disk/paxos when they boot. It's then read out of their to support this command. If you could create a ticket at tracker.ceph.com for this, that would be good! Ok, I can create the tracker. Another possibility could be the osd daemons are not able to talk with monitor/manager after the upgrade. I am seeing this error in the osd log 2017-08-17 09:31:22.036901 7f2eee6d8cc0 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.219788 seconds 2017-08-17 09:31:22.041983 7f2eee6d8cc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors 2017-08-17 09:31:22.041988 7f2eee6d8cc0 -1 osd.2 0 OSD:init: unable to mount object store 2017-08-17 09:31:22.041994 7f2eee6d8cc0 -1 ** ERROR: osd init failed: (5) Input/output error 2017-08-17 09:31:42.397497 7f302f72bcc0 0 set uid:gid to 500:167 (ceph:ceph) 2017-08-17 09:31:42.397509 7f302f72bcc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 9208 24 Nitin ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f