On 8/17/17, 9:59 AM, "Kamble, Nitin A" <Nitin.Kamble@xxxxxxxxxxxx> wrote:
On 8/17/17, 9:42 AM, "Vasu Kulkarni" <vakulkar@xxxxxxxxxx> wrote:
On Thu, Aug 17, 2017 at 9:18 AM, Kamble, Nitin A
<Nitin.Kamble@xxxxxxxxxxxx> wrote:
>
>
> On 8/16/17, 6:42 PM, "Dan Mick" <dmick@xxxxxxxxxx> wrote:
>
>
> > > "overall": {
> > > "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
> > > "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
> > > }
> > > }
> > >
> > >
> > > # ceph-osd --version
> > > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
> > >
> > > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
> >
> > You didn't say you restarted the osds. Did you?
> >
> > Yes, I did restart them, and that too more than once.
>
> Dunno, then. That seems impossible.
>
> It is showing the version of OSD before the upgrade.
> Is it possible the old version gets cached somewhere, and that is not getting updated after upgrade?
I wonder if its a systemd issue, Does the systemd restart actually
change the process id? (output of ps before/after would be good),
Hopefully all the pkgs are properly updated when you query them and
you dont see any old traces left behind.
On the osd nodes I am finding that OSD daemons are restarting with the errors mentioned in last email and shown below. Looks like because the restart time is ~5 minutes, system is not taking it as an restart loop error.
#grep " pid " /var/log/ceph/ceph-osd.2.log
2017-08-17 09:26:44.407213 7f2eee6d8cc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920308
2017-08-17 09:31:42.397509 7f302f72bcc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920824
2017-08-17 09:36:40.153434 7f56ee55bcc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921496
2017-08-17 09:38:01.971158 7fdb7857ecc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921839
2017-08-17 09:43:00.653569 7f690cedfcc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 922381
2017-08-17 09:47:58.645293 7fbc0f5accc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 923127
# grep ERROR -C 3 /var/log/ceph/ceph-osd.2.log
2017-08-17 09:31:22.036901 7f2eee6d8cc0 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.219788 seconds
2017-08-17 09:31:22.041983 7f2eee6d8cc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
2017-08-17 09:31:22.041988 7f2eee6d8cc0 -1 osd.2 0 OSD:init: unable to mount object store
2017-08-17 09:31:22.041994 7f2eee6d8cc0 -1 ** ERROR: osd init failed: (5) Input/output error
2017-08-17 09:31:42.397497 7f302f72bcc0 0 set uid:gid to 500:167 (ceph:ceph)
2017-08-17 09:31:42.397509 7f302f72bcc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920824
2017-08-17 09:31:42.402139 7f302f72bcc0 0 pidfile_write: ignore empty --pid-file
--
2017-08-17 09:36:19.832905 7f302f72bcc0 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.127693 seconds
2017-08-17 09:36:19.838803 7f302f72bcc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
2017-08-17 09:36:19.838809 7f302f72bcc0 -1 osd.2 0 OSD:init: unable to mount object store
2017-08-17 09:36:19.838815 7f302f72bcc0 -1 ** ERROR: osd init failed: (5) Input/output error
2017-08-17 09:36:40.153423 7f56ee55bcc0 0 set uid:gid to 500:167 (ceph:ceph)
2017-08-17 09:36:40.153434 7f56ee55bcc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921496
2017-08-17 09:36:40.158052 7f56ee55bcc0 0 pidfile_write: ignore empty --pid-file
--
2017-08-17 09:42:40.428889 7fdb7857ecc0 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 278.111786 seconds
2017-08-17 09:42:40.434401 7fdb7857ecc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
2017-08-17 09:42:40.434408 7fdb7857ecc0 -1 osd.2 0 OSD:init: unable to mount object store
2017-08-17 09:42:40.434414 7fdb7857ecc0 -1 ** ERROR: osd init failed: (5) Input/output error
2017-08-17 09:43:00.653558 7f690cedfcc0 0 set uid:gid to 500:167 (ceph:ceph)
2017-08-17 09:43:00.653569 7f690cedfcc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 922381
2017-08-17 09:43:00.658323 7f690cedfcc0 0 pidfile_write: ignore empty --pid-file
--
2017-08-17 09:47:38.476877 7f690cedfcc0 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.499695 seconds
2017-08-17 09:47:38.482565 7f690cedfcc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
2017-08-17 09:47:38.482572 7f690cedfcc0 -1 osd.2 0 OSD:init: unable to mount object store
2017-08-17 09:47:38.482578 7f690cedfcc0 -1 ** ERROR: osd init failed: (5) Input/output error
2017-08-17 09:47:58.645283 7fbc0f5accc0 0 set uid:gid to 500:167 (ceph:ceph)
2017-08-17 09:47:58.645293 7fbc0f5accc0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 923127
2017-08-17 09:47:58.649958 7fbc0f5accc0 0 pidfile_write: ignore empty --pid-file
There are no old version ceph packages on any of the ceph nodes. All the ceph packages are of version 12.1.4 now.
>
> Nitin
>
BTW I found a workaround which is working and giving the right versions now.
The workaround is mentioned here: https://github.com/ceph/ceph/pull/10825/files
Basically adding this line to global section of ceph.conf resolves the issue.
bluestore fsck on mount = 'false'
Thanks,
Nitin
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
