Re: Upgrading inconvenience for Luminous

[David Turner <drakonstein@xxxxxxxxx>]

Having all of the daemons in your cluster able to restart themselves at will sounds terrifying. What's preventing every osd from restarting at the same time? Also, ceph dot releases have been known to break environments. It's the nature of such a widely used software. I would recommend pinning the ceph version instead.

On Wed, Feb 21, 2018, 6:09 PM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:

Dear Cephalopodians,

we had our cluster (still in testing phase) configured for automatic updates so we got 12.2.3 "automagically" when it was released.

In /etc/sysconfig/ceph, we still have the default:
CEPH_AUTO_RESTART_ON_UPGRADE=no
so as expected, services were not restarted.

However, as soon as scrubs started to run, we got many scrub errors and inconsistent PGs.
Looking into the logs, I found that some ceph-osd processes (still running as 12.2.2) tried to load
the compression library (libsnappy) dynamically, and refused to do so since it was already updated to 12.2.3 on disk.
This appears to have caused the OSD to report read errors.

The situation was reasonably easy to fix (i.e. just restart all ceph-osd processes, and re-run a deep scrub some of the inconsistent PGs).
Still, I wonder whether this could be prevented by loading the libraries at OSD startup (and never unloading them),
or by shutting down the OSD in case of a library load failure.
Did anybody else experience this as of yet?

We will work around it either by version pinning or CEPH_AUTO_RESTART_ON_UPGRADE=yes (not decided yet).

Cheers,
        Oliver

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com