Dear Cephalopodians, we had our cluster (still in testing phase) configured for automatic updates so we got 12.2.3 "automagically" when it was released. In /etc/sysconfig/ceph, we still have the default: CEPH_AUTO_RESTART_ON_UPGRADE=no so as expected, services were not restarted. However, as soon as scrubs started to run, we got many scrub errors and inconsistent PGs. Looking into the logs, I found that some ceph-osd processes (still running as 12.2.2) tried to load the compression library (libsnappy) dynamically, and refused to do so since it was already updated to 12.2.3 on disk. This appears to have caused the OSD to report read errors. The situation was reasonably easy to fix (i.e. just restart all ceph-osd processes, and re-run a deep scrub some of the inconsistent PGs). Still, I wonder whether this could be prevented by loading the libraries at OSD startup (and never unloading them), or by shutting down the OSD in case of a library load failure. Did anybody else experience this as of yet? We will work around it either by version pinning or CEPH_AUTO_RESTART_ON_UPGRADE=yes (not decided yet). Cheers, Oliver
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
