Hi Richard, thank you for answer, it seems I can't go back I removed all luminous packages from all of 3 nodes and install jewel ones, on monitor node mon won't start, due disk features incompatibility root@ceph-node01:~# /usr/bin/ceph-mon -f --cluster ceph --id ceph-node01 --setuser ceph --setgroup ceph 2017-04-13 14:50:18.174662 7fe9ade9c700 -1 ERROR: on disk data includes unsupported features: compat={},rocompat={},incompat={8=support monmap features} 2017-04-13 14:50:18.174670 7fe9ade9c700 -1 error checking features: (1) Operation not permitted If I try to get something from OSDs it goes like: root@ceph-node01:~# export ms="/home/ceph/monstore"; for osd in /var/lib/ceph/osd/ceph-*; do ceph-objectstore-tool --data-path $osd --op update-mon-db --mon-store-path "$ms";done On-disk OSD incompatible features set compat={},rocompat={},incompat={14=explicit missing set,15=fastinfo pg attr} On-disk OSD incompatible features set compat={},rocompat={},incompat={14=explicit missing set,15=fastinfo pg attr} On-disk OSD incompatible features set compat={},rocompat={},incompat={14=explicit missing set,15=fastinfo pg attr} On-disk OSD incompatible features set compat={},rocompat={},incompat={14=explicit missing set,15=fastinfo pg attr}
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com