Hi all,
I did a pretty bit mistake doing our upgrade from hammer to luminous,
skipping the jewel release.
When I realized and tried to switch back to jewel, it was too late -
the cluster now won't start, complaining about "The disk uses features
unsupported by the executable.":
2017-11-17 01:27:26.190971 7fb446ab58c0 0 ceph version 0.94.10
(b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 19638
2017-11-17 01:27:26.209600 7fb446ab58c0 0
filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2017-11-17 01:27:26.277323 7fb446ab58c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
FIEMAP ioctl is supported and appears to work
2017-11-17 01:27:26.277353 7fb446ab58c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-11-17 01:27:26.302508 7fb446ab58c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2017-11-17 01:27:26.302668 7fb446ab58c0 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
disabled by conf
2017-11-17 01:27:26.325121 7fb446ab58c0 0
filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal
mode: checkpoint is not enabled
2017-11-17 01:27:26.343360 7fb446ab58c0 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 01:27:26.393876 7fb446ab58c0 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 01:27:26.394746 7fb446ab58c0 -1 osd.2 0 The disk uses
features unsupported by the executable.
2017-11-17 01:27:26.394758 7fb446ab58c0 -1 osd.2 0 ondisk features
compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,12=transaction
hints,13=pg meta object,14=explicit missing set,15=fastinfo pg
attr,16=deletes in missing set}
2017-11-17 01:27:26.394780 7fb446ab58c0 -1 osd.2 0 daemon features
compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded
objects,12=transaction hints,13=pg meta object}
2017-11-17 01:27:26.394794 7fb446ab58c0 -1 osd.2 0 Cannot write to disk!
Missing features: compat={},rocompat={},incompat={14=explicit missing
set,15=fastinfo pg attr,16=deletes in missing set}
2017-11-17 01:27:26.419854 7fb446ab58c0 1 journal close
/var/lib/ceph/osd/ceph-2/journal
2017-11-17 01:27:26.422687 7fb446ab58c0 -1 ESC[0;31m ** ERROR: osd init
failed: (95) Operation not supportedESC[0m
2017-11-17 01:27:26.863514 7fcc5f1428c0 0 ceph version 0.94.10
(b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 19731
2017-11-17 01:27:26.878617 7fcc5f1428c0 0
filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2017-11-17 01:27:26.880689 7fcc5f1428c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
FIEMAP ioctl is supported and appears to work
2017-11-17 01:27:26.880703 7fcc5f1428c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-11-17 01:27:26.898681 7fcc5f1428c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2017-11-17 01:27:26.898829 7fcc5f1428c0 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
disabled by conf
2017-11-17 01:27:26.906300 7fcc5f1428c0 0
filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal
mode: checkpoint is not enabled
2017-11-17 01:27:26.917013 7fcc5f1428c0 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 01:27:26.925628 7fcc5f1428c0 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 20: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 01:27:26.926496 7fcc5f1428c0 -1 osd.2 0 The disk uses
features unsupported by the executable.
2017-11-17 01:27:26.926509 7fcc5f1428c0 -1 osd.2 0 ondisk features
compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,12=transaction
hints,13=pg meta object,14=explicit missing set,15=fastinfo pg
attr,16=deletes in missing set}
2017-11-17 01:27:26.926533 7fcc5f1428c0 -1 osd.2 0 daemon features
compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded
objects,12=transaction hints,13=pg meta object}
2017-11-17 01:27:26.926553 7fcc5f1428c0 -1 osd.2 0 Cannot write to disk!
Missing features: compat={},rocompat={},incompat={14=explicit missing
set,15=fastinfo pg attr,16=deletes in missing set}
2017-11-17 01:27:26.927159 7fcc5f1428c0 1 journal close
/var/lib/ceph/osd/ceph-2/journal
2017-11-17 01:27:26.929073 7fcc5f1428c0 -1 ESC[0;31m ** ERROR: osd init
failed: (95) Operation not supportedESC[0m
2017-11-17 01:27:27.364931 7f16ccdc78c0 0 ceph version 0.94.10
(b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 19821
2017-11-17 01:27:27.379962 7f16ccdc78c0 0
filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2017-11-17 01:27:27.381509 7f16ccdc78c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
FIEMAP ioctl is supported and appears to work
2017-11-17 01:27:27.381524 7f16ccdc78c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-11-17 01:27:27.397192 7f16ccdc78c0 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2017-11-17 01:27:27.397324 7f16ccdc78c0 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
disabled by conf
2017-11-17 01:27:27.402018 7f16ccdc78c0 0
filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal
mode: checkpoint is not enabled
2017-11-17 01:27:27.412815 7f16ccdc78c0 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 19: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 01:27:27.421621 7f16ccdc78c0 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 19: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 01:27:27.422471 7f16ccdc78c0 -1 osd.2 0 The disk uses
features unsupported by the executable.
2017-11-17 01:27:27.422482 7f16ccdc78c0 -1 osd.2 0 ondisk features
compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,12=transaction
hints,13=pg meta object,14=explicit missing set,15=fastinfo pg
attr,16=deletes in missing set}
2017-11-17 01:27:27.422495 7f16ccdc78c0 -1 osd.2 0 daemon features
compat={},rocompat={},incompat={1=initial feature set(~v.18),2=pginfo
object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded
objects,12=transaction hints,13=pg meta object}
2017-11-17 01:27:27.422515 7f16ccdc78c0 -1 osd.2 0 Cannot write to disk!
Missing features: compat={},rocompat={},incompat={14=explicit missing
set,15=fastinfo pg attr,16=deletes in missing set}
2017-11-17 01:27:27.424247 7f16ccdc78c0 1 journal close
/var/lib/ceph/osd/ceph-2/journal
2017-11-17 01:27:27.426533 7f16ccdc78c0 -1 ESC[0;31m ** ERROR: osd init
failed: (95) Operation not supportedESC[0m
As the cluster won't start on Jewel, i can't give a ceph osd set
sortbitwise to complete a successful upgrade to Luminous. So trying to
start on Luminous i get the following errors from OSD:
2017-11-17 04:57:37.779635 7f04c28ead00 0 set uid:gid to 64045:64045
(ceph:ceph)
2017-11-17 04:57:37.779682 7f04c28ead00 0 ceph version 12.2.1
(3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process
(unknown), pid 8531
2017-11-17 04:57:37.783538 7f04c28ead00 -1 Public network was set, but
cluster network was not set
2017-11-17 04:57:37.783562 7f04c28ead00 -1 Using public network also
for cluster network
2017-11-17 04:57:37.790066 7f04c28ead00 0 pidfile_write: ignore empty
--pid-file
2017-11-17 04:57:37.800558 7f04c28ead00 0 load: jerasure load: lrc
load: isa
2017-11-17 04:57:37.801160 7f04c28ead00 0
filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2017-11-17 04:57:37.802397 7f04c28ead00 0
filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2017-11-17 04:57:37.802951 7f04c28ead00 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-11-17 04:57:37.802965 7f04c28ead00 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2017-11-17 04:57:37.802970 7f04c28ead00 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
splice() is disabled via 'filestore splice' config option
2017-11-17 04:57:37.817358 7f04c28ead00 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2017-11-17 04:57:37.817510 7f04c28ead00 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is
disabled by conf
2017-11-17 04:57:37.818348 7f04c28ead00 0
filestore(/var/lib/ceph/osd/ceph-2) start omap initiation
2017-11-17 04:57:37.818791 7f04c28ead00 1 leveldb: Recovering log #275311
2017-11-17 04:57:37.820333 7f04c28ead00 1 leveldb: Delete type=0 #275311
2017-11-17 04:57:37.820397 7f04c28ead00 1 leveldb: Delete type=3 #275310
2017-11-17 04:57:38.555491 7f04c28ead00 0
filestore(/var/lib/ceph/osd/ceph-2) mount(1758): enabling WRITEAHEAD
journal mode: checkpoint is not enabled
2017-11-17 04:57:38.559310 7f04c28ead00 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 28: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 04:57:38.561591 7f04c28ead00 1 journal _open
/var/lib/ceph/osd/ceph-2/journal fd 28: 5368709120 bytes, block size
4096 bytes, directio = 1, aio = 1
2017-11-17 04:57:38.563244 7f04c28ead00 1
filestore(/var/lib/ceph/osd/ceph-2) upgrade(1365)
2017-11-17 04:57:38.564625 7f04c28ead00 0 <cls>
/build/ceph-12.2.1/src/cls/hello/cls_hello.cc:296: loading cls_hello
2017-11-17 04:57:38.564658 7f04c28ead00 0 _get_class not permitted to
load lua
2017-11-17 04:57:38.567611 7f04c28ead00 0 <cls>
/build/ceph-12.2.1/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs
2017-11-17 04:57:38.568554 7f04c28ead00 0 _get_class not permitted to
load sdk
2017-11-17 04:57:38.568774 7f04c28ead00 0 _get_class not permitted to
load kvs
2017-11-17 04:57:38.568793 7f04c28ead00 1 osd.2 0 warning: got an error
loading one or more classes: (1) Operation not permitted
2017-11-17 04:57:38.569061 7f04c28ead00 0 osd.2 1531 crush map has
features 1107558400, adjusting msgr requires for clients
2017-11-17 04:57:38.569077 7f04c28ead00 0 osd.2 1531 crush map has
features 1107558400 was 8705, adjusting msgr requires for mons
2017-11-17 04:57:38.569085 7f04c28ead00 0 osd.2 1531 crush map has
features 1107558400, adjusting msgr requires for osds
2017-11-17 04:57:38.735392 7f04c28ead00 0 osd.2 1531 load_pgs
2017-11-17 04:57:38.740951 7f04c28ead00 -1 *** Caught signal (Aborted) **
in thread 7f04c28ead00 thread_name:ceph-osd
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e)
luminous (stable)
1: (()+0xa088b9) [0x5618d594b8b9]
2: (()+0x10330) [0x7f04c0e0d330]
3: (gsignal()+0x37) [0x7f04bfe2dc37]
4: (abort()+0x148) [0x7f04bfe31028]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f04c073c535]
6: (()+0x5e6d6) [0x7f04c073a6d6]
7: (()+0x5e703) [0x7f04c073a703]
8: (()+0x5e922) [0x7f04c073a922]
9: (object_stat_sum_t::decode(ceph::buffer::list::iterator&)+0x5cc)
[0x5618d563ec1c]
10:
(object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x54)
[0x5618d5659e14]
11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x1d5)
[0x5618d565a455]
12: (pg_info_t::decode(ceph::buffer::list::iterator&)+0x12a)
[0x5618d566059a]
13: (PG::read_info(ObjectStore*, spg_t, coll_t const&,
ceph::buffer::list&, pg_info_t&, PastIntervals&, unsigned char&)+0x231)
[0x5618d54d1231]
14: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x7b)
[0x5618d54d8c6b]
15: (OSD::load_pgs()+0x994) [0x5618d5435e74]
16: (OSD::init()+0x2127) [0x5618d544ddc7]
17: (main()+0x2ba8) [0x5618d5355798]
18: (__libc_start_main()+0xf5) [0x7f04bfe18f45]
19: (()+0x4b0826) [0x5618d53f3826]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
Thank you for any help, the storage cluster is hosting crucial data.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
