Just to report back the root cause of the above mentioned failures in " ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring"
It turns out the culprit was using Samsung SM883 SSD disks as DB/WAL partitions. Replacing SM883 with Intel S4510/4520 SSDs solved the issues.
It looks highly likely that the firmware SM883 has some compatibility issues.
best wishes,
Samuel
huxiaoyu@xxxxxxxxxxxx
发件人： huxiaoyu@xxxxxxxxxxxx
发送时间： 2022-03-05 09:07
收件人： ceph-users
主题： Failed in ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring
Dear Ceph folks,
I encountered a strange behavior with Luminous 12.2.13, when running the following
ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring
to create OSD,
what could be the root cause? Any ideas or suggestions?
best regards,
Samuel
*******************************************************Error log messages as below****************************************************************
+ ceph-osd -i 3 --mkkey
+ ceph-osd -i 3 --mkfs -k /var/lib/ceph/osd/ceph-3/keyring --osd-uuid b9e67369-1d36-4d1d-9f81-1aa679d4c5bf
/var/lib/ceph-bin/cephosd.sh: line 451: 33918 Aborted (core dumped) ceph-osd -i "${OSD_ID}" --mkfs -k "${OSD_DIR}"/keyring --osd-uuid "${OSD_UUID}" > ${log_file} 2>&1
+ '[' 134 -eq 0 ']'
+ echo osd mkfs fail
osd mkfs fail
------------------------------------------------ ${log_file}其中挂起的相关信息：
2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.452372 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.452390 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.452395 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.460573 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)
ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x559be313d662]
2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
6: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
7: (main()+0x11aa) [0x559be2a4424a]
8: (__libc_start_main()+0xf0) [0x7f73656cb840]
9: (_start()+0x29) [0x559be2ad3339]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2022-03-05 05:53:40.489602 7f7368465f80 -1 /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)
ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x559be313d662]
2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
6: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
7: (main()+0x11aa) [0x559be2a4424a]
8: (__libc_start_main()+0xf0) [0x7f73656cb840]
9: (_start()+0x29) [0x559be2ad3339]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-17> 2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-16> 2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-14> 2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-13> 2022-03-05 05:53:40.452372 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
-12> 2022-03-05 05:53:40.452390 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
-11> 2022-03-05 05:53:40.452395 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
-10> 2022-03-05 05:53:40.460573 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid
0> 2022-03-05 05:53:40.489602 7f7368465f80 -1 /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)
ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x559be313d662]
2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
6: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
7: (main()+0x11aa) [0x559be2a4424a]
8: (__libc_start_main()+0xf0) [0x7f73656cb840]
9: (_start()+0x29) [0x559be2ad3339]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7f7368465f80 thread_name:ceph-osd
ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
1: (()+0xab3b64) [0x559be30f9b64]
2: (()+0x11390) [0x7f7366745390]
3: (gsignal()+0x38) [0x7f73656e0438]
4: (abort()+0x16a) [0x7f73656e203a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x559be313d7ee]
6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
7: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
8: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
9: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
10: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
11: (main()+0x11aa) [0x559be2a4424a]
12: (__libc_start_main()+0xf0) [0x7f73656cb840]
13: (_start()+0x29) [0x559be2ad3339]
2022-03-05 05:53:40.492413 7f7368465f80 -1 *** Caught signal (Aborted) **
in thread 7f7368465f80 thread_name:ceph-osd
huxiaoyu@xxxxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
