thanks, Eugen. I am suspecting this perhaps could be related to https://tracker.ceph.com/issues/42223 huxiaoyu@xxxxxxxxxxxx From: Eugen Block Date: 2022-03-05 09:52 To: huxiaoyu CC: ceph-users Subject: Re: Failed in ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring Hi, there must be some mixup of the OSD IDs. Your command seems to use ID 3 but the log complains about ID 1. You should check your script and workflow. Zitat von huxiaoyu@xxxxxxxxxxxx: > Dear Ceph folks, > > I encountered a strange behavior with Luminous 12.2.13, when running > the following > ceph-osd -i ${osd_id} --mkfs -k > /var/lib/ceph/osd/ceph-${osd_id}/keyring > to create OSD, > > what could be the root cause? Any ideas or suggestions? > > best regards, > > Samuel > > > *******************************************************Error log > messages as > below**************************************************************** > > + ceph-osd -i 3 --mkkey > + ceph-osd -i 3 --mkfs -k /var/lib/ceph/osd/ceph-3/keyring > --osd-uuid b9e67369-1d36-4d1d-9f81-1aa679d4c5bf > /var/lib/ceph-bin/cephosd.sh: line 451: 33918 Aborted > (core dumped) ceph-osd -i "${OSD_ID}" --mkfs -k > "${OSD_DIR}"/keyring --osd-uuid "${OSD_UUID}" > ${log_file} 2>&1 > + '[' 134 -eq 0 ']' > + echo osd mkfs fail > osd mkfs fail > > ------------------------------------------------ ${log_file}其中挂起的相关信息： > 2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the following > dangerous and experimental features are enabled: bluestore,rocksdb > 2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the following > dangerous and experimental features are enabled: bluestore,rocksdb > 2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the following > dangerous and experimental features are enabled: bluestore,rocksdb > 2022-03-05 05:53:40.452372 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory > 2022-03-05 05:53:40.452390 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory > 2022-03-05 05:53:40.452395 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory > 2022-03-05 05:53:40.460573 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In > function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, > uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned > int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908 > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: > FAILED assert(available >= allocated) > ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) > luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x102) [0x559be313d662] > 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned > long)+0x685) [0x559be30ec865] > 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1] > 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c] > 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215] > 6: (OSD::mkfs(CephContext*, ObjectStore*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24] > 7: (main()+0x11aa) [0x559be2a4424a] > 8: (__libc_start_main()+0xf0) [0x7f73656cb840] > 9: (_start()+0x29) [0x559be2ad3339] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > 2022-03-05 05:53:40.489602 7f7368465f80 -1 > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In > function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, > uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned > int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908 > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: > FAILED assert(available >= allocated) > > ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) > luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x102) [0x559be313d662] > 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned > long)+0x685) [0x559be30ec865] > 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1] > 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c] > 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215] > 6: (OSD::mkfs(CephContext*, ObjectStore*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24] > 7: (main()+0x11aa) [0x559be2a4424a] > 8: (__libc_start_main()+0xf0) [0x7f73656cb840] > 9: (_start()+0x29) [0x559be2ad3339] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > -17> 2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the > following dangerous and experimental features are enabled: > bluestore,rocksdb > -16> 2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the > following dangerous and experimental features are enabled: > bluestore,rocksdb > -14> 2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the > following dangerous and experimental features are enabled: > bluestore,rocksdb > -13> 2022-03-05 05:53:40.452372 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory > -12> 2022-03-05 05:53:40.452390 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory > -11> 2022-03-05 05:53:40.452395 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory > -10> 2022-03-05 05:53:40.460573 7f7368465f80 -1 > bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid > 0> 2022-03-05 05:53:40.489602 7f7368465f80 -1 > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In > function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, > uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned > int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908 > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: > FAILED assert(available >= allocated) > > ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) > luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x102) [0x559be313d662] > 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned > long)+0x685) [0x559be30ec865] > 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1] > 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c] > 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215] > 6: (OSD::mkfs(CephContext*, ObjectStore*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24] > 7: (main()+0x11aa) [0x559be2a4424a] > 8: (__libc_start_main()+0xf0) [0x7f73656cb840] > 9: (_start()+0x29) [0x559be2ad3339] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > *** Caught signal (Aborted) ** > in thread 7f7368465f80 thread_name:ceph-osd > ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) > luminous (stable) > 1: (()+0xab3b64) [0x559be30f9b64] > 2: (()+0x11390) [0x7f7366745390] > 3: (gsignal()+0x38) [0x7f73656e0438] > 4: (abort()+0x16a) [0x7f73656e203a] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x28e) [0x559be313d7ee] > 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned > long)+0x685) [0x559be30ec865] > 7: (BlueFS::mount()+0x3d1) [0x559be30b4bc1] > 8: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c] > 9: (BlueStore::mkfs()+0x1225) [0x559be2ff7215] > 10: (OSD::mkfs(CephContext*, ObjectStore*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24] > 11: (main()+0x11aa) [0x559be2a4424a] > 12: (__libc_start_main()+0xf0) [0x7f73656cb840] > 13: (_start()+0x29) [0x559be2ad3339] > 2022-03-05 05:53:40.492413 7f7368465f80 -1 *** Caught signal (Aborted) ** > in thread 7f7368465f80 thread_name:ceph-osd > > > > huxiaoyu@xxxxxxxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |