Failed in ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear Ceph folks,

I encountered a strange behavior with Luminous 12.2.13, when running the following 
        ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring 
to create OSD, 

what could be the root cause? Any ideas or suggestions?

best regards,

Samuel


*******************************************************Error log messages as below****************************************************************

+ ceph-osd -i 3 --mkkey
+ ceph-osd -i 3 --mkfs -k /var/lib/ceph/osd/ceph-3/keyring --osd-uuid b9e67369-1d36-4d1d-9f81-1aa679d4c5bf
/var/lib/ceph-bin/cephosd.sh: line 451: 33918 Aborted                 (core dumped) ceph-osd -i "${OSD_ID}" --mkfs -k "${OSD_DIR}"/keyring --osd-uuid "${OSD_UUID}" > ${log_file} 2>&1
+ '[' 134 -eq 0 ']'
+ echo osd mkfs fail
osd mkfs fail

------------------------------------------------   ${log_file}其中挂起的相关信息:
2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.452372 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.452390 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.452395 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.460573 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)
 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x559be313d662]
 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
 6: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
 7: (main()+0x11aa) [0x559be2a4424a]
 8: (__libc_start_main()+0xf0) [0x7f73656cb840]
 9: (_start()+0x29) [0x559be2ad3339]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2022-03-05 05:53:40.489602 7f7368465f80 -1 /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)

 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x559be313d662]
 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
 6: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
 7: (main()+0x11aa) [0x559be2a4424a]
 8: (__libc_start_main()+0xf0) [0x7f73656cb840]
 9: (_start()+0x29) [0x559be2ad3339]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

   -17> 2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -16> 2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -14> 2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -13> 2022-03-05 05:53:40.452372 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
   -12> 2022-03-05 05:53:40.452390 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
   -11> 2022-03-05 05:53:40.452395 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
   -10> 2022-03-05 05:53:40.460573 7f7368465f80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid
     0> 2022-03-05 05:53:40.489602 7f7368465f80 -1 /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)

 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x559be313d662]
 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
 6: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
 7: (main()+0x11aa) [0x559be2a4424a]
 8: (__libc_start_main()+0xf0) [0x7f73656cb840]
 9: (_start()+0x29) [0x559be2ad3339]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

*** Caught signal (Aborted) **
 in thread 7f7368465f80 thread_name:ceph-osd
 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
 1: (()+0xab3b64) [0x559be30f9b64]
 2: (()+0x11390) [0x7f7366745390]
 3: (gsignal()+0x38) [0x7f73656e0438]
 4: (abort()+0x16a) [0x7f73656e203a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x559be313d7ee]
 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x685) [0x559be30ec865]
 7: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
 8: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
 9: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
 10: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
 11: (main()+0x11aa) [0x559be2a4424a]
 12: (__libc_start_main()+0xf0) [0x7f73656cb840]
 13: (_start()+0x29) [0x559be2ad3339]
2022-03-05 05:53:40.492413 7f7368465f80 -1 *** Caught signal (Aborted) **
 in thread 7f7368465f80 thread_name:ceph-osd



huxiaoyu@xxxxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux