Re: Failed in ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring

[Eugen Block <eblock@xxxxxx>]

I don't think that's an issue here unless you tried the same for osd.1  
but mixed up the error output from osd.3 and osd.1. But the error  
messages clearly state that there's no block device for osd.1 although  
you seem to try to create osd.3:

[...] bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label  
failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or  
directory

Can you clarify?

Zitat von huxiaoyu@xxxxxxxxxxxx:

> thanks, Eugen.
>
> I am suspecting this perhaps could be related to  
> https://tracker.ceph.com/issues/42223
>
>
>
>
> huxiaoyu@xxxxxxxxxxxx
>
> From: Eugen Block
> Date: 2022-03-05 09:52
> To: huxiaoyu
> CC: ceph-users
> Subject: Re:  Failed in ceph-osd -i ${osd_id} --mkfs -k  
> /var/lib/ceph/osd/ceph-${osd_id}/keyring
> Hi, there must be some mixup of the OSD IDs. Your command seems to use
> ID 3 but the log complains about ID 1. You should check your script
> and workflow.
>
> Zitat von huxiaoyu@xxxxxxxxxxxx:
>
> > Dear Ceph folks,
> >
> > I encountered a strange behavior with Luminous 12.2.13, when running
> > the following
> >         ceph-osd -i ${osd_id} --mkfs -k
> > /var/lib/ceph/osd/ceph-${osd_id}/keyring
> > to create OSD,
> >
> > what could be the root cause? Any ideas or suggestions?
> >
> > best regards,
> >
> > Samuel
> >
> >
> > *******************************************************Error log
> > messages as
> > below****************************************************************
> >
> > + ceph-osd -i 3 --mkkey
> > + ceph-osd -i 3 --mkfs -k /var/lib/ceph/osd/ceph-3/keyring
> > --osd-uuid b9e67369-1d36-4d1d-9f81-1aa679d4c5bf
> > /var/lib/ceph-bin/cephosd.sh: line 451: 33918 Aborted
> >  (core dumped) ceph-osd -i "${OSD_ID}" --mkfs -k
> > "${OSD_DIR}"/keyring --osd-uuid "${OSD_UUID}" > ${log_file} 2>&1
> > + '[' 134 -eq 0 ']'
> > + echo osd mkfs fail
> > osd mkfs fail
> >
> > ------------------------------------------------   ${log_file}其中挂起的相关信息：
> > 2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the following
> > dangerous and experimental features are enabled: bluestore,rocksdb
> > 2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the following
> > dangerous and experimental features are enabled: bluestore,rocksdb
> > 2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the following
> > dangerous and experimental features are enabled: bluestore,rocksdb
> > 2022-03-05 05:53:40.452372 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
> > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
> > 2022-03-05 05:53:40.452390 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
> > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
> > 2022-03-05 05:53:40.452395 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
> > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
> > 2022-03-05 05:53:40.460573 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid
> > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In
> > function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t,
> > uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned
> > int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
> > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757:
> > FAILED assert(available >= allocated)
> >  ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
> > luminous (stable)
> >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x102) [0x559be313d662]
> >  2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
> > long)+0x685) [0x559be30ec865]
> >  3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
> >  4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
> >  5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
> >  6: (OSD::mkfs(CephContext*, ObjectStore*,
> > std::__cxx11::basic_string<char, std::char_traits<char>,
> > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
> >  7: (main()+0x11aa) [0x559be2a4424a]
> >  8: (__libc_start_main()+0xf0) [0x7f73656cb840]
> >  9: (_start()+0x29) [0x559be2ad3339]
> >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > needed to interpret this.
> > 2022-03-05 05:53:40.489602 7f7368465f80 -1
> > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In
> > function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t,
> > uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned
> > int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
> > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757:
> > FAILED assert(available >= allocated)
> >
> >  ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
> > luminous (stable)
> >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x102) [0x559be313d662]
> >  2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
> > long)+0x685) [0x559be30ec865]
> >  3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
> >  4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
> >  5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
> >  6: (OSD::mkfs(CephContext*, ObjectStore*,
> > std::__cxx11::basic_string<char, std::char_traits<char>,
> > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
> >  7: (main()+0x11aa) [0x559be2a4424a]
> >  8: (__libc_start_main()+0xf0) [0x7f73656cb840]
> >  9: (_start()+0x29) [0x559be2ad3339]
> >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > needed to interpret this.
> >
> >    -17> 2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the
> > following dangerous and experimental features are enabled:
> > bluestore,rocksdb
> >    -16> 2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the
> > following dangerous and experimental features are enabled:
> > bluestore,rocksdb
> >    -14> 2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the
> > following dangerous and experimental features are enabled:
> > bluestore,rocksdb
> >    -13> 2022-03-05 05:53:40.452372 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
> > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
> >    -12> 2022-03-05 05:53:40.452390 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
> > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
> >    -11> 2022-03-05 05:53:40.452395 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
> > open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
> >    -10> 2022-03-05 05:53:40.460573 7f7368465f80 -1
> > bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid
> >      0> 2022-03-05 05:53:40.489602 7f7368465f80 -1
> > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In
> > function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t,
> > uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned
> > int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
> > /build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757:
> > FAILED assert(available >= allocated)
> >
> >  ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
> > luminous (stable)
> >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x102) [0x559be313d662]
> >  2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
> > long)+0x685) [0x559be30ec865]
> >  3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
> >  4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
> >  5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
> >  6: (OSD::mkfs(CephContext*, ObjectStore*,
> > std::__cxx11::basic_string<char, std::char_traits<char>,
> > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
> >  7: (main()+0x11aa) [0x559be2a4424a]
> >  8: (__libc_start_main()+0xf0) [0x7f73656cb840]
> >  9: (_start()+0x29) [0x559be2ad3339]
> >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > needed to interpret this.
> >
> > *** Caught signal (Aborted) **
> >  in thread 7f7368465f80 thread_name:ceph-osd
> >  ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
> > luminous (stable)
> >  1: (()+0xab3b64) [0x559be30f9b64]
> >  2: (()+0x11390) [0x7f7366745390]
> >  3: (gsignal()+0x38) [0x7f73656e0438]
> >  4: (abort()+0x16a) [0x7f73656e203a]
> >  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x28e) [0x559be313d7ee]
> >  6: (BitmapAllocator::init_rm_free(unsigned long, unsigned
> > long)+0x685) [0x559be30ec865]
> >  7: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
> >  8: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
> >  9: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
> >  10: (OSD::mkfs(CephContext*, ObjectStore*,
> > std::__cxx11::basic_string<char, std::char_traits<char>,
> > std::allocator<char> > const&, uuid_d, int)+0x164) [0x559be2b1da24]
> >  11: (main()+0x11aa) [0x559be2a4424a]
> >  12: (__libc_start_main()+0xf0) [0x7f73656cb840]
> >  13: (_start()+0x29) [0x559be2ad3339]
> > 2022-03-05 05:53:40.492413 7f7368465f80 -1 *** Caught signal (Aborted) **
> >  in thread 7f7368465f80 thread_name:ceph-osd
> >
> >
> >
> > huxiaoyu@xxxxxxxxxxxx
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx