Hi Rainer, Try something like dd if=/dev/zero of=/dev/sdX bs=4096 To wipe/zap any Information on the disk. HTH Mehmet Am 14. Februar 2019 13:57:51 MEZ schrieb Rainer Krienke <krienke@xxxxxxxxxxxxxx>: >Hi, > >I am quite new to ceph and just try to set up a ceph cluster. Initially >I used ceph-deploy for this but when I tried to create a BlueStore osd >ceph-deploy fails. Next I tried the direct way on one of the OSD-nodes >using ceph-volume to create the osd, but this also fails. Below you can >see what ceph-volume says. > >I ensured that there was no left over lvm VG and LV on the disk sdg >before I started the osd creation for this disk. The very same error >happens also on other disks not just for /dev/sdg. All the disk have >4TB >in size and the linux system is Ubuntu 18.04 and finally ceph is >installed in version 13.2.4-1bionic from this repo: >https://download.ceph.com/debian-mimic. > >There is a VG and two LV's on the system for the ubuntu system itself >that is installed on two separate disks configured as software raid1 >and >lvm on top of the raid. But I cannot imagine that this might do any >harm >to cephs osd creation. > >Does anyone have an idea what might be wrong? > >Thanks for hints >Rainer > >root@ceph1:~# wipefs -fa /dev/sdg >root@ceph1:~# ceph-volume lvm prepare --bluestore --data /dev/sdg >Running command: /usr/bin/ceph-authtool --gen-print-key >Running command: /usr/bin/ceph --cluster ceph --name >client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring >-i - osd new 14d041d6-0beb-4056-8df2-3920e2febce0 >Running command: /sbin/vgcreate --force --yes >ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b /dev/sdg > stdout: Physical volume "/dev/sdg" successfully created. > stdout: Volume group "ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b" >successfully created >Running command: /sbin/lvcreate --yes -l 100%FREE -n >osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 >ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b >stdout: Logical volume "osd-block-14d041d6-0beb-4056-8df2-3920e2febce0" >created. >Running command: /usr/bin/ceph-authtool --gen-print-key >Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 >--> Absolute path not found for executable: restorecon >--> Ensure $PATH environment variable contains common executable >locations >Running command: /bin/chown -h ceph:ceph >/dev/ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b/osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 >Running command: /bin/chown -R ceph:ceph /dev/dm-8 >Running command: /bin/ln -s >/dev/ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b/osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 >/var/lib/ceph/osd/ceph-0/block >Running command: /usr/bin/ceph --cluster ceph --name >client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring >mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap > stderr: got monmap epoch 1 >Running command: /usr/bin/ceph-authtool >/var/lib/ceph/osd/ceph-0/keyring >--create-keyring --name osd.0 --add-key >AQAAY2VcU968HxAAvYWMaJZmriUc4H9bCCp8XQ== > stdout: creating /var/lib/ceph/osd/ceph-0/keyring >added entity osd.0 auth auth(auid = 18446744073709551615 >key=AQAAY2VcU968HxAAvYWMaJZmriUc4H9bCCp8XQ== with 0 caps) >Running command: /bin/chown -R ceph:ceph >/var/lib/ceph/osd/ceph-0/keyring >Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ >Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore >bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap >--keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid >14d041d6-0beb-4056-8df2-3920e2febce0 --setuser ceph --setgroup ceph > stderr: 2019-02-14 13:45:54.788 7f3fcecb3240 -1 >bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid > stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In >function 'virtual int KernelDevice::read(uint64_t, uint64_t, >ceph::bufferlist*, IOContext*, bool)' thread 7f3fcecb3240 time >2019-02-14 13:45:54.841130 > stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: >FAILED assert((uint64_t)r == len) > stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) >mimic (stable) > stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >char const*)+0x102) [0x7f3fc60d33e2] > stderr: 2: (()+0x26d5a7) [0x7f3fc60d35a7] > stderr: 3: (KernelDevice::read(unsigned long, unsigned long, >ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] > stderr: 4: (BlueFS::_read(BlueFS::FileReader*, >BlueFS::FileReaderBuffer*, unsigned long, unsigned long, >ceph::buffer::list*, char*)+0x435) [0x5613713065c5] > stderr: 5: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] > stderr: 6: (BlueFS::mount()+0x1f1) [0x561371310c81] > stderr: 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] > stderr: 8: (BlueStore::mkfs()+0x805) [0x561371267fe5] > stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, >std::__cxx11::basic_string<char, std::char_traits<char>, >std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] > stderr: 10: (main()+0x4222) [0x561370cf7462] > stderr: 11: (__libc_start_main()+0xe7) [0x7f3fc3695b97] > stderr: 12: (_start()+0x2a) [0x561370dc095a] > stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` >is needed to interpret this. > stderr: 2019-02-14 13:45:54.840 7f3fcecb3240 -1 >/build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In function >'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, >IOContext*, bool)' thread 7f3fcecb3240 time 2019-02-14 13:45:54.841130 > stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: >FAILED assert((uint64_t)r == len) > stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) >mimic (stable) > stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >char const*)+0x102) [0x7f3fc60d33e2] > stderr: 2: (()+0x26d5a7) [0x7f3fc60d35a7] > stderr: 3: (KernelDevice::read(unsigned long, unsigned long, >ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] > stderr: 4: (BlueFS::_read(BlueFS::FileReader*, >BlueFS::FileReaderBuffer*, unsigned long, unsigned long, >ceph::buffer::list*, char*)+0x435) [0x5613713065c5] > stderr: 5: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] > stderr: 6: (BlueFS::mount()+0x1f1) [0x561371310c81] > stderr: 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] > stderr: 8: (BlueStore::mkfs()+0x805) [0x561371267fe5] > stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, >std::__cxx11::basic_string<char, std::char_traits<char>, >std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] > stderr: 10: (main()+0x4222) [0x561370cf7462] > stderr: 11: (__libc_start_main()+0xe7) [0x7f3fc3695b97] > stderr: 12: (_start()+0x2a) [0x561370dc095a] > stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` >is needed to interpret this. > stderr: -15> 2019-02-14 13:45:54.788 7f3fcecb3240 -1 >bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid > stderr: 0> 2019-02-14 13:45:54.840 7f3fcecb3240 -1 >/build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In function >'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, >IOContext*, bool)' thread 7f3fcecb3240 time 2019-02-14 13:45:54.841130 > stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: >FAILED assert((uint64_t)r == len) > stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) >mimic (stable) > stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >char const*)+0x102) [0x7f3fc60d33e2] > stderr: 2: (()+0x26d5a7) [0x7f3fc60d35a7] > stderr: 3: (KernelDevice::read(unsigned long, unsigned long, >ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] > stderr: 4: (BlueFS::_read(BlueFS::FileReader*, >BlueFS::FileReaderBuffer*, unsigned long, unsigned long, >ceph::buffer::list*, char*)+0x435) [0x5613713065c5] > stderr: 5: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] > stderr: 6: (BlueFS::mount()+0x1f1) [0x561371310c81] > stderr: 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] > stderr: 8: (BlueStore::mkfs()+0x805) [0x561371267fe5] > stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, >std::__cxx11::basic_string<char, std::char_traits<char>, >std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] > stderr: 10: (main()+0x4222) [0x561370cf7462] > stderr: 11: (__libc_start_main()+0xe7) [0x7f3fc3695b97] > stderr: 12: (_start()+0x2a) [0x561370dc095a] > stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` >is needed to interpret this. > stderr: *** Caught signal (Aborted) ** > stderr: in thread 7f3fcecb3240 thread_name:ceph-osd > stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) >mimic (stable) > stderr: 1: (()+0x92aa40) [0x561371357a40] > stderr: 2: (()+0x12890) [0x7f3fc47d7890] > stderr: 3: (gsignal()+0xc7) [0x7f3fc36b2e97] > stderr: 4: (abort()+0x141) [0x7f3fc36b4801] > stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, >char const*)+0x250) [0x7f3fc60d3530] > stderr: 6: (()+0x26d5a7) [0x7f3fc60d35a7] > stderr: 7: (KernelDevice::read(unsigned long, unsigned long, >ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] > stderr: 8: (BlueFS::_read(BlueFS::FileReader*, >BlueFS::FileReaderBuffer*, unsigned long, unsigned long, >ceph::buffer::list*, char*)+0x435) [0x5613713065c5] > stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] > stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] > stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] > stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] > stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, >std::__cxx11::basic_string<char, std::char_traits<char>, >std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] > stderr: 14: (main()+0x4222) [0x561370cf7462] > stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] > stderr: 16: (_start()+0x2a) [0x561370dc095a] > stderr: 2019-02-14 13:45:54.840 7f3fcecb3240 -1 *** Caught signal >(Aborted) ** > stderr: in thread 7f3fcecb3240 thread_name:ceph-osd > stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) >mimic (stable) > stderr: 1: (()+0x92aa40) [0x561371357a40] > stderr: 2: (()+0x12890) [0x7f3fc47d7890] > stderr: 3: (gsignal()+0xc7) [0x7f3fc36b2e97] > stderr: 4: (abort()+0x141) [0x7f3fc36b4801] > stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, >char const*)+0x250) [0x7f3fc60d3530] > stderr: 6: (()+0x26d5a7) [0x7f3fc60d35a7] > stderr: 7: (KernelDevice::read(unsigned long, unsigned long, >ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] > stderr: 8: (BlueFS::_read(BlueFS::FileReader*, >BlueFS::FileReaderBuffer*, unsigned long, unsigned long, >ceph::buffer::list*, char*)+0x435) [0x5613713065c5] > stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] > stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] > stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] > stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] > stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, >std::__cxx11::basic_string<char, std::char_traits<char>, >std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] > stderr: 14: (main()+0x4222) [0x561370cf7462] > stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] > stderr: 16: (_start()+0x2a) [0x561370dc095a] > stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` >is needed to interpret this. > stderr: 0> 2019-02-14 13:45:54.840 7f3fcecb3240 -1 *** Caught signal >(Aborted) ** > stderr: in thread 7f3fcecb3240 thread_name:ceph-osd > stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) >mimic (stable) > stderr: 1: (()+0x92aa40) [0x561371357a40] > stderr: 2: (()+0x12890) [0x7f3fc47d7890] > stderr: 3: (gsignal()+0xc7) [0x7f3fc36b2e97] > stderr: 4: (abort()+0x141) [0x7f3fc36b4801] > stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, >char const*)+0x250) [0x7f3fc60d3530] > stderr: 6: (()+0x26d5a7) [0x7f3fc60d35a7] > stderr: 7: (KernelDevice::read(unsigned long, unsigned long, >ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] > stderr: 8: (BlueFS::_read(BlueFS::FileReader*, >BlueFS::FileReaderBuffer*, unsigned long, unsigned long, >ceph::buffer::list*, char*)+0x435) [0x5613713065c5] > stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] > stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] > stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] > stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] > stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, >std::__cxx11::basic_string<char, std::char_traits<char>, >std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] > stderr: 14: (main()+0x4222) [0x561370cf7462] > stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] > stderr: 16: (_start()+0x2a) [0x561370dc095a] > stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` >is needed to interpret this. >--> Was unable to complete a new OSD, will rollback changes >Running command: /usr/bin/ceph --cluster ceph --name >client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring >osd purge-new osd.0 --yes-i-really-mean-it > stderr: purged osd.0 >--> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd >--cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap >/var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data >/var/lib/ceph/osd/ceph-0/ --osd-uuid >14d041d6-0beb-4056-8df2-3920e2febce0 --setuser ceph --setgroup ceph _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |