Hi folks,
I am trying to install create a bluestore osd manually with ceph-volume tool on a Ubuntu 14.04 system, but with no luck. The Ceph version I used is Luminous 12.2.2.
I do this manually instead of ceph-deploy command, because I want to seperate data to HDD, then WAL and DB to SSD. This is the command I ran:
$ sudo ceph-volume lvm prepare --bluestore --data /dev/sdy --block.wal /dev/nvme0n1p1 --block.db /dev/nvme0n1p1
I noticed several Linux commands' parameter or output format used by ceph-volume are not compatible with Ubuntu 14.04:
lvscreate command doesn't support "--yes" parameter -- I remove "--yes" parameter
blkid command doesn't show PARTUUID -- I trucate sgdisk command's output as the result
I modified ceph-volume code to bypass these problem, but it still failed. When ceph-volume tried to run this command, ceph raised a core dump:
sudo ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 96 --monmap /var/lib/ceph/osd/ceph-96/activate.monmap --key **************************************** --bluestore-block-wal-path /dev/nvme0n1p1 --bluestore-block-db-path /dev/nvme0n1p1 --osd-data /var/lib/ceph/osd/ceph-96/ --osd-uuid 0e30f7f8-3fef-4a91-b748-443b55fbb1d6 --setuser ceph --setgroup ceph
The --key parameter seems not normal to me, but, as an outsider, I am not sure which part goes wrong.
Hope you can help to find some clue.
The ceph-volume.log output is quite long. I put it in the attachment. You check it for detail and I put the error information here:
[2017-12-13 20:49:50,735][ceph_volume.process][INFO ] Running command: sudo ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 96 --monmap /var/lib/ceph/osd/ceph-96/activate.monmap --key **************************************** --bluestore-block-wal-path /dev/nvme0n1p1 --bluestore-block-db-path /dev/nvme0n1p1 --osd-data /var/lib/ceph/osd/ceph-96/ --osd-uuid 0e30f7f8-3fef-4a91-b748-443b55fbb1d6 --setuser ceph --setgroup ceph
[2017-12-13 20:49:50,763][ceph_volume.process][INFO ] stderr 2017-12-13 20:49:50.763027 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
[2017-12-13 20:49:50,763][ceph_volume.process][INFO ] stderr 2017-12-13 20:49:50.763233 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
[2017-12-13 20:49:50,763][ceph_volume.process][INFO ] stderr 2017-12-13 20:49:50.763393 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
2017-12-13 20:49:50.763452 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96/) _read_fsid unparsable uuid
[2017-12-13 20:49:51,784][ceph_volume.process][INFO ] stderr /build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f2789e87d00 time 2017-12-13 20:49:51.784626
/build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: 300: FAILED assert(rm.empty())
[2017-12-13 20:49:51,787][ceph_volume.process][INFO ] stderr ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x55cb5ba7d82e]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x291a) [0x55cb5ba2e66a]
3: (BlueFS::mount()+0x275) [0x55cb5ba0f9c5]
4: (BlueStore::_open_db(bool)+0x1680) [0x55cb5b92d2a0]
5: (BlueStore::_fsck(bool, bool)+0x3ca) [0x55cb5b95ebba]
6: (BlueStore::mkfs()+0xf20) [0x55cb5b936480]
7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x2a5) [0x55cb5b5299a5]
8: (main()+0x114e) [0x55cb5b441b1e]
9: (__libc_start_main()+0xf5) [0x7f27873b1f45]
10: (()+0x4b36d6) [0x55cb5b4e16d6]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-12-13 20:49:51.787609 7f2789e87d00 -1 /build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f2789e87d00 time 2017-12-13 20:49:51.784626
/build
[2017-12-13 20:49:51,787][ceph_volume.process][INFO ] stderr /ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: 300: FAILED assert(rm.empty())
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x55cb5ba7d82e]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x291a) [0x55cb5ba2e66a]
3: (BlueFS::mount()+0x275) [0x55cb5ba0f9c5]
4: (BlueStore::_open_db(bool)+0x1680) [0x55cb5b92d2a0]
5: (BlueStore::_fsck(bool, bool)+0x3ca) [0x55cb5b95ebba]
6: (BlueStore::mkfs()+0xf20) [0x55cb5b936480]
7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x2a5) [0x55cb5b5299a5]
8: (main()+0x114e) [0x55cb5b441b1e]
9: (__libc_start_main()+0xf5) [0x7f27873b1f45]
10: (()+0x4b36d6) [0x55cb5b4e16d6]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-214> 2017-12-13 20:49:50.763027 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: b
[2017-12-13 20:49:51,788][ceph_volume.process][INFO ] stderr uffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
-213> 2017-12-13 20:49:50.763233 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
-212> 2017-12-13 20:49:50.763393 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
-211> 2017-12-13 20:49:50.763452 7f2789e87d00 -1 bluestore(/var/lib/ceph/osd/ceph-96/) _read_fsid unparsable uuid
[2017-12-13 20:49:51,788][ceph_volume.process][INFO ] stderr 0> 2017-12-13 20:49:51.787609 7f2789e87d00 -1 /build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f2789e87d00 time 2017-12-13 20:49:51.784626
/build/ceph-12.2.2/src/os/bluestore/StupidAllocator.cc: 300: FAILED assert(rm.empty())
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x55cb5ba7d82e]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x291a) [0x55cb5ba2e66a]
3: (BlueFS::mount()+0x275) [0x55cb5ba0f9c5]
4: (BlueStore::_open_db(bool)+0x1680) [0x55cb5b92d2a0]
5: (BlueStore::_fsck(bool, bool)+0x3ca) [0x55cb5b95ebba]
6: (BlueStore::mkfs()+0xf20) [0x55cb5b936480]
7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x2a5) [0x55cb5b5299a5]
8: (main()+0x114e) [0x55cb5b441b1e]
9: (__libc_start_main()+0xf5) [0x7f27873b1f45]
10: (()+0x4b36d6) [0x55cb5b4e16d6]
NO
[2017-12-13 20:49:51,788][ceph_volume.process][INFO ] stderr TE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7f2789e87d00 thread_name:ceph-osd
[2017-12-13 20:49:51,791][ceph_volume.process][INFO ] stderr ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (()+0xa12e99) [0x55cb5ba40e99]
2: (()+0x10330) [0x7f27883a6330]
3: (gsignal()+0x37) [0x7f27873c6c37]
4: (abort()+0x148) [0x7f27873ca028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x55cb5ba7d9a0]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x291a) [0x55cb5ba2e66a]
7: (BlueFS::mount()+0x275) [0x55cb5ba0f9c5]
8: (BlueStore::_open_db(bool)+0x1680) [0x55cb5b92d2a0]
9: (BlueStore::_fsck(bool, bool)+0x3ca) [0x55cb5b95ebba]
10: (BlueStore::mkfs()+0xf20) [0x55cb5b936480]
11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x2a5) [0x55cb5b5299a5]
12: (main()+0x114e) [0x55cb5b441b1e]
13: (__libc_start_main()+0xf5) [0x7f27873b1f45]
14: (()+0x4b36d6) [0x55cb5b4e16d6]
2017-12-13 20:49:51.791532 7f2789e87d00 -1 *** Caught signal (Aborted) **
in thread 7f2789e87d00 thread_name:ceph-osd
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e
[2017-12-13 20:49:51,791][ceph_volume.process][INFO ] stderr 5ba) luminous (stable)
1: (()+0xa12e99) [0x55cb5ba40e99]
2: (()+0x10330) [0x7f27883a6330]
3: (gsignal()+0x37) [0x7f27873c6c37]
4: (abort()+0x148) [0x7f27873ca028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x55cb5ba7d9a0]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x291a) [0x55cb5ba2e66a]
7: (BlueFS::mount()+0x275) [0x55cb5ba0f9c5]
8: (BlueStore::_open_db(bool)+0x1680) [0x55cb5b92d2a0]
9: (BlueStore::_fsck(bool, bool)+0x3ca) [0x55cb5b95ebba]
10: (BlueStore::mkfs()+0xf20) [0x55cb5b936480]
11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x2a5) [0x55cb5b5299a5]
12: (main()+0x114e) [0x55cb5b441b1e]
13: (__libc_start_main()+0xf5) [0x7f27873b1f45]
14: (()+0x4b36d6) [0x55cb5b4e16d6]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
0> 2017-12-13 20:49:51.791532 7f2789e87d00 -1 *** Caught signal (Aborted) **
in thread 7f2789e87d00 thread_name:ceph-osd
ceph version 12
[2017-12-13 20:49:51,792][ceph_volume.process][INFO ] stderr .2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (()+0xa12e99) [0x55cb5ba40e99]
2: (()+0x10330) [0x7f27883a6330]
3: (gsignal()+0x37) [0x7f27873c6c37]
4: (abort()+0x148) [0x7f27873ca028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x55cb5ba7d9a0]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x291a) [0x55cb5ba2e66a]
7: (BlueFS::mount()+0x275) [0x55cb5ba0f9c5]
8: (BlueStore::_open_db(bool)+0x1680) [0x55cb5b92d2a0]
9: (BlueStore::_fsck(bool, bool)+0x3ca) [0x55cb5b95ebba]
10: (BlueStore::mkfs()+0xf20) [0x55cb5b936480]
11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x2a5) [0x55cb5b5299a5]
12: (main()+0x114e) [0x55cb5b441b1e]
13: (__libc_start_main()+0xf5) [0x7f27873b1f45]
14: (()+0x4b36d6) [0x55cb5b4e16d6]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
[2017-12-13 20:49:51,906][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python2.7/dist-packages/ceph_volume/main.py", line 144, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line 131, in dispatch
instance.main()
File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/main.py", line 38, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line 131, in dispatch
instance.main()
File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", line 293, in main
self.prepare(args)
File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", line 231, in prepare
fsid=osd_fsid,
File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", line 72, in prepare_bluestore
db=db
File "/usr/lib/python2.7/dist-packages/ceph_volume/util/prepare.py", line 225, in osd_mkfs_bluestore
process.run(command, obfuscate='--key')
File "/usr/lib/python2.7/dist-packages/ceph_volume/process.py", line 135, in run
raise RuntimeError(msg)
RuntimeError: command returned non-zero exit status: 134
Bruce J.
Attachment:
ceph-volume.log
Description: Binary data
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
