I am trying to stand up ceph (luminous) on 3 72 disk supermicro servers running ubuntu 16.04 with HWE enabled (for a 4.10 kernel for cephfs). I am not sure how this is possible but even though I am running the following line to wipe all disks of their partitions, once I run ceph-disk to partition the drive udev or device mapper automatically mounts a lockbox partition and ceph-disk fails::
wipe line::
for disk in $(lsblk --output MODEL,NAME | grep -iE "HGST|SSDSC2BA40" | awk '{print $NF}'); do sgdisk -Z /dev/${disk}; dd if=/dev/zero of=/dev/${disk} bs=1024 count=10000; ceph-disk zap /dev/${disk}; sgdisk -o /dev/${disk}; sgdisk -G /dev/${disk}; done
ceph-disk line:
prior to running that on a single disk all of the drives are empty except the OS drives
-----------------------------------------
but as soon as I run that cephcmd (which worked prior to upgrading to the 4.10 kernel:
----------------------------------------------------------------------------------------------------------------------------
ceph-disk line:
cephcmd="ceph-disk -v prepare --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --block.db /dev/${pssd} --block.wal /dev/${pssd} --bluestore --cluster ceph --fs-type xfs
-- /dev/${phdd}"
prior to running that on a single disk all of the drives are empty except the OS drives
root@kg15-1:/home/ceph-admin# lsblk --fs
NAME FSTYPE LABEL UUID MOUNTPOINT
sdbu
sdy
sdam
sdbb
sdf
sdau
sdab
sdbk
sdo
sdbs
sdw
sdak
sdd
sdas
sdbi
sdm
sdbq
sdu
sdai
sdb
sdaq
sdbg
sdk
sdaz
sds
sdag
sdbe
sdi
sdax
sdq
sdae
sdbn
sdbv
├─sdbv3 linux_raid_member kg15-1:2 664f69b7-2dd7-7012-75e3-a920ba7416b8
│ └─md2 ext4 6696d9f5-3385-47cb-8e8b-058637f8a1b8 /
├─sdbv1 linux_raid_member kg15-1:0 c4c78d8b-5c0b-6d51-d0a4-ecd40432f98c
│ └─md0 ext4 44f76d8d-0333-49a7-ab89-dafe70f6f12d /boot
└─sdbv2 linux_raid_member kg15-1:1 e3a74474-502c-098c-9415-7b99abcbd2e1
└─md1 swap 37e071a9-9361-456b-a740-87ddc99a8260 [SWAP]
sdz
sdan
sdbc
sdg
sdav
sdac
sdbl
sdbt
sdx
sdal
sdba
sde
sdat
sdaa
sdbj
sdn
sdbr
sdv
sdaj
sdc
sdar
sdbh
sdl
sdbp
sdt
sdah
sda
├─sda2 linux_raid_member kg15-1:1 e3a74474-502c-098c-9415-7b99abcbd2e1
│ └─md1 swap 37e071a9-9361-456b-a740-87ddc99a8260 [SWAP]
├─sda3 linux_raid_member kg15-1:2 664f69b7-2dd7-7012-75e3-a920ba7416b8
│ └─md2 ext4 6696d9f5-3385-47cb-8e8b-058637f8a1b8 /
└─sda1 linux_raid_member kg15-1:0 c4c78d8b-5c0b-6d51-d0a4-ecd40432f98c
└─md0 ext4 44f76d8d-0333-49a7-ab89-dafe70f6f12d /boot
sdap
sdbf
sdj
sday
sdr
sdaf
sdbo
sdao
sdbd
sdh
sdaw
sdp
sdad
sdbm
but as soon as I run that cephcmd (which worked prior to upgrading to the 4.10 kernel:
----------------------------------------------------------------------------------------------------------------------------
ceph-disk -v prepare --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --block.db /dev/sdd --block.wal /dev/sdd --bluestore --cluster ceph --fs-type xfs -- /dev/sdbu
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
set_type: Will colocate block with data on /dev/sdbu
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_size
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_db_size
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_size
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_wal_size
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
Traceback (most recent call last):
File "/usr/sbin/ceph-disk", line 9, in <module>
load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5704, in run
main(sys.argv[1:])
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5655, in main
args.func(args)
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2091, in main
Prepare.factory(args).prepare()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2080, in prepare
self._prepare()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2154, in _prepare
self.lockbox.prepare()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2842, in prepare
verify_not_in_use(self.args.lockbox, check_partitions=True)
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 950, in verify_not_in_use
raise Error('Device is mounted', partition)
ceph_disk.main.Error: Error: Device is mounted: /dev/sdbu5
----------------------------------------------------------------------------------------------------------------------------
So it says sdbu is mounted. I unmount it and again it errors saying it can't create the partition it just tried to create.
So it says sdbu is mounted. I unmount it and again it errors saying it can't create the partition it just tried to create.
root@kg15-1:/# mount | grep sdbu
/dev/sdbu5 on /var/lib/ceph/osd-lockbox/0e3baee9-a5dd-46f0-ae53-0e7dd2b0b257 type ext4 (rw,relatime,stripe=4,data="">
so I run:: /dev/sdbu5; sgdisk -o /dev/sdbu; sgdisk -Z /dev/sdbu; sgdisk -G /dev/sdbu
and try again:
--------------------------------------------------------------
now i'm back to where i was 5 seconds ago. I must be doing something wrong but I have no idea what it is.
so I run:: /dev/sdbu5; sgdisk -o /dev/sdbu; sgdisk -Z /dev/sdbu; sgdisk -G /dev/sdbu
and try again:
--------------------------------------------------------------
command_check_call: Running command: /sbin/sgdisk --new=5:0:+10M --change-name=5:ceph lockbox --partition-guid=5:None --typecode=5:fb3aabf9-d25f-47cc-bf5e-721d181642be --mbrtogpt -- /dev/sdbu
Setting name!
partNum is 4
REALLY setting name!
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdbu
command_check_call: Running command: /sbin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdbu /sbin/partprobe /dev/sdbu
command_check_call: Running command: /sbin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdbu5 uuid path is /sys/dev/block/68:133/dm/uuid
populate: Creating lockbox fs on %s: mkfs -t ext4 /dev/sdbu5
command_check_call: Running command: /sbin/mkfs -t ext4 /dev/sdbu5
mke2fs 1.42.13 (17-May-2015)
/dev/sdbu5 contains a ext4 file system
last mounted on /var/lib/ceph/osd-lockbox/0e3baee9-a5dd-46f0-ae53-0e7dd2b0b257 on Fri Oct 20 21:39:03 2017
Proceed anyway? (y,n) y
Creating filesystem with 10240 1k blocks and 2560 inodes
Filesystem UUID: 4f90660a-4ca8-4cd5-a42c-f947ef3c628f
Superblock backups stored on blocks:
8193
Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done
populate: Mounting lockbox mount -t ext4 /dev/sdbu5 /var/lib/ceph/osd-lockbox/8ef450fe-d60b-4199-a1aa-66c9b4681ab7
command_check_call: Running command: /bin/mount -t ext4 /dev/sdbu5 /var/lib/ceph/osd-lockbox/8ef450fe-d60b-4199-a1aa-66c9b4681ab7
command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd-lockbox/8ef450fe-d60b-4199-a1aa-66c9b4681ab7/osd-uuid.18288.tmp
command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd-lockbox/8ef450fe-d60b-4199-a1aa-66c9b4681ab7/ceph_fsid.18288.tmp
command: Running command: /usr/bin/ceph-authtool --gen-print-key
__init__: stderr
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
command: Running command: /usr/bin/ceph-authtool --gen-print-key
__init__: stderr
command_with_stdin: Running command with stdin: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8ef450fe-d60b-4
199-a1aa-66c9b4681ab7
command_with_stdin:
command_with_stdin: 2017-10-20 21:46:04.678460 7fe8b53bb700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
2017-10-20 21:46:04.678478 7fe8b53bb700 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication
2017-10-20 21:46:04.678481 7fe8b53bb700 0 librados: client.bootstrap-osd initialization error (2) No such file or directory
[errno 2] error connecting to the cluster
'['ceph', '--cluster', 'ceph', '--name', 'client.bootstrap-osd', '--keyring', '/var/lib/ceph/bootstrap-osd/ceph.keyring', '-i', '-', 'osd', 'new', '8ef450fe-d60b-4199-a1aa-66c
9b4681ab7']' failed with status code 1
----------------------FAILURE--------------------------
ceph-disk -v prepare --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --block.db /dev/sdd --block.wal /dev/sdd --bluestore --cluster ceph --fs-type xfs -- /dev/sdbu
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
set_type: Will colocate block with data on /dev/sdbu
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_size
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_db_size
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_size
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup bluestore_block_wal_size
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdbu uuid path is /sys/dev/block/68:128/dm/uuid
Traceback (most recent call last):
File "/usr/sbin/ceph-disk", line 9, in <module>
load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5704, in run
main(sys.argv[1:])
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5655, in main
args.func(args)
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2091, in main
Prepare.factory(args).prepare()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2080, in prepare
self._prepare()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2154, in _prepare
self.lockbox.prepare()
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 2842, in prepare
verify_not_in_use(self.args.lockbox, check_partitions=True)
File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 950, in verify_not_in_use
raise Error('Device is mounted', partition)
now i'm back to where i was 5 seconds ago. I must be doing something wrong but I have no idea what it is.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
