After looking to code in ceph-disk I came to the same conclusion, problem is with the mapping.
Here are quote form ceph-disk
def get_partition_dev(dev, pnum):
"""
get the device name for a partition
assume that partitions are named like the base dev, with a number, and optionally
some intervening characters (like 'p'). e.g.,
sda 1 -> sda1
cciss/c0d1 1 -> cciss!c0d1p1
"""
Script are looking for partitions labeled as "sdb[X]" or "p[X]", where [x] means number of partitions (counted from 1).
Dm-crypt are creating some new mapping in /dev/mapper/, example /dev/mapper/osd0 as main block device and /dev/mapper/osdp1 as first partition and /dev/mapper/osdp2 as second partition.
But real path to osd0 device is NOT /dev/mapper/osd0 but /dev/dm-0 (sic!), and /dev/dm-1 is as first partition (osdp1), /dev/dm-2 is as second partition (osdp2).
Conlusion. If we are using dm-crypt the script in ceph-disk should not looking partitions like sda partition 1 -> sda1 or osd0 partition 1-> osdp1 but should looking for partitions labeled as /dev/dm-X (counted from 1).
Block device Real path
/dev/mapper/osd0 -> /dev/dm-0
First partition Real path
/dev/mapper/osd0p1 -> /dev/dm-1
Second partition Real path
/dev/mapper/osd0p2 -> /dev/dm-2
Continuing, 'ceph-disk activate' should mount dm-crypted partitions not by using /dev/disk/by-partuuid, but /dev/disk/by-uuid
--
Best regards,
Michel Lukzak
>> ceph-disk-prepare --fs-type xfs --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --cluster ceph -- /dev/sdb
>> ceph-disk: Error: Device /dev/sdb2 is in use by a device-mapper mapping (dm-crypt?): dm-0
> It sounds like device-mapper still thinks it's using the the volume,
> you might be able to track it down with this:
> for i in `ls -1 /sys/block/ | grep sd`; do echo $i: `ls
> /sys/block/$i/${i}1/holders/`; done
> Then it's a matter of making sure there are no open file handles on
> the encrypted volume and unmounting it. You will still need to
> completely clear out the partition table on that disk, which can be
> tricky with GPT because it's not as simple as dd'in the start of the
> volume. This is what the zapdisk parameter is for in
> ceph-disk-prepare, I don't know enough about ceph-deploy to know if
> you can somehow pass it.
> After you know the device/dm mapping you can use udevadm to find out
> where it should map to (uuids replaced with xxx's):
> udevadm test /block/sdc/sdc1
> <snip>
> run: '/sbin/cryptsetup --key-file /etc/ceph/dmcrypt-keys/xxxxx
> --key-size 256 create xxxx /dev/sdc1'
> run: '/bin/bash -c 'while [ ! -e /dev/mapper/xxxxx ];do sleep 1; done''
> run: '/usr/sbin/ceph-disk-activate /dev/mapper/xxxxx'
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
