Were running a cluster with Hammer v94.2 and are running into issues with the Luks encrypted OSD data and journal partitions. The installation goes smoothly and everything runs OK, but we've had to reboot a couple of the storage nodes for various reasons and when they come back online, a large number of OSD processes fail to start because the LUKS encrypted partitions are not getting mounted correctly. I'm not sure if it is a udev issue or a problem with the OSD process itself, but the encrypted partitions end up getting mounted as "temporary-cryptsetup-PID" and they never recover. From below, you can see that some of the OSDs did come up correctly, but the majority do not. We've seen this problem now on several storage nodes, and it only occurs for those OSDs that used luks (the new default). The only recovery that we've found is to wipe them all out and rebuild them using "plain" dmcrypt (as it used to be). Using "blkid" on a partition that is in the "temporary-cryptsetup" state, does show that it has the right ID_PART_ENTRY_UUID and TYPE values and I can confirm that there is an associated key in /etc/ceph/dmcrypt-keys, but it still isn't mounting correctly. $ sudo blkid -p -o udev /dev/sdv2 ID_FS_UUID=87008c17-9e57-487d-8f8b-160f8f803d8b ID_FS_UUID_ENC=87008c17-9e57-487d-8f8b-160f8f803d8b ID_FS_VERSION=1 ID_FS_TYPE=crypto_LUKS ID_FS_USAGE=crypto ID_PART_ENTRY_SCHEME=gpt ID_PART_ENTRY_NAME=ceph\x20journal ID_PART_ENTRY_UUID=e3eda67b-a2e0-4d22-a62e-d9bda5ecf8b1 ID_PART_ENTRY_TYPE=45b0969e-9b03-4f30-b4c6-35865ceff106 ID_PART_ENTRY_NUMBER=2 ID_PART_ENTRY_OFFSET=2048 ID_PART_ENTRY_SIZE=20969473 ID_PART_ENTRY_DISK=65:80 So Im checking to see if this is a known issue or if we are missing something in the installation or configuration that would fix this problem. -Wyllys Ingersoll Ex: $ lsblk -l NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 111.8G 0 disk sda1 8:1 0 15.3G 0 part [SWAP] sda2 8:2 0 1K 0 part sda5 8:5 0 96.5G 0 part / sdb 8:16 0 3.7T 0 disk sdb1 8:17 0 3.6T 0 part e8bc1531-a187-4fd2-9e3f-cf90255f89d0 (dm-0) 252:0 0 3.6T 0 crypt sdb2 8:18 0 10G 0 part temporary-cryptsetup-1235 (dm-6) 252:6 0 125K 1 crypt sdc 8:32 0 3.7T 0 disk sdc1 8:33 0 3.6T 0 part temporary-cryptsetup-1788 (dm-37) 252:37 0 125K 1 crypt sdc2 8:34 0 10G 0 part temporary-cryptsetup-1789 (dm-36) 252:36 0 125K 1 crypt sdd 8:48 0 3.7T 0 disk sdd1 8:49 0 3.6T 0 part temporary-cryptsetup-1252 (dm-1) 252:1 0 125K 1 crypt sdd2 8:50 0 10G 0 part temporary-cryptsetup-1246 (dm-3) 252:3 0 125K 1 crypt sde 8:64 0 3.7T 0 disk sde1 8:65 0 3.6T 0 part temporary-cryptsetup-1260 (dm-14) 252:14 0 125K 1 crypt sde2 8:66 0 10G 0 part temporary-cryptsetup-1255 (dm-12) 252:12 0 125K 1 crypt sdf 8:80 0 3.7T 0 disk sdf1 8:81 0 3.6T 0 part temporary-cryptsetup-1268 (dm-15) 252:15 0 125K 1 crypt sdf2 8:82 0 10G 0 part temporary-cryptsetup-1245 (dm-5) 252:5 0 125K 1 crypt sdg 8:96 0 3.7T 0 disk sdg1 8:97 0 3.6T 0 part temporary-cryptsetup-1271 (dm-17) 252:17 0 125K 1 crypt sdg2 8:98 0 10G 0 part temporary-cryptsetup-1278 (dm-2) 252:2 0 125K 1 crypt sdh 8:112 0 3.7T 0 disk sdh1 8:113 0 3.6T 0 part 69dcd1e1-6e11-41ec-af19-1e0d90013957 (dm-43) 252:43 0 3.6T 0 crypt /var/lib/ceph/osd/ceph-42 sdh2 8:114 0 10G 0 part 3382723d-b0d9-4b50-affe-fb9f5df78d6f (dm-45) 252:45 0 10G 0 crypt sdi 8:128 0 3.7T 0 disk sdi1 8:129 0 3.6T 0 part temporary-cryptsetup-1265 (dm-20) 252:20 0 125K 1 crypt sdi2 8:130 0 10G 0 part temporary-cryptsetup-1277 (dm-16) 252:16 0 125K 1 crypt sdj 8:144 0 3.7T 0 disk sdj1 8:145 0 3.6T 0 part temporary-cryptsetup-1359 (dm-13) 252:13 0 125K 1 crypt sdj2 8:146 0 10G 0 part temporary-cryptsetup-1280 (dm-4) 252:4 0 125K 1 crypt sdk 8:160 0 3.7T 0 disk sdk1 8:161 0 3.6T 0 part temporary-cryptsetup-1760 (dm-34) 252:34 0 125K 1 crypt sdk2 8:162 0 10G 0 part temporary-cryptsetup-1761 (dm-31) 252:31 0 125K 1 crypt sdl 8:176 0 3.7T 0 disk sdl1 8:177 0 3.6T 0 part c3175d9f-ae12-4852-bbbc-b1d2c344c4ac (dm-38) 252:38 0 3.6T 0 crypt /var/lib/ceph/osd/ceph-32 sdl2 8:178 0 10G 0 part e4e10521-985a-4d94-a766-56d6de26443a (dm-41) 252:41 0 10G 0 crypt sdm 8:192 0 3.7T 0 disk sdm1 8:193 0 3.6T 0 part temporary-cryptsetup-1407 (dm-9) 252:9 0 125K 1 crypt sdm2 8:194 0 10G 0 part temporary-cryptsetup-1423 (dm-19) 252:19 0 125K 1 crypt sdn 8:208 0 3.7T 0 disk sdn1 8:209 0 3.6T 0 part temporary-cryptsetup-1442 (dm-11) 252:11 0 125K 1 crypt sdn2 8:210 0 10G 0 part temporary-cryptsetup-1433 (dm-7) 252:7 0 125K 1 crypt sdo 8:224 0 3.7T 0 disk sdo1 8:225 0 3.6T 0 part temporary-cryptsetup-1600 (dm-23) 252:23 0 125K 1 crypt sdo2 8:226 0 10G 0 part temporary-cryptsetup-1602 (dm-24) 252:24 0 125K 1 crypt sdp 8:240 0 3.7T 0 disk sdp1 8:241 0 3.6T 0 part temporary-cryptsetup-1634 (dm-27) 252:27 0 125K 1 crypt sdp2 8:242 0 10G 0 part temporary-cryptsetup-1638 (dm-25) 252:25 0 125K 1 crypt sdq 65:0 0 3.7T 0 disk sdq1 65:1 0 3.6T 0 part temporary-cryptsetup-1428 (dm-18) 252:18 0 125K 1 crypt sdq2 65:2 0 10G 0 part temporary-cryptsetup-1430 (dm-10) 252:10 0 125K 1 crypt sdr 65:16 0 3.7T 0 disk sdr1 65:17 0 3.6T 0 part temporary-cryptsetup-1727 (dm-29) 252:29 0 125K 1 crypt sdr2 65:18 0 10G 0 part temporary-cryptsetup-1728 (dm-32) 252:32 0 125K 1 crypt sds 65:32 0 3.7T 0 disk sds1 65:33 0 3.6T 0 part temporary-cryptsetup-1366 (dm-8) 252:8 0 125K 1 crypt sds2 65:34 0 10G 0 part temporary-cryptsetup-1611 (dm-21) 252:21 0 125K 1 crypt sdt 65:48 0 3.7T 0 disk sdt1 65:49 0 3.6T 0 part temporary-cryptsetup-1734 (dm-30) 252:30 0 125K 1 crypt sdt2 65:50 0 10G 0 part temporary-cryptsetup-1735 (dm-28) 252:28 0 125K 1 crypt sdu 65:64 0 3.7T 0 disk sdu1 65:65 0 3.6T 0 part temporary-cryptsetup-1605 (dm-22) 252:22 0 125K 1 crypt sdu2 65:66 0 10G 0 part temporary-cryptsetup-1607 (dm-26) 252:26 0 125K 1 crypt sdv 65:80 0 3.7T 0 disk sdv1 65:81 0 3.6T 0 part temporary-cryptsetup-1739 (dm-33) 252:33 0 125K 1 crypt sdv2 65:82 0 10G 0 part temporary-cryptsetup-1772 (dm-35) 252:35 0 125K 1 crypt sdw 65:96 0 3.7T 0 disk sdw1 65:97 0 3.6T 0 part 3171a1b9-e0f8-4521-a31a-821fcb549731 (dm-46) 252:46 0 3.6T 0 crypt /var/lib/ceph/osd/ceph-14 sdw2 65:98 0 10G 0 part 8c5882fd-21ef-4d9c-b62b-676248236514 (dm-47) 252:47 0 10G 0 crypt sdx 65:112 0 3.7T 0 disk sdx1 65:113 0 3.6T 0 part a576166d-07c4-468c-a704-c4080290a12e (dm-40) 252:40 0 3.6T 0 crypt /var/lib/ceph/osd/ceph-7 sdx2 65:114 0 10G 0 part 1a93e588-dbd4-4ce4-9955-e2f450576314 (dm-42) 252:42 0 10G 0 crypt sdy 65:128 0 3.7T 0 disk sdy1 65:129 0 3.6T 0 part da2f4e17-f2ba-49ce-bc11-fa699fbf0ba2 (dm-39) 252:39 0 3.6T 0 crypt /var/lib/ceph/osd/ceph-2 sdy2 65:130 0 10G 0 part 14422a1f-083c-44a8-ac6d-d2b4fe20650e (dm-44) 252:44 0 10G 0 crypt -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
