Hi, we want upgrade our ceph clusters from jewel to luminous. And also want to migrate the osds to ceph-volume described in http://docs.ceph.com/docs/luminous/ceph-volume/simple/scan/#ceph-volume-simple-scan The clusters are running since dumpling and are setup with dmcrypt. Since dumpling there are until now three different types of dmcrypt plain dmcrypt with keys local luks with keys local luks with keys on the ceph monitors Now it seems only the last type can be migrated to ceph-volume. ceph-volume simple scan trys to mount a lockbox which does not exists on the older OSDs. Are those OSDs not supported with ceph-volume? This are the errors: # ceph-volume simple scan /var/lib/ceph/osd/ceph-183 stderr: lsblk: /var/lib/ceph/osd/ceph-183: not a block device stderr: lsblk: /var/lib/ceph/osd/ceph-183: not a block device Running command: /usr/sbin/cryptsetup status /dev/mapper/21ad7722-002f-464c-b460-a8976a7b4872 Running command: /usr/sbin/cryptsetup status 21ad7722-002f-464c-b460-a8976a7b4872 Running command: mount -v /tmp/tmp3t1WRC stderr: mount: is write-protected, mounting read-only stderr: mount: unknown filesystem type '(null)' --> RuntimeError: command returned non-zero exit status: 32 and this is in the ceph-volume.log [2019-01-22 12:39:31,456][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -p /dev/mapper/9b68b7e9-854e-498a-8381-4eef128a9d7a [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] detecting if argument is a device or a directory: /var/lib/ceph/osd/ceph-183 [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] will scan directly, path is a directory [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] will scan encrypted OSD directory at path: /var/lib/ceph/osd/ceph-183 [2019-01-22 12:39:31,534][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -s PARTUUID -o value /dev/sdv1 [2019-01-22 12:39:31,539][ceph_volume.process][INFO ] stdout 21ad7722-002f-464c-b460-a8976a7b4872 [2019-01-22 12:39:31,540][ceph_volume.process][INFO ] Running command: /usr/sbin/cryptsetup status 21ad7722-002f-464c-b460-a8976a7b4872 [2019-01-22 12:39:31,546][ceph_volume.process][INFO ] stdout /dev/mapper/21ad7722-002f-464c-b460-a8976a7b4872 is active and is in use. [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout type: PLAIN [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout cipher: aes-cbc-essiv:sha256 [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout keysize: 256 bits [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout key location: dm-crypt [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout device: /dev/sdv1 [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout sector size: 512 [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout offset: 0 sectors [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout size: 7805646479 sectors [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout mode: read/write [2019-01-22 12:39:31,548][ceph_volume.process][INFO ] Running command: mount -v /tmp/tmp3t1WRC [2019-01-22 12:39:31,597][ceph_volume.process][INFO ] stderr mount: is write-protected, mounting read-only [2019-01-22 12:39:31,622][ceph_volume.process][INFO ] stderr mount: unknown filesystem type '(null)' [2019-01-22 12:39:31,622][ceph_volume][ERROR ] exception caught by decorator Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch instance.main() File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line 33, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch instance.main() File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 353, in main self.scan(args) File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 244, in scan osd_metadata = self.scan_encrypted(osd_path) File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 169, in scan_encrypted with system.tmp_mount(lockbox) as lockbox_path: File "/usr/lib/python2.7/site-packages/ceph_volume/util/system.py", line 145, in __enter__ self.path File "/usr/lib/python2.7/site-packages/ceph_volume/process.py", line 153, in run raise RuntimeError(msg) RuntimeError: command returned non-zero exit status: 32 ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) luminous (stable) Regards Manuel -- Manuel Lausch Systemadministrator Storage Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lausch@xxxxxxxx | Web: www.1und1.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |