On Tue, Jan 22, 2019 at 6:45 AM Manuel Lausch <manuel.lausch@xxxxxxxx> wrote: > > Hi, > > we want upgrade our ceph clusters from jewel to luminous. And also want > to migrate the osds to ceph-volume described in > http://docs.ceph.com/docs/luminous/ceph-volume/simple/scan/#ceph-volume-simple-scan > > The clusters are running since dumpling and are setup with dmcrypt. > Since dumpling there are until now three different types of dmcrypt > > plain dmcrypt with keys local > luks with keys local > luks with keys on the ceph monitors > > Now it seems only the last type can be migrated to ceph-volume. > > ceph-volume simple scan trys to mount a lockbox which does not exists > on the older OSDs. Are those OSDs not supported with ceph-volume? This is one case we didn't anticipate :/ We supported the wonky lockbox setup and thought we wouldn't need to go further back, although we did add support for both plain and luks keys. Looking through the code, it is very tightly couple to storing/retrieving keys from the monitors, and I don't know what workarounds might be possible here other than throwing away the OSD and deploying a new one (I take it this is not an option for you at all) > > This are the errors: > > # ceph-volume simple scan /var/lib/ceph/osd/ceph-183 > stderr: lsblk: /var/lib/ceph/osd/ceph-183: not a block device > stderr: lsblk: /var/lib/ceph/osd/ceph-183: not a block device > Running command: /usr/sbin/cryptsetup status /dev/mapper/21ad7722-002f-464c-b460-a8976a7b4872 > Running command: /usr/sbin/cryptsetup status 21ad7722-002f-464c-b460-a8976a7b4872 > Running command: mount -v /tmp/tmp3t1WRC > stderr: mount: is write-protected, mounting read-only > stderr: mount: unknown filesystem type '(null)' > --> RuntimeError: command returned non-zero exit status: 32 > > > and this is in the ceph-volume.log > > [2019-01-22 12:39:31,456][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -p /dev/mapper/9b68b7e9-854e-498a-8381-4eef128a9d7a > [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] detecting if argument is a device or a directory: /var/lib/ceph/osd/ceph-183 > [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] will scan directly, path is a directory > [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] will scan encrypted OSD directory at path: /var/lib/ceph/osd/ceph-183 > [2019-01-22 12:39:31,534][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -s PARTUUID -o value /dev/sdv1 > [2019-01-22 12:39:31,539][ceph_volume.process][INFO ] stdout 21ad7722-002f-464c-b460-a8976a7b4872 > [2019-01-22 12:39:31,540][ceph_volume.process][INFO ] Running command: /usr/sbin/cryptsetup status 21ad7722-002f-464c-b460-a8976a7b4872 > [2019-01-22 12:39:31,546][ceph_volume.process][INFO ] stdout /dev/mapper/21ad7722-002f-464c-b460-a8976a7b4872 is active and is in use. > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout type: PLAIN > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout cipher: aes-cbc-essiv:sha256 > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout keysize: 256 bits > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout key location: dm-crypt > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout device: /dev/sdv1 > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout sector size: 512 > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout offset: 0 sectors > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout size: 7805646479 sectors > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout mode: read/write > [2019-01-22 12:39:31,548][ceph_volume.process][INFO ] Running command: mount -v /tmp/tmp3t1WRC > [2019-01-22 12:39:31,597][ceph_volume.process][INFO ] stderr mount: is write-protected, mounting read-only > [2019-01-22 12:39:31,622][ceph_volume.process][INFO ] stderr mount: unknown filesystem type '(null)' > [2019-01-22 12:39:31,622][ceph_volume][ERROR ] exception caught by decorator > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc > return f(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148, in main > terminal.dispatch(self.mapper, subcommand_args) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line 33, in main > terminal.dispatch(self.mapper, self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 353, in main > self.scan(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 244, in scan > osd_metadata = self.scan_encrypted(osd_path) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 169, in scan_encrypted > with system.tmp_mount(lockbox) as lockbox_path: > File "/usr/lib/python2.7/site-packages/ceph_volume/util/system.py", line 145, in __enter__ > self.path > File "/usr/lib/python2.7/site-packages/ceph_volume/process.py", line 153, in run > raise RuntimeError(msg) > RuntimeError: command returned non-zero exit status: 32 > > > > ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) > luminous (stable) > > > Regards > Manuel > > > > > -- > Manuel Lausch > > Systemadministrator > Storage Services > > 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany > Phone: +49 721 91374-1847 > E-Mail: manuel.lausch@xxxxxxxx | Web: www.1und1.de > > Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 > > Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer > > > Member of United Internet > > Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. > > This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |