# ceph-volume lvm zap --destroy osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
Running command: /usr/sbin/cryptsetup status /dev/mapper/
--> Zapping: osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
--> Destroying physical volume osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz because --destroy was given
Running command: /usr/sbin/pvremove -v -f -f osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
stderr: Device osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz not found.
--> Unable to remove vg osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
--> RuntimeError: command returned non-zero exit status: 5
Running command: /usr/sbin/cryptsetup status /dev/mapper/
--> Zapping: osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
--> Destroying physical volume osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz because --destroy was given
Running command: /usr/sbin/pvremove -v -f -f osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
stderr: Device osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz not found.
--> Unable to remove vg osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
--> RuntimeError: command returned non-zero exit status: 5
This is how destroy failed before I started deleting volumes.
On Thu, Apr 18, 2019 at 2:26 PM Alfredo Deza <adeza@xxxxxxxxxx> wrote:
On Thu, Apr 18, 2019 at 3:01 PM Sergei Genchev <sgenchev@xxxxxxxxx> wrote:
>
> Thank you Alfredo
> I did not have any reasons to keep volumes around.
> I tried using ceph-volume to zap these stores, but none of the command worked, including yours 'ceph-volume lvm zap osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz'
If you do not want to keep them around you would need to use --destroy
and use the lv path as input:
ceph-volume lvm zap --destroy osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
>
> I ended up manually removing LUKS volumes and then deleting LVM LV, VG, and PV
>
> cryptsetup remove /dev/mapper/AeV0iG-odWF-NRPE-1bVK-0mxH-OgHL-fneTzr
> cryptsetup remove /dev/mapper/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
> lvremove /dev/ceph-f4efa78f-a467-4214-b550-81653da1c9bd/osd-block-097d59be-bbe6-493a-b785-48b259d2ff35
> sgdisk -Z /dev/sdd
>
> # ceph-volume lvm zap osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
> Running command: /usr/sbin/cryptsetup status /dev/mapper/
> --> Zapping: osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
> Running command: /usr/sbin/wipefs --all osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
> stderr: wipefs: error: osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz: probing initialization failed: No such file or directory
> --> RuntimeError: command returned non-zero exit status: 1
In this case, you removed the LV so the wipefs failed because that LV
no longer exists. Do you have output on how it failed before?
>
>
> On Thu, Apr 18, 2019 at 10:10 AM Alfredo Deza <adeza@xxxxxxxxxx> wrote:
>>
>> On Thu, Apr 18, 2019 at 10:55 AM Sergei Genchev <sgenchev@xxxxxxxxx> wrote:
>> >
>> > Hello,
>> > I have a server with 18 disks, and 17 OSD daemons configured. One of the OSD daemons failed to deploy with ceph-deploy. The reason for failing is unimportant at this point, I believe it was race condition, as I was running ceph-deploy inside while loop for all disks in this server.
>> > Now I have two left over LVM dmcrypded volumes that I am not sure how clean up. The command that failed and did not quite clean up after itself was:
>> > ceph-deploy osd create --bluestore --dmcrypt --data /dev/sdd --block-db osvg/sdd-db ${SERVERNAME}
>> >
>> > # lsblk
>> > .......
>> > sdd 8:48 0 7.3T 0 disk
>> > └─ceph--f4efa78f--a467--4214--b550--81653da1c9bd-osd--block--097d59be--bbe6--493a--b785--48b259d2ff35
>> > 253:32 0 7.3T 0 lvm
>> > └─AeV0iG-odWF-NRPE-1bVK-0mxH-OgHL-fneTzr 253:33 0 7.3T 0 crypt
>> >
>> > sds 65:32 0 223.5G 0 disk
>> > ├─sds1 65:33 0 512M 0 part /boot
>> > └─sds2 65:34 0 223G 0 part
>> > .......
>> > ├─osvg-sdd--db 253:8 0 8G 0 lvm
>> > │ └─2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz 253:34 0 8G 0 crypt
>> >
>> > # ceph-volume inventory /dev/sdd
>> >
>> > ====== Device report /dev/sdd ======
>> >
>> > available False
>> > rejected reasons locked
>> > path /dev/sdd
>> > scheduler mode deadline
>> > rotational 1
>> > vendor SEAGATE
>> > human readable size 7.28 TB
>> > sas address 0x5000c500a6b1d581
>> > removable 0
>> > model ST8000NM0185
>> > ro 0
>> > --- Logical Volume ---
>> > cluster name ceph
>> > name osd-block-097d59be-bbe6-493a-b785-48b259d2ff35
>> > osd id 39
>> > cluster fsid 8e7a3953-7647-4133-9b9a-7f4a2e2b7da7
>> > type block
>> > block uuid AeV0iG-odWF-NRPE-1bVK-0mxH-OgHL-fneTzr
>> > osd fsid 097d59be-bbe6-493a-b785-48b259d2ff35
>> >
>> > I was trying to run
>> > ceph-volume lvm zap --destroy /dev/sdd but it errored out. Osd id on this volume is the same as on next drive, /dev/sde, and osd.39 daemon is running. This command was trying to zap running osd.
>> >
>> > What is the proper way to clean both data and block db volumes, so I can rerun ceph-deploy again, and add them to the pool?
>> >
>>
>> Do you want to keep the LVs around or you want to complete get rid of
>> them? If you are passing /dev/sdd to 'zap' you are telling the tool to
>> destroy everything that is in there, regardless of who owns it
>> (including running
>> OSDs).
>>
>> If you want to keep LVs around then you can omit the --destroy flag
>> and pass the LVs as input, or if using a recent enough version you can
>> use --osd-fsid to zap:
>>
>> ceph-volume lvm zap osvg-sdd-db/2ukzAx-g9pZ-IyxU-Sp9h-fHv2-INNY-1vTpvz
>>
>> If you don't want the LVs around you can add --destroy, but use the LV
>> as input (not the device)
>>
>> > Thank you!
>> >
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
