Re: Filestore -> Bluestore

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Jun 12, 2018 at 10:06 AM, Vadim Bulst
<vadim.bulst@xxxxxxxxxxxxxx> wrote:
> Thanks Alfredo - I can imagine why. I edited the filter in lvm.conf and well
> it is definitely not generic.   Is there a other way to get this setup
> working? Or do I have to go back to filestore?

If you have an LV on top of your multipath device and use that for
ceph-volume it should work (we don't test this, but we do ensure we
work with LVs)

What is not supported is ceph-volume creating an LV from a multipath
device as input.

This is not related to the objectstore type.

>
> Cheers,
>
> Vadim
>
>
>
>
> On 12.06.2018 14:41, Alfredo Deza wrote:
>>
>> On Tue, Jun 12, 2018 at 7:04 AM, Vadim Bulst <vadim.bulst@xxxxxxxxxxxxxx>
>> wrote:
>>>
>>> I cannot release this lock! This is an expansion shelf connected with two
>>> cables to the controller. If there is no multipath management, the os
>>> would
>>> see every disk at least twice. Ceph has to deal with it somehow. I guess
>>> I'm
>>> not the only one who has a setup like this.
>>>
>> Do you have an LV on top of that dm? We don't support multipath devices:
>>
>>
>> http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#multipath-support
>>
>>> Best,
>>>
>>> Vadim
>>>
>>>
>>>
>>>
>>> On 12.06.2018 12:55, Alfredo Deza wrote:
>>>>
>>>> On Tue, Jun 12, 2018 at 6:47 AM, Vadim Bulst
>>>> <vadim.bulst@xxxxxxxxxxxxxx>
>>>> wrote:
>>>>>
>>>>> Hi Alfredo,
>>>>>
>>>>> thanks for your help. Yust to make this clear /dev/dm-0 is the name of
>>>>> my
>>>>> multipath disk:
>>>>>
>>>>> root@polstor01:/home/urzadmin# ls -la /dev/disk/by-id/ | grep dm-0
>>>>> lrwxrwxrwx 1 root root   10 Jun 12 07:50 dm-name-35000c500866f8947 ->
>>>>> ../../dm-0
>>>>> lrwxrwxrwx 1 root root   10 Jun 12 07:50
>>>>> dm-uuid-mpath-35000c500866f8947
>>>>> ->
>>>>> ../../dm-0
>>>>> lrwxrwxrwx 1 root root   10 Jun 12 07:50 scsi-35000c500866f8947 ->
>>>>> ../../dm-0
>>>>> lrwxrwxrwx 1 root root   10 Jun 12 07:50 wwn-0x5000c500866f8947 ->
>>>>> ../../dm-0
>>>>>
>>>>> If I run pvdisplay this device is not listed.
>>>>
>>>> Either way, you should not use dm devices directly. If this is a
>>>> multipath disk, then you must use that other name instead of /dev/dm-*
>>>>
>>>> I am not sure what kind of setup you have, but that mapper must
>>>> release its lock so that you can zap. We ensure that works with LVM, I
>>>> am not sure
>>>> how to do that in your environment.
>>>>
>>>> For example, with dmcrypt you get into similar issues, that is why we
>>>> check crypsetup, so that we can make dmcrypt release that device
>>>> before zapping.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Vadim
>>>>>
>>>>>
>>>>>
>>>>> On 12.06.2018 12:40, Alfredo Deza wrote:
>>>>>>
>>>>>> On Tue, Jun 12, 2018 at 4:37 AM, Vadim Bulst
>>>>>> <vadim.bulst@xxxxxxxxxxxxxx>
>>>>>> wrote:
>>>>>>>
>>>>>>> no change:
>>>>>>>
>>>>>>>
>>>>>>> root@polstor01:/home/urzadmin# ceph-volume lvm zap --destroy
>>>>>>> /dev/dm-0
>>>>>>> --> Zapping: /dev/dm-0
>>>>>>
>>>>>> This is the problem right here. Your script is using the dm device
>>>>>> that belongs to an LV.
>>>>>>
>>>>>> What you want to do here is destroy/zap the LV. Not the dm device that
>>>>>> belongs to the LV.
>>>>>>
>>>>>> To make this clear in the future, I've created:
>>>>>> http://tracker.ceph.com/issues/24504
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Running command: /sbin/cryptsetup status /dev/mapper/
>>>>>>>     stdout: /dev/mapper/ is inactive.
>>>>>>> --> Skipping --destroy because no associated physical volumes are
>>>>>>> found
>>>>>>> for
>>>>>>> /dev/dm-0
>>>>>>> Running command: wipefs --all /dev/dm-0
>>>>>>>     stderr: wipefs: error: /dev/dm-0: probing initialization failed:
>>>>>>> Device
>>>>>>> or
>>>>>>> resource busy
>>>>>>> -->  RuntimeError: command returned non-zero exit status: 1
>>>>>>>
>>>>>>>
>>>>>>> On 12.06.2018 09:03, Linh Vu wrote:
>>>>>>>
>>>>>>> ceph-volume lvm zap --destroy $DEVICE
>>>>>>>
>>>>>>> ________________________________
>>>>>>> From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of
>>>>>>> Vadim
>>>>>>> Bulst <vadim.bulst@xxxxxxxxxxxxxx>
>>>>>>> Sent: Tuesday, 12 June 2018 4:46:44 PM
>>>>>>> To: ceph-users@xxxxxxxxxxxxxx
>>>>>>> Subject: Re:  Filestore -> Bluestore
>>>>>>>
>>>>>>>
>>>>>>> Thanks Sergey.
>>>>>>>
>>>>>>> Could you specify your answer a bit more? When I look into the
>>>>>>> manpage
>>>>>>> of
>>>>>>> ceph-volume I couldn't find an option named "--destroy".
>>>>>>>
>>>>>>> I just like to make clear - this script has already migrated several
>>>>>>> servers. The problem is appearing when it should migrate devices in
>>>>>>> the
>>>>>>> expansion shelf.
>>>>>>>
>>>>>>> "-->  RuntimeError: Cannot use device (/dev/dm-0). A vg/lv path or an
>>>>>>> existing device is needed"
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Vadim
>>>>>>>
>>>>>>>
>>>>>>> I would say the handling of devices
>>>>>>> On 11.06.2018 23:58, Sergey Malinin wrote:
>>>>>>>
>>>>>>> “Device or resource busy” error rises when no “--destroy” option is
>>>>>>> passed
>>>>>>> to ceph-volume.
>>>>>>> On Jun 11, 2018, 22:44 +0300, Vadim Bulst
>>>>>>> <vadim.bulst@xxxxxxxxxxxxxx>,
>>>>>>> wrote:
>>>>>>>
>>>>>>> Dear Cephers,
>>>>>>>
>>>>>>> I'm trying to migrate our OSDs to Bluestore using this little script:
>>>>>>>
>>>>>>> #!/bin/bash
>>>>>>> HOSTNAME=$(hostname -s)
>>>>>>> OSDS=`ceph osd metadata | jq -c '[.[] | select(.osd_objectstore |
>>>>>>> contains("filestore")) ]' | jq '[.[] | select(.hostname |
>>>>>>> contains("'${HOSTNAME}'")) ]' | jq '.[].id'`
>>>>>>> IFS=' ' read -a OSDARRAY <<<$OSDS
>>>>>>> for OSD in "${OSDARRAY[@]}"; do
>>>>>>>      DEV=/dev/`ceph osd metadata | jq -c '.[] | select(.id=='${OSD}')
>>>>>>> |
>>>>>>> .backend_filestore_dev_node' | sed 's/"//g'`
>>>>>>>      echo "=== Migrating OSD nr ${OSD} on device ${DEV} ==="
>>>>>>>      ceph osd out ${OSD}
>>>>>>>        while ! ceph osd safe-to-destroy ${OSD} ; do echo "waiting for
>>>>>>> full
>>>>>>> evacuation"; sleep 60 ; done
>>>>>>>          systemctl stop ceph-osd@${OSD}
>>>>>>>          umount /var/lib/ceph/osd/ceph-${OSD}
>>>>>>>          /usr/sbin/ceph-volume lvm zap ${DEV}
>>>>>>>          ceph osd destroy ${OSD} --yes-i-really-mean-it
>>>>>>>          /usr/sbin/ceph-volume lvm create --bluestore --data ${DEV}
>>>>>>> --osd-id ${OSD}
>>>>>>> done
>>>>>>>
>>>>>>> Unfortunately - under normal circumstances this works flawlessly. In
>>>>>>> our
>>>>>>> case we have expansion shelfs connected as multipath devices to our
>>>>>>> nodes.
>>>>>>>
>>>>>>> /usr/sbin/ceph-volume lvm zap ${DEV}  is breaking with an error:
>>>>>>>
>>>>>>> OSD(s) 1 are safe to destroy without reducing data durability.
>>>>>>> --> Zapping: /dev/dm-0
>>>>>>> Running command: /sbin/cryptsetup status /dev/mapper/
>>>>>>>     stdout: /dev/mapper/ is inactive.
>>>>>>> Running command: wipefs --all /dev/dm-0
>>>>>>>     stderr: wipefs: error: /dev/dm-0: probing initialization failed:
>>>>>>> Device or resource busy
>>>>>>> -->  RuntimeError: command returned non-zero exit status: 1
>>>>>>> destroyed osd.1
>>>>>>> Running command: /usr/bin/ceph-authtool --gen-print-key
>>>>>>> Running command: /usr/bin/ceph --cluster ceph --name
>>>>>>> client.bootstrap-osd --keyring
>>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring
>>>>>>> osd tree -f json
>>>>>>> Running command: /usr/bin/ceph --cluster ceph --name
>>>>>>> client.bootstrap-osd --keyring
>>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring
>>>>>>> -i - osd new 74f6ff02-d027-4fc6-9b93-3a96d753
>>>>>>> 5c8f 1
>>>>>>> --> Was unable to complete a new OSD, will rollback changes
>>>>>>> --> OSD will be destroyed, keeping the ID because it was provided
>>>>>>> with
>>>>>>> --osd-id
>>>>>>> Running command: ceph osd destroy osd.1 --yes-i-really-mean-it
>>>>>>>     stderr: destroyed osd.1
>>>>>>>
>>>>>>> -->  RuntimeError: Cannot use device (/dev/dm-0). A vg/lv path or an
>>>>>>> existing device is needed
>>>>>>>
>>>>>>>
>>>>>>> Does anybody know how to solve this problem?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Vadim
>>>>>>>
>>>>>>> --
>>>>>>> Vadim Bulst
>>>>>>>
>>>>>>> Universität Leipzig / URZ
>>>>>>> 04109 Leipzig, Augustusplatz 10
>>>>>>>
>>>>>>> phone: +49-341-97-33380
>>>>>>> mail: vadim.bulst@xxxxxxxxxxxxxx
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Vadim Bulst
>>>>>>>
>>>>>>> Universität Leipzig / URZ
>>>>>>> 04109  Leipzig, Augustusplatz 10
>>>>>>>
>>>>>>> phone: ++49-341-97-33380
>>>>>>> mail:    vadim.bulst@xxxxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Vadim Bulst
>>>>>>>
>>>>>>> Universität Leipzig / URZ
>>>>>>> 04109  Leipzig, Augustusplatz 10
>>>>>>>
>>>>>>> phone: ++49-341-97-33380
>>>>>>> mail:    vadim.bulst@xxxxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>
>>>>> --
>>>>> Vadim Bulst
>>>>>
>>>>> Universität Leipzig / URZ
>>>>> 04109  Leipzig, Augustusplatz 10
>>>>>
>>>>> phone: ++49-341-97-33380
>>>>> mail:    vadim.bulst@xxxxxxxxxxxxxx
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>> --
>>> Vadim Bulst
>>>
>>> Universität Leipzig / URZ
>>> 04109  Leipzig, Augustusplatz 10
>>>
>>> phone: ++49-341-97-33380
>>> mail:    vadim.bulst@xxxxxxxxxxxxxx
>>>
>>>
>
> --
> Vadim Bulst
>
> Universität Leipzig / URZ
> 04109  Leipzig, Augustusplatz 10
>
> phone: ++49-341-97-33380
> mail:    vadim.bulst@xxxxxxxxxxxxxx
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux