On Wed, Jul 11, 2018 at 12:57 PM, Huseyin Cotuk <hcotuk@xxxxxxxxx> wrote: > Hi Paul, > > Thanks for your reply. I did not mention any special parameter while > upgrading to luminous. So this ceph-deploy version is the one coming from > the official debian luminous repo. That is because ceph-volume came out in Luminous and fully supports most everything ceph-disk provided. Another alternative is to allow ceph-volume to take over the management of your ceph-disk OSDs, although that is not provided via ceph-deploy. In the meantime, you can still use ceph-disk in Luminous, but like Paul mentioned it will require you to downgrade ceph-deploy to 1.5.39 > > I will try to downgrade ceph-deploy and try to add osd again. To prevent any > inconsistency, maybe you can check the version of ceph-deploy package on the > repository. > > My best, > Huseyin > On 11 Tem 2018 19:49 +0300, Paul Emmerich <paul.emmerich@xxxxxxxx>, wrote: > > BlueStore is really stable and mature nowadays. > > You seem to be using ceph-deploy 2.0.0 which I would not call mature and > stable at the moment ;) > Anyways, it uses ceph-volume instead of ceph-disk and I think you have to > specify the actual partition here. > But I'd just downgrade to ceph-deploy 1.5.39 when running Luminous (not a > long-term solution as ceph-disk will > be removed in Nautilus) > > Paul > > 2018-07-11 18:37 GMT+02:00 Huseyin Cotuk <hcotuk@xxxxxxxxx>: >> >> Hello everybody, >> >> I have just upgraded my ceph cluster from kraken to luminous. I just want >> to go on with filestore based objectstore for OSDs until Redhat announces >> bluestore as stable. It is still in technical preview. >> >> So my question is: “What is the right procedure of adding an filestore >> based OSD into the existing cluster with an NVME journal?" >> >> My NVME journal contains ceph journal partitions for existing OSD created >> with kraken. >> >> root@ank-ceph10:~# lsblk /dev/nvme0n1 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >> nvme0n1 259:0 0 2.9T 0 disk >> |-nvme0n1p1 259:1 0 40G 0 part >> |-nvme0n1p2 259:2 0 40G 0 part >> |-nvme0n1p3 259:3 0 40G 0 part >> |-nvme0n1p4 259:4 0 40G 0 part >> |-nvme0n1p5 259:5 0 40G 0 part >> |-nvme0n1p6 259:6 0 40G 0 part >> |-nvme0n1p7 259:7 0 40G 0 part >> |-nvme0n1p8 259:8 0 40G 0 part >> |-nvme0n1p9 259:9 0 40G 0 part >> |-nvme0n1p10 259:10 0 40G 0 part >> |-nvme0n1p11 259:11 0 40G 0 part >> `-nvme0n1p12 259:12 0 40G 0 part >> >> When i try to add a new osd with the following command, >> >> ceph-deploy osd create --filestore --journal /dev/nvme0n1 --data /dev/sdl >> ank-ceph10 >> >> I get the following error: >> >> [ank-ceph10][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster >> ceph lvm create --filestore --data /dev/sdl --journal /dev/nvme0n1 >> [ank-ceph10][WARNIN] --> RuntimeError: unable to use device >> [ank-ceph10][DEBUG ] Running command: /usr/bin/ceph-authtool >> --gen-print-key >> [ank-ceph10][DEBUG ] Running command: /usr/bin/ceph --cluster ceph --name >> client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - >> osd new 2d203f03-e547-4a8a-9140-53f48ed52e06 >> [ank-ceph10][DEBUG ] Running command: vgcreate --force --yes >> ceph-35465726-457d-439d-9f59-a8a050f5a486 /dev/sdl >> [ank-ceph10][DEBUG ] stderr: /run/lvm/lvmetad.socket: connect failed: No >> such file or directory >> [ank-ceph10][DEBUG ] WARNING: Failed to connect to lvmetad. Falling back >> to internal scanning. >> [ank-ceph10][DEBUG ] stdout: Physical volume "/dev/sdl" successfully >> created >> [ank-ceph10][DEBUG ] stdout: Volume group >> "ceph-35465726-457d-439d-9f59-a8a050f5a486" successfully created >> [ank-ceph10][DEBUG ] Running command: lvcreate --yes -l 100%FREE -n >> osd-data-2d203f03-e547-4a8a-9140-53f48ed52e06 >> ceph-35465726-457d-439d-9f59-a8a050f5a486 >> [ank-ceph10][DEBUG ] stderr: /run/lvm/lvmetad.socket: connect failed: No >> such file or directory >> [ank-ceph10][DEBUG ] WARNING: Failed to connect to lvmetad. Falling back >> to internal scanning. >> [ank-ceph10][DEBUG ] stdout: Logical volume >> "osd-data-2d203f03-e547-4a8a-9140-53f48ed52e06" created. >> [ank-ceph10][DEBUG ] --> blkid could not detect a PARTUUID for device: >> /dev/nvme0n1 >> [ank-ceph10][DEBUG ] --> Was unable to complete a new OSD, will rollback >> changes >> [ank-ceph10][DEBUG ] --> OSD will be fully purged from the cluster, >> because the ID was generated >> [ank-ceph10][DEBUG ] Running command: ceph osd purge osd.119 >> --yes-i-really-mean-it >> [ank-ceph10][DEBUG ] stderr: purged osd.119 >> [ank-ceph10][ERROR ] RuntimeError: command returned non-zero exit status: >> 1 >> [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume >> --cluster ceph lvm create --filestore --data /dev/sdl --journal /dev/nvme0n1 >> [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs >> >> AFAIU, blkid looks for PARTUUID for the journal device, but it does not >> have one. Only partitions have. I do not want to format my journal device. >> Any recommendation about this? >> >> root@ank-ceph10:~# blkid /dev/nvme0n1* >> /dev/nvme0n1: PTUUID="a6431404-5693-4076-98c9-ffbe84224e1b" PTTYPE="gpt" >> /dev/nvme0n1p1: PARTLABEL="ceph journal" >> PARTUUID="43440fab-a30f-4e42-9c15-35f375dde033" >> /dev/nvme0n1p10: PARTLABEL="ceph journal" >> PARTUUID="c9c9f459-98a1-4a6a-9350-9942a6fc02f6" >> /dev/nvme0n1p11: PARTLABEL="ceph journal" >> PARTUUID="3f64ddc1-ac5d-4b7b-ace3-ad35d44e4fd3" >> /dev/nvme0n1p12: PARTLABEL="ceph journal" >> PARTUUID="0fdff4d6-2833-4e6e-a832-9fb2452bc396" >> /dev/nvme0n1p2: PARTLABEL="ceph journal" >> PARTUUID="5ce0b4e8-3571-4297-974a-9ef648fac1a8" >> /dev/nvme0n1p3: PARTLABEL="ceph journal" >> PARTUUID="228cee11-06e3-4691-963a-77e74e099716" >> /dev/nvme0n1p4: PARTLABEL="ceph journal" >> PARTUUID="b7c09c3e-e4ae-42be-8686-5daf9e40c407" >> /dev/nvme0n1p5: PARTLABEL="ceph journal" >> PARTUUID="60d9115c-ebb1-4eaf-85ae-31379a5e9450" >> /dev/nvme0n1p6: PARTLABEL="ceph journal" >> PARTUUID="5a057b30-b697-4598-84c0-1794c608d70c" >> /dev/nvme0n1p7: PARTLABEL="ceph journal" >> PARTUUID="c22c272d-5b75-40ca-970e-87b1b303944c" >> /dev/nvme0n1p8: PARTLABEL="ceph journal" >> PARTUUID="ed9fd194-1490-42b1-a2b4-ae36b2a4f8ce" >> /dev/nvme0n1p9: PARTLABEL="ceph journal" >> PARTUUID="d5589315-4e47-49c4-91f5-48e1a55011d2" >> >> While using kraken, I used to add OSDs with journals via the follwing >> command: >> >> ceph-deploy osd prepare ank-ceph10:sdl:/dev/nvme0n1 >> >> >> Thanks for any recommendation. >> >> Best regards, >> >> Dr. Huseyin COTUK >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |