Specify OSD size and OSD journal size with ceph-ansible

[Robert LeBlanc <robert@xxxxxxxxxxxxx>]

I have a new cluster and I'd like to put the DB on the NVMe device, but only make it 30GB, then use 100GB of the rest of the NVMe as an OSD for the RGW metadata pool.

I set up the disks like the conf below without the block_db_size and it created all the LVs on the HDDs and one LV on the NVMe that took up all the space.

I've tried using block_db_size in vars, and also as a property in the list for each OSD disk but neither work.

With block_db_size in the vars I get:

failed: [sun-gcs02-osd01] (item={u'db': u'/dev/nvme0n1', u'data': u'/dev/sda', u'crush_device_class': u'hdd'}) => changed=true
 ansible_loop_var: item
 cmd:                                                                                                                                                                                            
 - docker                                                                                                                                                                                         - run 

 - --rm 

 - --privileged 

 - --net=host 

 - --ipc=host 

 - --ulimit 

 - nofile=1024:1024 

 - -v 

 - /run/lock/lvm:/run/lock/lvm:z 

 - -v 

 - /var/run/udev/:/var/run/udev/:z 

 - -v 

 - /dev:/dev 

 - -v 

 - /etc/ceph:/etc/ceph:z 

 - -v 

 - /run/lvm/:/run/lvm/ 

 - -v 

 - /var/lib/ceph/:/var/lib/ceph/:z 

 - -v 

 - /var/log/ceph/:/var/log/ceph/:z 

 - --entrypoint=ceph-volume 

 - docker.io/ceph/daemon:latest 

 - --cluster 

 - ceph 

 - lvm 

 - prepare 

 - --bluestore 

 - --data 

 - /dev/sda 

 - --block.db 

 - /dev/nvme0n1 

 - --crush-device-class 

 - hdd 

 delta: '0:00:05.004777' 

 end: '2019-08-28 23:26:39.074850' 

 item: 

   crush_device_class: hdd 

   data: /dev/sda
   db: /dev/nvme0n1
 msg: non-zero return code
 rc: 1
 start: '2019-08-28 23:26:34.070073'
 stderr: '-->  RuntimeError: unable to use device'
 stderr_lines: <omitted>
 stdout: |-
   Running command: /bin/ceph-authtool --gen-print-key
   Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new bcc7b3c3-6203-47c7-9f34-7b2e2060bf59
   Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-76cd6a80-17dd-4a89-a35b-0844026bc9d4 /dev/sda
    stdout: Physical volume "/dev/sda" successfully created.
    stdout: Volume group "ceph-76cd6a80-17dd-4a89-a35b-0844026bc9d4" successfully created
   Running command: /usr/sbin/lvcreate --yes -l 100%FREE -n osd-block-bcc7b3c3-6203-47c7-9f34-7b2e2060bf59 ceph-76cd6a80-17dd-4a89-a35b-0844026bc9d4
    stdout: Logical volume "osd-block-bcc7b3c3-6203-47c7-9f34-7b2e2060bf59" created.
   --> blkid could not detect a PARTUUID for device: /dev/nvme0n1
   --> Was unable to complete a new OSD, will rollback changes
   Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.21 --yes-i-really-mean-it
    stderr: purged osd.21
  stdout_lines: <omitted>

... (One for each device) ...

And the LVs are created for all the HDD OSDs and none on the NVMe.

Looking through the code I don't see a way to set a size for the OSD, but maybe I'm just missing it as I'm really new to Ansible.

        osds:
         hosts:
           sun-gcs02-osd[01:43]:
           sun-gcs02-osd[45:60]:
         vars:
           block_db_size: 32212254720
           lvm_volumes:
             - data: '/dev/sda'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdb'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdc'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdd'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sde'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdf'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdg'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdh'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdi'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdj'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdk'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/sdl'
               db: '/dev/nvme0n1'
               crush_device_class: 'hdd'
             - data: '/dev/nvme0n1'      # Use the rest for metadata
               crush_device_class: 'nvme'

With block_db_size set for each disk, I got an error during the parameter checking phase in Ansible and no LVs were created.

Please help me understand how to configure what I would like to do.

Thank you,

Robert LeBlanc

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com