Dear team, Kindly help on this , I am completely blocked. Best Regards Michel On Thu, Jan 5, 2023 at 2:45 PM Michel Niyoyita <micou12@xxxxxxxxx> wrote: > Dear team, > > I have deployed the ceph cluster in production using ceph-ansible on > ubuntu OS 20.04 it consists of 3 monitors and 3 osds (each osd has 20 disks > , 16 hdd and 4 ssd) after deployment the ceph cluster healthy is OK , but > instead of getting 60 disk it appears 48 disks which are hdd only > (according to the output of ceph osd df tree) below are output of ceph osd > df tree and lsblk command. > > hdd: 7.71349 T > ssd: 7 T > > ceph osd df tree > root@ceph-mon3:~# ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS TYPE NAME > -1 370.24731 - 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB > 349 TiB 5.66 1.00 - root default > -5 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 403 MiB > 116 TiB 5.66 1.00 - host ceph-osd1 > 0 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 6 up osd.0 > 3 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 8 up osd.3 > 6 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB > 7.3 TiB 5.66 1.00 12 up osd.6 > 9 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.9 > 12 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.12 > 16 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 3 up osd.16 > 18 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.18 > 21 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 6 up osd.21 > 24 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB > 7.3 TiB 5.66 1.00 7 up osd.24 > 27 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.27 > 30 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.30 > 33 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.33 > 36 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.36 > 39 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 9 up osd.39 > 42 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 6 up osd.42 > 45 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.45 > -3 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 397 MiB > 116 TiB 5.66 1.00 - host ceph-osd2 > 1 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 7 up osd.1 > 5 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 7 up osd.5 > 8 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 6 up osd.8 > 11 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 5 up osd.11 > 14 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.14 > 15 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 12 up osd.15 > 19 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 5 up osd.19 > 22 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 4 up osd.22 > 25 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 2 up osd.25 > 28 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 3 up osd.28 > 31 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 6 up osd.31 > 34 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 7 up osd.34 > 37 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.37 > 40 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.40 > 43 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 6 up osd.43 > 46 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 12 up osd.46 > -7 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 409 MiB > 116 TiB 5.66 1.00 - host ceph-osd3 > 2 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.2 > 4 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 7 up osd.4 > 7 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.7 > 10 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.10 > 13 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 12 up osd.13 > 17 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 6 up osd.17 > 20 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.20 > 23 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 8 up osd.23 > 26 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.26 > 29 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.29 > 32 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.32 > 35 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.35 > 38 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 9 up osd.38 > 41 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 3 up osd.41 > 44 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 4 up osd.44 > 47 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.47 > TOTAL 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB > 349 TiB 5.66 > > lsblk out put > ========== > root@ceph-osd1:~# lsblk > NAME > MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > loop0 > 7:0 0 55.4M 1 loop /snap/core18/1944 > loop1 > 7:1 0 69.9M 1 loop /snap/lxd/19188 > loop2 > 7:2 0 31.1M 1 loop /snap/snapd/10707 > sda > 8:0 0 7.3T 0 disk > └─ceph--f011a10f--3f22--4785--8fe8--1eb4e64d380f-osd--block--1c3eaee6--60a7--4b2d--aff1--dff0a909d48e > 253:1 0 7.3T 0 lvm > sdb > 8:16 0 7.3T 0 disk > └─ceph--ba3fef60--84a2--4cd7--9aeb--4af05a17733d-osd--block--a4926772--4913--4ad1--8c01--348e8cb30159 > 253:3 0 7.3T 0 lvm > sdc > 8:32 0 7.3T 0 disk > └─ceph--14676d53--0a82--47b2--acc2--10ebae5e2cb1-osd--block--b1032109--80e0--4065--923a--80541669c316 > 253:5 0 7.3T 0 lvm > sdd > 8:48 0 7.3T 0 disk > └─ceph--2e7875da--a136--4b39--b354--21cfa8643951-osd--block--5a1ad840--225c--4d79--ab23--8589f13b9f3d > 253:7 0 7.3T 0 lvm > sde > 8:64 0 7T 0 disk > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--be9d5f0f--ac60--42e1--850d--bf2baf913b3b > 253:26 0 447.1G 0 lvm > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--5e7310c4--2b18--4938--ab6e--dc37e860d9d5 > 253:28 0 447.1G 0 lvm > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--d2d78859--2e60--4ccc--b417--d33ff55837b6 > 253:30 0 447.1G 0 lvm > └─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--31ff57f6--e9d3--4bf1--b08d--95939083b4da > 253:32 0 447.1G 0 lvm > sdf > 8:80 0 7T 0 disk > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--7a8d393e--3af3--4c89--a99f--d3efcbb78066 > 253:18 0 447.1G 0 lvm > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--5abd44ca--aed8--48af--8f12--a64a6ae7938c > 253:20 0 447.1G 0 lvm > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--cd5768aa--b9ce--4c3b--9bc8--38c01b5bdc49 > 253:22 0 447.1G 0 lvm > └─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--364e42ff--6e66--4a76--9df2--b2bb04080cd0 > 253:24 0 447.1G 0 lvm > sdg > 8:96 0 7T 0 disk > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--3c469d07--0f1d--4544--861a--dd84283332fe > 253:10 0 447.1G 0 lvm > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--61d08fe1--8d5a--4ef8--ba2d--ba1fc84636db > 253:12 0 447.1G 0 lvm > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--2b2a8af7--33f5--4ca5--b09b--2f24ebf583b3 > 253:14 0 447.1G 0 lvm > └─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--fd2e105a--4fdf--4f15--a48b--1303717b8961 > 253:16 0 447.1G 0 lvm > sdh > 8:112 0 7T 0 disk > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--6e289081--5404--4294--a75a--f4a1a8fabe52 > 253:2 0 447.1G 0 lvm > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--e89285a3--4efa--4239--bbb7--a95064ced7f2 > 253:4 0 447.1G 0 lvm > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--0b7f1fd0--bc32--47b7--b8e4--c6f0f9f21047 > 253:6 0 447.1G 0 lvm > └─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--614e085d--ce04--4f5d--a485--ddca51bae65b > 253:8 0 447.1G 0 lvm > sdi > 8:128 0 7.3T 0 disk > └─ceph--8343ded8--765b--4038--9855--5e6684b55906-osd--block--054a0d83--4aea--4d4f--b623--77db554e207d > 253:9 0 7.3T 0 lvm > sdj > 8:144 0 7.3T 0 disk > └─ceph--88f64697--ab62--4fed--a1b4--88ef3563eaea-osd--block--97189221--a7f6--4d9d--b98a--a474fec95fc7 > 253:11 0 7.3T 0 lvm > sdk > 8:160 0 7.3T 0 disk > └─ceph--592f15db--face--49d8--b028--15fee31d77cd-osd--block--776b773d--347f--4a5c--8de7--66d4a9d56feb > 253:13 0 7.3T 0 lvm > sdl > 8:176 0 7.3T 0 disk > └─ceph--3ef3da28--0aa0--4bfe--80fa--7871a0ebbb2a-osd--block--9b91fccb--3562--4543--916a--6fd0baa00efd > 253:15 0 7.3T 0 lvm > sdm > 8:192 0 7.3T 0 disk > └─ceph--604f22a2--bf89--40a8--bd37--bb5c317f0525-osd--block--ff387140--c528--40e2--ac75--3add53fd5bbb > 253:17 0 7.3T 0 lvm > sdn > 8:208 0 7.3T 0 disk > └─ceph--b702f897--06ad--43fe--b04b--3e03477ffbc3-osd--block--07cc426f--b4c1--4459--a6bc--81ee7a034060 > 253:19 0 7.3T 0 lvm > sdo > 8:224 0 7.3T 0 disk > └─ceph--5e1b4c03--f01b--45d7--971f--a02fe109ef13-osd--block--653b2370--dcb0--46bc--a71b--f62af6777feb > 253:21 0 7.3T 0 lvm > sdp > 8:240 0 7.3T 0 disk > └─ceph--e3ed1f19--15f6--4bd2--81f0--93f9720d7fad-osd--block--4121fcc9--1c00--4da5--b01e--a041280d330a > 253:23 0 7.3T 0 lvm > sdq > 65:0 0 7.3T 0 disk > └─ceph--417cb977--615a--4b86--a4a9--d6af7587b670-osd--block--e77b7eab--7dd8--4300--bd98--27d010c2e17e > 253:25 0 7.3T 0 lvm > sdr > 65:16 0 7.3T 0 disk > └─ceph--ad66d2d6--581d--4985--b945--154586b39903-osd--block--6b1b3de2--8e13--4361--bcd0--16ad609177b0 > 253:27 0 7.3T 0 lvm > sds > 65:32 0 7.3T 0 disk > └─ceph--66aebaac--de6b--42ff--94f1--a489a755e51a-osd--block--a581f3e1--7d86--4f8f--9d0a--de87ef713436 > 253:29 0 7.3T 0 lvm > sdt > 65:48 0 7.3T 0 disk > └─ceph--1558f2d7--741a--4feb--9e68--a900b823eac7-osd--block--52d32f21--9f3a--491e--a26c--41304d63a0ee > 253:31 0 7.3T 0 lvm > sdu > 65:64 0 223.5G 0 disk > ├─sdu1 > 65:65 0 512M 0 part /boot/efi > ├─sdu2 > 65:66 0 1G 0 part /boot > └─sdu3 > 65:67 0 222G 0 part > └─ubuntu--vg-ubuntu--lv > 253:0 0 111G 0 lvm / > > Kindly help me to configure in order to get all disk up and work. > > Best Regards > > Michel > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |