Re: Missing SSDs disk on ceph deployment

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear team,

Kindly help on this , I am completely blocked.

Best Regards

Michel

On Thu, Jan 5, 2023 at 2:45 PM Michel Niyoyita <micou12@xxxxxxxxx> wrote:

> Dear team,
>
> I have deployed the ceph cluster in production using ceph-ansible on
> ubuntu OS 20.04 it consists of 3 monitors and 3 osds (each osd has 20 disks
> , 16 hdd and 4 ssd) after deployment the ceph cluster healthy is OK , but
> instead of getting 60 disk it appears 48 disks which are hdd only
> (according to the output of ceph osd df tree)  below are output of ceph osd
> df tree and lsblk command.
>
> hdd: 7.71349 T
> ssd: 7 T
>
> ceph osd df tree
> root@ceph-mon3:~# ceph osd df tree
> ID  CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE  DATA     OMAP  META
> AVAIL    %USE  VAR   PGS  STATUS  TYPE NAME
> -1         370.24731         -  370 TiB   21 TiB  249 MiB   0 B  1.2 GiB
>  349 TiB  5.66  1.00    -          root default
> -5         123.41577         -  123 TiB  7.0 TiB   83 MiB   0 B  403 MiB
>  116 TiB  5.66  1.00    -              host ceph-osd1
>  0    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.0
>  3    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    8      up          osd.3
>  6    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   35 MiB
>  7.3 TiB  5.66  1.00   12      up          osd.6
>  9    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    4      up          osd.9
> 12    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.12
> 16    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    3      up          osd.16
> 18    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    5      up          osd.18
> 21    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.21
> 24    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   35 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.24
> 27    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.27
> 30    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.30
> 33    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.33
> 36    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    5      up          osd.36
> 39    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    9      up          osd.39
> 42    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.42
> 45    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.45
> -3         123.41577         -  123 TiB  7.0 TiB   83 MiB   0 B  397 MiB
>  116 TiB  5.66  1.00    -              host ceph-osd2
>  1    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   26 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.1
>  5    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   26 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.5
>  8    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   26 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.8
> 11    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    5      up          osd.11
> 14    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.14
> 15    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   26 MiB
>  7.3 TiB  5.66  1.00   12      up          osd.15
> 19    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    5      up          osd.19
> 22    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   26 MiB
>  7.3 TiB  5.66  1.00    4      up          osd.22
> 25    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    2      up          osd.25
> 28    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    3      up          osd.28
> 31    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.31
> 34    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.34
> 37    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    8      up          osd.37
> 40    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    8      up          osd.40
> 43    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.43
> 46    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00   12      up          osd.46
> -7         123.41577         -  123 TiB  7.0 TiB   83 MiB   0 B  409 MiB
>  116 TiB  5.66  1.00    -              host ceph-osd3
>  2    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    4      up          osd.2
>  4    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.4
>  7    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    5      up          osd.7
> 10    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   21 MiB
>  7.3 TiB  5.66  1.00    4      up          osd.10
> 13    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00   12      up          osd.13
> 17    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   26 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.17
> 20    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    7      up          osd.20
> 23    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    8      up          osd.23
> 26    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    8      up          osd.26
> 29    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.29
> 32    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    6      up          osd.32
> 35    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    8      up          osd.35
> 38    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    9      up          osd.38
> 41    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   20 MiB
>  7.3 TiB  5.66  1.00    3      up          osd.41
> 44    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   25 MiB
>  7.3 TiB  5.66  1.00    4      up          osd.44
> 47    hdd    7.71349   1.00000  7.7 TiB  447 GiB  5.2 MiB   0 B   30 MiB
>  7.3 TiB  5.66  1.00    8      up          osd.47
>                          TOTAL  370 TiB   21 TiB  249 MiB   0 B  1.2 GiB
>  349 TiB  5.66
>
> lsblk out put
> ==========
> root@ceph-osd1:~# lsblk
> NAME
>                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> loop0
>                               7:0    0  55.4M  1 loop /snap/core18/1944
> loop1
>                               7:1    0  69.9M  1 loop /snap/lxd/19188
> loop2
>                               7:2    0  31.1M  1 loop /snap/snapd/10707
> sda
>                               8:0    0   7.3T  0 disk
> └─ceph--f011a10f--3f22--4785--8fe8--1eb4e64d380f-osd--block--1c3eaee6--60a7--4b2d--aff1--dff0a909d48e
> 253:1    0   7.3T  0 lvm
> sdb
>                               8:16   0   7.3T  0 disk
> └─ceph--ba3fef60--84a2--4cd7--9aeb--4af05a17733d-osd--block--a4926772--4913--4ad1--8c01--348e8cb30159
> 253:3    0   7.3T  0 lvm
> sdc
>                               8:32   0   7.3T  0 disk
> └─ceph--14676d53--0a82--47b2--acc2--10ebae5e2cb1-osd--block--b1032109--80e0--4065--923a--80541669c316
> 253:5    0   7.3T  0 lvm
> sdd
>                               8:48   0   7.3T  0 disk
> └─ceph--2e7875da--a136--4b39--b354--21cfa8643951-osd--block--5a1ad840--225c--4d79--ab23--8589f13b9f3d
> 253:7    0   7.3T  0 lvm
> sde
>                               8:64   0     7T  0 disk
> ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--be9d5f0f--ac60--42e1--850d--bf2baf913b3b
>    253:26   0 447.1G  0 lvm
> ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--5e7310c4--2b18--4938--ab6e--dc37e860d9d5
>    253:28   0 447.1G  0 lvm
> ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--d2d78859--2e60--4ccc--b417--d33ff55837b6
>    253:30   0 447.1G  0 lvm
> └─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--31ff57f6--e9d3--4bf1--b08d--95939083b4da
>    253:32   0 447.1G  0 lvm
> sdf
>                               8:80   0     7T  0 disk
> ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--7a8d393e--3af3--4c89--a99f--d3efcbb78066
>    253:18   0 447.1G  0 lvm
> ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--5abd44ca--aed8--48af--8f12--a64a6ae7938c
>    253:20   0 447.1G  0 lvm
> ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--cd5768aa--b9ce--4c3b--9bc8--38c01b5bdc49
>    253:22   0 447.1G  0 lvm
> └─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--364e42ff--6e66--4a76--9df2--b2bb04080cd0
>    253:24   0 447.1G  0 lvm
> sdg
>                               8:96   0     7T  0 disk
> ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--3c469d07--0f1d--4544--861a--dd84283332fe
>    253:10   0 447.1G  0 lvm
> ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--61d08fe1--8d5a--4ef8--ba2d--ba1fc84636db
>    253:12   0 447.1G  0 lvm
> ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--2b2a8af7--33f5--4ca5--b09b--2f24ebf583b3
>    253:14   0 447.1G  0 lvm
> └─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--fd2e105a--4fdf--4f15--a48b--1303717b8961
>    253:16   0 447.1G  0 lvm
> sdh
>                               8:112  0     7T  0 disk
> ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--6e289081--5404--4294--a75a--f4a1a8fabe52
>    253:2    0 447.1G  0 lvm
> ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--e89285a3--4efa--4239--bbb7--a95064ced7f2
>    253:4    0 447.1G  0 lvm
> ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--0b7f1fd0--bc32--47b7--b8e4--c6f0f9f21047
>    253:6    0 447.1G  0 lvm
> └─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--614e085d--ce04--4f5d--a485--ddca51bae65b
>    253:8    0 447.1G  0 lvm
> sdi
>                               8:128  0   7.3T  0 disk
> └─ceph--8343ded8--765b--4038--9855--5e6684b55906-osd--block--054a0d83--4aea--4d4f--b623--77db554e207d
> 253:9    0   7.3T  0 lvm
> sdj
>                               8:144  0   7.3T  0 disk
> └─ceph--88f64697--ab62--4fed--a1b4--88ef3563eaea-osd--block--97189221--a7f6--4d9d--b98a--a474fec95fc7
> 253:11   0   7.3T  0 lvm
> sdk
>                               8:160  0   7.3T  0 disk
> └─ceph--592f15db--face--49d8--b028--15fee31d77cd-osd--block--776b773d--347f--4a5c--8de7--66d4a9d56feb
> 253:13   0   7.3T  0 lvm
> sdl
>                               8:176  0   7.3T  0 disk
> └─ceph--3ef3da28--0aa0--4bfe--80fa--7871a0ebbb2a-osd--block--9b91fccb--3562--4543--916a--6fd0baa00efd
> 253:15   0   7.3T  0 lvm
> sdm
>                               8:192  0   7.3T  0 disk
> └─ceph--604f22a2--bf89--40a8--bd37--bb5c317f0525-osd--block--ff387140--c528--40e2--ac75--3add53fd5bbb
> 253:17   0   7.3T  0 lvm
> sdn
>                               8:208  0   7.3T  0 disk
> └─ceph--b702f897--06ad--43fe--b04b--3e03477ffbc3-osd--block--07cc426f--b4c1--4459--a6bc--81ee7a034060
> 253:19   0   7.3T  0 lvm
> sdo
>                               8:224  0   7.3T  0 disk
> └─ceph--5e1b4c03--f01b--45d7--971f--a02fe109ef13-osd--block--653b2370--dcb0--46bc--a71b--f62af6777feb
> 253:21   0   7.3T  0 lvm
> sdp
>                               8:240  0   7.3T  0 disk
> └─ceph--e3ed1f19--15f6--4bd2--81f0--93f9720d7fad-osd--block--4121fcc9--1c00--4da5--b01e--a041280d330a
> 253:23   0   7.3T  0 lvm
> sdq
>                              65:0    0   7.3T  0 disk
> └─ceph--417cb977--615a--4b86--a4a9--d6af7587b670-osd--block--e77b7eab--7dd8--4300--bd98--27d010c2e17e
> 253:25   0   7.3T  0 lvm
> sdr
>                              65:16   0   7.3T  0 disk
> └─ceph--ad66d2d6--581d--4985--b945--154586b39903-osd--block--6b1b3de2--8e13--4361--bcd0--16ad609177b0
> 253:27   0   7.3T  0 lvm
> sds
>                              65:32   0   7.3T  0 disk
> └─ceph--66aebaac--de6b--42ff--94f1--a489a755e51a-osd--block--a581f3e1--7d86--4f8f--9d0a--de87ef713436
> 253:29   0   7.3T  0 lvm
> sdt
>                              65:48   0   7.3T  0 disk
> └─ceph--1558f2d7--741a--4feb--9e68--a900b823eac7-osd--block--52d32f21--9f3a--491e--a26c--41304d63a0ee
> 253:31   0   7.3T  0 lvm
> sdu
>                              65:64   0 223.5G  0 disk
> ├─sdu1
>                             65:65   0   512M  0 part /boot/efi
> ├─sdu2
>                             65:66   0     1G  0 part /boot
> └─sdu3
>                             65:67   0   222G  0 part
>   └─ubuntu--vg-ubuntu--lv
>                             253:0    0   111G  0 lvm  /
>
> Kindly help me to configure in order to get all disk up and work.
>
> Best Regards
>
> Michel
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux