On 26.05.2021 18:12, Eugen Block wrote:
Could you share the output of
lsblk -o name,rota,size,type
from the affected osd node?
# lsblk -o name,rota,size,type
NAME
ROTA SIZE TYPE
loop0
1 71.3M loop
loop1
1 55M loop
loop2
1 29.9M loop
sda
1 223G disk
├─sda1
1 512M part
├─sda2
1 1G part
└─sda3
1 221.5G part
sdb
1 12.5T disk
└─ceph--block--1b5ad7e7--2e24--4315--8a05--7439ab782b45-osd--block--2da790bc--a74c--41da--8772--3b8aac77001c
1 12.5T lvm
sdc
1 12.5T disk
└─ceph--block--44ae73e8--726f--4556--978c--8e7d6570c867-osd--block--daeb5218--c10c--45e1--a864--2d60de44e594
1 12.5T lvm
sdd
1 12.5T disk
└─ceph--block--38e361f5--257f--47a5--85dc--16dbdd5fb905-osd--block--a3e1511f--8644--4c1e--a3dd--f365fcb27fc6
1 12.5T lvm
sde
0 1.8T disk
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--f5d822a2--d42a--4a2c--985f--c65977f4d020
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--9ed8d8a4--11bc--4669--bb8f--d66806181a7d
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--0fdff890--f5e4--4762--83f6--8d02ee63c399
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--facaa59d--1f44--4605--bdb8--5a7d58271323
0 357.7G lvm
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--44fbc70f--4169--41aa--9c2f--6638b226065a
0 357.7G lvm
sdf
0 1.8T disk
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--7c82d5e2--6ce1--49c6--9691--2e156a8fd9c0
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--135169cd--e0c8--4949--a137--c5e7da12bc52
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--fcb5b6cc--ee95--4dfa--a978--0a768da8bc66
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--44b74f22--d2f1--485a--971b--d89a82849c6e
0 357.7G lvm
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--1dd5ee57--e141--40aa--b4a6--5cb20dc51cc0
0 357.7G lvm
sdg
0 1.8T disk
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--274c882c--da8b--4251--a195--309ee9cbc36f
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--087b2305--2bbe--4583--b1ca--dda7416efefc
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--3a97a94b--92bd--41d9--808a--b79aa210fe11
0 357.7G lvm
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--441521b4--52ce--4a03--b5b2--da9392d037bc
0 357.7G lvm
sdi
1 12.5T disk
└─ceph--block--1c122cd0--b28a--4409--b17b--535d95029dda-osd--block--9a6456d9--4c23--4585--a6df--85eda27ae651
1 12.5T lvm
sdj
1 12.5T disk
└─ceph--block--4cbe20eb--6aa3--4e4a--bdda--d556144e2a83-osd--block--2fd1bc6c--0eb3--48aa--b495--db0423b5be28
1 12.5T lvm
sdk
1 12.5T disk
└─ceph--block--3a5c3abd--77f6--4823--b313--1f5900470b8f-osd--block--c350620a--32f5--4da1--bda2--423fe3644f17
1 12.5T lvm
sdl
1 12.5T disk
└─ceph--block--6c7d3bea--a19c--4efc--9a36--dc38a06dad9c-osd--block--f23293d3--4ef8--4127--92c3--7235055797df
1 12.5T lvm
sdm
1 12.5T disk
└─ceph--block--7b48b8d4--cca2--4600--a046--7e1bbc03e1d7-osd--block--bfe62b63--f4fc--4f9d--9bc3--028cba0bacc6
1 12.5T lvm
sdn
1 12.5T disk
└─ceph--block--b99edf97--4162--43e6--ae2e--143250e4c4a8-osd--block--c6cd1240--a171--4ac8--8db4--e282d6fd63c3
1 12.5T lvm
sdo
1 12.5T disk
└─ceph--block--5c324c26--f569--4073--8915--b242a6b56e1c-osd--block--9ab9a615--a87b--40b3--90f1--660fa0da5c93
1 12.5T lvm
sdp
1 12.5T disk
└─ceph--block--2ae62d0b--5f71--446d--b1a9--083842227895-osd--block--6f80035f--59b3--4740--9504--c63be09201f6
1 12.5T lvm
sdq
1 12.5T disk
└─ceph--block--76711aee--28a0--4e19--a4da--74492ab56b77-osd--block--f587fb82--30b9--4133--a000--1b7f649c92aa
1 12.5T lvm
sdr
1 12.5T disk
└─ceph--block--bd6f560d--f196--448b--8bc0--840ea381e798-osd--block--37cc1a13--9f4b--4593--85ef--829b33f4c82f
1 12.5T lvm
sds
1 12.5T disk
└─ceph--block--f6826b43--6594--4b1c--ba0d--654d3075656a-osd--block--16b26a2d--8a7b--4c60--abf0--5412d8da1446
1 12.5T lvm
sdt
1 12.5T disk
My spec file is for a tiny lab cluster, in your case the db drive size
should be something like '5T:6T' to specify a range.
How large are the HDDs?
The VG has 357.74GB of free space of total 5.24TB so I did actually
tried different values like "30G:", "30G", "300G:", "300G", "357G".
I also tied some crazy high numbers and some ranges, but don't remember
the values. But none of them worked.
The HDD is 15x14TB and SSD 3x1.92TB.
As you can see from lsblk or pvs under, Cephadm created one VG
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b from the 3 SSD and
created 15 LV on that VG, one for each HDD.
It only has 14 LV now since I zap the LV of the disk that died, the new
disk is /dev/sdt.
Here is also the output of pvs
# pvs
PV VG Fmt
Attr PSize PFree
/dev/sdb ceph-block-1b5ad7e7-2e24-4315-8a05-7439ab782b45 lvm2
a-- 12.47t 0
/dev/sdc ceph-block-44ae73e8-726f-4556-978c-8e7d6570c867 lvm2
a-- 12.47t 0
/dev/sdd ceph-block-38e361f5-257f-47a5-85dc-16dbdd5fb905 lvm2
a-- 12.47t 0
/dev/sde ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b lvm2
a-- <1.75t 16.00m
/dev/sdf ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b lvm2
a-- <1.75t 16.00m
/dev/sdg ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b lvm2
a-- <1.75t 357.71g
/dev/sdi ceph-block-1c122cd0-b28a-4409-b17b-535d95029dda lvm2
a-- 12.47t 0
/dev/sdj ceph-block-4cbe20eb-6aa3-4e4a-bdda-d556144e2a83 lvm2
a-- 12.47t 0
/dev/sdk ceph-block-3a5c3abd-77f6-4823-b313-1f5900470b8f lvm2
a-- 12.47t 0
/dev/sdl ceph-block-6c7d3bea-a19c-4efc-9a36-dc38a06dad9c lvm2
a-- 12.47t 0
/dev/sdm ceph-block-7b48b8d4-cca2-4600-a046-7e1bbc03e1d7 lvm2
a-- 12.47t 0
/dev/sdn ceph-block-b99edf97-4162-43e6-ae2e-143250e4c4a8 lvm2
a-- 12.47t 0
/dev/sdo ceph-block-5c324c26-f569-4073-8915-b242a6b56e1c lvm2
a-- 12.47t 0
/dev/sdp ceph-block-2ae62d0b-5f71-446d-b1a9-083842227895 lvm2
a-- 12.47t 0
/dev/sdq ceph-block-76711aee-28a0-4e19-a4da-74492ab56b77 lvm2
a-- 12.47t 0
/dev/sdr ceph-block-bd6f560d-f196-448b-8bc0-840ea381e798 lvm2
a-- 12.47t 0
/dev/sds ceph-block-f6826b43-6594-4b1c-ba0d-654d3075656a lvm2
a-- 12.47t 0
Also maybe you should use the option
'filter_logic: AND', but I'm not sure if that's already the default, I
remember that there were issues in Nautilus because the default was
OR. I tried this just recently with a version similar to this, I
believe it was 15.2.8 and it worked for me, but again, it's just a
tiny virtual lab cluster.
Yes, AND is default, I tried adding 'filter_logic: AND' but with the
same result.
In you virtual lab cluster do you have multiple HDD sharing the same SSD
as I do?
To me it looks like Cephadm can't find or use the 357.71GB free space on
the VG, it can only find devices that is available.
Here is how my "orch device ls" is for that host
$ ceph orch device ls --wide | egrep "Hostname|hd-7"
Hostname Path Type Vendor Model Size Available
Reject Reasons
pech-hd-7 /dev/sdt hdd WDC WUH721414AL5200 13.7T Yes
pech-hd-7 /dev/sdb hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdc hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdd hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sde ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
LVM detected, locked
pech-hd-7 /dev/sdf ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
LVM detected, locked
pech-hd-7 /dev/sdg ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
LVM detected, locked
pech-hd-7 /dev/sdi hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdj hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdk hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdl hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdm hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdn hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdo hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdp hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdq hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdr hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sds hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
--
Kai Stian Olstad
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
