BlueStore vs Filestore doesn’t matter beyond each OSD. Filestore is very deprecated so you’ll want to redeploy any Filestore OSDs when you can. `ceph osd metadata` can survey. I’ve had multiple issues over time with the MG spinners fwiw. For what SAS spinners cost with some effort you can get SATA SSDs for legacy systems. Do you have those NVMe drives for WAL+DB mirrored? You generally don’t want to go higher than 10:1 > On Sep 30, 2024, at 3:00 PM, Özkan Göksu <ozkangksu@xxxxxxxxx> wrote: > > Hello folks! I hope you are doing well :) > > I have a general question about XFS and LVM backend OSD performance > and possible effects if they are used together in the same pool. > > I built a cluster 5 years ago with Nautilus and I used the XFS backend for > OSD's. > After 5 years they reached me back with ERR state and I see 10K++ slow ops, > 29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down > due to compactions problem. > > At the first check-up I found some drives were replaced by others and they > used LVM backend for the replaced drives without using wal+db on nvme. > In the cluster I have mostly XFS backend drives wal+db on nvme and some LVM > drives without wal+db. > > We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3 > workload. RGW index stored on SSD pool. > 10 nodes: with > - 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL > - 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool > - 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD > > Total HDD raw size 2.8PiB, SSD size 26TiB > > I started fixing all the problems one by one and I'm gonna recreate these > LVM drives without wal+db and I wonder 2 questions: > 1- Are there any speed or latency differences on XFS and LVM backend OSD's > for 16TB 7200rpm NL-SAS drives. > 2- Mixing XFS and LVM backend on the same cluster does have any > negative effect or problems? > > Best regards: > > Extra note: If you wonder please check the LSBLK output for 1/10 server: > NODE-01# lsblk > NAME > MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda > 8:0 0 14.6T 0 disk > ├─sda1 > 8:1 0 100M 0 part > /var/lib/ceph/osd/ceph-180 > └─sda2 > 8:2 0 14.6T 0 part > sdb > 8:16 0 14.6T 0 disk > ├─sdb1 > 8:17 0 100M 0 part > /var/lib/ceph/osd/ceph-181 > └─sdb2 > 8:18 0 14.6T 0 part > sdc > 8:32 0 14.6T 0 disk > ├─sdc1 > 8:33 0 100M 0 part > /var/lib/ceph/osd/ceph-182 > └─sdc2 > 8:34 0 14.6T 0 part > sdd > 8:48 0 14.6T 0 disk > ├─sdd1 > 8:49 0 100M 0 part > /var/lib/ceph/osd/ceph-183 > └─sdd2 > 8:50 0 14.6T 0 part > sde > 8:64 0 14.6T 0 disk > ├─sde1 > 8:65 0 100M 0 part > /var/lib/ceph/osd/ceph-185 > └─sde2 > 8:66 0 14.6T 0 part > sdf > 8:80 0 14.6T 0 disk > └─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5 > 253:2 0 14.6T 0 lvm > sdg > 8:96 0 14.6T 0 disk > ├─sdg1 > 8:97 0 100M 0 part > /var/lib/ceph/osd/ceph-186 > └─sdg2 > 8:98 0 14.6T 0 part > sdh > 8:112 0 14.6T 0 disk > ├─sdh1 > 8:113 0 100M 0 part > /var/lib/ceph/osd/ceph-187 > └─sdh2 > 8:114 0 14.6T 0 part > sdi > 8:128 0 14.6T 0 disk > ├─sdi1 > 8:129 0 100M 0 part > /var/lib/ceph/osd/ceph-188 > └─sdi2 > 8:130 0 14.6T 0 part > sdj > 8:144 0 14.6T 0 disk > ├─sdj1 > 8:145 0 100M 0 part > /var/lib/ceph/osd/ceph-189 > └─sdj2 > 8:146 0 14.6T 0 part > sdk > 8:160 0 14.6T 0 disk > ├─sdk1 > 8:161 0 100M 0 part > /var/lib/ceph/osd/ceph-190 > └─sdk2 > 8:162 0 14.6T 0 part > sdl > 8:176 0 14.6T 0 disk > ├─sdl1 > 8:177 0 100M 0 part > /var/lib/ceph/osd/ceph-191 > └─sdl2 > 8:178 0 14.6T 0 part > sdm > 8:192 0 14.6T 0 disk > ├─sdm1 > 8:193 0 100M 0 part > /var/lib/ceph/osd/ceph-192 > └─sdm2 > 8:194 0 14.6T 0 part > sdn > 8:208 0 14.6T 0 disk > ├─sdn1 > 8:209 0 100M 0 part > /var/lib/ceph/osd/ceph-193 > └─sdn2 > 8:210 0 14.6T 0 part > sdo > 8:224 0 14.6T 0 disk > ├─sdo1 > 8:225 0 100M 0 part > /var/lib/ceph/osd/ceph-194 > └─sdo2 > 8:226 0 14.6T 0 part > sdp > 8:240 0 14.6T 0 disk > ├─sdp1 > 8:241 0 100M 0 part > /var/lib/ceph/osd/ceph-195 > └─sdp2 > 8:242 0 14.6T 0 part > sdq > 65:0 0 14.6T 0 disk > ├─sdq1 > 65:1 0 100M 0 part > /var/lib/ceph/osd/ceph-196 > └─sdq2 > 65:2 0 14.6T 0 part > sdr > 65:16 0 14.6T 0 disk > ├─sdr1 > 65:17 0 100M 0 part > /var/lib/ceph/osd/ceph-197 > └─sdr2 > 65:18 0 14.6T 0 part > sds > 65:32 0 14.6T 0 disk > ├─sds1 > 65:33 0 100M 0 part > /var/lib/ceph/osd/ceph-198 > └─sds2 > 65:34 0 14.6T 0 part > sdt > 65:48 0 894.3G 0 disk > ├─sdt1 > 65:49 0 100M 0 part > /var/lib/ceph/osd/ceph-199 > └─sdt2 > 65:50 0 894.2G 0 part > sdu > 65:64 0 894.3G 0 disk > └─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607 > 253:3 0 894.3G 0 lvm > sdv > 65:80 0 894.3G 0 disk > └─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4 > 253:1 0 894.3G 0 lvm > sdw > 65:96 0 14.6T 0 disk > └─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845 > 253:0 0 14.6T 0 lvm > sdx > 65:112 0 223.5G 0 disk > ├─sdx1 > 65:113 0 1M 0 part > └─sdx2 > 65:114 0 223.5G 0 part > rbd0 > 252:0 0 15G 0 disk > rbd1 > 252:16 0 6T 0 disk > nvme0n1 > 259:2 0 1.5T 0 disk > └─md0 > 9:0 0 1.5T 0 raid1 > ├─md0p1 > 259:4 0 50G 0 md > ├─md0p2 > 259:5 0 50G 0 md > ├─md0p3 > 259:6 0 50G 0 md > ├─md0p4 > 259:7 0 50G 0 md > ├─md0p5 > 259:8 0 50G 0 md > ├─md0p6 > 259:9 0 50G 0 md > ├─md0p7 > 259:10 0 50G 0 md > ├─md0p8 > 259:11 0 50G 0 md > ├─md0p9 > 259:12 0 50G 0 md > ├─md0p10 > 259:13 0 50G 0 md > ├─md0p11 > 259:14 0 50G 0 md > ├─md0p12 > 259:15 0 50G 0 md > ├─md0p13 > 259:16 0 50G 0 md > ├─md0p14 > 259:17 0 50G 0 md > ├─md0p15 > 259:18 0 50G 0 md > ├─md0p16 > 259:19 0 50G 0 md > ├─md0p17 > 259:20 0 50G 0 md > ├─md0p18 > 259:21 0 50G 0 md > └─md0p19 > 259:22 0 50G 0 md > nvme1n1 > 259:3 0 1.5T 0 disk > └─md0 > 9:0 0 1.5T 0 raid1 > ├─md0p1 > 259:4 0 50G 0 md > ├─md0p2 > 259:5 0 50G 0 md > ├─md0p3 > 259:6 0 50G 0 md > ├─md0p4 > 259:7 0 50G 0 md > ├─md0p5 > 259:8 0 50G 0 md > ├─md0p6 > 259:9 0 50G 0 md > ├─md0p7 > 259:10 0 50G 0 md > ├─md0p8 > 259:11 0 50G 0 md > ├─md0p9 > 259:12 0 50G 0 md > ├─md0p10 > 259:13 0 50G 0 md > ├─md0p11 > 259:14 0 50G 0 md > ├─md0p12 > 259:15 0 50G 0 md > ├─md0p13 > 259:16 0 50G 0 md > ├─md0p14 > 259:17 0 50G 0 md > ├─md0p15 > 259:18 0 50G 0 md > ├─md0p16 > 259:19 0 50G 0 md > ├─md0p17 > 259:20 0 50G 0 md > ├─md0p18 > 259:21 0 50G 0 md > └─md0p19 > 259:22 0 50G 0 md > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |