Hello Anthony. Good to hear from you. You miss understood somehow, I don't use filestore, I have XFS based bluestore and LVM based bluestore mixed in the same Nautilus cluster. Yes NVME's are mdadm raid1. I'm %100 agree with you and when I'm gonna rebuild, I will pin 1/2 to Nvme1 and 2/2 to Nvme2 without mirror. I build this system with raid1 nvme because it was one time project and I wanted to be safe rather than fast. Anthony D'Atri <anthony.datri@xxxxxxxxx>, 30 Eyl 2024 Pzt, 22:10 tarihinde şunu yazdı: > BlueStore vs Filestore doesn’t matter beyond each OSD. Filestore is very > deprecated so you’ll want to redeploy any Filestore OSDs when you can. > `ceph osd metadata` can survey. > > I’ve had multiple issues over time with the MG spinners fwiw. For what > SAS spinners cost with some effort you can get SATA SSDs for legacy > systems. Do you have those NVMe drives for WAL+DB mirrored? You > generally don’t want to go higher than 10:1 > > > On Sep 30, 2024, at 3:00 PM, Özkan Göksu <ozkangksu@xxxxxxxxx> wrote: > > > > Hello folks! I hope you are doing well :) > > > > I have a general question about XFS and LVM backend OSD performance > > and possible effects if they are used together in the same pool. > > > > I built a cluster 5 years ago with Nautilus and I used the XFS backend > for > > OSD's. > > After 5 years they reached me back with ERR state and I see 10K++ slow > ops, > > 29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down > > due to compactions problem. > > > > At the first check-up I found some drives were replaced by others and > they > > used LVM backend for the replaced drives without using wal+db on nvme. > > In the cluster I have mostly XFS backend drives wal+db on nvme and some > LVM > > drives without wal+db. > > > > We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3 > > workload. RGW index stored on SSD pool. > > 10 nodes: with > > - 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL > > - 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool > > - 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD > > > > Total HDD raw size 2.8PiB, SSD size 26TiB > > > > I started fixing all the problems one by one and I'm gonna recreate these > > LVM drives without wal+db and I wonder 2 questions: > > 1- Are there any speed or latency differences on XFS and LVM backend > OSD's > > for 16TB 7200rpm NL-SAS drives. > > 2- Mixing XFS and LVM backend on the same cluster does have any > > negative effect or problems? > > > > Best regards: > > > > Extra note: If you wonder please check the LSBLK output for 1/10 server: > > NODE-01# lsblk > > NAME > > MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > sda > > 8:0 0 14.6T 0 disk > > ├─sda1 > > 8:1 0 100M 0 part > > /var/lib/ceph/osd/ceph-180 > > └─sda2 > > 8:2 0 14.6T 0 part > > sdb > > 8:16 0 14.6T 0 disk > > ├─sdb1 > > 8:17 0 100M 0 part > > /var/lib/ceph/osd/ceph-181 > > └─sdb2 > > 8:18 0 14.6T 0 part > > sdc > > 8:32 0 14.6T 0 disk > > ├─sdc1 > > 8:33 0 100M 0 part > > /var/lib/ceph/osd/ceph-182 > > └─sdc2 > > 8:34 0 14.6T 0 part > > sdd > > 8:48 0 14.6T 0 disk > > ├─sdd1 > > 8:49 0 100M 0 part > > /var/lib/ceph/osd/ceph-183 > > └─sdd2 > > 8:50 0 14.6T 0 part > > sde > > 8:64 0 14.6T 0 disk > > ├─sde1 > > 8:65 0 100M 0 part > > /var/lib/ceph/osd/ceph-185 > > └─sde2 > > 8:66 0 14.6T 0 part > > sdf > > 8:80 0 14.6T 0 disk > > > └─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5 > > 253:2 0 14.6T 0 lvm > > sdg > > 8:96 0 14.6T 0 disk > > ├─sdg1 > > 8:97 0 100M 0 part > > /var/lib/ceph/osd/ceph-186 > > └─sdg2 > > 8:98 0 14.6T 0 part > > sdh > > 8:112 0 14.6T 0 disk > > ├─sdh1 > > 8:113 0 100M 0 part > > /var/lib/ceph/osd/ceph-187 > > └─sdh2 > > 8:114 0 14.6T 0 part > > sdi > > 8:128 0 14.6T 0 disk > > ├─sdi1 > > 8:129 0 100M 0 part > > /var/lib/ceph/osd/ceph-188 > > └─sdi2 > > 8:130 0 14.6T 0 part > > sdj > > 8:144 0 14.6T 0 disk > > ├─sdj1 > > 8:145 0 100M 0 part > > /var/lib/ceph/osd/ceph-189 > > └─sdj2 > > 8:146 0 14.6T 0 part > > sdk > > 8:160 0 14.6T 0 disk > > ├─sdk1 > > 8:161 0 100M 0 part > > /var/lib/ceph/osd/ceph-190 > > └─sdk2 > > 8:162 0 14.6T 0 part > > sdl > > 8:176 0 14.6T 0 disk > > ├─sdl1 > > 8:177 0 100M 0 part > > /var/lib/ceph/osd/ceph-191 > > └─sdl2 > > 8:178 0 14.6T 0 part > > sdm > > 8:192 0 14.6T 0 disk > > ├─sdm1 > > 8:193 0 100M 0 part > > /var/lib/ceph/osd/ceph-192 > > └─sdm2 > > 8:194 0 14.6T 0 part > > sdn > > 8:208 0 14.6T 0 disk > > ├─sdn1 > > 8:209 0 100M 0 part > > /var/lib/ceph/osd/ceph-193 > > └─sdn2 > > 8:210 0 14.6T 0 part > > sdo > > 8:224 0 14.6T 0 disk > > ├─sdo1 > > 8:225 0 100M 0 part > > /var/lib/ceph/osd/ceph-194 > > └─sdo2 > > 8:226 0 14.6T 0 part > > sdp > > 8:240 0 14.6T 0 disk > > ├─sdp1 > > 8:241 0 100M 0 part > > /var/lib/ceph/osd/ceph-195 > > └─sdp2 > > 8:242 0 14.6T 0 part > > sdq > > 65:0 0 14.6T 0 disk > > ├─sdq1 > > 65:1 0 100M 0 part > > /var/lib/ceph/osd/ceph-196 > > └─sdq2 > > 65:2 0 14.6T 0 part > > sdr > > 65:16 0 14.6T 0 disk > > ├─sdr1 > > 65:17 0 100M 0 part > > /var/lib/ceph/osd/ceph-197 > > └─sdr2 > > 65:18 0 14.6T 0 part > > sds > > 65:32 0 14.6T 0 disk > > ├─sds1 > > 65:33 0 100M 0 part > > /var/lib/ceph/osd/ceph-198 > > └─sds2 > > 65:34 0 14.6T 0 part > > sdt > > 65:48 0 894.3G 0 disk > > ├─sdt1 > > 65:49 0 100M 0 part > > /var/lib/ceph/osd/ceph-199 > > └─sdt2 > > 65:50 0 894.2G 0 part > > sdu > > 65:64 0 894.3G 0 disk > > > └─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607 > > 253:3 0 894.3G 0 lvm > > sdv > > 65:80 0 894.3G 0 disk > > > └─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4 > > 253:1 0 894.3G 0 lvm > > sdw > > 65:96 0 14.6T 0 disk > > > └─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845 > > 253:0 0 14.6T 0 lvm > > sdx > > 65:112 0 223.5G 0 disk > > ├─sdx1 > > 65:113 0 1M 0 part > > └─sdx2 > > 65:114 0 223.5G 0 part > > rbd0 > > 252:0 0 15G 0 disk > > rbd1 > > 252:16 0 6T 0 disk > > nvme0n1 > > 259:2 0 1.5T 0 disk > > └─md0 > > 9:0 0 1.5T 0 raid1 > > ├─md0p1 > > 259:4 0 50G 0 md > > ├─md0p2 > > 259:5 0 50G 0 md > > ├─md0p3 > > 259:6 0 50G 0 md > > ├─md0p4 > > 259:7 0 50G 0 md > > ├─md0p5 > > 259:8 0 50G 0 md > > ├─md0p6 > > 259:9 0 50G 0 md > > ├─md0p7 > > 259:10 0 50G 0 md > > ├─md0p8 > > 259:11 0 50G 0 md > > ├─md0p9 > > 259:12 0 50G 0 md > > ├─md0p10 > > 259:13 0 50G 0 md > > ├─md0p11 > > 259:14 0 50G 0 md > > ├─md0p12 > > 259:15 0 50G 0 md > > ├─md0p13 > > 259:16 0 50G 0 md > > ├─md0p14 > > 259:17 0 50G 0 md > > ├─md0p15 > > 259:18 0 50G 0 md > > ├─md0p16 > > 259:19 0 50G 0 md > > ├─md0p17 > > 259:20 0 50G 0 md > > ├─md0p18 > > 259:21 0 50G 0 md > > └─md0p19 > > 259:22 0 50G 0 md > > nvme1n1 > > 259:3 0 1.5T 0 disk > > └─md0 > > 9:0 0 1.5T 0 raid1 > > ├─md0p1 > > 259:4 0 50G 0 md > > ├─md0p2 > > 259:5 0 50G 0 md > > ├─md0p3 > > 259:6 0 50G 0 md > > ├─md0p4 > > 259:7 0 50G 0 md > > ├─md0p5 > > 259:8 0 50G 0 md > > ├─md0p6 > > 259:9 0 50G 0 md > > ├─md0p7 > > 259:10 0 50G 0 md > > ├─md0p8 > > 259:11 0 50G 0 md > > ├─md0p9 > > 259:12 0 50G 0 md > > ├─md0p10 > > 259:13 0 50G 0 md > > ├─md0p11 > > 259:14 0 50G 0 md > > ├─md0p12 > > 259:15 0 50G 0 md > > ├─md0p13 > > 259:16 0 50G 0 md > > ├─md0p14 > > 259:17 0 50G 0 md > > ├─md0p15 > > 259:18 0 50G 0 md > > ├─md0p16 > > 259:19 0 50G 0 md > > ├─md0p17 > > 259:20 0 50G 0 md > > ├─md0p18 > > 259:21 0 50G 0 md > > └─md0p19 > > 259:22 0 50G 0 md > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |