Hello folks! I hope you are doing well :)
I have a general question about XFS and LVM backend OSD performance
and possible effects if they are used together in the same pool.
I built a cluster 5 years ago with Nautilus and I used the XFS backend for
OSD's.
After 5 years they reached me back with ERR state and I see 10K++ slow ops,
29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down
due to compactions problem.
At the first check-up I found some drives were replaced by others and they
used LVM backend for the replaced drives without using wal+db on nvme.
In the cluster I have mostly XFS backend drives wal+db on nvme and some LVM
drives without wal+db.
We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3
workload. RGW index stored on SSD pool.
10 nodes: with
- 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL
- 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool
- 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD
Total HDD raw size 2.8PiB, SSD size 26TiB
I started fixing all the problems one by one and I'm gonna recreate these
LVM drives without wal+db and I wonder 2 questions:
1- Are there any speed or latency differences on XFS and LVM backend OSD's
for 16TB 7200rpm NL-SAS drives.
2- Mixing XFS and LVM backend on the same cluster does have any
negative effect or problems?
Best regards:
Extra note: If you wonder please check the LSBLK output for 1/10 server:
NODE-01# lsblk
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda
8:0 0 14.6T 0 disk
├─sda1
8:1 0 100M 0 part
/var/lib/ceph/osd/ceph-180
└─sda2
8:2 0 14.6T 0 part
sdb
8:16 0 14.6T 0 disk
├─sdb1
8:17 0 100M 0 part
/var/lib/ceph/osd/ceph-181
└─sdb2
8:18 0 14.6T 0 part
sdc
8:32 0 14.6T 0 disk
├─sdc1
8:33 0 100M 0 part
/var/lib/ceph/osd/ceph-182
└─sdc2
8:34 0 14.6T 0 part
sdd
8:48 0 14.6T 0 disk
├─sdd1
8:49 0 100M 0 part
/var/lib/ceph/osd/ceph-183
└─sdd2
8:50 0 14.6T 0 part
sde
8:64 0 14.6T 0 disk
├─sde1
8:65 0 100M 0 part
/var/lib/ceph/osd/ceph-185
└─sde2
8:66 0 14.6T 0 part
sdf
8:80 0 14.6T 0 disk
└─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5
253:2 0 14.6T 0 lvm
sdg
8:96 0 14.6T 0 disk
├─sdg1
8:97 0 100M 0 part
/var/lib/ceph/osd/ceph-186
└─sdg2
8:98 0 14.6T 0 part
sdh
8:112 0 14.6T 0 disk
├─sdh1
8:113 0 100M 0 part
/var/lib/ceph/osd/ceph-187
└─sdh2
8:114 0 14.6T 0 part
sdi
8:128 0 14.6T 0 disk
├─sdi1
8:129 0 100M 0 part
/var/lib/ceph/osd/ceph-188
└─sdi2
8:130 0 14.6T 0 part
sdj
8:144 0 14.6T 0 disk
├─sdj1
8:145 0 100M 0 part
/var/lib/ceph/osd/ceph-189
└─sdj2
8:146 0 14.6T 0 part
sdk
8:160 0 14.6T 0 disk
├─sdk1
8:161 0 100M 0 part
/var/lib/ceph/osd/ceph-190
└─sdk2
8:162 0 14.6T 0 part
sdl
8:176 0 14.6T 0 disk
├─sdl1
8:177 0 100M 0 part
/var/lib/ceph/osd/ceph-191
└─sdl2
8:178 0 14.6T 0 part
sdm
8:192 0 14.6T 0 disk
├─sdm1
8:193 0 100M 0 part
/var/lib/ceph/osd/ceph-192
└─sdm2
8:194 0 14.6T 0 part
sdn
8:208 0 14.6T 0 disk
├─sdn1
8:209 0 100M 0 part
/var/lib/ceph/osd/ceph-193
└─sdn2
8:210 0 14.6T 0 part
sdo
8:224 0 14.6T 0 disk
├─sdo1
8:225 0 100M 0 part
/var/lib/ceph/osd/ceph-194
└─sdo2
8:226 0 14.6T 0 part
sdp
8:240 0 14.6T 0 disk
├─sdp1
8:241 0 100M 0 part
/var/lib/ceph/osd/ceph-195
└─sdp2
8:242 0 14.6T 0 part
sdq
65:0 0 14.6T 0 disk
├─sdq1
65:1 0 100M 0 part
/var/lib/ceph/osd/ceph-196
└─sdq2
65:2 0 14.6T 0 part
sdr
65:16 0 14.6T 0 disk
├─sdr1
65:17 0 100M 0 part
/var/lib/ceph/osd/ceph-197
└─sdr2
65:18 0 14.6T 0 part
sds
65:32 0 14.6T 0 disk
├─sds1
65:33 0 100M 0 part
/var/lib/ceph/osd/ceph-198
└─sds2
65:34 0 14.6T 0 part
sdt
65:48 0 894.3G 0 disk
├─sdt1
65:49 0 100M 0 part
/var/lib/ceph/osd/ceph-199
└─sdt2
65:50 0 894.2G 0 part
sdu
65:64 0 894.3G 0 disk
└─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607
253:3 0 894.3G 0 lvm
sdv
65:80 0 894.3G 0 disk
└─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4
253:1 0 894.3G 0 lvm
sdw
65:96 0 14.6T 0 disk
└─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845
253:0 0 14.6T 0 lvm
sdx
65:112 0 223.5G 0 disk
├─sdx1
65:113 0 1M 0 part
└─sdx2
65:114 0 223.5G 0 part
rbd0
252:0 0 15G 0 disk
rbd1
252:16 0 6T 0 disk
nvme0n1
259:2 0 1.5T 0 disk
└─md0
9:0 0 1.5T 0 raid1
├─md0p1
259:4 0 50G 0 md
├─md0p2
259:5 0 50G 0 md
├─md0p3
259:6 0 50G 0 md
├─md0p4
259:7 0 50G 0 md
├─md0p5
259:8 0 50G 0 md
├─md0p6
259:9 0 50G 0 md
├─md0p7
259:10 0 50G 0 md
├─md0p8
259:11 0 50G 0 md
├─md0p9
259:12 0 50G 0 md
├─md0p10
259:13 0 50G 0 md
├─md0p11
259:14 0 50G 0 md
├─md0p12
259:15 0 50G 0 md
├─md0p13
259:16 0 50G 0 md
├─md0p14
259:17 0 50G 0 md
├─md0p15
259:18 0 50G 0 md
├─md0p16
259:19 0 50G 0 md
├─md0p17
259:20 0 50G 0 md
├─md0p18
259:21 0 50G 0 md
└─md0p19
259:22 0 50G 0 md
nvme1n1
259:3 0 1.5T 0 disk
└─md0
9:0 0 1.5T 0 raid1
├─md0p1
259:4 0 50G 0 md
├─md0p2
259:5 0 50G 0 md
├─md0p3
259:6 0 50G 0 md
├─md0p4
259:7 0 50G 0 md
├─md0p5
259:8 0 50G 0 md
├─md0p6
259:9 0 50G 0 md
├─md0p7
259:10 0 50G 0 md
├─md0p8
259:11 0 50G 0 md
├─md0p9
259:12 0 50G 0 md
├─md0p10
259:13 0 50G 0 md
├─md0p11
259:14 0 50G 0 md
├─md0p12
259:15 0 50G 0 md
├─md0p13
259:16 0 50G 0 md
├─md0p14
259:17 0 50G 0 md
├─md0p15
259:18 0 50G 0 md
├─md0p16
259:19 0 50G 0 md
├─md0p17
259:20 0 50G 0 md
├─md0p18
259:21 0 50G 0 md
└─md0p19
259:22 0 50G 0 md
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
