I found some seemingly related information in https://access.redhat.com/solutions/5173092. However, these bugs were fixed in luminous and I'm observing this on mimic. Therefore, I wonder if I hit an undiscovered issue with bluestore. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@xxxxxx> Sent: 10 January 2022 12:25:57 To: ceph-users Subject: OSD META usage growing without bounds Hi, I'm observing a strange behaviour on a small test cluster (13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable)). The cluster is up for about half a year and almost empty. We did a few rbd bench runs and created a file system, but there was zero client IO for at least 3 months. It looks like recently the OSD META usage of some OSDs started to increase for no apparent reason. One OSD already died with 100% usage and another is on its way. I can't see any obvious reason for this strange behaviour. If anyone has an idea, please let me know. Some diagnostic output: [root@rit-tceph ~]# ceph status cluster: id: bf1f51f5-b381-4cf7-b3db-88d044c1960c health: HEALTH_WARN 1 nearfull osd(s) 3 pool(s) nearfull services: mon: 3 daemons, quorum tceph-01,tceph-02,tceph-03 mgr: tceph-01(active), standbys: tceph-02, tceph-03 mds: testfs-1/1/1 up {0=tceph-01=up:active}, 2 up:standby osd: 9 osds: 8 up, 8 in data: pools: 3 pools, 500 pgs objects: 24 objects, 2.3 KiB usage: 746 GiB used, 1.4 TiB / 2.2 TiB avail pgs: 500 active+clean [root@rit-tceph ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.2 TiB 1.4 TiB 746 GiB 33.49 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS test 1 19 B 0 81 GiB 2 testfs_data 2 0 B 0 81 GiB 0 testfs_metadata 3 2.2 KiB 0 81 GiB 22 [root@rit-tceph ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 2.44707 - 2.2 TiB 746 GiB 120 MiB 34 KiB 746 GiB 1.4 TiB 33.49 1.00 - root default -5 0.81569 - 557 GiB 195 GiB 30 MiB 3 KiB 195 GiB 362 GiB 35.04 1.05 - host tceph-01 0 hdd 0.27190 1.00000 278 GiB 38 GiB 15 MiB 3 KiB 38 GiB 241 GiB 13.61 0.41 260 osd.0 3 hdd 0.27190 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.3 8 hdd 0.27190 1.00000 278 GiB 157 GiB 15 MiB 0 B 157 GiB 121 GiB 56.47 1.69 240 osd.8 -3 0.81569 - 835 GiB 113 GiB 45 MiB 3 KiB 113 GiB 723 GiB 13.48 0.40 - host tceph-02 2 hdd 0.27190 1.00000 278 GiB 18 GiB 15 MiB 0 B 18 GiB 261 GiB 6.30 0.19 157 osd.2 4 hdd 0.27190 1.00000 278 GiB 48 GiB 15 MiB 0 B 48 GiB 231 GiB 17.21 0.51 172 osd.4 6 hdd 0.27190 1.00000 278 GiB 47 GiB 15 MiB 3 KiB 47 GiB 231 GiB 16.93 0.51 171 osd.6 -7 0.81569 - 835 GiB 438 GiB 45 MiB 28 KiB 438 GiB 397 GiB 52.48 1.57 - host tceph-03 1 hdd 0.27190 1.00000 278 GiB 238 GiB 15 MiB 25 KiB 238 GiB 41 GiB 85.35 2.55 171 osd.1 5 hdd 0.27190 1.00000 278 GiB 200 GiB 15 MiB 3 KiB 200 GiB 79 GiB 71.68 2.14 160 osd.5 7 hdd 0.27190 1.00000 278 GiB 1.1 GiB 15 MiB 0 B 1.1 GiB 277 GiB 0.40 0.01 169 osd.7 TOTAL 2.2 TiB 746 GiB 120 MiB 34 KiB 746 GiB 1.4 TiB 33.49 MIN/MAX VAR: 0.01/2.55 STDDEV: 30.50 2 hours later: [root@rit-tceph ~]# ceph status cluster: id: bf1f51f5-b381-4cf7-b3db-88d044c1960c health: HEALTH_WARN 1 nearfull osd(s) 3 pool(s) nearfull services: mon: 3 daemons, quorum tceph-01,tceph-02,tceph-03 mgr: tceph-01(active), standbys: tceph-02, tceph-03 mds: testfs-1/1/1 up {0=tceph-01=up:active}, 2 up:standby osd: 9 osds: 8 up, 8 in data: pools: 3 pools, 500 pgs objects: 24 objects, 2.3 KiB usage: 748 GiB used, 1.4 TiB / 2.2 TiB avail pgs: 500 active+clean The usage is increasing surprisingly fast. Thanks for any pointers! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |