Re: Ceph Nautilous 14.2.22 slow OSD memory leak?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear Frederic,

Thanks a lot for the suggestions. We are using the valilla Linux 4.19 LTS version. Do you think we may be suffering from the same bug?

best regards,

Samuel



huxiaoyu@xxxxxxxxxxxx
 
From: Frédéric Nass
Date: 2024-01-12 09:19
To: huxiaoyu
CC: ceph-users
Subject: Re:  Ceph Nautilous 14.2.22 slow OSD memory leak?
Hello,
 
We've had a similar situation recently where OSDs would use way more memory than osd_memory_target and get OOM killed by the kernel.
It was due to a kernel bug related to cgroups [1].
 
If num_cgroups below keeps increasing then you may hit this bug.
 
$ cat /proc/cgroups | grep -e subsys -e blkio | column -t
   #subsys_name  hierarchy  num_cgroups  enabled
   blkio         4          1099         1
 
If you hit this bug, upgrading OSDs nodes kernels should get you through. If you can't access the Red Hat KB [1], let me know your current nodes kernel version and I'll check for you.
 
Regards,
Frédéric.
 
[1] https://access.redhat.com/solutions/7014337


De: huxiaoyu <huxiaoyu@xxxxxxxxxxxx>
à: ceph-users <ceph-users@xxxxxxx>
Envoyé: mercredi 10 janvier 2024 19:21 CET
Sujet :  Ceph Nautilous 14.2.22 slow OSD memory leak?

Dear Ceph folks, 

I am responsible for two Ceph clusters, running Nautilius 14.2.22 version, one with replication 3, and the other with EC 4+2. After around 400 days runing quietly and smoothly, recently the two clusters occured with similar problems: some of OSDs consume ca 18 GB while the memory target is setting at 2GB. 

What could wrong in the background? Does it mean any slow OSD memory leak issues with 14.2.22 which i do not know yet? 

I would be highly appreciated if some some provides any clues, ideas, comments ...... 

best regards, 

Samuel 



huxiaoyu@xxxxxxxxxxxx 
_______________________________________________ 
ceph-users mailing list -- ceph-users@xxxxxxx 
To unsubscribe send an email to ceph-users-leave@xxxxxxx 
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux