Hi all, I seem to have gotten the MDS up to the point that it reports stats. Does this mean anything: 2025-01-10T20:50:25.256+0100 7f87ccd5f700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s 2025-01-10T20:50:25.256+0100 7f87ccd5f700 0 mds.beacon.ceph-12 Skipping beacon heartbeat to monitors (last acked 156.027s ago); MDS internal heartbeat is not healthy! I hope it doesn't get failed by some king of timeout now. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Spencer Macphee <spencerofsydney@xxxxxxxxx> Sent: Friday, January 10, 2025 7:16 PM To: Frank Schilder Cc: ceph-users@xxxxxxx Subject: Re: Help needed, ceph fs down due to large stray dir I had a similar issue some months ago that ended up using around 300 gigabytes of RAM for a similar number of strays. You can get an idea of the strays kicking around by checking the omapkeys of the stray objects in the cephfs metadata pool. Strays are tracked in objects: 600.00000000, 601.00000000, 602.00000000, etc... etc... That would also give you an indication if it's progressing at each restart. On Fri, Jan 10, 2025 at 1:30 PM Frank Schilder <frans@xxxxxx<mailto:frans@xxxxxx>> wrote: Hi all, we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted. I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM. Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank? Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |