some further investigation about three mds with trimming behind problem: logs captured over two days show that, some log segments are stuck in trimming process. It looks like a bug with trimming log segment? Any thoughts? ==========log capture============ 9/26: debug 2023-09-26T16:50:59.004+0000 7fc74d95e700 10 mds.3.log _trim_expired_segments waiting for 197465903/720757956586 to expire debug 2023-09-26T16:45:22.788+0000 7f9c253a8700 10 mds.4.log _trim_expired_segments waiting for 195341004/893374309813 to expire debug 2023-09-26T14:11:37.681+0000 7f8f047cf700 10 mds.5.log _trim_expired_segments waiting for 189942575/642272326541 to expire 9/27: debug 2023-09-27T15:35:04.841+0000 7fc74d95e700 10 mds.3.log _trim_expired_segments waiting for 197465903/720757956586 to expire debug 2023-09-27T15:18:51.442+0000 7f9c29bb1700 10 mds.4.log _trim_expired_segments waiting for 195341004/893374309813 to expire debug 2023-09-27T15:27:33.024+0000 7f8f047cf700 10 mds.5.log _trim_expired_segments waiting for 189942575/642272326541 to expire Regards, Ben Ben <ruidong.gao@xxxxxxxxx> 于2023年9月26日周二 20:31写道： > Hi, > see below for details of warnings. > the cluster is running 17.2.5. the warnings have been around for a while. > one concern of mine is num_segments growing over time. clients with warn > of MDS_CLIENT_OLDEST_TID increase from 18 to 25 as well. The nodes are > with kernel 4.19.0-91.82.42.uelc20.x86_64. It looks like bugs with client > library. And rebooting nodes with problem will fix it for short period of > time? Any suggestions from community for fixing? > > Thanks, > Ben > > > [root@8cd2c0657c77 /]# ceph health detail > > HEALTH_WARN 6 hosts fail cephadm check; 2 clients failing to respond to > capability release; 25 clients failing to advance oldest client/flush tid; > 3 MDSs report slow requests; 3 MDSs behind on trimming > > [WRN] CEPHADM_HOST_CHECK_FAILED: 6 hosts fail cephadm check > > host host15w (192.168.31.33) failed check: Unable to reach remote > host host15w. Process exited with non-zero exit status 1 > > host host20w (192.168.31.38) failed check: Unable to reach remote > host host20w. Process exited with non-zero exit status 1 > > host host19w (192.168.31.37) failed check: Unable to reach remote > host host19w. Process exited with non-zero exit status 1 > > host host17w (192.168.31.35) failed check: Unable to reach remote > host host17w. Process exited with non-zero exit status 1 > > host host18w (192.168.31.36) failed check: Unable to reach remote > host host18w. Process exited with non-zero exit status 1 > > host host16w (192.168.31.34) failed check: Unable to reach remote > host host16w. Process exited with non-zero exit status 1 > > [WRN] MDS_CLIENT_LATE_RELEASE: 2 clients failing to respond to capability > release > > mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to > respond to capability release client_id: 460983 > > mds.code-store.host16w.vucirx(mds.3): Client failing to respond to > capability release client_id: 460983 > > [WRN] MDS_CLIENT_OLDEST_TID: 25 clients failing to advance oldest > client/flush tid > > mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to > advance its oldest client/flush tid. client_id: 460983 > > mds.code-store.host18w.fdsqff(mds.1): Client failing to advance its > oldest client/flush tid. client_id: 460226 > > mds.code-store.host18w.fdsqff(mds.1): Client k8s-node32 failing to > advance its oldest client/flush tid. client_id: 239797 > > mds.code-store.host15w.reolpx(mds.5): Client k8s-node34 failing to > advance its oldest client/flush tid. client_id: 460226 > > mds.code-store.host15w.reolpx(mds.5): Client k8s-node32 failing to > advance its oldest client/flush tid. client_id: 239797 > > mds.code-store.host15w.reolpx(mds.5): Client failing to advance its > oldest client/flush tid. client_id: 460983 > > mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node34 failing to > advance its oldest client/flush tid. client_id: 460226 > > mds.code-store.host18w.rtyvdy(mds.7): Client failing to advance its > oldest client/flush tid. client_id: 239797 > > mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node36 failing to > advance its oldest client/flush tid. client_id: 460983 > > mds.code-store.host17w.kcdopb(mds.2): Client failing to advance its > oldest client/flush tid. client_id: 239797 > > mds.code-store.host17w.kcdopb(mds.2): Client failing to advance its > oldest client/flush tid. client_id: 460983 > > mds.code-store.host17w.kcdopb(mds.2): Client k8s-node34 failing to > advance its oldest client/flush tid. client_id: 460226 > > mds.code-store.host17w.kcdopb(mds.2): Client k8s-node24 failing to > advance its oldest client/flush tid. client_id: 12072730 > > mds.code-store.host20w.bfoftp(mds.4): Client k8s-node32 failing to > advance its oldest client/flush tid. client_id: 239797 > > mds.code-store.host20w.bfoftp(mds.4): Client k8s-node36 failing to > advance its oldest client/flush tid. client_id: 460983 > > mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node24 failing to > advance its oldest client/flush tid. client_id: 12072730 > > mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node34 failing to > advance its oldest client/flush tid. client_id: 460226 > > mds.code-store.host19w.ywrmiz(mds.6): Client failing to advance its > oldest client/flush tid. client_id: 239797 > > mds.code-store.host19w.ywrmiz(mds.6): Client failing to advance its > oldest client/flush tid. client_id: 460983 > > mds.code-store.host16w.vucirx(mds.3): Client failing to advance its > oldest client/flush tid. client_id: 460983 > > mds.code-store.host16w.vucirx(mds.3): Client failing to advance its > oldest client/flush tid. client_id: 460226 > > mds.code-store.host16w.vucirx(mds.3): Client failing to advance its > oldest client/flush tid. client_id: 239797 > > mds.code-store.host17w.pdziet(mds.0): Client k8s-node32 failing to > advance its oldest client/flush tid. client_id: 239797 > > mds.code-store.host17w.pdziet(mds.0): Client k8s-node34 failing to > advance its oldest client/flush tid. client_id: 460226 > > mds.code-store.host17w.pdziet(mds.0): Client k8s-node36 failing to > advance its oldest client/flush tid. client_id: 460983 > > [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests > > mds.code-store.host15w.reolpx(mds.5): 4 slow requests are blocked > 5 > secs > > mds.code-store.host20w.bfoftp(mds.4): 6 slow requests are blocked > 5 > secs > > mds.code-store.host16w.vucirx(mds.3): 97 slow requests are blocked > > 5 secs > > [WRN] MDS_TRIM: 3 MDSs behind on trimming > > mds.code-store.host15w.reolpx(mds.5): Behind on trimming (25831/128) > max_segments: 128, num_segments: 25831 > > mds.code-store.host20w.bfoftp(mds.4): Behind on trimming (27605/128) > max_segments: 128, num_segments: 27605 > > mds.code-store.host16w.vucirx(mds.3): Behind on trimming (28676/128) > max_segments: 128, num_segments: 28676 > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |