Dear Xiubo, could you explain how to enable kernel debug logs (I assume this is on the client)? Thanks, Manuel On Fri, May 13, 2022 at 9:39 AM Xiubo Li <xiubli@xxxxxxxxxx> wrote: > > On 5/12/22 12:06 AM, Stefan Kooman wrote: > > Hi List, > > > > We have quite a few linux kernel clients for CephFS. One of our > > customers has been running mainline kernels (CentOS 7 elrepo) for the > > past two years. They started out with 3.x kernels (default CentOS 7), > > but upgraded to mainline when those kernels would frequently generate > > MDS warnings like "failing to respond to capability release". That > > worked fine until 5.14 kernel. 5.14 and up would use a lot of CPU and > > *way* more bandwidth on CephFS than older kernels (order of > > magnitude). After the MDS was upgraded from Nautilus to Octopus that > > behavior is gone (comparable CPU / bandwidth usage as older kernels). > > However, the newer kernels are now the ones that give "failing to > > respond to capability release", and worse, clients get evicted > > (unresponsive as far as the MDS is concerned). Even the latest 5.17 > > kernels have that. No difference is observed between using messenger > > v1 or v2. MDS version is 15.2.16. > > Surprisingly the latest stable kernels from CentOS 7 work flawlessly > > now. Although that is good news, newer operating systems come with > > newer kernels. > > > > Does anyone else observe the same behavior with newish kernel clients? > > There have some known bugs, which have been fixed or under fixing > recently, even in the mainline and, not sure whether are they related. > Such as [1][2][3][4]. More detail please see ceph-client repo testing > branch [5]. > > I have never see the "failing to respond to capability release" issue > yet, if you have the MDS logs(debug_mds = 25 and debug_ms = 1) and > kernel debug logs will be better to help debug it further, or provide > the steps to reproduce it. > > [1] https://tracker.ceph.com/issues/55332 > [2] https://tracker.ceph.com/issues/55421 > [3] https://bugzilla.redhat.com/show_bug.cgi?id=2063929 > [4] https://tracker.ceph.com/issues/55377 > [5] https://github.com/ceph/ceph-client/commits/testing > > Thanks > > -- Xiubo > > > > > Gr. Stefan > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |