Hi! What kind of client (kernel vs. FUSE) do you use? I experience a lot of the following problems with the most recent ubuntu 18.04.3 kernel 4.15.0-66-generic : kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but object is from ceph_inode_info Other clients with older kernels (e.g. 4.15.0-47-generic) work without interruption on the same CephFS. Lars Mon, 28 Oct 2019 22:10:25 +0000 Kári Bertilsson <karibertils@xxxxxxxxx> ==> Patrick Donnelly <pdonnell@xxxxxxxxxx> : > Any ideas or tips on how to debug further ? > > On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <karibertils@xxxxxxxxx> > wrote: > > > Hello Patrick, > > > > Here is output from those commands > > https://pastebin.com/yUmuQuYj > > > > 5 clients have the file system mounted, but only 2 of them have most of > > the activity. > > > > > > > > On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <pdonnell@xxxxxxxxxx> > > wrote: > > > >> Hello Kári, > >> > >> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <karibertils@xxxxxxxxx> > >> wrote: > >> > This seems to happen mostly when listing folders containing 10k+ > >> folders. > >> > > >> > The dirlisting hangs indefinitely or until i restart the active MDS and > >> then the hanging "ls" command will finish running. > >> > > >> > Every time restarting the active MDS fixes the problem for a while. > >> > >> Please share details about your cluster. `fs dump`, `ceph status`, and > >> `ceph versions`. How many clients are using the file system? > >> > >> -- > >> Patrick Donnelly, Ph.D. > >> He / Him / His > >> Senior Software Engineer > >> Red Hat Sunnyvale, CA > >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > >> > >> -- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |