Re: Dirlisting hangs with cephfs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I jumped the gun too quickly, dirlisting is still hanging with no entries in `ceph osd blacklist ls`.

But when i restart the active MDS and the standby goes active dirlisting finishes and i get 2 entries in blacklist with the IP address on the previously active MDS.

On Tue, Oct 29, 2019 at 1:03 PM Kári Bertilsson <karibertils@xxxxxxxxx> wrote:
I am noticing i have many entries in `ceph osd blacklist ls` and dirlisting works again after i removed all entries.
What can cause this and is there any way to disable blacklisting ?

On Tue, Oct 29, 2019 at 11:56 AM Kári Bertilsson <karibertils@xxxxxxxxx> wrote:
The file system was created on luminous and the problems started after upgrading from luminous to nautilus.
All CephFS configuration should be pretty much default except i enabled snapshots which was disabled by default on luminous.

On Tue, Oct 29, 2019 at 11:48 AM Kári Bertilsson <karibertils@xxxxxxxxx> wrote:
All clients are using the kernel client on proxmox kernel version 5.0.21-3-pve.

The mds logs are not showing anything interesting and have very little in them except for the restarts, maybe i need to increase debug level ?

On Tue, Oct 29, 2019 at 6:33 AM Lars Täuber <taeuber@xxxxxxx> wrote:
Hi!

What kind of client (kernel vs. FUSE) do you use?
I experience a lot of the following problems with the most recent ubuntu 18.04.3 kernel 4.15.0-66-generic :
kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but object is from ceph_inode_info

Other clients with older kernels (e.g. 4.15.0-47-generic) work without interruption on the same CephFS.


Lars


Mon, 28 Oct 2019 22:10:25 +0000
Kári Bertilsson <karibertils@xxxxxxxxx> ==> Patrick Donnelly <pdonnell@xxxxxxxxxx> :
> Any ideas or tips on how to debug further ?
>
> On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <karibertils@xxxxxxxxx>
> wrote:
>
> > Hello Patrick,
> >
> > Here is output from those commands
> > https://pastebin.com/yUmuQuYj
> >
> > 5 clients have the file system mounted, but only 2 of them have most of
> > the activity.
> >
> >
> >
> > On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <pdonnell@xxxxxxxxxx>
> > wrote:
> > 
> >> Hello Kári,
> >>
> >> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <karibertils@xxxxxxxxx>
> >> wrote: 
> >> > This seems to happen mostly when listing folders containing 10k+ 
> >> folders. 
> >> >
> >> > The dirlisting hangs indefinitely or until i restart the active MDS and 
> >> then the hanging "ls" command will finish running. 
> >> >
> >> > Every time restarting the active MDS fixes the problem for a while. 
> >>
> >> Please share details about your cluster. `fs dump`, `ceph status`, and
> >> `ceph versions`. How many clients are using the file system?
> >>
> >> --
> >> Patrick Donnelly, Ph.D.
> >> He / Him / His
> >> Senior Software Engineer
> >> Red Hat Sunnyvale, CA
> >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> >>
> >> 


--
                            Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23                      10117 Berlin
Tel.: +49 30 20370-352           http://www.bbaw.de
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux