On Tue, 2022-01-04 at 10:49 +0100, Bastian Blank wrote:
> Moin
>
> A customer reported panics inside memory management. Before all
> occurances there are reports about SLAB missmatch in the log. The
> "crash" tool shows freelist corruption in the memory dump. This makes
> this problem a use-after-free somewhere inside the ceph module.
>
> The crashs happen during high load situations, while copying data
> between two cephfs.
>
> > [152791.777454] ceph: dropping dirty+flushing - state for 00000000c039d4cc 1099526092092
> > [152791.777457] ------------[ cut here ]------------
> > [152791.777458] cache_from_obj: Wrong slab cache. jbd2_journal_handle but object is from kmalloc-256
> > [152791.777473] WARNING: CPU: 76 PID: 2676615 at mm/slab.h:521 kmem_cache_free+0x260/0x2b0
> […]
> > [152791.777530] CPU: 76 PID: 2676615 Comm: kworker/76:2 Kdump: loaded Not tainted 5.4.0-81-generic #91-Ubuntu
> > [152791.777531] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/28/2021
> > [152791.777540] Workqueue: ceph-msgr ceph_con_workfn [libceph]
> > [152791.777542] RIP: 0010:kmem_cache_free+0x260/0x2b0
> […]
> > [152791.777550] Call Trace:
> > [152791.777562] ceph_free_cap_flush+0x1d/0x20 [ceph]
> > [152791.777568] remove_session_caps_cb+0xcf/0x4b0 [ceph]
> > [152791.777573] ceph_iterate_session_caps+0xc8/0x2a0 [ceph]
> > [152791.777578] ? wake_up_session_cb+0xe0/0xe0 [ceph]
> > [152791.777583] remove_session_caps+0x55/0x190 [ceph]
> > [152791.777587] ? cleanup_session_requests+0x104/0x130 [ceph]
> > [152791.777592] handle_session+0x4c7/0x5e0 [ceph]
> > [152791.777597] dispatch+0x279/0x610 [ceph]
> > [152791.777602] try_read+0x566/0x8c0 [libceph]
>
> They reported the same in all tested kernels since 5.4, up to 5.15.5 or
> so. Sadly I have no tests with newer builds available.
>
> Any ideas how I can debug this further?
>
> Regards,
> Bastian
>
At first blush, this looks like the same problem as:
https://tracker.ceph.com/issues/52283
...but that should have been fixed in v5.14.
Do you have a more complete stack trace, preferably from your v5.15-ish
kernel? Log messages leading up to the WARNING may also be helpful. It
may be best to open a bug at https://tracker.ceph.com.
The log message before the [ cut here ] line, indicates that the client
was trying to drop caps in response to a session message from the MDS or
maybe a map change. Was the mount force-umounted or the client
blacklisted or something?
You may also want to try v5.16-rc8 if you're able to build your own
kernels. There were some patches that went in to improve how the client
handles inodes that become inaccessible.
--
Jeff Layton <jlayton@xxxxxxxxxx>
