We also had a full MDS crash a couple of weeks ago due to what seems to
be another back-ported feature going wrong. As soon as I deployed the
16.2.9 ceph-fuse client, someone asked for an unknown ceph xattr, which
crashed the MDS and kept crashing all the standby MDS's as well as they
got activated.
The reason seems to be that a new message is sent by the 16.2.9
ceph-fuse (and NOT by the 16.2.7 one) to the MDS for unknown ceph xattr
(now called vxattr). The MDS code has a couple of switch statements by
message type that do an abort when a message is received that the MDS
does not expect, which is exactly what happened. Then, as the MDS
crashed, a standby got activated, the clients got to replay their
pending requests - and ... the next MDS crashed due to the same message
- bringing the whole file system down.Finally I found the culprit client
- killed it, which got the cluster back to working. I ended up patching
the 16.2.9 client not to send this message and deployed that change quickly.
My question is - why do we backport significant new features to stable
releases? Especially ones that change the messaging API? I absolutely
not expect such a crash when updating a point release of a client.
Andras
On 7/12/22 07:01, Frank Schilder wrote:
Hi Gregory,
Thanks for your fast reply. I created https://urldefense.com/v3/__https://tracker.ceph.com/issues/56529__;!!DSb-azq1wVFtOg!Rp8Drbtzjblq3DeEE3NuOkr3ERgUTUvU7IZekfMqL7bAlq6UREUe8Dlk2Q46rssYChOsfegNZAOvNBRSDLR_$ and attached the standard logs. In case you need more, please let me know. Note that I added some more buggy behaviour, the vxattr handling seems broken more or less all the way around.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum@xxxxxxxxxx>
Sent: 11 July 2022 19:14:26
To: Frank Schilder
Cc: ceph-users@xxxxxxx
Subject: Re: ceph-fs crashes on getfattr
On Mon, Jul 11, 2022 at 8:26 AM Frank Schilder <frans@xxxxxx> wrote:
Hi all,
we made a very weird observation on our ceph test cluster today. A simple getfattr with a misspelled attribute name sends the MDS cluster into a crash+restart loop. Something as simple as
getfattr -n ceph.dir.layout.po /mnt/cephfs
kills a ceph-fs completely. The problem can be resolved if one executes a "umount -f /mnt/cephfs" on the host where the getfattr was executed. The MDS daemons need a restart. One might also need to clear the OSD blacklist.
We observe this with a kernel client on 5.18.6-1.el7.elrepo.x86_64 (Centos 7) with mimic and I'm absolutely sure I have not seen this problem with mimic on earlier 5.9.X-kernel versions.
Is this known to be a kernel client bug? Possibly fixed already?
That obviously shouldn't happen. Please file a tracker ticket.
There's been a fair bit of churn in how we handle the "vxattrs" so my
guess is an incompatibility got introduced between newer clients and
the old server implementation, but obviously we want it to work and we
especially shouldn't be crashing the MDS. Skimming through it I'm
actually not seeing what a client *could* do in that path to crash the
server so I'm a bit confused...
Oh. I think I see it now, but I'd like to confirm. Yeah, please make
that tracker ticket and attach the backtrace you get.
Thanks,
-Greg
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
