Re: mds dump inode crashes file system

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 5/16/23 17:44, Frank Schilder wrote:
Hi Xiubo,

forgot to include these, the inodes i tried to dump and which caused a crash are

ceph tell "mds.ceph-10" dump inode 2199322355147 <-- original file/folder causing trouble
ceph tell "mds.ceph-10" dump inode 2199322727209 <-- copy also causing trouble (after taking snapshot??)

Other folders all the way up in the hierarchy did not lead to a crash, the dump worked fine for these.

The debug settings during the first tries were:

ceph config set mds.ceph-10 debug_mds 20/5
ceph config set mds.ceph-10 debug_ms 5/0

Just set the 'debug_ms' to 1 should be enough. A higher level will introduce many noises.


The hex codes are 0x20011d3e5cb and 0x20011d99329. The dump commands are also in the log. I tried a bit around to find out where the problem is localized. I didn't have the high debug settings all the time due to disk space constraints. I can pull specific logs over a short time for anything that is reproducible (short sequence of specific commands).

Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________________________________________
From: Frank Schilder <frans@xxxxxx>
Sent: Monday, May 15, 2023 6:33 PM
To: Xiubo Li; ceph-users@xxxxxxx
Subject:  Re: mds dump inode crashes file system

Dear Xiubo,

I uploaded the cache dump, the MDS log and the dmesg log containing the snaptrace dump to

ceph-post-file: 763955a3-7d37-408a-bbe4-a95dc687cd3f

Sorry, I forgot to add user and description this time.

A question about trouble shooting. I'm pretty sure I know the path where the error is located. Would a "ceph tell mds.1 scrub start / recursive repair" be able to discover and fix broken snaptraces? If not I'm awaiting further instructions.

Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________________________________________
From: Xiubo Li <xiubli@xxxxxxxxxx>
Sent: Friday, May 12, 2023 3:44 PM
To: Frank Schilder; ceph-users@xxxxxxx
Subject: Re:  Re: mds dump inode crashes file system


On 5/12/23 20:27, Frank Schilder wrote:
Dear Xiubo and others.

I have never heard about that option until now. How do I check that and how to I disable it if necessary?
I'm in meetings pretty much all day and will try to send some more info later.
$ mount|grep ceph
I get

MON-IPs:SRC on DST type ceph (rw,relatime,name=con-fs2-rit-pfile,secret=<hidden>,noshare,acl,mds_namespace=con-fs2,_netdev)

so async dirop seems disabled.
Yeah.


Yeah, the kclient just received a corrupted snaptrace from MDS.
So the first thing is you need to fix the corrupted snaptrace issue in cephfs and then continue.
Ooookaaayyyy. I will take it as a compliment that you seem to assume I know how to do that. The documentation gives 0 hits. Could you please provide me with instructions of what to look for and/or what to do first?
There is no doc about this as I know.

If possible you can parse the above corrupted snap message to check what exactly corrupted.
I haven't get a chance to do that.
Again, how would I do that? Is there some documentation and what should I expect?
Currently there is no easy way to do this as I know, last time I have
parsed the corrupted binary data to the corresponding message manully.

And then we could know what exactly has happened for the snaptrace.


You seems didn't enable the 'osd blocklist' cephx auth cap for mon:
I can't find anything about an osd blocklist client auth cap in the documentation. Is this something that came after octopus? Our caps are as shown in the documentation for a ceph fs client (https://docs.ceph.com/en/octopus/cephfs/client-auth/), the one for mon is "allow r":

          caps mds = "allow rw path=/shares"
          caps mon = "allow r"
          caps osd = "allow rw tag cephfs data=con-fs2"
Yeah, it seems the 'osd blocklist' was disabled. As I remembered if
enabled it should be something likes:

caps mon = "allow r, allow command \"osd blocklist\""

I checked that but by reading the code I couldn't get what had cause the MDS crash.
There seems something wrong corrupt the metadata in cephfs.
He wrote something about an invalid xattrib (empty value). It would be really helpful to get a clue how to proceed. I managed to dump the MDS cache with the critical inode in cache. Would this help with debugging? I also managed to get debug logs with debug_mds=20 during a crash caused by an "mds dump inode" command. Would this contain something interesting? I can also pull the rados objects out and can upload all of these files.
Yeah, possibly. Where is the logs ?


I managed to track the problem down to a specific folder with a few files (I'm not sure if this coincides with the snaptrace issue, we might have 2 issues here). I made a copy of the folder and checked that an "mds dump inode" for the copy does not crash the MDS. I then moved the folders for which this command causes a crash to a different location outside the mounts. Do you think this will help? I'm wondering if after taking our daily snapshot tomorrow we end up in the degraded situation again.

I really need instructions for how to check what is broken without an MDS crash and then how to fix it.
Firstly we need to know where the corrupted metadata is.

I think the mds debug logs and the above corrupted snaptrace could help.
Need to parse that corrupted binary data.

Thanks
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux