Re: mds dump inode crashes file system

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Xiubo,

forgot to include these, the inodes i tried to dump and which caused a crash are

ceph tell "mds.ceph-10" dump inode 2199322355147 <-- original file/folder causing trouble
ceph tell "mds.ceph-10" dump inode 2199322727209 <-- copy also causing trouble (after taking snapshot??)

Other folders all the way up in the hierarchy did not lead to a crash, the dump worked fine for these.

The debug settings during the first tries were:

ceph config set mds.ceph-10 debug_mds 20/5
ceph config set mds.ceph-10 debug_ms 5/0

The hex codes are 0x20011d3e5cb and 0x20011d99329. The dump commands are also in the log. I tried a bit around to find out where the problem is localized. I didn't have the high debug settings all the time due to disk space constraints. I can pull specific logs over a short time for anything that is reproducible (short sequence of specific commands).

Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________________________________________
From: Frank Schilder <frans@xxxxxx>
Sent: Monday, May 15, 2023 6:33 PM
To: Xiubo Li; ceph-users@xxxxxxx
Subject:  Re: mds dump inode crashes file system

Dear Xiubo,

I uploaded the cache dump, the MDS log and the dmesg log containing the snaptrace dump to

ceph-post-file: 763955a3-7d37-408a-bbe4-a95dc687cd3f

Sorry, I forgot to add user and description this time.

A question about trouble shooting. I'm pretty sure I know the path where the error is located. Would a "ceph tell mds.1 scrub start / recursive repair" be able to discover and fix broken snaptraces? If not I'm awaiting further instructions.

Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________________________________________
From: Xiubo Li <xiubli@xxxxxxxxxx>
Sent: Friday, May 12, 2023 3:44 PM
To: Frank Schilder; ceph-users@xxxxxxx
Subject: Re:  Re: mds dump inode crashes file system


On 5/12/23 20:27, Frank Schilder wrote:
> Dear Xiubo and others.
>
>>> I have never heard about that option until now. How do I check that and how to I disable it if necessary?
>>> I'm in meetings pretty much all day and will try to send some more info later.
>> $ mount|grep ceph
> I get
>
> MON-IPs:SRC on DST type ceph (rw,relatime,name=con-fs2-rit-pfile,secret=<hidden>,noshare,acl,mds_namespace=con-fs2,_netdev)
>
> so async dirop seems disabled.

Yeah.


>> Yeah, the kclient just received a corrupted snaptrace from MDS.
>> So the first thing is you need to fix the corrupted snaptrace issue in cephfs and then continue.
> Ooookaaayyyy. I will take it as a compliment that you seem to assume I know how to do that. The documentation gives 0 hits. Could you please provide me with instructions of what to look for and/or what to do first?

There is no doc about this as I know.

>> If possible you can parse the above corrupted snap message to check what exactly corrupted.
>> I haven't get a chance to do that.
> Again, how would I do that? Is there some documentation and what should I expect?

Currently there is no easy way to do this as I know, last time I have
parsed the corrupted binary data to the corresponding message manully.

And then we could know what exactly has happened for the snaptrace.


>> You seems didn't enable the 'osd blocklist' cephx auth cap for mon:
> I can't find anything about an osd blocklist client auth cap in the documentation. Is this something that came after octopus? Our caps are as shown in the documentation for a ceph fs client (https://docs.ceph.com/en/octopus/cephfs/client-auth/), the one for mon is "allow r":
>
>          caps mds = "allow rw path=/shares"
>          caps mon = "allow r"
>          caps osd = "allow rw tag cephfs data=con-fs2"
Yeah, it seems the 'osd blocklist' was disabled. As I remembered if
enabled it should be something likes:

caps mon = "allow r, allow command \"osd blocklist\""

>
>> I checked that but by reading the code I couldn't get what had cause the MDS crash.
>> There seems something wrong corrupt the metadata in cephfs.
> He wrote something about an invalid xattrib (empty value). It would be really helpful to get a clue how to proceed. I managed to dump the MDS cache with the critical inode in cache. Would this help with debugging? I also managed to get debug logs with debug_mds=20 during a crash caused by an "mds dump inode" command. Would this contain something interesting? I can also pull the rados objects out and can upload all of these files.

Yeah, possibly. Where is the logs ?


> I managed to track the problem down to a specific folder with a few files (I'm not sure if this coincides with the snaptrace issue, we might have 2 issues here). I made a copy of the folder and checked that an "mds dump inode" for the copy does not crash the MDS. I then moved the folders for which this command causes a crash to a different location outside the mounts. Do you think this will help? I'm wondering if after taking our daily snapshot tomorrow we end up in the degraded situation again.
>
> I really need instructions for how to check what is broken without an MDS crash and then how to fix it.

Firstly we need to know where the corrupted metadata is.

I think the mds debug logs and the above corrupted snaptrace could help.
Need to parse that corrupted binary data.

Thanks
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux