Hi list,
I have a serious problem now... I think.
One of my users just informed me that a file he created (.doc file) has
a different content then before. It looks like the file's inode is
completely wrong and points to the wrong object. I myself have found
another file with the same symptoms. I'm afraid my (production) FS is
corrupt now, unless there is a possibility to fix the inodes.
Timeline of what happend:
Last week I upgraded our Ceph Jewel to Luminous.
This went without any problem.
I already had 5 MDS available and went with the Multi-MDS feature and
enabled it. The seemed to work okay, but after a while my MDS went
beserk and went flapping (crashed -> replay -> rejoin -> crashed)
The only way to fix this and get the FS back online was the disaster
recovery procedure:
cephfs-journal-tool event recover_dentries summary
ceph fs set cephfs cluster_down true
cephfs-table-tool all reset session
cephfs-table-tool all reset inode
cephfs-journal-tool --rank=cephfs:0 journal reset
ceph mds fail 0
ceph fs reset cephfs --yes-i-really-mean-it
Restarted the MDS and I was back online. Shortly after I was getting a
lot of "loaded dup inode". In the meanwhile the MDS kept crashing. It
looks like it had trouble creating new inodes. Right before the crash
it mostly complained something like:
-2> 2018-07-05 05:05:01.614290 7f8f8574b700 4 mds.0.server
handle_client_request client_request(client.324932014:1434 create
#0x10000360346/pyfiles.txt 2018-07-05 05:05:01.607458 caller_uid=0,
caller_gid=0{}) v2
-1> 2018-07-05 05:05:01.614320 7f8f7e73d700 5 mds.0.log
_submit_thread 24100753876035~1070 : EOpen [metablob 0x10000360346, 1
dirs], 1 open files
0> 2018-07-05 05:05:01.661155 7f8f8574b700 -1 /build/ceph-
12.2.5/src/mds/MDCache.cc: In function 'void
MDCache::add_inode(CInode*)' thread 7f8f8574b700 time 2018-07-05
05:05:01.615123
/build/ceph-12.2.5/src/mds/MDCache.cc: 262: FAILED assert(!p)
I also tried to counter the create inode crash by doing the following:
cephfs-journal-tool event recover_dentries
cephfs-journal-tool journal reset
cephfs-table-tool all reset session
cephfs-table-tool all reset inode
cephfs-table-tool all take_inos 100000
I'm worried that my FS is corrupt because files are not linked
correctly and have different content then they should.
Please help.
On Thu, 2018-07-05 at 10:35 +0200, Dennis Kramer (DT) wrote:
> Hi,
>
> I'm getting a bunch of "loaded dup inode" errors in the MDS logs.
> How can this be fixed?
>
> logs:
> 2018-07-05 10:20:05.591948 mds.mds05 [ERR] loaded dup inode 0x10000991921
> [2,head] v160 at <file path>, but inode 0x10000991921.head v146 already
> exists at <another file path>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.comAttachment:
signature.asc
Description: This is a digitally signed message part
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
