Re: cephfs metadata damage and scrub error

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Any update to this?  I also have the same problem....

# for i in $(cat pg_dump | grep 'stale+active+clean' | awk {'print $1'}); do echo -n "$i: "; rados list-inconsistent-obj $i; echo; done
107.ff: {"epoch":10762,"inconsistents":[]}
.....
and so on for 49 pg's that I think I had a problem with

# ceph tell mds.ceph damage ls | python -m "json.tool"
2017-07-18 16:28:08.657673 7f766e629700  0 client.1923797 ms_handle_reset on 192.168.1.10:6800/1268574779
2017-07-18 16:28:08.665693 7f76577fe700  0 client.1923798 ms_handle_reset on 192.168.1.10:6800/1268574779
[
    {
        "damage_type": "dir_frag",
        "frag": "*",
        "id": 4153356868,
        "ino": 1099511661266
    }
]

# cat dirs_ceph_inodes | grep 1099511661266
1099511661266 drwxr-xr-x 1 chris chris 1 May  4  2017 /mnt/ceph/2017/05/04/

# rm -rf /mnt/ceph/2017/05/04/
rm: cannot remove ‘/mnt/ceph/2017/05/04/’: Directory not empty

# ceph -v
ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)

All osd's are on 11.2.0


I am A OK with losing the directory and the contents...just need to get to happy pastures

Thanks,
/Chris Callegari



On Tue, May 30, 2017 at 7:42 AM, James Eckersall <james.eckersall@xxxxxxxxx> wrote:
Further to this, we managed to repair the inconsistent PG by comparing the object digests and removing the one that didn't match (3 of 4 replicas had the same digest, 1 didn't) and then issuing a pg repair and scrub.
This has removed the inconsistent flag on the PG, however, we are still seeing the mds report damage.

We tried removing the damage from the mds with damage rm, then ran a recursive stat across the problem directory, but the damage re-appeared.
Tried dong a scrub_path, but the command returned code -2 and the mds log shows that the scrub started and finished less than 1ms later.

Any further help is greatly appreciated.

On 17 May 2017 at 10:58, James Eckersall <james.eckersall@xxxxxxxxx> wrote:
An update to this.  The cluster has been upgraded to Kraken, but I've still got the same PG reporting inconsistent and the same error message about mds metadata damaged.
Can anyone offer any further advice please?
If you need output from the ceph-osdomap-tool, could you please explain how to use it?  I haven't been able to find any docs that explain.

Thanks
J

On 3 May 2017 at 14:35, James Eckersall <james.eckersall@xxxxxxxxx> wrote:
Hi David,

Thanks for the reply, it's appreciated.
We're going to upgrade the cluster to Kraken and see if that fixes the metadata issue.

J

On 2 May 2017 at 17:00, David Zafman <dzafman@xxxxxxxxxx> wrote:


James,

    You have an omap corruption.  It is likely caused by a bug which has already been identified.  A fix for that problem is available but it is still pending backport for the next Jewel point release.  All 4 of your replicas have different "omap_digest" values.

Instead of the xattrs the ceph-osdomap-tool --command dump-objects-with-keys output from OSDs 3, 10, 11, 23 would be interesting to compare.

***WARNING*** Please backup your data before doing any repair attempts.

If you can upgrade to Kraken v11.2.0, it will auto repair the omaps on ceph-osd start up.  It will likely still require a ceph pg repair to make the 4 replicas consistent with each other.  The final result may be the reappearance of removed MDS files in the directory.

If you can recover the data, you could remove the directory entirely and rebuild it.  The original bug was triggered during omap deletion typically in a large directory which corresponds to an individual unlink in cephfs.

If you can build a branch in github to get the newer ceph-osdomap-tool you could try to use it to repair the omaps.

David



On 5/2/17 5:05 AM, James Eckersall wrote:
Hi,

I'm having some issues with a ceph cluster.  It's an 8 node cluster rnning
Jewel ceph-10.2.7-0.el7.x86_64 on CentOS 7.
This cluster provides RBDs and a CephFS filesystem to a number of clients.

ceph health detail is showing the following errors:

pg 2.9 is active+clean+inconsistent, acting [3,10,11,23]
1 scrub errors
mds0: Metadata damage detected


The pg 2.9 is in the cephfs_metadata pool (id 2).

I've looked at the OSD logs for OSD 3, which is the primary for this PG,
but the only thing that appears relating to this PG is the following:

log_channel(cluster) log [ERR] : 2.9 deep-scrub 1 errors

After initiating a ceph pg repair 2.9, I see the following in the primary
OSD log:

log_channel(cluster) log [ERR] : 2.9 repair 1 errors, 0 fixed
log_channel(cluster) log [ERR] : 2.9 deep-scrub 1 errors


I found the below command in a previous ceph-users post.  Running this
returns the following:

# rados list-inconsistent-obj 2.9
{"epoch":23738,"inconsistents":[{"object":{"name":"10000411194.00000000","nspace":"","locator":"","snap":"head","version":14737091},"errors":["omap_digest_mismatch"],"union_shard_errors":[],"selected_object_info":"2:9758b358:::10000411194.00000000:head(33456'14737091
mds.0.214448:248532 dirty|omap|data_digest s 0 uv 14737091 dd
ffffffff)","shards":[{"osd":3,"errors":[],"size":0,"omap_digest":"0x6748eef3","data_digest":"0xffffffff"},{"osd":10,"errors":[],"size":0,"omap_digest":"0xa791d5a4","data_digest":"0xffffffff"},{"osd":11,"errors":[],"size":0,"omap_digest":"0x53f46ab0","data_digest":"0xffffffff"},{"osd":23,"errors":[],"size":0,"omap_digest":"0x97b80594","data_digest":"0xffffffff"}]}]}


So from this, I think that the object in PG 2.9 with the problem is
10000411194.00000000.

This is what I see on the filesystem on the 4 OSD's this PG resides on:

-rw-r--r--. 1 ceph ceph 0 Apr 27 12:31
/var/lib/ceph/osd/ceph-3/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2
-rw-r--r--. 1 ceph ceph 0 Apr 15 22:05
/var/lib/ceph/osd/ceph-10/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2
-rw-r--r--. 1 ceph ceph 0 Apr 15 22:07
/var/lib/ceph/osd/ceph-11/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2
-rw-r--r--. 1 ceph ceph 0 Apr 16 03:58
/var/lib/ceph/osd/ceph-23/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2

The extended attrs are as follows, although I have no idea what any of them
mean.

# file:
var/lib/ceph/osd/ceph-11/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2
user.ceph._=0sDwj5AAAABAM1AAAAAAAAABQAAAAxMDAwMDQxMTE5NC4wMDAwMDAwMP7/////////6RrNGgAAAAAAAgAAAAAAAAAGAxwAAAACAAAAAAAAAP////8AAAAAAAAAAP//////////AAAAABUn4QAAAAAAu4IAAK4m4QAAAAAAu4IAAAICFQAAAAIAAAAAAAAAAOSZDAAAAAAAsEUDAAAAAAAAAAAAjUoIWUgWsQQCAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVJ+EAAAAAAAAAAAAAAAAAABwAAACNSghZESm8BP///w==
user.ceph._@1=0s//////8=
user.ceph._layout=0sAgIYAAAAAAAAAAAAAAAAAAAA//////////8AAAAA
user.ceph._parent=0sBQRPAQAAlBFBAAABAAAIAAAAAgIjAAAAjxFBAAABAAAPAAAAdHViZWFtYXRldXIubmV0qdgAAAAAAAACAh0AAAB/EUEAAAEAAAkAAAB3cC1yb2NrZXREAAAAAAAAAAICGQAAABYNQQAAAQAABQAAAGNhY2hlUgAAAAAAAAACAh4AAAAQDUEAAAEAAAoAAAB3cC1jb250ZW50NAMAAAAAAAACAhgAAAANDUEAAAEAAAQAAABodG1sIAEAAAAAAAACAikAAADagTMAAAEAABUAAABuZ2lueC1waHA3LWNsdmdmLWRhdGGJAAAAAAAAAAICMwAAADkAAAAAAQ==
user.ceph._parent@1
=0sAAAfAAAANDg4LTU3YjI2NTdmMmZhMTMtbWktcHJveWVjdG8tMXSQCAAAAAAAAgIcAAAAAQAAAAAAAAAIAAAAcHJvamVjdHPBAgcAAAAAAAIAAAAAAAAAAAAAAA==
user.ceph.snapset=0sAgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==
user.cephos.seq=0sAQEQAAAAgcAqFAAAAAAAAAAAAgAAAAA=
user.cephos.spill_out=0sMAA=getfattr: Removing leading '/' from absolute
path names

# file:
var/lib/ceph/osd/ceph-3/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2
user.ceph._=0sDwj5AAAABAM1AAAAAAAAABQAAAAxMDAwMDQxMTE5NC4wMDAwMDAwMP7/////////6RrNGgAAAAAAAgAAAAAAAAAGAxwAAAACAAAAAAAAAP////8AAAAAAAAAAP//////////AAAAABUn4QAAAAAAu4IAAK4m4QAAAAAAu4IAAAICFQAAAAIAAAAAAAAAAOSZDAAAAAAAsEUDAAAAAAAAAAAAjUoIWUgWsQQCAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVJ+EAAAAAAAAAAAAAAAAAABwAAACNSghZESm8BP///w==
user.ceph._@1=0s//////8=
user.ceph._layout=0sAgIYAAAAAAAAAAAAAAAAAAAA//////////8AAAAA
user.ceph._parent=0sBQRPAQAAlBFBAAABAAAIAAAAAgIjAAAAjxFBAAABAAAPAAAAdHViZWFtYXRldXIubmV0qdgAAAAAAAACAh0AAAB/EUEAAAEAAAkAAAB3cC1yb2NrZXREAAAAAAAAAAICGQAAABYNQQAAAQAABQAAAGNhY2hlUgAAAAAAAAACAh4AAAAQDUEAAAEAAAoAAAB3cC1jb250ZW50NAMAAAAAAAACAhgAAAANDUEAAAEAAAQAAABodG1sIAEAAAAAAAACAikAAADagTMAAAEAABUAAABuZ2lueC1waHA3LWNsdmdmLWRhdGGJAAAAAAAAAAICMwAAADkAAAAAAQ==
user.ceph._parent@1
=0sAAAfAAAANDg4LTU3YjI2NTdmMmZhMTMtbWktcHJveWVjdG8tMXSQCAAAAAAAAgIcAAAAAQAAAAAAAAAIAAAAcHJvamVjdHPBAgcAAAAAAAIAAAAAAAAAAAAAAA==
user.ceph.snapset=0sAgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==
user.cephos.seq=0sAQEQAAAAZaQ9GwAAAAAAAAAAAgAAAAA=
user.cephos.spill_out=0sMAA=getfattr: Removing leading '/' from absolute
path names

# file:
var/lib/ceph/osd/ceph-10/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2
user.ceph._=0sDwj5AAAABAM1AAAAAAAAABQAAAAxMDAwMDQxMTE5NC4wMDAwMDAwMP7/////////6RrNGgAAAAAAAgAAAAAAAAAGAxwAAAACAAAAAAAAAP////8AAAAAAAAAAP//////////AAAAABUn4QAAAAAAu4IAAK4m4QAAAAAAu4IAAAICFQAAAAIAAAAAAAAAAOSZDAAAAAAAsEUDAAAAAAAAAAAAjUoIWUgWsQQCAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVJ+EAAAAAAAAAAAAAAAAAABwAAACNSghZESm8BP///w==
user.ceph._@1=0s//////8=
user.ceph._layout=0sAgIYAAAAAAAAAAAAAAAAAAAA//////////8AAAAA
user.ceph._parent=0sBQRPAQAAlBFBAAABAAAIAAAAAgIjAAAAjxFBAAABAAAPAAAAdHViZWFtYXRldXIubmV0qdgAAAAAAAACAh0AAAB/EUEAAAEAAAkAAAB3cC1yb2NrZXREAAAAAAAAAAICGQAAABYNQQAAAQAABQAAAGNhY2hlUgAAAAAAAAACAh4AAAAQDUEAAAEAAAoAAAB3cC1jb250ZW50NAMAAAAAAAACAhgAAAANDUEAAAEAAAQAAABodG1sIAEAAAAAAAACAikAAADagTMAAAEAABUAAABuZ2lueC1waHA3LWNsdmdmLWRhdGGJAAAAAAAAAAICMwAAADkAAAAAAQ==
user.ceph._parent@1
=0sAAAfAAAANDg4LTU3YjI2NTdmMmZhMTMtbWktcHJveWVjdG8tMXSQCAAAAAAAAgIcAAAAAQAAAAAAAAAIAAAAcHJvamVjdHPBAgcAAAAAAAIAAAAAAAAAAAAAAA==
user.ceph.snapset=0sAgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==
user.cephos.seq=0sAQEQAAAA1T1dEQAAAAAAAAAAAgAAAAA=
user.cephos.spill_out=0sMAA=getfattr: Removing leading '/' from absolute
path names

# file:
var/lib/ceph/osd/ceph-23/current/2.9_head/DIR_9/DIR_E/DIR_A/DIR_1/10000411194.00000000__head_1ACD1AE9__2
user.ceph._=0sDwj5AAAABAM1AAAAAAAAABQAAAAxMDAwMDQxMTE5NC4wMDAwMDAwMP7/////////6RrNGgAAAAAAAgAAAAAAAAAGAxwAAAACAAAAAAAAAP////8AAAAAAAAAAP//////////AAAAABUn4QAAAAAAu4IAAK4m4QAAAAAAu4IAAAICFQAAAAIAAAAAAAAAAOSZDAAAAAAAsEUDAAAAAAAAAAAAjUoIWUgWsQQCAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVJ+EAAAAAAAAAAAAAAAAAABwAAACNSghZESm8BP///w==
user.ceph._@1=0s//////8=
user.ceph._layout=0sAgIYAAAAAAAAAAAAAAAAAAAA//////////8AAAAA
user.ceph._parent=0sBQRPAQAAlBFBAAABAAAIAAAAAgIjAAAAjxFBAAABAAAPAAAAdHViZWFtYXRldXIubmV0qdgAAAAAAAACAh0AAAB/EUEAAAEAAAkAAAB3cC1yb2NrZXREAAAAAAAAAAICGQAAABYNQQAAAQAABQAAAGNhY2hlUgAAAAAAAAACAh4AAAAQDUEAAAEAAAoAAAB3cC1jb250ZW50NAMAAAAAAAACAhgAAAANDUEAAAEAAAQAAABodG1sIAEAAAAAAAACAikAAADagTMAAAEAABUAAABuZ2lueC1waHA3LWNsdmdmLWRhdGGJAAAAAAAAAAICMwAAADkAAAAAAQ==
user.ceph._parent@1
=0sAAAfAAAANDg4LTU3YjI2NTdmMmZhMTMtbWktcHJveWVjdG8tMXSQCAAAAAAAAgIcAAAAAQAAAAAAAAAIAAAAcHJvamVjdHPBAgcAAAAAAAIAAAAAAAAAAAAAAA==
user.ceph.snapset=0sAgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA==
user.cephos.seq=0sAQEQAAAADiM7AAAAAAAAAAAAAgAAAAA=
user.cephos.spill_out=0sMAA=getfattr: Removing leading '/' from absolute
path names


With metadata damage issue, I can get the list of inodes with the command
below.

$ ceph tell mds.0 damage ls | python -m "json.tool"
[
    {
        "damage_type": "dir_frag",
        "frag": "*",
        "id": 5129156,
        "ino": 1099556021325
    },
    {
        "damage_type": "dir_frag",
        "frag": "*",
        "id": 8983971,
        "ino": 1099548098243
    },
    {
        "damage_type": "dir_frag",
        "frag": "*",
        "id": 33278608,
        "ino": 1099548257921
    },
    {
        "damage_type": "dir_frag",
        "frag": "*",
        "id": 33455691,
        "ino": 1099548271575
    },
    {
        "damage_type": "dir_frag",
        "frag": "*",
        "id": 38203788,
        "ino": 1099548134708
    },
...

All of the inodes (approx 800 of them) are for various directories within a
wordpress cache directory.
I ran an rm -rf on each of the directories as I do not need the content.
The content of the directories was removed, but the directories are unable
to be removed as rmdir reports they are not empty, despite having 0 files
listed with ls.

I'm not sure if these two issues are related to each other.  They were
noticed within a day of each other.  I think the metadata damage error
appeared before the scrub error.

I'm at a bit of a loss with how to proceed and I don't want to make things
worse.

I'd really appreciate any help that anyone can give to try and resolve
these problems.

Thanks

J




_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux