On Mon, Jun 3, 2019 at 3:06 PM James Wilkins
<james.wilkins@xxxxxxxxxxxxx> wrote:
>
> Hi all,
>
> After a bit of advice to ensure we’re approaching this the right way.
>
> (version: 12.2.12, multi-mds, dirfrag is enabled)
>
> We have corrupt meta-data as identified by ceph
>
> health: HEALTH_ERR
> 2 MDSs report damaged metadata
>
> Asking the mds via damage ls
>
> {
> "damage_type": "dir_frag",
> "id": 2265410500,
> "ino": 2199349051809,
> "frag": "*",
> "path": "/projects/17343-5bcdaf07f4055-managed-server-0/apache-echfq-data/html/shop/app/cache/prod/smarty/cache/iqitreviews/simple/21832/1"
> }
>
>
> We’ve done the steps outlined here -> http://docs.ceph.com/docs/luminous/cephfs/disaster-recovery/ namely
>
> cephfs-journal-tool –fs:all journal reset (both ranks)
> cephfs-data-scan scan extents / inodes / links has completed
>
> However when attempting to access the named folder we get:
>
> 2019-05-31 03:16:04.792274 7f56f6fb5700 -1 log_channel(cluster) log [ERR] : dir 0x200136b41a1 object missing on disk; some files may be lost (/projects/17343-5bcdaf07f4055-managed-server-0/apache-echfq-data/html/shop/app/cache/prod/smarty/cache/iqitreviews/simple/21832/1)
>
> We get this error followed shortly by an MDS failover
>
> Two questions really
>
> What’s not immediately clear from the documentation is should we/do we also need to run the below?
>
> # Session table
> cephfs-table-tool 0 reset session
> # SnapServer
> cephfs-table-tool 0 reset snap
> # InoTable
> cephfs-table-tool 0 reset inode
> # Root inodes ("/" and MDS directory)
> cephfs-data-scan init
>
No, don't do this.
> And secondly – our current train of thought is we need to grab the inode number of the parent folder and delete this from the metadata pool via rados rmomapkey – is this correct?
>
Yes, find inode number of directory 21832. check if omap key '1_head'
exist in object <inode of directory in hex>.00000000. If it exists,
remove it.
> Any input appreciated
>
> Cheers,
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
