Re: CEPH MDS Damaged Metadata - recovery steps

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



(Thanks Yan for confirming fix - we'll implement now)

@Marc

Yep - x3 replica on meta-data pools

We have 4 clusters (all running same version) and have experienced meta-data corruption on the majority of them at some time or the other - normally a scan fixes - I suspect due to the use case - think LAMP stacks with various drupal/wordpress caching plugins - which are running within openshift containers and utilising CephFS as a storage backend.  These clusters have all been life-cycled up from Jewel if that matters.


Example;

# ceph osd dump | grep metadata
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 528539 flags hashpspool stripe_width 0 application cephfs

Only other thing of note is on this particular cluster the meta-data pool is quite large for the number of files - see below re 281GB - other clusters metadata is a lot smaller for similar dataset.

# ceph df
GLOBAL:
    SIZE        AVAIL       RAW USED     %RAW USED
    66.9TiB     29.1TiB      37.8TiB         56.44
POOLS:
    NAME                ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                 0      8.86TiB     64.57       4.86TiB      2637530
    cephfs_data         1      2.59TiB     34.72       4.86TiB     25341863
    cephfs_metadata     2       281GiB      5.34       4.86TiB      6755178

Cheers,

James

On 04/06/2019, 08:59, "Marc Roos" <M.Roos@xxxxxxxxxxxxxxxxx> wrote:

     
    How did this get damaged? You had 3x replication on the pool?
    
    
    
    -----Original Message-----
    From: Yan, Zheng [mailto:ukernel@xxxxxxxxx] 
    Sent: dinsdag 4 juni 2019 1:14
    To: James Wilkins
    Cc: ceph-users
    Subject: Re:  CEPH MDS Damaged Metadata - recovery steps
    
    On Mon, Jun 3, 2019 at 3:06 PM James Wilkins 
    <james.wilkins@xxxxxxxxxxxxx> wrote:
    >
    > Hi all,
    >
    > After a bit of advice to ensure we’re approaching this the right way.
    >
    > (version: 12.2.12, multi-mds, dirfrag is enabled)
    >
    > We have corrupt meta-data as identified by ceph
    >
    >     health: HEALTH_ERR
    >             2 MDSs report damaged metadata
    >
    > Asking the mds via damage ls
    >
    >     {
    >         "damage_type": "dir_frag",
    >         "id": 2265410500,
    >         "ino": 2199349051809,
    >         "frag": "*",
    >         "path": 
    "/projects/17343-5bcdaf07f4055-managed-server-0/apache-echfq-data/html/s
    hop/app/cache/prod/smarty/cache/iqitreviews/simple/21832/1"
    >     }
    >
    >
    > We’ve done the steps outlined here -> 
    > http://docs.ceph.com/docs/luminous/cephfs/disaster-recovery/ namely
    >
    > cephfs-journal-tool –fs:all journal reset (both ranks) 
    > cephfs-data-scan scan extents / inodes / links has completed
    >
    > However when attempting to access the named folder we get:
    >
    > 2019-05-31 03:16:04.792274 7f56f6fb5700 -1 log_channel(cluster) log 
    > [ERR] : dir 0x200136b41a1 object missing on disk; some files may be 
    > lost 
    > (/projects/17343-5bcdaf07f4055-managed-server-0/apache-echfq-data/html
    > /shop/app/cache/prod/smarty/cache/iqitreviews/simple/21832/1)
    >
    > We get this error followed shortly by an MDS failover
    >
    > Two questions really
    >
    > What’s not immediately clear from the documentation is should we/do 
    we also need to run the below?
    >
    > # Session table
    > cephfs-table-tool 0 reset session
    > # SnapServer
    > cephfs-table-tool 0 reset snap
    > # InoTable
    > cephfs-table-tool 0 reset inode
    > # Root inodes ("/" and MDS directory)
    > cephfs-data-scan init
    >
    
    No, don't do this.
    
    > And secondly – our current train of thought is we need to grab the 
    inode number of the parent folder and delete this from the metadata pool 
    via rados rmomapkey – is this correct?
    >
    
    Yes, find inode number of directory 21832. check if omap key '1_head'
    exist in object <inode of directory in hex>.00000000. If it exists, 
    remove it.
    
    > Any input appreciated
    >
    > Cheers,
    >
    >
    > _______________________________________________
    > ceph-users mailing list
    > ceph-users@xxxxxxxxxxxxxx
    > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
    _______________________________________________
    ceph-users mailing list
    ceph-users@xxxxxxxxxxxxxx
    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
    
    
    

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux