Hi all An accidental power failure happened. That resulted CephFS offline and cannot be mounted. I have 3 MDS daemons but it complains "1 mds daemon damaged". It seems a PG of cephfs_metadata is inconsistent. I tried to repair, but doesn't get it repaired. How do I repair the damaged MDS and bring the CephFS up/online? Details are included below. Many thanks in advance. Sagara # ceph -s cluster: id: abc... health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 4 scrub errors Possible data damage: 1 pg inconsistent services: mon: 3 daemons, quorum a,b,c (age 107s) mgr: a(active, since 22m), standbys: b, c mds: cephfs:0/1 3 up:standby, 1 damaged osd: 3 osds: 3 up (since 96s), 3 in (since 96s) data: pools: 3 pools, 192 pgs objects: 281.05k objects, 327 GiB usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail pgs: 191 active+clean 1 active+clean+inconsistent # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged; 4 scrub errors; Possible data damage: 1 pg inconsistent FS_DEGRADED 1 filesystem is degraded fs cephfs is degraded MDS_ALL_DOWN 1 filesystem is offline fs cephfs is offline because no MDS is active for it. MDS_DAMAGE 1 mds daemon damaged fs cephfs mds.0 is damaged OSD_SCRUB_ERRORS 4 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 2.44 is active+clean+inconsistent, acting [0,2,1] # ceph osd lspools 2 cephfs_metadata 3 cephfs_data 4 rbd # ceph pg repair 2.44 # ceph -w 2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid 2:22efaf6a:::200.00006048:head : candidate size 1441792 info size 1555896 mismatch 2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid 2:22efaf6a:::200.00006048:head : failed to pick suitable object info 2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44 2:22efaf6a:::200.00006048:head : on disk size (1540096) does not match object info size (1555896) adjusted for ondisk to (1555896) 2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed --- End of detail --- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |