Hi, Please share “ceph -s” output. - Etienne Menguy etienne.menguy@xxxxxxxx > On 29 Oct 2021, at 10:03, Michel Niyoyita <micou12@xxxxxxxxx> wrote: > > Hello team > > I am running a ceph cluster with 3 monitors and 4 OSDs nodes running 3osd > each , I deployed my ceph cluster using ansible and ubuntu 20.04 as OS , > the ceph version is Octopus. yesterday , My server which hosts OSDs nodes > restarted because of power issue and to comeback on its status one of the > monitor is out of quorum and some Pg marks as damaged . please help me to > solve this issue. below are health detail status I am finding. and the 4 > OSDs node are the same which are running monitors (3 of them). > > Best regards. > > Michel > > > root@ceph-mon1:~# ceph health detail > HEALTH_ERR 1/3 mons down, quorum ceph-mon1,ceph-mon3; 14/47195 objects > unfound (0.030%); Possible data damage: 13 pgs recovery_unfound; Degraded > data redundancy: 42/141585 objects degraded (0.030%), 13 pgs degraded; 2 > slow ops, oldest one blocked for 322 sec, daemons [osd.0,osd.7] have slow > ops. > [WRN] MON_DOWN: 1/3 mons down, quorum ceph-mon1,ceph-mon3 > mon.ceph-mon4 (rank 2) addr [v2: > 10.10.29.154:3300/0,v1:10.10.29.154:6789/0] is down (out of quorum) > [WRN] OBJECT_UNFOUND: 14/47195 objects unfound (0.030%) > pg 5.77 has 1 unfound objects > pg 5.6d has 2 unfound objects > pg 5.6a has 1 unfound objects > pg 5.65 has 1 unfound objects > pg 5.4a has 1 unfound objects > pg 5.30 has 1 unfound objects > pg 5.28 has 1 unfound objects > pg 5.25 has 1 unfound objects > pg 5.19 has 1 unfound objects > pg 5.1a has 1 unfound objects > pg 5.1 has 1 unfound objects > pg 5.b has 1 unfound objects > pg 5.8 has 1 unfound objects > [ERR] PG_DAMAGED: Possible data damage: 13 pgs recovery_unfound > pg 5.1 is active+recovery_unfound+degraded+remapped, acting [5,8,7], 1 > unfound > pg 5.8 is active+recovery_unfound+degraded+remapped, acting [6,11,8], 1 > unfound > pg 5.b is active+recovery_unfound+degraded+remapped, acting [7,0,5], 1 > unfound > pg 5.19 is active+recovery_unfound+degraded+remapped, acting [0,5,7], 1 > unfound > pg 5.1a is active+recovery_unfound+degraded, acting [10,11,8], 1 unfound > pg 5.25 is active+recovery_unfound+degraded+remapped, acting [0,10,11], > 1 unfound > pg 5.28 is active+recovery_unfound+degraded+remapped, acting [6,11,8], > 1 unfound > pg 5.30 is active+recovery_unfound+degraded+remapped, acting [7,5,0], 1 > unfound > pg 5.4a is active+recovery_unfound+degraded, acting [0,11,7], 1 unfound > pg 5.65 is active+recovery_unfound+degraded+remapped, acting [0,10,11], > 1 unfound > pg 5.6a is active+recovery_unfound+degraded, acting [0,11,7], 1 unfound > pg 5.6d is active+recovery_unfound+degraded+remapped, acting [7,2,0], 2 > unfound > pg 5.77 is active+recovery_unfound+degraded+remapped, acting [5,6,8], 1 > unfound > [WRN] PG_DEGRADED: Degraded data redundancy: 42/141585 objects degraded > (0.030%), 13 pgs degraded > pg 5.1 is active+recovery_unfound+degraded+remapped, acting [5,8,7], 1 > unfound > pg 5.8 is active+recovery_unfound+degraded+remapped, acting [6,11,8], 1 > unfound > pg 5.b is active+recovery_unfound+degraded+remapped, acting [7,0,5], 1 > unfound > pg 5.19 is active+recovery_unfound+degraded+remapped, acting [0,5,7], 1 > unfound > pg 5.1a is active+recovery_unfound+degraded, acting [10,11,8], 1 unfound > pg 5.25 is active+recovery_unfound+degraded+remapped, acting [0,10,11], > 1 unfound > pg 5.28 is active+recovery_unfound+degraded+remapped, acting [6,11,8], > 1 unfound > pg 5.30 is active+recovery_unfound+degraded+remapped, acting [7,5,0], 1 > unfound > pg 5.4a is active+recovery_unfound+degraded, acting [0,11,7], 1 unfound > pg 5.65 is active+recovery_unfound+degraded+remapped, acting [0,10,11], > 1 unfound > pg 5.6a is active+recovery_unfound+degraded, acting [0,11,7], 1 unfound > pg 5.6d is active+recovery_unfound+degraded+remapped, acting [7,2,0], 2 > unfound > pg 5.77 is active+recovery_unfound+degraded+remapped, acting [5,6,8], 1 > unfound > [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 322 sec, daemons > [osd.0,osd.7] have slow ops. > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |