Re: ceph mons and osds are down

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



What does

‘ceph osd tree’ show?

How many OSD’s should you have 7 or 10?

> On 22 Feb 2022, at 14:40, Michel Niyoyita <micou12@xxxxxxxxx> wrote:
> 
> Actually one of my colleagues tried to reboot all nodes and he did not prepare the node like setting noout , norecover ......, once all node are up the cluster is no longer accessible and above are messages we are getting. I did not remove any osd . except are marked down.
> below is my ceph.conf:
> 
> mon initial members = ceph-mon1,ceph-mon2,ceph-mon3
> mon_allow_pool_delete = True
> mon_clock_drift_allowed = 0.5
> mon_max_pg_per_osd = 400
> mon_osd_allow_primary_affinity = 1
> mon_pg_warn_max_object_skew = 0
> mon_pg_warn_max_per_osd = 0
> mon_pg_warn_min_per_osd = 0
> osd pool default crush rule = -1
> osd_pool_default_min_size = 1
> osd_pool_default_size = 2
> public network = 0.0.0.0/0 <http://0.0.0.0/0>
> 
> On Tue, Feb 22, 2022 at 4:32 PM <ashley@xxxxxxxxxxxxxx <mailto:ashley@xxxxxxxxxxxxxx>> wrote:
> You have 1 OSD offline, has this disk failed or you aware of what has caused this to go offline?
> Shows you have 10 OSD’s but only 7in, have you removed the other 3? Was the data fully drained off these first?
> 
> I see you have 11 Pool’s what are these setup as, type and min/max size?
> 
> > On 22 Feb 2022, at 14:15, Michel Niyoyita <micou12@xxxxxxxxx <mailto:micou12@xxxxxxxxx>> wrote:
> > 
> > Dear Ceph Users,
> > 
> > Kindly help me to repair my cluster is down from yesterday up to now I am
> > not able to make it up and running . below are some findings:
> > 
> >    id:     6ad86187-2738-42d8-8eec-48b2a43c298f
> >    health: HEALTH_ERR
> >            mons are allowing insecure global_id reclaim
> >            1/3 mons down, quorum ceph-mon1,ceph-mon3
> >            10/32332 objects unfound (0.031%)
> >            1 osds down
> >            3 scrub errors
> >            Reduced data availability: 124 pgs inactive, 60 pgs down, 411
> > pgs stale
> >            Possible data damage: 9 pgs recovery_unfound, 1 pg
> > backfill_unfound, 1 pg inconsistent
> >            Degraded data redundancy: 6009/64664 objects degraded (9.293%),
> > 55 pgs degraded, 80 pgs undersized
> >            11 pgs not deep-scrubbed in time
> >            5 slow ops, oldest one blocked for 1638 sec, osd.9 has slow ops
> > 
> >  services:
> >    mon: 3 daemons, quorum ceph-mon1,ceph-mon3 (age 3h), out of quorum:
> > ceph-mon2
> >    mgr: ceph-mon1(active, since 9h), standbys: ceph-mon2
> >    osd: 10 osds: 6 up (since 7h), 7 in (since 9h); 43 remapped pgs
> > 
> >  data:
> >    pools:   11 pools, 560 pgs
> >    objects: 32.33k objects, 159 GiB
> >    usage:   261 GiB used, 939 GiB / 1.2 TiB avail
> >    pgs:     11.429% pgs unknown
> >             10.714% pgs not active
> >             6009/64664 objects degraded (9.293%)
> >             1384/64664 objects misplaced (2.140%)
> >             10/32332 objects unfound (0.031%)
> >             245 stale+active+clean
> >             70  active+clean
> >             64  unknown
> >             48  stale+down
> >             45  stale+active+undersized+degraded
> >             37  stale+active+clean+remapped
> >             28  stale+active+undersized
> >             12  down
> >             2   stale+active+recovery_unfound+degraded
> >             2   stale+active+recovery_unfound+undersized+degraded
> >             2   stale+active+recovery_unfound+undersized+degraded+remapped
> >             2   active+recovery_unfound+undersized+degraded+remapped
> >             1   active+clean+inconsistent
> >             1   stale+active+recovery_unfound+degraded+remapped
> >             1   stale+active+backfill_unfound+undersized+degraded+remapped
> > 
> > If someone faced same issue please help me.
> > 
> > Best Regards.
> > 
> > Michel
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx <mailto:ceph-users-leave@xxxxxxx>
> 
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx <mailto:ceph-users-leave@xxxxxxx>

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux