Re: Remove an OSD with hardware issue caused rgw 503

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Sounds good. Thank you Kevin and have a nice day!

Best Regards,
Mary

On Tue, Apr 30, 2024, 8:21 AM Frank Schilder <frans@xxxxxx> wrote:

> I think you are panicking way too much. Chances are that you will never
> need that command, so don't get fussed out by an old post.
>
> Just follow what I wrote and, in the extremely rare case that recovery
> does not complete due to missing information, send an e-mail to this list
> and state that you still have the disk of the down OSD. Someone will send
> you the export/import commands within a short time.
>
> So stop worrying and just administrate your cluster with common storage
> admin sense.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Mary Zhang <maryzhang0920@xxxxxxxxx>
> Sent: Tuesday, April 30, 2024 5:00 PM
> To: Frank Schilder
> Cc: Eugen Block; ceph-users@xxxxxxx; Wesley Dillingham
> Subject: Re:  Re: Remove an OSD with hardware issue caused rgw
> 503
>
> Thank you Frank for sharing such valuable experience! I really appreciate
> it.
> We observe similar timelines: it took more than 1 week to drain our OSD.
> Regarding export PGs from failed disk and inject it back to the cluster,
> do you have any documentations? I find this online Ceph.io — Incomplete PGs
> -- OH MY!<https://ceph.io/en/news/blog/2015/incomplete-pgs-oh-my/>, but
> not sure whether it's the standard process.
>
> Thanks,
> Mary
>
> On Tue, Apr 30, 2024 at 3:27 AM Frank Schilder <frans@xxxxxx<mailto:
> frans@xxxxxx>> wrote:
> Hi all,
>
> I second Eugen's recommendation. We have a cluster with large HDD OSDs
> where the following timings are found:
>
> - drain an OSD: 2 weeks.
> - down an OSD and let cluster recover: 6 hours.
>
> The drain OSD procedure is - in my experience - a complete waste of time,
> actually puts your cluster at higher risk of a second failure (its not
> guaranteed that the bad PG(s) is/are drained first) and also screws up all
> sorts of internal operations like scrub etc for an unnecessarily long time.
> The recovery procedure is much faster, because it uses all-to-all recovery
> while drain is limited to no more than max_backfills PGs at a time and your
> broken disk sits much longer in the cluster.
>
> On SSDs the "down OSD"-method shows a similar speed-up factor.
>
> For a security measure, don't destroy the OSD right away, wait for
> recovery to complete and only then destroy the OSD and throw away the disk.
> In case an error occurs during recovery, you can almost always still export
> PGs from a failed disk and inject it back into the cluster. This, however,
> requires to take disks out as soon as they show problems and before they
> fail hard. Keep a little bit of life time to have a chance to recover data.
> Look at the manual of ddrescue why it is important to stop IO from a
> failing disk as soon as possible.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Eugen Block <eblock@xxxxxx<mailto:eblock@xxxxxx>>
> Sent: Saturday, April 27, 2024 10:29 AM
> To: Mary Zhang
> Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>; Wesley Dillingham
> Subject:  Re: Remove an OSD with hardware issue caused rgw 503
>
> If the rest of the cluster is healthy and your resiliency is
> configured properly, for example to sustain the loss of one or more
> hosts at a time, you don’t need to worry about a single disk. Just
> take it out and remove it (forcefully) so it doesn’t have any clients
> anymore. Ceph will immediately assign different primary OSDs and your
> clients will be happy again. ;-)
>
> Zitat von Mary Zhang <maryzhang0920@xxxxxxxxx<mailto:
> maryzhang0920@xxxxxxxxx>>:
>
> > Thank you Wesley for the clear explanation between the 2 methods!
> > The tracker issue you mentioned https://tracker.ceph.com/issues/44400
> talks
> > about primary-affinity. Could primary-affinity help remove an OSD with
> > hardware issue from the cluster gracefully?
> >
> > Thanks,
> > Mary
> >
> >
> > On Fri, Apr 26, 2024 at 8:43 AM Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx
> <mailto:wes@xxxxxxxxxxxxxxxxx>>
> > wrote:
> >
> >> What you want to do is to stop the OSD (and all its copies of data it
> >> contains) by stopping the OSD service immediately. The downside of this
> >> approach is it causes the PGs on that OSD to be degraded. But the
> upside is
> >> the OSD which has bad hardware is immediately no  longer participating
> in
> >> any client IO (the source of your RGW 503s). In this situation the PGs
> go
> >> into degraded+backfilling
> >>
> >> The alternative method is to keep the failing OSD up and in the cluster
> >> but slowly migrate the data off of it, this would be a long drawn out
> >> period of time in which the failing disk would continue to serve client
> >> reads and also facilitate backfill but you wouldnt take a copy of the
> data
> >> out of the cluster and cause degraded PGs. In this scenario the PGs
> would
> >> be remapped+backfilling
> >>
> >> I tried to find a way to have your cake and eat it to in relation to
> this
> >> "predicament" in this tracker issue:
> https://tracker.ceph.com/issues/44400
> >> but it was deemed "wont fix".
> >>
> >> Respectfully,
> >>
> >> *Wes Dillingham*
> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> >> wes@xxxxxxxxxxxxxxxxx<mailto:wes@xxxxxxxxxxxxxxxxx>
> >>
> >>
> >>
> >>
> >> On Fri, Apr 26, 2024 at 11:25 AM Mary Zhang <maryzhang0920@xxxxxxxxx
> <mailto:maryzhang0920@xxxxxxxxx>>
> >> wrote:
> >>
> >>> Thank you Eugen for your warm help!
> >>>
> >>> I'm trying to understand the difference between 2 methods.
> >>> For method 1, or "ceph orch osd rm osd_id", OSD Service — Ceph
> >>> Documentation
> >>> <https://docs.ceph.com/en/latest/cephadm/services/osd/#remove-an-osd>
> >>> says
> >>> it involves 2 steps:
> >>>
> >>>    1.
> >>>
> >>>    evacuating all placement groups (PGs) from the OSD
> >>>    2.
> >>>
> >>>    removing the PG-free OSD from the cluster
> >>>
> >>> For method 2, or the procedure you recommended, Adding/Removing OSDs —
> >>> Ceph
> >>> Documentation
> >>> <
> >>>
> https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/#removing-osds-manual
> >>> >
> >>> says
> >>> "After the OSD has been taken out of the cluster, Ceph begins
> rebalancing
> >>> the cluster by migrating placement groups out of the OSD that was
> removed.
> >>> "
> >>>
> >>> What's the difference between "evacuating PGs" in method 1 and
> "migrating
> >>> PGs" in method 2? I think method 1 must read the OSD to be removed.
> >>> Otherwise, we would not see slow ops warning. Does method 2 not involve
> >>> reading this OSD?
> >>>
> >>> Thanks,
> >>> Mary
> >>>
> >>> On Fri, Apr 26, 2024 at 5:15 AM Eugen Block <eblock@xxxxxx<mailto:
> eblock@xxxxxx>> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > if you remove the OSD this way, it will be drained. Which means that
> >>> > it will try to recover PGs from this OSD, and in case of hardware
> >>> > failure it might lead to slow requests. It might make sense to
> >>> > forcefully remove the OSD without draining:
> >>> >
> >>> > - stop the osd daemon
> >>> > - mark it as out
> >>> > - osd purge <id|osd.id<http://osd.id>> [--force]
> [--yes-i-really-mean-it]
> >>> >
> >>> > Regards,
> >>> > Eugen
> >>> >
> >>> > Zitat von Mary Zhang <maryzhang0920@xxxxxxxxx<mailto:
> maryzhang0920@xxxxxxxxx>>:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > We recently removed an osd from our Cepth cluster. Its underlying
> disk
> >>> > has
> >>> > > a hardware issue.
> >>> > >
> >>> > > We use command: ceph orch osd rm osd_id --zap
> >>> > >
> >>> > > During the process, sometimes ceph cluster enters warning state
> with
> >>> slow
> >>> > > ops on this osd. Our rgw also failed to respond to requests and
> >>> returned
> >>> > > 503.
> >>> > >
> >>> > > We restarted rgw daemon to make it work again. But the same failure
> >>> > occured
> >>> > > from time to time. Eventually we noticed that rgw 503 error is a
> >>> result
> >>> > of
> >>> > > osd slow ops.
> >>> > >
> >>> > > Our cluster has 18 hosts and 210 OSDs. We expect remove an osd with
> >>> > > hardware issue won't impact cluster performance & rgw availbility.
> Is
> >>> our
> >>> > > expectation reasonable? What's the best way to handle osd with
> >>> hardware
> >>> > > failures?
> >>> > >
> >>> > > Thank you in advance for any comments or suggestions.
> >>> > >
> >>> > > Best Regards,
> >>> > > Mary Zhang
> >>> > > _______________________________________________
> >>> > > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:
> ceph-users@xxxxxxx>
> >>> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx>
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:
> ceph-users@xxxxxxx>
> >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx>
> >>> >
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx<mailto:
> ceph-users@xxxxxxx>
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx>
> >>>
> >>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux