Re: Very slow backfilling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Forgot to do a reply all.

What does

ceph osd df
ceph osd dump | grep pool return?

Are you using auto scaling? 289pg with 272tb of data and 60 osds, that
seems like 3-4 pg per osd at almost 1TB each. Unless I'm thinking of this
wrong.

On Thu, Mar 2, 2023, 17:37 Joffrey <joff.au@xxxxxxxxx> wrote:

> My Ceph Version is 17.2.5 and all configuration about osd_scrub* are
> defaults. I tried some updates on osd-max-backfills but no change.
> I have many HDD with NVME for db and all are connected in a 25G network.
>
> Yes, it's the same PG since 4 days.
>
> I got a failure on a HDD and get many days of recovery+backfilling last  2
> weeks.   Perhaps the 'not in time' warning is related to this.
>
> 'Jof
>
> Le jeu. 2 mars 2023 à 14:25, Anthony D'Atri <aad@xxxxxxxxxxxxxx> a écrit :
>
> > Run `ceph health detail`.
> >
> > Is it the same PG backfilling for a long time, or a different one over
> > time?
> >
> > That it’s remapped makes me think that what you’re seeing is the balancer
> > doing its job.
> >
> > As far as the scrubbing, do you limit the times when scrubbing can
> happen?
> > Are these HDDs? EC?
> >
> > > On Mar 2, 2023, at 07:20, Joffrey <joff.au@xxxxxxxxx> wrote:
> > >
> > > Hi,
> > >
> > > I have many 'not {deep-}scrubbed in time' and a1 PG
> remapped+backfilling
> > > and I don't understand why this backfilling is taking so long.
> > >
> > > root@hbgt-ceph1-mon3:/# ceph -s
> > >  cluster:
> > >    id:     c300532c-51fa-11ec-9a41-0050569c3b55
> > >    health: HEALTH_WARN
> > >            15 pgs not deep-scrubbed in time
> > >            13 pgs not scrubbed in time
> > >
> > >  services:
> > >    mon: 3 daemons, quorum
> hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3
> > > (age 36h)
> > >    mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys:
> > > hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm
> > >    osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs
> > >    rgw: 3 daemons active (3 hosts, 2 zones)
> > >
> > >  data:
> > >    pools:   13 pools, 289 pgs
> > >    objects: 67.74M objects, 127 TiB
> > >    usage:   272 TiB used, 769 TiB / 1.0 PiB avail
> > >    pgs:     288 active+clean
> > >             1   active+remapped+backfilling
> > >
> > >  io:
> > >    client:   3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr
> > >    recovery: 790 KiB/s, 0 objects/s
> > >
> > >
> > > What can I do to understand this slow recovery (is it the backfill
> > action ?)
> > >
> > > Thanks you
> > >
> > > 'Jof
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux