Re: Ceph recovery network speed

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Jun 27, 2022 at 11:08 PM Frank Schilder <frans@xxxxxx> wrote:

> Do you, by any chance have SMR drives? This may not be stated on the
> drive, check what the internet has to say. I also would have liked to see
> the beginning of the ceph status, number of hosts, number of OSDs, up in
> down whatever. Can you also send the result of ceph osd df tree?
>
     As far as I can tell none of the drives are SMR drives.
I did have some inconsistent pop up, scrubs are still running.

cluster:
    id:     1684fe88-aae0-11ec-9593-df430e3982a0
    health: HEALTH_ERR
            10 scrub errors
            Possible data damage: 4 pgs inconsistent

  services:
    mon: 5 daemons, quorum cephmgr,cephmon1,cephmon2,cephmon3,cephmgr2 (age
8w)
    mgr: cephmon1.fxtvtu(active, since 2d), standbys: cephmon2.wrzwwn,
cephmgr2.hzsrdo, cephmgr.bazebq
    osd: 44 osds: 44 up (since 3d), 44 in (since 3d); 28 remapped pgs
    rgw: 2 daemons active (2 hosts, 1 zones)

  data:
    pools:   11 pools, 369 pgs
    objects: 2.45M objects, 9.2 TiB
    usage:   21 TiB used, 59 TiB / 80 TiB avail
    pgs:     503944/9729081 objects misplaced (5.180%)
             337 active+clean
             28  active+remapped+backfilling
             4   active+clean+inconsistent

  io:
    client:   1000 KiB/s rd, 717 KiB/s wr, 81 op/s rd, 57 op/s wr
    recovery: 34 MiB/s, 8 objects/s

ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE   DATA      OMAP     META
  AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
-1         80.05347         -   80 TiB    21 TiB    21 TiB   32 MiB   69
GiB   59 TiB  26.23  1.00    -          root default
-5         20.01337         -   20 TiB   5.3 TiB   5.3 TiB  1.4 MiB   19
GiB   15 TiB  26.47  1.01    -              host hyperion01
 1    hdd   1.81940   1.00000  1.8 TiB   749 GiB   747 GiB  224 KiB  2.2
GiB  1.1 TiB  40.19  1.53   36      up          osd.1
 3    hdd   1.81940   1.00000  1.8 TiB   531 GiB   530 GiB    3 KiB  1.9
GiB  1.3 TiB  28.52  1.09   31      up          osd.3
 5    hdd   1.81940   1.00000  1.8 TiB   167 GiB   166 GiB   36 KiB  1.2
GiB  1.7 TiB   8.98  0.34   18      up          osd.5
 7    hdd   1.81940   1.00000  1.8 TiB   318 GiB   316 GiB   83 KiB  1.2
GiB  1.5 TiB  17.04  0.65   26      up          osd.7
 9    hdd   1.81940   1.00000  1.8 TiB  1017 GiB  1014 GiB  139 KiB  2.6
GiB  846 GiB  54.59  2.08   38      up          osd.9
11    hdd   1.81940   1.00000  1.8 TiB   569 GiB   567 GiB    4 KiB  2.1
GiB  1.3 TiB  30.56  1.17   29      up          osd.11
13    hdd   1.81940   1.00000  1.8 TiB   293 GiB   291 GiB  338 KiB  1.5
GiB  1.5 TiB  15.72  0.60   23      up          osd.13
15    hdd   1.81940   1.00000  1.8 TiB   368 GiB   366 GiB  641 KiB  1.6
GiB  1.5 TiB  19.74  0.75   23      up          osd.15
17    hdd   1.81940   1.00000  1.8 TiB   369 GiB   367 GiB    2 KiB  1.5
GiB  1.5 TiB  19.80  0.75   26      up          osd.17
19    hdd   1.81940   1.00000  1.8 TiB   404 GiB   403 GiB    7 KiB  1.1
GiB  1.4 TiB  21.69  0.83   31      up          osd.19
45    hdd   1.81940   1.00000  1.8 TiB   639 GiB   637 GiB    2 KiB  2.0
GiB  1.2 TiB  34.30  1.31   32      up          osd.45
-3         20.01337         -   20 TiB   5.2 TiB   5.2 TiB  2.0 MiB   18
GiB   15 TiB  26.15  1.00    -              host hyperion02
 0    hdd   1.81940   1.00000  1.8 TiB   606 GiB   604 GiB  302 KiB  2.0
GiB  1.2 TiB  32.52  1.24   33      up          osd.0
 2    hdd   1.81940   1.00000  1.8 TiB    58 GiB    58 GiB  112 KiB  249
MiB  1.8 TiB   3.14  0.12   14      up          osd.2
 4    hdd   1.81940   1.00000  1.8 TiB   254 GiB   252 GiB   14 KiB  1.6
GiB  1.6 TiB  13.63  0.52   28      up          osd.4
 6    hdd   1.81940   1.00000  1.8 TiB   574 GiB   572 GiB    1 KiB  1.8
GiB  1.3 TiB  30.81  1.17   26      up          osd.6
 8    hdd   1.81940   1.00000  1.8 TiB   201 GiB   200 GiB  618 KiB  743
MiB  1.6 TiB  10.77  0.41   23      up          osd.8
10    hdd   1.81940   1.00000  1.8 TiB   628 GiB   626 GiB    4 KiB  2.2
GiB  1.2 TiB  33.72  1.29   37      up          osd.10
12    hdd   1.81940   1.00000  1.8 TiB   355 GiB   353 GiB  361 KiB  1.2
GiB  1.5 TiB  19.03  0.73   30      up          osd.12
14    hdd   1.81940   1.00000  1.8 TiB   1.1 TiB   1.1 TiB    1 KiB  2.7
GiB  708 GiB  62.00  2.36   38      up          osd.14
16    hdd   1.81940   1.00000  1.8 TiB   240 GiB   239 GiB    4 KiB  1.2
GiB  1.6 TiB  12.90  0.49   20      up          osd.16
18    hdd   1.81940   1.00000  1.8 TiB   300 GiB   298 GiB  542 KiB  1.6
GiB  1.5 TiB  16.08  0.61   21      up          osd.18
32    hdd   1.81940   1.00000  1.8 TiB   989 GiB   986 GiB   45 KiB  2.7
GiB  874 GiB  53.09  2.02   36      up          osd.32
-7         20.01337         -   20 TiB   5.2 TiB   5.2 TiB  2.9 MiB   17
GiB   15 TiB  26.06  0.99    -              host hyperion03
22    hdd   1.81940   1.00000  1.8 TiB   449 GiB   448 GiB  443 KiB  1.5
GiB  1.4 TiB  24.10  0.92   31      up          osd.22
23    hdd   1.81940   1.00000  1.8 TiB   299 GiB   298 GiB    5 KiB  1.4
GiB  1.5 TiB  16.05  0.61   26      up          osd.23
24    hdd   1.81940   1.00000  1.8 TiB   735 GiB   733 GiB    8 KiB  2.3
GiB  1.1 TiB  39.45  1.50   33      up          osd.24
25    hdd   1.81940   1.00000  1.8 TiB   519 GiB   517 GiB    5 KiB  1.4
GiB  1.3 TiB  27.85  1.06   26      up          osd.25
26    hdd   1.81940   1.00000  1.8 TiB   483 GiB   481 GiB  614 KiB  1.7
GiB  1.3 TiB  25.94  0.99   28      up          osd.26
27    hdd   1.81940   1.00000  1.8 TiB   226 GiB   225 GiB  1.5 MiB  1.0
GiB  1.6 TiB  12.11  0.46   17      up          osd.27
28    hdd   1.81940   1.00000  1.8 TiB   443 GiB   441 GiB   24 KiB  1.5
GiB  1.4 TiB  23.76  0.91   21      up          osd.28
29    hdd   1.81940   1.00000  1.8 TiB   801 GiB   799 GiB    7 KiB  2.2
GiB  1.0 TiB  42.98  1.64   31      up          osd.29
30    hdd   1.81940   1.00000  1.8 TiB   523 GiB   522 GiB  174 KiB  1.2
GiB  1.3 TiB  28.09  1.07   29      up          osd.30
31    hdd   1.81940   1.00000  1.8 TiB   322 GiB   321 GiB    4 KiB  1.2
GiB  1.5 TiB  17.30  0.66   26      up          osd.31
44    hdd   1.81940   1.00000  1.8 TiB   541 GiB   540 GiB  136 KiB  1.4
GiB  1.3 TiB  29.06  1.11   24      up          osd.44
-9         20.01337         -   20 TiB   5.3 TiB   5.2 TiB   25 MiB   16
GiB   15 TiB  26.25  1.00    -              host hyperion04
33    hdd   1.81940   1.00000  1.8 TiB   466 GiB   465 GiB  469 KiB  1.4
GiB  1.4 TiB  25.02  0.95   28      up          osd.33
34    hdd   1.81940   1.00000  1.8 TiB   508 GiB   506 GiB    2 KiB  1.8
GiB  1.3 TiB  27.28  1.04   30      up          osd.34
35    hdd   1.81940   1.00000  1.8 TiB   521 GiB   520 GiB    2 KiB  1.4
GiB  1.3 TiB  27.98  1.07   32      up          osd.35
36    hdd   1.81940   1.00000  1.8 TiB   872 GiB   870 GiB    3 KiB  2.3
GiB  991 GiB  46.81  1.78   40      up          osd.36
37    hdd   1.81940   1.00000  1.8 TiB   443 GiB   441 GiB  136 KiB  1.2
GiB  1.4 TiB  23.75  0.91   25      up          osd.37
38    hdd   1.81940   1.00000  1.8 TiB   138 GiB   137 GiB   24 MiB  647
MiB  1.7 TiB   7.40  0.28   27      up          osd.38
39    hdd   1.81940   1.00000  1.8 TiB   638 GiB   637 GiB  622 KiB  1.7
GiB  1.2 TiB  34.26  1.31   33      up          osd.39
40    hdd   1.81940   1.00000  1.8 TiB   444 GiB   443 GiB   14 KiB  1.4
GiB  1.4 TiB  23.85  0.91   25      up          osd.40
41    hdd   1.81940   1.00000  1.8 TiB   477 GiB   476 GiB  264 KiB  1.3
GiB  1.4 TiB  25.60  0.98   31      up          osd.41
42    hdd   1.81940   1.00000  1.8 TiB   514 GiB   513 GiB   35 KiB  1.2
GiB  1.3 TiB  27.61  1.05   29      up          osd.42
43    hdd   1.81940   1.00000  1.8 TiB   358 GiB   356 GiB  111 KiB  1.2
GiB  1.5 TiB  19.19  0.73   24      up          osd.43
                        TOTAL   80 TiB    21 TiB    21 TiB   32 MiB   69
GiB   59 TiB  26.23
MIN/MAX VAR: 0.12/2.36  STDDEV: 12.47

>
> The number of objects in flight looks small. Your objects seem to have an
> average size of 4MB and should recover with full bandwidth. Check with top
> how much IO wait percentage you have on the OSD hosts.
>
iowait is 3.3% and load avg is 3.7, nothing crazy from what I can tell.


>
> The one thing that jumps to my eye though is, that you only have 22 dirty
> PGs and they are all recovering/backfilling already. I wonder if you have a
> problem with your crush rules, they might not do what you think they do.
> You said you increased the PG count for EC-22-Pool to 128 (from what?) but
> it doesn't really look like a suitable number of PGs has been marked for
> backfilling. Can you post the output of "ceph osd pool get EC-22-Pool all"?
>
>From 32 to 128
ceph osd pool get EC-22-Pool all
size: 4
min_size: 3
pg_num: 128
pgp_num: 48
crush_rule: EC-22-Pool
hashpspool: true
allow_ec_overwrites: true
nodelete: false
nopgchange: false
nosizechange: false
write_fadvise_dontneed: false
noscrub: false
nodeep-scrub: false
use_gmt_hitset: 1
erasure_code_profile: EC-22-Pro
fast_read: 0
pg_autoscale_mode: on
eio: false
bulk: false



>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Curt <lightspd@xxxxxxxxx>
> Sent: 27 June 2022 19:41:06
> To: Robert Gallop
> Cc: Frank Schilder; ceph-users@xxxxxxx
> Subject: Re:  Re: Ceph recovery network speed
>
> I would love to see those types of speeds. I tried setting it all the way
> to 0 and nothing, I did that before I sent the first email, maybe it was
> your old post I got it from.
>
> osd_recovery_sleep_hdd                           0.000000
>
>
>                                  override  (mon[0.000000])
>
> On Mon, Jun 27, 2022 at 9:27 PM Robert Gallop <robert.gallop@xxxxxxxxx
> <mailto:robert.gallop@xxxxxxxxx>> wrote:
> I saw a major boost after having the sleep_hdd set to 0.  Only after that
> did I start staying at around 500MiB to 1.2GiB/sec and 1.5k obj/sec to 2.5k
> obj/sec.
>
> Eventually it tapered back down, but for me sleep was the key, and
> specifically in my case:
>
> osd_recovery_sleep_hdd
>
> On Mon, Jun 27, 2022 at 11:17 AM Curt <lightspd@xxxxxxxxx<mailto:
> lightspd@xxxxxxxxx>> wrote:
> On Mon, Jun 27, 2022 at 8:52 PM Frank Schilder <frans@xxxxxx<mailto:
> frans@xxxxxx>> wrote:
>
> > I think this is just how ceph is. Maybe you should post the output of
> > "ceph status", "ceph osd pool stats" and "ceph df" so that we can get an
> > idea whether what you look at is expected or not. As I wrote before,
> object
> > recovery is throttled and the recovery bandwidth depends heavily on
> object
> > size. The interesting question is, how many objects per second are
> > recovered/rebalanced
> >
>  data:
>     pools:   11 pools, 369 pgs
>     objects: 2.45M objects, 9.2 TiB
>     usage:   20 TiB used, 60 TiB / 80 TiB avail
>     pgs:     512136/9729081 objects misplaced (5.264%)
>              343 active+clean
>              22  active+remapped+backfilling
>
>   io:
>     client:   2.0 MiB/s rd, 344 KiB/s wr, 142 op/s rd, 69 op/s wr
>     recovery: 34 MiB/s, 8 objects/s
>
> Pool 12 is the only one with any stats.
>
> pool EC-22-Pool id 12
>   510048/9545052 objects misplaced (5.344%)
>   recovery io 36 MiB/s, 9 objects/s
>   client io 1.8 MiB/s rd, 404 KiB/s wr, 86 op/s rd, 72 op/s wr
>
> --- RAW STORAGE ---
> CLASS    SIZE   AVAIL    USED  RAW USED  %RAW USED
> hdd    80 TiB  60 TiB  20 TiB    20 TiB      25.45
> TOTAL  80 TiB  60 TiB  20 TiB    20 TiB      25.45
>
> --- POOLS ---
> POOL                        ID  PGS   STORED  OBJECTS     USED  %USED  MAX
> AVAIL
> .mgr                         1    1  152 MiB       38  457 MiB      0
>  9.2 TiB
> 21BadPool                    3   32    8 KiB        1   12 KiB      0
> 18 TiB
> .rgw.root                    4   32  1.3 KiB        4   48 KiB      0
>  9.2 TiB
> default.rgw.log              5   32  3.6 KiB      209  408 KiB      0
>  9.2 TiB
> default.rgw.control          6   32      0 B        8      0 B      0
>  9.2 TiB
> default.rgw.meta             7    8  6.7 KiB       20  203 KiB      0
>  9.2 TiB
> rbd_rep_pool                 8   32  2.0 MiB        5  5.9 MiB      0
>  9.2 TiB
> default.rgw.buckets.index    9    8  2.0 MiB       33  5.9 MiB      0
>  9.2 TiB
> default.rgw.buckets.non-ec  10   32  1.4 KiB        0  4.3 KiB      0
>  9.2 TiB
> default.rgw.buckets.data    11   32  232 GiB   61.02k  697 GiB   2.41
>  9.2 TiB
> EC-22-Pool                  12  128  9.8 TiB    2.39M   20 TiB  41.55
> 14 TiB
>
>
>
> > Maybe provide the output of the first two commands for
> > osd_recovery_sleep_hdd=0.05 and osd_recovery_sleep_hdd=0.1 each (wait a
> bit
> > after setting these and then collect the output). Include the applied
> > values for osd_max_backfills* and osd_recovery_max_active* for one of the
> > OSDs in the pool (ceph config show osd.ID | grep -e osd_max_backfills -e
> > osd_recovery_max_active).
> >
>
> I didn't notice any speed difference with sleep values changed, but I'll
> grab the stats between changes when I have a chance.
>
> ceph config show osd.19 | egrep 'osd_max_backfills|osd_recovery_max_active'
> osd_max_backfills                                1000
>
>
>                                 override  mon[5]
> osd_recovery_max_active                          1000
>
>
>                                 override
> osd_recovery_max_active_hdd                      1000
>
>
>                                 override  mon[5]
> osd_recovery_max_active_ssd                      1000
>
>
>                                 override
>
> >
> > I don't really know if on such a small cluster one can expect more than
> > what you see. It has nothing to do with network speed if you have a 10G
> > line. However, recovery is something completely different from a full
> > link-speed copy.
> >
> > I can tell you that boatloads of tiny objects are a huge pain for
> > recovery, even on SSD. Ceph doesn't raid up sections of disks against
> each
> > other, but object for object. This might be a feature request: that PG
> > space allocation and recovery should follow the model of LVM extends
> > (ideally match with LVM extends) to allow recovery/rebalancing larger
> > chunks of storage in one go, containing parts of a large or many small
> > objects.
> >
> > Best regards,
> > =================
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > ________________________________________
> > From: Curt <lightspd@xxxxxxxxx<mailto:lightspd@xxxxxxxxx>>
> > Sent: 27 June 2022 17:35:19
> > To: Frank Schilder
> > Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
> > Subject: Re:  Re: Ceph recovery network speed
> >
> > Hello,
> >
> > I had already increased/changed those variables previously.  I increased
> > the pg_num to 128. Which increased the number of PG's backfilling, but
> > speed is still only at 30 MiB/s avg and has been backfilling 23 pg for
> the
> > last several hours.  Should I increase it higher than 128?
> >
> > I'm still trying to figure out if this is just how ceph is or if there is
> > a bottleneck somewhere.  Like if I sftp a 10G file between servers it's
> > done in a couple min or less.  Am I thinking of this wrong?
> >
> > Thanks,
> > Curt
> >
> > On Mon, Jun 27, 2022 at 12:33 PM Frank Schilder <frans@xxxxxx<mailto:
> frans@xxxxxx><mailto:
> > frans@xxxxxx<mailto:frans@xxxxxx>>> wrote:
> > Hi Curt,
> >
> > as far as I understood, a 2+2 EC pool is recovering, which makes 1 OSD
> per
> > host busy. My experience is, that the algorithm for selecting PGs to
> > backfill/recover is not very smart. It could simply be that it doesn't
> find
> > more PGs without violating some of these settings:
> >
> > osd_max_backfills
> > osd_recovery_max_active
> >
> > I have never observed the second parameter to change anything (try any
> > ways). However, the first one has a large impact. You could try
> increasing
> > this slowly until recovery moves faster. Another parameter you might want
> > to try is
> >
> > osd_recovery_sleep_[hdd|ssd]
> >
> > Be careful as this will impact client IO. I could reduce the sleep for my
> > HDDs to 0.05. With your workload pattern, this might be something you can
> > tune as well.
> >
> > Having said that, I think you should increase your PG count on the EC
> pool
> > as soon as the cluster is healthy. You have only about 20 PGs per OSD and
> > large PGs will take unnecessarily long to recover. A higher PG count will
> > also make it easier for the scheduler to find PGs for recovery/backfill.
> > Aim for a number between 100 and 200. Give the pool(s) with most data
> > (#objects) the most PGs.
> >
> > Best regards,
> > =================
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > ________________________________________
> > From: Curt <lightspd@xxxxxxxxx<mailto:lightspd@xxxxxxxxx><mailto:
> lightspd@xxxxxxxxx<mailto:lightspd@xxxxxxxxx>>>
> > Sent: 24 June 2022 19:04
> > To: Anthony D'Atri; ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx
> ><mailto:ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>>
> > Subject:  Re: Ceph recovery network speed
> >
> > 2 PG's shouldn't take hours to backfill in my opinion.  Just 2TB
> enterprise
> > HD's.
> >
> > Take this log entry below, 72 minutes and still backfilling undersized?
> > Should it be that slow?
> >
> > pg 12.15 is stuck undersized for 72m, current state
> > active+undersized+degraded+remapped+backfilling, last acting
> > [34,10,29,NONE]
> >
> > Thanks,
> > Curt
> >
> >
> > On Fri, Jun 24, 2022 at 8:53 PM Anthony D'Atri <anthony.datri@xxxxxxxxx
> <mailto:anthony.datri@xxxxxxxxx>
> > <mailto:anthony.datri@xxxxxxxxx<mailto:anthony.datri@xxxxxxxxx>>>
> > wrote:
> >
> > > Your recovery is slow *because* there are only 2 PGs backfilling.
> > >
> > > What kind of OSD media are you using?
> > >
> > > > On Jun 24, 2022, at 09:46, Curt <lightspd@xxxxxxxxx<mailto:
> lightspd@xxxxxxxxx><mailto:
> > lightspd@xxxxxxxxx<mailto:lightspd@xxxxxxxxx>>> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I'm trying to understand why my recovery is so slow with only 2 pg
> > > > backfilling.  I'm only getting speeds of 3-4/MiB/s on a 10G
> network.  I
> > > > have tested the speed between machines with a few tools and all
> confirm
> > > 10G
> > > > speed.  I've tried changing various settings of priority and recovery
> > > sleep
> > > > hdd, but still the same. Is this a configuration issue or something
> > else?
> > > >
> > > > It's just a small cluster right now with 4 hosts, 11 osd's per.
> Please
> > > let
> > > > me know if you need more information.
> > > >
> > > > Thanks,
> > > > Curt
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:
> ceph-users@xxxxxxx><mailto:
> > ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>>
> > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx><mailto:
> > ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>>
> > >
> > >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx
> ><mailto:ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>>
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx><mailto:
> > ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>>
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux