hanging/stopped recovery/rebalance in Nautilus

["Philippe D'Anjou" <danjou.philippe@xxxxxxxx>]

Hi,

I often observed now that the recovery/rebalance in Nautilus starts quite fast but gets extremely slow (2-3 objects/s) even if there are like 20 OSDs involved. Right now I am moving (reweighted to 0) 16x8TB disks, it's running since 4 days and since 12h it's kind of stuck now at

  cluster:
    id:     2f525d60-aada-4da6-830f-7ba7b46c546b
    health: HEALTH_WARN
            Degraded data redundancy: 1070/899796274 objects degraded (0.000%), 1 pg degraded, 1 pg undersized
            1216 pgs not deep-scrubbed in time
            1216 pgs not scrubbed in time
     
  services:
    mon: 1 daemons, quorum km-fsn-1-dc4-m1-797678 (age 8w)
    mgr: km-fsn-1-dc4-m1-797678(active, since 6w)
    mds: xfd:1 {0=km-fsn-1-dc4-m1-797678=up:active}
    osd: 151 osds: 151 up (since 3d), 151 in (since 7d); 24 remapped pgs
    rgw: 1 daemon active (km-fsn-1-dc4-m1-797680)
 
  data:
    pools:   13 pools, 10433 pgs
    objects: 447.45M objects, 282 TiB
    usage:   602 TiB used, 675 TiB / 1.2 PiB avail
    pgs:     1070/899796274 objects degraded (0.000%)
             261226/899796274 objects misplaced (0.029%)
             10388 active+clean
             24    active+clean+remapped
             19    active+clean+scrubbing+deep
             1     active+undersized+degraded
             1     active+clean+scrubbing
 
  io:
    client:   10 MiB/s rd, 18 MiB/s wr, 141 op/s rd, 292 op/s wr

osd-max-backfill is at 16 for all OSDs.

Anyone got an idea why rebalance completely stopped?

Thanks

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com