Re: hanging/stopped recovery/rebalance in Nautilus

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,I often observed now that the recovery/rebalance in Nautilus starts quite fast but gets extremely slow (2-3 objects/s) even if there are like 20 OSDs involved. Right now I am moving (reweighted to 0) 16x8TB disks, it's running since 4 days and since 12h it's kind of stuck now at
  cluster:
    id:     2f525d60-aada-4da6-830f-7ba7b46c546b
    health: HEALTH_WARN
            Degraded data redundancy: 1070/899796274 objects degraded (0.000%), 1 pg degraded, 1 pg undersized
            1216 pgs not deep-scrubbed in time
            1216 pgs not scrubbed in time
      
  services:
    mon: 1 daemons, quorum km-fsn-1-dc4-m1-797678 (age 8w)
    mgr: km-fsn-1-dc4-m1-797678(active, since 6w)
    mds: xfd:1 {0=km-fsn-1-dc4-m1-797678=up:active}
    osd: 151 osds: 151 up (since 3d), 151 in (since 7d); 24 remapped pgs
    rgw: 1 daemon active (km-fsn-1-dc4-m1-797680)
 
  data:
    pools:   13 pools, 10433 pgs
    objects: 447.45M objects, 282 TiB
    usage:   602 TiB used, 675 TiB / 1.2 PiB avail
    pgs:     1070/899796274 objects degraded (0.000%)
             261226/899796274 objects misplaced (0.029%)
             10388 active+clean
             24    active+clean+remapped
             19    active+clean+scrubbing+deep
             1     active+undersized+degraded
             1     active+clean+scrubbing
 
  io:
    client:   10 MiB/s rd, 18 MiB/s wr, 141 op/s rd, 292 op/s wr


osd-max-backfill is at 16 for all OSDs.
Anyone got an idea why rebalance completely stopped?
Thanks

Try to lower sleep options:

osd_recovery_sleep_hdd -> if hdd without RocksDB on NVMe;
osd_recovery_sleep_hybrid -> for hybrid solutions, i.e. RocksDB on NVMe;
osd_recovery_sleep_ssd -> for non rotational devices;



k

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux