Reweighting causes whole cluster to peer/activate

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I'm seeing something that seems to be odd behavior when reweighting OSDs. I've just upgraded to 12.2.5 and am adding in a new osd server to the cluster. I gradually weight the 10TB OSDs into the cluster by doing a +1, letting things backfill for a while, then +1 until I reach my desired weight. This hasn't been a problem in the past, a proportionate amount of PGs would get remapped, peer and activate across this cluster. Now on 12.2.5 when I do this, almost all PGs peer and reactivate. Sometimes it recovers within a minute, other times it takes longer, this last time actually saw some OSDs on the new node crash and caused a longer time for peering/activating. Regardless of recovery time, this is a fairly violent reaction to reweighting.

Has anyone else seen behavior like this or have any ideas what's going on?

For example...

[root@sephmon1 ~]# ceph -s
  cluster:
    id:     bc2a1488-74f8-4d87-b2f6-615ae26bf7c9
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum sephmon1,sephmon2,sephmon3
    mgr: sephmon2(active), standbys: sephmon1, sephmon3
    mds: cephfs1-1/1/1 up  {0=sephmon1=up:active}, 1 up:standby
    osd: 789 osds: 789 up, 789 in
 
  data:
    pools:   7 pools, 39168 pgs
    objects: 74046k objects, 2989 TB
    usage:   3756 TB used, 1517 TB / 5273 TB avail
    pgs:     39137 active+clean
             26    active+clean+scrubbing+deep
             5     active+clean+scrubbing
 
  io:
    client:   3522 MB/s rd, 118 MB/s wr, 1295 op/s rd, 833 op/s wr
 
[root@sephmon1 ~]# for i in {771..779}; do ceph osd crush reweight osd.${i} 6.5; done
reweighted item id 771 name 'osd.771' to 6.5 in crush map
reweighted item id 772 name 'osd.772' to 6.5 in crush map
reweighted item id 773 name 'osd.773' to 6.5 in crush map
reweighted item id 774 name 'osd.774' to 6.5 in crush map
reweighted item id 775 name 'osd.775' to 6.5 in crush map
reweighted item id 776 name 'osd.776' to 6.5 in crush map
reweighted item id 777 name 'osd.777' to 6.5 in crush map
reweighted item id 778 name 'osd.778' to 6.5 in crush map
reweighted item id 779 name 'osd.779' to 6.5 in crush map
[root@sephmon1 ~]# ceph -s
  cluster:
    id:     bc2a1488-74f8-4d87-b2f6-615ae26bf7c9
    health: HEALTH_WARN
            2 osds down
            78219/355096089 objects misplaced (0.022%)
            Reduced data availability: 668 pgs inactive, 1920 pgs down, 551 pgs peering, 29 pgs incomplete
            Degraded data redundancy: 803425/355096089 objects degraded (0.226%), 204 pgs degraded
            3 slow requests are blocked > 32 sec
 
  services:
    mon: 3 daemons, quorum sephmon1,sephmon2,sephmon3
    mgr: sephmon2(active), standbys: sephmon1, sephmon3
    mds: cephfs1-1/1/1 up  {0=sephmon1=up:active}, 1 up:standby
    osd: 789 osds: 787 up, 789 in; 257 remapped pgs
 
  data:
    pools:   7 pools, 39168 pgs
    objects: 73964k objects, 2985 TB
    usage:   3756 TB used, 1517 TB / 5273 TB avail
    pgs:     0.028% pgs unknown
             94.904% pgs not active
             803425/355096089 objects degraded (0.226%)
             78219/355096089 objects misplaced (0.022%)
             20215 peering
             14335 activating
             1882  active+clean
             1788  down
             205   remapped+peering
             167   stale+peering
             142   activating+undersized+degraded
             127   activating+undersized
             126   stale+down
             57    active+undersized+degraded
             39    stale+active+clean
             27    incomplete
             17    activating+remapped
             11    unknown
             7     stale+activating
             6     down+remapped
             3     stale+activating+undersized
             2     stale+incomplete
             2     active+undersized
             2     stale+activating+undersized+degraded
             1     activating+undersized+degraded+remapped
             1     stale+active+undersized+degraded
             1     remapped
             1     active+clean+scrubbing
             1     active+undersized+degraded+remapped+backfill_wait
             1     stale+remapped+peering
             1     active+clean+remapped
             1     active+remapped+backfilling
 
  io:
    client:   3896 GB/s rd, 339 GB/s wr, 8004 kop/s rd, 320 kop/s wr
    recovery: 726 GB/s, 11172 objects/s
 

Thanks,
Kevin


  

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux