Den sön 25 nov. 2018 kl 22:10 skrev Stefan Kooman <stefan@xxxxxx>: > > Hi List, > > Another interesting and unexpected thing we observed during cluster > expansion is the following. After we added extra disks to the cluster, > while "norebalance" flag was set, we put the new OSDs "IN". As soon as > we did that a couple of hundered objects would become degraded. During > that time no OSD crashed or restarted. Every "ceph osd crush add $osd > weight host=$storage-node" would cause extra degraded objects. > > I don't expect objects to become degraded when extra OSDs are added. > Misplaced, yes. Degraded, no > > Someone got an explantion for this? > Yes, when you add a drive (or 10), some PGs decide they should have one or more replicas on the new drives, a new empty PG is created there, and _then_ that replica will make that PG get into the "degraded" mode, meaning if it had 3 fine active+clean replicas before, it now has 2 active+clean and one needing backfill to get into shape. It is a slight mistake in reporting it in the same way as an error, even if it looks to the cluster just as if it was in error and needs fixing. This gives the new ceph admins a sense of urgency or danger whereas it should be perfectly normal to add space to a cluster. Also, it could have chosen to add a fourth PG in a repl=3 PG and fill from the one going out into the new empty PG and somehow keep itself with 3 working replicas, but ceph chooses to first discard one replica, then backfill into the empty one, leading to this kind of "error" report. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |