Re: osds with different disk sizes may killing performance (?? ?)

[David Turner <drakonstein@xxxxxxxxx>]

You'll find it said time and time agin on the ML... avoid disks of different sizes in the same cluster.  It's a headache that sucks.  It's not impossible, it's not even overly hard to pull off... but it's very easy to cause a mess and a lot of headaches.  It will also make it harder to diagnose performance issues in the cluster.

There is no way to fill up all disks evenly with the same number of Bytes and then stop filling the small disks when they're full and only continue filling the larger disks.  What will happen if you are filling all disks evenly with Bytes instead of % is that the small disks will get filled completely and all writes to the cluster will block until you do something to reduce the amount used on the full disks.

On Fri, Apr 13, 2018 at 1:28 AM Ronny Aasen <ronny+ceph-users@xxxxxxxx> wrote:

On 13. april 2018 05:32, Chad William Seys wrote:
> Hello,
>    I think your observations suggest that, to a first approximation,
> filling drives with bytes to the same absolute level is better for
> performance than filling drives to the same percentage full. Assuming
> random distribution of PGs, this would cause the smallest drives to be
> as active as the largest drives.
>    E.g. if every drive had 1TB of data, each would be equally likely to
> contain the PG of interest.
>    Of course, as more data was added the smallest drives could not hold
> more and the larger drives become more active, but at least the smaller
> drives would as active as possible.

but in this case you would have a steep drop off of performance. when
you reach the fill level where small drives do not accept more data,
suddenly you would have a performance cliff where only your larger disks
are doing new writes. and only larger disks doing reads on new data.


it is also easier to make the logical connection while you are
installing new nodes/disks. then a year later when your cluster just
happen to reach that fill level.

it would also be an easier job balancing disks between nodes when you
are adding osd's anyway and the new ones are mostly empty. rather then
when your small osd's are full and your large disks have significant
data on them.



kind regards
Ronny Aasen
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com