> Op 30 december 2016 om 11:06 schreef Kees Meijs <kees@xxxxxxxx>:
>
>
> Hi Asley,
>
> We experience (using Hammer) a similar issue. Not that I have a perfect
> solution to share, but I felt like mentioning a "me too". ;-)
>
> On a side note: we configured correct weight per drive as well.
>
Ceph will never balance 100% perfect for a few reasons:
- CRUSH is not perfect nor will it ever be
- Object sizes vary
- The amount of Placement Groups matters
For this reason you can do a OSD reweight by running the 'ceph osd reweight-by-utilization' command or do it manually with 'ceph osd reweight X 0-1'
Wido
> Regards,
> Kees
>
> On 29-12-16 11:54, Ashley Merrick wrote:
> >
> > Hello,
> >
> >
> >
> > I currently have 5 servers within my CEPH Cluster
> >
> >
> >
> > 2 x (10 * 8TB Disks)
> >
> > 3 x (10 * 4TB Disks)
> >
> >
> >
> > Currently seeing a larger difference in OSD use across the two
> > separate server types, as well as within the server itself.
> >
> >
> >
> > For example on one 4TB server I have an OSD at 64% and one at 84%,
> > where on the 8TB servers the OSD range from 49% to 64%, where the
> > highest used OSD’s are on the 4TB.
> >
> >
> >
> > Each drive has a weight set correctly for the drive size and each
> > server has the correct weight set, below is my crush map. Apart from
> > running the command to adjust the re-weight is there anything I am
> > doing wrong or should change for better spread of data, not looking
> > for near perfect but where the 8TB drives are sitting at 64% max and
> > 4TB are sitting at 80%’s causes a big inbalance.
> >
> >
> >
> > # begin crush map
> >
> > tunable choose_local_tries 0
> >
> > tunable choose_local_fallback_tries 0
> >
> > tunable choose_total_tries 50
> >
> > tunable chooseleaf_descend_once 1
> >
> > tunable chooseleaf_vary_r 1
> >
> > tunable straw_calc_version 1
> >
> > tunable allowed_bucket_algs 54
> >
> >
> >
> > # buckets
> >
> > host sn1 {
> >
> > id -2 # do not change unnecessarily
> >
> > # weight 72.800
> >
> > alg straw2
> >
> > hash 0 # rjenkins1
> >
> > item osd.0 weight 7.280
> >
> > item osd.1 weight 7.280
> >
> > item osd.3 weight 7.280
> >
> > item osd.4 weight 7.280
> >
> > item osd.2 weight 7.280
> >
> > item osd.5 weight 7.280
> >
> > item osd.6 weight 7.280
> >
> > item osd.7 weight 7.280
> >
> > item osd.8 weight 7.280
> >
> > item osd.9 weight 7.280
> >
> > }
> >
> > host sn3 {
> >
> > id -6 # do not change unnecessarily
> >
> > # weight 72.800
> >
> > alg straw2
> >
> > hash 0 # rjenkins1
> >
> > item osd.10 weight 7.280
> >
> > item osd.11 weight 7.280
> >
> > item osd.12 weight 7.280
> >
> > item osd.13 weight 7.280
> >
> > item osd.14 weight 7.280
> >
> > item osd.15 weight 7.280
> >
> > item osd.16 weight 7.280
> >
> > item osd.17 weight 7.280
> >
> > item osd.18 weight 7.280
> >
> > item osd.19 weight 7.280
> >
> > }
> >
> > host sn4 {
> >
> > id -7 # do not change unnecessarily
> >
> > # weight 36.060
> >
> > alg straw2
> >
> > hash 0 # rjenkins1
> >
> > item osd.20 weight 3.640
> >
> > item osd.21 weight 3.640
> >
> > item osd.22 weight 3.640
> >
> > item osd.23 weight 3.640
> >
> > item osd.24 weight 3.640
> >
> > item osd.25 weight 3.640
> >
> > item osd.26 weight 3.640
> >
> > item osd.27 weight 3.640
> >
> > item osd.28 weight 3.640
> >
> > item osd.29 weight 3.300
> >
> > }
> >
> > host sn5 {
> >
> > id -8 # do not change unnecessarily
> >
> > # weight 36.060
> >
> > alg straw2
> >
> > hash 0 # rjenkins1
> >
> > item osd.30 weight 3.640
> >
> > item osd.31 weight 3.640
> >
> > item osd.32 weight 3.640
> >
> > item osd.33 weight 3.640
> >
> > item osd.34 weight 3.640
> >
> > item osd.35 weight 3.640
> >
> > item osd.36 weight 3.640
> >
> > item osd.37 weight 3.640
> >
> > item osd.38 weight 3.640
> >
> > item osd.39 weight 3.640
> >
> > }
> >
> > host sn6 {
> >
> > id -9 # do not change unnecessarily
> >
> > # weight 36.060
> >
> > alg straw2
> >
> > hash 0 # rjenkins1
> >
> > item osd.40 weight 3.640
> >
> > item osd.41 weight 3.640
> >
> > item osd.42 weight 3.640
> >
> > item osd.43 weight 3.640
> >
> > item osd.44 weight 3.640
> >
> > item osd.45 weight 3.640
> >
> > item osd.46 weight 3.640
> >
> > item osd.47 weight 3.640
> >
> > item osd.48 weight 3.640
> >
> > item osd.49 weight 3.640
> >
> > }
> >
> > root default {
> >
> > id -1 # do not change unnecessarily
> >
> > # weight 253.780
> >
> > alg straw2
> >
> > hash 0 # rjenkins1
> >
> > item sn1 weight 72.800
> >
> > item sn3 weight 72.800
> >
> > item sn4 weight 36.060
> >
> > item sn5 weight 36.060
> >
> > item sn6 weight 36.060
> >
> > }
> >
> >
> >
> > # rules
> >
> > rule replicated_ruleset {
> >
> > ruleset 0
> >
> > type replicated
> >
> > min_size 1
> >
> > max_size 10
> >
> > step take default
> >
> > step chooseleaf firstn 0 type host
> >
> > step emit
> >
> > }
> >
> >
> >
> > Thanks,
> >
> > Ashley
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
