Try 'ceph osd reweight-by-pg <int>' right after creating the pools? What is the typical object size in the cluster? Thanks, Guang ---------------------------------------- > To: ceph-users@xxxxxxxxxxxxxx > From: vedran.furac@xxxxxxxxx > Date: Thu, 13 Aug 2015 14:58:11 +0200 > Subject: OSD space imbalance > > Hello, > > I'm having an issue where disk usages between OSDs aren't well balanced > thus causing disk space to be wasted. Ceph is latest 0.94.2, used > exclusively through cephfs. Re-weighting helps, but just slightly, and > it has to be done on a daily basis causing constant refills. In the end > I get OSD with 65% usage with some other going over 90%. I also set the > "ceph osd crush tunables optimal", but I didn't notice any changes when > it comes to disk usage. Is there anything I can do to get them within > 10% range at least? > > health HEALTH_OK > mdsmap e2577: 1/1/1 up, 2 up:standby > osdmap e25239: 48 osds: 48 up, 48 in > pgmap v3188836: 5184 pgs, 3 pools, 18028 GB data, 6385 kobjects > 36156 GB used, 9472 GB / 45629 GB avail > 5184 active+clean > > > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR > 37 0.92999 1.00000 950G 625G 324G 65.85 0.83 > 21 0.92999 1.00000 950G 649G 300G 68.35 0.86 > 32 0.92999 1.00000 950G 670G 279G 70.58 0.89 > 7 0.92999 1.00000 950G 676G 274G 71.11 0.90 > 17 0.92999 1.00000 950G 681G 268G 71.73 0.91 > 40 0.92999 1.00000 950G 689G 260G 72.55 0.92 > 20 0.92999 1.00000 950G 690G 260G 72.62 0.92 > 25 0.92999 1.00000 950G 691G 258G 72.76 0.92 > 2 0.92999 1.00000 950G 694G 256G 73.03 0.92 > 39 0.92999 1.00000 950G 697G 253G 73.35 0.93 > 18 0.92999 1.00000 950G 703G 247G 74.00 0.93 > 47 0.92999 1.00000 950G 703G 246G 74.05 0.93 > 23 0.92999 0.86693 950G 704G 245G 74.14 0.94 > 6 0.92999 1.00000 950G 726G 224G 76.39 0.96 > 8 0.92999 1.00000 950G 727G 223G 76.54 0.97 > 5 0.92999 1.00000 950G 728G 222G 76.62 0.97 > 35 0.92999 1.00000 950G 728G 221G 76.66 0.97 > 11 0.92999 1.00000 950G 730G 220G 76.82 0.97 > 43 0.92999 1.00000 950G 730G 219G 76.87 0.97 > 33 0.92999 1.00000 950G 734G 215G 77.31 0.98 > 38 0.92999 1.00000 950G 736G 214G 77.49 0.98 > 12 0.92999 1.00000 950G 737G 212G 77.61 0.98 > 31 0.92999 0.85184 950G 742G 208G 78.09 0.99 > 28 0.92999 1.00000 950G 745G 205G 78.41 0.99 > 27 0.92999 1.00000 950G 751G 199G 79.04 1.00 > 10 0.92999 1.00000 950G 754G 195G 79.40 1.00 > 13 0.92999 1.00000 950G 762G 188G 80.21 1.01 > 9 0.92999 1.00000 950G 763G 187G 80.29 1.01 > 16 0.92999 1.00000 950G 764G 186G 80.37 1.01 > 0 0.92999 1.00000 950G 778G 171G 81.94 1.03 > 3 0.92999 1.00000 950G 780G 170G 82.11 1.04 > 41 0.92999 1.00000 950G 780G 169G 82.13 1.04 > 34 0.92999 0.87303 950G 783G 167G 82.43 1.04 > 14 0.92999 1.00000 950G 784G 165G 82.56 1.04 > 42 0.92999 1.00000 950G 786G 164G 82.70 1.04 > 46 0.92999 1.00000 950G 788G 162G 82.93 1.05 > 30 0.92999 1.00000 950G 790G 160G 83.12 1.05 > 45 0.92999 1.00000 950G 804G 146G 84.59 1.07 > 44 0.92999 1.00000 950G 807G 143G 84.92 1.07 > 1 0.92999 1.00000 950G 817G 132G 86.05 1.09 > 22 0.92999 1.00000 950G 825G 125G 86.81 1.10 > 15 0.92999 1.00000 950G 826G 123G 86.97 1.10 > 19 0.92999 1.00000 950G 829G 120G 87.30 1.10 > 36 0.92999 1.00000 950G 831G 119G 87.48 1.10 > 24 0.92999 1.00000 950G 831G 118G 87.50 1.10 > 26 0.92999 1.00000 950G 851G 101692M 89.55 1.13 > 29 0.92999 1.00000 950G 851G 101341M 89.59 1.13 > 4 0.92999 1.00000 950G 860G 92164M 90.53 1.14 > MIN/MAX VAR: 0.83/1.14 STDDEV: 5.94 > TOTAL 45629G 36156G 9473G 79.24 > > Thanks, > Vedran > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |