Hello Sage, are there any development plans to improve PG distribution to OSDs? We use CephFS on Infernalis and objects are distributed very well across the PGs. But the automatic PG distribution creates large fluctuations, even in the simplest case for same-sized OSDs and a flat hierarchy using 3x replication. Testing with crushtool, we would need an unreasonable high number of PG/OSD to have a nearly-flat distribution. Reweighting the PGs with "ceph osd reweight" helped for our cache-pool since it has a flat hierarchy (one OSD/host). The EC pool on the other hand has a hierarchy with around 2-4 OSDs (1-4TB / OSD) per host and redundancy is distributed across hosts (6-8TB / host). I assume, there would be no way around tuning the host weights in the crush map in this case? Markus -----Original Message----- From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Sage Weil Sent: Dienstag, 7. Juni 2016 15:22 To: M Ranga Swami Reddy <swamireddy@xxxxxxxxx> Cc: ceph-devel <ceph-devel@xxxxxxxxxxxxxxx>; ceph-users <ceph-users@xxxxxxxxxxxxxx> Subject: Re: [ceph-users] un-even data filled on OSDs On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote: > OK, understood... > To fix the nearfull warn, I am reducing the weight of a specific OSD, > which filled >85%.. > Is this work-around advisable? Sure. This is what reweight-by-utilization does for you, but automatically. sage > > Thanks > Swami > > On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote: > > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote: > >> Hi Sage, > >> >Jewel and the latest hammer point release have an improved > >> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... > >> >to dry > >> > run) to correct this. > >> > >> Thank you....But not planning to upgrade the cluster soon. > >> So, in this case - are there any tunable options will help? like > >> "crush tunable optimal" or so? > >> OR any other configuration options change will help? > > > > Firefly also has reweight-by-utilization... it's just a bit less > > friendly than the newer versions. CRUSH tunables don't generally > > help here unless you have lots of OSDs that are down+out. > > > > Note that firefly is no longer supported. > > > > sage > > > > > >> > >> > >> Thanks > >> Swami > >> > >> > >> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote: > >> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote: > >> >> Hello, > >> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs > >> >> filled with >85% of data and few OSDs filled with ~60%-70% of data. > >> >> > >> >> Any reason why the unevenly OSDs filling happned? do I need to > >> >> any tweaks on configuration to fix the above? Please advise. > >> >> > >> >> PS: Ceph version is - 0.80.7 > >> > > >> > Jewel and the latest hammer point release have an improved > >> > reweight-by-utilization (ceph osd test-reweight-by-utilization > >> > ... to dry > >> > run) to correct this. > >> > > >> > sage > >> > > >> -- > >> To unsubscribe from this list: send the line "unsubscribe > >> ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > >> > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
