I've seen some odd teuthology in the last week or two which seems to be anomalous rjenkins hash behavior as well. http://tracker.ceph.com/issues/12231 -Sam ----- Original Message ----- From: "Sage Weil" <sweil@xxxxxxxxxx> To: "Gleb Borisov" <borisov.gleb@xxxxxxxxx> Cc: ceph-devel@xxxxxxxxxxxxxxx Sent: Thursday, July 9, 2015 3:06:00 PM Subject: Re: Strange issue with CRUSH On Fri, 10 Jul 2015, Gleb Borisov wrote: > Hi Sage, > > Sorry for mailing you in person, I realize that you're quite busy at redhat, > but I wanted you have a look on an issue with CRUSH map. No problem. I hope you don't mind I've added ceph-devel to the cc list. > I've described very first insights here: > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-July/002897.html > > We are continue our research and found that distribution of PG count by OSD > is very strange and after digging into CRUSH source code found rjenkins1 > hash function. > > After some testing we realized that rjenkins1's value distribution is > exponential, and this can cause our disbalance. Any issue with rjenkins1's hash function is very interesting and concerning. Can you describe your analysis and what you mean by the distribution being exponential? > What do you think about adding additional hashing algorithm to CRUSH? It > seems that it could improve distribution. I am definitely open to adding new hash functions, especially if the current ones are flawed. The current hash was created by making ad hoc combinations of rjenkins' mix function with various numbers of arguments--hardly scientific or methodical. We did an analysis a couple years back and found that it effectively modeled a uniform distribution, but if we missed something or were wrong we should definitely correct it! In any case, the important step is to quantify what is wrong with the current hash so that we can ensure any new one is not flawed in the same way. Thanks- sage > We have also tried to generate some syntetic crushmaps (another bucket > types, more OSDs per host, more/less hosts by rack, different cound of > racks, linear osd ids, random osd ids, etc), but didn't found any > combination with better distribution of PG across OSD. > > Thanks and one more sorry for bothering you in person. > -- > Best regards, > Gleb M Borisov > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
