On 12/11/2013 11:10 PM, Duan, Jiangang wrote:
Mark,
Thanks for the comments.
One more question: is there bad impact if we use a higher PG# per OSD? E.g. 200x (I think a lot of people use this?) or 400x?
E.g. more memory consumption or lock contention?
I have not seen performance issues directly related to the number of PGs
per OSD, but rather based on the total number of PGs in the cluster. At
one point this was somewhere around 100K PGs with the hardware I was
testing, but some of the work we did last summer may have improved this.
The symptoms were mons not responding quickly to requests and generally
strange behaviour.
-jiangang
-----Original Message-----
From: Mark Nelson [mailto:mark.nelson@xxxxxxxxxxx]
Sent: Wednesday, December 11, 2013 10:38 PM
To: Duan, Jiangang
Cc: Sage Weil; Zhang, Jian; ceph-devel@xxxxxxxxxxxxxxx; He, Yujie
Subject: Re: question on BG# and its performance impact
On 12/11/2013 08:24 AM, Mark Nelson wrote:
Hi Jiangang,
To answer your earlier question about Uniformity:
What I saw in my testing was that the PG count increases, things did
tend to get more uniform, ie the standard deviation of the percentages
distributed over the set of OSDs slowly decreased with more PGs.
Primarily what I am interested in though is whether or not any
specific OSD has more PGs than the rest as that's all it will take to
screw up performance. As far as performance goes though, in my
testing it didn't necessarily seem to be strongly correlated with the
PG distribution, except for very small numbers of PGs. Much more
rigorous testing is probably needed to draw much of a conclusion.
Sage and I had a conversation a while ago about how to deal with
situations where you have uneven distributions (either through not
having enough PGs to ensure even distribution, or simply bad luck at
psuedo-random roulette). I proposed that we might iterate through
multiple possible pool distributions using different seed values until
we found one we liked with good psudorandom distribution. Perhaps you
could get even fancier by looking at what happens when you lose and
OSD or two. As this is all during pool creation, a little extra time
finding a nice initial distribution doesn't really hurt.
Sage mentioned though that it may be better to simple take whatever
distribution is generated and simply re-weight it to deal with
uniformity imperfections. I can't see any reason why this wouldn't
also work and has the benefit that it works no matter how the
distribution changes. Arguably this technique could go beyond just
looking at PG distributions and look at actual data distribution too
if the user wants extremely even data uniformity at the expense of a re-weighting tweak.
In any event, with very large clusters with lots of pools, I think we
will likely need to at some point adopt some kind of scheme that lets
us get away with fewer PGs per pool than our current recommendations.
Ha, replying to my own reply! Thinking about this a little more, these two techniques may in fact still be complementary. For very large clusters where the PG counts per OSD may be low, I suspect we will want to at least make sure the initial map guarantees that every OSD has at least 1 PG so we can do proper re-weighting down the road. In fact the better the initial distribution is, the less crazy we'll have to get with re-weighting, so it may not be a bad idea to use both techniques.
Mark
On 12/11/2013 12:22 AM, Duan, Jiangang wrote:
Cc the mail list as Sage suggested.
-----Original Message-----
From: Sage Weil [mailto:sage@xxxxxxxxxxx]
Sent: Wednesday, December 11, 2013 2:10 PM
To: Zhang, Jian
Cc: Mark Nelson; Duan, Jiangang
Subject: RE: question on BG# and its performance impact
On Wed, 11 Dec 2013, Zhang, Jian wrote:
Thanks for the suggestions, I will take a look on the ls output.
No, we didn't use the optimal crush tunables.
Hopefully that is part of it... try repeating the test with the
optimal tunables (now the default in master)!
s
Jian
-----Original Message-----
From: Sage Weil [mailto:sage@xxxxxxxxxxx]
Sent: Wednesday, December 11, 2013 1:36 PM
To: Zhang, Jian
Cc: Mark Nelson; Duan, Jiangang
Subject: RE: question on BG# and its performance impact
I might be worthwhile here to get teh actual list of objects (rados
-p $pool ls list.txt) and calculate the pg and osd mappings for each
of them to verify things are uniform.
One thing: are you using the 'optimal' crush tunables (ceph osd
crush tunables optimal)?
Also, can we cc ceph-devel?
sage
On Wed, 11 Dec 2013, Zhang, Jian wrote:
Mark,
Thanks for the help.
For the performance dip, I think it should casued by the directory
splitting, just check several OSD, it does has many sub directories.
For the pg # and distribution, see if I understand you correctly:
When you said "a slow trend toward uniformity" do you mean the pg #
for each pool is unforim? But from sheet2, the pg # on OSD10 is
103, while the pg # on OSD8 is 72, there is still a 30% gap. And I
think that's the reason we saw performance drop of 10M read with
1280 pgs
- pg # on the OSD is not balance.
Thanks
Jian
-----Original Message-----
From: Mark Nelson [mailto:mark.nelson@xxxxxxxxxxx]
Sent: Wednesday, December 11, 2013 12:01 PM
To: Duan, Jiangang
Cc: Sage Weil (sage@xxxxxxxxxxx); Zhang, Jian
Subject: Re: question on BG# and its performance impact
Hi Jiangang,
My results are rather old at this point, but I did similar testing
last spring to look at PG distribution and performance (with larger
writes) with varying numbers of PGs. I saw what looked like a slow
trend toward uniformity. Performance however was quite variable.
The performance dip you saw after many hours may have been due to
directory splitting on the underlying OSDs. When this happens
depends on the number of objects that are written out and the
number of PGs in the pool. Eventually, when enough objects are
written, the filestore will create a deeper nested directory
structure to store objects to keep the maximum number of objects
per directory below a certain maximum.
This is governed by two settings:
filestore merge threshold = 10
filestore split multiple = 2
The total number of objects per directory is by default 10 * 2 * 16
= 320. With small PG counts this can cause quite a bit of
directory splitting if there are many objects.
I believe that it is likely these defaults are lower than necessary
and we could allow more objects per directory, potentially reducing
the number of seeks for dentry lookups (though theoretically this
should be cached). We definitely have seen this have a large
performance impact with RGW though on clusters with small numbers
of PGs. With more PGs, and more relaxed thresholds, directory
splitting doesn't happen until many many millions of objects are
written out, and performance degradation as the disk fills up
appears to be less severe.
Mark
On 12/10/2013 09:05 PM, Duan, Jiangang wrote:
Sage/mark,
We find object# unbalance condition in our Ceph setup for both RBD
and object. Refer to the attached pdf.
Increase the PG# does increase performance however result in
unstable issues ?
Is this a known issue and do you have any BKM to fix this?
-jiangang
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
