Hi Wido,
On 02/21/2017 02:04 PM, Wido den Hollander wrote:
Hi,
I'm about to start a test where I'll be putting a lot of objects into BlueStore and see how it holds.
The reasoning behind is that I have a customer which has 165M objects in it's cluster which results in some PGs having 900k objects.
For FileStore with XFS this is quite heavy. A simple scrub takes ages.
The problem is that we can't simply increase the number of PGs since that will overload the OSDs as well.
On the other hand we could add hardware, but that also takes time.
So just for the sake of testing I'm looking at trying to replicate this situation using BlueStore from master.
Is there anything I should take into account? I'll probably be just creating a lot (millions) of 100 byte objects in the cluster with just a few PGs.
Couple of general things:
I don't anticipate you'll run into the same kind of pg splitting
slowdowns that you see with filestore, but you still may see some
slowdown as the object count increases since rocksdb will have more
key/value pairs to deal with. I expect you'll see a lot of metadata
movement between levels as it tries to keep things organized. One thing
to note is that it's possible you may see rocksdb bottlenecks as the OSD
volume size increases. This is one of the things the guys at Sandisk
were trying to tackle with Zetascale.
If you can put the rocksdb DB and WAL on SSDs that will likely help, but
you'll want to be mindful of how full the SSDs are getting. I'll be
very curious to see how your tests go, it's been a while since we've
thrown that many objects on a bluestore cluster (back around the
newstore timeframe we filled bluestore with many 10s of millions of
objects and from what I remember it did pretty well).
Mark
Wido
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
