Re: OSD tuning no longer required?

[Xavier Trilla <xavier.trilla@xxxxxxxxxxx>]

Hi there,

 

I would like just to note, that for some scenarios defaults are not good enough.

 

Recently we upgraded one of our clusters from Jewel to Luminous, during the upgrade we removed all the custom tuning we had done on it over the years from ceph.conf -I was extremely excited to get rid of all those parameters!!- but performance after the upgrade was really bad.

 

The cluster runs mainly over write cache backed HDDs (via dedicated SAS controller), so looks like the defaults for HDDs are for really low performance HDD setups.

 

So, at the end we had to reapply our previous config, and tune some other -not as well documented as I would like- parameters. And now, we have even more tuning parameters than we did… :/

 

I think the main issue is the lack of a updated guide to tune ceph OSD performance. Or at least the proper documentation in order to know what each parameter really does (And I’m running ceph clusters since cuttlefish, so I did spend my time checking documentation, presentations, etc…)

 

It’s a pity so many new users become disappointed with ceph, because they don’t get even near to the performance they are expecting.

 

I still have to try the new default values in a pure NVMe cluster, I hope the result will be better.

 

I think a good document about how to tune OSD performance, would really help Ceph :)

 

Cheers!

 

Saludos Cordiales,

Xavier Trilla P.

Clouding.io

 

¿Un Servidor Cloud con SSDs, redundado

y disponible en menos de 30 segundos?

 

¡Pruébalo ahora en Clouding.io!

 

De: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> En nombre de Robert Stanford
Enviado el: lunes, 16 de julio de 2018 21:40
Para: Gregory Farnum <gfarnum@xxxxxxxxxx>
CC: ceph-users <ceph-users@xxxxxxxxxxxxxx>; Konstantin Shalygin <k0ste@xxxxxxxx>
Asunto: Re: [ceph-users] OSD tuning no longer required?

 

 

 Golden advice.  Thank you Greg

 

On Mon, Jul 16, 2018 at 1:45 PM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:

On Fri, Jul 13, 2018 at 2:50 AM Robert Stanford <rstanford8896@xxxxxxxxx> wrote:

 

 This is what leads me to believe it's other settings being referred to as well:

https://ceph.com/community/new-luminous-rados-improvements/

 

"There are dozens of documents floating around with long lists of Ceph configurables that have been tuned for optimal performance on specific hardware or for specific workloads.  In most cases these ceph.conf fragments tend to induce funny looks on developers’ faces because the settings being adjusted seem counter-intuitive, unrelated to the performance of the system, and/or outright dangerous.  Our goal is to make Ceph work as well as we can out of the box without requiring any tuning at all, so we are always striving to choose sane defaults.  And generally, we discourage tuning by users. "

 

To me it's not just bluestore settings / sdd vs. hdd they're talking about ("dozens of documents floating around"... "our goal... without any tuning at all".  Am I off base?

 

Ceph is *extremely* tunable, because whenever we set up a new behavior (snapshot trimming sleeps, scrub IO priorities, whatever) and we're not sure how it should behave we add a config option. Most of these config options we come up with some value through testing or informed guesswork, set it in the config, and expect that users won't ever see it. Some of these settings we don't know what they should be, and we really hope the whole mechanism gets replaced before users see it, but they don't. Some of the settings should be auto-tuning or manually set to a different value for each deployment to get optimal performance.

So there are lots of options for people to make things much better or much worse for themselves.

 

However, by far the biggest impact and most common tunables are those that basically vary on if the OSD is using a hard drive or an SSD for its local storage — those are order-of-magnitude differences in expected latency and throughput. So we now have separate default tunables for those cases which are automatically applied.

 

Could somebody who knows what they're doing tweak things even better for a particular deployment? Undoubtedly. But do *most* people know what they're doing that well? They don't.

In particular, the old "fix it" configuration settings that a lot of people were sharing and using starting in the Cuttlefish days are rather dangerously out of date, and we no longer have defaults that are quite as stupid as some of those were.

 

So I'd generally recommend you remove any custom tuning you've set up unless you have specific reason to think it will do better than the defaults for your currently-deployed release.

-Greg

 

 

 Regards

 

On Thu, Jul 12, 2018 at 9:12 PM, Konstantin Shalygin <k0ste@xxxxxxxx> wrote:

  I saw this in the Luminous release notes:

  "Each OSD now adjusts its default configuration based on whether the
backing device is an HDD or SSD. Manual tuning generally not required"

  Which tuning in particular?  The ones in my configuration are
osd_op_threads, osd_disk_threads, osd_recovery_max_active,
osd_op_thread_suicide_timeout, and osd_crush_chooseleaf_type, among
others.  Can I rip these out when I upgrade to
Luminous?

 

This mean that some "bluestore_*" settings tuned for nvme/hdd separately.

Also with Luminous we have:

osd_op_num_shards_(ssd|hdd)

osd_op_num_threads_per_shard_(ssd|hdd)

osd_recovery_sleep_(ssd|hdd)




k

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com