Can you describe the osd and client set up (number of nodes, number of osds per node, journal disks, replication level, and osd disks)? Looks like a lot of time is spent looking up objects in the filestore (lfn_open, etc). -Sam On Fri, Nov 9, 2012 at 2:21 AM, Stefan Priebe - Profihost AG <s.priebe@xxxxxxxxxxxx> wrote: > New graph from today. fsetxattr seems to take a lot of CPU too. > > Am 09.11.2012 11:09, schrieb Stefan Priebe - Profihost AG: > >> >> Disabling the logging with: >> debug lockdep = 0/0 >> debug context = 0/0 >> debug crush = 0/0 >> debug buffer = 0/0 >> debug timer = 0/0 >> debug journaler = 0/0 >> debug osd = 0/0 >> debug optracker = 0/0 >> debug objclass = 0/0 >> debug filestore = 0/0 >> debug journal = 0/0 >> debug ms = 0/0 >> debug monc = 0/0 >> debug tp = 0/0 >> debug auth = 0/0 >> debug finisher = 0/0 >> debug heartbeatmap = 0/0 >> debug perfcounter = 0/0 >> debug asok = 0/0 >> debug throttle = 0/0 >> >> reduced the CPU load about 50% ! So each OSD process now takes only one >> whole 3.6Ghz core instead of two. >> >> Have you looked at my latest profile graph with disabled debug options? >> >> Greets, >> Stefan >> >> >> Am 08.11.2012 17:06, schrieb Mark Nelson: >>> >>> On 11/08/2012 09:45 AM, Stefan Priebe - Profihost AG wrote: >>>> >>>> Am 08.11.2012 16:01, schrieb Sage Weil: >>>>> >>>>> On Thu, 8 Nov 2012, Stefan Priebe - Profihost AG wrote: >>>>>> >>>>>> Is there any way to find out why a ceph-osd process takes around 10 >>>>>> times more >>>>>> load on rand 4k writes than on 4k reads? >>>>> >>>>> >>>>> Something like perf or oprofile is probably your best bet. perf can be >>>>> tedious to deploy, depending on where your kernel is coming from. >>>>> oprofile seems to be deprecated, although I've had good results with >>>>> it in >>>>> the past. >>>> >>>> >>>> I've recorded 10s with perf - it is now a 300MB perf.data file. Sadly >>>> i've no idea what todo with it next. >>> >>> >>> Pour yourself a stiff drink! (haha!) >>> >>> Try just doing a "perf report" in the directory where you've got the >>> data file. Here's a nice tutorial: >>> >>> https://perf.wiki.kernel.org/index.php/Tutorial >>> >>> Also, if you see missing symbols you might benefit by chowning the file >>> to root and running perf report as root. If you still see missing >>> symbols, you may want to just give up and try sysprof. >>> >>>> >>>>> would love to see where the CPU is spending most of it's time. >>>>> This is >>>>> on current master? >>>> >>>> Yes >>>> >>>>> I expect there are still some low-hanging fruit that >>>>> can bring CPU utilization down (or even boost iops). >>>> >>>> Would be great to find them. >>>> >>>> Stefan >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
