How did you obtain those numbers? Were the 8k and 16k numbers per osd, or the raw throughput of 1 client? -Sam On Fri, Nov 9, 2012 at 1:34 PM, Stefan Priebe - Profihost AG <s.priebe@xxxxxxxxxxxx> wrote: > Am 09.11.2012 um 22:21 schrieb Samuel Just <sam.just@xxxxxxxxxxx>: > >> Can you describe the osd and client set up (number of nodes, number of >> osds per node, journal disks, replication level, and osd disks)? >> Looks like a lot of time is spent looking up objects in the filestore >> (lfn_open, etc). > > Sure. I have 5 nodes each with 4 ssds one per osd. The graph is from one osd process. Replication level was set to two. Journal is on tmpfs. > > Anything else you need to know? > > Stefan > >> -Sam >> >> On Fri, Nov 9, 2012 at 2:21 AM, Stefan Priebe - Profihost AG >> <s.priebe@xxxxxxxxxxxx> wrote: >>> New graph from today. fsetxattr seems to take a lot of CPU too. >>> >>> Am 09.11.2012 11:09, schrieb Stefan Priebe - Profihost AG: >>> >>>> >>>> Disabling the logging with: >>>> debug lockdep = 0/0 >>>> debug context = 0/0 >>>> debug crush = 0/0 >>>> debug buffer = 0/0 >>>> debug timer = 0/0 >>>> debug journaler = 0/0 >>>> debug osd = 0/0 >>>> debug optracker = 0/0 >>>> debug objclass = 0/0 >>>> debug filestore = 0/0 >>>> debug journal = 0/0 >>>> debug ms = 0/0 >>>> debug monc = 0/0 >>>> debug tp = 0/0 >>>> debug auth = 0/0 >>>> debug finisher = 0/0 >>>> debug heartbeatmap = 0/0 >>>> debug perfcounter = 0/0 >>>> debug asok = 0/0 >>>> debug throttle = 0/0 >>>> >>>> reduced the CPU load about 50% ! So each OSD process now takes only one >>>> whole 3.6Ghz core instead of two. >>>> >>>> Have you looked at my latest profile graph with disabled debug options? >>>> >>>> Greets, >>>> Stefan >>>> >>>> >>>> Am 08.11.2012 17:06, schrieb Mark Nelson: >>>>> >>>>> On 11/08/2012 09:45 AM, Stefan Priebe - Profihost AG wrote: >>>>>> >>>>>> Am 08.11.2012 16:01, schrieb Sage Weil: >>>>>>> >>>>>>> On Thu, 8 Nov 2012, Stefan Priebe - Profihost AG wrote: >>>>>>>> >>>>>>>> Is there any way to find out why a ceph-osd process takes around 10 >>>>>>>> times more >>>>>>>> load on rand 4k writes than on 4k reads? >>>>>>> >>>>>>> >>>>>>> Something like perf or oprofile is probably your best bet. perf can be >>>>>>> tedious to deploy, depending on where your kernel is coming from. >>>>>>> oprofile seems to be deprecated, although I've had good results with >>>>>>> it in >>>>>>> the past. >>>>>> >>>>>> >>>>>> I've recorded 10s with perf - it is now a 300MB perf.data file. Sadly >>>>>> i've no idea what todo with it next. >>>>> >>>>> >>>>> Pour yourself a stiff drink! (haha!) >>>>> >>>>> Try just doing a "perf report" in the directory where you've got the >>>>> data file. Here's a nice tutorial: >>>>> >>>>> https://perf.wiki.kernel.org/index.php/Tutorial >>>>> >>>>> Also, if you see missing symbols you might benefit by chowning the file >>>>> to root and running perf report as root. If you still see missing >>>>> symbols, you may want to just give up and try sysprof. >>>>> >>>>>> >>>>>>> would love to see where the CPU is spending most of it's time. >>>>>>> This is >>>>>>> on current master? >>>>>> >>>>>> Yes >>>>>> >>>>>>> I expect there are still some low-hanging fruit that >>>>>>> can bring CPU utilization down (or even boost iops). >>>>>> >>>>>> Would be great to find them. >>>>>> >>>>>> Stefan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
