How are the 4 ssds per osd set up? RAID 5? -Sam On Sat, Nov 10, 2012 at 12:36 PM, Samuel Just <sam.just@xxxxxxxxxxx> wrote: > How did you obtain those numbers? Were the 8k and 16k numbers per > osd, or the raw throughput of 1 client? > -Sam > > On Fri, Nov 9, 2012 at 1:34 PM, Stefan Priebe - Profihost AG > <s.priebe@xxxxxxxxxxxx> wrote: >> Am 09.11.2012 um 22:21 schrieb Samuel Just <sam.just@xxxxxxxxxxx>: >> >>> Can you describe the osd and client set up (number of nodes, number of >>> osds per node, journal disks, replication level, and osd disks)? >>> Looks like a lot of time is spent looking up objects in the filestore >>> (lfn_open, etc). >> >> Sure. I have 5 nodes each with 4 ssds one per osd. The graph is from one osd process. Replication level was set to two. Journal is on tmpfs. >> >> Anything else you need to know? >> >> Stefan >> >>> -Sam >>> >>> On Fri, Nov 9, 2012 at 2:21 AM, Stefan Priebe - Profihost AG >>> <s.priebe@xxxxxxxxxxxx> wrote: >>>> New graph from today. fsetxattr seems to take a lot of CPU too. >>>> >>>> Am 09.11.2012 11:09, schrieb Stefan Priebe - Profihost AG: >>>> >>>>> >>>>> Disabling the logging with: >>>>> debug lockdep = 0/0 >>>>> debug context = 0/0 >>>>> debug crush = 0/0 >>>>> debug buffer = 0/0 >>>>> debug timer = 0/0 >>>>> debug journaler = 0/0 >>>>> debug osd = 0/0 >>>>> debug optracker = 0/0 >>>>> debug objclass = 0/0 >>>>> debug filestore = 0/0 >>>>> debug journal = 0/0 >>>>> debug ms = 0/0 >>>>> debug monc = 0/0 >>>>> debug tp = 0/0 >>>>> debug auth = 0/0 >>>>> debug finisher = 0/0 >>>>> debug heartbeatmap = 0/0 >>>>> debug perfcounter = 0/0 >>>>> debug asok = 0/0 >>>>> debug throttle = 0/0 >>>>> >>>>> reduced the CPU load about 50% ! So each OSD process now takes only one >>>>> whole 3.6Ghz core instead of two. >>>>> >>>>> Have you looked at my latest profile graph with disabled debug options? >>>>> >>>>> Greets, >>>>> Stefan >>>>> >>>>> >>>>> Am 08.11.2012 17:06, schrieb Mark Nelson: >>>>>> >>>>>> On 11/08/2012 09:45 AM, Stefan Priebe - Profihost AG wrote: >>>>>>> >>>>>>> Am 08.11.2012 16:01, schrieb Sage Weil: >>>>>>>> >>>>>>>> On Thu, 8 Nov 2012, Stefan Priebe - Profihost AG wrote: >>>>>>>>> >>>>>>>>> Is there any way to find out why a ceph-osd process takes around 10 >>>>>>>>> times more >>>>>>>>> load on rand 4k writes than on 4k reads? >>>>>>>> >>>>>>>> >>>>>>>> Something like perf or oprofile is probably your best bet. perf can be >>>>>>>> tedious to deploy, depending on where your kernel is coming from. >>>>>>>> oprofile seems to be deprecated, although I've had good results with >>>>>>>> it in >>>>>>>> the past. >>>>>>> >>>>>>> >>>>>>> I've recorded 10s with perf - it is now a 300MB perf.data file. Sadly >>>>>>> i've no idea what todo with it next. >>>>>> >>>>>> >>>>>> Pour yourself a stiff drink! (haha!) >>>>>> >>>>>> Try just doing a "perf report" in the directory where you've got the >>>>>> data file. Here's a nice tutorial: >>>>>> >>>>>> https://perf.wiki.kernel.org/index.php/Tutorial >>>>>> >>>>>> Also, if you see missing symbols you might benefit by chowning the file >>>>>> to root and running perf report as root. If you still see missing >>>>>> symbols, you may want to just give up and try sysprof. >>>>>> >>>>>>> >>>>>>>> would love to see where the CPU is spending most of it's time. >>>>>>>> This is >>>>>>>> on current master? >>>>>>> >>>>>>> Yes >>>>>>> >>>>>>>> I expect there are still some low-hanging fruit that >>>>>>>> can bring CPU utilization down (or even boost iops). >>>>>>> >>>>>>> Would be great to find them. >>>>>>> >>>>>>> Stefan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
