Hi Mark, There are rgw and rados-level throttling parameters. There are known issues of fairness. The only scenario we know of where something like the "deadlock" you're theorizing can happen is possible only when byte-throttling is incorrectly configured. Matt ----- Original Message ----- > From: "Mark Nelson" <mnelson@xxxxxxxxxx> > To: "Orit Wasserman" <owasserm@xxxxxxxxxx> > Cc: "Matt Benjamin" <mbenjamin@xxxxxxxxxx>, "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>, cbt@xxxxxxxxxxxxxx, "Mark > Seger" <mjseger@xxxxxxxxx>, "Kyle Bader" <kbader@xxxxxxxxxx>, "Karan Singh" <karan@xxxxxxxxxx>, "Brent Compton" > <bcompton@xxxxxxxxxx> > Sent: Tuesday, February 7, 2017 10:23:05 AM > Subject: Re: CBT: New RGW getput benchmark and testing diary > > > > On 02/07/2017 09:03 AM, Orit Wasserman wrote: > > On Tue, Feb 7, 2017 at 4:47 PM, Mark Nelson <mnelson@xxxxxxxxxx> wrote: > >> Hi Orit, > >> > >> This was a pull from master over the weekend: > >> 5bf39156d8312d65ef77822fbede73fd9454591f > >> > >> Btw, I've been noticing that it appears when bucket index sharding is > >> used, > >> there's a higher likelyhood that client connection attempts are delayed or > >> starved out entirely under high concurrency. I haven't looked at the code > >> yet, does this match with what you'd expect to happen? I assume the > >> threadpool is shared? > >> > > yes it is shared. > > Ok, so that probably explains the behavior I'm seeing. Perhaps a more > serious issue: Do we have anything in place to stop a herd of clients > from connecting, starving out bucket index lookups, and making > everything deadlock? > > > > >> Mark > >> > >> > >> On 02/07/2017 07:50 AM, Orit Wasserman wrote: > >>> > >>> Mark, > >>> On what version did you run the tests? > >>> > >>> Orit > >>> > >>> On Mon, Feb 6, 2017 at 7:07 PM, Mark Nelson <mnelson@xxxxxxxxxx> wrote: > >>>> > >>>> > >>>> > >>>> On 02/06/2017 11:02 AM, Orit Wasserman wrote: > >>>>> > >>>>> > >>>>> On Mon, Feb 6, 2017 at 5:44 PM, Matt Benjamin <mbenjamin@xxxxxxxxxx> > >>>>> wrote: > >>>>>> > >>>>>> > >>>>>> Keep in mind, RGW does most of its request processing work in civetweb > >>>>>> threads, so high utilization there does not necessarily imply > >>>>>> civetweb-internal processing. > >>>>>> > >>>>> > >>>>> True but the request processing is not a CPU intensive operation. > >>>>> It does seems to indicate that the civetweb threading model simply > >>>>> doesn't scale (we already noticed it already) or maybe it can point to > >>>>> some locking issue. We need to run a profiler to understand what is > >>>>> consuming CPU. > >>>>> It maybe a simple fix until we move to asynchronous frontend. > >>>>> It worth investigating as the CPU usage mark is seeing is really high. > >>>> > >>>> > >>>> > >>>> The initial profiling I did definitely showed a lot of tcmalloc > >>>> threading > >>>> activity, which diminshed after increasing threadcache. This is quite > >>>> similar to what we saw in simplemessenger with low threadcache values, > >>>> though likely is less true with async messenger. Sadly a profiler like > >>>> perf > >>>> probably isn't going to help much with debugging lock contention. > >>>> grabbing > >>>> GDB stack traces might help, or lttng. > >>>> > >>>>> > >>>>> Mark, > >>>>> How many concurrent request were handled? > >>>> > >>>> > >>>> > >>>> Most of the tests had 128 concurrent IOs per radosgw daemon. The max > >>>> thread > >>>> count was increased to 512. It was very obvious when exceeding the > >>>> thread > >>>> count since some getput processes will end up stalling and doing their > >>>> writes after others, leading to bogus performance data. > >>>> > >>>> > >>>>> > >>>>> Orit > >>>>> > >>>>>> Matt > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>>> > >>>>>>> > >>>>>>> From: "Mark Nelson" <mnelson@xxxxxxxxxx> > >>>>>>> To: "Matt Benjamin" <mbenjamin@xxxxxxxxxx> > >>>>>>> Cc: "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>, cbt@xxxxxxxxxxxxxx, > >>>>>>> "Mark > >>>>>>> Seger" <mjseger@xxxxxxxxx>, "Kyle Bader" > >>>>>>> <kbader@xxxxxxxxxx>, "Karan Singh" <karan@xxxxxxxxxx>, "Brent > >>>>>>> Compton" > >>>>>>> <bcompton@xxxxxxxxxx> > >>>>>>> Sent: Monday, February 6, 2017 10:42:04 AM > >>>>>>> Subject: Re: CBT: New RGW getput benchmark and testing diary > >>>>>>> > >>>>>>> Just based on what I saw during these tests, it looks to me like a > >>>>>>> lot > >>>>>>> more time was spent dealing with civetweb's threads than RGW. I > >>>>>>> didn't > >>>>>>> look too closely, but it may be worth looking at whether there's any > >>>>>>> low > >>>>>>> hanging fruit in civetweb itself. > >>>>>>> > >>>>>>> Mark > >>>>>>> > >>>>>>> On 02/06/2017 09:33 AM, Matt Benjamin wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> Thanks for the detailed effort and analysis, Mark. > >>>>>>>> > >>>>>>>> As we get closer to the L time-frame, it should become relevant to > >>>>>>>> look > >>>>>>>> at > >>>>>>>> the relative boost::asio frontend rework i/o paths, which are the > >>>>>>>> open > >>>>>>>> effort to reduce CPU overhead/revise threading model, in general. > >>>>>>>> > >>>>>>>> Matt > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> From: "Mark Nelson" <mnelson@xxxxxxxxxx> > >>>>>>>>> To: "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>, cbt@xxxxxxxxxxxxxx > >>>>>>>>> Cc: "Mark Seger" <mjseger@xxxxxxxxx>, "Kyle Bader" > >>>>>>>>> <kbader@xxxxxxxxxx>, > >>>>>>>>> "Karan Singh" <karan@xxxxxxxxxx>, "Brent > >>>>>>>>> Compton" <bcompton@xxxxxxxxxx> > >>>>>>>>> Sent: Monday, February 6, 2017 12:55:20 AM > >>>>>>>>> Subject: CBT: New RGW getput benchmark and testing diary > >>>>>>>>> > >>>>>>>>> Hi All, > >>>>>>>>> > >>>>>>>>> Over the weekend I took a stab at improving our ability to run RGW > >>>>>>>>> performance tests in CBT. Previously the only way to do this was > >>>>>>>>> to > >>>>>>>>> use > >>>>>>>>> the cosbench plugin, which required a fair amount of additional > >>>>>>>>> setup and while quite powerful can be overkill in situations where > >>>>>>>>> you > >>>>>>>>> want to rapidly iterate over tests looking for specific issues. A > >>>>>>>>> while > >>>>>>>>> ago Mark Seger from HP told me he had created a swift benchmark > >>>>>>>>> called > >>>>>>>>> "getput" that is written in python and is much more convenient to > >>>>>>>>> run > >>>>>>>>> quickly in an automated fashion. Normally getput is used in > >>>>>>>>> conjunction > >>>>>>>>> with gpsuite, a tool for coordinating benchmarking multiple getput > >>>>>>>>> processes. This is how you would likely use getput on a typical > >>>>>>>>> ceph > >>>>>>>>> or > >>>>>>>>> swift cluster, but since CBT builds the cluster and has it's own > >>>>>>>>> way > >>>>>>>>> for > >>>>>>>>> launching multiple benchmark processes, it uses getput directly. > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> Matt Benjamin > >>>>>> Red Hat, Inc. > >>>>>> 315 West Huron Street, Suite 140A > >>>>>> Ann Arbor, Michigan 48103 > >>>>>> > >>>>>> http://www.redhat.com/en/technologies/storage > >>>>>> > >>>>>> tel. 734-821-5101 > >>>>>> fax. 734-769-8938 > >>>>>> cel. 734-216-5309 > >>>>>> -- > >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > >>>>>> in > >>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx > >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
