Thanks Matt!
Just so I understand, how does the byte throttling impact the number of
threads used under a heavy client connection scenario? IE if you have
2000 threads and 2000 clients connect (1 thread per client?), What
ensures that additional threads are available for bucket index lookups?
Sorry for the weedy questions, just trying to make sure I understand how
this all works since I've never really looked closely at it and I'm
seeing some strange behavior.
Mark
On 02/07/2017 10:02 AM, Matt Benjamin wrote:
Hi Mark,
There are rgw and rados-level throttling parameters. There are known issues of fairness. The only scenario we know of where something like the "deadlock" you're theorizing can happen is possible only when byte-throttling is incorrectly configured.
Matt
----- Original Message -----
From: "Mark Nelson" <mnelson@xxxxxxxxxx>
To: "Orit Wasserman" <owasserm@xxxxxxxxxx>
Cc: "Matt Benjamin" <mbenjamin@xxxxxxxxxx>, "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>, cbt@xxxxxxxxxxxxxx, "Mark
Seger" <mjseger@xxxxxxxxx>, "Kyle Bader" <kbader@xxxxxxxxxx>, "Karan Singh" <karan@xxxxxxxxxx>, "Brent Compton"
<bcompton@xxxxxxxxxx>
Sent: Tuesday, February 7, 2017 10:23:05 AM
Subject: Re: CBT: New RGW getput benchmark and testing diary
On 02/07/2017 09:03 AM, Orit Wasserman wrote:
On Tue, Feb 7, 2017 at 4:47 PM, Mark Nelson <mnelson@xxxxxxxxxx> wrote:
Hi Orit,
This was a pull from master over the weekend:
5bf39156d8312d65ef77822fbede73fd9454591f
Btw, I've been noticing that it appears when bucket index sharding is
used,
there's a higher likelyhood that client connection attempts are delayed or
starved out entirely under high concurrency. I haven't looked at the code
yet, does this match with what you'd expect to happen? I assume the
threadpool is shared?
yes it is shared.
Ok, so that probably explains the behavior I'm seeing. Perhaps a more
serious issue: Do we have anything in place to stop a herd of clients
from connecting, starving out bucket index lookups, and making
everything deadlock?
Mark
On 02/07/2017 07:50 AM, Orit Wasserman wrote:
Mark,
On what version did you run the tests?
Orit
On Mon, Feb 6, 2017 at 7:07 PM, Mark Nelson <mnelson@xxxxxxxxxx> wrote:
On 02/06/2017 11:02 AM, Orit Wasserman wrote:
On Mon, Feb 6, 2017 at 5:44 PM, Matt Benjamin <mbenjamin@xxxxxxxxxx>
wrote:
Keep in mind, RGW does most of its request processing work in civetweb
threads, so high utilization there does not necessarily imply
civetweb-internal processing.
True but the request processing is not a CPU intensive operation.
It does seems to indicate that the civetweb threading model simply
doesn't scale (we already noticed it already) or maybe it can point to
some locking issue. We need to run a profiler to understand what is
consuming CPU.
It maybe a simple fix until we move to asynchronous frontend.
It worth investigating as the CPU usage mark is seeing is really high.
The initial profiling I did definitely showed a lot of tcmalloc
threading
activity, which diminshed after increasing threadcache. This is quite
similar to what we saw in simplemessenger with low threadcache values,
though likely is less true with async messenger. Sadly a profiler like
perf
probably isn't going to help much with debugging lock contention.
grabbing
GDB stack traces might help, or lttng.
Mark,
How many concurrent request were handled?
Most of the tests had 128 concurrent IOs per radosgw daemon. The max
thread
count was increased to 512. It was very obvious when exceeding the
thread
count since some getput processes will end up stalling and doing their
writes after others, leading to bogus performance data.
Orit
Matt
----- Original Message -----
From: "Mark Nelson" <mnelson@xxxxxxxxxx>
To: "Matt Benjamin" <mbenjamin@xxxxxxxxxx>
Cc: "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>, cbt@xxxxxxxxxxxxxx,
"Mark
Seger" <mjseger@xxxxxxxxx>, "Kyle Bader"
<kbader@xxxxxxxxxx>, "Karan Singh" <karan@xxxxxxxxxx>, "Brent
Compton"
<bcompton@xxxxxxxxxx>
Sent: Monday, February 6, 2017 10:42:04 AM
Subject: Re: CBT: New RGW getput benchmark and testing diary
Just based on what I saw during these tests, it looks to me like a
lot
more time was spent dealing with civetweb's threads than RGW. I
didn't
look too closely, but it may be worth looking at whether there's any
low
hanging fruit in civetweb itself.
Mark
On 02/06/2017 09:33 AM, Matt Benjamin wrote:
Thanks for the detailed effort and analysis, Mark.
As we get closer to the L time-frame, it should become relevant to
look
at
the relative boost::asio frontend rework i/o paths, which are the
open
effort to reduce CPU overhead/revise threading model, in general.
Matt
----- Original Message -----
From: "Mark Nelson" <mnelson@xxxxxxxxxx>
To: "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>, cbt@xxxxxxxxxxxxxx
Cc: "Mark Seger" <mjseger@xxxxxxxxx>, "Kyle Bader"
<kbader@xxxxxxxxxx>,
"Karan Singh" <karan@xxxxxxxxxx>, "Brent
Compton" <bcompton@xxxxxxxxxx>
Sent: Monday, February 6, 2017 12:55:20 AM
Subject: CBT: New RGW getput benchmark and testing diary
Hi All,
Over the weekend I took a stab at improving our ability to run RGW
performance tests in CBT. Previously the only way to do this was
to
use
the cosbench plugin, which required a fair amount of additional
setup and while quite powerful can be overkill in situations where
you
want to rapidly iterate over tests looking for specific issues. A
while
ago Mark Seger from HP told me he had created a swift benchmark
called
"getput" that is written in python and is much more convenient to
run
quickly in an automated fashion. Normally getput is used in
conjunction
with gpsuite, a tool for coordinating benchmarking multiple getput
processes. This is how you would likely use getput on a typical
ceph
or
swift cluster, but since CBT builds the cluster and has it's own
way
for
launching multiple benchmark processes, it uses getput directly.
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
