On Wed, 13 May 2015, Deneau, Tom wrote: > > > -----Original Message----- > > From: Sage Weil [mailto:sage@xxxxxxxxxxxx] > > Sent: Monday, May 11, 2015 12:04 PM > > To: Deneau, Tom > > Cc: ceph-devel > > Subject: Re: rados bench single instance vs. multiple instances > > > > On Mon, 11 May 2015, Deneau, Tom wrote: > > > I have noticed the following while running rados bench seq read tests > > > with a 40M object size > > > > > > single rados bench, 4 concurrent ops, bandwidth = > > 190 MB/s > > > 4 copies of rados bench, 1 concurrent op each, aggregate > > > bandwidth = 310 MB/s > > > > > > and in fact the single rados bench seems limited to 190, no matter how many > > concurrent ops. > > > > > > I don't see this kind of behavior with a 4M object size. > > > > > > (The above are with caches dropped on the osd targets) > > > > > > It doesn't seem to be related to the total number of bytes being > > > processed by the single because if I don't drop the caches, both the > > > single rados bench and the 4-copy rados bench get much higher numbers > > > (600 vs. 900) but still the single rados bench appears limited, no matter > > how many concurrent ops are used. > > > > > > Is there kind of throttling going on by design here? > > > > It might be the librados throttles: > > > > OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight > > data (both directions) > > OPTION(objecter_inflight_ops, OPT_U64, 1024) // max in-flight > > ios > > > > IIRC these only affect librados.. which would include 'rados bench'. > > > > sage > > > > Just a follow-up that changing the limits of the two options mentioned above > did indeed solve my problem. Great! > Also, my naïve understanding of the architecture was that things like RBD and RGW were > layered on librados as shown in http://ceph.com/docs/master/architecture/. So wouldn't these > throttles apply to those stacks as well? Yep! sage
