> -----Original Message----- > From: Sage Weil [mailto:sage@xxxxxxxxxxxx] > Sent: Monday, May 11, 2015 12:04 PM > To: Deneau, Tom > Cc: ceph-devel > Subject: Re: rados bench single instance vs. multiple instances > > On Mon, 11 May 2015, Deneau, Tom wrote: > > I have noticed the following while running rados bench seq read tests > > with a 40M object size > > > > single rados bench, 4 concurrent ops, bandwidth = > 190 MB/s > > 4 copies of rados bench, 1 concurrent op each, aggregate > > bandwidth = 310 MB/s > > > > and in fact the single rados bench seems limited to 190, no matter how many > concurrent ops. > > > > I don't see this kind of behavior with a 4M object size. > > > > (The above are with caches dropped on the osd targets) > > > > It doesn't seem to be related to the total number of bytes being > > processed by the single because if I don't drop the caches, both the > > single rados bench and the 4-copy rados bench get much higher numbers > > (600 vs. 900) but still the single rados bench appears limited, no matter > how many concurrent ops are used. > > > > Is there kind of throttling going on by design here? > > It might be the librados throttles: > > OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight > data (both directions) > OPTION(objecter_inflight_ops, OPT_U64, 1024) // max in-flight > ios > > IIRC these only affect librados.. which would include 'rados bench'. > > sage > I noticed those, and tried setting them higher but it didn't seem to have an effect. But maybe I didn't change it correctly. What I did: * stop all osds * add the following lines to ceph.conf [osd] objecter_inflight_op_bytes = 1048576000 * restart all osds * admin-daemon osd config show shows the new value * made sure /etc/ceph/ceph.conf on the client also had those lines added (although I was not sure if that was needed?) -- Tom -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
