Re: rados bench seq throttling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




> -----Original Message-----
> From: Gregory Farnum [mailto:gfarnum@xxxxxxxxxx]
> Sent: Monday, September 14, 2015 5:32 PM
> To: Deneau, Tom
> Cc: ceph-users
> Subject: Re:  rados bench seq throttling
> 
> On Thu, Sep 10, 2015 at 1:02 PM, Deneau, Tom <tom.deneau@xxxxxxx> wrote:
> > Running 9.0.3 rados bench on a 9.0.3 cluster...
> > In the following experiments this cluster is only 2 osd nodes, 6 osds
> > each and a separate mon node (and a separate client running rados
> bench).
> >
> > I have two pools populated with 4M objects.  The pools are replicated
> > x2 with identical parameters.  The objects appear to be spread evenly
> across the 12 osds.
> >
> > In all cases I drop caches on all nodes before doing a rados bench seq
> test.
> > In all cases I run rados bench seq for identical times (30 seconds)
> > and in that time we do not run out of objects to read from the pool.
> >
> > I am seeing significant bandwidth differences between the following:
> >
> >    * running a single instance of rados bench reading from one pool with
> 32 threads
> >      (bandwidth approx 300)
> >
> >    * running two instances rados bench each reading from one of the two
> pools
> >      with 16 threads per instance (combined bandwidth approx. 450)
> >
> > I have already increased the following:
> >   objecter_inflight_op_bytes = 104857600000
> >   objecter_inflight_ops = 8192
> >   ms_dispatch_throttle_bytes = 1048576000  #didn't seem to have any
> > effect
> >
> > The disks and network are not reaching anywhere near 100% utilization
> >
> > What is the best way to diagnose what is throttling things in the one-
> instance case?
> 
> Pretty sure the rados bench main threads are just running into their
> limits. There's some work that Piotr (I think?) has been doing to make it
> more efficient if you want to browse the PRs, but I don't think they're
> even in a dev release yet.
> -Greg

Some further experiments with numbers of rados-bench clients:
   * All of the following are reading 4M sized objects with dropped caches as
     described above:
   * When we run multiple clients, they are run on different pools but from
     the same separate client node, which is not anywhere near CPU or network-limited
    * threads is the total across all clients, as is BW

Case 1: two node cluster, 3 osds on each node
total          BW      BW      BW 
threads       1 cli   2cli    4cli
-------       -----   ----    ----  
  4            174     185     194
  8            214     273     301
 16            198     309     399
 32            226     309     409
 64            246     341     421


Case 2: one node cluster, 6 osds on one node.
total          BW      BW      BW 
threads       1 cli   2cli    4cli
-------       -----   ----    ----  
  4           339      262     236
  8           465      426     383
 16           467      433     353
 32           470      432     339
 64           471      429     345

So, from the above data, having multiple clients definitely helps
in the 2-node case (Case 1) but hurts in the single-node case.
Still interested in any tools that would help analyze this more deeply...

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux