On Thu, May 28, 2015 at 4:09 PM, Deneau, Tom <tom.deneau@xxxxxxx> wrote: > I've noticed that > * with a single node cluster with 4 osds > * and running rados bench rand on that same node so no network traffic > * with a number of objects small enough so that everything is in the cache so no disk traffic > > we still peak out at about 1600 MB/sec. > > And the cpu is 40% idle. (and a good chunk of the cpu activity is the rados benchmark itself) > > What is likely causing the throttling here? Well, rados bench itself is essentially single-threaded, so if it's using 100% CPU that's probably the bottleneck you're hitting. Otherwise, by default it will limit itself to 100MB of outstanding IO (there's an objecter config value you can change for this; it's been discussed recently) and that might not be enough given the latencies of hopping packets across different CPUs, and the OSDs have a slightly-embarrassing amount of CPU computation and thread hopping they have to perform on every op (around half a millisecond's worth on each read, I think?). -Gerg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
