On Mon, Dec 21, 2015 at 9:15 PM, Ben Hines <bhines@xxxxxxxxx> wrote: > I'd be curious to compare benchmarks. What size objects are you putting? As stated earlier, I ran rest-bench with 70KB objects which is a good approximation of the average object size in the underperforming system. > 10gig end to end from client to RGW server to OSDs? No but that shouldn't matter at all. You see, a reference system pulls 250 PUTs/s which amounts to approx. 17MB/s (which falls far short of what that same system can do for raw RADOS, in which case it happily saturates a Gigabit network). The problematic system is at less than a quarter of even that. > I wouldn't be surprised > if mine is pretty slow though in comparison, since we still don't have SSD > journals. So I have not paid much attention to upload speed. > > Our omap dirs are about 400MB on each OSD, and we have ~100 OSDs. ~20 > buckets with ~23 shards each and 500k-1M objects each, so the layout is much > different. Yeah, on the problematic system the per-OSD omap directories is about 50 times that. I'm still hoping for someone who is more familiar with LevelDB internals than me to explain the performance impact of large omap directories, and assuming smaller omap directories are beneficial, I wonder how it would be possible to get them back down to manageable sizes. Cheers, Florian _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |