On Tue, Feb 3, 2015 at 9:31 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: > Maybe more detail number can help us a bit. Here's what we're testing with and what we observe: Hardware: 2x6-core hyperthreaded Xeon E5-2620 v2 2.10GHz CPU 8x8 GiB DDR3 RAM 4x4 TB 7200 RPM 8 ms 183 MB/s SAS rotary disks Software: CentOS 7 CEPH 0.91 4 OSDs osd pool default size = 1 (just for testing!) keyvaluestore op threads = 16 keyvaluestore backend = mysql (our own backend) one MySQL process per OSD, each writing to a separate disk Test setup: rados bench on a fresh install 256 concurrent writes 360 seconds 2700 byte objects, and 1 MiB objects measure throughput with rados bench measure CPU usage by observing top measure max concurrent transaction submits by instrumenting the KeyValueDB interface With this setup, we observe that, with 2700 byte objects: 7.4 MiB/s (~2900 ops/s) throughput, 170%/170%/60%/60% OSD CPU usage, 200%/200%/65%/65% MySQL CPU usage, and 3/3/1/1 maximum concurrent transaction submits; and with 1 MiB objects: 50.7 MiB/s (~51 ops/s) throughput, 14%/14%/4%/4% OSD CPU usage, 50%/50%/15%/15% MySQL CPU usage, and 3/3/1/1 maximum concurrent transaction submits. We know that our transaction concurrency measurement is not buggy, as it will consistently report up to `keyvaluestore op threads` concurrent submits both on OSD startup on this same hardware, and during benchmarking in a resource-constrained VM. We are pretty sure MySQL is not the bottleneck, since we've been able to throw much more at it (concurrently); at least 10 kops/s per instance. (Sequentially it is not so good; hence our fixation on the low transaction concurrency!) Let me know if there are any other figures which would be helpful in diagnosing why the OSDs are not issuing as many concurrent transactions as we'd like, or why they are using so much CPU. Thanks for your help. - Chris -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
