Wow, thanks. I think that¹s the tidbit of info I needed to explain why increasing numjobs doesn¹t (anymore) scale performance as expected. Warren Wang On 8/4/16, 7:49 AM, "ceph-users on behalf of Jason Dillaman" <ceph-users-bounces@xxxxxxxxxxxxxx on behalf of jdillama@xxxxxxxxxx> wrote: >With exclusive-lock, only a single client can have write access to the >image at a time. Therefore, if you are using multiple fio processes >against the same image, they will be passing the lock back and forth >between each other and you can expect bad performance. > >If you have a use-case where you really need to share the same image >between multiple concurrent clients, you will need to disable the >exclusive-lock feature (this can be done with the RBD cli on existing >images or by passing "--image-shared" when creating new images). > >On Thu, Aug 4, 2016 at 5:52 AM, Alexandre DERUMIER <aderumier@xxxxxxxxx> >wrote: >> Hi, >> >> I think this is because of exclusive-lock feature enabled by default >>since jessie on rbd image >> >> >> ----- Mail original ----- >> De: "Zhiyuan Wang" <zhiyuan.wang@xxxxxxxxxxx> >> À: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >> Envoyé: Jeudi 4 Août 2016 11:37:04 >> Objet: Bad performance when two fio write to the same image >> >> >> >> Hi Guys >> >> I am testing the performance of Jewel (10.2.2) with FIO, but found the >>performance would drop dramatically when two process write to the same >>image. >> >> My environment: >> >> 1. Server: >> >> One mon and four OSDs running on the same server. >> >> Intel P3700 400GB SSD which have 4 partitions, and each for one osd >>journal (journal size is 10GB) >> >> Inter P3700 400GB SSD which have 4 partitions, and each format to XFS >>for one osd data (each data is 90GB) >> >> 10GB network >> >> CPU: Intel(R) Xeon(R) CPU E5-2660 (it is not the bottleneck) >> >> Memory: 256GB (it is not the bottleneck) >> >> 2. Client >> >> 10GB network >> >> CPU: Intel(R) Xeon(R) CPU E5-2660 (it is not the bottleneck) >> >> Memory: 256GB (it is not the bottleneck) >> >> 3. Ceph >> >> Default configuration expect use async messager (have tried simple >>messager, got nearly the same result) >> >> 10GB image with 256 pg num >> >> Test Case >> >> 1. One Fio process: bs 4KB; iodepth 256; direct 1; ioengine rbd; >>randwrite >> >> The performance is nearly 60MB/s and IOPS is nearly 15K >> >> Four osd are nearly the same busy >> >> 2. Two Fio process: bs 4KB; iodepth 256; direct 1; ioengine rbd; >>randwrite (write to the same image) >> >> The performance is nearly 4MB/s each, and IOPS is nearly 1.5K each >>Terrible >> >> And I found that only one osd is busy, the other three are much more >>idle on CPU >> >> And I also run FIO on two clients, the same result >> >> 3. Two Fio process: bs 4KB; iodepth 256; direct 1; ioengine rbd >>randwrite (one to image1, one to image2) >> >> The performance is nearly 35MB/s each and IOPS is nearly 8.5K each >>Reasonable >> >> Four osd are nearly the same busy >> >> >> >> >> >> Could someone help to explain the reason of TEST 2 >> >> >> >> Thanks >> >> >> Email Disclaimer & Confidentiality Notice >> >> This message is confidential and intended solely for the use of the >>recipient to whom they are addressed. If you are not the intended >>recipient you should not deliver, distribute or copy this e-mail. Please >>notify the sender immediately by e-mail and delete this e-mail from your >>system. Copyright © 2016 by Istuary Innovation Labs, Inc. All rights >>reserved. >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > >-- >Jason >_______________________________________________ >ceph-users mailing list >ceph-users@xxxxxxxxxxxxxx >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com This email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error destroy it immediately. *** Walmart Confidential *** _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |