Hi, I am just curious would this issue be also applied to other DB like postgresql? My team is currently looking at this but we are using a VM and install postgresql 9.3 on an attached device using RBD (via libvirt). We had yet to complete our planned tests but initial observations did indicate possible performance issues though we had yet to go through all possible options within postgresql. Regards, Luke -----Original Message----- From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Haomai Wang Sent: Wednesday, 21 May, 2014 6:22 PM To: ceph-devel@xxxxxxxxxxxxxxx Subject: Re: [Performance] Improvement on DB Performance I pushed the commit to fix this problem(https://github.com/ceph/ceph/pull/1848). With test program(Each sync request is issued with ten write request), a significant improvement is noticed. aio_flush sum: 914750 avg: 1239 count: 738 max: 4714 min: 1011 flush_set sum: 904200 avg: 1225 count: 738 max: 4698 min: 999 flush sum: 641648 avg: 173 count: 3690 max: 1340 min: 128 Compared to last mail, it reduce each aio_flush request to 1239 ns instead of 24145 ns. I hope it's the root cause for db on rbd performance. On Wed, May 21, 2014 at 6:15 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: > Hi all, > > I remember there exists discuss about DB(mysql) performance on rbd. > Recently I test mysql-bench with rbd and found awful performance. So I > dive into it and find that main cause is "flush" request from guest. > As we know, applications such as mysql, ceph has own journal for > durable and journal usually send sync&direct io. If fs barrier is on, > each sync io operation make kernel issue "sync"(barrier) request to > block device. Here, qemu will call "rbd_aio_flush" to apply. > > Via systemtap, I found a amazing thing: > aio_flush sum: 4177085 avg: 24145 count: > 173 max: 28172 min: 22747 > flush_set sum: 4172116 avg: 24116 count: > 173 max: 28034 min: 22733 > flush sum: 3029910 avg: 4 count: > 670477 max: 1893 min: 3 > > This statistic info is gathered in 5s. Most of consuming time is on > "ObjectCacher::flush". What's more, with time increasing, the flush > count will be increasing. > > After view source, I find the root cause is "ObjectCacher::flush_set", > it will iterator the "object_set" and look for dirty buffer. And > "object_set" contains all objects ever opened. For example: > > 2014-05-21 18:01:37.959013 7f785c7c6700 0 objectcacher flush_set > total: 5919 flushed: 5 > 2014-05-21 18:01:37.999698 7f785c7c6700 0 objectcacher flush_set > total: 5919 flushed: 5 > 2014-05-21 18:01:38.038405 7f785c7c6700 0 objectcacher flush_set > total: 5920 flushed: 5 > 2014-05-21 18:01:38.080118 7f785c7c6700 0 objectcacher flush_set > total: 5920 flushed: 5 > 2014-05-21 18:01:38.119792 7f785c7c6700 0 objectcacher flush_set > total: 5921 flushed: 5 > 2014-05-21 18:01:38.162004 7f785c7c6700 0 objectcacher flush_set > total: 5922 flushed: 5 > 2014-05-21 18:01:38.202755 7f785c7c6700 0 objectcacher flush_set > total: 5923 flushed: 5 > 2014-05-21 18:01:38.243880 7f785c7c6700 0 objectcacher flush_set > total: 5923 flushed: 5 > 2014-05-21 18:01:38.284399 7f785c7c6700 0 objectcacher flush_set > total: 5923 flushed: 5 > > These logs record the iteration info, the loop will check 5920 objects > but only 5 objects are dirty. > > So I think the solution is make "ObjectCacher::flush_set" only > iterator the objects which is dirty. > > -- > Best Regards, > > Wheat -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html ________________________________ DISCLAIMER: This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments). MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law. ------------------------------------------------------------------ - - DISCLAIMER: This e-mail (including any attachments) is for the addressee(s) only and may contain confidential information. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message. MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e- mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law. ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
