Thanks We tried the settings you recommened. For my cluster on Ubuntu 13.10, I have 4 OSDs on one host, I set : osd client message size cap = 52428800 osd client message cap = 50 >From client side, the write speed between 50~100MB/s, avg is 79MB/s. But on the centos 6.4 cluster, I upgrade the kernel to 3.8, it seems no performance improve: I have 2 hosts and each runs 3 OSDs(total 6 OSDs) The write speed is smooth, but speed is less than that run on Ubuntu. I think the replica write may result to this difference. -----邮件原件----- 发件人: Samuel Just [mailto:sam.just@xxxxxxxxxxx] 发送时间: 2013年10月8日 7:55 收件人: hjwsm1989@xxxxxxxxx 抄送: huangyellowhuang; ceph-devel 主题: Re: ceph on ubuntu and centos You might try osd client message size cap = 26214400 osd client message cap = 25. osd op threads = 8 and filestore op threads = 8 might also be good. Let us know what you find! Sounds like the the kernel is the most obvious candidate for slowness on Centos 6.4, is there a 3.0+ kernel around for Centos 6.4 you could try? -Sam On Mon, Oct 7, 2013 at 3:16 PM, hjwsm1989@xxxxxxxxx <hjwsm1989@xxxxxxxxx> wrote: > thanks for your reply ! > the ubuntu is 3.8,centos kernel version is 2.6.32. > which setting item should we change to get the smoth write speed ? > we tried tune some parameters: > osd op threads= 8 > filestore op threads = 8 > filestore max op queue = 30 > which one will have the largest effect on performance? > > thanks > > Samuel Just <sam.just@xxxxxxxxxxx>编写： > >>Interesting! What kernel versions were running on the 13.10 and >>centos 6.4 clusters? >>-Sam >> >>On Fri, Oct 4, 2013 at 6:33 PM, huangyellowhuang >><huangyellowhuang@xxxxxxx> wrote: >>> Hi,all >>> We test the ceph version 0.69 >>> (6ca6f2f9f754031f4acdb971b71c92c9762e18c3) on Ubuntu server 13.10 >>> and centos 6.4 final Our cluster configuration: >>> 3 host machine, each runs 3 OSDs(use XFS as backend fs),MON and MDS >>> runs on one of the three host, We have one KClient on Ubuntu server >>> 13.10 >>> >>> The cluster runs on Ubuntu works fine and a few ‘slow requests’ >>> msgs, about 100MB/s write speed. >>> But the cluster runs on centos is very bad, avg 30MB/s write speed, >>> many osd requests slow: >>> 2013-10-05 08:35:09.931145 mon.0 [INF] pgmap v928: 192 pgs: 192 >>> active+clean ; 50873 MB data, 101716 MB used, 13857 GB / 13956 GB >>> avail; 115 MB/s wr, 28 op/s >>> 2013-10-05 08:35:12.087614 mon.0 [INF] pgmap v929: 192 pgs: 192 >>> active+clean ; 50901 MB data, 101780 MB used, 13857 GB / 13956 GB >>> avail; 32593 KB/s wr, 8 op/s >>> 2013-10-05 08:35:03.963979 osd.0 [WRN] 37 slow requests, 1 included >>> below; o ldest blocked for > 798.235962 secs >>> 2013-10-05 08:35:03.963984 osd.0 [WRN] slow request 240.831078 >>> seconds old, received at 2013-10-05 08:31:03.132836: >>> osd_op(mds.0.1:375 200.00000000 [wri tefull 0~84] 1.844f3494 e47) v4 >>> currently no flag points reached >>> 2013-10-05 08:35:08.965134 osd.0 [WRN] 37 slow requests, 1 included >>> below; o ldest blocked for > 803.237127 secs >>> 2013-10-05 08:35:08.965139 osd.0 [WRN] slow request 480.312618 >>> seconds old, received at 2013-10-05 08:27:08.652461: >>> osd_op(mds.0.1:307 200.00000000 [wri tefull 0~84] 1.844f3494 e47) v4 >>> currently no flag points reached >>> 2013-10-05 08:35:10.965619 osd.0 [WRN] 37 slow requests, 1 included >>> below; o ldest blocked for > 805.237600 secs >>> 2013-10-05 08:35:10.965624 osd.0 [WRN] slow request 120.946652 >>> seconds old, received at 2013-10-05 08:33:10.018900: >>> osd_op(mds.0.1:404 200.00000000 [wri tefull 0~84] 1.844f3494 e47) v4 >>> currently no flag points reached >>> 2013-10-05 08:35:11.965986 osd.0 [WRN] 37 slow requests, 1 included >>> below; o ldest blocked for > 806.237800 secs >>> 2013-10-05 08:35:11.965992 osd.0 [WRN] slow request 60.474314 >>> seconds old, r eceived at 2013-10-05 08:34:11.491438: >>> osd_op(mds.0.1:430 200.00000000 [writ efull 0~84] 1.844f3494 e47) v4 >>> currently no flag points reached >>> >>> And we need to build a cluster have 4 hosts, each has 18 OSDs and 1 >>> kclient, every kclient server as samba server that serves 4 samba clients. >>> 1) Which linux distrubtion should we used? Centos or Ubuntu? >>> 2) What results in ceph performances so large on different distribution? >>> 3) It seems the bottleneck is underlayer fs can not handle requests >>> fast as ceph expect, bc the ‘slow requests’ shows if a request does >>> not handle after 30s. >>> >>> Thanks! >>> >>> >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe >>> ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>-- >>To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo >>info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
