+1 with 0.80 on Ubuntu 14.04. I have a 7-node cluster with 10 OSDs per node on a 10Gbps network. 3 of the nodes are also acting as monitors. Each node has a single 6-core CPU, 32GB of memory, 1 SSD for journals, and 36 3TB hard drives. I'm currently using 11 of the hard drives in each node, one for the OS and 10 for OSDs. I do have one SSD (75% full) providing journals for all 10 OSDs on each node, but the same setup used to be much faster. I had previously been testing writes with rbd-fuse and librbd on 0.72 and Ubuntu 12.04 on this cluster. I was averaging about 200MB/s with peaks well beyond that. This week I upgraded (clean install, not a true upgrade) all 7 nodes to Ubuntu 14.04 and Ceph 0.80, and now I'm getting 5MB/s writes with rbd-fuse. I think my next step is to go back to 0.72 on 14.04 to see if it's somehow related to the OS, but this feels like an issue related to the new Firefly features so far. I have tested both with an EC pool using an overlaid, writeback cache pool and with a 2x replicated pool directly with the same results. Any suggestions are welcome. I'll let you know if I find anything. Steve From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of wsnote Sent: Thursday, May 08, 2014 7:45 PM To: ceph-users at lists.ceph.com Subject: too slowly upload on ceph object storage Hi, everyone! I am testing ceph rgw and found it upload slowly. I don't know where may be the bottleneck. OS: CentOS 6.5 Version: Ceph 0.79 Hardware: CPU: 2 * quad-core Mem: 32GB Disk: 2TB*1+3TB*11 Network: 1*1GB Ethernet NIC Ceph Cluster: My cluster was composed of 4 servers(called ceph1-4). I installed a monister, a radosgw and 11 osd in every server. So the cluster had 4 servers(ceph1-4), 4 moniter, 4 radosgw, 44 osd. I configured ceph as ceph.com's documents said and didn't do any special config. Then I use s3cmd to test the Ceph Cluster. Test 1: In ceph1, upload a big file to the rgw in ceph1.Do test several times. The speed is about 10MB/s! It's too slowly! I upload files from ceph1 to ceph1. There is not any network latency. Test 2: In ceph1, upload a big file to the rgw in ceph2.Do test several times. The speed is also about 10MB/s! Test3: In each ceph server, upload a big file to the rgw in their own rgw in the mean time.Do test several times. The speed in each ceph server is about 1-3MB/s. The sum of the speeds is about 10MB/s! I use the command "iostat -kx 3" to watch the stress of disks. When testing, the iowait is lower than 1%, and %util is lower than 1% too. There may be some problem.The speed in one server is too slowly.And the total speed didn't increase with the number of rgw. Can anyone give some suggestion? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140509/60ffe5bb/attachment.htm>
 |