On 10/20/2014 06:27 AM, Mark Wu wrote:
Test result Update: Number of Hosts Maximum single volume IOPS Maximum aggregated IOPS SSD Disk IOPS SSD Disk Utilization 7 14k 45k 9800+ 90% 8 21k 50k 9800+ 90% 9 30k 56k 9800+ 90% 10 40k 54k 8200+ 70% Note: the disk average request size is about 20 sectors, not same as client side (4k) I have two questions about the result: 1. No matter how many nodes the cluster has, the backend write throughput is always almost 8 times of client side. Is it normal behavior in Ceph, or caused by some wrong configuration in my setup?
Are you counting journal writes and replication into this? Also note that journal writes will be slightly larger and padded to a 4K boundary for each write due to header information. I suspect for coalesced journal writes we may be able to pack the headers together to reduce this overhead.
The following data is captured in the 9 hosts test. Roughly, the aggregated backend write throughput is 1000 * 22 * 512 * 2 * 9 = 1980M/s The client side is 56k * 4 = 244M/s Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s wBlk_dir/s rBlk_svr/s wBlk_svr/s ops/s rops/s wops/s Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.33 0.00 1.33 0.00 10.67 8.00 0.00 0.00 0.00 0.00 sdb 0.00 6.00 0.00 10219.67 0.00 223561.67 21.88 4.08 0.40 0.09 89.43 sdc 0.00 6.00 0.00 9750.67 0.00 220286.67 22.59 2.47 0.25 0.09 89.83 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 1.33 0.00 10.67 8.00 0.00 0.00 0.00 0.00 Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s wBlk_dir/s rBlk_svr/s wBlk_svr/s ops/s rops/s wops/s Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 1.00 0.00 26.67 26.67 0.00 0.00 0.00 0.00 sdb 0.00 6.33 0.00 10389.00 0.00 224668.67 21.63 3.78 0.36 0.09 89.23 sdc 0.00 4.33 0.00 10106.67 0.00 217986.00 21.57 3.83 0.38 0.09 91.10 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 1.00 0.00 26.67 26.67 0.00 0.00 0.00 0.00 2. For the scalability issue ( 10 hosts performs worse than 9 hosts), is there any tuning suggestion to improve it?
Can you post exactly the test you are running and on how many hosts/volumes? That would help us debug.
Thanks! Mark _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
