Hello All,
I am trying ceph-firefly 0.80.8
(69eaad7f8308f21573c604f121956e64679a52a7) with 9 OSD ,all Samsung SSD
850 EVO on 3 servers with 24 G RAM,16 cores @2.27 Ghz Ubuntu 14.04 LTS
with 3.16-3 kernel.All are connected to 10G ports with maximum
MTU.There are no extra disks for journaling and also there are no
separate network for replication and data transfer.All 3 nodes are
also hosting monitoring process.Operating system runs on SATA disk.
When doing a sequential benchmark using "dd" on RBD, mounted on client
as ext4 its taking 110s to write 100Mb data at an average speed of
926Kbps.
time dd if=/dev/zero of=hello bs=4k count=25000 oflag=direct
25000+0 records in
25000+0 records out
102400000 bytes (102 MB) copied, 110.582 s, 926 kB/s
real 1m50.585s
user 0m0.106s
sys 0m2.233s
While doing this directly on ssd mount point shows:
time dd if=/dev/zero of=hello bs=4k count=25000
oflag=direct
25000+0 records in
25000+0 records out
102400000 bytes (102 MB) copied, 1.38567
s, 73.9 MB/s
OSDs are in XFS with these extra arguments :
rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M
ceph.conf
[global]
fsid = 7d889081-7826-439c-9fe5-d4e57480d9be
mon_initial_members = ceph1, ceph2, ceph3
mon_host = 10.99.10.118,10.99.10.119,10.99.10.120
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 450
osd_pool_default_pgp_num = 450
max_open_files = 131072
[osd]
osd_mkfs_type = xfs
osd_op_threads = 8
osd_disk_threads = 4
osd_mount_options_xfs = "rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M"
on our traditional storage with Full SAS disk, same "dd" completes in
16s with an average write speed of 6Mbps.
Rados bench:
rados bench -p rbd 10 write
Maintaining 16 concurrent writes of 4194304 bytes for up to 10
seconds or 0 objects
Object prefix: benchmark_data_ceph1_2977
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 16 94 78 311.821 312 0.041228 0.140132
2 16 192 176 351.866 392 0.106294 0.175055
3 16 275 259 345.216 332 0.076795 0.166036
4 16 302 286 285.912 108 0.043888 0.196419
5 16 395 379 303.11 372 0.126033 0.207488
6 16 501 485 323.242 424 0.125972 0.194559
7 16 621 605 345.621 480 0.194155 0.183123
8 16 730 714 356.903 436 0.086678 0.176099
9 16 814 798 354.572 336 0.081567 0.174786
10 16 832 816 326.313 72 0.037431 0.182355
11 16 833 817 297.013 4 0.533326 0.182784
Total time run: 11.489068
Total writes made: 833
Write size: 4194304
Bandwidth (MB/sec): 290.015
Stddev Bandwidth: 175.723
Max bandwidth (MB/sec): 480
Min bandwidth (MB/sec): 0
Average Latency: 0.220582
Stddev Latency: 0.343697
Max latency: 2.85104
Min latency: 0.035381
Our ultimate aim is to replace existing SAN with ceph,but for that it
should meet minimum 8000 iops.Can any one help me with this,OSD are
SSD,CPU has good clock speed,backend network is good but still we are
not able to extract full capability of SSD disks.
Thanks,
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
