I've reduced OSDs to 12 and moved journal to ssd drives and now have
"boost" with writes to ~33-35MB/s. Is it maximum without full ssd
pools?
Best,
Stan
2017-04-06 9:34 GMT+02:00 Stanislav Kopp <staskopp@xxxxxxxxx>:
> Hello,
>
> I'm evaluate ceph cluster, to see if you can use it for our
> virtualization solution (proxmox). I'm using 3 nodes, running Ubuntu
> 16.04 with stock ceph (10.2.6), every OSD uses separate 8 TB spinning
> drive (XFS), MONITORs are installed on the same nodes, all nodes are
> connected via 10G switch.
>
> The problem is, on client I have only ~25-30 MB/s with seq. write. (dd
> with "oflag=direct"). Proxmox uses Firefly, which is old, I know. But
> I have the same performance on my desktop running the same version as
> ceph nodes using rbd mount, iperf shows full speed (1GB or 10GB up to
> client).
> I know that this setup is not optimal and for production I will use
> separate MON nodes and ssd for OSDs, but was wondering is this
> performance still normal. This is my cluster status.
>
> cluster 3ea55c7e-5829-46d0-b83a-92c6798bde55
> health HEALTH_OK
> monmap e5: 3 mons at
> {ceph01=10.1.8.31:6789/0,ceph02=10.1.8.32:6789/0,ceph03=10.1.8.33:6789/0}
> election epoch 60, quorum 0,1,2 ceph01,ceph02,ceph03
> osdmap e570: 42 osds: 42 up, 42 in
> flags sortbitwise,require_jewel_osds
> pgmap v14784: 1024 pgs, 1 pools, 23964 MB data, 6047 objects
> 74743 MB used, 305 TB / 305 TB avail
> 1024 active+clean
>
> btw, bench on nodes itself looks good as far I see.
>
> ceph01:~# rados bench -p rbd 10 write
> ....
> Total time run: 10.159667
> Total writes made: 1018
> Write size: 4194304
> Object size: 4194304
> Bandwidth (MB/sec): 400.801
> Stddev Bandwidth: 38.2018
> Max bandwidth (MB/sec): 472
> Min bandwidth (MB/sec): 344
> Average IOPS: 100
> Stddev IOPS: 9
> Max IOPS: 118
> Min IOPS: 86
> Average Latency(s): 0.159395
> Stddev Latency(s): 0.110994
> Max latency(s): 1.1069
> Min latency(s): 0.0432668
>
>
> Thanks,
> Stan
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
