Hi Cephers,
I've setup a production environment Ceph cluster with the Jewel release
(10.1.0 (96ae8bd25f31862dbd5302f304ebf8bf1166aba6)) consisting of 3 MON
Servers and 6 OSD Servers:
3x MON Servers:
2x Intel Xeon E5-2630v3@2.40Ghz
384GB RAM
2x 200G Intel DC3700 in RAID-1 for OS
1x InfiniBand ConnectX-3 ADPT DP
6x OSD Servers:
2x Intel Xeon E5-2650v2@2.60Ghz
128GB RAM
2x 200G Intel DC3700 in RAID-1 for OS
12x 800G Intel DC3510 (osd & journal) on same device
1x InfiniBand ConnectX-3 ADPT DP (one port on PUB network and the other
on the CLUS network)
ceph.conf file is:
[global]
fsid = xxxxxxxxxxxxxxxxxxxxxxxxxxx
mon_initial_members = cibm01, cibm02, cibm03
mon_host = xx.xx.xx.1,xx.xx.xx.2,xx.xx.xx.3
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
public_network = xx.xx.16.0/20
cluster_network = xx.xx.32.0/20
[mon]
[mon.cibm01]
host = cibm01
mon_addr = xx.xx.xx.1:6789
[mon.cibm02]
host = cibm02
mon_addr = xx.xx.xx.2:6789
[mon.cibm03]
host = cibm03
mon_addr = xx.xx.xx.3:6789
[osd]
osd_pool_default_size = 2
osd_pool_default_min_size = 1
## OSD Configuration ##
[osd.0]
host = cibn01
public_addr = xx.xx.17.1
cluster_addr = xx.xx.32.1
[osd.1]
host = cibn01
public_addr = xx.xx.17.1
cluster_addr = xx.xx.32.1
...
They are all running *Ubuntu 14.04.4 LTS*. Journals are 5GB partitions
on each disk, since all the OSD daemons are SSD disks (Intel DC3510
800G). For example:
sdc 8:32 0 745.2G 0 disk
|-sdc1 8:33 0 740.2G 0 part
/var/lib/ceph/osd/ceph-0
`-sdc2 8:34 0 5G 0 part
The purpose of this cluster will be to serve as a backend storage for
Cinder volumes (RBD) and Glance images in an OpenStack cloud, most of
the clusters on OpenStack will be non-relational databases like
Cassandra with many instances each.
All of the nodes of the cluster are running InfiniBand FDR 56Gb/s with
Mellanox Technologies MT27500 Family [ConnectX-3] adapters.
So I assume that performance will be really nice, right?...but.. I'm
getting some numbers that I think they could be really more important.
# rados --pool rbd bench 10 write -t 16
Total writes made: 1964
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): *755.435*
Stddev Bandwidth: 90.3288
Max bandwidth (MB/sec): 884
Min bandwidth (MB/sec): 612
Average IOPS: 188
Stddev IOPS: 22
Max IOPS: 221
Min IOPS: 153
Average Latency(s): 0.0836802
Stddev Latency(s): 0.147561
Max latency(s): 1.50925
Min latency(s): 0.0192736
Then I connect to another server (this one is running on QDR - so I
would expect something between 2-3Gb/s), I map a RBD on the host, then
create a ext4 fs and mount it, and finally run a fio test:
# fio --rw=randwrite --bs=4M --numjobs=8 --iodepth=32 --runtime=22
--time_based --size=10G --loops=1 --ioengine=libaio --direct=1
--invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap
--group_reporting --exitall --name cephV1 --filename=/mnt/host01v1/test1
fio-2.1.3
Starting 8 processes
cephIBV1: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 7 (f=7): [wwwwww_w] [100.0% done] [0KB/431.6MB/0KB /s] [0/107/0
iops] [eta 00m:00s]
cephIBV1: (groupid=0, jobs=8): err= 0: pid=6203: Thu Apr 7 15:24:12 2016
write: io=15284MB, bw=676412KB/s, iops=165, runt= 23138msec
slat (msec): min=1, max=480, avg=46.15, stdev=63.68
clat (msec): min=64, max=8966, avg=1459.91, stdev=1252.64
lat (msec): min=87, max=8969, avg=1506.06, stdev=1253.63
clat percentiles (msec):
| 1.00th=[ 235], 5.00th=[ 478], 10.00th=[ 611], 20.00th=[ 766],
| 30.00th=[ 889], 40.00th=[ 988], 50.00th=[ 1106], 60.00th=[ 1237],
| 70.00th=[ 1434], 80.00th=[ 1680], 90.00th=[ 2474], 95.00th=[ 4555],
| 99.00th=[ 6915], 99.50th=[ 7439], 99.90th=[ 8291], 99.95th=[ 8586],
| 99.99th=[ 8979]
bw (KB /s): min= 3091, max=209877, per=12.31%, avg=83280.51,
stdev=35226.98
lat (msec) : 100=0.16%, 250=0.97%, 500=4.61%, 750=12.93%, 1000=22.61%
lat (msec) : 2000=45.04%, >=2000=13.69%
cpu : usr=0.87%, sys=4.77%, ctx=6803, majf=0, minf=16337
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.7%, 16=3.3%, 32=93.5%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=3821/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=15284MB, aggrb=676411KB/s, minb=676411KB/s,
maxb=676411KB/s, mint=23138msec, maxt=23138msec
Disk stats (read/write):
rbd1: ios=0/4189, merge=0/26613, ticks=0/2852032, in_queue=2857996,
util=99.08%
Does it look acceptable? I mean for an InfiniBand network, I guess that
throughput need to be better. How much more can I expect to achieve by
tuning the servers? The MTU on the OSD servers is:
MTU: 65520
Any drop packet found
txqueuelen:256
Also I've setup on the openib.conf file:
...
SET_IPOIB_CM=yes
IPOIB_MTU=65520
...
And on mlnx.conf file:
...
options mlx4_core enable_sys_tune=1
options mlx4_core log_num_mgm_entry_size=-7
Anyone here with experience on Infiniband setups can give me any hint in
order to 'improve' performance, I'm getting similar numbers with another
cluster on a 10GbE network :S
