On Wed, Jul 29, 2015 at 11:23 AM, Mark Nelson <mnelson@xxxxxxxxxx> wrote:
On 07/29/2015 10:13 AM, Jake Young wrote:
On Tue, Jul 28, 2015 at 11:48 AM, SCHAER Frederic<frederic.schaer@xxxxxx <mailto:frederic.schaer@xxxxxx>> wrote:
>
> Hi again,
>
> So I have tried
> - changing the cpus frequency : either 1.6GHZ, or 2.4GHZ on all cores
> - changing the memory configuration, from "advanced ecc mode" to
"performance mode", boosting the memory bandwidth from 35GB/s to 40GB/s
> - plugged a second 10GB/s link and setup a ceph internal network
> - tried various "tuned-adm profile" such as "throughput-performance"
>
> This changed about nothing.
>
> If
> - the CPUs are not maxed out, and lowering the frequency doesn't
change a thing
> - the network is not maxed out
> - the memory doesn't seem to have an impact
> - network interrupts are spread across all 8 cpu cores and receive
queues are OK
> - disks are not used at their maximum potential (iostat shows my dd
commands produce much more tps than the 4MB ceph transfers...)
>
> Where can I possibly find a bottleneck ?????
>
> I'm /(almost) out of ideas/ ... :'(
>
> Regards
>
>
Frederic,
I was trying to optimize my ceph cluster as well and I looked at all of
the same things you described, which didn't help my performance noticeably.
The following network kernel tuning settings did help me significantly.
This is my /etc/sysctl.conf file on all of my hosts: ceph mons, ceph
osds and any client that connects to my ceph cluster.
# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104)
for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
#net.core.rmem_max = 56623104
#net.core.wmem_max = 56623104
# Use 128M buffers
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 67108864
net.core.wmem_default = 67108864
net.core.optmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
# Also increase the max packet backlog
net.core.somaxconn = 1024
# Increase the length of the processor input queue
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 10
# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0
# If your servers talk UDP, also up these limits
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192
# Disable source routing and redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
# Recommended when jumbo frames are enabled
net.ipv4.tcp_mtu_probing = 1
I have 40 Gbps links on my osd nodes, and 10 Gbps links on everything else.
Let me know if that helps.
Hi Jake,
Could you talk a little bit about what scenarios you've seen tuning this help? I noticed improvement in RGW performance in some cases with similar TCP tunings, but it would be good to understand what other folks are seeing and in what situations.
_______________________________________________
Jake
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
Hey Mark,
I'm only using RBD. My clients are all VMware, so I have a few iSCSI proxy VMs (using rbd enabled tgt). My workload is typically light random read/write, except for the periodic eager zeroing of multi terabyte volumes. Since there is no VAAI in tgt, this turns into heavy sequential writing.
I found the network tuning above helped to "open up" the connection from a single iSCSI proxy VM to the cluster.
Note that my osd nodes have both a public network interface as well as a dedicated private network interface, which are both 40G. I believe the network tuning also has another effect of improving the performance of the cluster network (where the replication data is sent across), because initially I had only applied the kernel tuning to the osd nodes and saw a performance improvement before I implemented it on the iSCSI proxy VMs.
I should mention that I did all of my testing back in firefly (about 1 year ago) and I haven't tried to remove these parameters from my cluster to see if there is a performance degradation now that I'm running Hammer.
I guess there is a similar dataflow with RGW and using RBD with an iSCSI proxy server. Both have few RADOS clients, which funnel the requests of many HTTP/iSCSI clients.
Jake
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
