On 09/13/2016 06:46 AM, Owen Synge wrote:
Dear all,
Often issues arise with badly configured network switches, vlans, and
such like. Knowing each node routes to is a major deployment fail and
can be difficult to diagnose.
The brief looks like this:
Description:
* Diagnose network issues quickly for ceph.
* Identify network issues before deploying ceph.
A typical deployment will have 2 networks and potentially 3.
* External network for client access.
* Internal network for data replication. (Strongly recommended)
* Administration network. (Optional)
Typically we will have salt available on all nodes, but it would be the
same for ansible or an other config management solution.
* This will make injection of IP addresses and hosts trivial.
Before I go any further with developing a solution, does anyone know of
a pre made solution, that will avoid me writing much code, or ideally
any code.
So far I have only found tools for testing connectivity between 1 point
and another, not for testing 1:N or N:N.
We simulated 1:N and N:N throughput tests by wrapping iperf in simple
shell scripts and simply executing it in parallel on every host using
pdsh. These are really good tests to perform as it can show problems in
the switch itself. In one instance, a customer's bonding setup was
causing extreme throughput degradation, but only on certain routes, but
it only showed up in 1:N and N:N situations (primarily N:N). These were
just throw away scripts, but the idea works really well in practice so
something a bit more substantial might work very well. Way down on the
todo list is to add something like this to cbt for a pre-benchmark
network test.
1:N
---
Script to launch iperf servers:
#!/bin/bash
for i in 8 9 10 11 12 13
do
val=$((62+$i))
pdsh -R ssh -w osd[$i] iperf -s -B 192.168.1.$val &
done
Script to launch iperf clients:
#!/bin/bash
for i in 0 1 2 3 4 5 6 7
do
for val in 70 71 72 73 74 75
do
pdsh -R ssh -w client[$i] iperf -c 192.168.1.$val -f m -t 60 -P 1 >
/tmp/iperf_client${i}_to_${val}.out &
done
done
N:N
---
for i in 8 9 10 11 12 13
do
val=$((62+$i))
pdsh -R ssh -w osd[$i] iperf -s -B 192.168.1.$val &
done
#!/bin/bash
for i in 8 9 10 11 12 13
do
for val in 70 71 72 73 74 75
do
pdsh -R ssh -w osd[$i] iperf -c 192.168.1.$val -P 1 -f m -t 60 -P 1
> /tmp/iperf_${i}_to_${val}.out &
done
done
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
