Hello, I have a converged setup; as you suggest in https://github.com/gluster/gluster-kubernetes. It's a pretty standard installation. 3 nodes are dedicated. Each one with a system + data drives. The data drive is 500 GiB. I am suffering, for quite a while now, of extreme slowness. Here're some network tests. root@k8s2:~# iperf3 -Vc glusterfs0 iperf 3.1.3 Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 Time: Wed, 11 Mar 2020 01:47:45 GMT Connecting to host glusterfs0, port 5201 Cookie: k8s2.test.gva.cloudsigma.com.1583891 TCP MSS: 1448 (default) [ 4] local 10.0.77.3 port 53642 connected to 10.0.77.33 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.02 GBytes 8.77 Gbits/sec 1459 3.15 MBytes [ 4] 1.00-2.00 sec 1022 MBytes 8.58 Gbits/sec 2284 3.15 MBytes [ 4] 2.00-3.00 sec 1005 MBytes 8.43 Gbits/sec 2230 3.15 MBytes [ 4] 3.00-4.00 sec 982 MBytes 8.24 Gbits/sec 2334 2.38 MBytes [ 4] 4.00-5.00 sec 988 MBytes 8.29 Gbits/sec 668 2.65 MBytes [ 4] 5.00-6.00 sec 983 MBytes 8.25 Gbits/sec 1370 2.86 MBytes [ 4] 6.00-7.00 sec 1010 MBytes 8.47 Gbits/sec 1017 2.12 MBytes [ 4] 7.00-8.00 sec 995 MBytes 8.35 Gbits/sec 2220 1.27 MBytes [ 4] 8.00-9.00 sec 923 MBytes 7.75 Gbits/sec 324 1.64 MBytes [ 4] 9.00-10.00 sec 1001 MBytes 8.40 Gbits/sec 532 1.96 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 9.72 GBytes 8.35 Gbits/sec 14438 sender [ 4] 0.00-10.00 sec 9.72 GBytes 8.35 Gbits/sec receiver CPU Utilization: local/sender 28.6% (0.5%u/28.0%s), remote/receiver 22.5% (1.1%u/21.4%s) iperf Done. root@k8s2:~# iperf3 -Vc glusterfs1 iperf 3.1.3 Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 Time: Wed, 11 Mar 2020 01:49:10 GMT Connecting to host glusterfs1, port 5201 Cookie: k8s2.test.gva.cloudsigma.com.1583891 TCP MSS: 1448 (default) [ 4] local 10.0.77.3 port 42554 connected to 10.0.77.34 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 941 MBytes 7.89 Gbits/sec 4309 2.03 MBytes [ 4] 1.00-2.00 sec 1017 MBytes 8.53 Gbits/sec 2577 2.21 MBytes [ 4] 2.00-3.03 sec 962 MBytes 7.83 Gbits/sec 943 2.44 MBytes [ 4] 3.03-4.00 sec 965 MBytes 8.35 Gbits/sec 2441 2.63 MBytes [ 4] 4.00-5.00 sec 951 MBytes 7.98 Gbits/sec 1785 2.86 MBytes [ 4] 5.00-6.00 sec 900 MBytes 7.55 Gbits/sec 1199 3.00 MBytes [ 4] 6.00-7.00 sec 988 MBytes 8.28 Gbits/sec 888 100 KBytes [ 4] 7.00-8.00 sec 892 MBytes 7.49 Gbits/sec 2382 2.34 MBytes [ 4] 8.00-9.00 sec 967 MBytes 8.11 Gbits/sec 808 2.61 MBytes [ 4] 9.00-10.00 sec 979 MBytes 8.21 Gbits/sec 3070 2.76 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 9.34 GBytes 8.02 Gbits/sec 20402 sender [ 4] 0.00-10.00 sec 9.34 GBytes 8.02 Gbits/sec receiver CPU Utilization: local/sender 27.4% (0.5%u/26.9%s), remote/receiver 24.7% (0.7%u/23.9%s) iperf Done. root@k8s2:~# iperf3 -Vc glusterfs2 iperf 3.1.3 Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 Time: Wed, 11 Mar 2020 02:57:33 GMT Connecting to host glusterfs2, port 5201 Cookie: k8s2.test.gva.cloudsigma.com.1583895 TCP MSS: 1448 (default) [ 4] local 10.0.77.3 port 59230 connected to 10.0.77.35 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1020 MBytes 8.56 Gbits/sec 1423 1.96 MBytes [ 4] 1.00-2.00 sec 997 MBytes 8.36 Gbits/sec 2377 2.28 MBytes [ 4] 2.00-3.00 sec 1022 MBytes 8.58 Gbits/sec 1409 2.56 MBytes [ 4] 3.00-4.00 sec 976 MBytes 8.19 Gbits/sec 1184 2.79 MBytes [ 4] 4.00-5.00 sec 1011 MBytes 8.48 Gbits/sec 1063 3.00 MBytes [ 4] 5.00-6.00 sec 1016 MBytes 8.52 Gbits/sec 2608 3.00 MBytes [ 4] 6.00-7.00 sec 979 MBytes 8.21 Gbits/sec 1337 1.73 MBytes [ 4] 7.00-8.00 sec 1011 MBytes 8.48 Gbits/sec 746 2.11 MBytes [ 4] 8.00-9.00 sec 1013 MBytes 8.50 Gbits/sec 2190 2.43 MBytes [ 4] 9.00-10.00 sec 930 MBytes 7.80 Gbits/sec 1284 2.67 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 9.74 GBytes 8.37 Gbits/sec 15621 sender [ 4] 0.00-10.00 sec 9.74 GBytes 8.37 Gbits/sec receiver CPU Utilization: local/sender 26.5% (0.4%u/26.0%s), remote/receiver 30.1% (1.2%u/28.9%s) iperf Done. And some dd tests: /mnt # dd if=/dev/zero of=file.img bs=4k count=1M ^C2082+0 records in 2082+0 records out 8527872 bytes (8.5 MB, 8.1 MiB) copied, 178.228 s, 47.8 kB/s So, to me, it seems that something is completely wrong here and I dunno why. I'd appreciate a little help. Installations are timing out and stuff. Any performance tips would be appreciated. -- Renich Bon Ćirić Senior SysAdmin CloudSigma AG ==================== This email is from CLOUDSIGMA AG. The contents of this email and any attachments are confidential to the intended recipient. They may not be disclosed to or used by or copied in any way by anyone other than the intended recipient. If this email is received in error, please contact CLOUDSIGMA AG on +41 (0)44 585 39 07 quoting the name of the sender and the email address to which it has been sent and then delete it. Please note that neither CLOUDSIGMA AG nor the sender accepts any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. CLOUDSIGMA AG is a limited company registered in Canton Zürich, Switzerland (registered number CHE-115.204.434) with registered offices at Badenerstrasse 549, 8048 Zurich, Switzerland. For further information, please refer to www.cloudsigma.com . ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
