Storage really slow on k8s

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I have a converged setup; as you suggest in
https://github.com/gluster/gluster-kubernetes.

It's a pretty standard installation. 3 nodes are dedicated. Each one
with a system + data drives. The data drive is 500 GiB.

I am suffering, for quite a while now, of extreme slowness. Here're
some network tests.

root@k8s2:~# iperf3 -Vc glusterfs0
iperf 3.1.3
Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP
Tue Nov 26 12:20:02 UTC 2019 x86_64
Time: Wed, 11 Mar 2020 01:47:45 GMT
Connecting to host glusterfs0, port 5201
      Cookie: k8s2.test.gva.cloudsigma.com.1583891
      TCP MSS: 1448 (default)
[  4] local 10.0.77.3 port 53642 connected to 10.0.77.33 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting
0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.02 GBytes  8.77 Gbits/sec  1459   3.15 MBytes
[  4]   1.00-2.00   sec  1022 MBytes  8.58 Gbits/sec  2284   3.15 MBytes
[  4]   2.00-3.00   sec  1005 MBytes  8.43 Gbits/sec  2230   3.15 MBytes
[  4]   3.00-4.00   sec   982 MBytes  8.24 Gbits/sec  2334   2.38 MBytes
[  4]   4.00-5.00   sec   988 MBytes  8.29 Gbits/sec  668   2.65 MBytes
[  4]   5.00-6.00   sec   983 MBytes  8.25 Gbits/sec  1370   2.86 MBytes
[  4]   6.00-7.00   sec  1010 MBytes  8.47 Gbits/sec  1017   2.12 MBytes
[  4]   7.00-8.00   sec   995 MBytes  8.35 Gbits/sec  2220   1.27 MBytes
[  4]   8.00-9.00   sec   923 MBytes  7.75 Gbits/sec  324   1.64 MBytes
[  4]   9.00-10.00  sec  1001 MBytes  8.40 Gbits/sec  532   1.96 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.72 GBytes  8.35 Gbits/sec  14438             sender
[  4]   0.00-10.00  sec  9.72 GBytes  8.35 Gbits/sec                  receiver
CPU Utilization: local/sender 28.6% (0.5%u/28.0%s), remote/receiver
22.5% (1.1%u/21.4%s)

iperf Done.

root@k8s2:~# iperf3 -Vc glusterfs1
iperf 3.1.3
Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP
Tue Nov 26 12:20:02 UTC 2019 x86_64
Time: Wed, 11 Mar 2020 01:49:10 GMT
Connecting to host glusterfs1, port 5201
      Cookie: k8s2.test.gva.cloudsigma.com.1583891
      TCP MSS: 1448 (default)
[  4] local 10.0.77.3 port 42554 connected to 10.0.77.34 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting
0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   941 MBytes  7.89 Gbits/sec  4309   2.03 MBytes
[  4]   1.00-2.00   sec  1017 MBytes  8.53 Gbits/sec  2577   2.21 MBytes
[  4]   2.00-3.03   sec   962 MBytes  7.83 Gbits/sec  943   2.44 MBytes
[  4]   3.03-4.00   sec   965 MBytes  8.35 Gbits/sec  2441   2.63 MBytes
[  4]   4.00-5.00   sec   951 MBytes  7.98 Gbits/sec  1785   2.86 MBytes
[  4]   5.00-6.00   sec   900 MBytes  7.55 Gbits/sec  1199   3.00 MBytes
[  4]   6.00-7.00   sec   988 MBytes  8.28 Gbits/sec  888    100 KBytes
[  4]   7.00-8.00   sec   892 MBytes  7.49 Gbits/sec  2382   2.34 MBytes
[  4]   8.00-9.00   sec   967 MBytes  8.11 Gbits/sec  808   2.61 MBytes
[  4]   9.00-10.00  sec   979 MBytes  8.21 Gbits/sec  3070   2.76 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.34 GBytes  8.02 Gbits/sec  20402             sender
[  4]   0.00-10.00  sec  9.34 GBytes  8.02 Gbits/sec                  receiver
CPU Utilization: local/sender 27.4% (0.5%u/26.9%s), remote/receiver
24.7% (0.7%u/23.9%s)

iperf Done.

root@k8s2:~# iperf3 -Vc glusterfs2
iperf 3.1.3
Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP
Tue Nov 26 12:20:02 UTC 2019 x86_64
Time: Wed, 11 Mar 2020 02:57:33 GMT
Connecting to host glusterfs2, port 5201
      Cookie: k8s2.test.gva.cloudsigma.com.1583895
      TCP MSS: 1448 (default)
[  4] local 10.0.77.3 port 59230 connected to 10.0.77.35 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting
0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1020 MBytes  8.56 Gbits/sec  1423   1.96 MBytes
[  4]   1.00-2.00   sec   997 MBytes  8.36 Gbits/sec  2377   2.28 MBytes
[  4]   2.00-3.00   sec  1022 MBytes  8.58 Gbits/sec  1409   2.56 MBytes
[  4]   3.00-4.00   sec   976 MBytes  8.19 Gbits/sec  1184   2.79 MBytes
[  4]   4.00-5.00   sec  1011 MBytes  8.48 Gbits/sec  1063   3.00 MBytes
[  4]   5.00-6.00   sec  1016 MBytes  8.52 Gbits/sec  2608   3.00 MBytes
[  4]   6.00-7.00   sec   979 MBytes  8.21 Gbits/sec  1337   1.73 MBytes
[  4]   7.00-8.00   sec  1011 MBytes  8.48 Gbits/sec  746   2.11 MBytes
[  4]   8.00-9.00   sec  1013 MBytes  8.50 Gbits/sec  2190   2.43 MBytes
[  4]   9.00-10.00  sec   930 MBytes  7.80 Gbits/sec  1284   2.67 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.74 GBytes  8.37 Gbits/sec  15621             sender
[  4]   0.00-10.00  sec  9.74 GBytes  8.37 Gbits/sec                  receiver
CPU Utilization: local/sender 26.5% (0.4%u/26.0%s), remote/receiver
30.1% (1.2%u/28.9%s)

iperf Done.

And some dd tests:

/mnt # dd if=/dev/zero of=file.img bs=4k count=1M
^C2082+0 records in
2082+0 records out
8527872 bytes (8.5 MB, 8.1 MiB) copied, 178.228 s, 47.8 kB/s

So, to me, it seems that something is completely wrong here and I
dunno why. I'd appreciate a little help. Installations are timing out
and stuff.

Any performance tips would be appreciated.

-- 
Renich Bon Ćirić
Senior SysAdmin
CloudSigma AG

====================
This email is from CLOUDSIGMA AG. The contents of this email and any
attachments are confidential to the intended recipient. They may not
be disclosed to or used by or copied in any way by anyone other than
the intended recipient. If this email is received in error, please
contact CLOUDSIGMA AG on +41 (0)44 585 39 07 quoting the name of the
sender and the email address to which it has been sent and then delete
it. Please note that neither CLOUDSIGMA AG nor the sender accepts any
responsibility for viruses and it is your responsibility to scan or
otherwise check this email and any attachments. CLOUDSIGMA AG is a
limited company registered in Canton Zürich, Switzerland (registered
number  CHE-115.204.434) with registered offices at Badenerstrasse
549, 8048 Zurich, Switzerland. For further information, please refer
to www.cloudsigma.com .
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux