Hi, Serkan
I’m using Glusterfs as my Kubernetes storage backend, and Prometheus is running inside Kubernetes cluster. If this is not recommended I’ll consider running Prometheus outside of Kubernetes. But why performance is totally different on two
nodes? Will data be written to both nodes at the same time?
From: Serkan Çoban
Sent: Friday, March 31, 2017 4:08 PM
To: Yong Zhang
Cc: gluster-users@xxxxxxxxxxx
Subject: Re: [Gluster-users] High CPU/load, High latency on FSYNC
I do not recommend to use data volume of prometheus on glusterfs.
Hi, All
I have glusterfs 3.10.0 running with two replicated nodes, but one of the node has a bad CPU performance:
Node1/Node2: 4 core, 8GB, using thin LVM for volumes
This is node1:
And this is node2:
Currently I have only Prometheus using glusterfs volumes, so I started profile on volume prometheus and got this:
Brick: 10.10.0.100:/glusterfsvolumes/
devops/devops-prometheus/brick ------------------------------
------------------------------ ------- Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575 bytes
Data Written: 83056358475 bytes
Brick: 10.10.0.101:/glusterfsvolumes/
devops/devops-prometheus/brick ------------------------------
------------------------------ ------- Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0
0 0 No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Seems latency of FSYNC is very high. Sometimes cpu wait is also very high:
top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70
Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, 36.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem
Any ideas?
Thanks.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
