|
Hi, All I have glusterfs 3.10.0 running with two replicated nodes, but one of the node has a bad CPU performance: Node1/Node2: 4 core, 8GB, using thin LVM for volumes This is node1:
And this is node2:
Currently I have only Prometheus using glusterfs volumes, so I started profile on volume prometheus and got this: Brick: 10.10.0.100:/glusterfsvolumes/devops/devops-prometheus/brick ------------------------------------------------------------------- Cumulative Stats: Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+ No. of Reads: 993 15290 No. of Writes: 56237 581071 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 205094 FORGET 0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE 0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR 0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR 0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR 0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR 0.03 84.33 us 40.00 us 164.00 us 6 FSTAT 0.10 64.35 us 34.00 us 177.00 us 23 STATFS 0.16 62.87 us 29.00 us 172.00 us 38 STAT 0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK 0.18 41.66 us 24.00 us 95.00 us 62 FLUSH 0.22 198.44 us 138.00 us 339.00 us 16 CREATE 0.32 101.65 us 65.00 us 184.00 us 46 OPEN 0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK 0.52 56.22 us 25.00 us 228.00 us 136 FINODELK 0.77 129.15 us 49.00 us 1138.00 us 87 READ 1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP 2.44 159.13 us 41.00 us 5836.00 us 224 WRITE 3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP 89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC Duration: 362318 seconds Data Read: 2132763575 bytes Data Written: 83056358475 bytes Interval 0 Stats: Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+ No. of Reads: 993 15290 No. of Writes: 56237 581071 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 205094 FORGET 0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE 0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR 0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR 0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR 0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR 0.03 84.33 us 40.00 us 164.00 us 6 FSTAT 0.10 64.35 us 34.00 us 177.00 us 23 STATFS 0.16 62.87 us 29.00 us 172.00 us 38 STAT 0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK 0.18 41.66 us 24.00 us 95.00 us 62 FLUSH 0.22 198.44 us 138.00 us 339.00 us 16 CREATE 0.32 101.65 us 65.00 us 184.00 us 46 OPEN 0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK 0.52 56.22 us 25.00 us 228.00 us 136 FINODELK 0.77 129.15 us 49.00 us 1138.00 us 87 READ 1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP 2.44 159.13 us 41.00 us 5836.00 us 224 WRITE 3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP 89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC Duration: 362318 seconds Data Read: 2132763575 bytes Data Written: 83056358475 bytes Brick: 10.10.0.101:/glusterfsvolumes/devops/devops-prometheus/brick ------------------------------------------------------------------- Cumulative Stats: Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+ No. of Reads: 893 14392 No. of Writes: 56237 581071 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 205094 FORGET 0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE 0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR 0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR 0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR 0.02 57.00 us 51.00 us 61.00 us 3 FSTAT 0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR 0.12 474.00 us 451.00 us 497.00 us 2 READDIRP 0.18 64.00 us 28.00 us 113.00 us 23 STATFS 0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK 0.31 41.33 us 18.00 us 87.00 us 60 FLUSH 0.33 66.12 us 28.00 us 246.00 us 40 STAT 0.57 102.39 us 62.00 us 236.00 us 44 OPEN 0.58 664.86 us 311.00 us 855.00 us 7 UNLINK 0.86 51.37 us 22.00 us 101.00 us 134 FINODELK 1.07 118.28 us 46.00 us 437.00 us 72 READ 3.83 137.39 us 41.00 us 2431.00 us 222 WRITE 4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP 5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE 7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP 73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC Duration: 362152 seconds Data Read: 2005452176 bytes Data Written: 83056358475 bytes Interval 0 Stats: Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+ No. of Reads: 893 14392 No. of Writes: 56237 581071 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 205094 FORGET 0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE 0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR 0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR 0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR 0.02 57.00 us 51.00 us 61.00 us 3 FSTAT 0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR 0.12 474.00 us 451.00 us 497.00 us 2 READDIRP 0.18 64.00 us 28.00 us 113.00 us 23 STATFS 0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK 0.31 41.33 us 18.00 us 87.00 us 60 FLUSH 0.33 66.12 us 28.00 us 246.00 us 40 STAT 0.57 102.39 us 62.00 us 236.00 us 44 OPEN 0.58 664.86 us 311.00 us 855.00 us 7 UNLINK 0.86 51.37 us 22.00 us 101.00 us 134 FINODELK 1.07 118.28 us 46.00 us 437.00 us 72 READ 3.83 137.39 us 41.00 us 2431.00 us 222 WRITE 4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP 5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE 7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP 73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC Duration: 362152 seconds Data Read: 2005452176 bytes Data Written: 83056358475 bytes Seems latency of FSYNC is very high. Sometimes cpu wait is also very high: top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70 Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, 36.5
wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem Any ideas? Thanks. |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
