Hello:
I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica 3 volume. My replica 3 volume has been having lots of problems for a while now.
First, after a week or two, I notice a lot of RAM usage (half of my physical ram, plus lots of swap) for the replica 3 gluster process. The amount of ram in use varies by machine, but they all have had the problem show up (one at a time). Often the OOM killer kills it off before I catch it.
This last time, I caught it. I migrated all the machines off the problem server, and rebooted it. That was yesterday (thurs at 9am). Shortly after coming back up, ALL 3 of my servers are showing very high load averages (cpu % as reported by top is fairly low, althoguh iowait will spike to 25%). The one process that is using the most CPU is the glusterfsd process for the replica 3 volume. It is consistant on all 3 machines, and it loads down the machine and IO enough to be causing issues with the VMs running on it (even IO to a seprate disk on the same machine) .
I watched network I/O and I can see occasional spikes to 50MB/s (gig link), but its not consistant, and often spends considerable time near zero or in kB/s range. the repica 3 has its own dedicated gig network.
As of last night I still had 4 unhealed items for the server I rebooted. This AM, I'm down to 1, but its still there, and its still causing considerable performance issues. On a server, if I stop that glusterfsd process, then the loadavg returns to normal quickly.
HELP...what do I do? Lately it seems that all my outages and performace issues have been glusterfsd taking down my servers that they are running on. Is gluster ready for production, or did I misunderstand? My reliability has been abismal on my vm cluster, and I still have 3 VMs that won't boot anymore due to disk corruption caused by my last gluster outage (which was the high RAM until it crashed on the server due to kernel OOM killer).
Here's some info on the affected gluster volume:
Volume Name: data-hdd
Type: Replicate
Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.172.1.11:/gluster/brick3/data-hdd
Brick2: 172.172.1.12:/gluster/brick3/data-hdd
Brick3: 172.172.1.13:/gluster/brick3/data-hdd
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
transport.address-family: inet
performance.readdir-ahead: on
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
server.allow-insecure: on
network.ping-timeout: 30
---------------
[root@ovirt3 ~]# gluster volume profile data-hdd info
Brick: 172.172.1.13:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 188488 6126 17234
No. of Writes: 80 331655 3899
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 17228 1371069 265013
No. of Writes: 6365 26617689 15776028
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 350572 636120 189538
No. of Writes: 4475250 1557330 604455
Block Size: 131072b+ 262144b+
No. of Reads: 6442414 0
No. of Writes: 5194964 13
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 63 FORGET
0.00 0.00 us 0.00 us 0.00 us 9356 RELEASE
0.00 0.00 us 0.00 us 0.00 us 113830 RELEASEDIR
0.00 765.33 us 102.00 us 2091.00 us 3 FGETXATTR
0.00 352.90 us 263.00 us 1397.00 us 31 TRUNCATE
0.00 1094.05 us 360.00 us 5643.00 us 21 MKDIR
0.00 647.01 us 260.00 us 4059.00 us 83 MKNOD
0.00 899.64 us 68.00 us 27531.00 us 105 SETXATTR
0.00 139.19 us 26.00 us 4619.00 us 740 ENTRYLK
0.00 2121.54 us 1250.00 us 9726.00 us 63 RENAME
0.00 254.93 us 18.00 us 169316.00 us 932 FLUSH
0.00 79.37 us 14.00 us 3120.00 us 3210 LK
0.00 9377.76 us 374.00 us 197032.00 us 63 CREATE
0.00 4777.26 us 91.00 us 757927.00 us 197 FTRUNCATE
0.00 1787.09 us 117.00 us 505178.00 us 767 XATTROP
0.00 180.43 us 13.00 us 796920.00 us 33471 GETXATTR
0.01 398.80 us 72.00 us 2581011.00 us 18792 REMOVEXATTR
0.01 378.54 us 21.00 us 264034.00 us 32293 READDIR
0.01 152.76 us 1.00 us 952682.00 us 113830 OPENDIR
0.02 1460.88 us 64.00 us 4693687.00 us 18918 SETATTR
0.02 524.47 us 30.00 us 359603.00 us 56913 READDIRP
0.02 3564.23 us 59.00 us 4818020.00 us 9307 OPEN
0.04 125.61 us 21.00 us 355789.00 us 400177 STAT
0.11 132.13 us 24.00 us 460615.00 us 1143725 STATFS
0.26 111.57 us 18.00 us 3533816.00 us 3153747 FSTAT
0.81 383.95 us 22.00 us 3896810.00 us 2827323 LOOKUP
5.51 74.04 us 13.00 us 1883274.00 us 100195900 FINODELK
5.59 258.57 us 13.00 us 11221929.00 us 29091404 INODELK
6.50 922.80 us 25.00 us 7548801.00 us 9483802 READ
10.20 358.55 us 56.00 us 14993728.00 us 38295759 FXATTROP
12.42 306.31 us 47.00 us 30915095.00 us 54567728 WRITE
58.45 2495.18 us 43.00 us 21663381.00 us 31520936 FSYNC
Duration: 1946380 seconds
Data Read: 894661069137 bytes
Data Written: 1142499928350 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 3 0
No. of Writes: 15 3702 126
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 9 58
No. of Writes: 198 62716 20361
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 13 1 1
No. of Writes: 13640 9197 3948
Block Size: 131072b+ 262144b+
No. of Reads: 1826585 0
No. of Writes: 7387 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 15 FORGET
0.00 0.00 us 0.00 us 0.00 us 3648 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2421 RELEASEDIR
0.00 293.86 us 263.00 us 327.00 us 7 TRUNCATE
0.00 72.55 us 30.00 us 152.00 us 90 ENTRYLK
0.00 546.00 us 374.00 us 788.00 us 15 CREATE
0.00 475.86 us 178.00 us 2572.00 us 51 XATTROP
0.00 75.50 us 24.00 us 301.00 us 424 FLUSH
0.00 2889.13 us 1557.00 us 9171.00 us 15 RENAME
0.00 195.00 us 109.00 us 866.00 us 313 REMOVEXATTR
0.00 225.66 us 117.00 us 858.00 us 341 SETATTR
0.00 80.26 us 25.00 us 534.00 us 1041 LK
0.00 284.35 us 24.00 us 3965.00 us 408 READDIR
0.01 134.66 us 2.00 us 17518.00 us 2421 OPENDIR
0.01 202.33 us 34.00 us 15815.00 us 2413 FSTAT
0.01 624.48 us 23.00 us 279600.00 us 820 GETXATTR
0.04 107.65 us 29.00 us 1004.00 us 14106 STATFS
0.45 73.59 us 15.00 us 60009.00 us 250150 FINODELK
0.45 5080.05 us 62.00 us 1940405.00 us 3631 OPEN
0.97 869.22 us 35.00 us 1942304.00 us 45625 LOOKUP
1.50 378.77 us 58.00 us 1833454.00 us 163094 FXATTROP
1.77 599.60 us 55.00 us 1933975.00 us 121295 WRITE
15.58 5494.71 us 50.00 us 1958942.00 us 116418 FSYNC
29.74 668.39 us 52.00 us 1945979.00 us 1826670 READ
49.46 243.86 us 13.00 us 1945794.00 us 8325886 INODELK
Duration: 42487 seconds
Data Read: 239414982144 bytes
Data Written: 2475540892 bytes
Brick: 172.172.1.11:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 7915 524 274
No. of Writes: 31 9934 169
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 351 36927 5673
No. of Writes: 278 621338 316601
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 6003 2366 3082
No. of Writes: 110118 45664 16295
Block Size: 131072b+ 262144b+
No. of Reads: 3710107 0
No. of Writes: 556659 6
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 29 FORGET
0.00 0.00 us 0.00 us 0.00 us 6295 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5574 RELEASEDIR
0.00 164.00 us 164.00 us 164.00 us 1 SETXATTR
0.00 92.33 us 66.00 us 130.00 us 3 FGETXATTR
0.00 308.93 us 215.00 us 408.00 us 15 TRUNCATE
0.00 70.94 us 22.00 us 271.00 us 186 ENTRYLK
0.00 70.06 us 21.00 us 265.00 us 669 FLUSH
0.00 1974.90 us 1490.00 us 6096.00 us 31 RENAME
0.00 76.58 us 24.00 us 3869.00 us 2582 LK
0.00 3252.80 us 175.00 us 244398.00 us 106 XATTROP
0.00 377.53 us 22.00 us 33026.00 us 1292 READDIR
0.00 940.44 us 85.00 us 563301.00 us 755 REMOVEXATTR
0.00 134434.50 us 119.00 us 563308.00 us 6 FTRUNCATE
0.00 677.32 us 19.00 us 641166.00 us 2303 GETXATTR
0.00 296.94 us 2.00 us 605194.00 us 5575 OPENDIR
0.00 3015.49 us 112.00 us 2198121.00 us 793 SETATTR
0.00 144086.32 us 461.00 us 1973731.00 us 31 CREATE
0.00 122.50 us 21.00 us 470389.00 us 37921 STATFS
0.00 322.28 us 25.00 us 2826178.00 us 17238 STAT
0.01 151.48 us 22.00 us 460622.00 us 77757 FSTAT
0.01 4641.92 us 35.00 us 7009636.00 us 3665 READDIRP
0.01 3514.85 us 61.00 us 9109607.00 us 6278 OPEN
0.04 800.50 us 23.00 us 8211795.00 us 109977 LOOKUP
0.60 1279.23 us 59.00 us 237262387.00 us 1023399 FXATTROP
1.50 872.91 us 35.00 us 6929426.00 us 3773222 READ
1.61 2096.57 us 50.00 us 237261777.00 us 1677093 WRITE
6.17 16780.27 us 51.00 us 22990372.00 us 804858 FSYNC
11.35 1890.83 us 14.00 us 1066342526.00 us 13144090 INODELK
78.70 73624.16 us 15.00 us 759064386.00 us 2340006 FINODELK
Duration: 84992 seconds
Data Read: 486964889723 bytes
Data Written: 84167404788 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 3861 337 173
No. of Writes: 15 3702 126
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 159 5308 857
No. of Writes: 198 62716 20361
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 1008 1004 621
No. of Writes: 13640 9197 3948
Block Size: 131072b+ 262144b+
No. of Reads: 2341344 0
No. of Writes: 7387 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 15 FORGET
0.00 0.00 us 0.00 us 0.00 us 3647 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2541 RELEASEDIR
0.00 267.86 us 215.00 us 322.00 us 7 TRUNCATE
0.00 66.18 us 22.00 us 144.00 us 90 ENTRYLK
0.00 450.02 us 197.00 us 2730.00 us 51 XATTROP
0.00 66.45 us 21.00 us 265.00 us 423 FLUSH
0.00 2049.80 us 1504.00 us 6096.00 us 15 RENAME
0.00 193.90 us 85.00 us 1329.00 us 313 REMOVEXATTR
0.00 74.23 us 24.00 us 3869.00 us 1041 LK
0.00 258.23 us 112.00 us 12048.00 us 341 SETATTR
0.00 384.60 us 22.00 us 2284.00 us 647 READDIR
0.00 415.01 us 19.00 us 122660.00 us 1306 GETXATTR
0.00 495.05 us 2.00 us 605194.00 us 2542 OPENDIR
0.00 98.96 us 30.00 us 2284.00 us 14106 STATFS
0.00 194.92 us 25.00 us 628797.00 us 8675 STAT
0.00 849.86 us 35.00 us 369113.00 us 2005 READDIRP
0.00 585.27 us 61.00 us 279889.00 us 3630 OPEN
0.00 141763.87 us 461.00 us 1973731.00 us 15 CREATE
0.00 155.27 us 23.00 us 169047.00 us 21344 FSTAT
0.02 442.07 us 26.00 us 544143.00 us 45744 LOOKUP
0.11 641.69 us 59.00 us 2227486.00 us 163094 FXATTROP
0.16 1215.83 us 58.00 us 2628020.00 us 121295 WRITE
1.25 9910.79 us 51.00 us 5116269.00 us 116418 FSYNC
1.30 511.11 us 35.00 us 2493646.00 us 2354672 READ
13.86 1535.67 us 14.00 us 688678811.00 us 8321169 INODELK
83.28 309339.72 us 16.00 us 688564320.00 us 248271 FINODELK
Duration: 42487 seconds
Data Read: 307016229315 bytes
Data Written: 2475540892 bytes
Brick: 172.172.1.12:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 3706 247 142
No. of Writes: 15 3600 124
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 94 5848 956
No. of Writes: 194 60615 19772
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 1287 12947 1248
No. of Writes: 13126 8886 3810
Block Size: 131072b+ 262144b+
No. of Reads: 959349 0
No. of Writes: 3178716 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13 FORGET
0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR
0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR
0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK
0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE
0.00 160.27 us 17.00 us 63213.00 us 1041 LK
0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE
0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP
0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR
0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME
0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE
0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP
0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR
0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS
0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR
0.18 3713522.62 us 14.00 us 372969001.00 us 427 FLUSH
0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN
0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR
0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR
0.74 235518.84 us 19.00 us 384250227.00 us 28134 FSTAT
1.10 1203934.76 us 16.00 us 407840076.00 us 8208 STAT
2.41 22009.35 us 29.00 us 421083048.00 us 985824 READ
2.53 93906.00 us 13.00 us 118521408.00 us 241629 FINODELK
3.78 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK
3.84 218447.60 us 83.00 us 498122410.00 us 158007 FXATTROP
4.02 817162.39 us 19.00 us 409893532.00 us 44246 LOOKUP
25.08 68523.67 us 49.00 us 516930780.00 us 3288855 WRITE
54.73 4364231.40 us 51.00 us 688577405.00 us 112686 FSYNC
Duration: 41424 seconds
Data Read: 126333064115 bytes
Data Written: 418096774044 bytes
Interval 0 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 3706 247 142
No. of Writes: 15 3600 124
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 94 5848 956
No. of Writes: 194 60615 19772
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 1287 12947 1248
No. of Writes: 13126 8886 3810
Block Size: 131072b+ 262144b+
No. of Reads: 959349 0
No. of Writes: 3178716 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13 FORGET
0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR
0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR
0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK
0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE
0.00 160.27 us 17.00 us 63213.00 us 1041 LK
0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE
0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP
0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR
0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME
0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE
0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP
0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR
0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS
0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR
0.18 3720010.69 us 14.00 us 372969001.00 us 427 FLUSH
0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN
0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR
0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR
0.74 235509.69 us 19.00 us 384250227.00 us 28134 FSTAT
1.10 1204610.00 us 16.00 us 407840076.00 us 8208 STAT
2.41 22009.63 us 29.00 us 421083048.00 us 985824 READ
2.52 93906.03 us 13.00 us 118521408.00 us 241629 FINODELK
3.77 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK
3.84 218446.23 us 83.00 us 498122410.00 us 158007 FXATTROP
4.02 817497.87 us 19.00 us 409893532.00 us 44246 LOOKUP
25.07 68548.46 us 49.00 us 516930780.00 us 3288855 WRITE
54.76 4369255.44 us 51.00 us 688577405.00 us 112686 FSYNC
Duration: 41424 seconds
Data Read: 126333064115 bytes
Data Written: 418096774044 bytes
-----------
[this command took a few MINUTES to return]
[root@ovirt1 ~]# gluster volume heal data-hdd info
Brick 172.172.1.11:/gluster/brick3/data-hdd
/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
Status: Connected
Number of entries: 1
Brick 172.172.1.12:/gluster/brick3/data-hdd
Status: Connected
Number of entries: 0
Brick 172.172.1.13:/gluster/brick3/data-hdd
/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
Status: Connected
Number of entries: 1
-----------
09:43:02 up 16:09, 1 user, load average: 30.75, 39.10, 34.75
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
