Hello,
It’s been a while since I last ran GlusterFS so I thought I might give it another try here at home in my lab. I am using the 3.7 branch on 2 systems with a 3rd being an arbiter node. Much like the last time I tried GlusterFS, I keep running into issues with the glusterfsd process eating up so many resources that the systems sometimes become all but unusable. A quick Google search tells me I am not the only one to run into this issue but I have yet to find a cure. The last time I ran GlusterFS, it was to host web sites and I just chalked the problem up to a large number of small files. This time, I am using it to host VM’s and there are only 7 of them and while they are running, they are not doing anything else.
When I run ‘gluster volume heal gv0 info’, it almost always shows a few files and sometimes they say “Possibly undergoing heal”. Next time, it will be different files. So I am wondering if the problems are heal related and I’m hoping there is a way I can tune for this.
The systems are dual-quad core processors with 72 GB of memory and Gigabit Nics. Network speed doesn't seem to be the issue as when I check it with iperf, I can consistently get over 850 Mbits/sec. Processor and/or drive speed seems to be where the issues are coming from but not sure which comes first or what to do as I would think I should have ample resources.
When I don’t have Gluster running, I can run “hdparm –Tt /dev/sdb” and consistently get over 110MB/sec. Granted I am using 7200 rpm SATA3 drives but for 7 VMs that are all very quiet, I find it hard to believe they are not fast enough. When I run GlusterFS with the bricks on that drive (and nothing else running on those drives), I have seen that speed slow down to 20MB/sec. As I type this, one of the systems has 5 VMs up on it and the other has 2 and both experience the same thing. A quick look at the output of “top” will show load average of 15-20 and sometimes even higher. Again, the VMs are running but I have checked and they are not doing much of anything other than running.
When I originally set this up, I didn't have any extra config settings in place but as you can see from the output, I do now, although none have helped thus far. I am connecting to GlusterFS utilizing the built-in NFS server (v3 but that doesn’t seem to matter either) with options noatime, nodiratime, rsize=65536, and wsize=65536 but they didn't seem to make much of a difference either.
So to sum up, the problem seems to be all the heals going on. What can I do to get this back under control? Luckily it’s in a lab environment so I can experiment a bit. Here is some additional info that may be of assistance and thank you in advance:
gluster volume info gv0:
Volume Name: gv0
Type: Replicate
Volume ID: 14e7bb9c-aa5e-4386-8dd2-83a88d93dc54
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: server1:/export/brick1
Brick2: server2:/export/brick1
Brick3: kvm:/export/brick1
Options Reconfigured:
nfs.acl: off
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
---
I gathered this with load averages of approximately 12 so it has been much higher
gluster volume status gv0 info:
Status of volume: gv0
------------------------------------------------------------------------------
Brick : Brick server1:/export/brick1
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 4409
File System : ext3
Device : /dev/sdb1
Mount Options : rw
Inode Size : 128
Disk Space Free : 1.7TB
Total Disk Space : 1.8TB
Inode Count : 244203520
Free Inodes : 244203413
------------------------------------------------------------------------------
Brick : Brick server2:/export/brick1
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 4535
File System : ext3
Device : /dev/sdb1
Mount Options : rw
Inode Size : 128
Disk Space Free : 1.7TB
Total Disk Space : 1.8TB
Inode Count : 244203520
Free Inodes : 244203405
---
gluster volume provile gv0 as follows:
Brick: server1:/export/brick1
--------------------------------
Cumulative Stats:
Block Size: 16b+ 512b+ 1024b+
No. of Reads: 3 1319 39679
No. of Writes: 3 10804 917
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 673 33872 7199
No. of Writes: 3642 219192 133087
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 8808 20790 2275
No. of Writes: 52815 180145 4910
Block Size: 131072b+
No. of Reads: 952554
No. of Writes: 5558597
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 5 FORGET
0.00 0.00 us 0.00 us 0.00 us 755 RELEASE
0.00 0.00 us 0.00 us 0.00 us 214 RELEASEDIR
0.00 4.00 us 4.00 us 4.00 us 2 OPENDIR
0.00 168.67 us 31.00 us 406.00 us 3 READDIR
0.00 535.40 us 84.00 us 1612.00 us 5 OPEN
0.00 975.33 us 916.00 us 1045.00 us 3 FTRUNCATE
0.00 305.36 us 15.00 us 14784.00 us 59 FLUSH
0.00 14988.50 us 9326.00 us 20651.00 us 2 XATTROP
0.00 16257.40 us 61.00 us 58866.00 us 10 LOOKUP
0.01 41136.33 us 37.00 us 219392.00 us 6 GETXATTR
0.01 146916.50 us 51093.00 us 242740.00 us 2 SETATTR
0.01 637.58 us 24.00 us 124014.00 us 651 READ
0.02 241.17 us 6.00 us 75127.00 us 2553 ACCESS
0.02 220.31 us 10.00 us 77807.00 us 3235 STAT
0.10 223016.80 us 72200.00 us 681213.00 us 15 FSYNC
0.79 1466.11 us 29.00 us 236061.00 us 18414 FXATTROP
1.20 1392.82 us 13.00 us 842550.00 us 29201 FINODELK
12.66 3079.72 us 19.00 us 1161311.00 us 139747 INODELK
85.17 34610.43 us 56.00 us 2401944.00 us 83630 WRITE
Duration: 81829 seconds
Data Read: 126295937593 bytes
Data Written: 740051667513 bytes
Interval 7 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 234 0
No. of Writes: 171 7 16
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 1162 472 278
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1 0 0
No. of Writes: 1593 30 36991
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASE
0.00 4.00 us 4.00 us 4.00 us 1 OPENDIR
0.00 406.00 us 406.00 us 406.00 us 1 READDIR
0.00 979.50 us 347.00 us 1612.00 us 2 OPEN
0.00 1005.00 us 965.00 us 1045.00 us 2 FTRUNCATE
0.00 9326.00 us 9326.00 us 9326.00 us 1 XATTROP
0.00 656.32 us 27.00 us 14784.00 us 25 FLUSH
0.00 51093.00 us 51093.00 us 51093.00 us 1 SETATTR
0.01 16878.60 us 164.00 us 47828.00 us 5 LOOKUP
0.01 363.44 us 24.00 us 47319.00 us 235 READ
0.01 112.50 us 10.00 us 18678.00 us 1162 STAT
0.02 78723.67 us 38.00 us 219392.00 us 3 GETXATTR
0.03 387.51 us 7.00 us 75127.00 us 918 ACCESS
0.16 426656.75 us 217375.00 us 681213.00 us 4 FSYNC
0.51 1440.22 us 37.00 us 175074.00 us 3820 FXATTROP
1.00 1311.97 us 16.00 us 842550.00 us 8218 FINODELK
13.44 1951.10 us 23.00 us 1161311.00 us 73993 INODELK
84.79 22359.52 us 56.00 us 2401944.00 us 40723 WRITE
Duration: 394 seconds
Data Read: 392192 bytes
Data Written: 4934906368 bytes
Brick: kvm:/export/brick1
--------------------------
Cumulative Stats:
Block Size: 16b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 5 10288 849
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 3399 206329 128093
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 50427 175784 4811
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 6260106
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 250 RELEASE
0.00 0.00 us 0.00 us 0.00 us 144 RELEASEDIR
Duration: 72903 seconds
Data Read: 0 bytes
Data Written: 831653482591 bytes
Interval 7 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 171 7 16
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 1162 472 278
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 0 0
No. of Writes: 1593 30 36068
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASE
Duration: 394 seconds
Data Read: 0 bytes
Data Written: 4813926912 bytes
Brick: server2:/export/brick1
--------------------------------
Cumulative Stats:
Block Size: 16b+ 512b+ 1024b+
No. of Reads: 2 1554 187
No. of Writes: 3 10813 920
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 205139 61200
No. of Writes: 3642 219205 133087
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 41437 311428 9278
No. of Writes: 52809 180141 4853
Block Size: 131072b+
No. of Reads: 5558571
No. of Writes: 292354
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 5 FORGET
0.00 0.00 us 0.00 us 0.00 us 795 RELEASE
0.00 0.00 us 0.00 us 0.00 us 196 RELEASEDIR
0.00 94.49 us 10.00 us 2660.00 us 59 FLUSH
0.00 6859.40 us 78.00 us 16605.00 us 5 OPEN
0.00 22617.50 us 11067.00 us 34168.00 us 2 XATTROP
0.01 20768.00 us 112.00 us 81026.00 us 8 LOOKUP
0.04 37279.69 us 19436.00 us 86934.00 us 13 FSYNC
0.85 385.97 us 10.00 us 212031.00 us 28913 FINODELK
1.33 951.03 us 34.00 us 221747.00 us 18400 FXATTROP
1.62 303.60 us 22.00 us 231359.00 us 70024 READ
9.25 868.44 us 16.00 us 407424.00 us 139714 INODELK
86.89 82862.96 us 59.00 us 638518.00 us 13759 WRITE
Duration: 82915 seconds
Data Read: 744922109478 bytes
Data Written: 49788871225 bytes
Interval 7 Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 30 0 0
No. of Writes: 171 7 16
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 55 4 25
No. of Writes: 1162 472 278
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 30 0 37001
No. of Writes: 1593 30 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASE
0.00 45.52 us 10.00 us 91.00 us 25 FLUSH
0.00 11067.00 us 11067.00 us 11067.00 us 1 XATTROP
0.00 9621.00 us 2637.00 us 16605.00 us 2 OPEN
0.03 44736.33 us 19436.00 us 86934.00 us 3 FSYNC
0.03 33648.00 us 129.00 us 81026.00 us 4 LOOKUP
1.21 1351.63 us 35.00 us 123188.00 us 3820 FXATTROP
1.21 634.89 us 10.00 us 212031.00 us 8156 FINODELK
2.96 339.80 us 22.00 us 231359.00 us 37145 READ
12.90 743.44 us 17.00 us 407424.00 us 74014 INODELK
81.64 93359.19 us 64.00 us 442704.00 us 3729 WRITE
Duration: 394 seconds
Data Read: 4851727360 bytes
Data Written: 86422016 bytes
Thank you.
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
