Hello Community,
I would be very grateful if you share your thoughts about my problem.
I'm quite new to gluster , so keep that in mind.
Setup:
I have 3 nodes for an ovirt setup (2 hosts + 1 gluster arbiter). The diagram of my setup is: ovirt
I'm having bad write & read performance from the VM, despite having a SSD (not an enterprise-grade one , but still a SSD) for LVM writeback cache. Writing directly to the bricks is far faster than my network can do (1 Gbit/s network). I have checked the bandwidth and it seems that maximum possible for me is 123MB/s, yet observed speeds via glusterfs fuse client (testing from one of the hosts) is no more than 56MB/s and from VMs is around 20-30 MB/s.
Here is my volume profile info (I'm not sure what to look for):
[root@ovirt2 tuned]# gluster volume profile data info
Brick: ovirt2.localdomain:/gluster_bricks/data/data
---------------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 5854 198 159
No. of Writes: 2 6025 1430
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 302 9950 6485
No. of Writes: 611 23513 6540
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 6952 1774 1699
No. of Writes: 6439 5870 5171
Block Size: 131072b+
No. of Reads: 48690
No. of Writes: 127023
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 14 FORGET
0.00 0.00 us 0.00 us 0.00 us 534 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2180 RELEASEDIR
0.00 44.00 us 44.00 us 44.00 us 1 GETXATTR
0.00 28.75 us 23.00 us 33.00 us 4 INODELK
0.00 85.00 us 85.00 us 85.00 us 2 SETATTR
0.00 52.50 us 31.00 us 113.00 us 8 OPENDIR
0.00 33.83 us 23.00 us 60.00 us 29 ENTRYLK
0.00 40.32 us 24.00 us 157.00 us 53 STAT
0.00 59.32 us 43.00 us 121.00 us 38 FSTAT
0.00 188.08 us 28.00 us 458.00 us 13 READDIRP
0.00 323.46 us 286.00 us 398.00 us 13 MKNOD
0.00 42.84 us 26.00 us 142.00 us 109 STATFS
0.01 147.36 us 71.00 us 365.00 us 200 LOOKUP
0.17 914.71 us 186.00 us 3887.00 us 533 READ
3.57 322.84 us 41.00 us 1552044.00 us 32308 FXATTROP
21.39 1090.04 us 15.00 us 1228946.00 us 57261 FINODELK
27.42 2527.96 us 98.00 us 1552471.00 us 31651 WRITE
47.43 12047.80 us 203.00 us 1891369.00 us 11489 FSYNC
Duration: 59012 seconds
Data Read: 6853540304 bytes
Data Written: 17667709568 bytes
Interval 0 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 5854 198 159
No. of Writes: 2 6025 1430
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 302 9950 6485
No. of Writes: 611 23513 6540
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 6952 1774 1699
No. of Writes: 6439 5870 5171
Block Size: 131072b+
No. of Reads: 48690
No. of Writes: 127023
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 14 FORGET
0.00 0.00 us 0.00 us 0.00 us 534 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2180 RELEASEDIR
0.00 44.00 us 44.00 us 44.00 us 1 GETXATTR
0.00 28.75 us 23.00 us 33.00 us 4 INODELK
0.00 85.00 us 85.00 us 85.00 us 2 SETATTR
0.00 52.50 us 31.00 us 113.00 us 8 OPENDIR
0.00 33.83 us 23.00 us 60.00 us 29 ENTRYLK
0.00 40.32 us 24.00 us 157.00 us 53 STAT
0.00 59.32 us 43.00 us 121.00 us 38 FSTAT
0.00 188.08 us 28.00 us 458.00 us 13 READDIRP
0.00 323.46 us 286.00 us 398.00 us 13 MKNOD
0.00 42.84 us 26.00 us 142.00 us 109 STATFS
0.01 147.36 us 71.00 us 365.00 us 200 LOOKUP
0.17 914.71 us 186.00 us 3887.00 us 533 READ
3.57 322.84 us 41.00 us 1552044.00 us 32308 FXATTROP
21.39 1090.04 us 15.00 us 1228946.00 us 57261 FINODELK
27.42 2527.96 us 98.00 us 1552471.00 us 31651 WRITE
47.43 12047.80 us 203.00 us 1891369.00 us 11489 FSYNC
Duration: 59012 seconds
Data Read: 6853540304 bytes
Data Written: 17667709568 bytes
Brick: ovirt3.localdomain:/gluster_bricks/data/data
---------------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 257997
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 53 FORGET
0.00 0.00 us 0.00 us 0.00 us 3180 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2672 RELEASEDIR
0.00 14.33 us 13.00 us 16.00 us 3 FLUSH
0.00 43.00 us 43.00 us 43.00 us 1 TRUNCATE
0.00 229.00 us 229.00 us 229.00 us 1 CREATE
0.00 80.00 us 48.00 us 114.00 us 3 XATTROP
0.00 66.83 us 57.00 us 79.00 us 6 SETATTR
0.00 29.76 us 1.00 us 72.00 us 34 OPENDIR
0.00 96.12 us 12.00 us 240.00 us 17 GETXATTR
0.01 375.00 us 216.00 us 790.00 us 6 READDIR
0.01 30.90 us 11.00 us 186.00 us 73 INODELK
0.01 178.89 us 36.00 us 541.00 us 19 OPEN
0.02 113.86 us 57.00 us 313.00 us 78 UNLINK
0.05 212.36 us 146.00 us 488.00 us 90 MKNOD
0.14 35.88 us 10.00 us 399.00 us 1636 ENTRYLK
0.42 80.78 us 21.00 us 495.00 us 2122 LOOKUP
7.92 44.74 us 12.00 us 10903.00 us 71878 WRITE
11.52 62.55 us 25.00 us 27390.00 us 74768 FXATTROP
12.00 27.54 us 9.00 us 7191.00 us 176968 FINODELK
67.90 2384.96 us 53.00 us 82033.00 us 11562 FSYNC
Duration: 75025 seconds
Data Read: 0 bytes
Data Written: 257997 bytes
Interval 2 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 201556
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 19 FORGET
0.00 0.00 us 0.00 us 0.00 us 2046 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2606 RELEASEDIR
0.00 38.50 us 35.00 us 42.00 us 2 GETXATTR
0.00 33.17 us 15.00 us 62.00 us 6 INODELK
0.00 71.00 us 63.00 us 79.00 us 3 SETATTR
0.00 41.75 us 24.00 us 72.00 us 12 OPENDIR
0.00 26.28 us 12.00 us 48.00 us 29 ENTRYLK
0.01 191.15 us 154.00 us 240.00 us 13 MKNOD
0.10 112.97 us 40.00 us 170.00 us 299 LOOKUP
4.22 45.65 us 14.00 us 1288.00 us 31732 WRITE
5.99 35.51 us 9.00 us 7124.00 us 57863 FINODELK
9.57 101.29 us 33.00 us 27390.00 us 32435 FXATTROP
80.11 2390.06 us 61.00 us 82033.00 us 11507 FSYNC
Duration: 74302 seconds
Data Read: 0 bytes
Data Written: 201556 bytes
Brick: ovirt1.localdomain:/gluster_bricks/data/data
---------------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 13408 183 2200
No. of Writes: 11 12677 1556
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 60 3393 1278
No. of Writes: 666 59118 22688
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 10658 574 954
No. of Writes: 32082 29900 61801
Block Size: 131072b+
No. of Reads: 55974
No. of Writes: 576153
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 58 FORGET
0.00 0.00 us 0.00 us 0.00 us 3357 RELEASE
0.00 0.00 us 0.00 us 0.00 us 4823 RELEASEDIR
0.00 26.00 us 24.00 us 28.00 us 3 FLUSH
0.00 64.00 us 52.00 us 81.00 us 3 FSTAT
0.00 202.00 us 202.00 us 202.00 us 1 TRUNCATE
0.00 271.00 us 271.00 us 271.00 us 1 CREATE
0.00 135.33 us 72.00 us 180.00 us 3 XATTROP
0.00 103.50 us 84.00 us 155.00 us 6 SETATTR
0.00 69.77 us 17.00 us 174.00 us 13 GETXATTR
0.00 438.67 us 399.00 us 479.00 us 3 READDIRP
0.00 45.12 us 3.00 us 94.00 us 34 OPENDIR
0.00 91.63 us 53.00 us 184.00 us 19 OPEN
0.00 331.57 us 262.00 us 381.00 us 7 READDIR
0.00 46.28 us 24.00 us 176.00 us 337 STATFS
0.00 223.68 us 101.00 us 630.00 us 78 UNLINK
0.00 318.33 us 176.00 us 456.00 us 90 MKNOD
0.00 237.58 us 18.00 us 148201.00 us 2122 LOOKUP
0.01 1323.15 us 104.00 us 73159.00 us 704 READ
0.02 1890.07 us 15.00 us 89692.00 us 1634 ENTRYLK
0.07 137914.67 us 22.00 us 1523239.00 us 73 INODELK
0.22 395.07 us 30.00 us 676921.00 us 74768 FXATTROP
1.16 13685.56 us 1739.00 us 650863.00 us 11562 FSYNC
3.08 5851.12 us 94.00 us 1033754.00 us 71879 WRITE
95.43 76480.58 us 16.00 us 5368628.00 us 170274 FINODELK
Duration: 134846 seconds
Data Read: 7665929388 bytes
Data Written: 84724663200 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 7432 130 1738
No. of Writes: 6 6816 1430
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 48 3099 1040
No. of Writes: 611 24732 6649
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 9382 442 832
No. of Writes: 6602 6234 5531
Block Size: 131072b+
No. of Reads: 20719
No. of Writes: 142946
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 19 FORGET
0.00 0.00 us 0.00 us 0.00 us 2046 RELEASE
0.00 0.00 us 0.00 us 0.00 us 2606 RELEASEDIR
0.00 63.00 us 47.00 us 79.00 us 2 GETXATTR
0.00 64.00 us 52.00 us 81.00 us 3 FSTAT
0.00 33.83 us 27.00 us 41.00 us 6 INODELK
0.00 116.33 us 92.00 us 155.00 us 3 SETATTR
0.00 479.00 us 479.00 us 479.00 us 1 READDIRP
0.00 48.75 us 32.00 us 83.00 us 12 OPENDIR
0.00 283.31 us 176.00 us 383.00 us 13 MKNOD
0.00 46.40 us 26.00 us 142.00 us 164 STATFS
0.00 1303.45 us 20.00 us 36916.00 us 29 ENTRYLK
0.00 1054.11 us 63.00 us 148201.00 us 299 LOOKUP
0.01 1302.85 us 104.00 us 73159.00 us 618 READ
0.20 812.06 us 40.00 us 676921.00 us 32435 FXATTROP
1.19 13697.54 us 1739.00 us 650863.00 us 11507 FSYNC
1.94 8117.26 us 98.00 us 1033754.00 us 31733 WRITE
96.67 229052.02 us 17.00 us 5368628.00 us 56103 FINODELK
Duration: 74302 seconds
Data Read: 2999932124 bytes
Data Written: 19810606944 bytes
Any help will be appreciated.
Best Regards,
Strahil Nikolov
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
