Hello Community, recently I have built a new lab based on oVirt and CentOS 7. During deployment I had some hicups, but now the engine is up and running - but gluster is causing me trouble. Symptoms: Slow VM install from DVD, poor write performance. The latter has been tested via: dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_data bs=1M count=1000 status=progress The reported speed is 60MB/s which is way too low for my setup. My lab design: https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing Gluster version is 3.12.15 So far I have done: 1. Added 'server.allow-insecure on' (with 'option rpc-auth-allow-insecure on' in glusterd.vol) Volume info after that change: Volume Name: data Type: Replicate Volume ID: 9b06a1e9-8102-4cd7-bc56-84960a1efaa2 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: ovirt1.localdomain:/gluster_bricks/data/data Brick2: ovirt2.localdomain:/gluster_bricks/data/data Brick3: ovirt3.localdomain:/gluster_bricks/data/data (arbiter) Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off storage.owner-uid: 36 storage.owner-gid: 36 network.ping-timeout: 30 performance.strict-o-direct: on cluster.granular-entry-heal: enable server.allow-insecure: on Seems no positive or negative effect so far. 2. Tested with tmpfs on all bricks -> ovirt1 mounted gluster volume -> max 60MB/s (bs=1M without 'oflag=direct') [root@ovirt1 data]# dd if=/dev/zero of=large_io bs=1M count=4000 status=progress 4177526784 bytes (4.2 GB) copied, 70.843409 s, 59.0 MB/s 4000+0 records in 4000+0 records out 4194304000 bytes (4.2 GB) copied, 71.1407 s, 59.0 MB/s [root@ovirt1 data]# rm -f large_io [root@ovirt1 data]# gluster volume profile data info Brick: ovirt1.localdomain:/gluster_bricks/data/data --------------------------------------------------- Cumulative Stats: Block Size: 131072b+ No. of Reads: 8 No. of Writes: 44968 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 3 FORGET 0.00 0.00 us 0.00 us 0.00 us 35 RELEASE 0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR 0.00 78.00 us 78.00 us 78.00 us 1 FSTAT 0.00 35.67 us 26.00 us 73.00 us 6 FLUSH 0.00 324.00 us 324.00 us 324.00 us 1 XATTROP 0.00 45.80 us 38.00 us 54.00 us 10 STAT 0.00 227.67 us 216.00 us 242.00 us 3 CREATE 0.00 113.38 us 68.00 us 381.00 us 8 READ 0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR 0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR 0.00 59.97 us 45.00 us 113.00 us 32 OPEN 0.00 24.41 us 13.00 us 89.00 us 161 INODELK 0.00 43.43 us 28.00 us 214.00 us 93 STATFS 0.00 246.35 us 11.00 us 1155.00 us 20 READDIR 0.00 283.00 us 233.00 us 353.00 us 18 READDIRP 0.00 153.23 us 122.00 us 259.00 us 87 MKNOD 0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP 0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP 0.77 124.24 us 87.00 us 604.00 us 44968 WRITE 0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK 1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK 96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK Duration: 380 seconds Data Read: 1048576 bytes Data Written: 5894045696 bytes Interval 0 Stats: Block Size: 131072b+ No. of Reads: 8 No. of Writes: 44968 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 3 FORGET 0.00 0.00 us 0.00 us 0.00 us 35 RELEASE 0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR 0.00 78.00 us 78.00 us 78.00 us 1 FSTAT 0.00 35.67 us 26.00 us 73.00 us 6 FLUSH 0.00 324.00 us 324.00 us 324.00 us 1 XATTROP 0.00 45.80 us 38.00 us 54.00 us 10 STAT 0.00 227.67 us 216.00 us 242.00 us 3 CREATE 0.00 113.38 us 68.00 us 381.00 us 8 READ 0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR 0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR 0.00 59.97 us 45.00 us 113.00 us 32 OPEN 0.00 24.41 us 13.00 us 89.00 us 161 INODELK 0.00 43.43 us 28.00 us 214.00 us 93 STATFS 0.00 246.35 us 11.00 us 1155.00 us 20 READDIR 0.00 283.00 us 233.00 us 353.00 us 18 READDIRP 0.00 153.23 us 122.00 us 259.00 us 87 MKNOD 0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP 0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP 0.77 124.24 us 87.00 us 604.00 us 44968 WRITE 0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK 1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK 96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK Duration: 380 seconds Data Read: 1048576 bytes Data Written: 5894045696 bytes Brick: ovirt3.localdomain:/gluster_bricks/data/data --------------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 39328 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2 FORGET 0.00 0.00 us 0.00 us 0.00 us 12 RELEASE 0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR 0.00 101.00 us 101.00 us 101.00 us 1 FSTAT 0.00 51.50 us 20.00 us 81.00 us 4 FLUSH 0.01 219.50 us 188.00 us 251.00 us 2 CREATE 0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR 0.01 62.30 us 38.00 us 119.00 us 10 OPEN 0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR 0.01 24.60 us 12.00 us 64.00 us 40 INODELK 0.02 176.30 us 10.00 us 765.00 us 10 READDIR 0.07 63.08 us 39.00 us 133.00 us 78 UNLINK 0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK 0.13 126.89 us 99.00 us 179.00 us 76 MKNOD 0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP 28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP 29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE 40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK Duration: 189 seconds Data Read: 0 bytes Data Written: 39328 bytes Interval 0 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 39328 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2 FORGET 0.00 0.00 us 0.00 us 0.00 us 12 RELEASE 0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR 0.00 101.00 us 101.00 us 101.00 us 1 FSTAT 0.00 51.50 us 20.00 us 81.00 us 4 FLUSH 0.01 219.50 us 188.00 us 251.00 us 2 CREATE 0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR 0.01 62.30 us 38.00 us 119.00 us 10 OPEN 0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR 0.01 24.60 us 12.00 us 64.00 us 40 INODELK 0.02 176.30 us 10.00 us 765.00 us 10 READDIR 0.07 63.08 us 39.00 us 133.00 us 78 UNLINK 0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK 0.13 126.89 us 99.00 us 179.00 us 76 MKNOD 0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP 28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP 29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE 40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK Duration: 189 seconds Data Read: 0 bytes Data Written: 39328 bytes Brick: ovirt2.localdomain:/gluster_bricks/data/data --------------------------------------------------- Cumulative Stats: Block Size: 512b+ 131072b+ No. of Reads: 0 0 No. of Writes: 36 76758 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6 FORGET 0.00 0.00 us 0.00 us 0.00 us 87 RELEASE 0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR 0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR 0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR 0.00 36.18 us 22.00 us 62.00 us 11 FLUSH 0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE 0.00 82.56 us 59.00 us 138.00 us 9 FSTAT 0.00 89.42 us 67.00 us 161.00 us 12 SETATTR 0.00 272.40 us 235.00 us 296.00 us 5 CREATE 0.01 154.28 us 88.00 us 320.00 us 18 XATTROP 0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR 0.01 86.69 us 30.00 us 379.00 us 62 STAT 0.01 64.30 us 47.00 us 169.00 us 84 OPEN 0.02 107.34 us 23.00 us 273.00 us 73 READDIRP 0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE 0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR 0.03 128.51 us 27.00 us 338.00 us 96 FSYNC 0.03 240.75 us 14.00 us 1943.00 us 52 READDIR 0.04 65.59 us 26.00 us 293.00 us 279 STATFS 0.06 180.77 us 118.00 us 306.00 us 148 MKNOD 0.14 37.98 us 17.00 us 192.00 us 1598 INODELK 0.67 91.68 us 12.00 us 1141.00 us 3186 LOOKUP 10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP 11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK 19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK 25.21 142.92 us 62.00 us 593.00 us 76794 WRITE 32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK Duration: 1206 seconds Data Read: 0 bytes Data Written: 10060843008 bytes Interval 0 Stats: Block Size: 512b+ 131072b+ No. of Reads: 0 0 No. of Writes: 36 76758 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6 FORGET 0.00 0.00 us 0.00 us 0.00 us 87 RELEASE 0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR 0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR 0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR 0.00 36.18 us 22.00 us 62.00 us 11 FLUSH 0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE 0.00 82.56 us 59.00 us 138.00 us 9 FSTAT 0.00 89.42 us 67.00 us 161.00 us 12 SETATTR 0.00 272.40 us 235.00 us 296.00 us 5 CREATE 0.01 154.28 us 88.00 us 320.00 us 18 XATTROP 0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR 0.01 86.69 us 30.00 us 379.00 us 62 STAT 0.01 64.30 us 47.00 us 169.00 us 84 OPEN 0.02 107.34 us 23.00 us 273.00 us 73 READDIRP 0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE 0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR 0.03 128.51 us 27.00 us 338.00 us 96 FSYNC 0.03 240.75 us 14.00 us 1943.00 us 52 READDIR 0.04 65.59 us 26.00 us 293.00 us 279 STATFS 0.06 180.77 us 118.00 us 306.00 us 148 MKNOD 0.14 37.98 us 17.00 us 192.00 us 1598 INODELK 0.67 91.66 us 12.00 us 1141.00 us 3186 LOOKUP 10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP 11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK 19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK 25.21 142.92 us 62.00 us 593.00 us 76794 WRITE 32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK Duration: 1206 seconds Data Read: 0 bytes Data Written: 10060843008 bytes This indicates to me that it's not a problem in Disk/LVM/FileSystem layout. Most probably I haven't created the volume properly or some option/feature is disabled ?!? Network shows OK for a gigabit: [root@ovirt1 data]# dd if=/dev/zero status=progress | nc ovirt2 9999 3569227264 bytes (3.6 GB) copied, 29.001052 s, 123 MB/s^C 7180980+0 records in 7180979+0 records out 3676661248 bytes (3.7 GB) copied, 29.8739 s, 123 MB/s I'm looking for any help... you can share your volume info also. Thanks in advance. Best Regards, Strahil Nikolov _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
