Hi,
Thanks for sharing the output. May be in the case of NVME there is no performance difference in SSL vs Non-SSL.
Can you please share the output of the command "gluster v info" and what client you are using to mount the volume?
If you are using fuse have u used any specific argument at the time of mounting the volume?
Please use default event-threads if you are not getting any performance improvement and share the command output
"top -bH -d 10" at the time of writing IO.
Thanks,
Mohit Agrawal
Thanks for sharing the output. May be in the case of NVME there is no performance difference in SSL vs Non-SSL.
Can you please share the output of the command "gluster v info" and what client you are using to mount the volume?
If you are using fuse have u used any specific argument at the time of mounting the volume?
Please use default event-threads if you are not getting any performance improvement and share the command output
"top -bH -d 10" at the time of writing IO.
Thanks,
Mohit Agrawal
On Mon, Jan 6, 2020 at 4:01 AM Michael Richardson <hello@xxxxxxxxxxxxxxxxxxxxx> wrote:
Hi Mohit and Strahil,Thank you for taking the time to share your advice. Strahil, unfortunately your message didn't come to my inbox, so I'm combining my reply to both yourself and Mohit.Mohit,I mentioned there was no performance difference between using SSL and not using SSL. I did try setting it to AES128 anyway, but that seemed to have no effect as well.
Strahil,I've had a go at various different settings this morning, including all of those you've supplied. I couldn't see any improvement in throughput when messing with these, but I could make it worse by adjusting things like event-threads.I've also tried adjusting the fio test types from one large file to many small files and a mix in between without any real change in peak performance.In answer to your other question, I'm running Gluster 7.1-1 on Debian 10. I have confirmed that blk-mq is on and my scheduler is 'none'. I don't seem to have an option to change it to 'noop' but I'm not sure why.Here's some profile output as well, in case that's helpful. These' values don't look so bad (to me, at least), but again this is much less than the raw nvme output that I was hoping to get closer to.Brick: node1:/data/nvme/testvol1/brick
-------------------------------------------------------------
Cumulative Stats:
Block Size: 16384b+
No. of Reads: 492397
No. of Writes: 2849474
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE
0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR
0.00 33.67 us 30.56 us 43.31 us 5 GETXATTR
0.01 26.05 us 14.36 us 128.33 us 870 ENTRYLK
0.02 53.48 us 43.23 us 87.31 us 435 FTRUNCATE
0.02 58.54 us 46.75 us 94.94 us 435 FALLOCATE
0.04 24.21 us 9.58 us 679.17 us 2436 FLUSH
0.04 145.95 us 116.32 us 211.72 us 435 CREATE
0.28 211.24 us 32.92 us 17442.92 us 2000 OPEN
0.38 82.16 us 34.00 us 15537.54 us 6929 LOOKUP
1.92 6683.77 us 437.30 us 28616.62 us 436 FSYNC
2.03 48.87 us 7.50 us 24840.37 us 63019 FINODELK
18.97 455.94 us 43.78 us 43701.58 us 63019 FXATTROP
28.06 134.43 us 41.87 us 36537.65 us 316217 WRITE
48.22 611.65 us 100.31 us 43842.80 us 119420 READ
Duration: 3505 seconds
Data Read: 8067432448 bytes
Data Written: 46685782016 bytes
Interval 0 Stats:
Block Size: 16384b+
No. of Reads: 492397
No. of Writes: 2849474
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE
0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR
0.00 33.67 us 30.56 us 43.31 us 5 GETXATTR
0.01 26.05 us 14.36 us 128.33 us 870 ENTRYLK
0.02 53.48 us 43.23 us 87.31 us 435 FTRUNCATE
0.02 58.54 us 46.75 us 94.94 us 435 FALLOCATE
0.04 24.21 us 9.58 us 679.17 us 2436 FLUSH
0.04 145.95 us 116.32 us 211.72 us 435 CREATE
0.28 211.24 us 32.92 us 17442.92 us 2000 OPEN
0.38 82.16 us 34.00 us 15537.54 us 6929 LOOKUP
1.92 6683.77 us 437.30 us 28616.62 us 436 FSYNC
2.03 48.87 us 7.50 us 24840.37 us 63019 FINODELK
18.97 455.94 us 43.78 us 43701.58 us 63019 FXATTROP
28.06 134.43 us 41.87 us 36537.65 us 316217 WRITE
48.23 611.65 us 100.31 us 43842.80 us 119420 READ
Duration: 3505 seconds
Data Read: 8067432448 bytes
Data Written: 46685782016 bytes
Brick: node2:/data/nvme/testvol1/brick
-------------------------------------------------------------
Cumulative Stats:
Block Size: 16384b+
No. of Reads: 0
No. of Writes: 2849474
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE
0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR
0.00 27.59 us 26.48 us 28.92 us 5 GETXATTR
0.06 16.31 us 11.92 us 43.37 us 870 ENTRYLK
0.07 40.53 us 36.62 us 163.89 us 435 FTRUNCATE
0.08 43.30 us 38.41 us 70.49 us 435 FALLOCATE
0.13 12.82 us 9.48 us 79.63 us 2436 FLUSH
0.21 114.31 us 96.53 us 166.02 us 435 CREATE
1.53 184.29 us 24.29 us 19723.10 us 2000 OPEN
1.82 63.25 us 25.23 us 19525.54 us 6929 LOOKUP
4.81 18.44 us 9.06 us 182.62 us 63019 FINODELK
11.69 6473.51 us 380.98 us 17129.47 us 436 FSYNC
17.41 66.69 us 44.52 us 12198.65 us 63019 FXATTROP
62.20 47.49 us 31.49 us 16215.18 us 316195 WRITE
Duration: 3486 seconds
Data Read: 0 bytes
Data Written: 46685782016 bytes
Interval 0 Stats:
Block Size: 16384b+
No. of Reads: 0
No. of Writes: 2849474
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE
0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR
0.00 27.59 us 26.48 us 28.92 us 5 GETXATTR
0.06 16.31 us 11.92 us 43.37 us 870 ENTRYLK
0.07 40.53 us 36.62 us 163.89 us 435 FTRUNCATE
0.08 43.30 us 38.41 us 70.49 us 435 FALLOCATE
0.13 12.82 us 9.48 us 79.63 us 2436 FLUSH
0.21 114.31 us 96.53 us 166.02 us 435 CREATE
1.53 184.29 us 24.29 us 19723.10 us 2000 OPEN
1.82 63.25 us 25.23 us 19525.54 us 6929 LOOKUP
4.81 18.44 us 9.06 us 182.62 us 63019 FINODELK
11.69 6473.51 us 380.98 us 17129.47 us 436 FSYNC
17.41 66.69 us 44.52 us 12198.65 us 63019 FXATTROP
62.20 47.49 us 31.49 us 16215.18 us 316195 WRITE
Duration: 3486 seconds
Data Read: 0 bytes
Data Written: 46685782016 bytes
Brick: node3:/data/nvme/testvol1/brick
-------------------------------------------------------------
Cumulative Stats:
Block Size: 16384b+
No. of Reads: 0
No. of Writes: 2849474
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE
0.00 0.00 us 0.00 us 0.00 us 19 RELEASEDIR
0.00 30.60 us 27.30 us 39.71 us 5 GETXATTR
0.06 17.42 us 12.10 us 61.92 us 870 ENTRYLK
0.07 40.77 us 36.20 us 58.29 us 435 FTRUNCATE
0.08 45.15 us 38.82 us 68.36 us 435 FALLOCATE
0.13 13.22 us 9.49 us 79.73 us 2436 FLUSH
0.21 117.47 us 98.17 us 169.38 us 435 CREATE
1.47 176.64 us 26.93 us 19510.96 us 2000 OPEN
1.78 61.78 us 25.32 us 17795.12 us 6929 LOOKUP
4.98 19.05 us 10.17 us 7635.73 us 63019 FINODELK
11.53 6369.75 us 392.09 us 16146.95 us 436 FSYNC
17.82 68.11 us 44.52 us 11326.21 us 63019 FXATTROP
61.85 47.11 us 30.86 us 3458.30 us 316194 WRITE
Duration: 3481 seconds
Data Read: 0 bytes
Data Written: 46685782016 bytes
Interval 0 Stats:
Block Size: 16384b+
No. of Reads: 0
No. of Writes: 2849474
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57 FORGET
0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE
0.00 0.00 us 0.00 us 0.00 us 19 RELEASEDIR
0.00 30.60 us 27.30 us 39.71 us 5 GETXATTR
0.06 17.42 us 12.10 us 61.92 us 870 ENTRYLK
0.07 40.77 us 36.20 us 58.29 us 435 FTRUNCATE
0.08 45.15 us 38.82 us 68.36 us 435 FALLOCATE
0.13 13.22 us 9.49 us 79.73 us 2436 FLUSH
0.21 117.47 us 98.17 us 169.38 us 435 CREATE
1.47 176.64 us 26.93 us 19510.96 us 2000 OPEN
1.78 61.78 us 25.32 us 17795.12 us 6929 LOOKUP
4.98 19.05 us 10.17 us 7635.73 us 63019 FINODELK
11.53 6369.75 us 392.09 us 16146.95 us 436 FSYNC
17.82 68.11 us 44.52 us 11326.21 us 63019 FXATTROP
61.85 47.12 us 30.86 us 3458.30 us 316194 WRITE
Duration: 3481 seconds
Data Read: 0 bytes
Data Written: 46685782016 bytesOn Mon, Jan 6, 2020 at 12:49 AM Mohit Agrawal <moagrawa@xxxxxxxxxx> wrote:Hi,________Along with previous tunning suggested by Strahil please configure "ssl.cipher-list"to AES128 for specific volume to improve the performance. As you mentioned youhave configured SSL on a volume and the performance is drop in case of SSL. To improve the same please configure the AES128 cipher list, I hope you should get sufficient performance improvement.Please share the result of a performance improvement after configuring the cipherAES128 if it is possible for you.Thanks,Mohit Agrawal
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
