#gluster vol set VOLNAME group nl-cache --> didn't know there are groups of options, after this command i got set the following:
performance.nl-cache-timeout: 600
performance.nl-cache: on
performance.parallel-readdir: on
performance.io-thread-count: 64
network.inode-lru-limit: 200000
to note that i had network.inode-lru-limit set to max and got reduced to 200000
then i added
performance.nl-cache-positive-entry: on
The volume options:
Options Reconfigured:
performance.nl-cache-timeout: 600
performance.nl-cache: on
performance.nl-cache-positive-entry: on
performance.parallel-readdir: on
performance.io-thread-count: 64
network.inode-lru-limit: 200000
nfs.disable: on
transport.address-family: inet
performance.readdir-ahead: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
performance.md-cache-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.cache-size: 10GB
network.ping-timeout: 5
diagnostics.client-log-level: WARNING
diagnostics.brick-log-level: WARNING
features.quota: off
features.inode-quota: off
performance.quick-read: on
untar completed in 08mins 30secs
increasing network.inode-lru-limit to 1048576 untar completed in the same time
I have attached the gluster profile results of the last test, with network.inode-lru-limit to 1048576
I guess the next test will be creating more bricks for the same volume to have a 2x3. Since i do not see bottlenecks at the disk level and i have limited hw ATM i will just carve out the bricks from LVs from the same 1 disk VG.
Also i have tried to look for a complete list of options/description unsuccessfully can you point at one?
thanks
Davide
On Thu, Aug 30, 2018 at 5:47 PM Poornima Gurusiddaiah <pgurusid@xxxxxxxxxx> wrote:
To enable nl-cache please use group option instead of single volume set:#gluster vol set VOLNAME group nl-cacheThis sets few other things including time out, invalidation etc.For enabling the option Raghavendra mentioned, you ll have to execute it explicitly, as it's not part of group option yet:#gluster vol set VOLNAME performance.nl-cache-positive-entry onAlso from the past experience, setting the below option has helped in performance:# gluster vol set VOLNAME network.inode-lru-limit 200000Regards,PoornimaOn Thu, Aug 30, 2018, 8:49 PM Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx> wrote:_______________________________________________On Thu, Aug 30, 2018 at 8:38 PM, Davide Obbi <davide.obbi@xxxxxxxxxxx> wrote:yes "performance.parallel-readdir on and 1x3 replicaThat's surprising. I thought performance.parallel-readdir will help only when distribute count is fairly high. This is something worth investigating further.On Thu, Aug 30, 2018 at 5:00 PM Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx> wrote:On Thu, Aug 30, 2018 at 8:08 PM, Davide Obbi <davide.obbi@xxxxxxxxxxx> wrote:Thanks Amar,i have enabled the negative lookups cache on the volume:I think enabling nl-cache-positive-entry might help for untarring or git clone into glusterfs. Its disabled by default. can you let us know the results?
Option: performance.nl-cache-positive-entry
Default Value: (null)
Description: enable/disable storing of entries that were lookedup and found to be present in the volume, thus lookup on non existent file is served from the cacheTo deflate a tar archive (not compressed) of 1.3GB it takes aprox 9mins which can be considered a slight improvement from the previous 12-15 however still not fast enough compared to local disk. The tar is present on the gluster share/volume and deflated inside the same folder structure.I am assuming this is with parallel-readdir enabled, right?Running the operation twice (without removing the already deflated files) also did not reduce the time spent.Running the operation with the tar archive on local disk made no differenceWhat really made a huge difference while git cloning was setting "performance.parallel-readdir on". During the phase "Receiving objects" , as i enabled the xlator it bumped up from 3/4MBs to 27MBsWhat is the distribute count? Is it 1x3 replica?So in conclusion i'm trying to make the untar operation working at an acceptable level, not expecting local disks speed but at least being within the 4minsI have attached the profiles collected at the end of the untar operations with the archive on the mount and outsidethanksDavide--On Tue, Aug 28, 2018 at 8:41 AM Amar Tumballi <atumball@xxxxxxxxxx> wrote:One of the observation we had with git clone like work load was, nl-cache (negative-lookup cache), helps here.Try 'gluster volume set $volume-name nl-cache enable'.Also sharing the 'profile info' during this performance observations also helps us to narrow down the situation.More on how to capture profile info @ https://hackmd.io/PhhT5jPdQIKxzfeLQmnjJQ?view-AmarOn Thu, Aug 23, 2018 at 7:11 PM, Davide Obbi <davide.obbi@xxxxxxxxxxx> wrote:Hello,
did anyone ever managed to achieve reasonable waiting time while performing metadata intensive operations such as git clone, untar etc...? Is this possible workload or will never be in scope for glusterfs?
I'd like to know, if possible, what would be the options that affect such volume performances.
Albeit i managed to achieve decent git status/git grep operations, 3 and 30 secs, the git clone and untarring a file from/to the same share take ages. for a git repo of aprox 6GB.I'm running a test environment with 3 way replica 128GB RAM and 24 cores are 2.40GHz, one internal SSD dedicated to the volume brick and 10Gb networkThe options set so far that affects volume performances are:
48 performance.readdir-ahead: on
49 features.cache-invalidation-timeout: 600
50 features.cache-invalidation: on
51 performance.md-cache-timeout: 600
52 performance.stat-prefetch: on
53 performance.cache-invalidation: on
54 performance.parallel-readdir: on
55 network.inode-lru-limit: 900000
56 performance.io-thread-count: 32
57 performance.cache-size: 10GB
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users--Amar Tumballi (amarts)Davide ObbiSystem AdministratorBooking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL NetherlandsDirect +31207031558The world's #1 accommodation site
43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day
No booking fees, best price always guaranteed
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users--Davide ObbiSystem AdministratorBooking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL NetherlandsDirect +31207031558The world's #1 accommodation site
43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day
No booking fees, best price always guaranteed
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
Davide Obbi
System Administrator
Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
Direct +31207031558
43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day
No booking fees, best price always guaranteed
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
Brick: glusterserver-1005:/srv/gfs/test01/brk01/brick
-----------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 3
No. of Writes: 294 116 1096
Block Size: 8b+ 16b+ 32b+
No. of Reads: 4 10 7
No. of Writes: 2557 7882 15309
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2 4 7
No. of Writes: 36962 45448 81464
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 4 10 10
No. of Writes: 172545 242644 248911
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 6 4 1
No. of Writes: 328818 219934 113897
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 5 3 97791
No. of Writes: 30425 29940 3538
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 896825 FORGET
0.00 0.00 us 0.00 us 0.00 us 924635 RELEASE
0.00 0.00 us 0.00 us 0.00 us 339641 RELEASEDIR
0.00 29.66 us 21.47 us 43.39 us 3 IPC
0.00 79.07 us 65.72 us 87.75 us 3 OPEN
0.00 134.74 us 101.38 us 164.15 us 4 XATTROP
0.00 59.56 us 30.56 us 119.36 us 28 STAT
0.00 147.49 us 8.67 us 20057.07 us 474 GETXATTR
0.01 201.57 us 122.66 us 831.05 us 708 READDIR
0.02 53.68 us 19.37 us 2509.50 us 4269 STATFS
0.55 61.48 us 1.23 us 4126.95 us 129250 OPENDIR
0.69 236.16 us 61.19 us 450562.83 us 42307 RMDIR
0.93 311.30 us 73.79 us 22016.10 us 43259 READDIRP
0.97 46.57 us 12.83 us 3028.93 us 300094 FSTAT
1.05 786.11 us 78.71 us 1608062.77 us 19308 SYMLINK
1.12 79.61 us 34.32 us 18196.28 us 202935 SETXATTR
1.91 34.59 us 8.54 us 3191.28 us 794395 FLUSH
2.51 356.26 us 102.22 us 937208.95 us 101455 MKDIR
3.85 35.95 us 9.20 us 98426.35 us 1544860 FINODELK
4.26 1558.86 us 34.30 us 2179743.97 us 39390 READ
5.98 82.55 us 23.84 us 33131.27 us 1044119 WRITE
6.64 91.22 us 27.44 us 106627.80 us 1049164 SETATTR
7.59 218.99 us 56.19 us 792203.93 us 499616 UNLINK
8.59 132.48 us 13.22 us 181848.34 us 933898 LOOKUP
8.73 39.29 us 6.58 us 105988.10 us 3201734 ENTRYLK
8.96 42.81 us 8.65 us 3840.32 us 3014442 INODELK
13.83 129.04 us 56.01 us 855297.69 us 1544872 FXATTROP
21.79 395.23 us 84.43 us 2698257.78 us 794395 CREATE
0.00 0.00 us 0.00 us 0.00 us 757637 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 757634 CI_FORGET
Duration: 75917 seconds
Data Read: 12818311007 bytes
Data Written: 12043293853 bytes
Interval 54 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
15.47 22.05 us 22.05 us 22.05 us 1 INODELK
30.23 43.10 us 43.10 us 43.10 us 1 STATFS
54.30 77.42 us 77.42 us 77.42 us 1 SETXATTR
Duration: 9 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: glusterserver-1008:/srv/gfs/test01/brk02/brick
-----------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 3
No. of Writes: 294 116 1096
Block Size: 8b+ 16b+ 32b+
No. of Reads: 4 8 7
No. of Writes: 2557 7882 15309
Block Size: 64b+ 128b+ 256b+
No. of Reads: 7 2 3
No. of Writes: 36962 45448 81464
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 8 6 6
No. of Writes: 172545 242644 248911
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 5 3 0
No. of Writes: 328818 219934 113897
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1 0 134
No. of Writes: 30425 29940 3538
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 896765 FORGET
0.00 0.00 us 0.00 us 0.00 us 924635 RELEASE
0.00 0.00 us 0.00 us 0.00 us 339635 RELEASEDIR
0.00 13.37 us 13.37 us 13.37 us 1 IPC
0.00 66.69 us 53.10 us 83.91 us 3 OPEN
0.00 163.49 us 98.12 us 221.13 us 4 XATTROP
0.00 58.06 us 26.68 us 99.72 us 48 STAT
0.00 111.60 us 8.91 us 10140.95 us 475 GETXATTR
0.01 180.16 us 120.31 us 827.79 us 708 READDIR
0.02 50.21 us 19.04 us 317.89 us 4269 STATFS
0.58 57.68 us 1.15 us 2371.81 us 129210 OPENDIR
0.74 225.68 us 58.56 us 523496.35 us 42294 RMDIR
0.80 39.83 us 13.41 us 1725.14 us 257666 FSTAT
1.06 312.56 us 79.89 us 21462.77 us 43289 READDIRP
1.20 75.64 us 34.17 us 36849.81 us 202935 SETXATTR
1.29 854.18 us 79.47 us 1818051.47 us 19308 SYMLINK
1.85 29.93 us 8.26 us 3317.58 us 794395 FLUSH
2.33 294.38 us 100.64 us 1317507.92 us 101455 MKDIR
3.78 31.41 us 9.04 us 10312.45 us 1544860 FINODELK
6.06 74.47 us 23.47 us 107814.46 us 1044119 WRITE
7.01 85.65 us 28.44 us 349303.92 us 1049164 SETATTR
7.86 201.82 us 54.81 us 902938.47 us 499515 UNLINK
8.80 35.25 us 6.83 us 17269.82 us 3201467 ENTRYLK
8.99 122.18 us 12.96 us 3501.46 us 943281 LOOKUP
9.05 38.50 us 8.57 us 3244.38 us 3013980 INODELK
14.45 119.98 us 58.56 us 1015242.53 us 1544872 FXATTROP
24.11 389.23 us 88.05 us 3212752.83 us 794395 CREATE
0.00 0.00 us 0.00 us 0.00 us 760755 UPCALL
0.00 0.00 us 0.00 us 0.00 us 4 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 760751 CI_FORGET
Duration: 75916 seconds
Data Read: 17721794 bytes
Data Written: 12043293853 bytes
Interval 54 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
20.95 42.06 us 42.06 us 42.06 us 1 STATFS
21.97 44.10 us 44.10 us 44.10 us 1 STAT
24.05 48.29 us 48.29 us 48.29 us 1 INODELK
33.03 66.32 us 66.32 us 66.32 us 1 SETXATTR
Duration: 9 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: glusterserver-1009:/srv/gfs/test01/brk03/brick
-----------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 294 116 1096
Block Size: 8b+ 16b+ 32b+
No. of Reads: 3 6 6
No. of Writes: 2557 7882 15309
Block Size: 64b+ 128b+ 256b+
No. of Reads: 4 8 4
No. of Writes: 36962 45448 81464
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 2 5 10
No. of Writes: 172545 242644 248911
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 8 5 2
No. of Writes: 328818 219934 113897
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1 0 0
No. of Writes: 30425 29940 3538
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 896824 FORGET
0.00 0.00 us 0.00 us 0.00 us 924635 RELEASE
0.00 0.00 us 0.00 us 0.00 us 339635 RELEASEDIR
0.00 10.74 us 10.74 us 10.74 us 1 IPC
0.00 161.89 us 161.89 us 161.89 us 1 READ
0.00 60.71 us 49.68 us 72.11 us 3 OPEN
0.00 133.84 us 101.36 us 221.90 us 4 XATTROP
0.00 55.63 us 22.18 us 114.32 us 33 STAT
0.00 107.00 us 8.75 us 10290.51 us 476 GETXATTR
0.01 183.07 us 118.05 us 814.73 us 708 READDIR
0.02 49.79 us 17.32 us 316.04 us 4269 STATFS
0.56 54.30 us 1.06 us 2465.17 us 129212 OPENDIR
0.75 220.71 us 59.96 us 362941.28 us 42295 RMDIR
0.81 39.33 us 13.09 us 2392.57 us 256957 FSTAT
1.15 338.92 us 83.33 us 19915.34 us 42509 READDIRP
1.22 74.94 us 34.92 us 16300.31 us 202935 SETXATTR
1.45 934.84 us 78.16 us 2381641.79 us 19308 SYMLINK
1.89 29.76 us 8.42 us 2740.79 us 794395 FLUSH
2.12 261.21 us 104.18 us 543133.73 us 101455 MKDIR
3.86 31.19 us 9.12 us 13211.42 us 1544860 FINODELK
6.13 73.29 us 24.00 us 16058.39 us 1044119 WRITE
6.98 83.07 us 27.11 us 20891.54 us 1049164 SETATTR
7.69 192.35 us 57.56 us 1016426.04 us 499525 UNLINK
8.90 34.72 us 7.26 us 34002.85 us 3201497 ENTRYLK
9.12 120.69 us 12.89 us 179277.82 us 943388 LOOKUP
9.20 38.10 us 8.54 us 3504.33 us 3014388 INODELK
14.47 116.99 us 57.00 us 74415.45 us 1544872 FXATTROP
23.67 372.07 us 86.98 us 2749754.20 us 794395 CREATE
0.00 0.00 us 0.00 us 0.00 us 760911 UPCALL
0.00 0.00 us 0.00 us 0.00 us 2 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 760909 CI_FORGET
Duration: 75916 seconds
Data Read: 232191 bytes
Data Written: 12043293853 bytes
Interval 54 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
22.83 44.61 us 44.61 us 44.61 us 1 INODELK
34.35 67.11 us 67.11 us 67.11 us 1 SETXATTR
42.81 83.64 us 83.64 us 83.64 us 1 STATFS
Duration: 9 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Attachment:
io-stats-pre-netmax.txt.test01
Description: Binary data
Attachment:
io-stats-post-netmax.txt.test01
Description: Binary data
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
