|
Hi Dan, Thank you for the response. The Gluster is only access by the application and in the startup of the application time, I see a huge difference. I you see the statistics, you see that the application is writing a 2184 small files. You point me in the direction of the cache options. Do you have an explanation for the cache, why it would take 120min instead of 2min to start the application ? Block Size: 1b+ 16b+ 32b+ No. of Reads: 0 0 0 No. of Writes: 342 25
575 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 0 0 No. of Writes: 143
898 118 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 1 4 11 No. of Writes: 82 0 0 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 11 31 39 No. of Writes: 0 0 0 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 59 148
555 No. of Writes: 0 0 0 Vriendelijke groeten, Stijn Van Renterghem
Message: 10 Date: Thu, 18 Feb 2016 10:14:59 +1000 From: Dan Mons <dmons@xxxxxxxxxxxxxxxxxx> To: Stefan Jakobs <stefan@xxxxxxxxxxxxx> Cc: gluster-users <gluster-users@xxxxxxxxxxx> Subject: Re: FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ? Message-ID: <CACa6Tydras9yd8=VmGE1hXzhKJVa6ipVat10Ee3pfr=GjTUbOA@xxxxxxxxxxxxxx> Content-Type: text/plain; charset=UTF-8 Without knowing the details, I'm putting my money on cache. Choosing how to mount Gluster is workload dependent. If you're doing a lot of small files with single threaded writes, I suggest NFS. Your client's nfscache will dramatically improve performance from the end-user's point of view. If you're doing heavy multi-threaded reads and writes, and you have very good bandwidth from your client (e.g.: 10GbE) FUSE+GlusterFS is better, as it allows your client to talk to all Gluster nodes. If you are using FUSE+GlusterFS, on the gluster nodes themselves, experiment with the "performance.write-behind-window-size" and "performance.cache-size" options. Note that these will affect the cache used by the clients, so don't set them so high as to exhaust the RAM of any client connecting (or, for low-memory clients, use NFS instead). Gluster ships with conservative defaults for cache, which is a good thing. It's up to the user to tweak for their optimal needs. There's no right or wrong answer here. Experiment with NFS and various cache allocations with FUSE+GlusterFS, and see how you go. And again, consider your workloads, and whether or not they're taking full advantage of the FUSE client's ability to deal with highly parallel workloads. -Dan ---------------- Dan Mons - VFX Sysadmin Cutting Edge From: Van Renterghem Stijn Hi, I have setup a server with a new installation of Gluster. The volume type is ‘Replicate’. 1) I mounted the volume with Fuse IP1:/app /srv/data glusterfs defaults,_netdev,backupvolfile-server=IP2,fetch-attempts=2 0 0 When I start my application, it takes 2h until the application is started Below you can see the stats after the application is started. I can see a very high LOOKUP value. Can you explain this high value ? The volume type is replicate, so I should think, I shouldn’t have LOOKUPs ? Interval2 Block Size: 1b+ 16b+ 32b+ No. of Reads: 0 0 0 No. of Writes: 342 25
575 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 0 0 No. of Writes: 143
898 118 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 1 4 11 No. of Writes: 82 0 0 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 11 31 39 No. of Writes: 0 0 0 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 59 148
555 No. of Writes: 0 0 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1 FORGET 0.00 0.00 us 0.00 us 0.00 us 201 RELEASE 0.00 0.00 us 0.00 us 0.00 us 54549 RELEASEDIR 0.00 47.00 us 47.00 us 47.00 us 1 REMOVEXATTR 0.00 94.00 us 74.00 us 114.00 us 2 XATTROP 0.00 191.00 us 191.00 us 191.00 us 1 TRUNCATE 0.00 53.50 us 35.00 us 74.00 us 4 STATFS 0.00 79.67 us 70.00 us 91.00 us 3 RENAME 0.00 37.33 us 27.00 us 68.00 us 15 INODELK 0.00 190.67 us 116.00 us 252.00 us 3 UNLINK 0.00 28.83 us 8.00 us 99.00 us 30 ENTRYLK 0.00 146.33 us 117.00 us 188.00 us 6 CREATE 0.00 37.63 us 12.00 us 73.00 us 84 READDIR 0.00 23.75 us 8.00 us 75.00 us 198 FLUSH 0.00 65.33 us 42.00 us 141.00 us 204 OPEN 0.01 45.78 us 11.00 us 191.00 us 944 FINODELK 0.01 80.34 us 31.00 us 211.00 us 859 READ 0.02 96.74 us 50.00 us 188.00 us 944 FXATTROP 0.02 55.84 us 24.00 us 140.00 us 1707 FSTAT 0.02 52.89 us 21.00 us 175.00 us 2183 WRITE 0.02 59.69 us 11.00 us 235.00 us 2312 GETXATTR 0.03 51.18 us 8.00 us 142.00 us 3091 STAT 0.46 48.66 us 1.00 us 179.00 us 54549 OPENDIR 1.13 135.93 us 18.00 us 16362.00 us 48124 READDIRP
98.29 70.46 us 16.00 us 2903.00 us
8104385
LOOKUP Duration: 7560 seconds Data Read: 91208567 bytes
= 91MB Data Written: 292007 bytes
= 0,292MB 2) I have tried some tuning options, but that didn’t changed anything: #gluster volume info app Volume Name: app Type: Replicate Volume ID: f1b59aec-adf8-41f8-ad95-839ace247041 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: IP1:/exports/app/app Brick2: IP2:/exports/app/app Options Reconfigured: cluster.readdir-optimize: on server.event-threads: 8 client.event-threads: 8 cluster.lookup-optimize: on diagnostics.count-fop-hits: on diagnostics.latency-measurement: on auth.allow: client1,client2 nfs.rpc-auth-allow: client1,client2 nfs.export-volumes: on nfs.addr-namelookup: off nfs.disable: off performance.readdir-ahead: on performance.io-thread-count: 64 3) I then have enabled NFS support. I stopped the application and unmounted the volume. I then mounted it again with nfs: IP1:/app /srv/data nfs rsize=4096,wsize=4096,hard,intr 0 0 I started the application again and it was running within 3minutes. The stats with NFS where very different then with Fuse. It seems that they are almost not logged. Interval 11 Stats: Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 0 0 No. of Writes: 9 1 1 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 0 No. of Writes: 1 5 8 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR 0.02 2.00 us 2.00 us 2.00 us 1 OPENDIR 0.57 34.00 us 19.00 us 49.00 us 2 READDIR 0.81 96.00 us 96.00 us 96.00 us 1 SETATTR 1.06 62.50 us 61.00 us 64.00 us 2 OPEN 1.39 164.00 us 164.00 us 164.00 us 1 TRUNCATE 1.39 41.25 us 30.00 us 52.00 us 4 GETXATTR 1.54 91.00 us 86.00 us 96.00 us 2 XATTROP 2.72 80.50 us 29.00 us 122.00 us 4
LOOKUP 2.81 33.30 us 17.00 us 56.00 us 10 INODELK 10.36 76.69 us 26.00 us 133.00 us 16 FLUSH 15.83 75.00 us 61.00 us 105.00 us 25 WRITE 17.22 48.55 us 13.00 us 78.00 us 42 FINODELK 44.28 124.83 us 62.00 us 161.00 us 42 FXATTROP Duration: 580 seconds Data Read: 0 bytes Data Written: 60839 bytes What is wrong with the Fuse client ? Why does my application start in 120min with Gluster+Fuse and in 3min with Gluster+NFS ? Regards, Stijn |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
