Re: FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Without knowing the details, I'm putting my money on cache.

Choosing how to mount Gluster is workload dependent.  If you're doing
a lot of small files with single threaded writes, I suggest NFS.  Your
client's nfscache will dramatically improve performance from the
end-user's point of view.

If you're doing heavy multi-threaded reads and writes, and you have
very good bandwidth from your client (e.g.: 10GbE) FUSE+GlusterFS is
better, as it allows your client to talk to all Gluster nodes.

If you are using FUSE+GlusterFS, on the gluster nodes themselves,
experiment with the "performance.write-behind-window-size" and
"performance.cache-size" options.  Note that these will affect the
cache used by the clients, so don't set them so high as to exhaust the
RAM of any client connecting (or, for low-memory clients, use NFS
instead).

Gluster ships with conservative defaults for cache, which is a good
thing.  It's up to the user to tweak for their optimal needs.

There's no right or wrong answer here.  Experiment with NFS and
various cache allocations with FUSE+GlusterFS, and see how you go.
And again, consider your workloads, and whether or not they're taking
full advantage of the FUSE client's ability to deal with highly
parallel workloads.

-Dan
----------------
Dan Mons - VFX Sysadmin
Cutting Edge
http://cuttingedge.com.au


On 18 February 2016 at 08:56, Stefan Jakobs <stefan@xxxxxxxxxxxxx> wrote:
> Van Renterghem Stijn:
>> Interval2
>> Block Size:                  1b+                  16b+                  32b+
>> No. of Reads:                    0                     0
>>  0 No. of Writes:                  342                    25
>>    575
>>
>>    Block Size:                 64b+                 128b+
>> 256b+ No. of Reads:                    0                     0
>>        0 No. of Writes:                  143                   898
>>          118
>>
>>    Block Size:                512b+                1024b+
>> 2048b+ No. of Reads:                    1                     4
>>        11 No. of Writes:                   82                     0
>>             0
>>
>>    Block Size:               4096b+                8192b+
>> 16384b+ No. of Reads:                   11                    31
>>         39 No. of Writes:                    0                     0
>>              0
>>
>>    Block Size:              32768b+               65536b+
>> 131072b+ No. of Reads:                   59                   148
>>         555 No. of Writes:                    0                     0
>>               0
>>
>> %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>> Fop ---------   -----------   -----------   -----------   ------------
>>   ---- 0.00       0.00 us       0.00 us       0.00 us              1
>> FORGET 0.00       0.00 us       0.00 us       0.00 us            201
>> RELEASE 0.00       0.00 us       0.00 us       0.00 us          54549
>> RELEASEDIR 0.00      47.00 us      47.00 us      47.00 us              1
>> REMOVEXATTR 0.00      94.00 us      74.00 us     114.00 us              2
>>   XATTROP 0.00     191.00 us     191.00 us     191.00 us              1
>> TRUNCATE 0.00      53.50 us      35.00 us      74.00 us              4
>> STATFS 0.00      79.67 us      70.00 us      91.00 us              3
>> RENAME 0.00      37.33 us      27.00 us      68.00 us             15
>> INODELK 0.00     190.67 us     116.00 us     252.00 us              3
>> UNLINK 0.00      28.83 us       8.00 us      99.00 us             30
>> ENTRYLK 0.00     146.33 us     117.00 us     188.00 us              6
>> CREATE 0.00      37.63 us      12.00 us      73.00 us             84
>> READDIR 0.00      23.75 us       8.00 us      75.00 us            198
>> FLUSH 0.00      65.33 us      42.00 us     141.00 us            204
>> OPEN 0.01      45.78 us      11.00 us     191.00 us            944
>> FINODELK 0.01      80.34 us      31.00 us     211.00 us            859
>>   READ 0.02      96.74 us      50.00 us     188.00 us            944
>> FXATTROP 0.02      55.84 us      24.00 us     140.00 us           1707
>>  FSTAT 0.02      52.89 us      21.00 us     175.00 us           2183
>> WRITE 0.02      59.69 us      11.00 us     235.00 us           2312
>> GETXATTR 0.03      51.18 us       8.00 us     142.00 us           3091
>>   STAT 0.46      48.66 us       1.00 us     179.00 us          54549
>> OPENDIR 1.13     135.93 us      18.00 us   16362.00 us          48124
>> READDIRP 98.29      70.46 us      16.00 us    2903.00 us        8104385
>>  LOOKUP
>>
>>     Duration: 7560 seconds
>>    Data Read: 91208567 bytes = 91MB
>> Data Written: 292007 bytes = 0,292MB
>
> How did you collect these statistics?
>
> Thanks
> Stefan
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux