Re: Gluster native mount is really slow compared to nfs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Title: RE: [Gluster-users] Gluster native mount is really slow compared to nfs

Hello,

 

 

While there are probably other interesting parameters and options in gluster itself, for us the largest difference with this speedtest and also for our website (real world performance) was the negative-timeout value during mount. Only 1 seems to solve so many problems, is there anyone knowledgeable why this is the case? 

 

This would better be default I suppose ... 

 

I'm still wondering if there is a big underlying issue in gluster causing the difference to be so gigantic.



Regards

Jo

 


 

-----Original message-----
From: Jo Goossens <jo.goossens@xxxxxxxxxxxxxxxx>
Sent: Tue 11-07-2017 18:48
Subject: RE: [Gluster-users] Gluster native mount is really slow compared to nfs
CC: gluster-users@xxxxxxxxxxx;
To: Vijay Bellur <vbellur@xxxxxxxxxx>;

PS: I just tested between these 2:

 

mount -t glusterfs -o negative-timeout=1,use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log 192.168.140.41:/www /var/www
 
mount -t glusterfs -o use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log 192.168.140.41:/www /var/www
 
So it means only 1 second negative timeout...
 

In this particular test: ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 --threads 8 --files 50000 --file-size 64 --record-size 64

 
 
The result is about 4 seconds with the negative timeout of 1 second defined and many many minutes without the negative timeout (I quit after 15 minutes of waiting)
 
I will go over to some real world tests now to see how it performs there.
 
 
Regards
Jo

 

 


 

-----Original message-----
From: Jo Goossens <jo.goossens@xxxxxxxxxxxxxxxx>
Sent: Tue 11-07-2017 18:23
Subject: Re: [Gluster-users] Gluster native mount is really slow compared to nfs
To: Vijay Bellur <vbellur@xxxxxxxxxx>;
CC: gluster-users@xxxxxxxxxxx;

Hello Vijay,

 

 

What do you mean exactly? What info is missing?

 

PS: I already found out that for this particular test all the difference is made by : negative-timeout=600 , when removing it, it's much much slower again.

 

 

Regards

Jo
 

-----Original message-----
From: Vijay Bellur <vbellur@xxxxxxxxxx>
Sent: Tue 11-07-2017 18:16
Subject: Re: Gluster native mount is really slow compared to nfs
To: Jo Goossens <jo.goossens@xxxxxxxxxxxxxxxx>;
CC: gluster-users@xxxxxxxxxxx; Joe Julian <joe@xxxxxxxxxxxxxxxx>;


On Tue, Jul 11, 2017 at 11:39 AM, Jo Goossens <jo.goossens@xxxxxxxxxxxxxxxx> wrote:

Hello Joe,

 

 

I just did a mount like this (added the bold):

 

mount -t glusterfs -o attribute-timeout=600,entry-timeout=600,negative-timeout=600,fopen-keep-cache,use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log 192.168.140.41:/www /var/www
 

Results:

 

root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 --threads 8 --files 5000 --file-size 64 --record-size 64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5000
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file /var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 1.232004,files = 5000,records = 0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 1.148738,files = 5000,records = 0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 1.130913,files = 5000,records = 0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 1.183088,files = 5000,records = 0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 1.220752,files = 5000,records = 0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 1.228039,files = 5000,records = 0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 1.216787,files = 5000,records = 0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 1.229036,files = 5000,records = 0,status = ok
total threads = 8
total files = 40000
100.00% of requested files processed, minimum is  70.00
1.232004 sec elapsed time
32467.428972 files/sec
 

 

root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 --threads 8 --files 50000 --file-size 64 --record-size 64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 50000
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file /var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 4.242312,files = 50000,records = 0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 4.250831,files = 50000,records = 0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 3.771269,files = 50000,records = 0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 4.060653,files = 50000,records = 0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 3.880653,files = 50000,records = 0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 3.847107,files = 50000,records = 0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 3.895537,files = 50000,records = 0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 3.966394,files = 50000,records = 0,status = ok
total threads = 8
total files = 400000
100.00% of requested files processed, minimum is  70.00
4.250831 sec elapsed time
94099.245073 files/sec
root@app1:~/smallfile-master#
 

 

As you can see it's now crazy fast, I think close to or faster than nfs !! What the hell!??!

 

I'm so exited I already post. Any suggestions for those parameters? I will do additional testing over here , because this is ridiculous. That woud mean defaults or no good at all...

 

 
Would it be possible to profile the client [1] with defaults and the set of options used now? That could help in understanding the performance delta better.
 
Thanks,
Vijay
 
 
 _______________________________________________
 Gluster-users mailing list
 Gluster-users@xxxxxxxxxxx
 http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux