FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

 

I have setup a server with a new installation of Gluster.

The volume type is ‘Replicate’.

 

1)

I mounted the volume with Fuse

IP1:/app       /srv/data               glusterfs       defaults,_netdev,backupvolfile-server=IP2,fetch-attempts=2      0 0

 

When I start my application, it takes 2h until the application is started

Below you can see the stats after the application is started. I can see a very high LOOKUP value.

Can you explain this high value ? The volume type is replicate, so I should think, I shouldn’t have LOOKUPs ?

 

Interval2

Block Size:                  1b+                  16b+                  32b+

No. of Reads:                    0                     0                     0

No. of Writes:                  342                    25                   575

 

   Block Size:                 64b+                 128b+                 256b+

No. of Reads:                    0                     0                     0

No. of Writes:                  143                   898                   118

 

   Block Size:                512b+                1024b+                2048b+

No. of Reads:                    1                     4                    11

No. of Writes:                   82                     0                     0

 

   Block Size:               4096b+                8192b+               16384b+

No. of Reads:                   11                    31                    39

No. of Writes:                    0                     0                     0

 

   Block Size:              32768b+               65536b+              131072b+

No. of Reads:                   59                   148                   555

No. of Writes:                    0                     0                     0

 

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop

---------   -----------   -----------   -----------   ------------        ----

      0.00       0.00 us       0.00 us       0.00 us              1      FORGET

      0.00       0.00 us       0.00 us       0.00 us            201     RELEASE

      0.00       0.00 us       0.00 us       0.00 us          54549  RELEASEDIR

      0.00      47.00 us      47.00 us      47.00 us              1 REMOVEXATTR

      0.00      94.00 us      74.00 us     114.00 us              2     XATTROP

      0.00     191.00 us     191.00 us     191.00 us              1    TRUNCATE

      0.00      53.50 us      35.00 us      74.00 us              4      STATFS

      0.00      79.67 us      70.00 us      91.00 us              3      RENAME

      0.00      37.33 us      27.00 us      68.00 us             15     INODELK

      0.00     190.67 us     116.00 us     252.00 us              3      UNLINK

      0.00      28.83 us       8.00 us      99.00 us             30     ENTRYLK

      0.00     146.33 us     117.00 us     188.00 us              6      CREATE

      0.00      37.63 us      12.00 us      73.00 us             84     READDIR

      0.00      23.75 us       8.00 us      75.00 us            198       FLUSH

      0.00      65.33 us      42.00 us     141.00 us            204        OPEN

      0.01      45.78 us      11.00 us     191.00 us            944    FINODELK

      0.01      80.34 us      31.00 us     211.00 us            859        READ

      0.02      96.74 us      50.00 us     188.00 us            944    FXATTROP

      0.02      55.84 us      24.00 us     140.00 us           1707       FSTAT

      0.02      52.89 us      21.00 us     175.00 us           2183       WRITE

      0.02      59.69 us      11.00 us     235.00 us           2312    GETXATTR

      0.03      51.18 us       8.00 us     142.00 us           3091        STAT

      0.46      48.66 us       1.00 us     179.00 us          54549     OPENDIR

      1.13     135.93 us      18.00 us   16362.00 us          48124    READDIRP

     98.29      70.46 us      16.00 us    2903.00 us        8104385      LOOKUP

 

    Duration: 7560 seconds

   Data Read: 91208567 bytes = 91MB

Data Written: 292007 bytes = 0,292MB

 

 

 

2)

I have tried some tuning options, but that didn’t changed anything:

 

#gluster volume info app

 

Volume Name: app

Type: Replicate

Volume ID: f1b59aec-adf8-41f8-ad95-839ace247041

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: IP1:/exports/app/app

Brick2: IP2:/exports/app/app

Options Reconfigured:

cluster.readdir-optimize: on

server.event-threads: 8

client.event-threads: 8

cluster.lookup-optimize: on

diagnostics.count-fop-hits: on

diagnostics.latency-measurement: on

auth.allow: client1,client2

nfs.rpc-auth-allow: client1,client2

nfs.export-volumes: on

nfs.addr-namelookup: off

nfs.disable: off

performance.readdir-ahead: on

performance.io-thread-count: 64

 

 

 

 

3)

I then have enabled NFS support.

I stopped the application and unmounted the volume. I then mounted it again with nfs:

IP1:/app        /srv/data               nfs             rsize=4096,wsize=4096,hard,intr  0 0

 

I started the application again and it was running within 3minutes.

The stats with NFS where very different then with Fuse. It seems that they are almost not logged.

 

Interval 11 Stats:

   Block Size:                128b+                 256b+                 512b+

No. of Reads:                    0                     0                     0

No. of Writes:                    9                     1                     1

 

   Block Size:               1024b+                2048b+                4096b+

No. of Reads:                    0                     0                     0

No. of Writes:                    1                     5                     8

 

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop

---------   -----------   -----------   -----------   ------------        ----

      0.00       0.00 us       0.00 us       0.00 us              2     RELEASE

      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR

      0.02       2.00 us       2.00 us       2.00 us              1     OPENDIR

      0.57      34.00 us      19.00 us      49.00 us              2     READDIR

      0.81      96.00 us      96.00 us      96.00 us              1     SETATTR

      1.06      62.50 us      61.00 us      64.00 us              2        OPEN

      1.39     164.00 us     164.00 us     164.00 us              1    TRUNCATE

      1.39      41.25 us      30.00 us      52.00 us              4    GETXATTR

      1.54      91.00 us      86.00 us      96.00 us              2     XATTROP

      2.72      80.50 us      29.00 us     122.00 us              4      LOOKUP

      2.81      33.30 us      17.00 us      56.00 us             10     INODELK

     10.36      76.69 us      26.00 us     133.00 us             16       FLUSH

     15.83      75.00 us      61.00 us     105.00 us             25       WRITE

     17.22      48.55 us      13.00 us      78.00 us             42    FINODELK

     44.28     124.83 us      62.00 us     161.00 us             42    FXATTROP

 

    Duration: 580 seconds

   Data Read: 0 bytes

Data Written: 60839 bytes

 

 

 

What is wrong with the Fuse client ?

Why does my application start in 120min with Gluster+Fuse and in 3min with Gluster+NFS ?

 

 

 

 

 

Regards,

Stijn

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux