Can't access volume during self-healing

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I'm evaluating gluster for use in our product, and I want to ensure that I understand the failover behavior.  What I'm seeing isn't great, but it doesn't look from the docs I've read that this is what everyone else is experiencing.


Is this normal?

Setup:

-          one volume, distributed, replicated (2), with two bricks on two different servers

-          35,000 files on volume, about 1MB each, all in one directory (I'm open to changing this, if that's the problem.  ls -l takes a really long time)

-          volume is mounted (mount -t gluster) on server 1

Procedure:

-          I stop glusterd and glusterfsd on server1, and send a few files to the volume.  This is fine.  I can write and read the files.

-          I start glusterd on server1, and this starts glusterfsd.  This triggers self-heal.

-          Send a file to the server, and try to read it.

-          Sending takes a couple of minutes.  Reading is immediate.

-          Once self-heal is done, subsequent sends and reads are immediate.

I tried profiling this operation, and it seems like it's stuck on locking the file:


(server1 is uca-amm3.cnda.avaya.com. server2 is uc-amm4.cnda.avaya.com)

Brick: uc-amm4.cnda.avaya.com:/media/data/brick1
------------------------------------------------
Cumulative Stats:
   Block Size:               1024b+                4096b+                8192b+
 No. of Reads:                    0                     0                     0
No. of Writes:                  112                  1216                 38216

   Block Size:              16384b+               32768b+               65536b+
 No. of Reads:                    0                  1765                 12554
No. of Writes:               144493                 15648                  3032

   Block Size:             131072b+
 No. of Reads:                91441
No. of Writes:                  247
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           5141      FORGET
      0.00       0.00 us       0.00 us       0.00 us          21270     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              6  RELEASEDIR
      0.00      61.00 us      61.00 us      61.00 us              1    GETXATTR
      0.00      62.00 us      62.00 us      62.00 us              1     OPENDIR
      0.00      36.86 us      27.00 us      74.00 us              7       FSTAT
      0.00      45.70 us      21.00 us      81.00 us             10       FLUSH
      0.00     123.00 us      83.00 us     135.00 us              5        OPEN
      0.00     118.29 us      41.00 us     315.00 us              7      STATFS
      0.00     419.60 us     266.00 us     539.00 us              5      CREATE
      0.00     422.69 us     118.00 us    2087.00 us             13     XATTROP
      0.00    1202.54 us      18.00 us   14631.00 us             13     ENTRYLK
      0.00     151.12 us      75.00 us     200.00 us            125        READ
      0.00      37.29 us      13.00 us    1232.00 us           1549    FINODELK
      0.00      80.78 us      43.00 us     151.00 us            762       WRITE
      0.00      74.75 us      40.00 us     371.00 us           1524    FXATTROP
      0.04    4004.48 us      95.00 us   17214.00 us           1156    READDIRP
     99.96   16538.92 us      58.00 us  976002.00 us         660602      LOOKUP
    Duration: 2820 seconds
   Data Read: 13651676656 bytes
Data Written: 3941825592 bytes
Interval 0 Stats:
   Block Size:               1024b+                4096b+                8192b+
 No. of Reads:                    0                     0                     0
No. of Writes:                  112                  1216                 38216

   Block Size:              16384b+               32768b+               65536b+
 No. of Reads:                    0                  1765                 12554
No. of Writes:               144493                 15648                  3032

   Block Size:             131072b+
 No. of Reads:                91441
No. of Writes:                  247
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           5141      FORGET
      0.00       0.00 us       0.00 us       0.00 us          21270     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              6  RELEASEDIR
      0.00      61.00 us      61.00 us      61.00 us              1    GETXATTR
      0.00      62.00 us      62.00 us      62.00 us              1     OPENDIR
      0.00      36.86 us      27.00 us      74.00 us              7       FSTAT
      0.00      45.70 us      21.00 us      81.00 us             10       FLUSH
      0.00     123.00 us      83.00 us     135.00 us              5        OPEN
      0.00     118.29 us      41.00 us     315.00 us              7      STATFS
      0.00     419.60 us     266.00 us     539.00 us              5      CREATE
      0.00     422.69 us     118.00 us    2087.00 us             13     XATTROP
      0.00    1202.54 us      18.00 us   14631.00 us             13     ENTRYLK
      0.00     151.12 us      75.00 us     200.00 us            125        READ
      0.00      37.29 us      13.00 us    1232.00 us           1549    FINODELK
      0.00      80.78 us      43.00 us     151.00 us            762       WRITE
      0.00      74.75 us      40.00 us     371.00 us           1524    FXATTROP
      0.04    4004.48 us      95.00 us   17214.00 us           1156    READDIRP
     99.96   16538.92 us      58.00 us  976002.00 us         660602      LOOKUP
    Duration: 2820 seconds
   Data Read: 13651676656 bytes
Data Written: 3941825592 bytes
Brick: uca-amm3.cnda.avaya.com:/media/data/brick1
-------------------------------------------------
Cumulative Stats:
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    1                    43                    72

   Block Size:              32768b+
 No. of Reads:                    0
No. of Writes:                    1
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              5     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR
      0.00     132.00 us     132.00 us     132.00 us              1     OPENDIR
      0.00      70.00 us      46.00 us      94.00 us              2       FLUSH
      0.00     175.00 us     175.00 us     175.00 us              1     XATTROP
      0.00     185.00 us     106.00 us     264.00 us              2      STATFS
      0.00     135.67 us      75.00 us     227.00 us              3    GETXATTR
      0.00     489.00 us     489.00 us     489.00 us              1      CREATE
      0.00     250.00 us     152.00 us     348.00 us              2     READDIR
      0.00     153.25 us     102.00 us     177.00 us              4        OPEN
      0.00     157.88 us      76.00 us     245.00 us              8     SETATTR
      0.00     330.25 us     257.00 us     430.00 us              4       MKNOD
      0.00      34.11 us      14.00 us     237.00 us            239    FINODELK
      0.00      83.54 us      62.00 us     179.00 us            117       WRITE
      0.00      99.28 us      42.00 us     298.00 us            234    FXATTROP
      0.14    5310.09 us     127.00 us   13588.00 us           1156    READDIRP
      2.55 22648341.40 us      74.00 us 113241113.00 us              5     ENTRYLK
     97.31   13061.69 us      16.00 us   47524.00 us         330308      LOOKUP
    Duration: 133 seconds
   Data Read: 0 bytes
Data Written: 1570968 bytes
Interval 0 Stats:
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    1                    43                    72

   Block Size:              32768b+
 No. of Reads:                    0
No. of Writes:                    1
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              5     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR
      0.00     132.00 us     132.00 us     132.00 us              1     OPENDIR
      0.00      70.00 us      46.00 us      94.00 us              2       FLUSH
      0.00     175.00 us     175.00 us     175.00 us              1     XATTROP
      0.00     185.00 us     106.00 us     264.00 us              2      STATFS
      0.00     135.67 us      75.00 us     227.00 us              3    GETXATTR
      0.00     489.00 us     489.00 us     489.00 us              1      CREATE
      0.00     250.00 us     152.00 us     348.00 us              2     READDIR
      0.00     153.25 us     102.00 us     177.00 us              4        OPEN
      0.00     157.88 us      76.00 us     245.00 us              8     SETATTR
      0.00     330.25 us     257.00 us     430.00 us              4       MKNOD
      0.00      34.11 us      14.00 us     237.00 us            239    FINODELK
      0.00      83.54 us      62.00 us     179.00 us            117       WRITE
      0.00      99.28 us      42.00 us     298.00 us            234    FXATTROP
      0.14    5310.09 us     127.00 us   13588.00 us           1156    READDIRP
      2.55 22648341.40 us      74.00 us 113241113.00 us              5     ENTRYLK
     97.31   13061.69 us      16.00 us   47524.00 us         330308      LOOKUP
    Duration: 133 seconds
   Data Read: 0 bytes
Data Written: 1570968 bytes



Any ideas?

Thanks,
Anne

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131009/267c9dbe/attachment.html>


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux