Re: Cannot run VMware Virtual Machines on GlusterFS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Tomoaki, this is very useful. I will look deeper soon.
Thanks!

Avati

On Thu, Jun 21, 2012 at 9:21 PM, Tomoaki Sato <tsato@xxxxxxxxxxxxx> wrote:
Avati,

tshark says ...
FH values that the linux kernel NFS server returns stays constant for every LOOKUP 'foo' but,
FH values that the GlusterFS(NFS) returns are non-constant.

operaions at the ESXi host:

~ # ./getcwd /vmfs/volumes/94925201-78f190e0/foo
========= sleep 30 ================
/vmfs/volumes/94925201-78f190e0/foo
~ #

tshark's output at the linux kernel NFS server:

# tshark -i 2 -R nfs
Running as user "root" and group "root". This could be dangerous.
Capturing on br0
/* chdir */
 2.056680 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff
 2.056990 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 13)  Directory mode:0755 uid:0 gid:0
 9.848666 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff
 9.848767 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 60)  Directory mode:0755 uid:0 gid:0
 9.848966 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo
 9.849049 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 62), FH:0xdb05b90a         <=====
 20.055508 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff
 20.055702 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 103)  Directory mode:0755 uid:0 gid:0
 29.054939 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff
 29.055180 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 132)  Directory mode:0755 uid:0 gid:0
 38.054338 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff
 38.054583 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 151)  Directory mode:0755 uid:0 gid:0
/* getcwd */
 39.849107 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0xdb05b90a/..
 39.849449 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 170), FH:0x1ffd38ff
 39.849676 192.168.1.23 -> 192.168.1.254 NFS V3 READDIRPLUS Call, FH:0x1ffd38ff
 39.849833 192.168.1.254 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 172) . .. foo
 39.850071 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo
 39.850149 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 174), FH:0xdb05b90a
 39.850746 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0xdb05b90a/..
 39.850814 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 176), FH:0x1ffd38ff
 39.851014 192.168.1.23 -> 192.168.1.254 NFS V3 READDIRPLUS Call, FH:0x1ffd38ff
 39.851095 192.168.1.254 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 178) . .. foo
 39.851329 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo
 39.851438 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 180), FH:0xdb05b90a            <=====

operations at the ESXi host:

~ # ./getcwd /vmfs/volumes/ef172a87-e5ae817f/foo
========= sleep 30 ================
getcwd: No such file or directory
~ #

tshark's output at the GlusterFS(NFS) server:

# tshark -i 1 -R nfs
Running as user "root" and group "root". This could be dangerous.
Capturing on eth0
/* chdir */
 1.228396 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5
 1.229406 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 6)  Directory mode:0755 uid:0 gid:0
 4.445894 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5
 4.446916 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 16)  Directory mode:0755 uid:0 gid:0
 4.447099 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x43976ad5/foo
 4.448147 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 18), FH:0x3f9fd887     <=====
 10.228438 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5
 10.229432 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 31)  Directory mode:0755 uid:0 gid:0
 19.228321 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5
 19.229309 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 47)  Directory mode:0755 uid:0 gid:0
 28.228139 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5
 28.229112 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 70)  Directory mode:0755 uid:0 gid:0
/* getcwd */
 34.448796 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x3f9fd887/..
 34.450119 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 81), FH:0x42966b36
 34.450343 192.168.1.23 -> 192.168.1.136 NFS V3 READDIRPLUS Call, FH:0x42966b36
 34.452105 192.168.1.136 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 83) .. foo .
 34.452311 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x42966b36/..
 34.453464 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 85), FH:0xbc1b2900
 34.453648 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x42966b36/foo
 34.454677 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 87), FH:0x3e9ed964     <======

Regards,

Tomo

(2012年06月20日 16:28), Tomoaki Sato wrote:
> Avati,
>
> I've tried following:
> 1) 'esxcfg-nas -d gluster_nfs' at the ESXi host.
> 2) 'volume set bar nfs.enable-ino32 on' at the 192.168.1.136 host.
> 3) 'volume stop bar' and 'volume start bar' at the 192.168.1.136 host.
> 4) 'esxcfg-nas -a -o 192.168.1.136 -s /bar gluster_nfs' at the ESXi host.
>
> on the ESXi host:
>
> ~ # uname -m
> x86_64
> ~ # mkdir /vmfs/volumes/ef172a87-e5ae817f/after-enable-ino32-on
> ~ # ls -liR /vmfs/volumes/ef172a87-e5ae817f
> /vmfs/volumes/ef172a87-e5ae817f:
> -2118204814 drwxr-xr-x 1 root root 4096 Jun 20 07:13 after-enable-ino32-on
> 1205893126 drwxr-xr-x 1 root root 4096 Jun 20 07:08 baz
> -1291907235 drwx------ 1 root root 16384 Jun 6 23:41 lost+found
>
> /vmfs/volumes/ef172a87-e5ae817f/after-enable-ino32-on:
>
> /vmfs/volumes/ef172a87-e5ae817f/baz:
> -1374929331 drwxr-xr-x 1 root root 4096 Jun 19 06:41 foo
>
> /vmfs/volumes/ef172a87-e5ae817f/baz/foo:
>
> /vmfs/volumes/ef172a87-e5ae817f/lost+found:
> ~ # ./getcwd /vmfs/volumes/ef172a87-e5ae817f/after-enable-ino32-on
> getcwd: No such file or directory
> ~ #
>
> on the 192.168.1.136 host:
>
> # gluster volume info bar
>
> Volume Name: bar
> Type: Distribute
> Volume ID: b2d75589-8370-4528-ab4e-b543b3abdc3b
> Status: Started
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: bar-1-private:/mnt/brick
> Options Reconfigured:
> diagnostics.brick-log-level: TRACE
> diagnostics.client-log-level: TRACE
> nfs.enable-ino32: on
>
> please fine attached nfs.log5.
>
> Regards,
>
> Tomo
>
> (2012/06/20 16:11), Anand Avati wrote:
>> -1374929331 drwxr-xr-x 1 root root 4096 Jun 19 06:41 foo
>>
>> ...
>>
>> 2920037965 drwxr-xr-x 2 root root 4096 Jun 19 15:41 foo
>>
>>
>> Ouch!
>>
>> -1374929331 == (int32_t) 2920037965
>>
>> 'uname -m' from the ESXi host please! Is it a 32bit OS? Can you try 'gluster volume set bar nfs.enable-ino32 on' and retry?
>>
>> Avati
>


[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux