Avati, tshark says ... FH values that the linux kernel NFS server returns stays constant for every LOOKUP 'foo' but, FH values that the GlusterFS(NFS) returns are non-constant. operaions at the ESXi host: ~ # ./getcwd /vmfs/volumes/94925201-78f190e0/foo ========= sleep 30 ================ /vmfs/volumes/94925201-78f190e0/foo ~ # tshark's output at the linux kernel NFS server: # tshark -i 2 -R nfs Running as user "root" and group "root". This could be dangerous. Capturing on br0 /* chdir */ 2.056680 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff 2.056990 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 13) Directory mode:0755 uid:0 gid:0 9.848666 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff 9.848767 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 60) Directory mode:0755 uid:0 gid:0 9.848966 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo 9.849049 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 62), FH:0xdb05b90a <===== 20.055508 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff 20.055702 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 103) Directory mode:0755 uid:0 gid:0 29.054939 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff 29.055180 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 132) Directory mode:0755 uid:0 gid:0 38.054338 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff 38.054583 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 151) Directory mode:0755 uid:0 gid:0 /* getcwd */ 39.849107 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0xdb05b90a/.. 39.849449 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 170), FH:0x1ffd38ff 39.849676 192.168.1.23 -> 192.168.1.254 NFS V3 READDIRPLUS Call, FH:0x1ffd38ff 39.849833 192.168.1.254 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 172) . .. foo 39.850071 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo 39.850149 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 174), FH:0xdb05b90a 39.850746 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0xdb05b90a/.. 39.850814 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 176), FH:0x1ffd38ff 39.851014 192.168.1.23 -> 192.168.1.254 NFS V3 READDIRPLUS Call, FH:0x1ffd38ff 39.851095 192.168.1.254 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 178) . .. foo 39.851329 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo 39.851438 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 180), FH:0xdb05b90a <===== operations at the ESXi host: ~ # ./getcwd /vmfs/volumes/ef172a87-e5ae817f/foo ========= sleep 30 ================ getcwd: No such file or directory ~ # tshark's output at the GlusterFS(NFS) server: # tshark -i 1 -R nfs Running as user "root" and group "root". This could be dangerous. Capturing on eth0 /* chdir */ 1.228396 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5 1.229406 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 6) Directory mode:0755 uid:0 gid:0 4.445894 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5 4.446916 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 16) Directory mode:0755 uid:0 gid:0 4.447099 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x43976ad5/foo 4.448147 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 18), FH:0x3f9fd887 <===== 10.228438 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5 10.229432 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 31) Directory mode:0755 uid:0 gid:0 19.228321 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5 19.229309 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 47) Directory mode:0755 uid:0 gid:0 28.228139 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5 28.229112 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 70) Directory mode:0755 uid:0 gid:0 /* getcwd */ 34.448796 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x3f9fd887/.. 34.450119 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 81), FH:0x42966b36 34.450343 192.168.1.23 -> 192.168.1.136 NFS V3 READDIRPLUS Call, FH:0x42966b36 34.452105 192.168.1.136 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 83) .. foo . 34.452311 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x42966b36/.. 34.453464 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 85), FH:0xbc1b2900 34.453648 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x42966b36/foo 34.454677 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 87), FH:0x3e9ed964 <====== Regards, Tomo (2012年06月20日 16:28), Tomoaki Sato wrote: > Avati, > > I've tried following: > 1) 'esxcfg-nas -d gluster_nfs' at the ESXi host. > 2) 'volume set bar nfs.enable-ino32 on' at the 192.168.1.136 host. > 3) 'volume stop bar' and 'volume start bar' at the 192.168.1.136 host. > 4) 'esxcfg-nas -a -o 192.168.1.136 -s /bar gluster_nfs' at the ESXi host. > > on the ESXi host: > > ~ # uname -m > x86_64 > ~ # mkdir /vmfs/volumes/ef172a87-e5ae817f/after-enable-ino32-on > ~ # ls -liR /vmfs/volumes/ef172a87-e5ae817f > /vmfs/volumes/ef172a87-e5ae817f: > -2118204814 drwxr-xr-x 1 root root 4096 Jun 20 07:13 after-enable-ino32-on > 1205893126 drwxr-xr-x 1 root root 4096 Jun 20 07:08 baz > -1291907235 drwx------ 1 root root 16384 Jun 6 23:41 lost+found > > /vmfs/volumes/ef172a87-e5ae817f/after-enable-ino32-on: > > /vmfs/volumes/ef172a87-e5ae817f/baz: > -1374929331 drwxr-xr-x 1 root root 4096 Jun 19 06:41 foo > > /vmfs/volumes/ef172a87-e5ae817f/baz/foo: > > /vmfs/volumes/ef172a87-e5ae817f/lost+found: > ~ # ./getcwd /vmfs/volumes/ef172a87-e5ae817f/after-enable-ino32-on > getcwd: No such file or directory > ~ # > > on the 192.168.1.136 host: > > # gluster volume info bar > > Volume Name: bar > Type: Distribute > Volume ID: b2d75589-8370-4528-ab4e-b543b3abdc3b > Status: Started > Number of Bricks: 1 > Transport-type: tcp > Bricks: > Brick1: bar-1-private:/mnt/brick > Options Reconfigured: > diagnostics.brick-log-level: TRACE > diagnostics.client-log-level: TRACE > nfs.enable-ino32: on > > please fine attached nfs.log5. > > Regards, > > Tomo > > (2012/06/20 16:11), Anand Avati wrote: >> -1374929331 drwxr-xr-x 1 root root 4096 Jun 19 06:41 foo >> >> ... >> >> 2920037965 drwxr-xr-x 2 root root 4096 Jun 19 15:41 foo >> >> >> Ouch! >> >> -1374929331 == (int32_t) 2920037965 >> >> 'uname -m' from the ESXi host please! Is it a 32bit OS? Can you try 'gluster volume set bar nfs.enable-ino32 on' and retry? >> >> Avati >
