Vijay, Thank you for your attention to this matter. I could not find strace command on the ESXi hosts. CentOS version of strace command says "strace+ ptrace(PTRACE_TRACEME, ...): Function not implemented" on the ESXi host. Please find attached log file. This log file is smaller than the previous version and covers only the 'getcwd' period. Regards, Tomo (2012/06/20 10:51), Vijay Bellur wrote:
On 06/18/2012 11:55 PM, Tomoaki Sato wrote:Vijay, please find attached test program and log file. test-code: $ cat getcwd.c #include <stdio.h> #include <unistd.h> int main(int argc, char* argv[]) { char cwd[1024]; if ( argc != 2 ) { printf("usage: %s dir\n", argv[0]); _exit(1); } if ( chdir(argv[1]) != 0 ) { perror("chdir"); _exit(1); } if ( getcwd(cwd, 1024) == NULL ) { perror("getcwd"); _exit(1); } printf("%s\n", getcwd(cwd, 1024)); _exit(0); } operations: ~ # ./getcwd /vmfs/volumes/1668aab4-a78e64bd/foo getcwd: No such file or directory ~ #Thanks for providing this test program. Can you please provide the output of strace -f ./getcwd /vmfs/volumes/1668aab4-a78e64bd/foo and the corresponding nfs log file? -Vijay
[2012-06-20 11:14:05.275468] T [rpcsvc.c:470:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 826 [2012-06-20 11:14:05.275523] T [rpcsvc-auth.c:305:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX [2012-06-20 11:14:05.275545] T [rpcsvc.c:382:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x3d7774f3, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3) from rpc-transport (socket.nfs-server) [2012-06-20 11:14:05.275570] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: localhost, uid: 0, gid: 0 [2012-06-20 11:14:05.275590] T [rpcsvc.c:211:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - LOOKUP [2012-06-20 11:14:05.275624] D [nfs3-helpers.c:1641:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: 3d7774f3, LOOKUP: args: FH: hashcount 2, exportid b2d75589-8370-4528-ab4e-b543b3abdc3b, gfid 3b2400dc-1951-426d-8420-6c44546f4ec3, name: .. [2012-06-20 11:14:05.275649] T [nfs3.c:1408:nfs3_lookup] 0-nfs-nfsv3: FH to Volume: bar [2012-06-20 11:14:05.275679] T [nfs3-helpers.c:3679:nfs3_fh_resolve_entry_hard] 0-nfs-nfsv3: FH hard resolution: gfid: 3b2400dc-1951-426d-8420-6c44546f4ec3 , entry: .. [2012-06-20 11:14:05.275710] T [nfs3-helpers.c:3687:nfs3_fh_resolve_entry_hard] 0-nfs-nfsv3: Entry needs lookup: /baz/foo/.. [2012-06-20 11:14:05.275729] T [nfs.c:437:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.275748] T [nfs.c:445:nfs_user_create] 0-nfs: gid: 0 [2012-06-20 11:14:05.275765] T [nfs-fops.c:335:nfs_fop_lookup] 0-nfs: Lookup: /baz/foo/.. [2012-06-20 11:14:05.275784] T [nfs-fops.c:131:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.275801] T [nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 0 [2012-06-20 11:14:05.275879] T [rpc-clnt.c:1310:rpc_clnt_record] 0-bar-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: [2012-06-20 11:14:05.275907] T [rpc-clnt.c:1190:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68 [2012-06-20 11:14:05.275977] T [rpc-clnt.c:1510:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11268x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (bar-client-0) [2012-06-20 11:14:05.276558] T [rpc-clnt.c:685:rpc_clnt_reply_init] 0-bar-client-0: received rpc message (RPC XID: 0x11268x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) from rpc-transport (bar-client-0) [2012-06-20 11:14:05.276611] D [nfs3-helpers.c:3458:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 3d7774f3, LOOKUP: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 3, exportid b2d75589-8370-4528-ab4e-b543b3abdc3b, gfid a4e1c22c-8d79-4b0b-b0a0-fff48b01ab4c [2012-06-20 11:14:05.276647] T [rpcsvc.c:1050:rpcsvc_submit_generic] 0-rpc-service: Tx message: 228 [2012-06-20 11:14:05.276669] T [rpcsvc.c:676:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 252, payload: 228, rpc hdr: 24 [2012-06-20 11:14:05.276714] T [rpcsvc.c:1087:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1031238899x, Program: NFS3, ProgVers: 3, Proc: 3) to rpc-transport (socket.nfs-server) [2012-06-20 11:14:05.276894] T [rpcsvc.c:470:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 826 [2012-06-20 11:14:05.276923] T [rpcsvc-auth.c:305:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX [2012-06-20 11:14:05.276942] T [rpcsvc.c:382:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x3d7774f4, Ver: 2, Program: 100003, ProgVers: 3, Proc: 17) from rpc-transport (socket.nfs-server) [2012-06-20 11:14:05.276991] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: localhost, uid: 0, gid: 0 [2012-06-20 11:14:05.277012] T [rpcsvc.c:211:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - READDIRPLUS [2012-06-20 11:14:05.277038] D [nfs3-helpers.c:3523:nfs3_log_readdir_call] 0-nfs-nfsv3: XID: 3d7774f4, READDIRPLUS: args: FH: hashcount 3, exportid b2d75589-8370-4528-ab4e-b543b3abdc3b, gfid a4e1c22c-8d79-4b0b-b0a0-fff48b01ab4c, dircount: 1008, maxcount: 8064 [2012-06-20 11:14:05.277059] T [nfs3.c:4310:nfs3_readdir] 0-nfs-nfsv3: FH to Volume: bar [2012-06-20 11:14:05.277079] T [nfs3-helpers.c:3726:nfs3_fh_resolve_inode] 0-nfs-nfsv3: FH needs inode resolution [2012-06-20 11:14:05.277097] T [nfs3-helpers.c:3535:nfs3_fh_resolve_inode_done] 0-nfs-nfsv3: FH inode resolved [2012-06-20 11:14:05.277121] T [nfs.c:437:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.277139] T [nfs.c:445:nfs_user_create] 0-nfs: gid: 0 [2012-06-20 11:14:05.277156] T [nfs-fops.c:610:nfs_fop_readdirp] 0-nfs: readdir [2012-06-20 11:14:05.277174] T [nfs-fops.c:131:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.277191] T [nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 0 [2012-06-20 11:14:05.277219] T [rpc-clnt.c:1310:rpc_clnt_record] 0-bar-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: [2012-06-20 11:14:05.277241] T [rpc-clnt.c:1190:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 [2012-06-20 11:14:05.277295] T [rpc-clnt.c:1510:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11269x Program: GlusterFS 3.1, ProgVers: 330, Proc: 40) to rpc-transport (bar-client-0) [2012-06-20 11:14:05.277740] T [rpc-clnt.c:685:rpc_clnt_reply_init] 0-bar-client-0: received rpc message (RPC XID: 0x11269x Program: GlusterFS 3.1, ProgVers: 330, Proc: 40) from rpc-transport (bar-client-0) [2012-06-20 11:14:05.277780] T [nfs.c:437:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.277799] T [nfs.c:445:nfs_user_create] 0-nfs: gid: 0 [2012-06-20 11:14:05.277816] T [nfs-fops.c:475:nfs_fop_fstat] 0-nfs: FStat [2012-06-20 11:14:05.277834] T [nfs-fops.c:131:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.277852] T [nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 0 [2012-06-20 11:14:05.277876] T [rpc-clnt.c:1310:rpc_clnt_record] 0-bar-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: [2012-06-20 11:14:05.277901] T [rpc-clnt.c:1190:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 [2012-06-20 11:14:05.277958] T [rpc-clnt.c:1510:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11270x Program: GlusterFS 3.1, ProgVers: 330, Proc: 25) to rpc-transport (bar-client-0) [2012-06-20 11:14:05.278326] T [rpc-clnt.c:685:rpc_clnt_reply_init] 0-bar-client-0: received rpc message (RPC XID: 0x11270x Program: GlusterFS 3.1, ProgVers: 330, Proc: 25) from rpc-transport (bar-client-0) [2012-06-20 11:14:05.278358] T [nfs3.c:4104:nfs3svc_readdir_fstat_cbk] 0-nfs-nfsv3: Reached end-of-directory [2012-06-20 11:14:05.278382] D [nfs3-helpers.c:3489:nfs3_log_readdirp_res] 0-nfs-nfsv3: XID: 3d7774f4, READDIRPLUS: NFS: 0(Call completed successfully.), POSIX: 0(Success), dircount: 1008, maxcount: 8064, cverf: 59641052, is_eof: 1 [2012-06-20 11:14:05.278405] T [nfs3-helpers.c:758:nfs3_fill_entryp3] 0-nfs-nfsv3: Entry: .., ino: 16777216 [2012-06-20 11:14:05.278426] T [nfs3-helpers.c:758:nfs3_fill_entryp3] 0-nfs-nfsv3: Entry: foo, ino: 2920037965 [2012-06-20 11:14:05.278445] T [nfs3-helpers.c:758:nfs3_fill_entryp3] 0-nfs-nfsv3: Entry: ., ino: 1205893126 [2012-06-20 11:14:05.278471] T [rpcsvc.c:1050:rpcsvc_submit_generic] 0-rpc-service: Tx message: 612 [2012-06-20 11:14:05.278493] T [rpcsvc.c:676:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 636, payload: 612, rpc hdr: 24 [2012-06-20 11:14:05.278534] T [rpcsvc.c:1087:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1031238900x, Program: NFS3, ProgVers: 3, Proc: 17) to rpc-transport (socket.nfs-server) [2012-06-20 11:14:05.278573] T [nfs3.c:447:nfs3_call_state_wipe] 0-nfs-nfsv3: fd 0x38e0cdc ref: 1 [2012-06-20 11:14:05.278753] T [rpcsvc.c:470:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 826 [2012-06-20 11:14:05.278782] T [rpcsvc-auth.c:305:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX [2012-06-20 11:14:05.278801] T [rpcsvc.c:382:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x3d7774f5, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3) from rpc-transport (socket.nfs-server) [2012-06-20 11:14:05.278821] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: localhost, uid: 0, gid: 0 [2012-06-20 11:14:05.278839] T [rpcsvc.c:211:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - LOOKUP [2012-06-20 11:14:05.278863] D [nfs3-helpers.c:1641:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: 3d7774f5, LOOKUP: args: FH: hashcount 3, exportid b2d75589-8370-4528-ab4e-b543b3abdc3b, gfid a4e1c22c-8d79-4b0b-b0a0-fff48b01ab4c, name: .. [2012-06-20 11:14:05.278884] T [nfs3.c:1408:nfs3_lookup] 0-nfs-nfsv3: FH to Volume: bar [2012-06-20 11:14:05.278906] T [nfs3-helpers.c:3679:nfs3_fh_resolve_entry_hard] 0-nfs-nfsv3: FH hard resolution: gfid: a4e1c22c-8d79-4b0b-b0a0-fff48b01ab4c , entry: .. [2012-06-20 11:14:05.278929] T [nfs3-helpers.c:3687:nfs3_fh_resolve_entry_hard] 0-nfs-nfsv3: Entry needs lookup: /baz/.. [2012-06-20 11:14:05.278947] T [nfs.c:437:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.278965] T [nfs.c:445:nfs_user_create] 0-nfs: gid: 0 [2012-06-20 11:14:05.278981] T [nfs-fops.c:335:nfs_fop_lookup] 0-nfs: Lookup: /baz/.. [2012-06-20 11:14:05.278999] T [nfs-fops.c:131:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.279016] T [nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 0 [2012-06-20 11:14:05.279070] T [rpc-clnt.c:1310:rpc_clnt_record] 0-bar-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: [2012-06-20 11:14:05.279094] T [rpc-clnt.c:1190:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68 [2012-06-20 11:14:05.279150] T [rpc-clnt.c:1510:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11271x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (bar-client-0) [2012-06-20 11:14:05.279557] T [rpc-clnt.c:685:rpc_clnt_reply_init] 0-bar-client-0: received rpc message (RPC XID: 0x11271x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) from rpc-transport (bar-client-0) [2012-06-20 11:14:05.279598] D [nfs3-helpers.c:3458:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 3d7774f5, LOOKUP: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 4, exportid b2d75589-8370-4528-ab4e-b543b3abdc3b, gfid 00000000-0000-0000-0000-000000000001 [2012-06-20 11:14:05.279626] T [rpcsvc.c:1050:rpcsvc_submit_generic] 0-rpc-service: Tx message: 228 [2012-06-20 11:14:05.279648] T [rpcsvc.c:676:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 252, payload: 228, rpc hdr: 24 [2012-06-20 11:14:05.279691] T [rpcsvc.c:1087:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1031238901x, Program: NFS3, ProgVers: 3, Proc: 3) to rpc-transport (socket.nfs-server) [2012-06-20 11:14:05.279938] T [rpcsvc.c:470:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 826 [2012-06-20 11:14:05.279967] T [rpcsvc-auth.c:305:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX [2012-06-20 11:14:05.279986] T [rpcsvc.c:382:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x3d7774f6, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3) from rpc-transport (socket.nfs-server) [2012-06-20 11:14:05.280006] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: localhost, uid: 0, gid: 0 [2012-06-20 11:14:05.280024] T [rpcsvc.c:211:rpcsvc_program_actor] 0-rpc-service: Actor found: NFS3 - LOOKUP [2012-06-20 11:14:05.280048] D [nfs3-helpers.c:1641:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: 3d7774f6, LOOKUP: args: FH: hashcount 3, exportid b2d75589-8370-4528-ab4e-b543b3abdc3b, gfid a4e1c22c-8d79-4b0b-b0a0-fff48b01ab4c, name: foo [2012-06-20 11:14:05.280080] T [nfs3.c:1408:nfs3_lookup] 0-nfs-nfsv3: FH to Volume: bar [2012-06-20 11:14:05.280103] T [nfs3-helpers.c:3679:nfs3_fh_resolve_entry_hard] 0-nfs-nfsv3: FH hard resolution: gfid: a4e1c22c-8d79-4b0b-b0a0-fff48b01ab4c , entry: foo [2012-06-20 11:14:05.280126] T [nfs.c:437:nfs_user_create] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.280144] T [nfs.c:445:nfs_user_create] 0-nfs: gid: 0 [2012-06-20 11:14:05.280160] T [nfs-fops.c:335:nfs_fop_lookup] 0-nfs: Lookup: /baz/foo [2012-06-20 11:14:05.280178] T [nfs-fops.c:131:nfs_create_frame] 0-nfs: uid: 0, gid 0, gids: 1 [2012-06-20 11:14:05.280195] T [nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 0 [2012-06-20 11:14:05.280249] T [rpc-clnt.c:1310:rpc_clnt_record] 0-bar-client-0: Auth Info: pid: 1, uid: 0, gid: 0, owner: [2012-06-20 11:14:05.280273] T [rpc-clnt.c:1190:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 88, rpc hdr: 68 [2012-06-20 11:14:05.280327] T [rpc-clnt.c:1510:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11272x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (bar-client-0) [2012-06-20 11:14:05.280668] T [rpc-clnt.c:685:rpc_clnt_reply_init] 0-bar-client-0: received rpc message (RPC XID: 0x11272x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) from rpc-transport (bar-client-0) [2012-06-20 11:14:05.280710] D [nfs3-helpers.c:3458:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 3d7774f6, LOOKUP: NFS: 0(Call completed successfully.), POSIX: 22(Invalid argument), FH: hashcount 4, exportid b2d75589-8370-4528-ab4e-b543b3abdc3b, gfid 3b2400dc-1951-426d-8420-6c44546f4ec3 [2012-06-20 11:14:05.280738] T [rpcsvc.c:1050:rpcsvc_submit_generic] 0-rpc-service: Tx message: 228 [2012-06-20 11:14:05.280760] T [rpcsvc.c:676:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 252, payload: 228, rpc hdr: 24 [2012-06-20 11:14:05.280801] T [rpcsvc.c:1087:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1031238902x, Program: NFS3, ProgVers: 3, Proc: 3) to rpc-transport (socket.nfs-server)
