Here is another problem with an inaccessible file. Consider this problem
on a 2x2 replicaed/rdistributed setup:
client# ls /gfs/usr/src/tools/compat/obj/ |grep strlcat.lo
strlcat.lo
client# ls /gfs/usr/src/tools/compat/obj/strlcat.lo
ls: /gfs/usr/src/tools/compat/obj/strlcat.lo: Socket is not connected
client log:
[2011-07-27 06:14:37.684558] I [client3_1-fops.c:411:client3_1_stat_cbk]
0-gfs-client-1: remote operation failed: No such file or directory
[2011-07-27 06:14:37.684716] W [client3_1-fops.c:2656:client3_1_stat]
0-gfs-client-0: failed to send the fop Socket is not connected
[2011-07-27 06:14:37.690212] I [client3_1-fops.c:411:client3_1_stat_cbk]
0-gfs-client-1: remote operation failed: No such file or directory
[2011-07-27 06:14:37.690336] W [client3_1-fops.c:2656:client3_1_stat]
0-gfs-client-0: failed to send the fop Socket is not connected
The "Socket is not connected" message is there because one of the servers
is down. That should work anyway, right?
On the running server, log for brick /export/wd1a
[2011-07-27 06:14:37.685180] I [server3_1-fops.c:1492:server_stat_cbk]
0-gfs-server: 5989901: STAT /usr/src/tools/compat/obj/strlcat.lo (0)
==> -1 (No such file or directory)
[2011-07-27 06:14:37.690954] I [server3_1-fops.c:1492:server_stat_cbk]
0-gfs-server: 5989902: STAT /usr/src/tools/compat/obj/strlcat.lo (0)
==> -1 (No such file or directory)
The file exists on the server and the gfid is safe:
server# ls -l /export/*/usr/src/tools/compat/obj/strlcat.lo
-rw-r--r-- 1 root wheel 595 Jul 22 09:20
/export/wd1a/usr/src/tools/compat/obj/strlcat.lo
---------T 1 root wheel 0 Jul 22 09:20
/export/wd3a/usr/src/tools/compat/obj/strlcat.lo
server# getextattr -x trusted.gfid /export/*/usr/src/tools/compat/obj/strlcat.lo
/export/wd1a/usr/src/tools/compat/obj/strlcat.lo
000 56 e9 57 ce f4 9a 46 08 aa b0 55 fa 2f 5d 13 d3 V.W...F...U./]..
/export/wd3a/usr/src/tools/compat/obj/strlcat.lo
000 56 e9 57 ce f4 9a 46 08 aa b0 55 fa 2f 5d 13 d3 V.W...F...U./]..
Here is the kernel trace on the server for brick /export/wd1a
490 4 glusterfsd CALL __lstat30(0xb9fff760,0xb9fff6b4)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd RET __lstat30 0
490 4 glusterfsd CALL lgetxattr(0xb9fff760,0xbba4c073,0xb9fff718,0x10)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd MISC xattr-name:
000 646966672e64657473757274 trusted.gfid
490 4 glusterfsd MISC xattr-val:
000 d3135d2ffa55b0aa08469af4ce57e956 V.W...F...U./]..
490 4 glusterfsd RET lgetxattr 16/0x10
490 4 glusterfsd CALL __lstat30(0xb9fff760,0xb9fff608)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd RET __lstat30 0
490 4 glusterfsd CALL lgetxattr(0xb9fff760,0xbba4c073,0xb9fff66c,0x10)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd MISC xattr-name:
000 646966672e64657473757274 trusted.gfid
490 4 glusterfsd MISC xattr-val:
000 d3135d2ffa55b0aa08469af4ce57e956 V.W...F...U./]..
490 4 glusterfsd RET lgetxattr 16/0x10
490 4 glusterfsd CALL __lstat30(0xb9fff760,0xb9fff6cc)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd RET __lstat30 0
490 4 glusterfsd CALL lgetxattr(0xb9fff760,0xbba4c073,0xb9fff66c,0x10)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd MISC xattr-name:
000 646966672e64657473757274 trusted.gfid
490 4 glusterfsd MISC xattr-val:
000 d3135d2ffa55b0aa08469af4ce57e956 V.W...F...U./]..
490 4 glusterfsd RET lgetxattr 16/0x10
490 4 glusterfsd CALL lgetxattr(0xb9fff760,0xb97031a0,0,0)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd MISC xattr-name:
000 6c63615f7869736f702e6d6574737973 system.posix_acl
010 746c75616665645f _default
490 4 glusterfsd RET lgetxattr -1 errno 93 Attribute not found
490 4 glusterfsd CALL lgetxattr(0xb9fff760,0xb9703100,0,0)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
490 4 glusterfsd MISC xattr-name:
000 6c63615f7869736f702e6d6574737973 system.posix_acl
010 7373656363615f _access
490 4 glusterfsd RET lgetxattr -1 errno 93 Attribute not found
490 4 glusterfsd CALL __lstat30(0xb970c040,0xb9fff6cc)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj"
490 4 glusterfsd RET __lstat30 0
490 4 glusterfsd CALL lgetxattr(0xb970c040,0xbba4c073,0xb9fff66c,0x10)
490 4 glusterfsd NAMI "/export/wd1a/usr/src/tools/compat/obj"
490 4 glusterfsd MISC xattr-name:
000 646966672e64657473757274 trusted.gfid
490 4 glusterfsd MISC xattr-val:
000 7eef14d5ffdb52a4aa46a593fbf50b80 ......F..R.....~
490 4 glusterfsd RET lgetxattr 16/0x10
"
(... and then the logged message )
I tried removing the posix-acl xlator but it does not change anything.
Any idea of what is wrong here?
--
Emmanuel Dreyfus
manu@xxxxxxxxxx
