Hi, thanks for the answer. I'm out of the office until next week, but
when I come back, I'll try to do the tests and send you the info.
Thank you very much,
Omar
El 2015-11-23 21:18, bfields@xxxxxxxxxxxx escribió:
On Fri, Nov 20, 2015 at 12:04:49PM +0100, Omar Walid Llorente wrote:
Hi, I'm Omar Walid Llorente and I am a systems administrator at the
Politechnical University of Madrid (UPM), Spain. I write you in the
hope you can help us manage a problem that have discovered recently
about our new datastore architecture in our teaching labs. We have
created a gluster distributed volume that we reexport with NFS to
our lab clients via intermediate servers.
First of all thanks for all your work and sorry if this isn't
related with your package, but I think it has a good chance. I'll
try explain myself as short as possible.
As introduced previously, we have a problem exporting with
nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously
mounted with gluster-3.7.4 via fuse mount.
The problem is quite simple to reproduce and always repeatable: if a
file has read-only permissions for owner and user wants to copy it,
permissions problem arises:
cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied
-r--r--r-- 1 cdc admincdc 7 nov 3 2015 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 3 2015 kk.txt
cdc@client:~$
If the file permissions are not read-only, there is no problem:
cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod
644 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt
-rw-r--r-- 1 cdc admincdc 7 nov 3 2015 644.txt
-rw-r--r-- 1 cdc admincdc 7 nov 3 2015 kk.txt
cdc@client:~$
If we track it down with strace, the problem arises exactly when
fsync() is called from cp.
So, cp is calling fsync on kk.txt before closing it?
Of course, if we try this combination of commands in other
directories not mounted by nfs (local ones) or mounted with
samba/cifs or even mounted with nfs-ganesha (both fuse mounted with
gluster), this doen't happen. This problem doesn't happen either if
the nfs-kernel-server exports a directory not mounted with fuse (any
local one).
Please, tell me if this is the right place to post the probem and
where is it if this is not. Let me know if we can help you any way
to solve or test it (we've developed a small program in c that shows
exactly the same behaviour).
It'd be interesting to see exactly which NFS operation is failing.
You
should be able to do this by running wireshark to sniff the NFS
client/server traffic while running the reproducer, then reading
through
the traffic to find the failing operation.
I'm also curious:
LOGS ON SERVER SIDE (glusterfs mount logs):
[2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2015-11-20 10:51:53.874917] W [MSGID: 114031]
[client-rpc-fops.c:1298:client3_3_removexattr_cbk]
0-home-lab-3-client-0: remote operation failed [Permission denied]
[2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]
0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1
(Permission denied)
[2015-11-20 10:51:53.881389] W [MSGID: 114031]
[client-rpc-fops.c:1298:client3_3_removexattr_cbk]
0-home-lab-3-client-3: remote operation failed [Permission denied]
[2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk]
0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1
(Permission denied)
I wonder what this is about? Why would an fsync on the NFS client
result in a REMOVEXATTR on the server side?
--b.
[2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk]
0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1
(Permission denied)
[2015-11-20 10:51:53.883057] W [MSGID: 114031]
[client-rpc-fops.c:1298:client3_3_removexattr_cbk]
0-home-lab-3-client-3: remote operation failed [Permission denied]
[2015-11-20 10:51:53.884003] E [MSGID: 114031]
[client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
remote operation failed. Path: /cdc/kk.txt
(3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied]
[2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk]
0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission
denied)
[2015-11-20 10:51:53.885619] W [MSGID: 114031]
[client-rpc-fops.c:1298:client3_3_removexattr_cbk]
0-home-lab-3-client-3: remote operation failed [Permission denied]
[2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk]
0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1
(Permission denied)
[2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk]
0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1
(Permission denied)
[2015-11-20 10:51:53.887891] W [MSGID: 114031]
[client-rpc-fops.c:1298:client3_3_removexattr_cbk]
0-home-lab-3-client-3: remote operation failed [Permission denied]
(NOTE: We have more gluster brick logs but we don't know if are
relevant)
--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: omar@xxxxxxxxxx Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs"
in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
