On Sat, 2018-11-03 at 10:30 +0530, Malahal Naineni wrote: > > Why does your server need to have multiple filehandles refer to the > > same file > > Trond, the first thing I want to understand is that this server's > multiple filehandles referring to the same file/directory is the > reason for the client's behavior described in this email. I am pretty > sure it is, but want a confirmation! We treat always different filehandles as if they refer to different files. It has long been the case that snapshots from several vendors are encoded to look like the same file (same fileid + same fsid) and differing only by filehandle. If we were to try to consolidate those inodes we would end up corrupting application data. Note that this should be consistent with NFSv4, where the state model requires the client to treat state that belongs to different filehandles as if it were referring to different files. The client is allowed to (but not required to) consolidate the file data caches in these cases, however it may not consolidate file delegation, layout, share or byte range locking state. Note also that the Linux VFS's dcache model does not at all support hard linked directories, nor does it really allow for aliased dentries. > Ganesha by design uses one file handle on big endian systems and > another one on little endian systems. If a mixed cluster is going to > serve the same file system, then the same file will have at least two > different file handles. This is probably done to avoid needless > host2net and net2host integer conversions. The Linux client will not treat those filehandles as if they refer to the same inode for the reasons stated above. I understand your performance motivation, but backward compatibility with existing setups must come first. Note that if each filesystem is served from only one server, then everything will work fine, It is when you start to play games with clustering IP addresses or similar that things break, and as I said, the NFSv4 protocol would be a horrible mess for this case. > Also, how do servers deal with version changes in file handle? Looks > like live migrations will be broken when there is a file handle > version change? Sure, but again no existing servers are using that feature and nobody so far has volunteered any interest in working on it. > Regards, Malahal. > > On Sat, Nov 3, 2018 at 1:56 AM Trond Myklebust < > trondmy@xxxxxxxxxxxxxxx> wrote: > > On Fri, 2018-11-02 at 21:24 +0530, Malahal Naineni wrote: > > > Ben, NFSv3 RFC1813.txt states: "If two file handles from the same > > > server are equal, they must refer to the same file, but if > > > they are > > > not equal, no conclusions can be drawn." Ganesha does return same > > > fileid here (inode). > > > > > > In NFSv4, they have introduced "unique_handles" attribute. I > > > don't > > > see > > > Linux NFS client using this at all though. > > > > Why does your server need to have multiple filehandles refer to the > > same file, and why do you expect clients to support this? > > > > Yes, the spec allows it, but that's not a sufficient reason. > > > > > Regards, Malahal. > > > On Fri, Nov 2, 2018 at 4:35 PM Benjamin Coddington < > > > bcodding@xxxxxxxxxx> wrote: > > > > On 2 Nov 2018, at 1:26, Malahal Naineni wrote: > > > > > > > > > Hi All, we are using NFS-Ganesha with Linux NFS clients. The > > > > > client's > > > > > shell reports the following. Based on lsof, the directory is > > > > > marked > > > > > deleted. "cd to ROOT and cd to the same home directory fixes > > > > > the > > > > > issue. The client behaves as though the directory is deleted > > > > > and > > > > > recreated! Our NFS-Ganesha server implementation uses > > > > > multiple > > > > > file > > > > > handles that point to the same object. NFS spec says this > > > > > should > > > > > be > > > > > fine, but Linux NFS seems to be broken in this regard. > > > > > tcpdump > > > > > does > > > > > indicate file handle change (note that all file handles are > > > > > permanent, > > > > > meaning they are valid at the server any time) around this > > > > > issue > > > > > time. > > > > > > > > > > "shell-init: error retrieving current directory: getcwd: > > > > > cannot > > > > > access > > > > > parent directories: No such file or directory" > > > > > sh 112544 malahal cwd DIR > > > > > 0,67 > > > > > 65536 45605209 /home/malahal (deleted) > > > > > (10.120.154.42:/nfs/malahal-export/) > > > > > > > > > > Function nfs_prime_dcache() seems to invalidate the dcache > > > > > entry > > > > > if > > > > > nfs_same_file() returns false. nfs_same_file() does seem to > > > > > return > > > > > false with the following change, if I read it correctly, if > > > > > there > > > > > is a > > > > > file handle change. Can this be the source of my issue? It > > > > > seems > > > > > that > > > > > the client should do this only if the file handle is NOT > > > > > valid > > > > > (e.g. > > > > > if it gets ESTALE), right? > > > > > > > > > > The following commit seems to assume that the objects are > > > > > different if > > > > > they have different file handles! > > > > > commit 7dc72d5f7a0ec97a53e126c46e2cbd2560757955 > > > > > Author: Trond Myklebust <trond.myklebust@xxxxxxxxxxxxxxx> > > > > > Date: Thu Sep 22 13:38:52 2016 -0400 > > > > > > > > > > NFS: Fix inode corruption in nfs_prime_dcache() > > > > > > > > My understanding is that for NFSv3 we have to assume that > > > > distinct > > > > filehandles are distinct objects, but maybe I'm wrong about > > > > this. > > > > > > > > For NFSv4.x, we can follow the guidance in RFCs 5661 or 7530 > > > > section 10.3.4 > > > > to determine if the differing filehandles are the same object, > > > > specifically > > > > the fileid recommended attribute needs to be implemented. Is > > > > Ganesha > > > > returning the same fileid for both filehandles? > > > > > > > > Ben > > -- > > Trond Myklebust > > CTO, Hammerspace Inc > > 4300 El Camino Real, Suite 105 > > Los Altos, CA 94022 > > www.hammer.space > > > > -- Trond Myklebust Linux NFS client maintainer, Hammerspace trond.myklebust@xxxxxxxxxxxxxxx
