On 10/7/20 7:38 PM, Ian Kent wrote:
On Wed, 2020-10-07 at 18:39 -0500, Patrick Goetz wrote:I have a rather puzzling timing issue which I hope someone can shed some light on. We are using a centralized authentication service (Microsoft AD) with authorization restricted to members of a particular security group. Home directories are housed on an NFS server (Ubuntu 20.04) and automounted to a collection of compute workstations (currently a mixture of Ubuntu 18.04 and 20.04 systems). Because there is a reasonably high level of turnover (grad students and postdocs) and because I don't want people to login directly to the NFS server any more (hence can't use pam_mkhomedir without compromising security by setting no_root_squash on the NFS server) I decided to try and automate the creation of home directories a different way. To wit, I run a pam_exec script in the session configuration: /etc/pam.d/common-session: ------------------------- session optional pam_exec.so log=/tmp/pam_exec.log /usr/local/sbin/make-nfs_homedir.sh Which runs this script: ----------------------------------------------- #!/bin/bash if [ $PAM_TYPE = "open_session" ]; then UTEID="$(cut -d'@' -f1 <<<"$PAM_USER")" USERGID=$(id -g $UTEID) PROC=$$ TEMPDIR=$UTEID$PROC if (($USERGID > 100000)); then mkdir /tmp/$TEMPDIR mount helios:/home /tmp/$TEMPDIR if [ ! -d /tmp/$TEMPDIR/$UTEID ]; then mkdir /tmp/$TEMPDIR/temp/$UTEID sleep 70 fi cd / umount /tmp/$TEMPDIR rmdir /tmp/$TEMPDIR fi fi ----------------------------------------------- A brief explanation is in order. nfs_server:/home is mounted to /tmp/$uid$pid on the client workstation and the script checks to see if the user's home directory already exists. If so do nothing. If not, create it in /home/temp. On the NFS server, the /home/temp directory has 1777 permissions, so anyone can write to it, including the nobody user from the workstations. Now for some systemd black magic. On the NFS server there is a systemd path unit file and accompanying service file. Whenever a directory is created in /home/temp, there's an NFS server script which moves it to /home and sets the appropriate user permissions. This process is nearly instantaneous; i.e. if I type `mkdir stuff` in /home/temp on the NFS server, I don't have time to type `ls` before the directory is moved to /home with appropriate user permissions set. One last detail: /etc/auto.home on the workstation: * -tcp,vers=4.2 helios.biosci.utexas.edu:/home/& OK, now for the puzzling part. Notice the sleep 70 directive in the client side pam_exec script. I've fiddled around with this quite a bit and anything less results in a message of Could not chdir to home directory /home/pgoetz: No such file or directory on first log in. I've checked, and indeed the directory is not mounted. If you hang around in / long enough (roughly 70 seconds) you can eventually cd to your automounted home directory. So, Question: Why the delay? The home directory on the NFS server is created nearly instantly, so it can't be that. And as mentioned, if I sleep for say, 60 seconds, the home directory isn't immediately accessible on login, although one can cd to it a few seconds later. I can't fathom why the required delay.It sounds like the NFS client isn't seeing the attribute change of the server directory /home. IIUC basically there are two things to worry about, first that the VFS path walk on the client actually results calls into the NFS client code (it might not for various reasons, like the VFS doesn't think the directory dentry needs revalidation), and second, the attributes used by the NFS client to detect staleness aren't being changed by the server operations being done so revalidation isn't done by the NFS client. Not sure what to do about it short of diving into the NFS client kernel code. Ian
Thanks, Ian. I just wanted to make sure I'm not overlooking something dumb. Looking at the NFS client code is not warranted, given the minor inconvenience this causes. While this is obviously a bit of an edge case, since there are other ways to get this done, given the ongoing shift to dynamic containerized computing these kinds of issues will continue to gain prominence though.
