On 10/7/20 7:38 PM, Ian Kent wrote:
On Wed, 2020-10-07 at 18:39 -0500, Patrick Goetz wrote:
I have a rather puzzling timing issue which I hope someone can shed
some
light on. We are using a centralized authentication service
(Microsoft
AD) with authorization restricted to members of a particular
security
group. Home directories are housed on an NFS server (Ubuntu 20.04)
and
automounted to a collection of compute workstations (currently a
mixture
of Ubuntu 18.04 and 20.04 systems).
Because there is a reasonably high level of turnover (grad students
and
postdocs) and because I don't want people to login directly to the
NFS
server any more (hence can't use pam_mkhomedir without compromising
security by setting no_root_squash on the NFS server) I decided to
try
and automate the creation of home directories a different way. To
wit, I
run a pam_exec script in the session configuration:
/etc/pam.d/common-session:
-------------------------
session optional pam_exec.so log=/tmp/pam_exec.log
/usr/local/sbin/make-nfs_homedir.sh
Which runs this script:
-----------------------------------------------
#!/bin/bash
if [ $PAM_TYPE = "open_session" ]; then
UTEID="$(cut -d'@' -f1 <<<"$PAM_USER")"
USERGID=$(id -g $UTEID)
PROC=$$
TEMPDIR=$UTEID$PROC
if (($USERGID > 100000)); then
mkdir /tmp/$TEMPDIR
mount helios:/home /tmp/$TEMPDIR
if [ ! -d /tmp/$TEMPDIR/$UTEID ]; then
mkdir /tmp/$TEMPDIR/temp/$UTEID
sleep 70
fi
cd /
umount /tmp/$TEMPDIR
rmdir /tmp/$TEMPDIR
fi
fi
-----------------------------------------------
A brief explanation is in order. nfs_server:/home is mounted to
/tmp/$uid$pid on the client workstation and the script checks to see
if
the user's home directory already exists. If so do nothing. If
not,
create it in /home/temp. On the NFS server, the /home/temp
directory
has 1777 permissions, so anyone can write to it, including the
nobody
user from the workstations.
Now for some systemd black magic. On the NFS server there is a
systemd
path unit file and accompanying service file. Whenever a directory
is
created in /home/temp, there's an NFS server script which moves it
to
/home and sets the appropriate user permissions. This process is
nearly
instantaneous; i.e. if I type `mkdir stuff` in /home/temp on the NFS
server, I don't have time to type `ls` before the directory is moved
to
/home with appropriate user permissions set.
One last detail: /etc/auto.home on the workstation:
* -tcp,vers=4.2 helios.biosci.utexas.edu:/home/&
OK, now for the puzzling part. Notice the
sleep 70
directive in the client side pam_exec script. I've fiddled around
with
this quite a bit and anything less results in a message of
Could not chdir to home directory /home/pgoetz: No such file or
directory
on first log in. I've checked, and indeed the directory is not
mounted.
If you hang around in / long enough (roughly 70 seconds) you can
eventually cd to your automounted home directory.
So, Question: Why the delay? The home directory on the NFS server
is
created nearly instantly, so it can't be that. And as mentioned, if
I
sleep for say, 60 seconds, the home directory isn't immediately
accessible on login, although one can cd to it a few seconds
later. I
can't fathom why the required delay.
It sounds like the NFS client isn't seeing the attribute change of
the server directory /home.
IIUC basically there are two things to worry about, first that the
VFS path walk on the client actually results calls into the NFS
client code (it might not for various reasons, like the VFS doesn't
think the directory dentry needs revalidation), and second, the
attributes used by the NFS client to detect staleness aren't being
changed by the server operations being done so revalidation isn't
done by the NFS client.
Not sure what to do about it short of diving into the NFS client
kernel code.
Ian
Thanks, Ian. I just wanted to make sure I'm not overlooking something
dumb. Looking at the NFS client code is not warranted, given the minor
inconvenience this causes. While this is obviously a bit of an edge
case, since there are other ways to get this done, given the ongoing
shift to dynamic containerized computing these kinds of issues will
continue to gain prominence though.