EACCES race when opening just-created file

[Noah Misch <noah@xxxxxxxxxxxx>]

With the Linux kernel as both NFS client and server, I'm seeing a race
condition where open() can return EACCES for a just-created file.  Is this
expected behavior?  POSIX implicitly allows it, but I failed to locate any
discussion of it.  I have attached a test program; here, it witnesses the race
for roughly a dozen of the 4096 files it creates (different files each time).
One router separates client from server, with overall ping times as follows:

100 packets transmitted, 100 received, 0% packet loss, time 99039ms
rtt min/avg/max/mdev = 0.173/0.259/4.447/0.422 ms

This server has three similar clients, each of which witnesses the race.  I
did not reproduce this in a different NFS setup where client and server were
virtual machines on the same box.  Those are the only two environments I have
tested so far.

-- Server Details
[nm@gcc45 0:0 00:36:42 ~]$ uname -a
Linux gcc45 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) i686 GNU/Linux
[nm@gcc45 0:0 00:36:37 ~]$ cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
/home   gcc24.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
/home   gcc23.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
/home   gcc22.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)

-- Client Details
[nm@erpro8-fsf3 0:1 00:37:28 nfs]$ uname -a
Linux erpro8-fsf3 4.1.4 #1 SMP PREEMPT Mon Aug 3 14:22:54 PDT 2015 mips64 GNU/Linux
[nm@erpro8-fsf3 0:1 00:37:49 nfs]$ mount|grep nfs
gcc45.tetaneutral.net:/home on /home type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2a03:7220:8083:9d00::1,local_lock=none,addr=2a03:7220:8081:8100::1)

Thanks,
nm

/*
 * Demonstrates open() returning EACCES for a recently-created file on NFS.  One
 * thread creates files nfs0 .. nfs4095 in the CWD, and another thread
 * continually attempts to open(O_RDWR) the latest file.  Prints a message
 * whenever the second thread sees open() fail in a way other than ENOENT.
 * Affected NFS configurations will print such messages for a selection of the
 * files, and immune configurations will print nothing.
 *
 * I abstracted this test case from a malfunction in the PostgreSQL database,
 * where one process opened a file just created in another process.  (See
 * src/interfaces/ecpg cases thread/thread and thread/prep.)
 */

#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <pthread.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

/* no reader/writer synchronization; occasional disagreement is okay */
static int pos;

static void *
creator_task(void *arg)
{
    for (pos = 0; pos < (1 << 12); ++pos)
    {
	char buf[128];
	int fd;

	sprintf(buf, "nfs%d", pos);
	fd = open(buf, O_RDWR | O_CREAT | O_EXCL, 0600);
	if (fd < 0)
	    printf("could not create %s: %s\n", buf, strerror(errno));
	else if (close(fd) != 0)
	    printf("could not close %s after creation: %s\n",
		   buf, strerror(errno));
    }

    pos = -1;

    return NULL;
}

static void *
tester_task(void *arg)
{
    while (pos >= 0)
    {
	char buf[128];
	int fd;

	sprintf(buf, "nfs%d", pos);
	fd = open(buf, O_RDWR);
	if (fd < 0)
	{
	    if (errno != ENOENT)
		printf("could not open %s: %s\n", buf, strerror(errno));
	}
	else if (close(fd) != 0)
	    printf("could not close %s: %s\n", buf, strerror(errno));
    }

    return NULL;
}

int
main(int argc, char **argv)
{
    pthread_t creator;

    pthread_create(&creator, NULL, creator_task, 0);
    tester_task(NULL);
    pthread_join(creator, NULL);

    return 0;
}