ceph-fuse (jewel 10.2.2): No such file or directory issues

[Goncalo Borges <goncalo.borges@xxxxxxxxxxxxx>]

Dear cephfsers :-)

We saw some weirdness in cephfs that we do not understand.

We were helping some user which complained that her batch system job outputs were not produced in cephfs.

Please note that we are using ceph-fuse (jewel 10.2.2) as client

We log in into the machine where her jobs run, and saw the following behavior:

# ls /coepp/cephfs/mel/user/foo/bar/stuff

ls: cannot access '/coepp/cephfs/mel/user/foo/bar/stuff': No such file or directory

If we went back 1 directory, still No such file

# ls /coepp/cephfs/mel/user/foo/bar

ls: cannot access '/coepp/cephfs/mel/user/foo/bar': No such file or directory

But if I did an ls in the user directory it was fine

# ls /coepp/cephfs/mel/user

....

And then trying to ls to the directories which failed previous worked fine

It seems like a cache issue and I wonder if there is a way to mitigate it.

It is also worthwhile to mention that this seems to happen while we are adding a new storage server to the underlying ceph infrastructure, so there was some data movement happening in the background.

Any suggestion on how to mitigate it?

Cheers
Goncalo and Sean

-- 
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW  2006
T: +61 2 93511937

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com