Hi Greg
Thanks for replying. Answer inline.
Dear cephfsers :-)
We saw some weirdness in cephfs that we do not understand.
We were helping some user which complained that her batch system job outputs
were not produced in cephfs.
Please note that we are using ceph-fuse (jewel 10.2.2) as client
We log in into the machine where her jobs run, and saw the following
behavior:
# ls /coepp/cephfs/mel/user/foo/bar/stuff
ls: cannot access '/coepp/cephfs/mel/user/foo/bar/stuff': No such file or
directory
If we went back 1 directory, still No such file
# ls /coepp/cephfs/mel/user/foo/bar
ls: cannot access '/coepp/cephfs/mel/user/foo/bar': No such file or
directory
But if I did an ls in the user directory it was fine
# ls /coepp/cephfs/mel/user
....
And then trying to ls to the directories which failed previous worked fine
It seems like a cache issue and I wonder if there is a way to mitigate it.
It is also worthwhile to mention that this seems to happen while we are
adding a new storage server to the underlying ceph infrastructure, so there
was some data movement happening in the background.
Any suggestion on how to mitigate it?
If you're really using 10.2.2 and not something earlier, I don't think
this is a bug we've heard about. It sounds like you could work around
it by dropping caches or listing down from the root gratuitously, but
otherwise we'll need to do some debugging. Can you narrow in on what
makes this user's workload different from the others? Did you try
doing any tracing to see where the ENOENT was coming from?
Really using 10.2.2 everywhere.
To debug it a bit further we have to wait for the next time it happens.
Than we can attach strace to the ceph-fuse process and get the
information which you are asking for.
Relative to the user workload, there is nothing special happening in
those directories. It is just a directory used to store logs (stderr and
stdout) from the batch system jobs.
We were thinking if setting
fuse_disable_pagecache = true
would actually solve the problem. In this way you force ceph-fuse to
read directly from osds, right?!
We understand about the performance issues that it might imply but we
are more concerned in having data coherence in the client.
Thoughts?
Cheers
--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW 2006
T: +61 2 93511937
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
