On Thu, Apr 26, 2018 at 10:00 PM, Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: > Dear Cephalopodians, > > just now that our Ceph cluster is under high I/O load, we get user reports of files not being seen on some clients, > but somehow showing up after forcing a stat() syscall. > > For example, one user had added several files to a directory via an NFS client attached to nfs-ganesha (which uses libcephfs), > and afterwards, all other nfs-ganesha servers saw it, and 44 of our Fuse-clients - > but one single client still saw the old contents of the directory, i.e. the files seemed missing(!). > This happened both when using "ls" on the directory or when trying to access the non-existent files directly. > > I could confirm this observation also in a fresh login shell on the machine. > > Then, on the "broken" client, I entered in the directory which seemed to contain only the "old" content, and I created a new file in there. > This worked fine, and all other clients saw the file immediately. > Also on the broken client, metadata was now updated and all other files appeared - i.e. everything was "in sync" again. > > There's nothing in the ceph-logs of our MDS, or in the syslogs of the client machine / MDS. > > > Another user observed the same, but not explicitly limited to one machine (it seems random). > He now uses a "stat" on the file he expects to exist (but which is not seen with "ls"). > The stat returns "No such file", a subsequent "ls" then however lists the file, and it can be accessed normally. > > This feels like something is messed up concerning the client caps - these are all 12.2.4 Fuse clients. > > Any ideas how to find the cause? > It only happens since recently, and under high I/O load with many metadata operations. > Sounds like bug in readdir cache. Could you try the attached patch. Regards Yan, Zheng > Cheers, > Oliver > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
Attachment:
readdir.patch
Description: Binary data
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
