cephfs - inconsistent nfs and samba directory listings

[Mike Carlson <mike@xxxxxxxxxxxx>]

Hello.

Since we upgraded to Infernalis last, we have noticed a severe problem with cephfs when we have it shared over Samba and NFS

Directory listings are showing an inconsistent view of the files:

$ ls /lts-mon/BD/xmlExport/ | wc -l

     100

$ sudo umount /lts-mon

$ sudo mount /lts-mon

$ ls /lts-mon/BD/xmlExport/ | wc -l

    3507

The only work around I have found is un-mounting and re-mounting the nfs share, that seems to clear it up

Same with samba, I'd post it here but its thousands of lines. I can add additional details on request.

This happened after our upgrade to infernalis. Is it possible the MDS is in an inconsistent state?

We have cephfs mounted on a server using the built in cephfs kernel module:

lts-mon:6789:/ /ceph ceph name=admin,secretfile=/etc/ceph/admin.secret,noauto,_netdev

We are running all of our ceph nodes on ubuntu 14.04 LTS. Samba is up to date, 4.1.6, and we export nfsv3 to linux and freebsd systems. All seem to exhibit the same behavior.

system info:

# uname -a

Linux lts-osd1 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@lts-osd1:~# lsb

lsblk        lsb_release  

root@lts-osd1:~# lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 14.04.3 LTS

Release: 14.04

Codename: trusty

package info:

 # dpkg -l|grep ceph

ii  ceph                                 9.2.0-1trusty                    amd64        distributed storage and file system

ii  ceph-common                          9.2.0-1trusty                    amd64        common utilities to mount and interact with a ceph storage cluster

ii  ceph-fs-common                       9.2.0-1trusty                    amd64        common utilities to mount and interact with a ceph file system

ii  ceph-mds                             9.2.0-1trusty                    amd64        metadata server for the ceph distributed file system

ii  libcephfs1                           9.2.0-1trusty                    amd64        Ceph distributed file system client library

ii  python-ceph                          9.2.0-1trusty                    amd64        Meta-package for python libraries for the Ceph libraries

ii  python-cephfs                        9.2.0-1trusty                    amd64        Python libraries for the Ceph libcephfs library

What is interesting, is a directory or file will not show up in a listing, however, if we directly access the file, it shows up in that instance:

# ls -al |grep SCHOOL

# ls -alnd SCHOOL667055

drwxrwsr-x  1 21695  21183  2962751438 Jan 13 09:33 SCHOOL667055

Any tips are appreciated!

Thanks,

Mike C

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com