Hey Greg,
The inconsistent view is only over nfs/smb on top of our /ceph mount.
When I look directly on the /ceph mount (which is using the cephfs kernel module), everything looks fine
It is possible that this issue just went unnoticed, and it only being a infernalis problem is just a red herring. With that, it is oddly coincidental that we just started seeing issues.
On Wed, Jan 13, 2016 at 11:30 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
On Wed, Jan 13, 2016 at 11:24 AM, Mike Carlson <mike@xxxxxxxxxxxx> wrote:
> Hello.
>
> Since we upgraded to Infernalis last, we have noticed a severe problem with
> cephfs when we have it shared over Samba and NFS
>
> Directory listings are showing an inconsistent view of the files:
>
>
> $ ls /lts-mon/BD/xmlExport/ | wc -l
> 100
> $ sudo umount /lts-mon
> $ sudo mount /lts-mon
> $ ls /lts-mon/BD/xmlExport/ | wc -l
> 3507
>
>
> The only work around I have found is un-mounting and re-mounting the nfs
> share, that seems to clear it up
> Same with samba, I'd post it here but its thousands of lines. I can add
> additional details on request.
>
> This happened after our upgrade to infernalis. Is it possible the MDS is in
> an inconsistent state?
So this didn't happen to you until after you upgraded? Are you seeing
missing files when looking at cephfs directly, or only over the
NFS/Samba re-exports? Are you also sharing Samba by re-exporting the
kernel cephfs mount?
Zheng, any ideas about kernel issues which might cause this or be more
visible under infernalis?
-Greg
> _______________________________________________
>
> We have cephfs mounted on a server using the built in cephfs kernel module:
>
> lts-mon:6789:/ /ceph ceph
> name=admin,secretfile=/etc/ceph/admin.secret,noauto,_netdev
>
>
> We are running all of our ceph nodes on ubuntu 14.04 LTS. Samba is up to
> date, 4.1.6, and we export nfsv3 to linux and freebsd systems. All seem to
> exhibit the same behavior.
>
> system info:
>
> # uname -a
> Linux lts-osd1 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC
> 2015 x86_64 x86_64 x86_64 GNU/Linux
> root@lts-osd1:~# lsb
> lsblk lsb_release
> root@lts-osd1:~# lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 14.04.3 LTS
> Release: 14.04
> Codename: trusty
>
>
> package info:
>
> # dpkg -l|grep ceph
> ii ceph 9.2.0-1trusty
> amd64 distributed storage and file system
> ii ceph-common 9.2.0-1trusty
> amd64 common utilities to mount and interact with a ceph storage
> cluster
> ii ceph-fs-common 9.2.0-1trusty
> amd64 common utilities to mount and interact with a ceph file system
> ii ceph-mds 9.2.0-1trusty
> amd64 metadata server for the ceph distributed file system
> ii libcephfs1 9.2.0-1trusty
> amd64 Ceph distributed file system client library
> ii python-ceph 9.2.0-1trusty
> amd64 Meta-package for python libraries for the Ceph libraries
> ii python-cephfs 9.2.0-1trusty
> amd64 Python libraries for the Ceph libcephfs library
>
>
> What is interesting, is a directory or file will not show up in a listing,
> however, if we directly access the file, it shows up in that instance:
>
>
> # ls -al |grep SCHOOL
> # ls -alnd SCHOOL667055
> drwxrwsr-x 1 21695 21183 2962751438 Jan 13 09:33 SCHOOL667055
>
>
> Any tips are appreciated!
>
> Thanks,
> Mike C
>
>
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
