CephFS: 'ls -alR' performance terrible unless Linux cache flushed

[negillen negillen <negillen@xxxxxxxxx>]

Hello everyone,

something very strange is driving me crazy with CephFS (kernel driver).

I copy a large directory on the CephFS from one node. If I try to perform a 'time ls -alR' on that directory it gets executed in less than one second.

If I try to do the same 'time ls -alR' from another node it takes several minutes. No matter how many times I repeat the command, the speed is always abysmal. The ls works fine on the node where the initial copy was executed from. This happens with any directory I have tried, no matter what kind of data is inside.

After lots of experimenting I have found that in order to have fast ls speed for that dir from every node I need to flush the Linux cache on the original node:
echo 3 > /proc/sys/vm/drop_caches

Unmounting and remounting the CephFS on that node does the trick too.

Anyone has a clue about what's happening here? Could this be a problem with the writeback fscache for the CephFS?

Any help appreciated! Thanks and regards. :)

# uname -r
3.10.80-1.el6.elrepo.x86_64
# ceph -v
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
# ceph -s
    cluster f9ffbbd7-186b-483a-96ea-90cdadb81f2a
     health HEALTH_OK
     monmap e1: 3 mons at {[omissis]}
            election epoch 60, quorum 0,1,2 [omissis]
     mdsmap e59: 1/1/1 up {0=[omissis]=up:active}, 2 up:standby
     osdmap e146: 2 osds: 2 up, 2 in
      pgmap v122287: 256 pgs, 2 pools, 30709 MB data, 75239 objects
            62432 MB used, 860 GB / 921 GB avail
                 256 active+clean

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com