On Sun, Mar 8, 2015 at 9:21 AM, Francois Lafont <flafdivers@xxxxxxx> wrote: > Hello, > > Thanks to Jcsp (John Spray I guess) that helps me on IRC. > > On 06/03/2015 04:04, Francois Lafont wrote: > >>> ~# mkdir /cephfs >>> ~# mount -t ceph 10.0.2.150,10.0.2.151,10.0.2.152:/ /cephfs/ -o name=cephfs,secretfile=/etc/ceph/ceph.client.cephfs.secret >>> >>> Then in ceph-testfs, I do: >>> >>> root@test-cephfs:~# mkdir /cephfs/d1 >>> root@test-cephfs:~# ll /cephfs/ >>> total 4 >>> drwxr-xr-x 1 root root 0 Mar 4 11:45 ./ >>> drwxr-xr-x 24 root root 4096 Mar 4 11:42 ../ >>> drwxr-xr-x 1 root root 0 Mar 4 11:45 d1/ >>> >>> After, in test-cephfs2, I do: >>> >>> root@test-cephfs2:~# ll /cephfs/ >>> total 4 >>> drwxr-xr-x 1 root root 0 Mar 4 11:45 ./ >>> drwxr-xr-x 24 root root 4096 Mar 4 11:42 ../ >>> drwxrwxrwx 1 root root 0 Mar 4 11:45 d1/ >>> >>> 1) Why the unix rights of d1/ are different when I'm in test-cephfs >>> and when I'm in test-cephfs2? It should be the same, isn't it? > > In fact, this problem is (maybe) a bug in the Linux kernel ceph > client version "3.16". Indeed, if I mount the cephfs with ceph-fuse > on the 2 client nodes, the problem doesn't happen and if I mount the > cephfs with "mount.ceph" but with the Linux kernel 3.13, the problem > doesn't happen too. I have made a bug report here : > > http://tracker.ceph.com/issues/11059 > It's a bug in ACL code, I have updated http://tracker.ceph.com/issues/11059 >>> 2) If I create 100 files in /cephfs/d1/ with test-cephfs: >>> >>> for i in $(seq 100) >>> do >>> echo "$(date +%s.%N)" >/cephfs/d1/f_$i >>> done >>> >>> sometimes, in test-cephfs2, when I do a simple: >>> >>> root@test-cephfs2:~# time \ls -la /cephfs >> >> Sorry error of copy and paste, of course it was: >> >> root@test-cephfs2:~# time \ls -la /cephfs/d1/ >> >>> the command can take 2 or 3 seconds which seems to me very long >>> for a directory with just 100 files. Generally, if I repeat the >>> command on test-cephfs2 just after, it's immediate but not always. >>> I can not reproduce the problem in a determinist way. Sometimes, >>> to reproduce the problem, I must remove all the files in /cephfs/ >>> on test-cepfs and recreate them. It's very strange. Sometimes and >>> randomly, something seems to be stalled but I don't know what. I >>> suspect a problem of mds tuning but, In fact, I don't know what >>> to do. >> >> I have the same problem with hammer too. >> But someone can confirm me that 3s (not always) for "ls -la" in >> a cephfs directory which contains 100 file it's pathological? After >> all, maybe is it normal? I don't have much experience with cephfs. > > In fact, according to what I was told on IRC, a such time for the > "ls -la" command could be normal because the client node requests > a "stat" for each file in the directory and each "stat" can take > a little time. But I'm still a little puzzled. Indeed the first > "ls -la" can take 2 or 3 seconds and the next "ls -la" is usually > faster but not always. Sometimes (it's very random), the second > "ls -la", even the third "ls -la" etc. can be very slow. I admit > that after a number of attempts, "ls -la" becomes faster but not > always from the second attempt. > > I'm still surprised by such times. For instance, It seems to me > that, with a mounted nfs share, commands like "ls -la" are very > fast in comparison (with a directory which contains the same number > of files). Can anyone explain to me why there is a such difference > between the nfs case and the cephfs case? This is absolutely not a > criticism but it's just to understand the concepts that come into > play. In the case of "ls -al" ie just reading (it is assumed that > there is no writing on the directory), the nfs and the cephfs cases > seem to me very similar: the client just requests a stat on each file > in the directory. Am I wrong? NFS has no cache coherence mechanism. It can't guarantee one client always see other client's change. The time variation is caused cache coherence. when client has valid information in its cache, 'stat' operation will be fast. Otherwise the client need to send request to MDS and wait for reply, which will be slow. > > -- > François Lafont > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |