On Friday, February 8, 2013, Yann Dupont wrote:
Le 08/02/2013 11:00, Tobias Prousa a écrit :
Hi,Hello,
I'm am evaluating a small ceph cluster with three nodes each running one mon, one mds, and 2-3 osds. Nodes are debian squeeze/wheezy (mixed) all running linux-3.2.35 from debian repos (from backports.debian.org on squeeze box). Ceph is using repo from ceph.com/debian-bobtail, installed 0.56.2.
The client is running debian wheezy but kernel 3.7.3 from debian experimental, ceph packages also installed using ceph.com/debian-bobtail repo.
There is only one active mds with two standby. One thing which might be unusual is that I do not mount the cephfs-root on the client but a subdir, i.e. mount -t ceph 172.16.0.4:6789:/home/ /home
I am experiencing truncated files where files get writte successfully to ceph and can be read without problem, i.e. md5sum is ok and even filesize shows correct value of maybe 3.7MiB. Then after waiting for some time without the file receiving any IO it starts showing a filesize of 2.0MiB and md5sum fails (as to be expected).
So far I only noticed that behaviour only in one subtree of my home folder where I prepare software packages using "tar cvfj ..." to create a .tar.bz2 bundle. Those files suddenly get truncated to 2.0MiB after some time, but it seems that only newly created files get truncated after some time. Similar .tar.bz2 files which I have there, since initial rsync to ceph, don't get truncated at all.
I've seen the same, but because I used to point subdirectories to different pool (using cephs set_layout).
and restricted the client to writing on this very pool.
caps: [osd] allow rw pool the-name-of-the-pool
All seems to works ok, but after some times (or after an unmount/remount) all files are still there, but all content is nullified (yes,filled with 0).
I ended up giving extend rights for the client ( caps: [osd] allow * ) and now it seems to works fine.
BTW, is cephfs set_layout supposed to work ?
If I set the layout for a subdirectory to, say pool 4, and restrict the client to another pool (say pool 5), I'm still able to mount, read and write the subdirectory.
This'll happen because the client first writes to page cache (of course), and it doesn't discover it can't write out to the pool until it goes to flush. You should see errors if you try and fsync the files. (We have a bug open to add more pro-active checking of OSD write permissions.)
There's a separate bug open where sometimes directories lose their layouts. Check if you're hitting that if you believe your OSD permissions are set correctly. I'm planning to spend a bunch of machine time trying to reproduce this in the next week or two in order to validate that a patch we have will fix the issue.
-Greg
(Beware, use cephfs set_layout on a recent kernel. For exemple, using cephfs on stock ubuntu 12.10 kernel gives a oops)
Cheers,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@xxxxxxxxxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
