|
Le 08/02/2013 17:29, Gregory Farnum a
écrit :
On Friday, February 8, 2013, Yann Dupont wrote:
Le 08/02/2013
11:00, Tobias Prousa a écrit :
Hi,
I'm am evaluating a small ceph cluster with three nodes each
running one mon, one mds, and 2-3 osds. Nodes are debian
squeeze/wheezy (mixed) all running linux-3.2.35 from debian
repos (from backports.debian.org
on squeeze box). Ceph is using repo from ceph.com/debian-bobtail, installed
0.56.2.
The client is running debian wheezy but kernel 3.7.3 from
debian experimental, ceph packages also installed using ceph.com/debian-bobtail repo.
There is only one active mds with two standby. One thing which
might be unusual is that I do not mount the cephfs-root on the
client but a subdir, i.e. mount -t ceph 172.16.0.4:6789:/home/
/home
I am experiencing truncated files where files get writte
successfully to ceph and can be read without problem, i.e.
md5sum is ok and even filesize shows correct value of maybe
3.7MiB. Then after waiting for some time without the file
receiving any IO it starts showing a filesize of 2.0MiB and
md5sum fails (as to be expected).
So far I only noticed that behaviour only in one subtree of my
home folder where I prepare software packages using "tar cvfj
..." to create a .tar.bz2 bundle. Those files suddenly get
truncated to 2.0MiB after some time, but it seems that only
newly created files get truncated after some time. Similar
.tar.bz2 files which I have there, since initial rsync to
ceph, don't get truncated at all.
Hello,
I've seen the same, but because I used to point subdirectories
to different pool (using cephs set_layout).
and restricted the client to writing on this very pool.
caps: [osd] allow rw pool the-name-of-the-pool
All seems to works ok, but after some times (or after an
unmount/remount) all files are still there, but all content is
nullified (yes,filled with 0).
I ended up giving extend rights for the client ( caps: [osd]
allow * ) and now it seems to works fine.
BTW, is cephfs set_layout supposed to work ?
If I set the layout for a subdirectory to, say pool 4, and
restrict the client to another pool (say pool 5), I'm still able
to mount, read and write the subdirectory.
This'll happen because the client first writes to page cache
(of course), and it doesn't discover it can't write out to the
pool until it goes to flush. You should see errors if you try
and fsync the files. (We have a bug open to add more pro-active
checking of OSD write permissions.)
Thanks Gregory for the explanation. That's more or less what I was
imagining. So, the fact that file and their length are still there
is because information is OK on mds, but no data is really written
on osd, right ?
Anyway, this explain the case for write, BUT why an unathorized
client can still acces and read data from a pool he isn't allowed
to do ? Still mds permissions ?
To be honest I see the directory layouts, but didn't try a md5sum on
a file to see if he is REALLY read...
There's a separate bug open where sometimes directories lose
their layouts. Check if you're hitting that if you believe your
OSD permissions are set correctly. I'm planning to spend a bunch
of machine time trying to reproduce this in the next week or two
in order to validate that a patch we have will fix the issue.
Will try to create a separate case for this monday.
Thanks for your time,
Cheers,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@xxxxxxxxxxxxxx
|
