On Thu, Jan 2, 2014 at 2:18 PM, Alex Pearson <alex@xxxxxxxxxxx> wrote: > Hi All, > Victory! Found the issue, it was a mistake on my part, however it does raise another questions... > > The issue was: > root@osh1:~# ceph --cluster apics auth list > installed auth entries: > <SNIP> > client.cuckoo > key: AQBjTblS4AFAARAAZyumzFyk2JS8d9AjutRoTQ== > caps: [mon] allow r > caps: [osd] allow rwx pool=staging, allow rwx pool=media2 > > > When I recreated the pool I changed from 'media2' to 'media3' - so there wasn't any authorization to the pool. I've corrected this (see below), then REMOUNTED the filesystem on the client (it didn't work until I'd done this) > > root@osh1:~# ceph --cluster apics auth caps client.cuckoo osd 'allow rwx pool=media3, allow rwx pool=staging' mon 'allow r' > Ref: http://www.sebastien-han.fr/blog/2013/07/26/ceph-update-cephx-keys/ > > > The BIG QUESTION though... The data was being stored - I verified this by MD5 summing the data after it was written. But it wasn't being accounted for anywhere and the permissions system looks to have failed. This looks like a big security hole, surely a permissions denied error should have occurred here? Also the data was being stored, but didn't appear in any DF commands, and couldn't be seen using 'ceph --cluster <name> ls -p <pool name>'? You aren't doing quite what you think here. What's actually happened is that the data was locally buffered for writing out, and when you read it (for the md5 sum) it was looking at that in-memory state. The CephFS client was then trying to flush that dirty file data out to the OSDs, and getting EPERM back. This is a pretty tricky situation for us to handle, and our (lack of a) solution right now is definitely not great. Unfortunately it's also about all that the POSIX spec lets us do — if you try and do a flush or a sync you would get back an error code, but short of that we have no mechanism for communicating to the user that they can't write to the place they're trying to write to. We've toyed with some sort of pre-emptive check that the user can write to the location their file is stored in (and returning an error on open if they can't), but it's actually quite a hard problem and hasn't gotten any serious attention yet. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |