On 08/13/2012 09:55 AM, Gregory Farnum wrote:
We've discussed some of the issues here a little bit before. See http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/7094 if you're interested. Josh, can you discuss the current status of the advisory locking? -Greg
Yehuda reworked it into a generic rados class so it can be used outside of rbd. It hasn't been re-integrated with rbd yet, and I haven't looked at it closely since the generalization. Yehuda could describe it in more detail. Josh
On Sun, Aug 12, 2012 at 8:44 AM, Sage Weil <sage@xxxxxxxxxxx> wrote:RBD image locking is on roadmap, but it's tricky. Almost all of the pieces are in place for exclusive locking of the image header, which will let the user know when other nodes have the image mapped, and give them the option to break their lock and take over ownership. The real challenge is fencing. Unlike move conventional options like SCSI, the RBD image is distributed across the entire cluster, so ensuring that the old guy doesn't still have IOs in flight that will stomp on the new owner means that potentially everyone needs to be informed that the bad guy should be locked out. I think there are a few options: 1- The user has their own fencing or STOGITH on top of rbd, informed by the rbd locking. Pull the plug, update your iptables, whatever. Not very friendly. 2- Extend the rados 'blacklist' functionality to let you ensure that every node in the cluster has received the updated osdmap+blacklist information, so that you can be sure no further IO from the old guy is possible. 3- Use the same approach that ceph-mds fencing uses, in which the old owner isn't known to be fenced away from a particular object until the new owner reads/touches that object. My hope is that we can get away with #3, in which case all of the basic pieces are in place and the real remaining work is integration and testing. The logic goes something like this: File systems write to blocks on disk in a somewhat ordered fashion. After writing a bunch of data, they approach a 'consistency point' where their journal and/or superblocks must be flushed and things 'commit' to disk. At that point, if the IO fails or blocks, it won't continue to clobber other parts of the disk. When an fs in mounted, those same critical areas are read (superblock, journal, etc.). The existing client/osd interaction ensures that if the new guy knows that the old guy is fenced, the act of reading ensures that the relevant ceph-osds will find out too and that paticular object will be fenced. The resulting conclusion is that if a file system (or application on top of it doing direct io) is sufficiently well-behaved that will be not corrupt itself when the disk reorders IOs (they do) and issues barrier/flush operations at the appropriate time (in modern kernels, they do), then it will work. I suppose it's roughly analogous to Schroedinger's cat: until the new owner reads a block, it may or may not still be modified/modifiable by the old guy, but as soon as it is observed, its state is known. What do you guys think? If that doesn't work, I think we're stuck with #2, which is expensive but doable. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
-- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
