On Thu, 24 Jan 2013, Mandell Degerness wrote: > The advisory locks are nice, but it would be really nice to have the > fencing. If a node is temporarily off the network and a heartbeat > monitor attempts to bring up a service on a different node, there is > no way to ensure that the first node will not write data to the rbd > after the rbd is mounted on the second node. It would be nice if, on > seeing that an advisory lock exists, you could tell ceph "Do not > accept data from node X until further notice". Just a reminder: you can use the information from the locks to fence. The basic process is: - identify old rbd lock holder (rbd lock list <img>) - blacklist old owner (ceph osd blacklist add <addr>) - break old rbd lock (rbd lock remove <img> <lockid> <addr>) - lock rbd image on new host (rbd lock add <img> <lockid>) - map rbd image on new host The oddity here is that the old VM can in theory continue to write up until the OSD hears about the blacklist via the internal gossip. This is okay because the act of the new VM touching any part of the image (and the OSD that stores it) ensures that that OSD gets the blacklist information. So on XFS, for example, the act of replaying the XFS journal ensures that any attempt by the old VM to write to the journal will get EIO. sage > > On Thu, Jan 24, 2013 at 11:50 AM, Josh Durgin <josh.durgin@xxxxxxxxxxx> wrote: > > On 01/24/2013 05:30 AM, Ugis wrote: > >> > >> Hi, > >> > >> I have rbd which contains non-cluster filesystem. If this rbd is > >> mapped+mounted on one host, it should not be mapped+mounted on the > >> other simultaneously. > >> How to protect such rbd from being mapped on the other host? > >> > >> At ceph level the only option is to use "lock add [image-name] > >> [lock-id]" and check for existance of this lock on the other client or > >> is it possible to protect rbd in a way that on other clients "rbd map > >> " command would just fail with something like Permission denied > >> without using arbitrary locks? In other words, can one limit the count > >> of clients that may map certain rbd? > > > > > > This is what the lock commands were added for. The lock add command > > will exit non-zero if the image is already locked, so you can run > > something like: > > > > rbd lock add [image-name] [lock-id] && rbd map [image-name] > > > > to avoid mapping an image that's in use elsewhere. > > > > The lock-id is user-defined, so you could (for example) use the > > hostname of the machine mapping the image to tell where it's > > in use. > > > > Josh > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
