This could work, thanks! P.S. Is there a way to tell which client has mapped certain rbd if no "rbd lock" is used? It would be useful to see that info in output of "rbd info <image>". Probably attribute for rbd like "max_map_count_allowed" would be useful in future - just to make sure rbd is not mapped from multiple clients if it must not. I suppose it can actually happen if multiple admins work with same rbds from multiple clients and no strict "rbd lock add.." procedure is followed. Ugis 2013/1/25 Sage Weil <sage@xxxxxxxxxxx>: > On Thu, 24 Jan 2013, Mandell Degerness wrote: >> The advisory locks are nice, but it would be really nice to have the >> fencing. If a node is temporarily off the network and a heartbeat >> monitor attempts to bring up a service on a different node, there is >> no way to ensure that the first node will not write data to the rbd >> after the rbd is mounted on the second node. It would be nice if, on >> seeing that an advisory lock exists, you could tell ceph "Do not >> accept data from node X until further notice". > > Just a reminder: you can use the information from the locks to fence. The > basic process is: > > - identify old rbd lock holder (rbd lock list <img>) > - blacklist old owner (ceph osd blacklist add <addr>) > - break old rbd lock (rbd lock remove <img> <lockid> <addr>) > - lock rbd image on new host (rbd lock add <img> <lockid>) > - map rbd image on new host > > The oddity here is that the old VM can in theory continue to write up > until the OSD hears about the blacklist via the internal gossip. This is > okay because the act of the new VM touching any part of the image (and the > OSD that stores it) ensures that that OSD gets the blacklist information. > So on XFS, for example, the act of replaying the XFS journal ensures that > any attempt by the old VM to write to the journal will get EIO. > > sage > > > >> >> On Thu, Jan 24, 2013 at 11:50 AM, Josh Durgin <josh.durgin@xxxxxxxxxxx> wrote: >> > On 01/24/2013 05:30 AM, Ugis wrote: >> >> >> >> Hi, >> >> >> >> I have rbd which contains non-cluster filesystem. If this rbd is >> >> mapped+mounted on one host, it should not be mapped+mounted on the >> >> other simultaneously. >> >> How to protect such rbd from being mapped on the other host? >> >> >> >> At ceph level the only option is to use "lock add [image-name] >> >> [lock-id]" and check for existance of this lock on the other client or >> >> is it possible to protect rbd in a way that on other clients "rbd map >> >> " command would just fail with something like Permission denied >> >> without using arbitrary locks? In other words, can one limit the count >> >> of clients that may map certain rbd? >> > >> > >> > This is what the lock commands were added for. The lock add command >> > will exit non-zero if the image is already locked, so you can run >> > something like: >> > >> > rbd lock add [image-name] [lock-id] && rbd map [image-name] >> > >> > to avoid mapping an image that's in use elsewhere. >> > >> > The lock-id is user-defined, so you could (for example) use the >> > hostname of the machine mapping the image to tell where it's >> > in use. >> > >> > Josh >> > >> > -- >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> > the body of a message to majordomo@xxxxxxxxxxxxxxx >> > More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
