On Thu, 2007-11-08 at 19:03 -0500, Christopher Barry wrote: > On Thu, 2007-11-08 at 15:32 -0600, Ryan O'Hara wrote: > > Christopher Barry wrote: > > > > > > Okay. I had some other issues to deal with, but now I'm back to this, > > > and let me get you all up to speed on what I have done, and what I do > > > not understand about all of this. > > > > > > status: > > > esx-01: contains nodes 1 thru 3 > > > esx-02: contains nodes 4 thru 6 > > > > > > esx-01: all 3 cluster nodes can mount gfs. > > > > > > esx-02: none can mount gfs. > > > esx-02: scsi reservation errors in dmesg > > > esx-02: mount fails w/ "can't read superblock" > > > > OK. So it looks like one of the nodes is still holding a reservation on > > the device. First, we need to determine which node has that reservation. > > From any node, you should be able to run the following commands: > > > > sg_persist -i -k /dev/sdc > > sg_persist -i -r /dev/sdc > > > > The first will list all the keys registered with the device. The second > > will show you which key is holding the reservation. At this point, I > > would expect that you will only see 1 key registered and that key will > > also be the reservation holder, but that it just a guess. > > > > The keys are unique to each node, so we can figure correlate a key to a > > node. The key is just the hex representation of the node's IP address. > > You can get this by running gethostip -x <hostname>. By doing this, you > > should be able to figure out which node is still holding a reservation. > > Once you determine this key/node, try running /etc/init.d/scsi_reserve > > stop from that node. Once that runs, use the sg_persist commands listed > > above to see if the reservation is cleared. > > > > > Oddly, with the gfs filesystem unmounted on all nodes, I can format the > > > gfs filesystem from the esx-02 box (from node4), and then mount it from > > > a node on esx-01, but cannot mount it on the node I just formatted it > > > from! > > > > > > fdisk -l shows /dev/sdc1 on nodes 4 thru 6 just fine. > > > > Hmm. I wonder if there is something goofy happening because the nodes > > are running within vmware. I have never tried this, so I have no idea. > > Either way, we should be able to clear up the problem. > > > > > # sg_persist -C --out /dev/sdc1 > > > fails to clear out the reservations > > > > Right. It believe this must be run from the node holding the > > reservation, or at the very least a node that is registered with the > > device. Also node that scsi reservations effect the entire LUN, so you > > can't issue registrations/reservations to a single partition (ie. sdc1). > > > > > I do not understand these reservations, maybe someone can summarize? > > > > I'll try to be brief. Each node in the cluster can register with a > > device, thus a device may have many registrations. Each node registers > > by using a unique key. Once registered, one of the nodes can issue a > > reservation. Only one node may hold the reservation, the reservations is > > created using that node's key. For our purposed, we use a > > write-exclusive, registrants only type of reservation. This means that > > only nodes that are registered with the device may write to it. As long > > as that reservation exists, that rule will be enforced. > > > > When it comes to to remove registrations, there it one caveat: the node > > that hold the reservation cannot unregister unless there are no other > > nodes registered with the device. This is due to the fact that the > > reservations holder must also be registered *and* if the reservation > > were to go away the write-exclusive, registrants-only policy would not > > longer be in effect. So ... what may have happened is that you tried to > > clear the reservation while other nodes were still registered, which > > will fail since that cannot happen. Once all the other nodes have > > "unregistered", you should be able to go back and clear the reservation. > > > > Yes, this is a limitation in our product. There is a notion of moving a > > reservation (in the case where the reservation holder wants to > > unregister), but that is not yet implemented. > > > > > I'm not at the box this sec (vpn-ing in will hork my evolution), but I > > > will provide any amount of data if either you Ryan, or anyone else has > > > stuff for me to try. > > > > Please let me know if you have questions or need further assistance > > clearing that pesky reservation for you. :) > > > > > Thanks all, > > > -C > > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster@xxxxxxxxxx > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > Ryan, > > Thank you so much for your replies. > > I tracked down the registration and reserve to the first cluster node, > by converting the hex value to the IP per your instructions. All nodes > reported only this one registration. > > On that node, I try: > # sg_persist -C --out /dev/sdc > > and it returns a failure, citing a scsi reservation conflict. > > I then try on kop-sds-01, the node holding the reservation: > > #/etc/init.d/scsi_reserve stop > connect() failed on local socket: Connection refused > No volume groups found > > > Now, I initially had clvmd running, and I had volume groups defined, but > since I'm running on a netapp that does all of that stuff, I decided to > simplify it and remove this stuff. I removed all that in the beginning, > after initially trying to troubleshoot this problem. Are these > reservations somehow stuck looking at an old lvm configuration > somewhere? > > Thanks! > > -C YAAAY! Looks like I might have it. I pulled out the command from /etc/init.d/scsi_reserve and used that on the node in question, and I am able to mount on node4. I'm rebooting the whole kaboodle now. I expect all nodes to mount when it comes up. Thanks Ryan! Now that I can mount, how bad will the performance be... ;) -C -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
