If clustat reports rgmanager as online, why would any clusvcadm operation fail with "Try again (resource groups locked)" ?
Is there any way to recover from that rgmanger failure/error besides resetting the entire cluster?
Details --
Yesterday evening a technician connected a Netgear GS748T switch to my network. The new switch somehow caused a storm of traffic that in turn caused a disruption of network connectivity across the entire LAN, including to all of my CS/GFS cluster nodes, for a few minutes until the new switch was removed from the network.
This morning when I finally had a chance to investigate I found that all of the cluster members that are supposed to be online were online and that the cluster was quorate. But rgmanager would not work and services running under rgmanager were hung. (The cluster must have become inquorate and blocked access to the shared GFS volume while the outage was in progress. But some of the services and rgmanager never recovered?)
I first tried resetting the "lead" member. (This is a pool of mirrored storage servers where the lead member creates a rsync batch off of a main fileserver and all of the other members then replay the rsync batch that is on a shared filesystem against their local filesystem mirror of the main fileserver)
No matter what I did rgmanager would not start. cman_tool services would report code "S-1,80,4" --
root@gfs05:~
(0)>cman_tool services
Service Name GID LID State Code
Fence Domain: "default" 1 2 run -
[2 1 4 3]
DLM Lock Space: "clvmd" 2 3 run -
[2 1 4 3]
User: "usrm::manager" 0 4 join S-1,80,4
[]
Other cluster members would report rgmanager as online, yet when I tried to operate on member services, the operation would fail with "Try again (resource groups locked)".
root@gfs06:~
(1)>clustat
Member Status: Quorate
Member Name Status
------ ---- ------
gfs04 Online, rgmanager
gfs05 Online
gfs06 Online, Local, rgmanager
gfs07 Online, rgmanager
gfs08 Offline
Service Name Owner (Last) State
------- ---- ----- ------ -----
mapsmirror1 gfs05 started
mapsmirror2 gfs06 started
mapsmirror3 gfs07 started
mapsmirror4 gfs04 started
mapsmirror5 (none) stopped
root@gfs06:~
(0)>clusvcadm -d mapsmirror1
Member gfs06 disabling mapsmirror1...failed: Try again (resource groups locked)
Eventually I just gave up and power cycled all cluster members at ounce. Everything, including rgmanger, then came back online OK.
-- Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
