qdiskd questions

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

I have a two node cluster providing NFS. I'm using a small partition on the
shared storage as a quorum disk, with a single heuristic to ping the default
gateway on the network. Both nodes are connected to the network with bonded
interfaces, but I have all of the heartbeat/cluster traffic running over a
crossover cable between the two. The hardware is a pair of Dell PowerEdge
servers with an MD3000 array between them and I'm using the DRAC interface
as the fence device. My cluster.conf looks like the following:

<cluster name="storage" config_version="46">
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
        <quorumd interval="1" tko="10" votes="1" device="/dev/mapper/md3000p1">
                <heuristic program="ping 192.168.30.254 -c3 -t2" score="1" interval="2" tko="3"/>
        </quorumd>
        <clusternodes>
                <clusternode name="node1-xover" votes="1" nodeid="1">
			<fence>
				...
			</fence>
		</clusternode>
                <clusternode name="node2-xover" votes="1" nodeid="2">
			<fence>
				...
			</fence>
		</clusternode>
        </clusternodes>
        <cman expected_votes="3"/>
	...
</cluster>

(It's exactly as per the qdisk(5) man page example)

It's been up and running for ages with no trouble, but recently I
had a problem where the default gateway, despite being an active/passive
pair of Cisco ASA firewalls configured for failover, took at least 30
seconds to fail over when the primary device developed a problem. This
caused the heuristic to fail for long enough, and both nodes rebooted
simultaneously which caused a loss of service. All I can see in the
logs is:

---8<---
Mar  5 07:38:50 node1 qdiskd[7967]: <info> Heuristic: 'ping 192.168.30.254 -c3 
-t2' DOWN (3/3) 
Mar  5 07:38:50 node1 qdiskd[7967]: <notice> Score insufficient for master oper
ation (0/1; required=1); downgrading 
Mar  5 07:38:50 node1 kernel: md: stopping all md devices.
Mar  5 07:38:51 node1 kernel: Synchronizing SCSI cache for disk sdd: 
Mar  5 07:38:51 node1 kernel: Synchronizing SCSI cache for disk sdb: 
Mar  5 07:38:51 node1 kernel: Synchronizing SCSI cache for disk sda: 
Mar  5 07:38:51 node1 kernel: ACPI: PCI interrupt for device 0000:0a:00.0 disab
led
Mar  5 07:38:51 node1 kernel: hub 1-1:1.0: cannot reset port 2 (err = -71)
Mar  5 07:42:34 node1 syslogd 1.4.1: restart.
---8<---

The time difference between the last two messages is obviously where the
node is rebooting. The timestamps on the logs from both nodes are identical
apart from a few seconds on that last message.

I'm a bit unsure what actually did the rebooting in this case, was it
qdiskd or each node shooting the other? Ideally I would like to prevent
this situation from happening again, is it a case of simply adding
reboot="0" to the <quorumd> directive? Does this introduce any different
problems?

Thanks

Matt

Attachment: pgplKXpS8TWbR.pgp
Description: PGP signature

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

[Index of Archives]     [Corosync Cluster Engine]     [GFS]     [Linux Virtualization]     [Centos Virtualization]     [Centos]     [Linux RAID]     [Fedora Users]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite Camping]

  Powered by Linux