I have a strange problem on one machine where eth0 gets killed when I add
a virtual interface. It's got something to do with the NIC ordering or
with the xen network script having a problem with multiple NICs and
virtual interfaces. I could need some help/comments on this.
Some history:
I added a NIC (chip identifies as Realtek Semiconductor Co., Ltd.
RTL8111/8168B PCI Express Gigabit Ethernet) to a Dell R200 server.
CentOS 5.3 with Xen 3.3.1 (gitco repo). eth0 and eth1 are the built-in
NICs, this is then eth2 (or it should be).
Works. Everything is fine until I add a virtual interface to eth0 and
reboot. I can add eth0:1 at runtime just fine. But if I let it stay in
network-scripts and boot the whole eth0 is killed (doesn't show up in
ifconfig and doesn't work). A network restart brings it up as if nothing
is wrong.
I first thought it might have something to do with the fact that eth0 is
actually a bridge on Xen > 3.2 and tried the same config on another
machine and there it works. It's not the exact same xen version, not 64bit
and it's got only 1 NIC. So there are differences, but it seems to rule
out the bridge as a cause.
I then checked the logs more thoroughly and found that CentOS changes the
NIC initialization order at boot-time.
Without the third NIC it's eth0=NIC1 and eth1=NIC2 (as shown on the
chassis). But with the third NIC it's most often that one that goes first.
Here's a typical excerpt from messages. tigon/tg3 is the driver for the
internal NICs which normally were on eth0 and eth1.
Apr 25 19:00:59 c4 kernel: eth0: RTL8168b/8111b at 0xffffc20000022000,
00:21:27:c9:d1:f5, XID 38000000 IRQ 16
Apr 25 19:00:59 c4 kernel: eth1: Tigon3 [partno(BCM95721) rev 4201 PHY
(5750)] (PCI Express) 10/100/1000Base-T Ethernet 00:1e:c9:fe:fb:ab
Apr 25 19:00:59 c4 kernel: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1]
WireSpeed[1] TSOcap[1]
Apr 25 19:00:59 c4 kernel: eth1: dma_rwctrl[76180000] dma_mask[64-bit]
Apr 25 19:00:59 c4 kernel: eth2: Tigon3 [partno(BCM95721) rev 4201 PHY
(5750)] (PCI Express) 10/100/1000Base-T Ethernet 00:1e:c9:fe:fb:ac
Apr 25 19:00:59 c4 kernel: eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1]
WireSpeed[1] TSOcap[1]
Apr 25 19:00:59 c4 kernel: eth2: dma_rwctrl[76180000] dma_mask[64-bit]
Apr 25 19:00:59 c4 kernel: tg3: eth0: Link is up at 1000 Mbps, full
duplex.
Apr 25 19:00:59 c4 kernel: tg3: eth0: Flow control is on for TX and on for
RX.
Apr 25 19:00:59 c4 kernel: r8169: eth2: link up
Apr 25 19:00:59 c4 kernel: r8169: eth2: link up
Apr 25 19:01:01 c4 ntpd[2461]: Listening on interface eth2,
192.168.2.4#123 Enabled
Apr 25 19:01:01 c4 ntpd[2461]: Listening on interface eth0,
192.168.1.24#123 Enabled
Apr 25 19:01:01 c4 ntpd[2461]: Listening on interface eth1,
192.168.2.3#123 Enabled
Apr 25 19:01:08 c4 uxmon: c4.net: started monitoring: lo eth2 eth0 eth1
Apr 25 19:01:18 c4 kernel: tg3: peth0: Link is up at 1000 Mbps, full
duplex.
Apr 25 19:01:18 c4 kernel: tg3: peth0: Flow control is on for TX and on
for RX.
Apr 25 19:01:18 c4 kernel: device peth0 entered promiscuous mode
Apr 25 19:01:18 c4 kernel: type=1700 audit(1240678878.244:3): dev=peth0
prom=256 old_prom=0 auid=4294967295 ses=4294967295
Apr 25 19:01:18 c4 kernel: eth0: topology change detected, propagating
Apr 25 19:01:18 c4 kernel: eth0: port 1(peth0) entering forwarding state
Repeated booting sometimes gives me a different order, e.g. the two tigon
come first, but this is rare.
Well, it seems this wasn't a problem until I added a virtual interface to
eth0. When the eth interfaces are brought up the system seems to
reenumerate the eth numbering according to the HWADDR matches and thus
eth0=NIC1 and so on. As soon as I add a virtual interface to eth0 this
breaks and all of eth0 is killed. At least that's what I figure.
So, the next obvious question is: How can I set a fixed order, so that
NIC1 is always brought up first as eth0?
I'm not sure if this would fix it, though. I have done too few reboots
yet, but it seems that at least once I got the "correct" initialization
order but eth0 got killed, anyway. So, it might not be the order but still
something in the Xen script which happens only when multiple NICs are
present and a virtual interface is added.
Any thoughts so far?
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
