PXE problem after CentOS reboot

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



Dear folks,

We are installing a large diskless cluster using CentOS 5.1. The hardware is pretty new - Supermicro X7DWT boards with Harpertown CPUs. Unfortunately we have some PXE-related problems described by the following scenario: 1) Set up DHCP, TFTP and NFS on a server, prepare PXE kernel and initrd - fine.
2) Start up the node using PXE for the first time - fine.
3) Reboot the node - PXE boot fails for all next attempts. We see that a server gets DHCP requests and answers them, but a node doesn't response with DHCP ack. The typical DHCP log is:
Jan  5 09:14:34 shoffner dhcpd: DHCPDISCOVER from 00:30:48:7e:24:a6 via eth1
Jan 5 09:14:34 shoffner dhcpd: DHCPOFFER on 10.1.5.2 to 00:30:48:7e:24:a6 via eth1
Jan  5 09:14:36 shoffner dhcpd: DHCPDISCOVER from 00:30:48:7e:24:a6 via eth1
Jan 5 09:14:36 shoffner dhcpd: DHCPOFFER on 10.1.5.2 to 00:30:48:7e:24:a6 via eth1
Jan  5 09:14:40 shoffner dhcpd: DHCPDISCOVER from 00:30:48:7e:24:a6 via eth1
Jan 5 09:14:40 shoffner dhcpd: DHCPOFFER on 10.1.5.2 to 00:30:48:7e:24:a6 via eth1
Jan  5 09:14:48 shoffner dhcpd: DHCPDISCOVER from 00:30:48:7e:24:a6 via eth1
Jan 5 09:14:48 shoffner dhcpd: DHCPOFFER on 10.1.5.2 to 00:30:48:7e:24:a6 via eth1 4) Anything like DHCP server restart, node reset, node power on/off doesn't help 5) The only thing that will enable system to boot again over PXE is to perform "bmc reset cold" command on a node using ipmitool - yes, we have IPMI card sharing the same Ethernet interface. After that we can boot CentOS again. 6) When Linux is loaded, if we reboot a node using "bmc power cycle" instead of reboot or shutdown, a node will boot for the next time without problems
7) There are no problems with a second GbE interface (without IPMI)
8) So our guess is that Linux on a reboot leaves Ethernet device in some state that cause brain damage for IPMI+PXE combination. We tried to play with some e1000 driver options, we are also tried latest Intel driver - nothing helps. Do you have any idea what goes wrong? Any help will be much appreciated. Below there is a system summary:

[root@node-05-03 ~]# uname -a
Linux node-05-03 2.6.18-53.1.4.el5 #1 SMP Fri Nov 30 00:45:55 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

[root@node-05-03 ~]# lspci
00:00.0 Host bridge: Intel Corporation Memory Controller Hub (rev 20)
00:01.0 PCI bridge: Intel Corporation PCI Express Port 1 (rev 20)
00:05.0 PCI bridge: Intel Corporation PCI Express Port 5 (rev 20)
00:07.0 PCI bridge: Intel Corporation PCI Express Port 7 (rev 20)
00:0f.0 System peripheral: Intel Corporation DMA/DCA Engine (rev 20)
00:10.0 Host bridge: Intel Corporation FSB Registers (rev 20)
00:10.1 Host bridge: Intel Corporation FSB Registers (rev 20)
00:10.2 Host bridge: Intel Corporation FSB Registers (rev 20)
00:10.3 Host bridge: Intel Corporation FSB Registers (rev 20)
00:10.4 Host bridge: Intel Corporation FSB Registers (rev 20)
00:11.0 Host bridge: Intel Corporation Unknown device 4031 (rev 20)
00:15.0 Host bridge: Intel Corporation FBD Registers (rev 20)
00:15.1 Host bridge: Intel Corporation FBD Registers (rev 20)
00:16.0 Host bridge: Intel Corporation FBD Registers (rev 20)
00:16.1 Host bridge: Intel Corporation FBD Registers (rev 20)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) 00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09) 00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09) 00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09) 00:1f.2 SATA controller: Intel Corporation 631xESB/632xESB SATA AHCI Controller (rev 09) 00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller (rev 09)
01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR] (rev a0)
02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) 02:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01) 03:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01) 03:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E3 (rev 01) 05:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01) 05:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
08:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)

Thanks in advance,
Andrey
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos

[Index of Archives]     [CentOS]     [CentOS Announce]     [CentOS Development]     [CentOS ARM Devel]     [CentOS Docs]     [CentOS Virtualization]     [Carrier Grade Linux]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [Xorg]     [Linux USB]
  Powered by Linux