This problem also exhibits in RHEL3-U4, so CentOS 3.4 inherited it.
CentOS 3.4 is treated as rhas3 by xcat. The first time you run 'nodeset
<host> install' for a particular OS, the /tftpboot/xcat/rhas3/<arch>
tree gets populated with files, including install.gz, which is a
compressed disk image which in turn contains all the kernel modules,
including tg3.
1.) First thing you need to do is delete (or rename) the
/tftpboot/xcat/rhas3/<arch>/ directory.
2.) Next is to fix a minor bug in xcat's 'mkinstall' located in
$XCATROOT/install/rhas3/ARCH/bin/. I posted the fix to the xcat user's
list, and have included the hacked mkinstall as an attachment. Please
note that this mkinstall is for 'x86', but it should be pretty
straightforward to see how to make the change for x86_64 if that's your
arch.
<< cut&paste'd from my post in the xcat user's list >>
I went the second route and dug around inside
$XCATROOT/install/rhas3/ARCH/bin/mkinstall. Turns out that there is a
fix already inside this script and a module already built, but that it
only will be applied to RHEL3-U4 and not CentOS 3.4
It's because of the file test at line 375 of mkinstall: the version
string in the kernel rpm that ships on the RHEL3-U4 cd's is 2.4.21-27,
but the CentOS-3.4 kernel version string is 2.4.21-27.0.1. This version
mismatch is echo'd at lines 383 and 386 also.
<< eo snip >>
3.) 'cd' to the $XCATROOT/install/rhas3/ARCH/ directory. The fix only
gets applied if you're in this directory, as the path the to bcm5700.o
is relative - Egan Ford probably knows why he did it this way, I'm just
hacking on his script and wanted to muck with it as little as possible.
4.) run 'nodeset <some_CentOS3.4_host> install'. Because there is not
an install.gz in the /tftpboot/xcat/rhas3/<arch> directory, it will be
built, and because you're in the $XCATROOT/install/rhas3/ARCH/
directory, the pre-built bcm5700.o module which lives in
$XCATROOT/install/rhas3/ARCH/data/27.EL will get put into the install.gz
as tg3.o.
5.) 'rreset' your blades and watch them properly install CentOS 3.4 via
nfs...
John Burk
Sr. Technical Director
Mainframe Entertainment
604.628.1019
Niswonger Charles M Contr AFRL/MNGG wrote:
> John,
>
> I'm running 14 older HS20's right now and have 14 of the new
> 8843 3.6 GHz EM64T blades coming in. Right now we're running SuSE 9.2
> and xcat, but we've seen the same tg3 bug that you mentioned. I'm
> curious as to the workaround that you used for CentOS. With the
> multi-level load needed to install the bootable image onto the blade
> we still don't have a satisfactory workaround...
>
> Mike Niswonger, Systems Engineer
> charles.niswonger@xxxxxxxxxxxx
> Air Force Research Lab / MNGG (KHILS),
> SYColeman Corp.
> 101 W Eglin Blvd, Bldg. 13, Rm. 115
> Eglin AFB, FL 32542-5427
> (850)882-4446, x1217
> (850)882-4451, fax
>
> -----Original Message-----
> From: centos-bounces@xxxxxxxxxx [mailto:centos-bounces@xxxxxxxxxx] On
> Behalf Of John Burk
> Sent: Friday, April 15, 2005 1:42 PM
> To: CentOS mailing list
> Subject: Re: IBM BladeCenter HS20 blades
>
> I have 60 HS20's running CentOS 3.4, but I do the install via the xcat
> toolset over nfs.
>
> Have you looked at xcat yet to manage the os on your blades? Well
> worth the time to setup, written primarily by IBM, and free.
>
> http://www.xcat.org/
>
> BTW, what processors are in your blades? I'm running the 3.2GHz
> Nocona's, if your are similar you'll probably need to use these kernel
> parameters in your grub.conf:
>
> acpi=noirq noexec=off noapic
>
> And if you do go with nfs installs via xcat, the tg3 driver in 3.4 is
> broken. E-mail me and I'll send you the work-around. It's already in
> xcat but it takes a bit of finesse to apply the fix.
>
> John Burk
>
> Sr. Technical Director
> Mainframe Entertainment
>
> 604.628.1019
>
>
>
>
> Brendon Colby wrote:
>
> > Greetings,
> >
> > We have purchased an IBM BladeCenter and I am in the process of
> > testing Linux installation on these things (boot off SAN i.e. qla2300
> > driver, not using internal drives). My distro of choice is Debian,
> > however, since I'm really not interested in trying to hand compile all
> > the drivers, I decided to try CentOS (which I'm so far very impressed
> > with).
> >
> > On boot, as with the Debian Sarge disk, the CDROM disappears. The
> > installation media complains that it cannot find the CD. I'm testing
> > using the DVD ISO of CentOS 4. Has anyone been successful in getting
> > CentOS 3/4 installed on a BladeCenter server? I noticed back in 11/04
> > there were a few posts on it.
> >
> > Thanks.
> >
> > Brendon Colby
> > _______________________________________________
> > CentOS mailing list
> > CentOS@xxxxxxxxxx
> > http://lists.centos.org/mailman/listinfo/centos
> >
> >
> _______________________________________________
> CentOS mailing list
> CentOS@xxxxxxxxxx
> http://lists.centos.org/mailman/listinfo/centos
>
-------------- next part --------------
#!/bin/ksh
#egan@xxxxxxxxxx
#See $XCATROOT/README for license.
#
if [ -z "$XCATROOT" ]
then
if [ -r /etc/sysconfig/xcat ]
then
. /etc/sysconfig/xcat
else
if [ -r /etc/rc.config ]
then
. /etc/rc.config
fi
fi
fi
if [ -z "$XCATROOT" ]
then
if [ -r "$(dirname $0)/../lib/xcatroot" ]
then
. $(dirname $0)/../lib/xcatroot
fi
fi
if [ -z "$XCATROOT" ]
then
echo "$(basename $0): env XCATROOT not defined!" >&2
exit 1
fi
if [ ! -d "$XCATROOT" ]
then
echo "$(basename $0): XCATROOT $XCATROOT does not exist!" >&2
exit 1
fi
. $XCATROOT/lib/functions
function usage {
echo "\nUsage: ${MYNAME} [singlenode]\n"
}
needhelp $* && usage >&2 && exit 0
if [ "$#" != "1" ]
then
usage >&2
exit 1
fi
NODE=$1
NODENAME=$1
if ! isanode $1 2>/dev/null
then
echo "${MYNAME}: $1 is not a node" >&2
exit 1
fi
if NODETYPE=$(tabdb $NODETYPETAB $NODE 3)
then
:
else
echo "${MYNAME}: $NODE nodetype image not defined in $NODETYPETAB" >&2
exit 1
fi
if OSVER=$(tabdb $NODETYPETAB $NODE 1)
then
:
else
echo "${MYNAME}: $NODE nodetype osver not defined in $NODETYPETAB" >&2
exit 1
fi
if ARCH=$(tabdb $NODETYPETAB $NODE 2)
then
:
else
echo "${MYNAME}: $NODE nodetype arch not defined in $NODETYPETAB" >&2
exit 1
fi
if NODERES=$(whatismyres2 $NODE)
then
RESNAME=$NODERES
else
echo "${MYNAME}: $NODE noderes not defined in $NODERESTAB" >&2
exit 1
fi
MYOSVER=rhas3
MYARCH=x86
if [ "$OSVER" != "$MYOSVER" -o "$ARCH" != "$MYARCH" ]
then
echo "${MYNAME}: Install directory does not match $NODE in $NODETYPETAB"
exit 1
fi
if INSTALLDIR=$(tabdb $SITETAB installdir 1)
then
:
else
echo "$MYNAME: installdir not defined in $SITETAB"
exit 1
fi
DISTDIR=$OSVER/$ARCH
CFDIR=scripts/$OSVER/$ARCH
PXET=install.pxe
GRUBT=install.grub
ELILOT=elilo.conf
KERNEL=installz
INITRD=install.gz
FILES="$INSTALLDIR/$DISTDIR/images/pxeboot/initrd.img $INSTALLDIR/$DISTDIR/images/pxeboot/vmlinuz /usr/share/terminfo/x/xterm $TFTPDIR/$TFTPXCATROOT $TFTPDIR"
if [ -r $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$KERNEL -a -r $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD ]
then
DOIMAGE=0
else
DOIMAGE=1
fi
EXIT=0
for i in $FILES
do
if [ ! -r $i ]
then
echo "${MYNAME}: $i missing"
EXIT=1
fi
if [ "$EXIT" = "1" ]
then
exit 1
fi
done
if [ -r $XCATPREFIX/tftp/pxelinux.0 ]
then
mkdir $TFTPDIR >/dev/null 2>&1
cp -f $XCATPREFIX/tftp/pxelinux.0 $TFTPDIR
else
echo "$MYNAME: $XCATPREFIX/tftp/pxelinux.0 missing"
exit 1
fi
if [ -r $XCATPREFIX/tftp/default ]
then
mkdir $TFTPDIR/pxelinux.cfg >/dev/null 2>&1
cp -f $XCATPREFIX/tftp/default $TFTPDIR/pxelinux.cfg/default
else
echo "$MYNAME: $XCATPREFIX/tftp/default missing"
exit 1
fi
MASTER=$(tabdb $SITETAB master 1) \
|| echo "${MYNAME}: master not defined in $SITETAB"
MASTER_IP=$(nametoip $MASTER)
#MASTER_IPS=$(
# IPS=""
# for i in $(dnsnr "@$MASTER(-eth[0-9])?")
# do
# if IP=$(nametoip $i)
# then
# IPS="$IPS $IP"
# fi
# done
# print $IPS
#)
MASTER_IPS=$(
IPS=""
for i in $(ifconfig | egrep '^eth[0-9]' | awk '{print $1}')
do
IP=$(ifconfig $i | perl -pi -e 's/ +/:/g' | grep inet:addr: | awk -F: '{print $4}')
IPS="$IPS $IP"
done
print $IPS
)
if CONFIG_SERVER=$(tabdb $NODERESTAB $RESNAME $noderes_config_server)
then
CONFIG_IP=$(nametoip $CONFIG_SERVER)
else
CONFIG_IP=$MASTER_IP
CONFIG_SERVER=$MASTER
fi
RSHC=$(tabdb $SITETAB rsh 1) \
|| echo "${MYNAME}: rsh not defined in $SITETAB"
if [ "$(basename $RSHC)" = "rsh" ]
then
RSH="Y"
else
RSH="N"
fi
SERIAL=$(tabdb $NODERESTAB $RESNAME $noderes_serial) \
|| echo "${MYNAME}: serial defined in $NODERESTAB"
INSTALL_NFS=$(tabdb $NODERESTAB $RESNAME $noderes_nfs_install) \
|| echo "${MYNAME}: nfs not defined in $NODERESTAB"
INSTALL_NFS_IP=$(nametoip $INSTALL_NFS)
if [ "$INSTALL_NFS_IP" = "noip" ]
then
echo "Invalid Install NFS for $NODENAME"
fi
INSTALL_NIC=$(tabdb $NODERESTAB $RESNAME $noderes_netdevice) \
|| echo "${MYNAME}: install_nic not defined in $NODERESTAB"
INSTALL_DIR=$(tabdb $NODERESTAB $RESNAME $noderes_install_dir) \
|| echo "${MYNAME}: nfs dir not defined in $NODERESTAB"
INSTALL_SRC_DIR=$INSTALL_DIR/$DISTDIR
INSTALL_CF_DIR=$INSTALL_DIR/$CFDIR
mkdir -p $INSTALL_CF_DIR
FILE=$INSTALLDIR/$CFDIR/$NODETYPE-$NODERES
INSTALL_CF_FILE=$NODETYPE-$NODERES
if [ ! -r $NODETYPE.tmpl ]
then
if [ ! -r base/$NODETYPE.tmpl ]
then
echo "${MYNAME}: cannot read template $NODETYPE.tmpl"
exit 1
else
TMPLDIR="base"
fi
else
TMPLDIR="."
fi
export MASTER
export MASTER_IP
export MASTER_IPS
export CONFIG_IP CONFIG_SERVER
export RSH
export OSVER
export ARCH
export NODE
export NODERES
export NODENAME
export NODETYPE
export INSTALL_NFS
export INSTALL_NFS_IP
export INSTALL_SRC_DIR
export INSTALL_CF_DIR
export INSTALL_CF_FILE
export KERNEL
export INITRD
perl -pi -e 's/\\\n$//' <$TMPLDIR/$NODETYPE.tmpl | LANG=C subvars.pl >$FILE
mkdir -p $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH
if [ -r "$PXET" ]
then
PXEF=$TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$NODETYPE-$NODERES.pxe
perl -pi -e 's/\\\n$//' <$PXET | LANG=C subvars.pl >$PXEF
if [ "$SERIAL" = "0" -o "$SERIAL" = "1" ]
then
:
else
perl -pi -e "s/^SERIAL/#SERIAL/g" $PXEF
# perl -pi -e "s/(console[^ ]* )//" $PXEF
perl -pi -e "s/(console[^ ]* ){2}//" $PXEF
fi
fi
if [ -r "$GRUBT" ]
then
GRUBF=$TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$NODETYPE-$NODERES.grub
perl -pi -e 's/\\\n$//' <$GRUBT | LANG=C subvars.pl >$GRUBF
if [ "$SERIAL" = "0" -o "$SERIAL" = "1" ]
then
:
else
perl -pi -e "s/^serial/#serial/g" $GRUBF
perl -pi -e "s/^terminal/#terminal/g" $GRUBF
# perl -pi -e "s/(console[^ ]* )//" $GRUBF
perl -pi -e "s/(console[^ ]* ){2}//" $GRUBF
fi
fi
if [ -r "$ELILOT" ]
then
ELILOC=$TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$NODETYPE-$NODERES.elilo
perl -pi -e 's/\\\n$//' <$ELILOT | LANG=C subvars.pl >$ELILOC
if [ "$SERIAL" = "0" -o "$SERIAL" = "1" ]
then
:
else
perl -pi -e "s/^SERIAL/#SERIAL/g" $ELILOC
# perl -pi -e "s/console[^ ]* //" $ELILOC
perl -pi -e "s/(console[^ ]* ){2}//" $ELILOC
fi
fi
if [ "$DOIMAGE" = "0" ]
then
exit 0
fi
cp -f $INSTALLDIR/$DISTDIR/images/pxeboot/vmlinuz $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$KERNEL
cp -f $INSTALLDIR/$DISTDIR/images/pxeboot/initrd.img $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD
TMPDIR=/tmp/.mkinstall-$OSVER-$ARCH
if getlock mkinstall-$OSVER-$ARCH 60
then
:
else
echo "${MYNAME} Failed to get $TMPDIR lock"
exit 1
fi
rm -rf $TMPDIR
mkdir -p $TMPDIR
gunzip -c $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD >$TMPDIR/initrd 2>/dev/null
mkdir -p $TMPDIR/root
mount -o loop $TMPDIR/initrd $TMPDIR/root
#mkdir -p $TMPDIR/root/etc/terminfo/x
#mkdir -p $TMPDIR/root/etc/terminfo/v
#mkdir -p $TMPDIR/root/etc/terminfo/b
#cp -f /usr/share/terminfo/x/xterm $TMPDIR/root/etc/terminfo/v/vt100
#cp -f /usr/share/terminfo/x/xterm $TMPDIR/root/etc/terminfo/b/bterm
#cp -f /usr/share/terminfo/x/xterm $TMPDIR/root/etc/terminfo/x/xterm
###
#rm -f $TMPDIR/root/etc/terminfo/v/vt100-nav
perl -pi -e 's/^.*qla2.00.*\n$//' $TMPDIR/root/modules/pcitable
#MYDIR=$(pwd)
#cd $TMPDIR
#gunzip -c root/modules/modules.cgz | cpio -id 2>/dev/null
#if [ -r $MYDIR/data/e100.o ]
#then
# rm -f 2.4.21-3.ELBOOT/eepro100.o
# cp -f $MYDIR/data/e100.o 2.4.21-3.ELBOOT/eepro100.o
# cat $MYDIR/data/e100.pcitable >> root/modules/pcitable
#fi
#if [ -r $MYDIR/data/bcm5700.o ]
#then
# cp -f $MYDIR/data/bcm5700.o 2.4.21-3.ELBOOT/bcm5700.o
# cat $MYDIR/data/bcm5700.pcitable >> root/modules/pcitable
# cat $MYDIR/data/bcm5700.module-info >> root/modules/module-info
#fi
#if [ -r $MYDIR/data/e1000.o ]
#then
# cp -f $MYDIR/data/e1000.o 2.4.21-3.ELBOOT/e1000.o
# cat $MYDIR/data/e1000.pcitable >> root/modules/pcitable
# cat $MYDIR/data/e1000.module-info >> root/modules/module-info
#fi
#find 2.4.21-3.ELBOOT -type f -print | cpio -o -H crc 2>/dev/null | gzip -c -9 >root/modules/modules.cgz
#cd $MYDIR
# default kernel version that ships with RHEL3-U4
KERNELVER=2.4.21-27
#
# try and detect if we're actually installing CentOS instead of RHEL
# ( tests cut&paste'd from XCATROOT/sbin/copycds )
#
if [ -f $INSTALLDIR/$DISTDIR/.discinfo ]
then
if egrep "1104951135.38" $INSTALLDIR/$DISTDIR/.discinfo >/dev/null 2>&1
then
if grep "i386" $INSTALLDIR/$DISTDIR/.discinfo >/dev/null 2>&1
then
# kernel version in the CentOS-3.4 distro
KERNELVER=2.4.21-27.0.1
fi
fi
fi
#
# fix the broken tg3 driver in RHEL3-U4/CentOS-3.4
#
if [ -f $INSTALLDIR/$DISTDIR/RedHat/RPMS/kernel-${KERNELVER}.EL.i686.rpm ]
then
MYDIR=$(pwd)
cd $TMPDIR
gunzip -c root/modules/modules.cgz | cpio -id 2>/dev/null
if [ -r $MYDIR/data/27.EL/bcm5700.o ]
then
cp -f $MYDIR/data/27.EL/bcm5700.o ${KERNELVER}.ELBOOT/i386/tg3.o
fi
find ${KERNELVER}.ELBOOT -type f -print | cpio -o -H crc 2>/dev/null | gzip -c -9 >root/modules/modules.cgz
cd $MYDIR
fi
umount $TMPDIR/root
gzip -9 -f $TMPDIR/initrd
cp -f $TMPDIR/initrd.gz $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD
rm -rf $TMPDIR
freelock mkinstall-$OSVER-$ARCH 0
cd $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH
chown -f root.root *.gz *.pxe *.nbi *.eb
chmod -f 444 *.gz *.pxe *.nbi *.eb
exit 0
