Re: Kickstart with multiple eth devices

"Jason Warr" <jason@xxxxxxxx> · Wed, 25 Feb 2015 16:53:34 -0600

On Wed, 25 Feb 2015 16:45:04 -0600, Ashley M. Kirchner <ashley@xxxxxxxxxx>  
wrote:

Out of order meaning, it puts the additional ethernet card as eth0, with  
the built-in ports as eth1 and eth2 respectively. WITHOUT the additional  
card >installed, it puts the built-in ports as eth0 and eth1, which is  
what I want it to do. The additional card should be eth2, at least  
that's what I want it to >do.

So if you really need them to be in that order your best bet may be to  
blacklist the kernel module for the add in card so it does not get  
enumerated during install.  If you do that and create the udev rules file  
as I do then on first boot udev should enforce your rules file and then  
enumerate the add in card as eth2.

Add "rdblacklist=MODULE_NAME" to your append line in pxelinux.conf file.

This should get you into a usable state post-install so you can setup  
additional interfaces.

Looking at persistent-net.rules, it always puts the additional card  
first, both without your script as well as with. I need it to be last.  
The system's built->ins should always be eth0 and eth1 respectively.

It looks like the add in card is in a bus slot that is getting enumerated,  
module loaded before the on-board interfaces.

And dmesg confirms it as well, it identifies the added card first (and  
assigns it eth0), then identifies the built-in ports. I'd grab the  
actual output except >I need to manually reconfigure the interfaces so I  
can actually get ON the machine. Right now I'm just looking at its  
console

On Wed, Feb 25, 2015 at 3:37 PM, Jason Warr <jason@xxxxxxxx> wrote:
Define out of order in this case just so I know for sure what you mean.

What my solution does, or at least does reliably in my case, is make  
sure the interfaces are in the same order once installed as the install  
kernel saw >>them.  It won't re-order them to be sequential based on  
bus, mac or driver.  I am working on that but it will also include  
naming the devices based on >>the module name, similar to how FreeBSD  
and Solaris do it.

Just to get an idea of what might be going on can you run "dmesg | grep  
eth" so I can see some of what udev is doing?

On Wed, 25 Feb 2015 16:28:31 -0600, Ashley M. Kirchner  
<ashley@xxxxxxxxxx> wrote:

Ok, when I run that, it creates a now "custom"  
70-persistent-net.rules, but
the interfaces are still out of order, with the added one listed  
first, and
the built-ins 2nd and 3rd.

On Wed, Feb 25, 2015 at 2:00 PM, Jason Warr <jason@xxxxxxxx> wrote:

Here is my script for post install if you want to try it.

In order for the shuffling to not occur you do need to create the udev
rules file somehow.  I am not sure how mangled this will be in email  
but it
is worth a try.  It should run OK with nothing else.  I have a better
version in the works but the enhancements are mainly useful for Fedora
19-21.

I did forget to say I also block NetworkManager from the interfaces.

############################

#!/bin/bash
## BIND MAC address to proper interfaces so they stay persistent
## I want them to stay as they were in kickstart

## This can cause issues with VLAN interfaces for both bond dev's and  
base
eth dev's.
##  The bond one was solved by adding in the "KERNEL="eth?*" as that  
will
only apply to physical
##  Devices.  Once we have VLAN's on a real device instead of just on
BOND's this then applies
##  to the hardware devices as well.  The core issue is that the MAC
address is inherited
##  by all of the children devices and thus the UDEV rule has to be
crafted to only apply
##  to the base physical device.
##  This one was solved via adding DRIVERS=="?*" as the VLAN int's  
wont
have one

   echo "[KICKSTART] Binding eth interfaces to the expected MAC  
address
in UDEV"
   echo "## Created by Kickstart to keep network interfaces in an
expected order" > \
       /etc/udev/rules.d/70-persistent-net.rules
   echo "" >> \
       /etc/udev/rules.d/70-persistent-net.rules

   cd /sys/class/net/
   for NETDEV in $(ls | grep eth | sort)
   do
       ## Create a UDEV rule for each eth interface
       echo "## ${NETDEV} interface" >> \
           /etc/udev/rules.d/70-persistent-net.rules

       ## We throw this one in here as it can contain some useful
information
       echo "## $(dmesg | grep ${NETDEV} | grep -i -v -e "console" -e
"Command line" | head -1)" >> \
           /etc/udev/rules.d/70-persistent-net.rules

       echo -n "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\",  
"

\

           /etc/udev/rules.d/70-persistent-net.rules
       echo -n "ATTR{address}==\"$(cat ${NETDEV}/address)\", " >> \
           /etc/udev/rules.d/70-persistent-net.rules
       echo -n "ATTR{dev_id}==\"0x0\", ATTR{type}==\"1\",
KERNEL==\"eth?*\", " >> \
           /etc/udev/rules.d/70-persistent-net.rules
       echo -e "NAME=\"${NETDEV}\"\n" >> \
           /etc/udev/rules.d/70-persistent-net.rules

       ## Make a log of the devices present during install
       echo -e "${NETDEV} $(cat ${NETDEV}/address)\n" >>
/root/ksnet-devices

       ## Also remove the HWADDR line from all of the net config files
       grep -v -e NAME -e HWADDR -e NM_CONTROLLED \
           /etc/sysconfig/network-scripts/ifcfg-${NETDEV} | sed
's/\"//g' \
           > /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp
       echo "NM_CONTROLLED=no" >> /etc/sysconfig/network-
scripts/ifcfg-${NETDEV}-tmp
       /usr/bin/perl -p -i -e 's/dhcp/none/' /etc/sysconfig/network-
scripts/ifcfg-${NETDEV}-tmp
       mv -f /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp \
             /etc/sysconfig/network-scripts/ifcfg-${NETDEV}
   done

###########################

On Wed, 25 Feb 2015 14:53:40 -0600, Ashley M. Kirchner  
<ashley@xxxxxxxxxx>
wrote:

Thanks for that Jason but it didn't solve the problem. The system is  
still
coming up with the interfaces shuffled. It seems to *always* want to  
use
the added ethernet card as eth0.

On Wed, Feb 25, 2015 at 1:37 PM, Jason Warr <jason@xxxxxxxx> wrote:

Starting back in RHEL/Cent 5 I found that the only way to make sure  
your
interface enumeration was consistent after install with what you had
during
install was to create a udev rules file using the mac addresses as  
the
key.  It is easy to run a short script in postinstall to create it  
based
on
how anaconda has seen them.

In order for this to work on Cent 6 you have to set biosdevname=0  
on the
kernel boot for the installed system.

PXE boot options:

label c6inst-sda
   kernel /linux-boot/cent6-x64/vmlinuz
   append initrd=/linux-boot/cent6-x64/initrd.img ksdevice=bootif
ip=dhcp ks=http://xx.xx.xx.xx/install/linux/ks/basic-cent6-sda.cfg
   ipappend 2

In kickstart:

BOOTOPTS="biosdevname=0"

Also in kickstart I do not specify the config for ANY network  
interfaces.
I let anaconda pull in only the config for the boot interface from  
PXE.
I
manually configure everything else.  The only thing I do to non-boot
interfaces is set the DHCP and ONBOOT to no.

On Wed, 25 Feb 2015 14:21:18 -0600, Ashley M. Kirchner <
ashley@xxxxxxxxxx>
wrote:

Version 6.6 ...

On Wed, Feb 25, 2015 at 1:17 PM, Jim Perrin <jperrin@xxxxxxxxxx>  
wrote:

<overly trimmed>

On 02/25/2015 01:56 PM, Ashley M. Kirchner wrote:
Ok, so some of this now works, but I'm still having problems.  
With
the
bootif option, the system now correctly configures and uses the  
same
interface to get its kickstart file. However, when the system is  
done
and
boots up, the interfaces are still messed up. So this is what I  
have
in
the
kickstart file:

What version of CentOS 6 is this?

In the PXE config file I have:

IPAPPEND 2
APPEND ks=http://192.168.x.x/ks/portico.ks
initrd=centos/x86_64/initrd.img
ramdisk_size=100000 ksdevice=bootif

As soon as I *remove* the additional ethernet card, the system  
will
boot
up
with the ports configured correctly (port 1 = eth0, port 2 =  
eth1).
So
why
is it that as soon as there is an additional one, all things go  
to
hell?
Why must the boot process shuffle them? More importantly, how do  
I
prevent
this so that the system comes up properly after a kickstart  
install?

The reason I ask the version, is this is exactly the sort of  
thing that
biosdevname is designed to solve. With biosdevname, you get  
devices
like
'em1, em2, p6p1', which aren't as friendly as 'eth0' but also keep
names
sane and avoid the hair-tearing issues you're experiencing  
currently.
You don't appear to be adding anything via your append line that  
would
disable biosdevname, so I must assume you're using a much older 6  
base
install.

In my experience biosdevname creates just as many problems as it
solves.
Dell can keep it.

--
Jim Perrin
The CentOS Project | http://www.centos.org
twitter: @BitIntegrity | GPG Key: FA09AD77
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos

_______________________________________________

CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos

_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos

_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos

_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos