Kickstart With Multiple Eth Devices

Home » CentOS » Kickstart With Multiple Eth Devices
CentOS 25 Comments

I have a Dell server that has two built-in ethernet devices. When I
kickstart the machine, they are correctly identified as eth0 and eth1
(correctly meaning they correspond to the physical device ports 1 and 2). I
need a third one and want that to come up as eth2. After adding the hardware, kickstart now fails because for some reason it goes through a rename process where it makes the newly added card eth1 (or eth0, I
forgot). Is there a way to stop this rename process so kickstart correctly uses the physical hardware the way they are, meaning physical port 1 eth0, port 2 = eth1, and the additional ethernet card then becomes eth2?

Should I be using the device’s MAC address when I set the ‘network‘ option in the kickstart file? So instead of ‘network –device=eth0′ I make it
network -device

25 thoughts on - Kickstart With Multiple Eth Devices

  • Correction to my own post: I know it’s not kickstart that’s doing the renaming, it’s the kernel that’s booting up the system.

  • 6.6-x86_64 …

    Keep in mind this is happening during the kickstart process. What I’m seeing is this:

    When the system first comes on, PXE kicks in and port 2 (or eth1) gets an IP from the network’s dhcp and is configured properly. PXE grabs the necessary files to start installation. However, as soon as it the install image is uncompressed it does a rename on the devices and what should be eth3 is now configured as either eth0 or eth1 (I forgot which one) and what
    *should* be eth1 is effectively no longer configured. At that point kickstart stops because it can no longer fetch the kickstart file. I have to physically remove the additional ethernet card for it to work properly.

  • Totally unrelated, but this is the reason I love discussions like this getting into the archives. I had no idea this option existed and it just solved an annoying problems I’ve been trying to think how to solve for ages!

    In PXE’s ‘default’;

    LABEL new-node1
    MENU LABEL ^1) New Node 1 – RHEL 6
    KERNEL boot/rhel6/x86_64/vmlinuz
    IPAPPEND 2
    APPEND initrd=boot/rhel6/x86_64/initrd.img ks=http://10.20.4.1/rhel6/x86_64/ks/pxe-ccrs-node2.ks ksdevice=bootif

    Then in kickstart;

    network –bootproto dhcp –onboot yes –hostname node1.example.com

    (not the lack of –device)

    With this, my nodes with 6 NICs reliably boot without asking the user to choose the NIC by MAC they want to install from.

    Thanks!!

  • Ok, so some of this now works, but I’m still having problems. With the bootif option, the system now correctly configures and uses the same interface to get its kickstart file. However, when the system is done and boots up, the interfaces are still messed up. So this is what I have in the kickstart file:

    #

  • What version of CentOS 6 is this?

    The reason I ask the version, is this is exactly the sort of thing that biosdevname is designed to solve. With biosdevname, you get devices like
    ’em1, em2, p6p1′, which aren’t as friendly as ‘eth0’ but also keep names sane and avoid the hair-tearing issues you’re experiencing currently. You don’t appear to be adding anything via your append line that would disable biosdevname, so I must assume you’re using a much older 6 base install.

  • Starting back in RHEL/Cent 5 I found that the only way to make sure your
    interface enumeration was consistent after install with what you had
    during install was to create a udev rules file using the mac addresses as
    the key. It is easy to run a short script in postinstall to create it
    based on how anaconda has seen them.

    In order for this to work on Cent 6 you have to set biosdevname=0 on the
    kernel boot for the installed system.

    PXE boot options:

    label c6inst-sda
    kernel /linux-boot/cent6-x64/vmlinuz
    append initrd=/linux-boot/cent6-x64/initrd.img ksdevice=bootif ip=dhcp
    ks=http://xx.xx.xx.xx/install/linux/ks/basic-cent6-sda.cfg
    ipappend 2

    In kickstart:

    BOOTOPTS=”biosdevname=0″

    Also in kickstart I do not specify the config for ANY network interfaces.
    I let anaconda pull in only the config for the boot interface from PXE. I
    manually configure everything else. The only thing I do to non-boot
    interfaces is set the DHCP and ONBOOT to no.

    In my experience biosdevname creates just as many problems as it solves.
    Dell can keep it.

  • Thanks for that Jason but it didn’t solve the problem. The system is still coming up with the interfaces shuffled. It seems to *always* want to use the added ethernet card as eth0.

  • Here is my script for post install if you want to try it.

    In order for the shuffling to not occur you do need to create the udev
    rules file somehow. I am not sure how mangled this will be in email but
    it is worth a try. It should run OK with nothing else. I have a better
    version in the works but the enhancements are mainly useful for Fedora
    19-21.

    I did forget to say I also block NetworkManager from the interfaces.

    ############################

    #!/bin/bash
    ## BIND MAC address to proper interfaces so they stay persistent
    ## I want them to stay as they were in kickstart

    ## This can cause issues with VLAN interfaces for both bond dev’s and base
    eth dev’s.
    ## The bond one was solved by adding in the “KERNEL=”eth?*” as that will
    only apply to physical
    ## Devices.

  • Ok, when I run that, it creates a now “custom” 70-persistent-net.rules, but the interfaces are still out of order, with the added one listed first, and the built-ins 2nd and 3rd.

  • Define out of order in this case just so I know for sure what you mean.

    What my solution does, or at least does reliably in my case, is make sure
    the interfaces are in the same order once installed as the install kernel
    saw them. It won’t re-order them to be sequential based on bus, mac or
    driver. I am working on that but it will also include naming the devices
    based on the module name, similar to how FreeBSD and Solaris do it.

    Just to get an idea of what might be going on can you run “dmesg | grep
    eth” so I can see some of what udev is doing?

  • Out of order meaning, it puts the additional ethernet card as eth0, with the built-in ports as eth1 and eth2 respectively. WITHOUT the additional card installed, it puts the built-in ports as eth0 and eth1, which is what I want it to do. The additional card should be eth2, at least that’s what I
    want it to do.

    Looking at persistent-net.rules, it always puts the additional card first, both without your script as well as with. I need it to be last. The system’s built-ins should always be eth0 and eth1 respectively.

    And dmesg confirms it as well, it identifies the added card first (and assigns it eth0), then identifies the built-in ports. I’d grab the actual output except I need to manually reconfigure the interfaces so I can actually get ON the machine. Right now I’m just looking at its console.

  • Ashley M. Kirchner wrote:

    Now that you have a 70-persistant-net.rules, what happens if you edit it, and name the interfaces in the correct order, then reboot?

    mark

  • Add “rdblacklist=MODULE_NAME” to your append line in pxelinux.conf file.

    Trying that next. It’ll have to wait till tomorrow as we’re under a serious blizzard/snow event right now and I’d like to get home before all of hell freezes over. However, question, if I blacklist the module, that means during kickstart it doesn’t know it’s there either, which would cause a
    ‘network’ definition for that interface to fail (and kickstart to stop) … I think.

    This should get you into a usable state post-install so you can setup

    My hope is that I don’t have to do anything after it’s done and reboots. The whole purpose of me doing this is in case something happens with the drive and I’m not here, someone can shove a new one in, reboot and let it go through the kickstart and be operational ten minutes later without them having done anything else. Maybe I’m aiming too damn high.

  • It will if you try to configure the now non-existent interface.

    No, your not. One thing you might be doing though is being to concerned
    about ordering. The actual numbering of interfaces is, in most cases,
    purely cosmetic. So if you do not have another reason for needing a nice,
    logical ordering you should ignore it. Your use case is almost exactly
    why I went the route I did in the first place.

    Since this appears to be a recovery plan for a specific system then your
    best course is to accept the ordering as kickstart sees it and adjust your
    network config lines to compensate. As long as you have a udev rules file
    the ordering will not change on reboot.

  • That’s what I figured, so I can remove it from the kickstart file, no problem. The question then becomes, if kickstart doesn’t configure it, what happens when the system reboots after install? It won’t know what to do with that interface, correct?

    Is this a case where I will need to put an ifcfg-eth2 file in place during post-install?

  • Upon reboot the system *should* generate a base one for you as it will see
    it as a new interface. Not a big deal if it does not though, just create
    one yourself. You will want to add it to the udev rules file though. You
    can re-run the script I sent to do that if you want. At that point it
    should be eth2. Or you can edit the existing one by copying a line and
    changing the MAC and eth* to whatever you need.

  • And after picking this back up this morning …. still no dice. I have now blacklisted the one module that would enumerate the add-in ethernet port so that is no longer an issue during the kickstart process, however the following is now happening:

    – kickstart completes successfully using the machine’s physical port 2 (or eth1) which is on a subnet with DHCP
    – when the system reboots, it brings up port 1 (eth0) with the correct static IP information, HOWEVER …
    – port 2 (eth1) is NOT configured properly. When I look at it’s ifcfg-eth1
    file, its bootproto is set to none when it should be set to dhcp.
    – the add-in card has not been enumerated, in fact the system doesn’t even know it’s there (dmesg has no mention of it and no module loaded)

    So for port 2 (eth1), the kickstart file has it configured as a dhcp interface, so why when the system reboots it comes up with bootproto=none?

  • Check the installed kernel append line to make sure the rdblacklist option
    is not being pulled from the kickstart boot line. If it is you can add
    this to the kickstart post install section:

    /usr/bin/perl -p -i -e ‘s/rdblacklist=MODULENAME//’ /boot/grub/grub.conf

    As I pointed out the script I sent changes all interfaces to DHCP=none.

    If you are using it and you don’t want it to do that then remove this line:

    /usr/bin/perl -p -i -e ‘s/dhcp/none/’
    /etc/sysconfig/network-scripts/ifcfg-${NETDEV}-tmp

  • Nope, it doesn’t add it to the kernel boot parameters. That was the first thing I checked.

    As for the bootproto … DUH. I didn’t check that. :) That being solved, yeah it’s not bringing up the add-in card now when it boots up.

  • What about a blacklist line somewhere in /etc/modprobe.d ?

    I have never noticed anaconda adding one there but it is worth checking.

    Also, when you check dmesg, are you looking for the expected module name
    or eth2?

  • Yeah, turned out bootif accomplishes the same thing, at least in my scenario. What happened afterwards though is different.