User Tools

Site Tools


ipmi_on_linux

This is an old revision of the document!


Running IPMI on Linux

What is IPMI?

IPMI is standard which allows remote server management, primarliy developed by Intel. IPMI cards, known as Baseboard Management Cards (BMCs) are primitive computers in their own right and are operational all the time, so long as the server has a power source. The server itself does not need to be powered on, or the operating system operational for the BMC to work, it just needs a power source to be connected to the server.

The primary benefits of IPMI are:

  • View server chassis and motherboard sensor output remotely, such as chassis status and intrusion detection.
  • Ability to remotely power on, power off, reboot the server and flash the identification light.
  • Ability to set up a console on a serial port and have the BMC redirect that console over a network port, which in cooperation with BIOS level console redirection, gives you the ability to view the BIOS, bootloader, bootup and shutdown procedures and console output should the machine hang or lock up, just as you would if you were interacting with the machine locally. This is called Serial Over Lan (SOL) and is available in IPMI v2.0 as a standard and using non-standard proprietary methods in v1.5.

Essentially, IPMI will save you from a few hundred to over a thousand GBP instead of buying a remote power control unit and SOL will save you the same instead of buying an IP KVM.

IPMI Revisions

There are currently 3 IPMI revisions (with details taken from http://www.ecst.csuchico.edu/~dranch/LINUX/IPMI/ipmi-on-linux.html):

  • IPMI v1.0 - Autonomous access, logging and control. IPMI messaging command sets, sensor data records and event messages. Access through system interfaces like memory mapped IO, I2C bus etc.
  • IPMI v1.5 - Ability to send IPMI messages to BMC over LAN, LAN alerting. No SOL.
  • IPMI v2.0 - Serial Over LAN enabling console redirection, access control, enhanced authentication, packet encryption, SMbus interface.

SOL is desirable as it allows you to interact with the server as though it were local in cases where the operating system locks up and SSH or telnet access are not available due to the operating system being inoperable. IPMI v1.5 still allows to you to power the system on and off and view sensor output.

Glossary

List of IPMI terms. Taken wholesale from http://www.ecst.csuchico.edu/~dranch/LINUX/IPMI/ipmi-on-linux.html

  • BMC : Baseboard Management Controllers. IPMI compliant micro controllers that handle system event management. These are usually available as cPCI cards.
  • GPCagent : A SuperMicro proprietary “Graceful Power” control agent. This agent provides graceful power control features for both Linux and windows platform. GPC means the OS will shutdown gracefully before a power shutdown. It is available from the SuperMicro website. It run as a daemon (smagent) on a Linux based managed system. It requires openIPMI kernel modules to be installed on the managed system to interact with the IPMI device.
  • i2c : A low speed (>=400khz) system management interface supported on most embedded systems. It is similar to the SMBus.
  • ipmicli : A SuperMicro proprietary command line interface for Linux, similar in function to IPMIView, available from the SuperMicro website.
  • IPMItool : An opensource tool for accessing the IPMI device through either local or remote access. Its a command line tool that can used to perform various commands for reading and writing to the IPMI device. It is equivalent to the ipmicli proprietary tool except for console redirection which is not available on this tool.
  • IPMIView : A SuperMicro proprietary java applet available from SuperMicro website. Runs on both windows (tested on windows2k) and Linux (fc3) platforms. This runs on the remote system and can be used to interact with the IPMI interface on the managed system. This software provides sensor monitoring, secure login, LAN/IP configuration, chassis power control and console redirection terminal. It also provides a graceful power shutdown/restart option that requires a daemon running on the managed system.
  • ipnmac : A SuperMicro proprietary command line tool for Linux to set the IP and mac address for the ipmi interface. This tool can be used to set the address locally on the managed system.
  • Managed system : system which is to be managed using IPMI. The IPMI card is installed on this machine. IPMI v2.0 supports both local and remote access to the BMC. Local access is provided through a system interface like kCS (IO port). Remote access is provided through the onboard LAN interface (on IPMI supported motherboards).
  • OpenIPMI : An opensource IPMI project that maintains linux drivers for the IPMI device. These drivers run on the managed system and provide a local interface to the IPMI card. They also support a primitive command line utility, equivalent to the ipmicli. The utility is meant more as a sample than a working tool.
  • Remote system : System from which the IPMI enabled server is managed. This is usually over the network.
  • SMbus : System Management bus. A low speed (<100khz) system management interface supported on most PC and server motherboards. It is similar to the I2C bus. The BMC uses the SMbus to communicate with motherboard sensors and Ethernet interface.
  • SuperoDoctor : A SuperMicro proprietary IPMI tool. Verision II is a command line tool for local access on the managed system to IPMI interface. Version III is a GUI based tool for local access but works only on windows platform. This requires the openIPMI kernel modules to be installed on the managed machine.

Installation

This guide covers the installation of IPMI tools on Dell 1425 servers and Supermicro servers with a PDSMi+ motherboard. The instructions will be relevent for other server models, but I make no promises. I make references in the links section to Supermicro X7DVL based hardware which comes with a different IPMI BMC but I haven't spent much time investigating the hardware.

For Supermicros servers, open up the server chassis and make a note of the MAC address on the IPMI port, you'll need this later.

In the BIOS, set the concole redirection to COM2, which is the BMC console port, it doesn't phsically exist). Disable BIOS redirection after POST, choose a baud rate (19200 is recommended a lot), a terminal type (vt100 works ok for me) and leave the other settings as they are.

Supermicro BMCs require you to boot from a CD and use their tool to flash the BMC prior to use, refer to the FTP link below and go up a few directories to get the latest IPMI CD image (the one shipped with the server caused me problems). Burn the image to a CD and boot from it. Use the utility to install the firmware for your IPMI version and then your motherboard version. I had to guess whether to use RCMP+ or RCMP. I have the Supermicro AOC-IPMI20-E and it wouldn't work with RCMP+, even though it will upload a firmware for you. It worked after I reflashed it with the RCMP version.

When the process finished, use the ipnmac utility while still booted from the CD, by typing ipnmac (if you're not in the right directory you'll have to navigate using cd and dir commands to find it). Give it a unique IP address from any other interface on the machine or on your network. The BMC needs to be network addressable in its own right. Also give it the MAC address you took from the IPMI socket on the motherboard, not the one written on the LAN ports as suggested by the official Supermicro docs.

Once this step is done, hit CTRL-Alt-Del and remove the CD.

For Dell 1425 servers (and probably other Dells), hit Alt-3 when prompted to enter the BMC setup and give it some unique network settings and some user settings (make your passwords secure!).

You can boot now into Linux, or from a Linux installer CD.

Note on network configuration:

Both Dell 1425s and Supermicro PDSMi+ motherboards use the first LAN port to redirect the IPMI traffic over when using SOL. For this reason, whether you intend to use SOL or not, it is a good idea to use the first LAN ports, almost certainly eth0, as your local network interface, rather than an Internet visible interface.

Once booted into Linux, install openipmi on all machines with a BMC and ipmitool on every machine from which you wish to run IPMI commands locally or to send IPMI commands to a remote machine. Red Hat, FC and Centos users will have to install OpenIPMI and OpenIPMI-tools. SUSE and other Linux users will have figure out what to do themselves for ipmitool. There are IPMItool packages on the website if they're not in your package-shallow distros ;). OpenIPMI is not required to send ipmi commands to remote machines. You only need OpenIPMI where you want to run IPMI commands locally, manage the BMC locally from the OS (which you want to do if you have a BMC in the machine) or I presume to do console redirection over the BMC.

Next you need to load the kernel modules. Fedora/Red Hat/Centos people just need to run setup, open the services tool and check the ipmi box then run /etc/init.d/ipmi start. On Ubuntu I had to do the following, Red Hat and derivatives could try this if their devices aren't found when starting the service. Try modprobing ipmi_si without any options at first, then build up the options if it fails. If you're not sure which method your BMC uses, try leaving out the type= parameter as the module will figure it out.

For kernel 2.6.x:

modprobe ipmi_msghandler
modprobe ipmi_devintf
modprobe ipmi_si type=kcs ports=0xca8 regspacings=4

If ipmi_si won't load, look at the output of dmidecode for the base address of your IPMI BMC and then use that base address for the ports=<base address> module option. The default ports option is 0xca2 so if your BMC is at that address according to dmidecode, then you don't need this option.

Kernel 2.4 people will have to follow the Debian IPMI instructions, as you're living in a world I haven't encountered with IPMI. You should note that ipmi_si is called ipmi_si_drv and its regspacings option is called si_regspacings. You may also have to make your own device node if you're not using devfs, as documented in the Debian instructions. After modprobing the relevent modules successfully, ls -l /dev/ipmi0 to see if you have a device node before trying to create one.

If this works without errors then Fedora/RH/Centos people are set, Ubuntu/Debian people need to add the modules and options to /etc/modules.conf or maybe add the modprobe commands to /etc/init.d/local if you have no other way.

If you then cat /proc/devices, you should see your IPMI device listed and ls -l /dev/ipmi0 should show your device node.

Setting Up Serial Consoles

Neither of the 2 brands of BMCs I have set up have a physical serial port, they are logical and are managed by the BMC.

You should have set up your BIOS for console redirection earlier, so now we will do the bootloader and init.

To allow your bootloader to redirect over the BMC's serial port, edit /boot/grub/grub.conf, sometimes known as /boot/grub/menu.list.

Disable splash screens by commenting out anything starting with splash outside of the OS boot menu section and remove any splash options from your kernel lines. A text console can't display them.

Add console=tty0 console=ttyS1,19200n8r to the end of your kernel lines, so it should look something like the following:

title           Ubuntu, kernel 2.6.15-26-amd64-server
root            (hd0,0)
kernel          /vmlinuz-2.6.15-26-amd64-server root=/dev/sda2 ro quiet console=tty0 console=ttyS1,19200n8r
initrd          /initrd.img-2.6.15-26-amd64-server
savedefault
boot

This gives you a serial console on the second serial port, which should be your BMC's serial port. The order of the console options above os above. The last listed will be the system's default console which will display the boot messages and kernel errors. This means that after halfway through the boot process, during shutdown and when there are kernel errors, only the serial console will see the messages. As you want to work remotely, this is the way it has to be. You can't have more than one default console.

About halfway through the bootup procedure, once the kernel has booted and loaded drivers for your hardware and has mounted the hard disks and so on, the bootloader hands over to init which brings up your services and network configurations etc. To get init and therefore your booting/booted Linux system redirecting the console over the BMC's serial port, edit /etc/inittab and add the following line to the console section for Debian/Ubuntu:

S1:2345:respawn:/sbin/getty -L ttyS1 19200 vt100

For Fedora/Red Hat/CentOS:

S1:2345:respawn:/sbin/agetty -h ttyS1 19200 vt100

This gives you a serial console on the second serial port, which should be your BMC's serial port, for the BIOS as configured earlier, the bootloader and init.

IPMI Commands

You should then be able to run some IPMI commands locally (probably need to be root as root owns the device node):

root@gw01:~# ipmitool -I open chassis power status
Chassis Power is on
root@gw01:~# ipmitool -I open chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     :
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
Sleep Button Disable : allowed
Diag Button Disable  : allowed
Reset Button Disable : allowed
Power Button Disable : allowed
Sleep Button Disabled: true
Diag Button Disabled : true
Reset Button Disabled: true
Power Button Disabled: true

Command dissection:

ipmitool -I open chassis power status

-I open           - use the local openipmi interface.
chassis           - run a command from the chassis set of commands.
power             - run a command relating to power.
status            - show the status of the chassis power status.

Thats pretty simple.

Assuming you get a local response, then try a remote command from another machine, you won't need to be root as you are talking to the BMC directly, rather than through the OS.

adam@ns0:~$ ipmitool -I lan -H 192.168.10.123 -U root -a chassis power status
Password:
Chassis Power is on
adam@ns0:~$ ipmitool -I lan -H 192.168.10.123 -U root -a chassis status
Password:
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     :
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
Sleep Button Disable : allowed
Diag Button Disable  : allowed
Reset Button Disable : allowed
Power Button Disable : allowed
Sleep Button Disabled: true
Diag Button Disabled : true
Reset Button Disabled: true
Power Button Disabled: true

Command dissection:

ipmitool -I lan -H 192.168.10.123 -U ipmiadmin -a chassis power status

-I lan                     - Run the command over the LAN.
-H 192.168.10.123          - -H means host followed by an IP address or hostname.
-U root                    - -U specifies a user, followed by the username set up on the BMC, not a local or remote OS user.
-a                         - prompt for password.
chassis power status       - as per the local command dissection.

If that all works for you then you can a woop and a holler.

IPMItool has a familiar UNIX or Cisco IOS syntax, which allows you to complete the command as you go along. If you type in an incomplete command, IPMItool will provide you with a list of options that can be used to complete your command.

Simply typing ipmitool -I open will offer the top-level sub-commands, which you can then choose from to add to your command.

root@gw01:~# ipmitool -I open
No command provided!
Commands:
      raw           Send a RAW IPMI request and print response
      i2c           Send an I2C Master Write-Read command and print response
      lan           Configure LAN Channels
      chassis       Get chassis status and set power state
      event         Send pre-defined events to MC
      mc            Management Controller status and global enables
      sdr           Print Sensor Data Repository entries and readings
      sensor        Print detailed sensor information
      fru           Print built-in FRU and scan SDR for FRU locators
      sel           Print System Event Log (SEL)
      pef           Configure Platform Event Filtering (PEF)
      sol           Configure IPMIv2.0 Serial-over-LAN
      isol          Configure IPMIv1.5 Serial-over-LAN
      user          Configure Management Controller users
      channel       Configure Management Controller channels
      session       Print session information
      sunoem        OEM Commands for Sun servers
      shell         Launch interactive IPMI shell
      exec          Run list of commands from file
      set           Set runtime variable for shell and exec
root@gw01:~# ipmitool -I open user
User Commands: summary [<channel number>]
                 list    [<channel number>]
                 set name     <user id> <username>
                 set password <user id> [<password>]
                 disable      <user id>
                 enable       <user id>
                 test         <user id> <16|20> [<password]>
root@gw01:~# ipmitool -I open chassis
Chassis Commands:  status, power, identify, policy, restart_cause, poh, bootdev
root@gw01:~# ipmitool -I open chassis power
chassis power Commands: status, on, off, cycle, reset, diag, soft

…And so on.

Beware, powering the server off or rebooting does not do a graceful shutdown like the OS would. They are the same as holding in the power button or pressing the reboot button. Supermicro offer a graceful shutdown daemon for Windows and Linux which will respond to the graceful showdown and reboot commands provided by Supermicro's graphical Java IPMI control application IPMIview.

Setting Up the BMC

The Supermicro BMCs send out arp requests gratuitously which can degrade performance, so we will turn them off:

ipmitool lan set 1 arp generate off

We will also set the BMC's netmask as well as it's default and backup gateways:

ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.1.1
ipmitool lan set 1 bakgw ipaddr 192.168.1.2

You can also set the gateway MAC addresses if you want to.

And now set it's SNMP community name so that we can send SNMP traps:

ipmitool lan set 1 snmp <community name>

Graphical IPMI

There is a Java application from Supermicro called IPMIView which allows you to graphically manage servers with IPMI BMCs. The primary advantage of this tool is that SOL works. The version which is shipped on the CD and on the latest CD image from the Supermicro FTP site doesn't work on FC6 or F7. Another version, shipped on the Supermicro FTP site for our dual processor machines is at ftp://ftp.supermicro.com/CDR-SIMIPMI_1.11_for_SIM_IPMI/IPMI_Solution/Linux/Administrator and this appears to work with Fedora 7.

Install as a regular user using sudo from the command line (by uncommenting the following line from /etc/sudoers if need to):

%wheel        ALL=(ALL)       ALL

…and adding your user to the wheel group:

usermod -G wheel <username>

Make the installer executable:

chmod +x ./IPMIView-Linux_2.6.31_071005.bin

and run it as sudo:

sudo ./IPMIView-Linux_2.6.31_071005.bin

It should now run. When the install finishes you need to chmod +x the bundled JRE directory as it's root only by default:

sudo chmod -R +x /opt/SUPERMICRO/IPMIView/_jvm/

Now you should be able to run

/opt/SUPERMICRO/IPMIView/IPMIView20.bin

as a regular user and create a desktop icon for it by right-clicking the desktop, clicking Create Launcher and providing the command as above and any other meta-data you wish to add, like description, name and and a nice icon from the installation directory.

Using the IPMIView application is covered in the Supermicro documentation (seems to be the only thing covered in them too…)

Adding an IPMI User With IPMIView

You should now be able to search for the host's IPMI device address in IPMIView and log in with the username and password of ADMIN. You should add a new user, change it's user level to administrator and delete the ADMIN user.

IPMI Standards

IPMI Software

IPMItool

OpenIPMI Linux IPMI kernel module support

Link to Supermicro patched version of IPMItool

Supermicro FTP server, Linux IPMI tools directory (version number in URL subject to change)

Newer version of IPMITool (version number in URL subject to change)

Supermicro bootable iso images for flashing BMCs (get the one with IPMI_Server_Managment.iso in the name)

IPMI Software Documentation

IPMI Howtos

Vendor IPMI Documentation

Dell IPMI howto Contains useful sample commands.

Other IPMI Documentation

Cern IPMI presentation PDF containing demonstration of IPMI usage, including reference to a version of IPMItool patched by Supermicro.

Todo

  • Include relevant info instead of referring to other docs in case they disappear, such as kernel 2.4 info from Debian howto
  • Cover Graceful Shutdown Daemon
ipmi_on_linux.1195667287.txt.gz · Last modified: 2016/11/25 22:38 (external edit)