User Tools

Site Tools


ipmi_on_linux

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
ipmi_on_linux [2008/04/22 13:58]
adam
ipmi_on_linux [2016/11/25 22:38] (current)
Line 1: Line 1:
 ===== Running IPMI on Linux ===== ===== Running IPMI on Linux =====
 +
  
 ==== What is IPMI? ==== ==== What is IPMI? ====
  
  
-IPMI is standard which allows remote server management, primarliy developed by Intel. IPMI cards, known as Baseboard Management Cards (BMCs) are primitive computers in their own right and are operational all the time, so long as the server has a power source. The server itself does not need to be powered on, or the operating system operational for the BMC to work, it just needs a power source to be connected to the server.+IPMI is standard which allows remote server management, primarily developed by Intel. IPMI cards, known as Baseboard Management Cards (BMCs) are primitive computers in their own right and are operational all the time, so long as the server has a power source. The server itself does not need to be powered on, or the operating system operational for the BMC to work, it just needs a power source to be connected to the server.
  
 The primary benefits of IPMI are: The primary benefits of IPMI are:
Line 12: Line 13:
   * Ability to set up a console on a serial port and have the BMC redirect that console over a network port, which in cooperation with BIOS level console redirection, gives you the ability to view the BIOS, bootloader, bootup and shutdown procedures and console output should the machine hang or lock up, just as you would if you were interacting with the machine locally. This is called Serial Over Lan (SOL) and is available in IPMI v2.0 as a standard and using non-standard proprietary methods in v1.5.   * Ability to set up a console on a serial port and have the BMC redirect that console over a network port, which in cooperation with BIOS level console redirection, gives you the ability to view the BIOS, bootloader, bootup and shutdown procedures and console output should the machine hang or lock up, just as you would if you were interacting with the machine locally. This is called Serial Over Lan (SOL) and is available in IPMI v2.0 as a standard and using non-standard proprietary methods in v1.5.
  
-Essentially, IPMI will save you from a few hundred to over a thousand GBP instead of buying a remote power control unit and SOL will save you the same instead of buying an IP KVM.+Essentially, IPMI will save you from a few hundred to over a thousand GBP instead of buying a remote power control unit and SOL will save you the same amount again over buying an IP KVM. 
  
  
Line 20: Line 22:
  
   * IPMI v1.0 - Autonomous access, logging and control. IPMI messaging command sets, sensor data records and event messages. Access through system interfaces like memory mapped IO, I2C bus etc.   * IPMI v1.0 - Autonomous access, logging and control. IPMI messaging command sets, sensor data records and event messages. Access through system interfaces like memory mapped IO, I2C bus etc.
-  * IPMI v1.5 - Ability to send IPMI messages to BMC over LAN, LAN alerting. No SOL as part of standard but some vendor specific SOL implementations. +  * IPMI v1.5 - Ability to send IPMI messages to BMC over LAN, LAN alerting. No SOL as part of the standard's specification but some vendor specific SOL implementations. 
-  * IPMI v2.0 - Serial Over LAN enabling console redirection, access control, enhanced authentication, packet encryption, SMbus interface.+  * IPMI v2.0 - Serial Over LAN enabling console redirection, access control, enhanced authentication, packet encryption using RCMP+, SMbus interface.
  
-IPMI version 2.0 is desirable as it allows you to use SOL to get a remote console on the server as though it were local in cases where the operating system locks up and SSH or telnet access are not available due to the operating system being inoperable; v 2.0 also allows you to encrypt the contents of the IPMI packets to protect the passwords sent to remote systems. IPMI v1.5 still allows to you to power the system on and off and view sensor output. Both 2.0 and 1.5 are in common usage and are both still sold on new servers.+IPMI version 2.0 is desirable as it allows you to use SOL to get a remote console on the server as though it were local in cases where the operating system locks up and SSH or (heaven forbid) telnet access are not available due to the operating system being inoperable. v2.0 also allows you to encrypt the contents of the IPMI packets sent to remote systems and so protects the BMC passwords and your commands on the network. IPMI v1.5 still allows to you to power the system on and off and view sensor output, but does not support packet encryption (and therefore sends your BMC password over the network in plain text) and does not support SOL in any standardised way. Both 2.0 and 1.5 are in common usage and are both still sold on new servers.
  
 ==== Glossary ==== ==== Glossary ====
Line 52: Line 54:
  
   * SuperoDoctor : A SuperMicro proprietary IPMI tool.  Verision II is a command line tool for local access on the managed system to IPMI interface. Version III is a GUI based tool for local access but works only on windows platform. This requires the openIPMI kernel modules to be installed on the managed machine.   * SuperoDoctor : A SuperMicro proprietary IPMI tool.  Verision II is a command line tool for local access on the managed system to IPMI interface. Version III is a GUI based tool for local access but works only on windows platform. This requires the openIPMI kernel modules to be installed on the managed machine.
- 
- 
- 
- 
- 
- 
- 
- 
- 
- 
  
 ==== Installation ==== ==== Installation ====
  
-This guide covers the installation of IPMI tools on Dell 1425 servers and Supermicro servers with a PDSMi+ motherboard. The instructions will be relevent for other server models, but I make no promises. I make references in the links section to Supermicro X7DVL based hardware which comes with a different IPMI BMC but I haven't spent much time investigating the hardware.+This guide covers the installation of IPMI tools on Dell 1425 servers and Supermicro servers with a PDSMi+ motherboard. The instructions will be relevant for other server models, but I make no promises. I make references in the links section to Supermicro X7DVL based hardware which comes with a different IPMI BMC but I haven't spent much time investigating the hardware.
  
-For Supermicros servers, open up the server chassis and make a note of the MAC address on the IPMI port, you'll need this later.+For Supermicro servers, open up the server chassis and make a note of the MAC address on the IPMI port. You're supposed need this later when flashing the BMC with it's firmware, however I found that it wasn't necessary to tell the BMC it's MAC address as it already knew and offered it as a default when asking for it. The wise amongst you will write this down anyway and compare it to what the flashing utility says. Beware also that the Supermicro docs erroneously tell you to get the MAC address from the LAN ports, not the BMC socket, this is wrong, you need the MAC written on the BMC's socket.
  
-In the BIOS, set the concole redirection to COM2, which is the BMC console port, it doesn'phsically exist). Disable BIOS redirection after POST, choose a baud rate (19200 is recommended lot), a terminal type (vt100 works ok for me) and leave the other settings as they are.+In the BIOS, set the console redirection to COM2, which is the BMC console port, it doesn'physically exist). Disable BIOS redirection after POST, choose a baud rate (19200 is default on PDSMI+ motherboards, so we used it as default on everything to keep things tidy, Supermicro X7DVL motherboards which use a SIMIPMI BMC could use a number of different baud rates, your choice is up to you), a terminal type (vt100 works ok for me) and leave the other settings as they are.
  
-Supermicro BMCs require you to boot from a CD and use their tool to flash the BMC prior to use, refer to the FTP link below and go up a few directories to get the latest IPMI CD image (the one shipped with the server caused me problems). Burn the image to a CD and boot from it. Use the utility to install the firmware for your IPMI version and then your motherboard version. I had to guess whether to use RCMP+ or RCMP. I have the [[http://www.supermicro.com/products/accessories/addon/AOC-IPMI20-E.cfm|Supermicro AOC-IPMI20-E]] and it wouldn't work with RCMP+, even though it will upload a firmware for you. It worked after I reflashed it with the RCMP version.+Supermicro BMCs require you to boot from a CD and use their tool to flash the BMC prior to use, refer to the FTP link below and go up a few directories to get the latest IPMI CD image (the one shipped with the server caused me problems). Burn the image to a CD and boot from it. Use the utility to install the firmware for your IPMI version and then your motherboard version. I had to guess whether to use RCMP+ (meaning it supports encryption) or RCMP (meaning no encryption). I have the [[http://www.supermicro.com/products/accessories/addon/AOC-IPMI20-E.cfm|Supermicro AOC-IPMI20-E]] and it wouldn't work with RCMP+, even though it will upload a firmware for you. It worked after I reflashed it with the RCMP version.
  
-When the process finished, use the ipnmac utility while still booted from the CD, by typing ipnmac (if you're not in the right directory you'll have to navigate using cd and dir commands to find it). Give it a unique IP address from any other interface on the machine or on your network. The BMC needs to be network addressable in its own right. Also give it the MAC address you took from the IPMI socket on the motherboard, not the one written on the LAN ports as suggested by the official Supermicro docs.+When the process finished, use the ipnmac utility while still booted from the CD, by typing ipnmac (if you're not in the right directory you'll have to navigate using cd and dir commands to find it). Give it a unique IP address from any other interface on the machine or on your network. The BMC needs to be network addressable in its own right. Also give it the MAC address you took from the IPMI socket on the motherboard, not the one written on the LAN ports as suggested by the official Supermicro docs. As I said before, I found that the flashing utility offered the correct MAC address as a default at this stage, but it would be wise to check it against what you wrote down earlier.
  
 Once this step is done, hit CTRL-Alt-Del and remove the CD. Once this step is done, hit CTRL-Alt-Del and remove the CD.
Line 93: Line 85:
   modprobe ipmi_msghandler   modprobe ipmi_msghandler
   modprobe ipmi_devintf   modprobe ipmi_devintf
 +  modprobe ipmi_si
 +
 +If ipmi_si won't load, you may need to use:
 +
   modprobe ipmi_si type=kcs ports=0xca8 regspacings=4   modprobe ipmi_si type=kcs ports=0xca8 regspacings=4
  
-If ipmi_si won't load, look at the output of dmidecode for the base address of your IPMI BMC and then use that base address for the ''ports=<base address>'' module option.  The default ports option is ''0xca2'' so if your BMC is at that address according to ''dmidecode'', then you don't need this option. A SLES 10 user tells me that they did have to specify the ports value 0xca2 on an HP DL380 G5, so perhaps it isn't always the default.+Or look at the output of dmidecode for the base address of your IPMI BMC and then use that base address for the ''ports=<base address>'' module option.  The default ports option is ''0xca2'' so if your BMC is at that address according to ''dmidecode'', then you don't need this option. A SLES 10 user tells me that they did have to specify the ports value 0xca2 on an HP DL380 G5, so perhaps it isn't always the default.
  
 Kernel 2.4 people will have to follow the Debian IPMI instructions, as you're living in a world I haven't encountered with IPMI. You should note that ipmi_si is called ipmi_si_drv and its regspacings option is called si_regspacings. You may also have to make your own device node if you're not using devfs, as documented in the Debian instructions. After modprobing the relevent modules successfully, ''ls -l /dev/ipmi0'' to see if you have a device node before trying to create one. Kernel 2.4 people will have to follow the Debian IPMI instructions, as you're living in a world I haven't encountered with IPMI. You should note that ipmi_si is called ipmi_si_drv and its regspacings option is called si_regspacings. You may also have to make your own device node if you're not using devfs, as documented in the Debian instructions. After modprobing the relevent modules successfully, ''ls -l /dev/ipmi0'' to see if you have a device node before trying to create one.
  
-If this works without errors then Fedora/RH/Centos people are set, Ubuntu/Debian people need to add the modules and options to /etc/modules.conf or maybe add the modprobe commands to /etc/init.d/local if you have no other way.+If this works without errors then Fedora/RH/Centos people are set, Ubuntu/Debian people need to add the modules and options to /etc/modules or maybe add the modprobe commands to /etc/init.d/local if you have no other way.
  
 If you then ''cat /proc/devices'', you should see your IPMI device listed and ''ls -l /dev/ipmi0'' should show your device node. If you then ''cat /proc/devices'', you should see your IPMI device listed and ''ls -l /dev/ipmi0'' should show your device node.
 +
  
 ==== Setting Up Serial Consoles ==== ==== Setting Up Serial Consoles ====
Line 109: Line 106:
 You should have set up your BIOS for console redirection earlier, so now we will do the bootloader and init. You should have set up your BIOS for console redirection earlier, so now we will do the bootloader and init.
  
-To allow your bootloader to redirect over the BMC's serial port, edit /boot/grub/grub.conf, sometimes known as /boot/grub/menu.list.+To allow your bootloader to redirect over the BMC's serial port, for Grub v1, edit /boot/grub/grub.conf, sometimes known as /boot/grub/menu.list add the following lines to grub.conf or menu.list: 
 + 
 +  serial --unit=1 --speed=19200 --word=8 --parity=no --stop=1  
 +  terminal --timeout=10 serial console
  
 Disable splash screens by commenting out anything starting with splash outside of the OS boot menu section and remove any splash options from your kernel lines. A text console can't display them. Disable splash screens by commenting out anything starting with splash outside of the OS boot menu section and remove any splash options from your kernel lines. A text console can't display them.
  
-Add ''console=tty0 console=ttyS1,19200n8r'' to the end of your kernel lines, so it should look something like the following:+To make kernel messages output over your BMC, add ''console=tty0 console=ttyS1,19200n8r'' to the end of your kernel lines, so it should look something like the following:
  
   title           Ubuntu, kernel 2.6.15-26-amd64-server   title           Ubuntu, kernel 2.6.15-26-amd64-server
Line 122: Line 122:
   boot   boot
  
-This gives you a serial console on the second serial port, which should be your BMC's serial port. The order of the console options above os above. The last listed will be the system's default console which will display the boot messages and kernel errors. This means that after halfway through the boot process, during shutdown and when there are kernel errors, only the serial console will see the messages. As you want to work remotely, this is the way it has to be. You can't have more than one default console.+For Grub2, look [[http://linux.xvx.cz/2009/08/debian-with-grub2-and-serial-connection/|here]] 
 + 
 +This gives you a serial console on the second serial port, which should be your BMC's serial port, at 19200Kb per second, which should match what you chose in the BIOS. The order of the console options above is important. The last listed will be the system's default console which will display the boot messages and kernel errors. This means that after halfway through the boot process, during shutdown and when there are kernel errors, only the serial console will see the messages. As you want to work remotely, this is the way it has to be. You can't have more than one default console. You can however interrupt grub at boot time and edit the kernel line for a single boot with different parameters if you need to see the default console locally but remember that they will persist until you reboot, which means that if you reboot but still want to see them locally, you'll have to interrupt and edit grub at boot time again and also, to see them remotely again, you need to remember to reboot.
  
 About halfway through the bootup procedure, once the kernel has booted and loaded drivers for your hardware and has mounted the hard disks and so on, the bootloader hands over to init which brings up your services and network configurations etc. To get init and therefore your booting/booted Linux system redirecting the console over the BMC's serial port, edit /etc/inittab and add the following line to the console section for Debian/Ubuntu: About halfway through the bootup procedure, once the kernel has booted and loaded drivers for your hardware and has mounted the hard disks and so on, the bootloader hands over to init which brings up your services and network configurations etc. To get init and therefore your booting/booted Linux system redirecting the console over the BMC's serial port, edit /etc/inittab and add the following line to the console section for Debian/Ubuntu:
Line 132: Line 134:
   S1:2345:respawn:/sbin/agetty -h ttyS1 19200 vt100   S1:2345:respawn:/sbin/agetty -h ttyS1 19200 vt100
    
-This gives you a serial console on the second serial port, which should be your BMC's serial port, for the BIOS as configured earlier, the bootloader and init. +This gives you a serial console on the second serial port, which should be your BMC's serial port, for the BIOS as configured earlier, the bootloader and init. Again, change 19200 to whatever you chose in your BIOS.
- +
  
 +Users of Ubuntu (and presumably Debian and it's other derivative distributions) can read [[https://help.ubuntu.com/community/IPMI]] for assistance with all of this stuff. I guess it will be useful for everybody else too, if you can recognise where it is Ubuntu specific.
  
 ==== IPMI Commands ==== ==== IPMI Commands ====
Line 168: Line 169:
   ipmitool -I open chassis power status   ipmitool -I open chassis power status
      
-  -I open           - use the local openipmi interface.+  -I open           - use the local openipmi interface. It seems that when the interface is local, it can be omitted from the command and the command will default to the local interface.
   chassis           - run a command from the chassis set of commands.   chassis           - run a command from the chassis set of commands.
   power             - run a command relating to power.   power             - run a command relating to power.
Line 175: Line 176:
 Thats pretty simple. Thats pretty simple.
  
-Assuming you get a local response, then try a remote command from another machine, you won't need to be root as you are talking to the BMC directly, rather than through the OS.+Assuming you get a local response, then try a remote command from another machine, you won't need to be root as you are talking to the BMC directly, rather than through the OS. IPMI 2.0 BMCs support both encrypted (RCMP+) and non-encrypted (RCMP) IPMI traffic sent over the network, 1.5 BMCs only support non-encrypted traffic. Encrypted lan traffic is specified by the ''lanplus'' parameter, where as non-encrypted network traffic is specified by the ''lan'' parameter.
  
-  adam@ns0:~$ ipmitool -I lan -H 192.168.10.123 -U root -a chassis power status+  adam@ns0:~$ ipmitool -I lanplus -H 192.168.10.123 -U root -a chassis power status
   Password:   Password:
   Chassis Power is on   Chassis Power is on
-  adam@ns0:~$ ipmitool -I lan -H 192.168.10.123 -U root -a chassis status+  adam@ns0:~$ ipmitool -I lanplus -H 192.168.10.123 -U root -a chassis status
   Password:   Password:
   System Power         : on   System Power         : on
Line 204: Line 205:
 Command dissection: Command dissection:
  
-  ipmitool -I lan -H 192.168.10.123 -U ipmiadmin -a chassis power status+  ipmitool -I lanplus -H 192.168.10.123 -U ipmiadmin -a chassis power status
      
-  -I lan                     - Run the command over the LAN.+  -I lanplus                 - Run the command over the LAN with encryption for IPMI 2.0 hosts. You can use ''-I lan'' to send commands over the lan for 1.5 hosts as they don't support the encryption used by lanplus. 2.0 hosts can also use ''-I lan'' but why would you want to when 2.0 hosts can use encryption.
   -H 192.168.10.123          - -H means host followed by an IP address or hostname.   -H 192.168.10.123          - -H means host followed by an IP address or hostname.
-  -U root                    - -U specifies a user, followed by the username set up on the BMC, not a local or remote OS user. +  -U root                    - -U specifies a user, followed by the username set up on the BMC, this is not a local or remote OS user. 
-  -a                         - prompt for password.+  -a                         - prompt for password. You can also possible to supply the password as an environment variable instead of using -a, but I guess this is less secure.
   chassis power status       - as per the local command dissection.   chassis power status       - as per the local command dissection.
  
 If that all works for you then you can a woop and a holler. If that all works for you then you can a woop and a holler.
  
-IPMItool has a familiar UNIX or Cisco IOS syntax, which allows you to complete the command as you go along. If you type in an incomplete command, IPMItool will provide you with a list of options that can be used to complete your command.+IPMItool has a familiar UNIX bash shell or Cisco IOS syntax, which allows you to complete the command as you go along. If you type in an incomplete command, IPMItool will provide you with a list of options that can be used to complete your command.
  
 Simply typing ipmitool -I open will offer the top-level sub-commands, which you can then choose from to add to your command. Simply typing ipmitool -I open will offer the top-level sub-commands, which you can then choose from to add to your command.
Line 275: Line 276:
  
   ipmitool lan set 1 snmp <community name>   ipmitool lan set 1 snmp <community name>
 +
 +==== Getting a Remote Console Using IPMI====
 +
 +To get a Serial Over LAN console, assuming you have set up the various BMC redirections correctly, run:
 +
 +  ipmitool -I lanplus -H 192.168.10.123 -U ipmiadmin -a sol activate
 +
 +SOL only works on IPMI 2.0 BMCs and so only works with devices which support the encrypted lanplus interface (though my Supermicro's would do it without encryption using the graphical tool described below).
 +
 +If SOL doesn't work or you simply want to quit your SOL session, you can use the key sequence:
 +
 +  ~.
 +
 +to get out. It seems that inside a SOL session, all of ipmitool's SOL session commands start with the character ''~''.
 +
 +Alternatively you may use a graphical tool as described below.
  
 ==== Graphical IPMI ==== ==== Graphical IPMI ====
  
-Graphical IPMI tools make life easier as you don't have to remember commands or syntax and as most GUI tools are provided by the IPMI device vendors, they may offer vendor specific functionality that isn't available in the generic command line tools. Of course, in all cases except regarding vendor specific (and thus not standards compliant) functionality, the command line tools should provide all necessary functionality and so GUI tools are not essential.+Graphical IPMI tools make life easier as you don't have to remember commands or syntax and as most GUI tools are provided by the IPMI device vendors, they may offer vendor specific functionality that isn't available in the generic command line tools. Of course, in all cases except regarding vendor specific (and thus not standards compliant) functionality, the command line tools should provide all necessary functionality and so GUI tools are not essential to get IPMI working.
  
 This section describes the installation and use of Supermicro's IPMIView. I am told that Dell's OpenManage suite also offers some GUI IPMI tools but I can't vouch for that as the Dell servers I used didn't seem to come with OpenManage. There are surely other GUI IPMI tools from other vendors, I just don't know about them. This section describes the installation and use of Supermicro's IPMIView. I am told that Dell's OpenManage suite also offers some GUI IPMI tools but I can't vouch for that as the Dell servers I used didn't seem to come with OpenManage. There are surely other GUI IPMI tools from other vendors, I just don't know about them.
  
-There is a Java application from Supermicro called IPMIView which allows you to graphically manage servers with IPMI BMCs. The primary advantage of this tool is that SOL works easily. The version which is shipped on the CD (as of late 2007) doesn't work on FC6 or F7, however the latest version from the FTP site does. Another version, shipped on the Supermicro FTP site for our dual processor machines is at [[ftp://ftp.supermicro.com/CDR-SIMIPMI_1.11_for_SIM_IPMI/IPMI_Solution/Linux/Administrator]] and this appears to work with Fedora 7 also.+There is a Java application from Supermicro called IPMIView which allows you to graphically manage servers with IPMI BMCs. The primary advantage of this tool is that SOL works easily and you can issue graceful reboot and shutdown commands if you install Supermicro's graceful shutdown daemon. The version of IPMIView which is shipped on the CD (as of late 2007) doesn't work on FC6 or F7, however the latest version from the FTP site does. Another version, shipped on the Supermicro FTP site for our dual processor machines is at [[ftp://ftp.supermicro.com/CDR-SIMIPMI_1.11_for_SIM_IPMI/IPMI_Solution/Linux/Administrator]] and this appears to work with Fedora 7 also.
  
 Install as a regular user using sudo from the command line (by uncommenting the following line from /etc/sudoers if need to): Install as a regular user using sudo from the command line (by uncommenting the following line from /etc/sudoers if need to):
Line 315: Line 332:
  
 You should now be able to search for the host's IPMI device address in IPMIView and log in with the username and password of ADMIN. You should add a new user, change it's user level to administrator and delete the ADMIN user. You should now be able to search for the host's IPMI device address in IPMIView and log in with the username and password of ADMIN. You should add a new user, change it's user level to administrator and delete the ADMIN user.
 +
 +
 +
  
 ==== Links ==== ==== Links ====
Line 347: Line 367:
  
 [[ftp://ftp.supermicro.com/CDR-SIMIPMI_1.11_for_SIM_IPMI/Manuals|Supermicro IPMI Manuals for Supermicro X7DVL motherboard with SIM IPMI devices]] (version number in URL subject to change) [[ftp://ftp.supermicro.com/CDR-SIMIPMI_1.11_for_SIM_IPMI/Manuals|Supermicro IPMI Manuals for Supermicro X7DVL motherboard with SIM IPMI devices]] (version number in URL subject to change)
 +
 +The April 2008 issue, number 89, of Linux Pro Magazine (aka Linux Magazine outside of Canada and the USA), contains a decent IPMI article written by Justin Penney. This doesn't appear to be web visible at the moment.
  
 === IPMI Howtos === === IPMI Howtos ===
  
-[[http://buttersideup.com/docs/howto/IPMI_on_Debian.html|Debian IPMI howto]]+[[https://help.ubuntu.com/community/IPMI|Ubuntu IPMI documentation]] New for Ubuntu 8.04 
 + 
 +[[http://buttersideup.com/docs/howto/IPMI_on_Debian.html|Older Debian IPMI howto]]
  
 [[http://www.ecst.csuchico.edu/~dranch/LINUX/IPMI/ipmi-on-linux.html|Supermicro IPMI on Linux]] Contains errata on the Supermicro Docs [[http://www.ecst.csuchico.edu/~dranch/LINUX/IPMI/ipmi-on-linux.html|Supermicro IPMI on Linux]] Contains errata on the Supermicro Docs
ipmi_on_linux.1208869105.txt.gz · Last modified: 2016/11/25 22:38 (external edit)