This shows you the differences between two versions of the page.
ipmi_on_linux [2008/05/12 09:10] adam |
ipmi_on_linux [2016/11/25 22:38] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ===== Running IPMI on Linux ===== | ||
- | |||
- | |||
- | ==== What is IPMI? ==== | ||
- | |||
- | |||
- | IPMI is standard which allows remote server management, primarily developed by Intel. IPMI cards, known as Baseboard Management Cards (BMCs) are primitive computers in their own right and are operational all the time, so long as the server has a power source. The server itself does not need to be powered on, or the operating system operational for the BMC to work, it just needs a power source to be connected to the server. | ||
- | |||
- | The primary benefits of IPMI are: | ||
- | |||
- | * View server chassis and motherboard sensor output remotely, such as chassis status and intrusion detection. | ||
- | * Ability to remotely power on, power off, reboot the server and flash the identification light. | ||
- | * Ability to set up a console on a serial port and have the BMC redirect that console over a network port, which in cooperation with BIOS level console redirection, | ||
- | |||
- | Essentially, | ||
- | |||
- | |||
- | |||
- | ==== IPMI Revisions ==== | ||
- | |||
- | There are currently 3 IPMI revisions (with details taken from [[http:// | ||
- | |||
- | * IPMI v1.0 - Autonomous access, logging and control. IPMI messaging command sets, sensor data records and event messages. Access through system interfaces like memory mapped IO, I2C bus etc. | ||
- | * IPMI v1.5 - Ability to send IPMI messages to BMC over LAN, LAN alerting. No SOL as part of the standard' | ||
- | * IPMI v2.0 - Serial Over LAN enabling console redirection, | ||
- | |||
- | IPMI version 2.0 is desirable as it allows you to use SOL to get a remote console on the server as though it were local in cases where the operating system locks up and SSH or (heaven forbid) telnet access are not available due to the operating system being inoperable. v2.0 also allows you to encrypt the contents of the IPMI packets sent to remote systems and so protects the BMC passwords and your commands on the network. IPMI v1.5 still allows to you to power the system on and off and view sensor output, but does not support packet encryption (and therefore sends your BMC password over the network in plain text) and does not support SOL in any standardised way. Both 2.0 and 1.5 are in common usage and are both still sold on new servers. | ||
- | |||
- | ==== Glossary ==== | ||
- | |||
- | List of IPMI terms. Taken wholesale from [[http:// | ||
- | |||
- | * BMC : Baseboard Management Controllers. IPMI compliant micro controllers that handle system event management. These are usually available as cPCI cards. | ||
- | |||
- | * GPCagent : A SuperMicro proprietary " | ||
- | |||
- | * i2c : A low speed (> | ||
- | |||
- | * ipmicli : A SuperMicro proprietary command line interface for Linux, similar in function to IPMIView, available from the SuperMicro website. | ||
- | |||
- | * IPMItool : An opensource tool for accessing the IPMI device through either local or remote access. Its a command line tool that can used to perform various commands for reading and writing to the IPMI device. It is equivalent to the ipmicli proprietary tool except for console redirection which is not available on this tool. | ||
- | |||
- | * IPMIView : A SuperMicro proprietary java applet available from SuperMicro website. Runs on both windows (tested on windows2k) and Linux (fc3) platforms. This runs on the remote system and can be used to interact with the IPMI interface on the managed system. This software provides sensor monitoring, secure login, LAN/IP configuration, | ||
- | |||
- | * ipnmac : A SuperMicro proprietary command line tool for Linux to set the IP and mac address for the ipmi interface. This tool can be used to set the address locally on the managed system. | ||
- | |||
- | * Managed system : system which is to be managed using IPMI. The IPMI card is installed on this machine. IPMI v2.0 supports both local and remote access to the BMC. Local access is provided through a system interface like kCS (IO port). Remote access is provided through the onboard LAN interface (on IPMI supported motherboards). | ||
- | |||
- | * OpenIPMI : An opensource IPMI project that maintains linux drivers for the IPMI device. These drivers run on the managed system and provide a local interface to the IPMI card. They also support a primitive command line utility, equivalent to the ipmicli. The utility is meant more as a sample than a working tool. | ||
- | |||
- | * Remote system : System from which the IPMI enabled server is managed. This is usually over the network. | ||
- | |||
- | * SMbus : System Management bus. A low speed (< | ||
- | |||
- | * SuperoDoctor : A SuperMicro proprietary IPMI tool. Verision II is a command line tool for local access on the managed system to IPMI interface. Version III is a GUI based tool for local access but works only on windows platform. This requires the openIPMI kernel modules to be installed on the managed machine. | ||
- | |||
- | ==== Installation ==== | ||
- | |||
- | This guide covers the installation of IPMI tools on Dell 1425 servers and Supermicro servers with a PDSMi+ motherboard. The instructions will be relevant for other server models, but I make no promises. I make references in the links section to Supermicro X7DVL based hardware which comes with a different IPMI BMC but I haven' | ||
- | |||
- | For Supermicro servers, open up the server chassis and make a note of the MAC address on the IPMI port. You're supposed need this later when flashing the BMC with it's firmware, however I found that it wasn't necessary to tell the BMC it's MAC address as it already knew and offered it as a default when asking for it. The wise amongst you will write this down anyway and compare it to what the flashing utility says. Beware also that the Supermicro docs erroneously tell you to get the MAC address from the LAN ports, not the BMC socket, this is wrong, you need the MAC written on the BMC's socket. | ||
- | |||
- | In the BIOS, set the console redirection to COM2, which is the BMC console port, it doesn' | ||
- | |||
- | Supermicro BMCs require you to boot from a CD and use their tool to flash the BMC prior to use, refer to the FTP link below and go up a few directories to get the latest IPMI CD image (the one shipped with the server caused me problems). Burn the image to a CD and boot from it. Use the utility to install the firmware for your IPMI version and then your motherboard version. I had to guess whether to use RCMP+ (meaning it supports encryption) or RCMP (meaning no encryption). I have the [[http:// | ||
- | |||
- | When the process finished, use the ipnmac utility while still booted from the CD, by typing ipnmac (if you're not in the right directory you'll have to navigate using cd and dir commands to find it). Give it a unique IP address from any other interface on the machine or on your network. The BMC needs to be network addressable in its own right. Also give it the MAC address you took from the IPMI socket on the motherboard, | ||
- | |||
- | Once this step is done, hit CTRL-Alt-Del and remove the CD. | ||
- | |||
- | For Dell 1425 servers (and probably other Dells), hit Alt-3 when prompted to enter the BMC setup and give it some unique network settings and some user settings (make your passwords secure!). | ||
- | |||
- | You can boot now into Linux, or from a Linux installer CD. | ||
- | |||
- | Note on network configuration: | ||
- | |||
- | Both Dell 1425s and Supermicro PDSMi+ motherboards use the first LAN port to redirect the IPMI traffic over when using SOL. For this reason, whether you intend to use SOL or not, it is a good idea to use the first LAN ports, almost certainly eth0, as your local network interface, rather than an Internet visible interface. | ||
- | |||
- | Once booted into Linux, install openipmi on all machines with a BMC and ipmitool on every machine from which you wish to run IPMI commands locally or to send IPMI commands to a remote machine. Red Hat, FC and Centos users will have to install OpenIPMI and OpenIPMI-tools. SUSE and other Linux users will have figure out what to do themselves for ipmitool. There are IPMItool packages on the website if they' | ||
- | |||
- | Next you need to load the kernel modules. Fedora/Red Hat/Centos people just need to run setup, open the services tool and check the ipmi box then run / | ||
- | |||
- | For kernel 2.6.x: | ||
- | |||
- | modprobe ipmi_msghandler | ||
- | modprobe ipmi_devintf | ||
- | modprobe ipmi_si type=kcs ports=0xca8 regspacings=4 | ||
- | |||
- | If ipmi_si won't load, look at the output of dmidecode for the base address of your IPMI BMC and then use that base address for the '' | ||
- | |||
- | Kernel 2.4 people will have to follow the Debian IPMI instructions, | ||
- | |||
- | If this works without errors then Fedora/ | ||
- | |||
- | If you then '' | ||
- | |||
- | |||
- | ==== Setting Up Serial Consoles ==== | ||
- | |||
- | Neither of the 2 brands of BMCs I have set up have a physical serial port, they are logical and are managed by the BMC. | ||
- | |||
- | You should have set up your BIOS for console redirection earlier, so now we will do the bootloader and init. | ||
- | |||
- | To allow your bootloader to redirect over the BMC's serial port, edit / | ||
- | |||
- | serial --unit=1 --speed=19200 --word=8 --parity=no --stop=1 | ||
- | terminal --timeout=10 serial console | ||
- | |||
- | Disable splash screens by commenting out anything starting with splash outside of the OS boot menu section and remove any splash options from your kernel lines. A text console can't display them. | ||
- | |||
- | To make kernel messages output over your BMC, add '' | ||
- | |||
- | title | ||
- | root (hd0,0) | ||
- | kernel | ||
- | initrd | ||
- | savedefault | ||
- | boot | ||
- | |||
- | This gives you a serial console on the second serial port, which should be your BMC's serial port, at 19200Kb per second, which should match what you chose in the BIOS. The order of the console options above is important. The last listed will be the system' | ||
- | |||
- | About halfway through the bootup procedure, once the kernel has booted and loaded drivers for your hardware and has mounted the hard disks and so on, the bootloader hands over to init which brings up your services and network configurations etc. To get init and therefore your booting/ | ||
- | |||
- | S1: | ||
- | |||
- | For Fedora/Red Hat/CentOS: | ||
- | |||
- | S1: | ||
- | |||
- | This gives you a serial console on the second serial port, which should be your BMC's serial port, for the BIOS as configured earlier, the bootloader and init. Again, change 19200 to whatever you chose in your BIOS. | ||
- | |||
- | Users of Ubuntu (and presumably Debian and it's other derivative distributions) can read [[https:// | ||
- | |||
- | ==== IPMI Commands ==== | ||
- | |||
- | You should then be able to run some IPMI commands locally (probably need to be root as root owns the device node): | ||
- | |||
- | root@gw01: | ||
- | Chassis Power is on | ||
- | root@gw01: | ||
- | System Power : on | ||
- | Power Overload | ||
- | Power Interlock | ||
- | Main Power Fault : false | ||
- | Power Control Fault : false | ||
- | Power Restore Policy : always-off | ||
- | Last Power Event : | ||
- | Chassis Intrusion | ||
- | Front-Panel Lockout | ||
- | Drive Fault : false | ||
- | Cooling/Fan Fault : false | ||
- | Sleep Button Disable : allowed | ||
- | Diag Button Disable | ||
- | Reset Button Disable : allowed | ||
- | Power Button Disable : allowed | ||
- | Sleep Button Disabled: true | ||
- | Diag Button Disabled : true | ||
- | Reset Button Disabled: true | ||
- | Power Button Disabled: true | ||
- | |||
- | Command dissection: | ||
- | |||
- | ipmitool -I open chassis power status | ||
- | | ||
- | -I open - use the local openipmi interface. It seems that when the interface is local, it can be omitted from the command and the command will default to the local interface. | ||
- | chassis | ||
- | power - run a command relating to power. | ||
- | status | ||
- | |||
- | Thats pretty simple. | ||
- | |||
- | Assuming you get a local response, then try a remote command from another machine, you won't need to be root as you are talking to the BMC directly, rather than through the OS. IPMI 2.0 BMCs support both encrypted (RCMP+) and non-encrypted (RCMP) IPMI traffic sent over the network, 1.5 BMCs only support non-encrypted traffic. Encrypted lan traffic is specified by the '' | ||
- | |||
- | adam@ns0:~$ ipmitool -I lanplus -H 192.168.10.123 -U root -a chassis power status | ||
- | Password: | ||
- | Chassis Power is on | ||
- | adam@ns0:~$ ipmitool -I lanplus -H 192.168.10.123 -U root -a chassis status | ||
- | Password: | ||
- | System Power : on | ||
- | Power Overload | ||
- | Power Interlock | ||
- | Main Power Fault : false | ||
- | Power Control Fault : false | ||
- | Power Restore Policy : always-off | ||
- | Last Power Event : | ||
- | Chassis Intrusion | ||
- | Front-Panel Lockout | ||
- | Drive Fault : false | ||
- | Cooling/Fan Fault : false | ||
- | Sleep Button Disable : allowed | ||
- | Diag Button Disable | ||
- | Reset Button Disable : allowed | ||
- | Power Button Disable : allowed | ||
- | Sleep Button Disabled: true | ||
- | Diag Button Disabled : true | ||
- | Reset Button Disabled: true | ||
- | Power Button Disabled: true | ||
- | |||
- | Command dissection: | ||
- | |||
- | ipmitool -I lanplus -H 192.168.10.123 -U ipmiadmin -a chassis power status | ||
- | | ||
- | -I lanplus | ||
- | -H 192.168.10.123 | ||
- | -U root - -U specifies a user, followed by the username set up on the BMC, this is not a local or remote OS user. | ||
- | -a - prompt for password. You can also possible to supply the password as an environment variable instead of using -a, but I guess this is less secure. | ||
- | chassis power status | ||
- | |||
- | If that all works for you then you can a woop and a holler. | ||
- | |||
- | IPMItool has a familiar UNIX bash shell or Cisco IOS syntax, which allows you to complete the command as you go along. If you type in an incomplete command, IPMItool will provide you with a list of options that can be used to complete your command. | ||
- | |||
- | Simply typing ipmitool -I open will offer the top-level sub-commands, | ||
- | |||
- | root@gw01: | ||
- | No command provided! | ||
- | Commands: | ||
- | raw Send a RAW IPMI request and print response | ||
- | i2c Send an I2C Master Write-Read command and print response | ||
- | lan | ||
- | chassis | ||
- | event Send pre-defined events to MC | ||
- | mc Management Controller status and global enables | ||
- | sdr Print Sensor Data Repository entries and readings | ||
- | sensor | ||
- | fru Print built-in FRU and scan SDR for FRU locators | ||
- | sel Print System Event Log (SEL) | ||
- | pef | ||
- | sol | ||
- | isol Configure IPMIv1.5 Serial-over-LAN | ||
- | user Configure Management Controller users | ||
- | channel | ||
- | session | ||
- | sunoem | ||
- | shell | ||
- | exec Run list of commands from file | ||
- | set Set runtime variable for shell and exec | ||
- | root@gw01: | ||
- | User Commands: summary [< | ||
- | | ||
- | set name < | ||
- | set password <user id> [< | ||
- | | ||
- | | ||
- | | ||
- | root@gw01: | ||
- | Chassis Commands: | ||
- | root@gw01: | ||
- | chassis power Commands: status, on, off, cycle, reset, diag, soft | ||
- | |||
- | ...And so on. | ||
- | |||
- | Beware, powering the server off or rebooting does not do a graceful shutdown like the OS would. They are the same as holding in the power button or pressing the reboot button. Supermicro offer a graceful shutdown daemon for Windows and Linux which will respond to the graceful showdown and reboot commands provided by Supermicro' | ||
- | |||
- | ==== Setting Up the BMC ==== | ||
- | |||
- | The Supermicro BMCs send out arp requests gratuitously which can degrade performance, | ||
- | |||
- | ipmitool lan set 1 arp generate off | ||
- | |||
- | We will also set the BMC's netmask as well as it's default and backup gateways: | ||
- | |||
- | ipmitool lan set 1 netmask 255.255.255.0 | ||
- | ipmitool lan set 1 defgw ipaddr 192.168.1.1 | ||
- | ipmitool lan set 1 bakgw ipaddr 192.168.1.2 | ||
- | |||
- | You can also set the gateway MAC addresses if you want to. | ||
- | |||
- | And now set it's SNMP community name so that we can send SNMP traps: | ||
- | |||
- | ipmitool lan set 1 snmp < | ||
- | |||
- | ==== Getting a Remote Console Using IPMI==== | ||
- | |||
- | To get a Serial Over LAN console, assuming you have set up the various BMC redirections correctly, run: | ||
- | |||
- | ipmitool -I lanplus -H 192.168.10.123 -U ipmiadmin -a sol activate | ||
- | |||
- | SOL only works on IPMI 2.0 BMCs and so only works with devices which support the encrypted lanplus interface (though my Supermicro' | ||
- | |||
- | If SOL doesn' | ||
- | |||
- | ~. | ||
- | |||
- | to get out. It seems that inside a SOL session, all of ipmitool' | ||
- | |||
- | Alternatively you may use a graphical tool as described below. | ||
- | |||
- | ==== Graphical IPMI ==== | ||
- | |||
- | Graphical IPMI tools make life easier as you don't have to remember commands or syntax and as most GUI tools are provided by the IPMI device vendors, they may offer vendor specific functionality that isn't available in the generic command line tools. Of course, in all cases except regarding vendor specific (and thus not standards compliant) functionality, | ||
- | |||
- | This section describes the installation and use of Supermicro' | ||
- | |||
- | There is a Java application from Supermicro called IPMIView which allows you to graphically manage servers with IPMI BMCs. The primary advantage of this tool is that SOL works easily and you can issue graceful reboot and shutdown commands if you install Supermicro' | ||
- | |||
- | Install as a regular user using sudo from the command line (by uncommenting the following line from / | ||
- | |||
- | %wheel | ||
- | |||
- | ...and adding your user to the wheel group: | ||
- | |||
- | usermod -G wheel < | ||
- | |||
- | Make the installer executable: | ||
- | |||
- | chmod +x ./ | ||
- | |||
- | and run it as sudo: | ||
- | |||
- | sudo ./ | ||
- | |||
- | It should now run. When the install finishes you need to chmod +x the bundled JRE directory as it's root only by default: | ||
- | |||
- | sudo chmod -R +x / | ||
- | |||
- | Now you should be able to run | ||
- | |||
- | / | ||
- | |||
- | as a regular user and create a desktop icon for it by right-clicking the desktop, clicking Create Launcher and providing the command as above and any other meta-data you wish to add, like description, | ||
- | |||
- | Using the IPMIView application is covered in the Supermicro documentation (seems to be the only thing covered in them too...) | ||
- | |||
- | ==== Adding an IPMI User With IPMIView ==== | ||
- | |||
- | You should now be able to search for the host's IPMI device address in IPMIView and log in with the username and password of ADMIN. You should add a new user, change it's user level to administrator and delete the ADMIN user. | ||
- | |||
- | |||
- | |||
- | |||
- | ==== Links ==== | ||
- | |||
- | === IPMI Standards === | ||
- | |||
- | [[http:// | ||
- | |||
- | [[http:// | ||
- | |||
- | === IPMI Software === | ||
- | |||
- | [[http:// | ||
- | |||
- | [[http:// | ||
- | |||
- | [[https:// | ||
- | |||
- | [[ftp:// | ||
- | |||
- | [[ftp:// | ||
- | |||
- | [[ftp:// | ||
- | |||
- | === IPMI Software Documentation === | ||
- | |||
- | [[http:// | ||
- | |||
- | [[http:// | ||
- | |||
- | [[ftp:// | ||
- | |||
- | [[ftp:// | ||
- | |||
- | The April 2008 issue, number 89, of Linux Pro Magazine (aka Linux Magazine outside of Canada and the USA), contains a decent IPMI article written by Justin Penney. This doesn' | ||
- | |||
- | === IPMI Howtos === | ||
- | |||
- | [[https:// | ||
- | |||
- | [[http:// | ||
- | |||
- | [[http:// | ||
- | |||
- | [[http:// | ||
- | |||
- | [[http:// | ||
- | |||
- | === Vendor IPMI Documentation === | ||
- | |||
- | [[http:// | ||
- | |||
- | === Other IPMI Documentation === | ||
- | |||
- | [[http:// | ||
- | |||
- | === Related Documentation === | ||
- | |||
- | [[http:// | ||
- | |||
- | ==== Todo ==== | ||
- | |||
- | * Include relevant info instead of referring to other docs in case they disappear, such as kernel 2.4 info from Debian howto | ||
- | * Cover Graceful Shutdown Daemon | ||