Kermit Project   C-Kermit   C-Kermit Tutorial   Kermit Script Library   ilosetup   ilodomany

iLO BLADE SERVER SETUP AUTOMATION SCRIPTS

Frank da Cruz
Columbia University
As of: ilosetup 1.15 / Mon Feb 7 13:14:21 2011

Columbia University HPC Cluster January 2009
Photo by Megan Pengelly-Anderson
Columbia University, like many other large institutions and companies, is replacing older computers with thin, rackmountable, interchangeable, modular computers called blades, in our case the ones from Hewlett-Packard, ProLiant 64-bit processors capable of running a variety of operating systems, including Windows and Linux.

The HP blade servers are set up and configured through an "Onboard Administrator" called Integrated Lights-Out (iLO), which is part of the machine itself, and not dependent on any operating system or other software. A Web-based GUI management tool is provided for manual configuration, testing, and monitoring of the blades, and there is also a text-mode "back door" accessible by Telnet (in iLO 2 but not iLO 3 and later) or SSH that provides a traditional command-line management interface.

When a company finds itself installing dozens or hundreds of blade servers, manual configuration through the Web interface is prohibitively repetitive, labor-intensive, and error-prone.

The task can be handled more efficiently and with less risk by using a program that automates the configuration dialog with each blade to yield a consistent configuration across any number of blades. A good tool for doing this is C-Kermit because it is designed specifically for automating dialogs across communications connections, including Telnet and SSH.

Two iLO configuration scripts have been written for C-Kermit, to be used on any reasonably modern Unix-based computer (Linux, Mac OS X, NetBSD, Solaris, HP-UX, AIX, etc). These scripts have been used to set up the 200 blade systems that have arrived so far and, according to the Senior Systems Engineer in charge of the deploying the blades, "we plan on using it for all future HP purchases - blade and otherwise. The primary advantages of this script are that we no longer need to attach a monitor and keyboard to each machine to manually configure the iLO – adding users, enabling ssh, configuring hostnames."

The main script, ilosetup, configures a blade (or any other HP server that has iLO). It needs to know the IP hostname for the blade and the username and password to create on the blade. It can also be given various option parameters.

The second script is ilodomany which simply executes ilosetup a number of times in sequence; once for each hostname given on the command line. This is so the created user ID and password do not need to be hardwired into the script or stored on a disk, or entered manually for each server. Instead, the access info is given once to ilodomany and then passed to each instance of ilosetup in a relatively secure manner.

The Serials File

The iLO scripts depend on the existence of a "serials file", a plain-text file that contains one line of information for each blade server. As the Data Center manager explains, "This is the file that the Unix group uses as an inventory of hosts they maintain for several purposes. Going forward we'd hope to have a CUIT-wide inventory (or Data Center inventory) in a database that can be queried by the Unix, Windows, and DC groups (or any other group that needs it). So, I'm thinking that the script would have to be able to query this database for the info that it is currently getting from the serials file."

Columbia University HPC Cluster January 2009
Photo by Stew Feuerstein
Each line in the serials file contains the following fields, separated by whitespace:

  1. IP Hostname
  2. Serial number
  3. iLO MAC address (hex)
  4. eth0 MAC address (hex)
  5. eth1 MAC address (hex)
  6. factory-set iLO access password
  7. Warranty
In our case, this file already existed, and is updated every time a new blade arrives. Obviously the serials file should be well secured.

By convention, the iLO Console must be registered as hostname-console. So, for example, if the server's IP name is SERVER123.CC.COLUMBIA.EDU then the iLO Console's IP name is SERVER123-CONSOLE.CC.COLUMBIA.EDU.

How to Use the iLO Setup Script

You should run the iloesetup script in a 24x80 text terminal window emulating a DEC VT100-series terminal (VT100, VT102, VT220, VT320, etc). The screen dimensions are important because iLO components do screen formatting and positioning based on this size; if you use a longer screen length, you will see text writing over other text.

If you start ilosetup without any command-line arguments you get a usage message like this one:

Usage: [ ilo_username=x ] [ ilo_userpass=y ] ilosetup hostname [ options ]
Options:
  1. bios
  2. connect
  3. debug
  4. help
  5. iloonly
  6. nopassword
  7. nopoweroff
  8. noraid
  9. noreset
 10. notelnet
 11. poweroff
 12. pwrmode
 13. quiet
 14. raid
 15. reset
 16. ssh
 17. telnet
 18. usepassword
 19. verbose
Default options: verbose nopoweroff telnet reset usepassword raid
Optional environment variables:
 ilo_userid:   user ID to create
 ilo_userpass: password for user ID
 DEBUG=1: turn on debugging
If created user ID and password are not found in the environment,
you are prompted for them.
The options are:
quiet Silent running, suitable for a cron job
verbose (default) The configuration dialog is shown on the screen.
connect Instead of configuring the server, the script logs in automatically to iLO and, if successful, presents an interactive iLO console session to the user.
bios Like connect, but takes you to the BIOS rbsu> prompt for an interactive BIOS session. It's your responsibility to get out gracefully.
debug For use when testing or debugging the script.
telnet (default) Try to connect with Telnet first and if that doesn't work try ssh
ssh Try to connect with ssh first and if that doesn't work try Telnet. NOTE: In iLO 3 Telnet is gone; you must specify ssh.
notelnet This says that the host does not support Telnet at all, so don't try to connect to it with Telnet (i.e. use only ssh), and don't try to set any Telnet-related configuration parameters.
usepassword Use password from serials file for console login.
nopassword Use a blank password for console login.
iloonly Only do iLO commands, no BIOS or RAID configurations (which require the server be rebooted). This option can be run without disturbing the server.
raid Configure RAID. This is like reformatting the disks, everything on them will be erased. Requires prior power cycle. A logical RAID 1 disk is configured from Physical Disks 1 and 2, if and only if Physical Disks 1 and 2 exist and there is not already a Logical RAID-1 Disk 1.
noraid (default) Skip the RAID configuration.
poweroff Power the host off when finished.
nopoweroff (default) Don't power the host off when finished. NOTE: currently, the host is always powered off prior to giving BIOS commands, otherwise it is not possible to reach the BIOS. The nopoweroff option does not inhibit this. The only way to run this script without the server being power cycled is with the iloonly option or with the connect option.
reset (default) Reboot iLO when finished (new iLO settings don't take effect until iLO is restarted).
noreset Don't reboot iLO.
pwrmode:n Change the power regulator mode. The value of n must be 1, 2, 3, or 4:
1 = Dynamic power savings mode
2 = Static low power mode
3 = Static high performance mode
4 = OS control mode
This option is "standalone" in the sense that no other configuration commands are given during the session. Power is cycled before and after because it must be.

Example 1:
ilosetup server123 nopoweroff reset

Example 2:
ilosetup server456 pwrmode:3

Example 3:
ilosetup server789 ssh connect    (makes an interactive SSH terminal connection to the iLO prompt)

To pass options from ilodomany to ilosetup, include an options clause on the ilodomany command line:
options="someoption anotheroption someotheroption"

Example using ilodomany to configure several hosts the same way:
ilodomany server123 server124 server125 options="nopoweroff reset"

The password options, usepassword and nopassword, are because sometimes a blade is delivered with a blank console password instead of the password that the manufacturer claims to have assigned. The 'usepassword' option (default) means to try the factory-set password taken from the serials file first, and if that doesn't work, to try a blank password; the 'nopassword' option means try a blank password first, and if that doesn't work, try the password from the serials file.

As noted, the created username and password are picked up from environment variables ilo_username and ilo_password, if defined; otherwise the script prompts for them. Then it looks up the host in the serials file to get the iLO access password. Then it makes an SSH connection to the blade and runs through the configuration dialog, checking each operation for success or failure, and logging everything that happens. At Columbia, the configuration goes as follows:

  1. A session log is opened to record the dialog.
  2. A Telnet (or SSH) connection is attempted.
  3. If this fails, the other kind of connection is attempted.
    As delivered, the blades have Telnet disabled and SSH enabled. However some problems have been noted with the iLO SSH implementation so Telnet is preferred here. Therefore the first time the script is run on a blade, SSH is used, but then the script itself enables Telnet, so subsequent accesses can be with Telnet.

    iLO 3 does not support telnet. However, unless instructed otherwise ilosetup tries first to connect with telnet and when this fails, it tries SSH. The script still works but an unnecessary and fruitless step is taken on iLO 3 boxes. Include the 'notelnet' option on the command line to avoid it. If desired the script can be changed to always try SSH first.

  4. If no connection was made the script writes a message to the log and exits.
  5. The script attempts to log in to the iLO console; upon failure exits with failure status code.
  6. Once logged in, the script prompts the user for the created ID and password if these weren't found in the environment.
  7. With the created user ID and password in hand, the script issues iLO commands to:
  8. If the iloonly option was given, the script exits (first resetting iLO, unless the noreset option was given). Otherwise it proceeds with the following steps, which power the server off and on (which will, of course, disrupt any work that is being done on it).
  9. Now we check to see if the host is powered on or off. If it's on, we shut it down and power it up again (otherwise we just power it up) in order to be able to access the BIOS. Then we wait up to 100 seconds for an indication from the server that it actually is on, and if this doesn't come we fail and exit.
  10. Once the server is on we enter the Virtual Serial Port (VSP) subsystem. This can fail for a variety of reasons, all of which are checked for and handled appropriately.
  11. If the raid option was given, we enter the Arrays CLI, and if the blade has two physical disks, and it is not already configured to have a single logical RAID 1 drive, we configure it to have one, and then we return to VSP.
  12. From VSP enter the RBSU subsystem, which is the BIOS, or fail and exit if we can't.
  13. In the BIOS we issue a series of commands at the RBSU prompt to configure a virtual serial port, a serial console, and an EMS console, checking for errors along the way. The connection from the iLO console to VSP to RBSU is not well flow controlled, so the rate at which we send characters has to be throttled.
  14. We exit from RBSU, "escape back" to the iLO console by sending the iLO's escape sequence.
  15. If "poweroff at end" was requested, we power off the server.
  16. Finally we reset iLO itself, so the iLO changes will take effect, exit, and close the connection.

The exit status is 0 if the script succeeded, nonzero if it failed.

Logs

The results of each run are logged in the logging directory. The names of the logs are:
xxx-connect.log
xxx-session.log
where xxx is the simple hostname. The first one just shows the connection and login process, to be checked in case the script failed to connect or authenticate. The second one is a log of the dialog with the iLO console after logging in.

The logs are regular Kermit session logs, transcripts of characters that were received from the server during the session, intermixed with lines that start with “%%” that are progress messages from the script itself, which can be grepped for a quick summary of the run:

grep ^%% fillmore-session.log

In C-Kermit 8.0, the logs contain everything that came in, including escape sequences. With C-Kermit 9.0, the escape sequences are stripped out.

Implementation Notes

Click to view or download...
ilosetup ilodomany
ilosetup ilodomany

This script was written for C-Kermit 8.0.212 Dev.28, an internal working copy that is to be released as C-Kermit 9.0, which is much better adapted to modern platforms than the previous release (8.0.211 of April 2004). The iLO scripts use several features that are new to this version:
FSEEK /FIND:string Seeks to the spot in a file opened by FOPEN that contains the specified string. In earlier Kermit versions this would be accomplished by reading the file a line at a time and checking each line for the string.
The \fkwval() function This is a handy way to parse strings of the form foo=bar; it creates a macro named foo with the value bar (foo could be any single word, bar can be anything at all, and "=" is the default separator but others, such as ":" can be specified).
The PUTENV command This is what allows ilodomany to pass the created user ID and password to ilsosetup in a secure manner: they are not stored in any file, nor do they appear on the command line where they might be seen with the Unix 'ps' or 'w' commands.
The \fkwval() function is used to parse status messages resulting from iLO commands:

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND SYNTAX ERROR
or:
status=0
status_tag=COMMAND COMPLETED
Global settings modified.
thus allowing the script to easily check whether a command succeeded or failed, and if it failed, to get the error message. All Kermit has to do is check the value of its dynamically created STATUS macro, and if it isn't 0, then its ERROR_TAG macro will contain the message. (Unfortunately, this model is not used consistently throughout the iLO console and its subsystems, which makes the script somewhat inelegant in spots.)

These new features are not yet in Kermit 95 (the Windows version of C-Kermit) because they were added after the most recent K95 release. In any case, as written, the scripts might depend on some Unixisms.

iLO and RBSU commands are issued by macros that take care the error handling according to each command, because not all errors should be fatal; for example if the created ID already exists, that's not a fatal error. The RBSU command macro, as noted, takes extreme measures to get the command to its destination despite the traps and pitfalls of the multi-leg communication pathway.

As always, C-Kermit depends on the external ssh client to make SSH connections. All modern Unixes should have this client; if not, it can be downloaded from here. C-Kermit's Telnet client is built-in.

iLO 3

In December 2010 the script was adapted to iLO 3, which following the current "industry best practices" makes numerous arbitrary and capricious changes to the dialogs and messages on which this or any script must depend; for example, changing a certain message from "Duplicate login name" to "Duplicate user name". Also, setting some parameter which had previously been set to the same value is now an error, where previously it was accepted. Furthermore, some features have been taken away such as Telnet access. Certain prompts no longer appear where they once did. And so on. Note to posterity: text interfaces should be stable because other programs often depend on them.

The ilosetup script was adapted to these changes in such a way that it still works with iLO 2. When the connection is made, the script extracts the iLO version number from the greeting herald, and uses it to decide when do something special for iLO 3.

Adapting the scripts to Different Sites

A C-Kermit script is like a shell script; you have to give it execute permission:
$ chmod +x ilosetup ilodomany
and you have to change the top line of the script to indicate the path of the C-Kermit program (8.0.212 or later) that will execute it, e.g.:
#!/usr/local/bin/kermit +
(in C-Kermit the '+' is required to indicate that command-line arguments are to be processed by the script and not by Kermit itself, as explained in the Kermit Scripting Tutorial.)

To use the iLO scripts at a new site:


(End)