BlueBEAR Cluster PP Guide

BlueBEAR

The system uses Torque and MOAB as the job manager and scheduler respectively. It uses the IBM GPFS system to provide the shared user filesystem. GPFS stands for General Parallel File System.

The system version

Using SL4 in a login session

To run just one particular script in SL4, you can do this without going into an SL4 login session. Let's say the first line of the script is #!/bin/sh. You can either enter:

         /bin4/sh myscript    any args

         myscript    any args

If you choose to do this /bin4/ script modification, note that inside the SL4 environment, the /bin4/ binaries also exist, but are simply soft-links to the /bin/ binaries, so behave in a consistent way, and so there is no need to keep a second copy of the script which has the conventional invocation as its first line.

Scripts which are invoked by other scripts already in an SL4 environment are run in that same SL4 environmnent. That applies in an interactive session and in a job. So there is no need to invoke those in a special way or go to the trouble of modifying them.

Using SL4 in a submitted job

To run a job totally within SL4, submit it with the qsub option -S /bin4/bash. That is, either use that option on the qsub command line, or put this in the submitted job script:

         #PBS -S /bin4/bash

Alternatively, you can choose that the script only (and not the initialisation of the job) is run under SL4, using one of the techniques described above for login sessions: using the /bin4/ binaries either to invoke a script, or as the first line of a script, possibly the job-script.

Knowing what system you're on

         release=$(lsb_release -r -s)
         echo You are running on release $release >&2
         PS1=$release-$PS1

(The redirection on the echo is important, as always inside a .bashrc, if you want scp and sftp to continue to work).

Differences from the BlueBEAR documentation

• it doesn't allow through the PBS variables like PBS_JOBID and PBS_O_WORKDIR,
• it doesn't accept arguments,
• it can't be used in a script #!interpreter line,
• it doesn't preserve the current directory,
• it doesn't preserve any changed umask.

So by preference, use the /bin4/ binaries. For any problems with those binaries, see me, rather than eHelpdesk.

How does it work?

Additional filesystems which you would normally expect to see in a native system (like the filesystem that contains your $HOME directory, and the /tmp filesystem) are specially mounted inside the SL4 image too.

When you enter one of the special SL4 commands like /bin4/sh, then the command changes the filesystem root to the root of the SL4 installation, changes the current directory to the same directory but in the SL4 image, and then invokes the command of the same name in the SL4 image. So /bin4/bash invokes bash in the changed-root system. Because your $HOME and other files are mounted in the SL4 image also, you can continue to see them.

(This is a bit different to the normal use of the chroot facility, which normally is used in order to give less access to system facilities, rather than similar access to a different system).

Your programs will therefore use the scripts and run-time libraries and other system files of the SL4 installation, rather than the native installation. So they are operating in a virtual application environment. Those libraries may call a system kernel interface to provide certain facilities, and in this case that kernel is the SL5 kernel, not a SL4 kernel. However, the assumption is that the kernel facilities are backward-compatible between SL5 and SL4.

As an analogy, consider a fully-resolved (static) program. If and when a program is compiled on one system with all library references resolved, and so with no run-time libraries, and you copy it to a second different system, you would expect it to work flawlessly, if the second system was later but backwards-compatible with the first system. It would be a very brave or foolhardy vendor or kernel developer who altered the kernel interfaces to cause such a static program to fail. In our case, we are not fully-resolving the library references but instead we are providing all the run-time libraries of the first system too; we rely equally on the backward-compatibility of the kernel interfaces, and so for the same reason we expect that method to work flawlessly too.

Why implement it that way?

Well, having worker nodes which run SL4 natively was certainly an option, and is still an option if for any reason the current technique fails to live up to expectations.

One thing in favour of the chosen method is that we don't need to consider the numbers game, of how many SL4 nodes and SL5 nodes. A worker node can run a mixture of SL4 and SL5 processes, rather than being dedicated to one or the other. Another benefit is running with a single kernel version, which the vendor believed would improve GPFS stability.

Another scenario could be to run the different systems virtually, using Xen or another virtualisation technique. However, that would involve GPFS running on different kernel versions, with two instances per worker node, and the cluster vendor was not in favour of that approach, having already declared a preference for having one kernel version for GPFS throughout the cluster.

Future of SL4 on BlueBEAR

Of course, having the two systems readily available makes it easy to test future migration to SL5, as will happen anyway. SL5 is a mature system, based on Red Hat Enterprise Linux (RHEL) 5, which was first available in March 2007.

There is a range of dates for the likely release of RHEL6, the earliest being March 2009.