BlueBEAR Grid UI

A guide to submitting jobs to the grid via BlueBEAR.

Introduction

The BlueBEAR log in nodes provide a Grid User Interface, that can be used by anybody to submit jobs to the grid. By default, jobs submitted from here will be submitted to the BlueBEAR grid nodes (representing about 10% of the cluster), but it is possible to submit jobs to other sites on the grid. A good introduction to the grid can be found on the GridPP website.

The Birmingham grid nodes all run Scientific Linux 5, and this is true for most worker nodes on the grid.

Limitations

When creating jobs to run in a grid environment, it is important to take note of a few limitations. For example, MPI is generally not supported on grid enabled clusters. Your home directory and software caches that are normally available on BlueBEAR will not be accessible on the grid nodes.

Jobs submitted to the grid should be written to be as robust as possible. One common problem is that jobs will not run under your username. They will instead be mapped to a grid pool account, so when writing bash scripts no hard references to local paths or usernames should be included - liberal use of the $HOME and $USER environment variables is highly recommended!

For more help and information, please email lcg-site-admin@hep.ph.bham.ac.uk.

Setup environment

Before submitting a job to the grid, you need to set up the Grid User Interface environment. This can be done by simply executing the script

   source /apps/hep/lcgui/lcguisetup

on one of the BlueBEAR login nodes. This will give you full access to all the tools you will need to submit jobs to the grid.

Grid Certificates

In order to do anything on the grid, you will need a digital certificate to prove who you are and membership of a Virtual Organisation (VO). For help with this please email lcg-site-admin@hep.ph.bham.ac.uk.

You can apply for a grid certificate here. There is a good guide on how to install your grid certificate here.

Unless you already know which VO you should be a member of (for example, particle physics users working on the ATLAS experiment should join the atlas vo), you should join the National Grid Service VO ( ngs.ac.uk). You can apply for NGS membership here.

Exporting your Grid Certificate

After a grid certificate has been issued, it will be stored in your web browser. In order to use the command line tools on BlueBEAR this certificate will need to be exported and converted into the right format.

To export the certificate ( in firefox!):

1. Click "Edit -> Preferences ->Advanced -> Encryption -> View Certificates"
2. Select the certificate and click "Backup".
3. Choose an appropriate name.

Copy this certificate into your home directory on BlueBEAR. Having exported the certificate you then need to convert it into the correct format. Make the directory $HOME/.globus, if it doesn't already exist, and use the following commands to convert the certificate:

   openssl pkcs12 -in your_exported_certificate.p12 -clcerts -nokeys -out ~/.globus/usercert.pem
   openssl pkcs12 -in your_exported_certificate.p12 -nocerts -out ~/.globus/userkey.pem

Creating a Proxy

Assuming that you already have a grid certificate available on the login nodes, and that you are a member of a suitable VO, you will need to create a proxy certificate before you can perform any tasks on the grid. The proxy certificate is a short lived instance of your grid credentials. When you submit a job to the grid, your proxy certificate will be automatically submitted along with the job, enabling access to, for example, computing elements and storage.

You can create a proxy certificate by using the command:

   voms-proxy-init --voms ngs.ac.uk

Note that the VO string (in this example ngs.ac.uk) can and should be substituted for your own VO (eg biomed or atlas etc). This command will prompt you for your grid certificate password. Once created, a proxy is valid for 12 hours, though it is possible to create a proxy with a longer life span if required using the -valid=XX:00 switch (where XX is the lifespan of the proxy in hours).

You can check a proxy by using the command voms-proxy-info -all.

Submitting via gqsub

The simplest way of submitting a job to the grid from BlueBEAR is to just use the gqsub command. Full details on how to use this command can be found here. Assuming that you have a suitable bash script (ie no hardcoded usernames, no references to BlueBEAR software caches etc), you can submit the job to the grid using the command:

   gqsub my_bash_script.sh

Your job will then be submitted by default to the BlueBEAR grid nodes, via the computing element epgr04.ph.bham.ac.uk. You can check the status of your job using the command gqstat.

Submitting WMS

In general, in order to submit a job to the grid the job needs to be specified in a JDL file. This file specifies the name of the executable or script, any input files, output files and most importantly, resource requirements. This JDL file, along with your bash script and proxy certificate are then sent to a Workload Management System (WMS). The WMS will compare the list of resource requirements you specified and submit your job to a suitable site. This can, theoretically, be anywhere in the world.

The gqsub script writes a simple JDL file based on your bash script, and take care of submission to the WMS. By default it specifies in the JDL file that jobs should run at Birmingham on the BlueBEAR worker nodes. It may be desirable to submit jobs to other sites (

A good guide to writing your own JDL files, along with the relevant commands to submit to a WMS can be found here.

Submitting via CreamCE

The WMS is a good, standardised method for submitting jobs to the grid. However, there can be a significant lag between a job completing on a WN and the job status on the WMS updating (20 minutes+ is not unusual). This can make submitting via the WMS less viable if you have a short job.

The latest computing elements (called Cream CEs) allow jobs to be directly submitted through a different set of commands. These tend to update their status much more quickly. Full details can be found here. Direct submission to the BlueBEAR Cream CE can be achieved using the command:

glite-ce-job-submit -a -o cream.id -r epgr07.ph.bham.ac.uk:8443/cream-pbs-glong my_jdl_file.jdl

It is possible to check the status of jobs using the command glite-ce-job-status -i cream.id

-- ChristopherCurtis - 15 Dec 2010