Torque: some background details

Access to your /home files

The worker nodes on our clusters see the same set of user home files that the master node does. This means that we are operating with a shared home file system. This is not always the case in Torque/PBS setups, but it is the case on ours. There are many benefits of using a shared home file system.

The job script

When you submit a job, you can either pipe the command(s) into qsub, or you can specify the name of the job script file as an argument to qsub. A simple example of a pipe submission is: echo 'date;  gcc myprog.c'  |  qsub . An example of a file submission is qsub myfile.job . In either case, there may be options you need to specify to qsub.

What you submit for processing is a script, in Unix/Linux parlance. This means that it is a human-readable set of commands. When creating a job script file, you will probably use a plain-text editor (and not Word!). Such files are processed by a shell, which is a system-provided command which takes commands as input and processes them.

If you don't specify a shell, then the default one will usually be bash, which is a typical Unix/Linux shell. Your input can be a mixture of comments (preceded by a # character), qsub options (preceded by #PBS), normal commands, and possibly built-in commands providing some control structure. If you're not familiar with a shell, then don't worry: you can do most things without using control commands or understanding how it all works.

The script file named on the qsub command is read in full at qsub-time and the contents submitted as part of the job. If you wish, you can then safely modify that script file or even delete it without affecting the job. When the script starts running, the current directory is your home directory (and not the directory from which you did the qsub command), unless you specify a different directory in a qsub -d option.

Multiple jobs

The qsub'd script file is unique in that its contents are copied as part of the job and there is no harm in modifying it or deleting it before the job runs. However, files that the job might use are not treated that way: Torque does not scan your script to see what files it might use and whether it might be useful to take a copy of them! And it certainly can't tell what files a compiled program might use, whose filenames might be embedded in the program's source.

So it's up to you to make sure that the files that a job needs exist and have the right contents for that job at the time the job runs. This is of course normally achieved by creating them before the qsub and not touching them until that job has finished!

If you are going to submit and/or run multiple similar (but slightly different) jobs at the same time, then you have to be careful how they are submitted. See Torque: multiple jobs for some ideas of tacking this issue.

Environmental variables

Certain environmental variables, see the table below, are available within the Torque/PBS environment. Your script may test or use these variables if you wish. (You can export additional variables using the -v option of qsub).

For example, your job could change to the directory you were working in at the time of the qsub by having the following in your script (remembering that by default a job starts execution in your home directory):

           cd  "$PBS_O_WORKDIR"

Bear in mind that you may want to use the same script to test things interactively before using qsub. In that case, be careful how you use these PBS variables, because they are undefined except within a running job. In the above example, if you follow its format exactly, including the double-quotes, there will be no change of directory except within a job, which is probably exactly what you want. if that's too subtle for your taste, just cd to the required directory explicitly!


Env Variable Value For example
PBS_O_HOME HOME value at qsub time /home/filer1/lowel
PBS_O_HOST Hostname at qsub time escience1
PBS_O_INITDIR set to the value of the qsub -d option directory (undefined by default)
PBS_O_LOGNAME LOGNAME value at qsub time lowel
PBS_O_MAIL MAIL value at qsub time /var/mail/lowel
PBS_O_PATH PATH value at qsub time /usr/local/bin:/bin:....
PBS_O_SHELL SHELL value at qsub time /bin/bash
PBS_O_QUEUE the name of queue to which the job was submitted esq
PBS_O_WORKDIR the absolute current directory at qsub time /home/filer1/lowel/proj1
PBS_ARRAYID For job arrays (qsub -t), the assigned id for this job instance 1
PBS_ENVIRONMENT set to PBS_BATCH or to PBS_INTERACTIVE PBS_BATCH
PBS_JOBCOOKIE a hexadecimal string unique to the job (unpredictable)
PBS_JOBID the job identifier assigned by the batch system 539.escience1
PBS_JOBNAME the job name defaulted or given by the user (-N)
 mytest
PBS_NODEFILE the filename of nodelist file (main node script only) /var/spool/pbs/aux/539.escience1
PBS_NODENUM
 0 for main or only node, 1:n-1 for others
 0
PBS_VNODENUM Under pbsdsh: 0 to c-1 where c is the number of cores
 0
PBS_TASKNUM
 Under pbsdsh: a unique task-instance counter
 2
PBS_QUEUE name of the queue from which the job is executed eslong
PBS_ROOTDIR For qsub -D, the system root dir for the job /image/SL4



L.S.Lowe