Torque: some background details

Access to your /home files

The worker nodes on the escience cluster see the same set of user home files that the escience1 master node does. This means that we are operating with a shared home file system. This is not always the case in Torque/PBS setups, but it is the case with this one. There are many benefits of using a shared home file system.

The job script

The file that you submit for processing is a script, in Unix/Linux parlance. This means that it is a human-readable file of commands, usually created by a plain-text editor (and not Word!). Such files are processed by a shell, which is a system-provided command which takes commands as input and processes them.

If you don't specify a shell, then the default one will usually be bash, which is a typical Unix/Linux shell. Your input can be a mixture of comments (preceded by a # character), qsub options (preceded by #PBS), normal commands, and possibly built-in commands providing some control structure. If you're not familiar with a shell, then don't worry: you can do most things without using control commands or understanding how it all works.

The script file named on the qsub command is read in full at qsub-time and the contents submitted as part of the job. If you wish, you can then safely modify that script file or even delete it without affecting the job.

When the script starts running, the current directory is your home directory (and not the directory from which you did the qsub command), unless you specify otherwise in a qsub option.

Multiple jobs

The qsub'd script file is unique in that its contents are copied as part of the job and there is no harm in modifying it or deleting it before the job runs. However, files that the job might use are not treated that way: Torque does not scan your script to see what files it might use and whether it might be useful to take a copy of them! And it certainly can't tell what files a compiled program might use, whose filenames might be embedded in the program's source.

So it's up to you to make sure that the files that a job needs exist and have the right contents for that job at the time the job runs. This is of course normally achieved by creating them before the qsub and not touching them until that job has finished!

If you are going to submit and/or run multiple similar (but slightly different) jobs at the same time, then you have to be careful how they are submitted. See Torque: multiple jobs for some ideas of tacking this issue.

Environmental variables

Certain environmental variables, see the table below, are available within the Torque/PBS environment. Your script may test or use these variables if you wish. (You can export additional variables using the -v option of qsub).

For example, your job could change to the directory you were working in at the time of the qsub by having the following in your script (remembering that by default a job starts execution in your home directory):

           cd  "$PBS_O_WORKDIR"

Bear in mind that you may want to use the same script to test things interactively before using qsub. In that case, be careful how you use these PBS variables, because they are undefined except within a running job. In the above example, if you follow its format exactly, including the double-quotes, there will be no change of directory except within a job, which is probably exactly what you want. if that's too subtle for your taste, just cd to the required directory explicitly!

Env Variable	Value	For example
PBS_O_HOME	HOME value at qsub time	/home/filer1/lowel
PBS_O_HOST	Hostname at qsub time	escience1
PBS_O_INITDIR	set to the value of the qsub -d option directory	(undefined by default)
PBS_O_LOGNAME	LOGNAME value at qsub time	lowel
PBS_O_MAIL	MAIL value at qsub time	/var/mail/lowel
PBS_O_PATH	PATH value at qsub time	/usr/local/bin:/bin:....
PBS_O_SHELL	SHELL value at qsub time	/bin/bash
PBS_O_QUEUE	the name of queue to which the job was submitted	esq
PBS_O_WORKDIR	the absolute current directory at qsub time	/home/filer1/lowel/proj1
PBS_ENVIRONMENT	set to PBS_BATCH or to PBS_INTERACTIVE	PBS_BATCH
PBS_JOBCOOKIE	a hexadecimal string unique to the job	(unpredictable)
PBS_JOBID	the job identifier assigned by the batch system	539.escience1
PBS_JOBNAME	the job name defaulted or given by the user (-N)	mytest
PBS_NODEFILE	the filename of nodelist file (main node script only)	/var/spool/pbs/aux/539.escience1
PBS_NODENUM	0 for main or only node, 1:n-1 for others	0
PBS_VNODENUM	Under pbsdsh: 0 to c-1 where c is the number of cores	0
PBS_TASKNUM	Under pbsdsh: a unique task-instance counter	2
PBS_QUEUE	name of the queue from which the job is executed	eslong
PBS_ROOTDIR	Root directory if specified in qsub -D option	/system/SL4

Queues

Torque/PBS has been set up so that jobs are submitted to a default queue, which then enqueues the jobs on the basis of required resources, typically CPU time, in different queues. There are limits on the number of jobs which can be processed simultaneously in each queue, and on the number of jobs each user or group of users can run simultaneously. This is done only to give a fair share to all users, is not rigid, and can be changed by contacting the mesc-cluster-approval team.

Debugging and Interactive Jobs

The master node can be used for compiling and debugging your programs to some extent, but be aware that the master is a shared resource and does not have as much memory as the worker nodes.

You can submit jobs which use your text-mode session as the input and output of the job, which can be useful for debugging. To do this, use the -I option of qsub, and do not supply a script (unless it just contains #PBS options). Note that your job will still be subject to CPU time and wall time restrictions. Please avoid tying up worker nodes for longer than necessary.

L.S.Lowe