Torque: Multiple jobs

File: torquemulti. This update: 20091212.

This note is about multiple similar jobs submitted to the Torque batch system.

The qsub'd script file is unique in that its contents are copied as part of the job, and there is no harm in modifying it or deleting it before the job runs. However, files that the job might use are not treated that way: Torque does not scan your script to see what files it might use and whether it might be useful to take a copy of them! And it certainly can't tell what files a compiled program might use, whose filenames might be embedded in the program's source.

So it's up to you to make sure that the files that a job needs exist and have the right contents for that job at the time the job runs. This is of course normally achieved by creating them before the qsub and not touching them until that job has finished.

If you are going to submit and/or run multiple similar (but slightly different) jobs at the same time, then you have to be careful how they are submitted. For example, if the jobs internally use a data input file whose contents are to be different for each job, then you have to find a sensible way of achieving that; if you simply modified the data input file and submitted a job, and then repeated that exercise, all the jobs would probably run with the same (final) version of the data!

There are various techniques that people use for this issue - when this is done many times it's usually worth writing a script which does the repetitive work for you:

Any further suggestions welcome!