Oxford Ganga Tutorial (17/12/2008)

Session 1 - General Ganga Usage

This session aims to go through the main principles of Ganga and it's general usage outside the particular experimental flavours.

Setting up and Running Ganga

Before running ganga properly, you should set up your own config file. To do this, run Ganga as follows:

 /home/slater/runGanga -g --disableLCG

The -g option sets up a file '.gangrc' in your home directory with a generic flavour. The options given in here will override anything else (except the command line). The --disableLCG option (specific to this script) turns off the Grid portion of Ganga. You are now ready to start Ganga properly:

   /home/slater/runGanga --disableLCG

If all is well, you should be presented with an IPython prompt (if you've never used IPython before, you will need to press return to get past the welcome message):

Ganga version not specified. Using LATEST version.
Running Ganga LATEST with the following parameters:
-o[LCG]EDG_ENABLE=False -o[LCG]GLITE_ENABLE=False


*** Welcome to Ganga ***
Version: Ganga-5-1-1
Documentation and support: http://cern.ch/ganga
Type help() or help('index') for online help.

This is free software (GPL), and you are welcome to redistribute it
under certain conditions; type license() for details.

GangaAtlas                         : INFO     Tasks read from file
Ganga.GPIDev.Lib.JobRegistry       : INFO     Found 0 jobs in "jobs", completed in 0 seconds
Ganga.GPIDev.Lib.JobRegistry       : INFO     Found 0 jobs in "templates", completed in 0 seconds


In [1]:         

Just to reiterate, this will setup Ganga in it's 'flavourless' mode - we will come on to using Atlas/LHCb software later!

Getting Help

Ganga is based completely on Python and so the usual Python commands can be entered at the IPython prompt. For the specific Ganga related parts, however, there is an online help system that can be accessed using:

In [1]: help() 
************************************

*** Welcome to Ganga ***
Version: Ganga-5-1-1
Documentation and support: http://cern.ch/ganga
Type help() or help('index') for online help.

This is free software (GPL), and you are welcome to redistribute it
under certain conditions; type license() for details.


This is an interactive help based on standard pydoc help.

Type 'index'  to see GPI help index.
Type 'python' to see standard python help screen.
Type 'interactive' to get online interactive help from an expert.
Type 'quit'   to return to Ganga.
************************************

help>                                   

Type 'index' at the prompt to see the Class list available. Then type the name of the particular object you're interested in to see the associated help. You can also do this directly from the IPython prompt using:

 In [1]: help(Job)

In preparation for the next exercise, access the help for the Job, Executable and Local objects.

Your First Job

Create a basic job object with default options and view it:

In [1]: j = Job()
In [2]: j
 Out[6]: Job (
 status = 'new' ,
 name = '' ,
 inputdir = '/home/slater/gangadir/workspace/mws/LocalAMGA/0/input/' ,
 outputdir = '/home/slater/gangadir/workspace/mws/LocalAMGA/0/output/' ,
 outputsandbox = [] ,
 id = 0 ,
 info = JobInfo (
    submit_counter = 0
    ) ,
 inputdata = None ,
 merger = None ,
 inputsandbox = [] ,
 application = Executable (
    exe = 'echo' ,
    env = {} ,
    args = ['Hello World']
    ) ,
 outputdata = None ,
 splitter = None ,
 subjobs = 'Job slice:  jobs(0).subjobs (0 jobs)
' ,
 backend = Local (
    actualCE = '' ,
    workdir = '' ,
    nice = 0 ,
    id = -1 ,
    exitcode = None
    )
 )

This is the summary of the object that Ganga uses to manage your job. These include the following parts:

  • application
  • backend
  • splitter
  • inputsandbox/outputsandbox
  • inputdata/outputdata
  • splitter

Now submit the job to the local backend by doing the following:

 In [3]: j.submit()

If all is well, the job will be submitted and you can then check it's progress using the following:

 In [4]: jobs

This will show a summary of all the jobs currently running. You're basic Hello World job will go through the following stages: 'submitted', 'running', 'completing' and 'completed'. When your job has reached the completed state, the standard output and error output are transferred to the output directory of the job (as listed in the job object). Have a look at them using:

 In [5]: !ls $j.outputdir 

where we have used the exclamation mark (!) to go into 'shell' mode and the dollar sign ($) to return to python. Check the output is valid by using the one of the following:

 In [6]: !emacs $j.outputdir/stdout
 In [7]: j.peek("stdout", "emacs") 

The peek command in the last line can be used to run a command (the second argument) on a file in the output directory (the first argument). These two lines are equivalent. With any luck, you will see Hello World! in the stdout. Congratulations, you've run your first job!

More on the Executable Application

The executable is the most basic Ganga application. For the job above, view the info about it using:

 In [8]: j.application  
Executable (
    exe = 'echo' ,
    env = {} ,
    args = ['Hello World']
    )

From this, you can see the exectuable to run ('echo'), any environemnt variables set ('env' - none in this case) and the arguments to give to the executable ('Hello World'). We'll now make a copy of this job object, change these parameters and resubmit the job:

 In [9]: j = jobs(0).copy()
 In [10]: j.application.exe = 'env' 
 In [11]: j.application.env['MYENVVAR'] = 'my environment variable'
 In [12]: j.application.args = []
 In [13]: j.submit()

After completion, check the output and your environment variable should be there, e.g.

In [13]: j.peek("stdout", "grep MYENVVAR")

You can also use the exectuable application to run your own scripts. First write the following into a file 'myscript.sh' and make it executable:

 #!/bin/env bash
echo "A simple script to list the directory contents:"

ls

Now, we'll create a new Executable application:

In [1]: exe_app = Executable()
In [2]: exe_app.exe = 'myscript.sh'

And now create a job object, assign the executable application we've just created and submit the job:

In [3]: j = Job()
In [4]: j.inputsandbox = [File('myscript.sh')]
In [5]: j.application = exe_app
In [6]: j.submit()

Note that we needed to add the script to the input sandbox so it got transferred with the job.

At this point, just to show the persistency of the job objects, exit Ganga (ctrl-D), restart and view your jobs with the 'jobs' command as before. All your jobs are stored and can be accessed using commands such as 'jobs()'.

Submitting to Different Backends

After looking at what can be done with the Executable application, we'll now look at the different backends that can be used. A 'backend' for Ganga describes where you want you job to actually run. Up until this point, we have been using the 'Local' backend which refers to the current computer you're using. Consequently, your computer name can be seen in the 'ActualCE' field. We'll now try using the local batch system to send jobs:

In [1]: j = Job()
In [1]: j.backend = PBS()
In [1]: j.submit()

Again, your job starts in the submitted state and then will go through running and completed states. That is all there is to running on the local batch system. Similarly, if you log in to lxplus and do the following, you can run on the LSF batch system there:

/afs/cern.ch/sw/ganga/install/5.1.2/bin/ganga
In [1]: j = Job()
In [1]: j.backend = LSF()
In [1]: j.submit()

Finally, it is also possible to submit a job to lxplus from Oxford using the Remote backend. By giving the host, username and ganga execution commands you can define a job locally and submit to the LSF queue at CERN (NOTE: do change the username and ganga_dir!):

In [1]: j = Job()
In [1]: j.backend = Remote()
In [1]: j.backend.host = "lxplus.cern.ch"
In [1]: j.backend.username = "mslater"
In [1]: j.backend.ganga_cmd = "/afs/cern.ch/sw/ganga/install/5.1.2/bin/ganga"
In [1]: j.backend.ganga_dir = "/afs/cern.ch/user/m/mslater/gangadir/remote_jobs"
In [1]: j.backend.remote_backend = LSF()
In [1]: j.submit()

Be aware that with this backend, when specifying input data, you are specifying the data at CERN, not locally! Just for experience, try submitting the previous jobs above (not just the basic Hello World job) to these other backends.

Basic Job Splitting and Merging

Where Ganga becomes very powerful is with the introduction of job splitting. This allows a single job to be defined but several to be jobs created. The splitting of jobs is controlled by the Job Splitter object - for these generic executable examples, we'll be using the ArgSplitter? which takes various argument options and creates a job to run on each. Here is an example that takes the basic 'echo' job and splits it with three arguments:

In [1]: s = ArgSplitter()
In [1]: s.args=[ ['A'], ['B'], ['C'] ]
In [1]: j.merger=TextMerger()
In [1]: j.merger.files=['stdout']
In [1]: j.merger.ignorefailed = True
In [1]: j = Job()
In [1]: j.splitter = s
In [1]: j.submit()

After the job has completed, the text merger completes automatically and you can view the output in the parent job output directory as before.

More Advanced Job Manipulation

Ganga provides many ways of manipulating jobs. Here I present a brief overview, but for the complete list have a look at:

http://ganga.web.cern.ch/ganga/user/html/GangaIntroduction/

Copying Jobs

You can copy a job regardless of it's status using the following:

j = Job()
j2 = j.copy()

The copied job is able to be submitted regardless of the original jobs status. Consequently, you can do the following:

j = jobs(2)
j.submit()

Job Status

Jobs can be killed and then resubmitted using the following:

j.kill()
j.resubmit()

The status of a job can be forced (e.g. if you think it has hung and you want to set it to failed) using the following:

j.force_status('failed')

Removing Jobs

To clean up your job repository, you can remove jobs using the 'remove' method:

j.remove()

-- MarkSlater - 24 Nov 2008


This topic: Computing > WebHome > Ganga > OxfordGangaTutorial
Topic revision: r4 - 03 Dec 2008 - _47C_61UK_47O_61eScience_47OU_61Birmingham_47L_61ParticlePhysics_47CN_61mark_32slater
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback