AliceGridAnalysis < ALICE

ALICE Web>AliceComputing>AliceGridAnalysis (01 Dec 2010, _47C_61UK_47O_61eScience_47OU_61Birmingham_47L_61ParticlePhysics_47CN_61arvinder_32palaha? ) (raw view)
EditAttach
---++ Running on the GRID

As more real data and MC production becomes available it will only be possible to access it all on the GRID. This page explains how to run an analysis task on the GRID.
---+++ Prerequisites

You need to be running on a computer with an up to date version of Alien and recent versions of ROOT, GEANT and !AliRoot. Installation of these packages is not covered here. You should consult the ALICE offline <a href="http://aliweb.cern.ch/Offline/" title="New offline site">pages</a>, in particular <a href="http://alien2.cern.ch/index.php?option=com_content&view=article&id=44&Itemid=118" title="Alien user installation">Alien</a> and <a href="http://aliweb.cern.ch/Offline/AliRoot/Installation.html#ROOT" title="ROOT Installation">ROOT</a>. There is also a useful guide to maintaining multiple combinations of versions <a href="http://newton.ph.unito.it/~berzano/w/doku.php?id=alice:compile" title="Dario's guide">here</a>.

You should also have an analysis task which can successfully be compiled and run locally and on a PROOF cluster (eg CAF or SKAF). The first requirement should ensure that when your code is sent onto the GRID it can be compiled on each worker node as necessary. The second condition should allow the output from your task to merged at the end as PROOF also does this. The task should have the usual run.C macro for running it.
---+++ Introduction

The simplest means to run on the GRID is to use the Alien plugin. This involves modifying one configuration macro followed by some changes to the macro used to run your task. For clarity these will be dealt with separately below. The very first task is to download the tutorial example from <a href="http://aliweb.cern.ch/secure/Offline/sites/aliceinfo.cern.ch.secure.Offline/files/uploads/AnalysisTrain/alienplugin.tgz" title="Alien tutorial code">http://aliweb.cern.ch/secure/Offline/sites/aliceinfo.cern.ch.secure.Offline/files/uploads/AnalysisTrain/alienplugin.tgz</a> as advised <a href="https://groups.cern.ch/group/alice-project-analysis-task-force/Lists/Archive/Flat.aspx?RootFolder=%2fgroup%2falice-project-analysis-task-force%2fLists%2fArchive%2fProblem+with+alienplugin+in+test+mode&FolderCTID=0x0120020020057F7ECFBF5441A3407964C7042DEF" title="Gheata email">here</a>.

This should be untar'd in a clean directory.
---+++ Configuration Modifications

Here is the !CreateAlienHandler.C macro. The lines with red portions show the changes made.
<pre>AliAnalysisGrid* CreateAlienHandler()</pre><pre>{</pre><pre>// Check if user has a valid token, otherwise make one. This has limitations.</pre><pre>// One can always follow the standard procedure of calling alien-token-init then</pre><pre>// source /tmp/gclient_env_$UID in the current shell.</pre><pre>  if (!AliAnalysisGrid::CreateToken()) return NULL;</pre><pre>  AliAnalysisAlien *plugin = new AliAnalysisAlien();</pre><pre>// Set the run mode (can be "full", "test", "offline", "submit" or "terminate")</pre><pre>  plugin->SetRunMode("full");</pre><pre>// Set versions of used packages</pre><pre>  plugin->SetAPIVersion("V1.1x");</pre><pre>  plugin->SetROOTVersion("%RED%v5-26-00b%ENDCOLOR%");</pre><pre>  plugin->SetAliROOTVersion("%RED%v4-19-04-AN%ENDCOLOR%");</pre><pre style="margin: 1em 0px; overflow: scroll; font-family: Verdana,Arial,Helvetica,sans-serif; font-size: 10px; width: auto; color: #7a4707">// Declare input data to be processed.</pre><pre>// Method 1: Create automatically XML collections using alien 'find' command.</pre><pre>// Define production directory LFN</pre><pre>  plugin->SetGridDataDir("%RED%/alice/data/2010/LHC10b%ENDCOLOR%");%</pre><pre>  // On real reconstructed data:</pre><pre>// plugin->SetGridDataDir("/alice/data/2009/LHC09d");</pre><pre>// Set data search pattern</pre><pre>  plugin->SetDataPattern("*ESDs.root");</pre><pre>// Data pattern for reconstructed data</pre><pre>// plugin->SetDataPattern("*ESDs/pass4/*ESDs.root");</pre><pre> plugin->SetRunPrefix("000"); // %RED% *uncomment* this for real data%ENDCOLOR%</pre><pre>// ...then add run numbers to be considered</pre><pre>  plugin->AddRunNumber(%RED%117054%ENDCOLOR%);</pre><pre>// plugin->AddRunNumber(104065); // real data</pre><pre>// plugin->SetOutputSingleFolder("output");</pre><pre>// plugin->SetOutputToRunNo();</pre><pre>// Method 2: Declare existing data files (raw collections, xml collections, root file)</pre><pre>// If no path mentioned data is supposed to be in the work directory (see SetGridWorkingDir())</pre><pre>// XML collections added via this method can be combined with the first method if</pre><pre>// the content is compatible (using or not tags)</pre><pre>// plugin->AddDataFile("tag.xml");</pre><pre>// plugin->AddDataFile("/alice/data/2008/LHC08c/000057657/raw/Run57657.Merged.RAW.tag.root");</pre><pre>// Define alien work directory where all files will be copied. Relative to alien $HOME.</pre><pre>  plugin->SetGridWorkingDir("%RED%pt117054%ENDCOLOR%");</pre><pre>// Declare alien output directory. Relative to working directory.</pre><pre>  plugin->SetGridOutputDir("%RED%output117054%ENDCOLOR%"); // In this case will be $HOME/work/output</pre><pre>// Declare the analysis source files names separated by blancs. To be compiled runtime</pre><pre>// using ACLiC on the worker nodes.</pre><pre>  plugin->SetAnalysisSource("AliAnalysisTaskPt.cxx");</pre><pre>// Declare all libraries (other than the default ones for the framework. These will be</pre><pre>// loaded by the generated analysis macro. Add all extra files (task .cxx/.h) here.</pre><pre>  plugin->SetAdditionalLibs("AliAnalysisTaskPt.h AliAnalysisTaskPt.cxx");</pre><pre>// Declare the output file names separated by blancs.</pre><pre>// (can be like: file.root or file.root@ALICE::Niham::File)</pre><pre>// plugin->SetOutputFiles("Pt.ESD.1.root");</pre><pre>  plugin->SetDefaultOutputs();</pre><pre>// Optionally define the files to be archived.</pre><pre>// plugin->SetOutputArchive("log_archive.zip:stdout,stderr@ALICE::NIHAM::File root_archive.zip:*.root@ALICE::NIHAM::File");</pre><pre>// plugin->SetOutputArchive("log_archive.zip:stdout,stderr"); // %RED%Comment out as now done automatically%ENDCOLOR%</pre><pre>// Optionally set a name for the generated analysis macro (default MyAnalysis.C)</pre><pre>  plugin->SetAnalysisMacro("TaskPt.C");</pre><pre>// Optionally set maximum number of input files/subjob (default 100, put 0 to ignore)</pre><pre>  plugin->SetSplitMaxInputFileNumber(100);</pre><pre>// Optionally modify the executable name (default analysis.sh)</pre><pre>  plugin->SetExecutable("TaskPt.sh");</pre><pre>// Optionally set number of failed jobs that will trigger killing waiting sub-jobs.</pre><pre>// plugin->SetMaxInitFailed(5);</pre><pre>// Optionally resubmit threshold.</pre><pre>// plugin->SetMasterResubmitThreshold(90);</pre><pre>// Optionally set time to live (default 30000 sec)</pre><pre>  plugin->SetTTL(30000);</pre><pre>// Optionally set input format (default xml-single)</pre><pre>  plugin->SetInputFormat("xml-single");</pre><pre>// Optionally modify the name of the generated JDL (default analysis.jdl)</pre><pre>  plugin->SetJDLName("TaskPt.jdl");</pre><pre>// Optionally modify job price (default 1)</pre><pre>  plugin->SetPrice(1);   </pre><pre>// Optionally modify split mode (default 'se')  </pre><pre>  plugin->SetSplitMode("se");</pre><pre>  return plugin;</pre><pre>}</pre>

---+++++ Explanation of changes

   1 Enable the most up to date ROOT and !AliRoot versions deployed in the GRID. There are regular announcements to the alice-project-analysis-task-force list (to which you should subscribe) telling you the latest versions. 
   1 Set the directory from which the search for data begins. Consult <a href="http://alimonitor.cern.ch/production/raw.jsp" title="Monalisa raw production">Monalisa</a> to find the period and pass you are interested in, click on it and see the directory name in the 'Output dir' column. 
   1 You need to uncomment this !SetRunPrefix line for real data as it uses the run number to search the directory hierarchy and the subdirectories have a 000 in their names. 
   1 Choose a run number to look at, again consulting <a href="http://alimonitor.cern.ch/production/raw.jsp" title="Monalisa raw production">Monalisa</a>. 
   1 Change the name of the working directory from the default - not compulsory but useful. 
   1 Change the name of the output directory from the default - again useful to see what it happening. 
   1 Comment out the line <span style="font-size: 10px; color: #7a4707; white-space: pre" class="Apple-style-span"> plugin->SetOutputArchive("log_archive.zip:stdout,stderr");</span> as this is now handled automatically. (Mail from A. Gheata 29/6/10). 
---++++ Testing

At this point you can test your changes to the !CreateAlienHandler.C macro simply by using the !runGrid.C which you downloaded to run the simple pT analysis task. First obtain an alien token in the usual way then do:
<pre>root runGrid.C</pre>

This should submit your jobs leaving you in the alien shell. You can check on the progress of your jobs using commands such as 'ps'. (Will add some more detailed notes and/or links on monitoring jobs). Basically they should start with status 'I' for inserting moving, 'W' for waiting and then to 'R' for running. You should be patient! It may take some time for the job to start running of there are already job(s) running at the site hosting the copie(s) of the files that you request. Ideally the will all end in status 'D' for done (and not EV, ESV etc. which are various error states).
---+++++ Merging

You can wait until all jobs are done and then exit the alien shell. The merging of the outputs from the subjobs will then take place and the output file returned to your working directory. In case you do want not wait for all your jobs to be done you can exit and merging of the competed jobs takes place. At a later time, after checking in aliensh that all the jobs are done, you can re-run the merging. To do that you should edit !CreateAlienHandler.C, changing the line
<pre>plugin->SetRunMode("full");</pre><pre>to </pre><pre>plugin->SetRunMode("terminate");</pre>

Don't forget to change it back next to you run.
---+++ Running your own analysis task

By running the procedure outlined above you have verified that all of the grid infrastructure is working correctly. However it is not very interesting to run the !AliAnalysisTaskPt task. You could modify it to do more interesting things but more likely it that you already have an analysis task that you do not want to re-write. To get your task running on the grid you need to make further modifications to the !CreateAlienHandler.C macro.
<pre>plugin->SetAnalysisSource("AliAnalysisTaskPt.cxx");</pre>

is changed to use your own source code, for example
<pre>plugin->SetAnalysisSource("AliMult.cxx");</pre> <pre>plugin->SetAdditionalLibs("AliAnalysisTaskPt.h <font color="#7a4707">AliAnalysisTaskPt</font>.cxx");</pre>

is changed to include all the .cxx and .h files that you need. Eg:
<pre>plugin->SetAdditionalLibs("AliMult.h <font color="#7a4707">AliMult</font>.cxx");</pre>

You should also change the following lines so that the names given to the various automatically produced files are more sensible, although this is not strictly necessary:
<pre>plugin->SetGridWorkingDir("pt117054");</pre>

to eg,
<pre>plugin->SetGridWorkingDir("mult117054");</pre><pre>plugin->SetAnalysisMacro("TaskPt.C");</pre>

to eg,

<pre>plugin->SetAnalysisMacro("TaskMult.C");</pre><pre>plugin->SetExecutable("TaskPt.sh");</pre>

to eg,
<pre>plugin->SetExecutable("TaskMult.sh");</pre><pre>plugin->SetJDLName("TaskPt.jdl");</pre>

to eg,

plugin->SetJDLName("TaskMult.jdl");

The next step is modify your macro which you use to run your job. It is not possible to specify exactly what needs to be done for each individual case. Basically you are merging the runGrid.C macro provided with your existing "run.C" macro. The following are generally needed though. Add the following to the libraries loaded at the start:
<pre>  gSystem->Load("libCore.so"); </pre> <pre>  gSystem->Load("libTree.so");</pre> <pre>  gSystem->Load("libGeom.so");</pre>

Add the following in place of the connection to PROOF;

<pre>// Create and configure the alien handler plugin</pre> <pre>  gROOT->LoadMacro("CreateAlienHandler.C");</pre> <pre>  AliAnalysisGrid *alienHandler = CreateAlienHandler(); </pre> <pre>  if (!alienHandler) return;</pre>

Remove
<pre>   gROOT->LoadMacro("$ALICE_ROOT/PWG0/CreateESDChain.C");</pre>

When creating your task change
<pre>  gProof->Load("AliMult.cxx++g"); </pre>

to
<pre>  gROOT->LoadMacro("AliMult.cxx++g"); </pre>

After creating the analysis manager, add the line
<pre>    mgr->SetGridHandler(alienHandler); </pre>

Finally change
<pre>    mgr->StartAnalysis("proof","/ALICE/pp007000/LHC10b_000114783_p1");</pre>

to
<pre>  mgr->StartAnalysis("grid");</pre>

I have attached two files showing the changes made to Arvinder's macro for running on SKAF to run on the grid to serve as an example.
   * <span style="color: blue; text-decoration: underline">[[%ATTACHURL%/skaf.C][skaf.C]]</span>: Original macro to run user task before adaption 

   * <span style="color: blue; text-decoration: underline">[[%ATTACHURL%/runGridAliMult.C][runGridAliMult.C]]</span>: Real example of macro adapted to run on Grid using the Alien plugin. 
---+++ Monitoring your task once you have submitted it

When you have submitted your grid job you are given a job number. You can check on all the jobs you have submitted from within aliensh with
<pre>  ps</pre>

or, for only the jobs that are currently running,
<pre>  ps -X</pre>

To check what is happening with a specific job, do
<pre>  ps -trace jobnumber all</pre>

where "all" is an optional argument that prints the entire status history of the job.

To look at the output of a job when it is in progress use the spy command:
<pre>  spy jobnumber workdir</pre>

or
<pre>  spy jobnumber stdout</pre> Where workdir prints the job working directory and stdout (obviously) prints the stdout. These only work when a job is running, ie has status "R."

There are other commands (see the offline bible/alien tutorial) but these are probably the most useful.

When you look at a job with ps, it can have various status codes. When a job runs without problems, it will go through them in the order below. For more information see the offline bible or this job status <a target="_blank" href="http://alimonitor.cern.ch/show?page=jobStatus.html" title="GRID job status">flow chart</a>:

I: inserting

W: waiting

ST: started/staging

R: running

SV: saving

SVD: saved

D: done

If there is a problem, it can give one of these error codes:

ESP: typically, the required dataset path does not exist. Check the run has been reconstructed, and the right pass chosen.

EI / EA: normally a service failure rather than a problem with your job. Try again later.

EIB: error in download of input files. Probably the input file does not exist in the storage element or the storage element is unreachable from the job worker node.

EV: job validation failed (your validation script returned nonzero)

ESV: an output file could not be saved, probably due to an unavailable storage element.

Z / EXP: job got lost on a worker node due to a node or network failure. Resubmit the job.

There is also an ESP error which doesn't seem to be documented anywhere, but seems to be due to a problem (like a typo) in the jdl.

An alternative monitoring solution is provided by <a href="http://alimonitor.cern.ch/" title="Monalisa">Monalisa</a> in the 'My Jobs' link in the bar at the top.

-- Main.PatrickScott - 20 May 2010
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
c	runGridAliMult.C	r1	manage	2.0 K	19 May 2010 - 17:13	UnknownUser	Real example of macro adapted to run on Grid using the Alien plugin.
c	skaf.C	r1	manage	2.0 K	19 May 2010 - 17:14	UnknownUser	Original macro to run user task before adaption
Topic revision: r11 - 01 Dec 2010 - _47C_61UK_47O_61eScience_47OU_61Birmingham_47L_61ParticlePhysics_47CN_61arvinder_32palaha?
ALICE
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback