Athena Access to the SE

You can list files on the SE using either rfdir or dpns-ls:

rfdir /dpm/ph.bham.ac.uk/home/atlas/atlasmcdisk/mc08/
dpns-ls -l /dpm/ph.bham.ac.uk/home/atlas/atlasmcdisk/mc08/

Athena ultimately requires a list of files, so perhaps a script to take a DQ2 dataset name, find it's location on the SE and then output filelist.py would suffice?

Setup (User)

RFIO relies on the library libshift.so, but Castor and DPM required two different versions, Castor being the default. To fix this, in the run directory:

ln -s /home/lcgui/SL4/prod/lcg/lib/libdpm.so libshift.so.2.1
export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH

This allows for (the very slow) access to the SE using RFIO. Users will also require a valid grid certificate.

Finding Files

The sancho.sh script may be used to locate files on the SE via their DQ2 label. It can be used to output a filelist in two formats - with and without Athena decorations.

Example 1:

source sancho.sh mc08.105502.AcerMC_tchan.recon.AOD.e352_s462_r541           #Prints a list of files suitable with one file per line (suitable for ganga)

Example 2:

source sancho.sh -a mc08.105502.AcerMC_tchan.recon.AOD.e352_s462_r541       #Prints a list of files with Athena decorations ("ServiceMgr...." etc)

Testing

Variable RFIO Buffer Size

The time taken to iterate over 1000 ttbar events from the sample mc08.105200.T1_McAtNlo_Jimmy.recon.AOD.e357_s462_r541/ for various RFIO buffer sizes is shown below. The test was completed on a desktop machine. The plot shows that whilst the absolute time is dependent on the number of persistent StoreGate collections retrieved, the overall trend for smaller buffers being better unless you can afford to read the entire file.

rfio.png

From a single user point of view, 4 kB buffers are the best solution. Will this put too much of a drain on the SE though? Presumeably this means that once the user has consumed 4kB, it will immediately request more data. I wonder if this scales as the number of clients increases?

-- ChristopherCurtis - 29 May 2009

  • sancho.sh: sanch.sh - tool for finding files on the SE and printing filelists
Topic attachments
ISorted ascending Attachment History Action Size Date Who Comment
Pngpng rfio.png r1 manage 24.3 K 11 Jun 2009 - 15:17 UnknownUser  
Shsh sancho.sh r1 manage 3.2 K 10 Jun 2009 - 17:37 UnknownUser sanch.sh - tool for finding files on the SE and printing filelists
Edit | Attach | Watch | Print version | History: r10 | r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 11 Jun 2009 - _47C_61UK_47O_61eScience_47OU_61Birmingham_47L_61ParticlePhysics_47CN_61christopher_32curtis?
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback