EnabGrid

Information about EnabGrid

Published on February 7, 2008

Author: Ulisse

Source: authorstream.com

Content

Enabling Grid Computer for HEP: James Cunha Werner [email protected] Enabling Grid Computer for HEP Babar Team at University of Manchester Resources: www.hep.man.ac.uk/u/jamwer Human resource strategy: James Cunha Werner [email protected] Human resource strategy * Jobs with 5 events instead Millions. Resources Strategy: James Cunha Werner [email protected] Resources Strategy Grid Test Bed: James Cunha Werner [email protected] Grid Test Bed Slide 5: James Cunha Werner [email protected] Slide 6: James Cunha Werner [email protected] Software: 850 packages. Tau Datasets: range between 60 files 1GB and 150 files 1GB Total 4,000 GB ~ 10,000 files Analysis Submission to Grid : James Cunha Werner [email protected] Analysis Submission to Grid Single command: ./easygrid dataset_name Perform Handlers management and submission Software based in State-machine Verify skimdata available: If not available perform BbkDatasetTCL to generate skimData. Each file will be a job. Verify if there are handlers pending If not, script generation (gera.c) with edg-job-submit and ClassAdds, and script execution. Nest for submission policy and optimisation. If yes, verify job status. When the all jobs ended, recover results in user folder. (Prototype) Generation and submission: James Cunha Werner [email protected] Generation and submission [[email protected] babar]$ ./easygrid SP-1005-Tau11-R14 Invalid configuration filename: /opt/edg/etc/vomses Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=james werner Enter GRID pass phrase for this identity: Creating temporary proxy ......................................................... Done Creating proxy .................................................... Done Searching pre selected skimdata. Searching previous handlers. Handlers not found. Submiting to GRID . Wait end of process... Job Status : James Cunha Werner [email protected] Job Status [[email protected] babar]$ ./easygrid SP-1005-Tau11-R14 Invalid configuration filename: /opt/edg/etc/vomses Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=james werner Enter GRID pass phrase for this identity: Creating temporary proxy ............................ Done Creating proxy ............................... Done Searching pre selected skimdata. Searching previous handlers. Checking if jobs finished. ### Handle -> https://lcgrb01.gridpp.rl.ac.uk:9000/foRHhWyeDBnbqA9JkDADLg Current Status: Scheduled https://lcgrb01.gridpp.rl.ac.uk:9000/foRHhWyeDBnbqA9JkDADLg still pendent. ### Handle -> https://lxn1188.cern.ch:9000/8DdK3xruxtevNpei3zZbaA Current Status: Scheduled https://lxn1188.cern.ch:9000/8DdK3xruxtevNpei3zZbaA still pendent. 4 jobs did not finished ! Try again later. Job Status and recovery: James Cunha Werner [email protected] Job Status and recovery [[email protected] babar]$ ./easygrid SP-1005-Tau11-R14 Invalid configuration filename: /opt/edg/etc/vomses Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=james werner Enter GRID pass phrase for this identity: Creating temporary proxy .......................................... Done Creating proxy ........................................................... Done Searching pre selected skimdata. Searching previous handlers. Checking if jobs finished. ### Handle -> https://lcgrb01.gridpp.rl.ac.uk:9000/foRHhWyeDBnbqA9JkDADLg Current Status: Done Exit code: 0 ### Handle -> https://lxn1188.cern.ch:9000/8DdK3xruxtevNpei3zZbaA Current Status: Done Exit code: 0 0 jobs did not finished ! Try again later. All jobs done. Recovering results in your folder. Results in the following folders: /home/jamwer/grid_sub/babar/jamwer_foRHhWyeDBnbqA9JkDADLg /home/jamwer/grid_sub/babar/jamwer_8DdK3xruxtevNpei3zZbaA Monte Carlo Submission to Grid: James Cunha Werner [email protected] Monte Carlo Submission to Grid Single Command: ./mcgrid JobName num_copies Perform Handlers management and submission. Software based in State-Machine: Verify if there are handlers pending If not, script generation (geramc.c) with edg-job-submit and ClassAdds for each copy, and script execution. Nest for submission policy and optimisation. If yes, verify job status. When the all jobs ended, recover results in user folder. (Prototype) MC Submission : James Cunha Werner [email protected] MC Submission [[email protected] mcgrid1]$ ./mcgrid MCteste 3 Invalid configuration filename: /opt/edg/etc/vomses Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=james werner Enter GRID pass phrase for this identity: Creating temporary proxy ................................. Done Creating proxy ....................................................... Done Searching previous handlers. Handlers not found. Submiting to GRID . Wait end of process... Job Status: James Cunha Werner [email protected] Job Status [[email protected] mcgrid1]$ ./mcgrid MCteste 3 Invalid configuration filename: /opt/edg/etc/vomses Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=james werner Enter GRID pass phrase for this identity: Creating temporary proxy ........................................ Done Creating proxy ....................................... Done Searching previous handlers. Checking if jobs finished. ### Handle -> https://lxn1188.cern.ch:9000/9WzceoIMEQoTK24a-UvOmw Current Status: Scheduled https://lxn1188.cern.ch:9000/9WzceoIMEQoTK24a-UvOmw still pendent. ### Handle -> https://lcgrb01.gridpp.rl.ac.uk:9000/c4iCB8vioozaGteI9hybIg Current Status: Ready https://lcgrb01.gridpp.rl.ac.uk:9000/c4iCB8vioozaGteI9hybIg still pendent. ### Handle -> https://lcgrb01.gridpp.rl.ac.uk:9000/L5BD1OE--eckTm5RXkp2nA Current Status: Ready https://lcgrb01.gridpp.rl.ac.uk:9000/L5BD1OE--eckTm5RXkp2nA still pendent. 3 jobs did not finished ! Try again later. Job status and recovery: James Cunha Werner [email protected] Job status and recovery [[email protected] mcgrid1]$ ./mcgrid MCteste 3 Invalid configuration filename: /opt/edg/etc/vomses Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=james werner Enter GRID pass phrase for this identity: Creating temporary proxy .................................................. Done Creating proxy .................................................... Done Searching previous handlers. Checking if jobs finished. ### Handle -> https://lxn1188.cern.ch:9000/9WzceoIMEQoTK24a-UvOmw Current Status: Done Exit code: 0 ### Handle -> https://lcgrb01.gridpp.rl.ac.uk:9000/c4iCB8vioozaGteI9hybIg Current Status: Done Exit code: 0 0 jobs did not finished ! Try again later. All jobs done. Recovering results in your folder. Results in the following folders: /home/jamwer/grid_sub/mcgrid1/jamwer_9WzceoIMEQoTK24a-UvOmw /home/jamwer/grid_sub/mcgrid1/jamwer_c4iCB8vioozaGteI9hybIg /home/jamwer/grid_sub/mcgrid1/jamwer_L5BD1OE--eckTm5RXkp2nA Testing Submission Script: James Cunha Werner [email protected] Testing Submission Script Load Range: Worker load x #Files 16 x 60 files = 960 jobs pendent 16 x 150 files = 2400 jobs pendent Test with Submission script * sslv3 alert handshake failure ** Please wait job enter the “Done” status. This never happens! Resource Broker not reliable or robust. Sometimes failure 3 days a week or takes hours to submit/dispatch to CE (empty!). Pending Infrastructure => Course of action: James Cunha Werner [email protected] Pending Infrastructure => Course of action Babar Software Know How is not available at Manchester => Web Page & Network skills. Quality Assurance => We are OK! from benchmark (E x P) Real Application to perform complete cycle, acquire know how, and grid prof-of-concept is missing => Partnership with physicists CERN does NOT recognise Babar Community => Lets reduce their priority! RB at Manchester => 60MB binaries and policies freedom. SE/RC at Manchester => policies and submission jobs freedom. Mass storage (10TB) for Babar purposes => CAP! UI in the AFS => wide access to Manchester farms. Apprenticeship at RAL and later at SLAC – production and experiment => improve where others fail Configuration for optimal job performance/submission at Tear 2 (1 Ce x 50 WN? Performance dCache with Babar Software? Why 10TB if Liverpool bought 80TB? Electricity bill? => analyse procedures to improve QoS and better Site Configuration Update (software and data) and operational policies => operational standards to achieve high QoS Aimed Hardware Architecture : James Cunha Werner [email protected] Aimed Hardware Architecture (Redundant RB with alternate access) Aimed Software Architecture: James Cunha Werner [email protected] Aimed Software Architecture Production Job Submission Package: James Cunha Werner [email protected] Production Job Submission Package Operational policies/integration with RB (application level). Recovery of aborted status. Resources optimisation. Integration with RC (application level) for replicas policies development. Interactive data visualisation (Useful?) Integration with GridSite (Data visualisation, analysis, performance monitor, and submission) Professional version. Integrate LCG2 and Job Submission with Babar/CM2 at University of Manchester for Tau Physics modelling, analysis and MC generation.: James Cunha Werner [email protected] Integrate LCG2 and Job Submission with Babar/CM2 at University of Manchester for Tau Physics modelling, analysis and MC generation. We aim to be soon… The largest site in UK. Leader in grid computing and HEP Summary Conclusion: James Cunha Werner [email protected] Conclusion Babar CM2 is running at Manchester! LCG2 Grid is running with real world experiment! Babar submission prototype to Grid is running ! LCG is not LHC software only! It is Babar’s. We are doing today what will take years to you to achieve. Lets work together!

#files presentations

Linux Commands - Cheat Sheet
30. 01. 2018
0 views

Linux Commands - Cheat Sheet

Bug Prediction and Analysis
09. 11. 2009
0 views

Bug Prediction and Analysis

Related presentations


Other presentations created by Ulisse

wtoprimitivo
08. 05. 2008
0 views

wtoprimitivo

M Naoi IPES2007
07. 05. 2008
0 views

M Naoi IPES2007

atos maxwell jamescfh
02. 05. 2008
0 views

atos maxwell jamescfh

06ESADE
30. 04. 2008
0 views

06ESADE

1 4 Eiropas Kopienas programmas
22. 04. 2008
0 views

1 4 Eiropas Kopienas programmas

5 11
21. 04. 2008
0 views

5 11

phan
18. 04. 2008
0 views

phan

Brown Velocity
17. 04. 2008
0 views

Brown Velocity

SME in Pakistan
10. 01. 2008
0 views

SME in Pakistan

visual perception 1
14. 01. 2008
0 views

visual perception 1

Ross Thompson Presentation
15. 01. 2008
0 views

Ross Thompson Presentation

ArmorofGod ShieldofFaithNew
17. 01. 2008
0 views

ArmorofGod ShieldofFaithNew

Hospital Employee Immunization 1
17. 01. 2008
0 views

Hospital Employee Immunization 1

EnhancingCoreWeb
10. 01. 2008
0 views

EnhancingCoreWeb

A C 8 US CHAPTER
10. 01. 2008
0 views

A C 8 US CHAPTER

6 SMS
18. 01. 2008
0 views

6 SMS

textiles grad 2006
22. 01. 2008
0 views

textiles grad 2006

SessIIBYeboahEtAl
22. 01. 2008
0 views

SessIIBYeboahEtAl

hfcourse frzn
19. 01. 2008
0 views

hfcourse frzn

Heezen overview of cases
22. 01. 2008
0 views

Heezen overview of cases

2c
23. 01. 2008
0 views

2c

language
14. 01. 2008
0 views

language

10 Cash Control
04. 02. 2008
0 views

10 Cash Control

gsoverview
04. 02. 2008
0 views

gsoverview

Cuisine
22. 01. 2008
0 views

Cuisine

schap011
07. 02. 2008
0 views

schap011

Frontier1
11. 02. 2008
0 views

Frontier1

resiliency
09. 01. 2008
0 views

resiliency

library orientation
28. 01. 2008
0 views

library orientation

WeddingPlanning
29. 01. 2008
0 views

WeddingPlanning

harpring
21. 01. 2008
0 views

harpring

owen holland
12. 01. 2008
0 views

owen holland

breakout 2ARavishankara
20. 02. 2008
0 views

breakout 2ARavishankara

Michelozzo and Bernardo
10. 01. 2008
0 views

Michelozzo and Bernardo

absent
15. 01. 2008
0 views

absent

p1 calyon de dumast
28. 02. 2008
0 views

p1 calyon de dumast

p548 lecture2v2
03. 03. 2008
0 views

p548 lecture2v2

SB 18 Productive Consultation
08. 03. 2008
0 views

SB 18 Productive Consultation

tortola
20. 01. 2008
0 views

tortola

what
20. 01. 2008
0 views

what

ONEPowerPointPresent ation
19. 03. 2008
0 views

ONEPowerPointPresent ation

hall peter
25. 03. 2008
0 views

hall peter

EID Intro
31. 03. 2008
0 views

EID Intro

cottrell icfa korea may05
03. 04. 2008
0 views

cottrell icfa korea may05

Maps
07. 04. 2008
0 views

Maps

ErgonomicsPresentati on 000
07. 03. 2008
0 views

ErgonomicsPresentati on 000

London2012
14. 04. 2008
0 views

London2012

Samurai Hoedown
07. 02. 2008
0 views

Samurai Hoedown

m2f2
04. 02. 2008
0 views

m2f2

Olympic Mascots
14. 04. 2008
0 views

Olympic Mascots

stdslides2003
12. 03. 2008
0 views

stdslides2003

Packaging ru
14. 01. 2008
0 views

Packaging ru

phpulsefeb
14. 02. 2008
0 views

phpulsefeb

ALADI May2 2005
25. 01. 2008
0 views

ALADI May2 2005

041014 TEAHA
10. 04. 2008
0 views

041014 TEAHA

tds gap
22. 01. 2008
0 views

tds gap

AIDS f06
16. 01. 2008
0 views

AIDS f06

UniversityForumFeb23 2005
16. 01. 2008
0 views

UniversityForumFeb23 2005

Presentation to LGA 23Jan07 V3
24. 01. 2008
0 views

Presentation to LGA 23Jan07 V3

NW web presentation7 03
29. 02. 2008
0 views

NW web presentation7 03

build ssp system
21. 01. 2008
0 views

build ssp system

laighside
29. 01. 2008
0 views

laighside

landis
15. 01. 2008
0 views

landis

Service design for active ageing
21. 11. 2016
0 views

Service design for active ageing