HPSS IU slides

Information about HPSS IU slides

Published on November 21, 2007

Author: Laurence

Source: authorstream.com

Content

IU's TeraGrid-accessible HPSS archival storage system:  IU's TeraGrid-accessible HPSS archival storage system Overview about HPSS Feeds and speeds, and particulars of IU's distributed HPSS environment Moving data to and from IU's HPSS via the TeraGrid and gridftp Hands on exercises with gridftp and HPSS HPSS Background:  HPSS Background IU installed HPSS system in 1998 HPSS is distributed between Indianapolis and Bloomington campuses. Over 1.8 petabytes of storage available and can scale to 4.2 petabytes In aggregate, HPSS can handle as much as 2GB/s of data flowing in or out. HPSS Design:  HPSS Design IUB Subsystem IUPUI Subsystem Research Network Research Network FC SAN FC SAN HPSS Core Servers HPSS Performance:  HPSS Performance 100MB/sec single thread over gigabit IU HPSS capable of >1GB/sec in parallel over multiple 10 gigabit 150TB of DDN disk for cache 40 IBM Jaguar drives - 100MB/sec transfers Storing Data:  Minimum file size should be one MB Prefer data to be at 10MB or larger Small files should be aggregated Maximum file size is 10TB By default, data is stored on tape at each campus Storing Data Class of Service:  Class of Service Each file has class of service (COS) COS defines maximum file size number of copies stripe width With GridFTP, COS is chosen automatically by file size User can override by appending ,,<COS> to the filename COS Listing:  COS Listing COS 1 - default; max file size 10GB COS 2 - max file size 40GB COS 3 - max file size 10TB COS 4 - max file size 10TB 16 way to disk 4 way to tape Single copy COS available Quotas:  Quotas Default is 1TB Can request up to 10TB Double these numbers if single-copy COS used Accessing MDSS:  Accessing MDSS Data goes to disk cache first Data copied to tape ASAP Copy sent to other campus by default Data purged from disk cache as space needed Data staged back to disk cache as needed Accessing MDSS:  Accessing MDSS Fastest Methods hsi (hsi.mdss.iu.edu) gridftp (gridftp.mdss.iu.edu) pftp_client (ftp.mdss.iu.edu) kerberized ftp (ftp.mdss.iu.edu) Accessing MDSS:  Accessing MDSS Convenient Methods sftp (sftp.mdss.iu.edu) https (www.mdss.iu.edu) Samba (smb.mdss.iu.eu) Hpssfs hsi example:  hsi example intrigue:~ seiffert$ hsi Principal: seiffert [seiffert]Password: Username: seiffert UID: 1529 CC: 1529 Copies: 1 [hsi.3.3 Wed Jan 17 11:31:07 EST 2007] /hpss/s/e/seiffert->put myfile.tar put ’myfile.tar' : '/hpss/s/e/seiffert/myfile.tar’\ ( 4051006 bytes, 4198.5 KBS (cos=1)) /hpss/s/e/seiffert->get myfile.tar get '/Users/seiffert/myfile.tar' :\ '/hpss/s/e/seiffert/myfile.tar' (2007/01/14 12:09:36\ 4051006 bytes, 8275.6 KBS ) /hpss/s/e/seiffert-> sftp example:  sftp example intrigue:~ seiffert$ sftp sftp.mdss.iu.edu Connecting to sftp.mdss.iu.edu... [email protected]'s password: sftp> put welcome.wmv Uploading welcome.wmv to /ssh\-chroot/cos1/hpss/s/e/seiffert/welcome.wmv welcome.wmv 100% 3956KB 3.9MB/s 00:01 sftp> get welcome.wmv Fetching /ssh-chroot/cos1/hpss/s/e/seiffert/welcome.wmv to welcome.wmv /ssh-chroot/cos1/hpss/s/e/seiffert/welcome.wmv \ 100% 3956KB 3.9MB/s 00:00 IU HPSS from TeraGrid:  IU HPSS from TeraGrid Accessible by either: gridftp.archive.iu.teragrid.org gridftp.hpss.iu.teragrid.org Need to add TeraGrid certificate to IU Big Red System uberftp and globus-url-copy tested and supported tgcp is not tested or supported globus-url-copy Example:  globus-url-copy Example Simple example to and from globus-url-copy -rp gsiftp://my.local.site/myfile.tar \ gsiftp://gridftp.archive.iu.teragrid.org/myfile.tar globus-url-copy -rp \ gsiftp://gridftp.archive.iu.teragrid.org/myfile.tar \ gsiftp://my.local.site/myfile.tar Parallel transfer to and from globus-url-copy -rp -stripe -bs 8388608 \ gsiftp://my.local.site/bigfile.tar \ gsiftp://gridftp.archive.iu.teragrid.org:2814/bigfile.tar,,4 globus-url-copy -rp -stripe \ gsiftp://gridftp.archive.iu.teragrid.org:2814/bigfile.tar \ gsiftp://my.local.site/ uberftp example:  uberftp example Simple Example uberftp> put myfile.tar myfile.tar uberftp> get myfile.tar Parallel Example uberftp> open -P 2814 gridftp.archive.iu.teragrid.org uberftp> lopen my.local.site uberftp> parallel 4 uberftp> blksize 8388608 uberftp> put bigfile.tar bigfile.tar,,4

Related presentations


Other presentations created by Laurence

Growing Grapes in Wisconsin05
04. 01. 2008
0 views

Growing Grapes in Wisconsin05

lec29w
06. 03. 2008
0 views

lec29w

Positive Side of Life
03. 10. 2007
0 views

Positive Side of Life

022Neo Dada
03. 10. 2007
0 views

022Neo Dada

Soil Fertility 101 Wheat focus
09. 10. 2007
0 views

Soil Fertility 101 Wheat focus

gluthurk
04. 10. 2007
0 views

gluthurk

confined space training
07. 11. 2007
0 views

confined space training

Vogelsang
13. 11. 2007
0 views

Vogelsang

LR TMSA Danaos conference
15. 11. 2007
0 views

LR TMSA Danaos conference

9 548 Pando UTB 01
19. 11. 2007
0 views

9 548 Pando UTB 01

ri aquaculture permit process
28. 12. 2007
0 views

ri aquaculture permit process

Ch8
01. 01. 2008
0 views

Ch8

EEP101 15
03. 01. 2008
0 views

EEP101 15

AliyaKassamRE
04. 01. 2008
0 views

AliyaKassamRE

Shiva Kumar Sharma
06. 12. 2007
0 views

Shiva Kumar Sharma

Rabies 1
16. 11. 2007
0 views

Rabies 1

USDA Cultivar Poster
23. 11. 2007
0 views

USDA Cultivar Poster

Bio1100Ch3W
03. 01. 2008
0 views

Bio1100Ch3W

FinalMCX
19. 02. 2008
0 views

FinalMCX

20070412100800
12. 03. 2008
0 views

20070412100800

fasolSlides
14. 03. 2008
0 views

fasolSlides

q3ex
18. 03. 2008
0 views

q3ex

1129130239620
21. 03. 2008
0 views

1129130239620

chonSlides
27. 03. 2008
0 views

chonSlides

penrosepresentation
07. 04. 2008
0 views

penrosepresentation

refupdate
30. 03. 2008
0 views

refupdate

ogan
26. 11. 2007
0 views

ogan

GiraFlyer
23. 11. 2007
0 views

GiraFlyer

netinfo070919
28. 11. 2007
0 views

netinfo070919

Quiz Powerpoint3
06. 11. 2007
0 views

Quiz Powerpoint3

MuseumInformatics
02. 10. 2007
0 views

MuseumInformatics