OpenMPI OpenIB

Information about OpenMPI OpenIB

Published on January 3, 2008

Author: UpBeat

Source: authorstream.com

Content

Open MPI:  Open MPI Overview and Current Status Tim Woodall - LANL Galen Shipman - LANL/UNM Overview:  Overview Point-to-Point Architecture OpenIB Implementation Results Future Work Point-to-Point Architecture:  Point-to-Point Architecture Component Architecture: “Plug-ins” for different capabilities (e.g. different networks) Tunable run-time parameters Three component frameworks: Point-to-point messaging layer (PML) implements MPI semantics Byte Transfer Layer (BTL) abstracts network interfaces Memory Pool (mpool) provides for memory management/registration PML Framework:  PML Framework Single PML manages multiple BTL modules Maintains set of BTLs on a per-peer basis Message fragmentation and scheduling Implements MPI semantics Synchronous / buffered / ready / normal sends Persistent requests / Request completion Eager/Rendezvous protocol Eager send of short messages Configurable threshold (short vs. long) Multiple long protocols PML Protocols:  PML Protocols Send / receive pipeline to / from pre-registered buffers (non-contiguous data) MPI_Alloc_mem support Red/black tree of memory registrations BTL associated with registration is used by scheduler Xfer of contiguous data with 1 RDMA (after match) “Leave pinned” run-time parameter Registration on first-use MRU cache (configurable size) of registrations Bandwidth equivalent to pre-registered buffers (MPI_Alloc_mem) PML Protocols (Continued):  PML Protocols (Continued) Dynamic memory registration/deregistration Fragment message and build pipeline of RDMA requests Overlap [de-]registration with RDMA Bandwidth 97% of pre-registered memory at large message sizes (8Mbytes) Performance impacted by bus type/bandwidth BTL Framework:  BTL Framework MPI agnostic Provides simple API to upper layers Tagged send/receive primitives One-sided put/get operations Access to data type engine for zero copy data transfer BTL modules natively support commodity networks: Current (self, shared memory, myrinet GM/MX, Infiniband mvapi/OpenIB, Portals, TCP) Planned (LAPI, Quadrics Elan4) OpenIB BTL:  OpenIB BTL BTL module initialization Resources allocation Connection management Small message Xfer Large message Xfer OpenIB Issues Future Work BTL module initialization:  BTL module initialization A separate BTL module is initialized for each port on each HCA The PML schedules across these BTL modules just as any other interconnect When multiple BTL modules exist peers establish QP connections by matching subnets Resource Allocation:  Resource Allocation SRQ Scalability :  SRQ Scalability K- multiplier based on number of nodes Connection management:  Connection management Addressing information is exchanged dynamically via an OOB channel This greatly improves scalability but at the cost of increased first message latency Connections are established with peers in the same subnet (local subnet routing only) Small Message Xfer:  Small Message Xfer Maintain list of pre-registered fragments for send and recv List grows dynamically in chunks (more efficient to register) Small messages are copied to/from pre-registered buffers Recv descriptors are posted as needed based on min/max thresholds Small Message Performance:  Small Message Performance * Send/Recv based protocol Large Message Xfer:  Large Message Xfer RDMA Write and RDMA Read are both supported RDMA Read provides better performance than RDMA Write - control messages are reduced RDMA pipeline protocol performance highly dependent on I/O Bus performance Results:  Results OpenMPI/OpenIB - All Results:  Results OpenMPI/OpenIB - All - Log Results:  Results OpenMPI/OpenIB - Eager limit Results:  Results Combined Results Results:  Results Combined Results - Log OpenIB Opportunities:  OpenIB Opportunities User level notification of VM activity Caching of memory registrations can be dangerous Need the ability to detect VM changes that effect memory registrations (such as sbrk and munmap) Reliable Multicast for collectives SRQ performance, 2/10 usec penalty, but who’s counting? Future Work:  Future Work Small message RDMA (using working set of peers) - optional Dynamic connection management using Unreliable Datagrams Dynamic connection teardown - optional Source Code Access:  Source Code Access Subversion repository Download client from: http://subversion.tigris.org/ v1.2.1 or later Check out with: svn co http://svn.open-mpi.org/svn/ompi/trunk ompi Anonymous, read-only access Questions?:  Questions? Tim Woodall Email: [email protected] Phone: 505-665-5224 Galen Shipman Email: [email protected] Hardware Specs:  Hardware Specs Dual Intel Xeon 3.2 GHz 1024 KB Cache 2 Gbytes memory Bus: Intel Corp. E7525/E7520/E7320 PCI Express Mellanox Technologies MT25208 InfiniHost III Ex 288 Port Voltaire switch

Related presentations


Other presentations created by UpBeat

Making Disciples
17. 06. 2007
0 views

Making Disciples

CVA Final Version 2006
06. 12. 2007
0 views

CVA Final Version 2006

Children Animals
03. 10. 2007
0 views

Children Animals

whole grains
04. 10. 2007
0 views

whole grains

07 peering and te with bgp
07. 10. 2007
0 views

07 peering and te with bgp

climate promoting sd china
12. 10. 2007
0 views

climate promoting sd china

ppt forest policy
27. 11. 2007
0 views

ppt forest policy

ch04dp3
01. 12. 2007
0 views

ch04dp3

Chairman Award 2 Hickok
02. 11. 2007
0 views

Chairman Award 2 Hickok

Nancy Kerrigan
05. 11. 2007
0 views

Nancy Kerrigan

Academia Sinica 05 30 Sep
15. 11. 2007
0 views

Academia Sinica 05 30 Sep

19361
15. 11. 2007
0 views

19361

Possessives Apostrophe
20. 11. 2007
0 views

Possessives Apostrophe

Malaysia
26. 11. 2007
0 views

Malaysia

compliance
19. 12. 2007
0 views

compliance

upload c beatles64 6n22
21. 12. 2007
0 views

upload c beatles64 6n22

Azerbaijan PPT VCMar02
29. 12. 2007
0 views

Azerbaijan PPT VCMar02

final mccc yougov report
30. 12. 2007
0 views

final mccc yougov report

MikeWhiteDDDMDec1106
05. 11. 2007
0 views

MikeWhiteDDDMDec1106

what is AI
05. 01. 2008
0 views

what is AI

Basic Plant Physiology
07. 01. 2008
0 views

Basic Plant Physiology

blm doe levy
07. 01. 2008
0 views

blm doe levy

Hedley
12. 11. 2007
0 views

Hedley

AC004
31. 12. 2007
0 views

AC004

ravi keynote
11. 10. 2007
0 views

ravi keynote

globalisation 2008
24. 02. 2008
0 views

globalisation 2008

19741
06. 03. 2008
0 views

19741

james rigney
21. 03. 2008
0 views

james rigney

ShipToAverage
07. 11. 2007
0 views

ShipToAverage

World Indigo Market Monkeys
27. 03. 2008
0 views

World Indigo Market Monkeys

ch09 1
28. 12. 2007
0 views

ch09 1

1992 stations cross esquivel
07. 04. 2008
0 views

1992 stations cross esquivel

A105 017 Blackholes
28. 11. 2007
0 views

A105 017 Blackholes

laurence millar
28. 03. 2008
0 views

laurence millar

lule
09. 04. 2008
0 views

lule

no5
10. 04. 2008
0 views

no5

Energy Scenarios
13. 04. 2008
0 views

Energy Scenarios

disnqmwrqn5ghio
17. 04. 2008
0 views

disnqmwrqn5ghio

Kids Landscape2004
22. 04. 2008
0 views

Kids Landscape2004

46 DNT Japan Tsuji
09. 10. 2007
0 views

46 DNT Japan Tsuji

EPUK presentation FINAL
16. 11. 2007
0 views

EPUK presentation FINAL

adetajan2007
13. 12. 2007
0 views

adetajan2007

ENCHANTED DOLLS
02. 10. 2007
0 views

ENCHANTED DOLLS

nclb overview
28. 11. 2007
0 views

nclb overview

180 07 lec9
16. 11. 2007
0 views

180 07 lec9

china standards retro
12. 10. 2007
0 views

china standards retro

Chapter 20 Outline
11. 12. 2007
0 views

Chapter 20 Outline

2004SellingInChina JamesColony
10. 10. 2007
0 views

2004SellingInChina JamesColony

Psychology and Political Thought
07. 01. 2008
0 views

Psychology and Political Thought

Love 2
17. 06. 2007
0 views

Love 2

Love 1
17. 06. 2007
0 views

Love 1

Love Your Body
17. 06. 2007
0 views

Love Your Body

Love Story Slides 4 Eric
17. 06. 2007
0 views

Love Story Slides 4 Eric

love not leave
17. 06. 2007
0 views

love not leave

Love Dynamics
17. 06. 2007
0 views

Love Dynamics

love eco
17. 06. 2007
0 views

love eco

Medieval lyrics
17. 06. 2007
0 views

Medieval lyrics

MDA USC 2007
17. 06. 2007
0 views

MDA USC 2007

MDA AAAI 2004
17. 06. 2007
0 views

MDA AAAI 2004

Lucy AM
17. 06. 2007
0 views

Lucy AM

lemonade 8
10. 10. 2007
0 views

lemonade 8

esslli07 3
22. 11. 2007
0 views

esslli07 3

BNLTRIP LArTPC Bock
08. 11. 2007
0 views

BNLTRIP LArTPC Bock

Love Your Body2
17. 06. 2007
0 views

Love Your Body2

Lucy PM
17. 06. 2007
0 views

Lucy PM

Season change1
02. 11. 2007
0 views

Season change1

prÃsentationecole
05. 11. 2007
0 views

prÃsentationecole

marcenasjourney
05. 11. 2007
0 views

marcenasjourney

Nets1
28. 12. 2007
0 views

Nets1

SPS Search MSDN evening 030604
12. 12. 2007
0 views

SPS Search MSDN evening 030604

From Crayon to Mouse
18. 12. 2007
0 views

From Crayon to Mouse

The Presentation
14. 03. 2008
0 views

The Presentation

ev
21. 11. 2007
0 views

ev

trt02 eccles
05. 11. 2007
0 views

trt02 eccles

Digging Dinos
12. 09. 2007
0 views

Digging Dinos