4th IEEE International Conference on Information Technology: New Generations ITNG 2007 April 2-4, 2007, Las

Vegas, Nevada, USA

Design and Implementation of a High Performance MPI for Large Scale

Computing System

Prabu D, Vanamala V, Sanjeeb Kumar Deka, Sridharan R, Prahlada Rao BB†, Mohanram N†
Centre for Development of Advanced Computing (C-DAC), Bangalore, INDIA
(prabud, vanamalav, sanjeebd, rsridharan, prahladab, mohan)@cdacb.ernet.in

Abstract
The necessity to process large application in
reasonable time has lead to the use of parallel
processing and the use of programming standards,
which enable the running of user application in a
parallel environment. This paper presents C-DAC
Message Passing Interface (C-MPI). C-MPI is an
optimized MPI for Cluster of Multiprocessors
(CLUMPS). It has been able to exhibit higher
performance compared to public domain MPI such as
MPICH. In this work we have presented the design
flow of C-MPI and compared the various performance
of C-MPI and MPICH over PARAMNet-II Interconnect
using KSHIPRA as the underlying protocol.
1. Introduction
The enormous advance in the field of PCs and high-
speed networks has led to low-cost clusters of personal
computers, which have been able to provide the
computing power equivalent to that of supercomputers.
As per the Moore’s law, the processor speed doubles
every 18 months [1]. The advances in hardware, CPU
technologies, and high-speed communication network
enable the parallel application developers to use the
Message Passing Interface (MPI) standard
implementation. A panel of parallel programming
industry leaders including representatives from the
national laboratories, universities, and key parallel
system vendors define the MPI standards [2]. This
leads to the development of commercial MPI having
low latency and high bandwidth. The central idea of the
paper is to provide the user a high performance
optimized parallel programming Message Passing
Interface over high speed interconnect to solve his
complex applications in a parallel programming
environment.
______________________________________________________
†Any further correspondence regarding this publication can be
addressed to (mohan, prahladab)@cdacb.ernet.in
Rest of the paper is organized as follows. Section 2
provides a brief overview of C-MPI. Section 3 is the
brief overview of MPICH. In Section 4, we have
presented a brief description of PARAMNet-II. In
Section 5, experimental results have been discussed
and compared to that of MPICH. Finally, in Section 6,
Conclusion and Future Work are presented.
2. C-DAC Message Passing Interface
C-MPI is a high performance implementation of the
MPI standard for a Cluster of Multiprocessors
(CLUMPS). By adhering to the MPI standard, C-MPI
supports the execution of the multitude of MPI
applications with enhanced performance on CLUMPS.
C-MPI can work on both TCP/IP using Ethernet and
KSHIPRA [3] over PARAMNet-II. It also leverages on
the fact that most of the high performance networks
provides a large exchange communication bandwidth.
It helps in performing send and receive operation of
messages in a simultaneous manner, consequently
reducing the number of hops in the transmission path.
In C-MPI, different MPI collective communication
calls have been optimized for CLUMPS architecture.
The C-MPI algorithms effectively use the shared
memory communications on multiprocessor nodes and
thereby reducing the computation time of the user
application. C-MPI provides the following advantages
over public domain MPI, MPICH [4]:
• Supports multiple protocols. It uses the
network for remote communication and shared
memory for local communication.
• Collective communication routines have been
optimized for CLUMPS to achieve minimized
execution time.
C-MPI is designed to achieve high performance and
portability. It is layered over Abstract Device Interface
(ADI) to maintain portability. On C-DAC’s PARAM
Padma [5], C-MPI employs both TCP/IP and
KSHIPRA in the underlying ADI layer. The C-MPI
4th IEEE International Conference on Information Technology: New Generations ITNG 2007 April 2-4, 2007, Las
Vegas, Nevada, USA
functions are implemented in terms of macros and
functions. The Fig 1 shows the C-MPI control flow.
The upper layer does the communication of control
information and the lower layer performs the transfer of
data from one process address space to another.
User Task (C-MPI Application)
API (upper layer)
Collective
Communication
Point-to-point Communication
Protocol Module (lower layer)
TCP/IP KSHIPRA Shared
Memory
PARAMNet-II/C-VIPL Library
Figure 1. C-
C-MPI Control Flow
For achieving optimal performance characteristics, C-
MPI can directly work over the System Area Network
(SAN) in the user space using lightweight
communication protocols (KSHIPRA). This results in
decrease in communication time for MPI point-to-point
communication protocols. The communication, at the
lowest level, occurs in point-to-point manner as shown
in the Fig. 1. Hence, reduction in the communication
time of the point-to-point communication calls lead to
reduced communication time for the collective
communication calls as well.
3. MPICH
MPICH[6] is an open source, portable
implementation of MPI standard which is a widely used
parallel programming paradigm for distributed memory
applications used in parallel computing. MPICH is
available for different flavors of Unix (including
Linux) and Microsoft Windows. MPICH is jointly
developed by Argonne National Laboratory and
Mississippi State University.
MPICH supports one-side communication, dynamic
processes, intercommunicator collective operations,
and expanded MPI-IO functionality. Moreover, it can
work over clusters consisting of both single-processor
and SMP nodes. MPICH implementations are not
thread safe. MPICH uses TCP/UDP socket interfaces
to communicate messages between nodes. Because of
this, there have been great efforts in reducing the
overhead incurred in processing the TCP/IP stacks. To
overcome this problem MPICH is now enabled to
support Virtual Interface Architecture (VIA). VIA[7]
has defined different mechanisms that enable bypassing
layers of protocol stacks and avoid intermediate copies
of data during sending and receiving of messages. This
allows significant increase in communication
performance and decrease in processor utilization by
the communication subsystem.
4. PARAMNet-II
PARAMNet-II is high bandwidth, low latency
System Area Network (SAN) developed by C-DAC for
its PARAM supercomputers. Its usability ranges from
commodity compute servers to high performance
scalable systems. The different components of
PARAMNet-II are: Network Interface Card (NIC), one
or more routing switches and software. PARAMNet-II
[8] is the second-generation System Area Network of
PARAMNet series.
Figure 2. PARAMNet-
PARAMNet-II Card
PARAMNet-II is having a 16-port non-blocking
crossbar based switch, which provides a bandwidth of
2.5 Gigabit per second. Multilevel switching strategy is
used for supporting the 16 ports of the switch. The
switch can operate in full duplex mode with latency as
low as 0.5 Microseconds. Distributed scheduler has
been used for controlling the crossbar. Distributed
schedulers allow individual routing table per port,
allowing for any network topology. Each of the
schedulers gets the input data from a flow control block
that handles the low-level hardware handshake between
the end points. For uniform distribution of bandwidth
in a group, the switch uses Group adaptive routing
4th IEEE International Conference on Information Technology: New Generations ITNG 2007 April 2-4, 2007, Las
Vegas, Nevada, USA

based on Least Recently Used (LRU) algorithm. The
PARAMNet-II CCPIII NIC is based on C-DAC’s
Communication Co-Processor III chip. It supports
direct user level access with protection and up to 1024
connections. Address translation, packetization and re-
assembly of messages are done in the hardware itself.
PARAMNet-II uses an Application Programming
Interface (API) called C-DAC’s Virtual Interface
Provider Library (C-VIPL). The later is the interface
for SAN that adheres to C-DAC’s KSHIPRA.
Fig 3. depicts the results of running HPL benchmark
on 62 4-way nodes for C-MPI and MPICH. The
sustained performance for C-MPI is found to be 532
Gigaflops against the calculated peak performance of
992 Gigaflops. However, it is 495.80 Gigaflops for
MPICH. The results show that C-MPI clearly
outperforms MPICH in Fig 3.
Performance of HPL on PARAM Padma
Block Size Nb =200
600

Performance
5. The Experimental Results

(Gflops)
400

200 C-MPI
5.1. Experimental Setup MPICH
0
The experimental evaluation for performance of C- 10000 43895 244000
Problem Size (N)
MPI is done on PARAM Padma at CTSF [9], C-DAC
Knowledge Park, Bangalore, India. The specification
of PARAM Padma shown in Table 1 currently has a Figure 3. Performance comparison for C-
C-MPI
peak performance of one Teraflop. The performance and MPICH for HPL Benchmark
testing of 32-bit C-MPI-1.0 for comparison with the
public domain MPI, 32-bit MPICH-1.2.4 is done on 2, 5.2.2. NAS: The NAS [11] Parallel Benchmarks suite
25 32 and 62, four way nodes running on AIX-5.1 is widely used to compare the performance of different
operating system. types of parallel computing platforms, since it contains
computational kernels that are representative of several
Table 1. Description of PARAM Padma different algorithms used in real-world applications.
NAS 2.4 constitutes eight CFD problems, coded in
Specification Compute node File servers
MPI and standard FORTAN 77/C.
62 nodes of 4 The results of NAS benchmarks on PARAM Padma
way SMP and 6 nodes. of are as follows. The performance interms of Mflops for
Configuration
one node. of 4 way SMP
selective benchmarks are shown in Table 2.
32 way SMP
Optimization Flags used are –qstrict –O3 –bmaxdata :
248(Power 4 24(UltraSparc
No. of processors 0x80000000 and RAND value is randi8
@1GHz) IV@900MHz)
Aggregate 0.5 96
memory Terabytes Gigabytes Table 2. NAS Performance of C-
C-MPI and MPICH
4.5 0.4 over PARAMNet-
PARAMNet-II
Internal storage Benchmarks Class / # of Performance (MFlops)
Terabytes Terabytes
Processors MPICH C-MPI
Operating system AIX Solaris
BT C/25 5223.28 5362.87
Peak computing CG D/32 757.68 834.97
power for 62 AIX 992GF(~ 1 TF) --
IS C/32 113.14 117.16
nodes
LU C/32 6325.97 9236.28
File system -- QFS D/32 7226.80 7681.42
MG C/32 5159.88 5906.21
D/32 6677.41 7056.79
5.2. Performance Evaluation
SP C/32 3139.60 3462.83

5.2.1. HPL: HPL [10] benchmark is a numerically

5.2.3. PMB: PMB (Pallas MPI Benchmark)[12] is
intensive test. It is a popular benchmark suite to
complex benchmark used for measuring MPI
evaluate the performance of Super Computers and
performance. It consists of a concise set of benchmarks
Clusters and involves solving a dense linear system in
targeted at evaluating most important MPI functions.
double precision (64 bits) arithmetic on distributed
The different benchmarks under PMB are PingPong,
memory system. HPL benchmark is used to test the
PingPing, Sendrecv, Exchange, Allreduce, Reduce,
PARAMPadma cluster efficiency.
4th IEEE International Conference on Information Technology: New Generations ITNG 2007 April 2-4, 2007, Las
Vegas, Nevada, USA
Reduce-scatter, Allgather, Allgatherv, Alltoall, Bcast
and Barrier. In Table 3, output of PMB (Pallas MPI
Benchmark) is depicted. The latency is obtained using
Pingpong benchmark across two AIX nodes. It is found
that the latency of C-MPI is smaller compared to
MPICH.
Table 3. Latency for C-
C-MPI and MPICH
Latency for zero byte
MPI Type
message
MPICH 29.18µs
C-MPI 21.38µs
5.2.4. P-COMS: P-COMS [13] comprises of a set of
MPI benchmarks used for measuring communication
overheads on large message passing clusters (such as
PARAM 10000, PARAM Padma). The benchmarks
have been implemented using Message Passing
Interface standard. The different benchmarks under P-
COMS are alltoall, ptp, advptp, cc, ccomp, gppong,
roundtrip, allgring, oneway, and circularshift. In Table
4, below shows the performance of the two MPIs for P-
COMS Benchmark. From the result it is evident that C-
MPI is having low latency and High bandwidth
compared to MPICH over PARAMNet-II interconnect.
Table 4. Latency and Bandwidth for C-MPI and
MPICH for P-COMS Benchmark under
PARAMNet-II
Communication
Overhead
MPICH C-MPI
Parameter
Latency for zero 22.39µs
29.82µs
byte message
Bandwidth for
101.13MBps 120.00MBps
10MB
Performance test using HPL, NAS, Pallas, and
PCOMS benchmarks clearly reveals that C-MPI
performs better than that of MPICH over PARAMNet-
II. (Note the all HPC benchmark results are taken for
best performance). From Table 4, we have seen that the
bandwidth provided by C-MPI is much higher and low
latency compared to MPICH.
6. Conclusions and Future work
In this paper we present the design flow and
Implementation of C-MPI over PARAMNet-II in
PARAM Padma. Based on the experimental results
shown in Section 5.0 using standard HPC benchmarks,
we conclude the performance of C-MPI is better than
that of MPICH over PARAMNet-II Interconnect
platform. This work can be extended to enable C-MPI
for InfiniBand Architecture Interconnect and also for
the Grid environments [14].
7. References
[1] Carlo Kopp, “Moore’s Law and it’s Implication for
Information Warfare,” The 3rd International Association of
Old Crows (AOC) Electronic Warfare Conference
Proceedings, Zurich, May 20-25, 2000.
http://www.ausairpower.net/moore-iw.pdf
[2] The Message Passing Interface Standard
http://www-unix.mcs.anl.gov/mpi/
[3] KSHIPRA - Scalable Communication Substrate for
Cluster of Multi Processors
http://www.cdac.in/HTmL/ssdgblr/kshipra.asp
[4] MPICH-A Portable Implementation of MPI
http://www-unix.mcs.anl.gov/mpi/mpich1/
[5] C-DAC, India, PARAM Padma Supercomputer
http://www.cdac.in/html/parampma.asp
[6] W. Gropp, E. Lusk, N. Doss and A. Skjellum, “A high-
performance, portable, implementation of the MPI Standard,
Parallel Computing: 22: 789-828, 1996.
http://www.globus.org/alliance/publications/papers/paper1.p
df
[7] Virtual Interface Architecture
www.intel.com/intelpress/via
[8] C-DAC, PARAMNet-II, NIC
http://www.cdac.in/HTML/htdg/products/pnic.asp
[9] C-DAC’s Tera Scale Supercomputing Facility, C-DAC
Bangalore www.cdac.in/html/ctsf/
[10] A. Petitet, R. C. Whaley, J. Dongarra, A. Cleary, “HPL-
A Portable Implementation of The High-Performance
Linpack Benchmark For Distributed-Memory Computers,”
Innovative Computing Laboratory, University Of Tennessee,
January 2004. http://www.netlib.org/benchmark/hpl/
[11] The Numerical Aerodynamic Simulation, NAS Parallel
Benchmarks. www.nas.nasa.gov/Software/NPB/
[12] Pallas MPI Benchmark (PMB), Intel,
http://www.pallas.com/e/products/index.htm
[13] PARAM- Communication Overhead Measurement
Suites (P-COMS), Center for Development of Advanced
Computing http://www.cdac.in/html/betatest/hpc.asp
[14] GARUDA C-DAC Bangalore INDIA,
The National Grid Computing Initiative.
http://www.garudaindia.in/tech_research.asp