New Parallel Programming tools

Ease of programming, portability and superior performance are the vital goals of achieving high productivity in the HEC. Although MPI provides portability and high performance, it is widely known as hard to program. That difficultly fuels the gap between the advances of parallel programming and programmers' knowledge I mentioned on my May 10th blog.

There are many alternatives proposed in the recent years to provide high productivity. I mention some of them in this post.

Global Arrays: this tool kit provides "shared memory" programming interface for distributed memory computers to develop MIMD parallel programs.

UPC: Unified Parallel C is an extension of the C programming language that provides the user with an explicit model of a distributed shared memory system.

Co-array Fortran is an extension to Fortran 95 for Single program multiple data (SPMD) parallel processing.

Cluster OpenMP is an OpenMP extension for clusters. OpenMP is a popular parallel programming paradigm for shared memory multiprocessors. Intel has extended this paradigm for clusters and provides as a component of Intel 9.1 compiler. The performance seems promising, but I have to test it someday to see how good it is.

Revitalization

High end computing (HEC) is a major strategic tool for science, engineering, and industry. HEC simulations in various areas of science enable to understand the world around us. They study the universe, enabling us to observe the systems that are too small (nanotechnology, biotechnology), too large (astrophysics, hurricanes, tsunamis, aircraft), or too dangerous (nuclear weapons) for direct experimental observation.

Over the past few decades, supercomputers have grown from a few Gigaflops of computing power to hundreds of Teraflops of power. New projects are already aiming for tens of Petaflops. If there is anyone who doubts whether that much computing power is needed, there are many applications in the fields mentioned above that require a lot more computation than Petaflops. Here is a talk by Dr. David Bailey of LBNL that explains about 21st century high-end computing.

In 2003 High End Computing Revitalization Task Force (HECRTF) is formed to develop a plan for undertaking and sustaining a robust Federal high-end computing program. In 2004, a Federal HEC plan is prepared. Numerous open research issues can be found in that plan.

Real Bottleneck

Here's an interesting article discussing that the gap between advances of parallel computing and the knowledge of parallel application developers (scientists) is a serious one. Even though there are many optimized libaries and rapidly growing supercomputers, making them available to scientists in various fields is an important issue. As for computer science researchers the fight to overcome the gap between peak performance and sustained system performance (divergence problem) is pivotal.

What's the next programming model for HEC?

Among existing programming models for high-end computing, MPI is the most widely used. While the MPI standard is for message-passing, other popular programming models are: OpenMP, PVM, HPF, UPC. Michael Ball's blog gives some numbers on usage of these models. As he mentioned, it is widely considered MPI is difficult to program. There is a search for new models.

With the arrival of new multi-core processors, clusters of SMPs (known as Clumps) are the future. There is a very high possibility that these clusters are going to have heterogeneous muti-core processors such as Cell. So, the question is what is the most effective and optimal programming model for these new architectures? Is a new programming model necessary just because MPI is difficult to program? If a new programming model is needed, what should be the features of it? Hybrid MPI/OpenMP seems to be a temporary solution for cluster of SMPs. What is the future of this temporary solution? These questions open research questions. I am in the quest for finding answers to these questions and will keep it posted here if I find any. If you know any existing solutions, please comment or e-mail me.

So many MPI implementations!

Here is an interesting article about "why there are so many MPI implementations" by Jeff Squyres. Some of the prominent MPI implementations are MPICH1, MPICH2, LAM/MPI, Open-MPI, Intel-MPI, MPI/PRO, WMPI, MVAPICH (MPI over Infiniband). List of some other implementations is available at http://www-unix.mcs.anl.gov/mpi/implementations.html. I have some experience in working with MPICH2, where we improved the performance of MPI derived datatypes. My fellow Ph.D. student has a version of MPI called MPI-Mitten to enable process migration in MPI. It seems, everyone wants to optimize MPI implementation for the resources (supercomputer, network interface, memory heirarchy in their machine etc.) available to them based on the need for their application. In my view this is something similar to a recent Cray motto: "adapt the system to the application - not the application to the system". With MPI, it is "adapt MPI implementation for the application".

Who am I?

I am Suren Byna, a Ph.D. candidate in Computer Science Dept. at Illinois Institute of Technology, Chicago. My research area is in High End Computing (HEC). In specific, my research work is related to optimizing memory performance for High End Computing. Recently there has been a lot of talk about whats next in HEC. Many articles and editorials are published in HPCwire. There is some discussion about what is the next HEC programming model. So, I thought I could compile more information regarding future HEC trends combined with my views and others'. I will be posting soon.