Parallel and Multi-Core Computing with C/C++

Where is OpenCL and OpenMP headed?

Michael_Wong — Mon, 26 Jan 2009 01:55:49 GMT

The following is a a private communication from an IBM engineer Matthew Markland who asked a great question. I do not claim great expertise but I feel that there is enough of an opinion piece that some folks may like to see this discussion or continue it. I have edited the response somewhat but it is largely in tact and reprinted with Matthew's permission. Note I have no insight into PGI or any other product other then what I read in public articles, and as such makes no product claim. Any opinion regarding other company remain necessarily my own and is not IBM's position.

Michael:

I hope that the new year finds everything well for you and yours.
I'm enjoying the C/C++ Cafe posts you guys put out immensely.

Please join in the discussion, or even bring this up as other experts will chime in.

I just wanted to get your opinion on some things that have been
going through my mind with respect to the multicore/hybrid
programming models that are being put out by various entities. It
seems that many people believe that the best model is an extension
to the language model, be it a pure language extension like what
CUDA and OpenCL have, or with a new model of pragmas like PGI is

OpenCL/CUDA is mostly a library based model and a language extension(modulo the 4 memory annotations). But yes I see where you are going with this ...

adding. I'm wondering, especially in the case of the PGI extensions,

I am assuming this is the pragma directives available in their technology preview:

#pragma acc directive-name [clause [,clause] ...]

whether they make sense given the existing OpenMP spec
Where do you

So there has been parallel languages that are directive based, language extensions, and library based. Usually they start off with library based because they are easy to port, and works on many vendors' compiler. Language-based solutions are harder to implement, and can not be easily corrected if wrong. Directive-based like OpenMP makes it easily adapted in an incremental manner, and keeps the base program running even on platforms that don't accept the directive. Today, we have examples of all three. MPI is a pure library based solution. Cilk is a pure language based solution and OpenMP is a directive-based solution (although it too has a library part).

A mostly library based language like OpenCL is in a sense a step backwards. So PGI is trying a directive based approach to send the computational kernel to the accelerator/GPGPU. This is a bet from their part. I am familiar with their chief compiler engineer on the OpenMP Committee Michael Wolfe, and respects his opinion.

see this headed from a personal perspective.

Having some involvement in OpenCL, I can see where it falls somewhat short, but is nevertheless a tremendous accomplishment. It is designed for today's GPGPU architecture, assumes a weak memory model, implicitly have a dual layer of scheduling policy between the host (outer asynchronous layer) and the thread processors (inner synchronous processors with local memory). This is in addition to it being still relatively hard to program,( though easier then DirectX or OpenGL) and for people who have to port a 100,000 line of code is a large commitment on a technology that may not be around. OpenCL, is still a stream processing language and as such is limited in the scope of the parallel programs it can speed-up. What PGI is probably looking for is a more generalized programming model which works in broader situation. That is why they introduced the scheduling clause, and tied it to OpenMP. I would not be surprised if some kind of heterogenous programming support would be in OpenMP in future.

I don't have any significant personal insight but also is involved in adapting the OpenMP paradigm to fit in the next programming model without knowing where to go.

In the end (and this is based on Michael Wolfe's excellent analogy in an HPC paper), OpenCL is basically designed for a hardware that is a large wide body air carrier that can handle massive number of passengers in one run, but requires special airport transportation to get the passengers to the plane because the plane doesn't fit in the terminal. So the speed it has (in terms of # of passenger-miles) is mitigated by the wait time (DMA access)of loading the plane. It works when everything fits.

If you don't have that many passengers, or have a variable number of passengers, it doesn't buy you any extra benefit and may penalize you with a super wide-body jet. And there are lots of other kinds of air carriers out there, including the super-fast kind for the payload just has to get there by 9 am the next day and the medium sized ones that can carry your particular amount of load.
As such, there will still be a place for OpenMP, MPI, TBB, futures, UPC, TM. We are suffering under an alarming number of these so-called parallel languages/extension/libraries lately and I can only see more as we all search for the right model. At one point, we had the same in terms of sequential languages, and over time we have dwindled down into a few General Purpose languages with many domain-specific languages. The same will likely happen in the parallel language world.

How does OpenMP 3.0 work better with C++ (Part 3)?

Michael_Wong — Fri, 12 Dec 2008 05:33:06 GMT

Let's get to the good stuff. OpenMP 2.5 did not really specify what constructors should be called with various private/first/lastprivate/threadprivate:

In some cases, it did not even specify that they should apply to non-PODs (Plain'ol Data, i.e. C structs).

OpenMP 3.0 changed that. Beside specifying non-PODs, it also specified precise rules for the constructor sequence that is in line with what the semantics would require.

Firstprivate:
For instance, it would specify that a firstprivate for a class type variable should expect an accessible copy constructor, since it is required to initialize each of the one or more list items private to a thread with the value that the corresponding original item has when the construct is encountered.

Lastprivate:
For a class type variable, it requires an accessible, unambiguous copy assignment operator for the class type. And it requires an accessible, unambiguous default constructor for the class type unless the variable is also specified in a firstprivate clause.

Threadprivate:
This is the most interesting as it differentiates three kinds of initializaton in C++.
1. Without initialization: Object1 o;

Default constructor is called

2. Direct initialization: Object1 o( (int)23 );

Constructor accepting the argument is called

3. Copy initialization: Object1 o = other_instance;

Copy constructor is called

The semantics of this is that for the master thread, global static objects and static class members are constructed before main() is entered in an undefined order.

For the slave threads, the exact point in time of object construction is unspecified, but is has to happen before the thread references it the first time

These changes were a long time coming. It causes what used to be vaguely implied by the 2.5 specification, now to be clearly specified so all compilers can conform. It also allows the users of C++ with OpenMP to have more consistent behavior.

OpenMP at SuperComputing 08

Michael_Wong — Mon, 08 Dec 2008 15:16:30 GMT

The OpenMP Birds-of-a-Feather session at SC08 was very well attended. The room was full to overflowing, with approximately 60-80 people. while OpenMP had BOFs at SC in prior years, this is actually the first year that OpenMP has had a Booth on the Exhibitor floor as well.

The BOF had many elements, including what is new in OpenMP 3.0. They were:
1. Welcome and summary of ARB news
- 5 mins
- Larry Meadows

2. The three greatest things about OpenMP 3.0 and the three most
important things left out of OpenMP 3.0
- 15 mins
- Tim Mattson

3. Tasking
- 10 to 15 mins
- Alex Duran

4. Extending the OpenMP profiling API for OpenMP 3.0
- 10 to 15 mins
- Oleg Mazurov

5. Announcements:
- 5 mins total
- IWOMP'09 announcement - Matthias Mueller
- OpenMP book examples - Ruud van der Pas

6. Panel "How to kill OpenMP by 2011"
- 35 mins
- Moderator:
Larry Meadows
- Panelists:
Tim Mattson
Bronis de Supinski
Christian Terboven
Jesus Labarta

7. Wrap up
- 5 mins
- Larry Meadows

For a description, you can find lots of detail here, including a downloadable summary card of the 3.0 Specification.

http://openmp.org/

I met a number of people during my afternoon session manning the booth on Wednesday 2-6 pm, including one of the original founder of OpenMP, an instructor on using OpenMP in graduate courses as well as a consultant, among many others. I invite them all to drop me a note here so we can continue our discussion.

One thing that was surprising to many folks was the many compilers that already have 3.0 implementation, despite the specification ratifying only in May 2008. The companies with 3.0 implementation includes IBM, Sun, PGI, and soon Intel. Note I am just reading this from the net and have no insight into other company's or even my own company's release schedule. GNU should have something by 4.4. Same disclaimer.

One thing that happened at the OpenMP BOF was a panel discussion on How to Kill OpenMP by 2012. This is kind of a fun session, especially at the end of a long day to not take yourself too seriously. The point is to showcase all the wrong ways in spreading a de-facto standard.

I have had some experience working with language designs through my various roles as standard rep, and compiler writer. So I thought I would give my $0.02 here on How to Kill OpenMP by 2012:
10. Don't implement the specification as stated.
9. Make it impossible to nail down ambiguities by having no way of addressing defects.
8. Ignore the user forum or suggestions
7. Add everyone's favorite feature, no matter how marginally useful.
6. Make the process as non-transparent as possible, so no one knows when you are ratifying, or even what you are doing.
5. Debate endlessly, on anything, not necessarily having anything to do with the language.
4. Design a feature as completely as possible before releasing it
3. Make no concession when taking a stance on objecting to someone's feature.
2. Form close-nit elitist groups and follow the NIH syndrome.
1. Don't organize any meetings, and when there are meetings, don't follow any rules.

Seriously, I have not found this to be a problem in any of the committees that I am a part of. Otherwise, we would not have made any progress. But we can always do better.

How does OpenMP 3.0 work better with C++?

Michael_Wong — Thu, 04 Dec 2008 02:51:20 GMT

This was one of the question asked at SC 08. I will try to answer that here. I will start and add more as I move through the various topics.

OpenMP 3.0 had better support for C++ in the following areas:

Parallelization of RandomAccess Iterator loops with strict canonical operators
Threadprivatization of static class members
Unsigned loop control variable support
Fully specify constructor call requirement in private/first/lastprivate/threadprivate
better match with the C++0x memory model

For-Worksharing with Iterator loops:

We specifically enabled C++ RandomAccess iterators and C pointeres to be parallelized with explicit directives.

#include <vector>
void iterator_example()
{
   std::vector<int> vec(23);
   std::vector<int>::iterator it;
   #pragma omp parallel for default(none) shared(vec)
   for (it = vec.begin(); it < vec.end(); it++)
   {
   // do work with *it //
   }
}

I will follow up with more examples.

How does OpenMP 3.0 work better with C++ (Part 2)?

Michael_Wong — Fri, 28 Nov 2008 17:04:13 GMT

This part will talk about enabling threadprivatization of static class member variables.

In 2.5, as a result of ambiguous language, the support for this was inconsistent. In general, it would claim that a threadprivate variable must be namespace, file or block scope.

In 3.0, this code is now allowed:

class T {
   public:
      static int i;
      #pragma omp threadprivate(i)
};

This may seem a trivial change, but for C++, it enables a powerful idiom of singletons and allocators, which all rely on static class member variable.

Next posting will continue with the semantics of private, first/lastprivate, threadprivate+copyin/copyprivate for C++.

OpenMP Booth at SuperComputing 08

Michael_Wong — Wed, 26 Nov 2008 22:21:44 GMT

I should include an obligatory photo of the OpenMP Booth with our CEO Larry Meadows. Many members worked tirelessly, especially Larry at making sure the Booth was put up, taken down and staffed properly.

We also met many people who specifically dropped by to see what was going on, in this first year where we had a booth.

Many Thanks.

Questions on C/C++ volatile from SC08 Tutorial S08: A Hands-on Introduction to OpenMP

Michael_Wong — Thu, 20 Nov 2008 16:53:19 GMT

On Sunday, I attended the OpenMP Tutorial hosted by some of my colleagues from the OpenMP committee, Tim Mattson and Larry Meadows. It was an excellent tutorial, full of features and pointing out the behavioral differences between OpenMP 2.5 and 3.0. Tim has been doing this tutorial for almost 20 years and Larry, as the OpenMP CEO has been involved in this for the same amount of time.

I must point out that IBM xl C/C++ 10.1 compiler for AIX and Linux both have support for OpenMP 3.0, in addiiton to the compilers mentioned during the tutorial. In fact, this compiler was one of the first to have support for OpenMP 3.0 in the industry since the ratification of the specification in May, 2008.

http://www-01.ibm.com/software/awdtools/xlcpp/aix/
http://www-01.ibm.com/software/awdtools/xlcpp/linux/

You can get a trial version from the link above.

There were a few C++ specific questions. During the discussion on OpenMP memory model, one questions involves how volatile in C/C++ is used in multithreaded program.

The C++ 0x Standard will clarify that the volatile keyword continues to have nothing to do with multithreading. It merely indicates that something from the environment may change the value. The role of an atomic variable will be used to indicate that another thread may change the value of this variable. It is possible to say that a variable is an atomic volatile to indicate that something from the environment AND another thread may change the value. The C standard, when it supports concurrency in C1x will likely adapt the same meaning.

So in short, C++ Volatile !=Jave volatile and C++ Volatile!= C++ atomic. During C++0x concurrency deliberations, we discussed at length and felt that there is too much history and presumed meanings involved with volatile. In fact, it may even have different meanings varying from compilers to platforms. So it was felt wiser to leave it alone, allowing implementers to retain whatever meaning they are used to, and adapt a new way of naming truly atomic types.

I hope this helps to answer one of the question regarding the role of volatile.

There was another question about how OpenMP 3.0 changed the role of what constructor (default, copy, assignment) is called by each of the private/threadprivate/first/lastprivate variable. Tim was right that we worked very hard to clarify this aspect, but the rules while somewhat intuitive is still fairly tedious to enumerate, given the many ways that C++ does initialization. I will clarify that in a subsequent post.