What is so Hard About Parallel Programming?

I've seen it again.  One more claim that "Parallel programming is hard" and then the claimant launching into some complex and convoluted solution to make parallel programming easier.  (Matt Wolfe has some interesting observations in this regard.) How hard is it to do/learn parallel programming?

Disclaimer: I've been doing parallel/concurrent/multithreaded programming in one form or another for over 20 years. I want my readers to understand that my opinions in this area may be tainted by my experience. I watch chef Gordon Ramsay whip up something in the kitchen, making it look easy, but I know I'd never be able to do the same thing with the same ease. He's had years of practice, and I know that I might be able to get closer to his culinary skills with some instruction and lots of practice. It's the same with parallel programming, IMO.

I do know something that is hard: Theory of Computation. O-M-G! All that stuff about grammars, finite state machines, context-free languages, pumping lemmas, Turing Machines, recursive functions, uncomputability, Church, Post, and Kleene. It was enough to make my head swim when first confronted with all of this. (If your head is swimming with the memory of these topics, too, pause now and take a deep calming breath.) With mega-liters of water under the bridge since those harrowing days, I no longer wake up in the middle of the night screaming about being asked to solve the Halting Problem.

Recently, I started reading Feynman Lectures on Computation (Perseus Publishing, 1996), edited by Tony Hey and Robin W. Allen. These are transcriptions of lectures delivered by Richard P. Feynman during the mid-80s at the California Institute of Technology. Chapter 3 deals with the Theory of Computation. It is 40 pages long and covers a range of topics from finite state machines to Turing Machines to uncomputable functions and the Halting Problem.  Forty pages(!) on stuff that took me a whole semester to cover. Can it really be that hard? Prof. Feynman (a Physics professor) certainly makes it look easy.

Granted, there was a lot of stuff I covered in my semester course that was left out of Feynman's lecture. But, on further reflection, he didn't need to cover it to make his connections and to give his students (likely non-CS majors) enough background to allow them to be conversant with the topic and to get started on their own investigations. I believe that Parallel Programming can well take this path, too.

My thoughts about the proponents of the "Parallel Programming is Hard" line is that they are probably thinking about the totality of all the things one needs to have mastered to be an expert and effective parallel programmer in all situations. That is a lot of stuff and could be thought of as "hard." But do we really need to teach it all at once, in one sitting as it were?  Couldn't we start by hitting the highlights and give some previews about what the future issues are that will give students enough information to be practical parallel programmers?

I think that learning parallel programming is no harder than learning a second programming language. C++ and Fortran and Java have syntax rules and methods of coding that must be mastered in order to write programs, as do Perl, Lisp, COBOL, Python, Ruby, Scheme, APL, C#, and a thousand other languages. Any given parallel or multithreaded programming model/method/API also has syntax rules and methods of coding that must be mastered in order to write programs.

Some may say that the methods and other pitfalls in parallel programming are so different from sequential programming, and this is what makes it "hard." Well, COBOL and Lisp are two very different programming languages, but I've been able to learn both of these during my career.  While I may not be proficient in either of these right now, I feel I could easily brush up on the details and be able to achieve at least a journeyman level of skill.

For any university degree program that involves computer programming, journeyman seems like a sufficient level of skill in parallel programming to ask a sophomore to have achieved. (Is there any university/college that wouldn't expect their rising juniors to have at least two programming languages under their belts?) If more specific parallel programming skills are needed, those niggly, more advanced details can get taught during the junior and senior years. To get to the journeyman level, we need to add parallel programming fundamentals and principles (not all the esoteric details) in the first programming course and continue to exercise those skills beyond that.

What about the creative side of parallel programming? I've often heard that programming is more art than science. I think that this is doubly true for parallel programming. We can easily teach the mechanics of programming. But, if there isn't an innate talent for logical thinking and breaking down a problem into a series of steps, a student isn't going to stick around in a CS program for very long. Once parallel programming becomes a part of the university curriculums, especially if it starts out on Day One, we'll still have people enroll in courses only to later discover that they can't grasp the art.

To summarize, I don't think parallel programming is any harder than learning a second programming language and, taking a cure from Richard Feynman, we don't need to fill in all the details of problems and pitfalls inherent in concurrent algorithms in order to start teaching parallel programming at the outset of a student's academic career. We don't need to breed a race of "Atomic Super-Programmers." Give them enough to get started with some chance of success in a controlled environment and fill in the blanks later, as needed. That doesn't sound all that "hard," does it?