Hitting the Wall

No, this isn’t going to be another fabricated memoir about drug addiction. This is my attempt to shed some light on what I believe is a rapidly approaching paradigm shift in parallel computing; General Purpose Graphics Processing Unit computing (GPGPU). I am a Systems Architect at a large financial institution. As you have probably read on the web, we robber barons of Wall Street use computer models concocted by PhD’ed rocket scientist and programmed by people like me. These sophisticated models enable us to build systems that virtually print money (well not really but pretty damn close). These systems require an enormous amount of computational resources to function. We typically satisfy these needs by parallelizing our computations across large farms of servers.

At its core parallel computing is using multiple computational resources in parallel to solve a computational problem. The goal of solving computational problems in parallel is to save time or money or both. Additionally some problems are so large that their only practical solution is through parallel computing. Parallel computing is used in the areas such as graphics processing, computational finance, data mining, seismology, mathematics and physics just to name a few.

Parallel computing is becoming more critical to the computing world due to the impending collapse of Moore's law. Gordon Moore, a co-founder of Intel, noted that the number of transistors that can be placed inexpensively on an integrated circuit increases exponentially doubling every two years. More transistors mean faster more powerful CPUs. This observation coined the phrase "Moore's Law". Gordon Moore also coined a lesser know phrase "Moore's Second Law".

Moore's second law states that the R&D, manufacturing and QA cost associated with semiconductor fabrication also increase exponentially over time. Moore’s second law hypothesized that at some point the increasing fabrication cost coupled with the physical limitations of semiconductor fabrication materials will erect a brick wall halting the exponential advance of CPU clock rates. While Gordon Moore did not predict when this halting would occur, recent products from Intel and AMD would suggest that they (we) have hit it. Both companies seem to have at least temporarily abandoned their relentless quest for faster CPUs in favor of CPUs consisting of many cores.

In the past if a programs performance was less than adequate for the user you could typically just buy more powerful hardware to make it faster. With the impending collapse of Moore's law this will no longer be true. So until they come up with subatomic computing, programmers will need to embrace parallel computing more in order to squeeze performance gains from their applications.

So why am I posting this? Because GPGPU is, in my opinion, going to have a major impact on the way programmers parallelize their applications in the future. By adopting technologies like CUDA / OpenCL programmers will be able to tap into a vast amount of computational resources at a very low cost. Those who quickly adapt to this paradigm shift will thrive and prosper… those who don’t will wither and die.

So you want to know more about this GPGPU thing...?
See my next post “GPGPU 101”.