History in the Making -- Pioneering and Promoting Parallelism

The University of Illinois has a long and distinguished history in parallel computing, beginning with the ILLIAC machines of the 1960s. In particular, ILLIAC IV is widely recognized as an early forerunner in the quest for parallelism, and it inspired many parallel computing projects. In the 1980s, the CEDAR machine and the systems software development led by David Kuck and co-workers at the Center for Supercomputing Research and Development (CSRD) brought Illinois to the forefront of parallel computing — a position it holds to this day. Software development at CSRD led to a start-up company, Kuck and Associates, which was purchased by Intel in 2000. The National Center for Supercomputing Applications and the Computational Science and Engineering Program have contributed to the development of the national expertise base in parallel scientific applications, providing leadership to academic institutions, government, and industry. Over 40 years of sustained effort has brought Illinois to a true leadership position across all areas of parallel computing research, from the design of groundbreaking machines to parallelizing compilers and other systems software, to parallel scientific applications. Thanks to corporate, state, and federal investments in our infrastructure and programs, we are now poised to bring the power of parallel computing to a wide range of applications which will enhance quality of life for all.

While the world at large entered a new era of parallel computing in 2003, Illinois had paved the initial road several decades earlier with the ILLIAC IV. In 1994 a team of University researchers correctly anticipated that speed increases could not continue to be attained indefinitely through the development of sequential computing processors. In that same year, Illinois researchers collaborated with Intel and Hewlett-Packard to create an automatic parallelizing compiler technology for parallel computing processors based on Instruction-Level Parallelism (ILP) techniques. This led to a new Intel microprocessor product line in 2000, the Itanium Processor Family. Since then, Intel has sold about 2 million Itanium processors and openly acknowledges Illinois contributions to the Itanium compiler technologies. This marks the first time in history that automatic parallelizing compiler technology was successfully deployed in mass-market commercial products.

At the same time that researchers in computer science and electrical and computer engineering were developing technology to support parallelism, the National Center for Supercomputing Applications, founded in 1986 as one of five NSF supercomputing centers to serve the national research community, made the shift to scientific applications of massively parallel processing, utilizing advanced architectures like the DARPA-funded Thinking Machines CM-2 and CM-5. Computational scientists teamed with computer scientists to get the most out of these and succeeding architectures, establishing a tradition of applications development for scalable systems that continues today.

The original NCSA proposal was aimed at providing high performance computing resources to the scientific research community. Illinois faculty are known for creating pioneering applications in a wide range of scientific and engineering disciplines, including physics, biology, atmospheric sciences, astrophysics and gravitation. These applications were used to pursue greater performance on each successive advanced computer architecture at NCSA. One well-known application is NAMD, developed by Klaus Schulten, Laxmikant Kale and collaborators, is one example of highly scalable code developed for conducting molecular dynamics simulations on parallel computers. A mainstay for the computational biology community, currently consuming 20% of the total computer cycles used at NCSA and the Pittsburgh Supercomputing Center, NAMD received a 2002 Gordon Bell award. NAMD scales to hundreds of processors on high-end parallel platforms and tens of processors on commodity clusters using gigabit Ethernet, and supports computational modeling in the chemical and pharmaceutical industries, as well as educational applications.

Among the early full-spectrum parallel applications environments is the Illinois Center for the Simulation of Advanced Rockets (CSAR) and the associated academic program in Computational Science and Engineering at the University of Illinois. CSAR is one of five university-based centers of excellence founded in 1997 and funded by the U.S. Department of Energy's Advanced Simulation and Computing program. CSAR's objective is to enable accurate predictions of the performance, reliability, and safety of complex physical systems through computational simulation. Parallel computing has figured significantly in the simulations. Parallel performance monitoring and predictions, meshing, partitioning, and visualization have all been conducted in parallel environments, marking CSAR at Illinois as one of the first comprehensive parallel research platforms.

In 2001, the positive momentum gained from the Itanium compiler development inspired Wen-mei Hwu to collaborate with the Department of Electrical & Computer Engineering and the Coordinated Science Laboratory on a proposal to create a parallel computing thrust in the then-new MARCO Center for Circuits, Systems, and Software (C2S2), funded by the Semiconductor Industry Association and the U.S. Department of Defense (DARPA). In 2006, the semiconductor industry drastically increased the funding level for parallel computing and established a new Concurrent Systems Theme in MARCO's Gigascale Systems Research Center (GSRC). The Concurrent Systems Theme, also led by Wen-mei Hwu, now comprises 14 researchers from 8 major universities, including Illinois, MIT, Berkeley, CMU, Princeton, Michigan, Penn State, and UCLA. As the lead institution of the GSRC Concurrent Theme, Illinois became well positioned to win subsequent competitions for major parallel computing research projects and centers funded by industry and government.

Among the first major success stories in this effort occurred in Fall 2007, when the Illinois/IBM Blue Waters team won the $208,000,000 NSF Petascale computer acquisition CFP competition, over several other highly regarded peer institutions and national laboratories. The Illinois team was led by Thom Dunning, Director of NCSA, with Co-PIs from NCSA and other Illinois academic departments, including Wen-mei Hwu (hardware), Marc Snir (software), and Rob Pennington (overall project management). The departments of ECE and CS and the CSL all contributed substantially to the proposal. As important as building the machine, which will be operational in 2011, are the Petascale Application Computing Teams (PACTs). Their job will be to facilitate the transition of several critical science and engineering applications onto the petascale platform. They will work through and with the Great Lakes Consortium, which comprises virtually all major universities in the computational science and engineering research community, to scale their applications to the petascale. The location of the Blue Waters computer on the Illinois campus, along with the leadership of the Great Lakes Consortium, establishes Illinois as the "unparalleled" leader of the high-end science and engineering parallel computing community.

A second momentous event in Fall 2007 propelled Illinois to the top of the parallel computing community in the eyes of industry as well. Illinois was named by Intel/Microsoft as one of two Universal Parallel Computing Research Centers (UPCRCs) in the nation. The UPCRC at Illinois is co-led by Wen-mei Hwu and Marc Snir, with $10 million in industry funding and $10 million from the Illinois campus. The departments of ECE and CS and the CSL all contributed to the successful proposal and are all integrally involved in the research. Proposals were submitted by 44 universities, and MIT, Stanford, Berkeley, and Illinois were the finalists. In the end, Illinois and Berkeley beat MIT and Stanford to become the two centers.

The focus of the UPCRC is client-based, consumer-oriented computing, intended to take advantage of an increased number of processsors per chip. With a tag line of "Making parallel programming easy," the UPCRC recognizes that multi- and many-core computing is becoming pervasive, and client-focused mass-market applications are now driving parallel programming. The new challenge in parallel computing is not only performance but productivity. This focus addresses the needs of the broad community of parallel application developers, and addresses one of the grant challenges in parallel computing: making parallel programming accessible to all programmers.

Many fundamental computations in the client computing domain share common characteristics with those used in the science and engineering computing communities. Thus the UPCRC extends Illinois's parallel computing leadership from science and engineering into mass consumer markets. This evolution paves the way for Illinois parallel computing research to directly impact the lives of average citizens worldwide.

In spring 2008, several NCSA researchers and UIUC faculty from the departments of Physics, Astronomy, Chemistry, Bioengineering, ECE, and CS joined forces to compete successfully to form a team on "Next-Generation Acceleration Systems for Advanced Science and Engineering Applications" in the newly established Institute for Advanced Computing Applications and Technology (IACAT). Led by Wen-mei Hwu, the faculty and students from Physics, Astronomy, Chemistry, and Bioengineering have successfully adapted several popular research applications for execution by multi-core CPUs and many-core GPUs. Initial successes in the applications projects have been published at the 2008 ACM Computing Frontier Conference, ACM Principles and Practice of Parallel Programming (PPoPP), and the Journal of Computational Chemistry. The team is currently working on the new compiler-adaptable framework approach, proposed to accelerate the parallelization of major science and engineering applications. Industry support has come in the form of major equipment donations from NVIDIA for a novel parallel computing server with 64 graphics processing units, called the "QP Cluster." The QP Cluster has allowed Illinois to conduct research and education work well ahead of MIT, Stanford, Berkeley, and CMU.

Further recognition for Illinois's primacy in parallel computing came in summer 2008, when NVIDIA, a major supplier of graphics processor units (GPUs) , named Illinois the world's first CUDA Center of Excellence, with Wen-mei Hwu as PI. Illinois was recognized as the national leader and granted the first center aimed at applying these powerful specialized devices to parallel computing. Three additional centers will be announced later in 2008. The Illinois CUDA Center is accompanied by a donation of 128 G200 GPUs, along with $500,000 in gift funds for research and education. Dr. David Kirk, the Chief Scientist of NVIDIA is fully engaged in the Center. Dr. Kirk has served as an adjunct professor in the Department of Electrical and Computer Engineering at Illinois and co-taught ECE498AL "Massively Parallel Programming," with Wen-mei Hwu for two semesters in 2007-2008. The Center is making it possible for Illinois to extend this parallel computing course into a global course offering. The Center is already attracting additional industry partners, including Microsoft, which is also beginning to participate in the Center with gift funding and staff contributions.

To complement Illinois parallel computing activities, Illinois is facilitating efforts to establish and lead a global parallel computing application development community supported by on-line technology. Parallel-programming.org tracks global parallel-programming research, from academia and industry, and provides complete coverage of parallel programming news, research papers, white papers, conferences, workshops, and blogs. Building on a long history of innovation in large-scale systems and parallel computing, the Illinois department of computer science has announced the Cloud Computing Testbed (CCT) , a joint research endeavor to investigate Internet-scale cloud computing systems, in collaboration with Hewlett-Packard, Intel, and Yahoo! The global partnership, which also includes the National Science Foundation (NSF), the Infocomm Development Authority of Singapore (IDA), and the Karlsruhe Institute of Technology in Germany, will result in the establishment of a globally distributed, six-site, Internet-scale testing environment, enabling advanced research in systems-level research issues such as automatic resource allocation, scheduling, monitoring, and management tasks that arise in processing and responding to large amounts of data. The proposed research will cover a breadth of research areas including networking, operating systems, virtual machines, distributed systems, data-mining, Web search, network measurement, and multimedia. By coupling this research with Illinois parallel computing expertise, it will be possible to develop advanced applications of data mining and natural language understanding, in areas ranging from social networking to counter-terrorism.

The CCT effort will build on other recent Illinois efforts to advance parallel computing across the spectrum, from Blue Waters to the UPCRC, accelerating research for compute- and data-intensive, Internet-scale computing and drive innovation for future systems. It will enable a better understanding of Internet characteristics, especially emerging online social networks. It will also enable development of open-source tools for network analysis, fast querying, and transfer of distributed logs, search engines yielding navigation maps, semantic Web tools, and configuration for multimedia environments. Strong synergy is expected between our cloud computing efforts and the DHS Center for Multimodal Information Access, led by CS faculty member Dan Roth.

Efforts are now underway to unite all of the above Illinois initiatives in parallel computing under a virtual umbrella to be known as Parallel@Illinois, to provide opportunities to leverage resources and personnel among the Parallel@Illinois affiliates and to help recruit new corporate participation and sponsorship. Parallel@Illinois will provide a unifying representation of parallel computing research at Illinois, while the website (www.parallel.illinois.com) will serve as a community forum for researchers and programmers. Future plans call for a distinguished lecture series, workshops and a summer school.

In summary, parallel computing has become simply . . . computing. From this point on, all computers will be parallel computers, but the research and education needed to build and use these computers are still far from mature. With strong support from industry, government sources, and our own University, Illinois is leading the way with Blue Waters, CSAR, UPCRC, IACAT, CUDA, PPRC, CCT, and Parallel@Illinois. Illinois is, without doubt, the globally recognized leader in parallel computing. We have envisioned and implemented parallel computing, we have developed and educated in parallel programming, and we are extending the impact of our work from science and engineering disciplines into everyday lives.