Parallel Programming (CS 4961)

Fall 2009

Schedule: Tues./Thur., 9:10-10:30 AM
Location: WEB L112
Instructor: Mary Hall
Office Hours: MEB 3466; Tuesdays, 10:45-11:15 AM; Wednesdays, 11:00-11:30 AM or by appointment
Mailing list: cs4961@list.eng.utah.edu
TA: Sriram Aananthakrishnan (office hours: MEB 3115; Mondays and Wednesdays, 2:00-3:00 PM)

Jump to: [Background] [Grading] [Textbook] [Schedule] [Assignments] [Policies]

 Background and Description

This course is a comprehensive exploration of parallel programming paradigms, examining core issues, surveying the set of contemporary parallel programming models, and providing context with a small set of parallel algorithms. In the last few years, this area has been the subject of significant interest due to a number of factors. Most significantly, the advent of multi-core microprocessors has made parallel computing available to the masses. At the high end, major vendors of large-scale parallel systems, including IBM, Cray and Sun, have recently introduced new parallel programming languages designed for applications that exploit tens of thousands of processors. Embedded devices can also be thought of as small distributed-memory multiprocessors. The convergence of these distinct markets offers an opportunity to finally provide application programmers with a productive way to express parallel computation.

The course will be structured as lectures, homeworks, programming assignments and a final project. Students will perform four programming projects to express algorithms using selected parallel programming models and measure their performance. The final project will consist of teams of 2-3 students who will implement codes by combining multiple programming models.

Prerequisites: CS 4400

 Grading

30%Programming projects(P1, P2, P3, P4)
25%Written homeworks
20%Midterm quizzes
25%Final project

Note:Late homeworks are not allowed. Late projects will incur a 20% penalty per day.

 Textbook

Principles of Parallel Programming by Calvin Lin and Lawrence Snyder (ISBN-10: 0-321-48790-7).

 Schedule (tentative)

As this is the first time I have taught this material to undergraduates and at University of Utah, the following schedule is tentative and subject to change. The readings listed should be completed ahead of class so that you can follow the lecture and respond to questions. Homeworks and programming assignments are due before class so that we can discuss them.

Date Topics Read Due Notes
25 Aug Introduction (ppt) (pdf)
Importance of parallel programming 		Chapter 1, pgs. 2-8, 25-26 - -
27 Aug Introduction to parallel algorithms and correctness (ppt) (pdf)
Concerns for parallelism correctness 		Chapter 1, pgs. 8-24 HW01 -
01 Sep Parallel Computing Platforms and Models of Execution (ppt) (pdf)
A diversity of parallel architectures, taxonomy, and 6 examples 		Chapter 2, pgs. 30-59 - -
Parallel Programming Model Concepts
03 Sep Data and Task Parallelism (ppt) (pdf)
Relate execution models and programming models, Peril-L syntax 		Chapter 2, pgs. 87-96 HW02 -
08 Sep Data and Task Parallelism, cont. (ppt) (pdf)
Abstractions for task parallelism and introduction to SIMD 		- - -
10 Sep
Single-Instruction Multiple Data (SIMD) Programming/Execution Model (ppt)
 (pdf)
Example: Intel SSE-3 Multimedia Extension 		- - -
15 Sep Detailed Example from Homework 2 (ppt)
(pdf)
And prep for proj1 		- P1 -
17 Sep
Introduction to Threads (ppt) (pdf)
Data Parallelism Expressed in OpenMP 		Chapter 6, pgs 145, 193-199 - -
22 Sep
Task Parallelism in OpenMP (ppt) (pdf)
Open MP 3.0 		- - -
24 Sep Thread Libraries (ppt) (pdf)
Example: Thread Building Blocks 		- P2 -
Reasoning about Performance
29 Sep Performance Concepts (ppt)
(pdf)
Speedup, parallelism overhead, locality 		Chapter 3, pp. 61-85 - -
01 Oct Data Locality I
Concept of data reuse, registers and caches 		- - -
06 Oct Data Locality II (ppt) (pdf)
Code restructuring techniques 		- - -
08 Oct Midterm Quiz I
- 		- - -
20 Oct Class cancelled
- 		- - -
22 Oct Performance analysis tools (intro)
Example: VTUNE 		- - -
27 Oct Discuss exam, performance analysis tools and locality, cont. (ppt)
(pdf)
Analyzing reuse 		- HW03 -
Message Passing and Distributed Memory
29 Oct Introduction to Message Passing (ppt)
(pdf)
What is MPI? Complexities of a distributed address space 		Chapter 7, pgs. 202-228 - -
03 Nov MPI Communication Operations (ppt)
(pdf)
Blocking, Buffered, Non-blocking, One-sided 		Chapter 2, p. 54-55 - -
Parallel Programming for GPUs
05 Nov Introduction to GPUs and CUDA (ppt)
(pdf)
Architecture and execution model 		- HW4 -
10 Nov Programming in CUDA (ppt)
(pdf)
Managing data and SIMT execution 		- - -
Parallel Algorithms
12 Nov Key Concepts in Optimizing Regular and Irregular Algorithms (case studies)
Locality, Parallelism, Communication, Load Imbalance 		- - -
Current and Future Directions
17 Nov Class cancelled - - -
19 Nov Guest Lecture, Ganesh Gopalakrishnan - - -
24 Nov Global Address Space and Global View Languages (ppt) (pdf) Chapel Chapter 8, 9 - -
01 Dec Future Directions for Parallel Computing (ppt)
(pdf)
Chapel and MapReduce 		Chapter 10 Midterm -
03 Dec Transactional Memory and Course Retrospective (ppt)
(pdf)
Parallel Computing Research, intersection of HPC and commodity 		Chapter 11 - -
Project Presentations
08 Dec Project Presentations, Dry Run - - -
10 Dec Project Presentations Poster Session - - -

 Assignments

All written homeworks are due a week from when they are assigned. See the schedule above for due dates. Use handin , and submit PDF files. Programming assignments P1, P2 and P3 will also be due a week from when they are assigned. Programming assignments P4 and P5 will be more in depth, so will have more time. P5 will be a group project for teams of 2 to 3 students.

Written Homework

Programming Projects The final project will be a group project with teams of 2 to 3 students. You will combine two of the parallel programming models we learned in class and develop a non-trivial application. I will suggest some assignments, but will also allow you to select your own. There are 3 parts to this assignment:

 Policies

Cheating: Any assignment or exam that is handed in must be your own work. It must be in your own words, and based on your own understanding of the solution. However, talking with one another to understand the material better is strongly encouraged. When taking an exam, you must work independently. Any collaboration during an exam will be considered cheating. Any student who is caught cheating will be given an E in the course and referred to the University Student Behavior Committee.

ADA: The University of Utah conforms to all standards of the Americans with Disabilities Act (ADA). If you wish to qualify for exemptions under this act, notify the Center for Disabled Students Services, 160 Union.

College guidelines: Document concerning adding, dropping, etc. here.