Syllabus

This course aims to teach you how to write fast code for parallel computers and specialized hardware accelerators. It is a spiritual follow-on to Software Performance Engineering (SPE), where we graduate from mainstream CPUs to things like GPUs. SPE or similar low-level programming, performance analysis, computer architecture, and performance optimization experience is expected as a prerequisite.

Key things we expect you to learn include:

• Why do parallel accelerators like GPUs look the way they do? We will both motivate and explain key architectural issues like SIMD, multithreading, latency hiding, memory systems, matrix units, and spatial architectures (FPGAs/CGRAs), and see how similar ideas manifest in everything from multicore CPUs, to GPUs, TPUs, IPUs, and DSPs.

• How can we program them for high performance? The course emphasizes developing practical experience writing & optimizing code on modern accelerators (mostly CUDA on NVIDIA GPUs), and how the major architectural ideas manifest in how we program them.

• How to reason about accelerator performance from first principles.

Topics:

• Parallel Programming and Parallel Algorithms

• GPU Architecture and CUDA Programming 🔥🔥🔥

• Hardware Accelerators and Hardware-aware Optimization

• Domain-Specific Programming Systems

• FPGA Acceleration & Storage

• Machine Learning Systems 🔥🔥🔥

• Message Passing Interface (MPI) and Multi-GPU

Grading

The overall course grade comprises:

• 80% lab assignments (8 * 10%)
• 15% paper reading (5 * 3%)
• 5% participation (primarily live lab attendance)

Late submissions

• You have 6 free “late days” to use on lab submissions over the course of the term.
• Beyond those 6, there is a 25% penalty for each additional day submissions are late.
• Late days may be used for both checkpoints and final submissions.
• We will automatically compute the application of late days to your submissions over the semester which maximizes your lab grade.
• Late days may not be used for the paper presentations, as there is no slack to allow for this.

We will generally also allow a short grace period after the deadline to accommodate minor technical hiccups, at our discretion. You don’t need to ask for accommodation if a bug delayed your submission by 5 minutes.

Late days are intended to cover everyday disruptions like a mild illness, interview travel, or a confluence of deadlines with other classes. Additional extensions over and above this generous baseline will only be considered in exceptional circumstances with S^3 support.

Attendance

You may miss up to 2 live labs, no questions asked. As with late days for lab submissions, this is intended to cover common minor disruptions, i and beyond this, excuses for exceptional circumstances will be considered with S^3 support.

Collaboration Policy

You are welcome and encouraged to discuss and learn from each other, but what you ultimately submit should represent your own individual final implementation.