Introduction to Parallel Programming for GPUs with CUDA

The goals for this CUDA tutorial are to gain a comprehensive understanding of the CUDA programming model, including SIMT threading, kernel structure, and GPU memory hierarchy, as well as practical knowledge of memory optimization techniques like global memory coalescing and shared memory bank conflicts.

Topics

1. Chapter 1: CPU vs GPU Architecture and Performance

   • 1. Introduction
   • 2. Efficiency of GPUs over CPUs
   • 3. Structural Differences: CPU vs GPU
   • 4. Streaming Multiprocessors (SMs) in GPUs
   • 5. Thread Hierarchy Organization in GPUs
   • 6. Two-Level Parallelism Latency hiding in GPUs
   • 7. Synchronization in GPU Threads
   • 8. Work Division: CPUs vs GPUs
   • 9. SIMT vs SIMD
   • 10. Limitations of GPUs

2. Chapter 2: CUDA Programming In Practice

   • 1. Introduction
   • 2. Host-Device Heterogeneous Programming Model
   • 3. Increment Array Example
   • 4. Necessary Steps for Host-Device Heterogeneous Code
   • 5. CUDA Syntax
   • 6. Matrix Multiplication Example

3. Chapter 3: GPU Memory Hierarchy and Optimization

   • 1. Introduction
   • 2. Common Challenges in GPU Programming
   • 3. GPU Memory Hierarchy
   • 4. Global Memory Coalescing
   • 5. Shared Memory Programming

References:

• Nvidia Cuda Programming Guide
• CS 4230/6230 (Parallel and High-Performance Computing) Lectures by Professor P. Sadayappan, The University of Utah

---

By Omid Asudeh