[TM10027100] - High Performance Computing

This course reviews high-performance computing systems from both aspects of hardware and software. The course talks about the importance of parallel processing, parallel system architectures, parallel algorithm design, parallel programming, and performance evaluation methodologies. Class materials are uploaded to Google Classroom, and students will do programming practice with their own PCs.

Class (hpc_01) #1

Notes about the first class: README.md

1. Class Schedule
2. Introduction to HPC
3. Supercomputers and Parallel Computing
4. The Modern Scientific Method
5. Target Applications
6. Floating-Point Operations (FLOPS)
7. Parallel Computing
8. Parallel Architectures
9. Power Calculation Example
10. System Overview
11. Software Overview
12. Benefits of Parallel Computing
13. Importance of Application Parallelism
14. Seeking Concurrency
15. Parallelism in Graphs
16. Parallelization Ratio and Amdahl’s Law
17. Parallel Programming
18. Accelerators and Heterogeneous Computing
19. System Performance Metrics
20. Heterogeneous Computing
21. Latency-Oriented Features
22. Cache Memory
23. Branch Prediction
24. Strategies for High Throughput
25. Conclusion

Class (hpc_02) #2

Notes about the second class: README.md

1. Today's Topics: Parallel Computers
2. Dependencies in Parallel Computing
3. Parallel Loops
4. Loop Dependence
5. Compiler Optimization
6. SIMD Overview
7. Vector Processors
8. Hardware Trends for Vectorization
9. Code optimization for NEC SX-AT
10. Shared-memory Computer
11. UMA Architecture
12. Shared Data
13. What is a Cache Memory?
14. The principle of locality
15. Message Collision
16. NUMA Architecture
17. Switch Networks
18. Cache coherence in NUMA
19. Distributed-Memory Computers
20. Hybrid Systems
21. Performance Metrics
22. Network Topology
23. Summary

Class (hpc_03)

Notes about the second class: README.md

1. SIMD Overview
2. Sharing a Memory Space
3. Basic Network Topology
4. Basic Network Topology (continued)
5. Parallel Algorithm Design
6. Today's Topics
7. How to Run a Program on HPC?
8. Why Job Submission is Needed
9. Job Script File
10. Job Submission
11. Job Scheduling
12. Workflows
13. Technical Challenges
14. Various Workloads
15. Task/Channel Model (Ian Foster, 1995)
16. Synchronous vs Asynchronous Communication
17. Execution Time
18. Foster’s Design Methodology
19. Finding the Sum
20. Parallel Reduction Design
21. n-Body Problem
22. Other Communication Patterns
23. Summary

Class (hpc_04)

1. Parallel Computers and Systems
2. Software and Resources
3. Parallel Algorithm Design
4. Running Programs on HPC
5. Introduction to MPI Programming
6. MPI Programming Basics
7. MPI Functions and Communication Patterns
8. Collective Communication
9. Deadlocks and Synchronization
10. Performance Measurement
11. Exercise - Monte Carlo π Calculation
12. Summary of MPI Programming Concepts
13. Sample Codes
    • Minimal MPI Program
    • Simple Parallel Reduction Code

Class (hpc_05)

1. Introduction to High-Performance Computing (HPC)
2. Parallel Programming Fundamentals
3. Core Concepts in Parallel Processing with OpenMP
4. Data Management in OpenMP
5. Task Parallelism
6. Heterogeneous Computing and Offloading
7. Performance and Scalability Considerations
8. Optimization Techniques

Class (hpc_06)

1. Overview of Parallel Computing
2. Parallel Algorithm Design
3. Parallel Programming Interfaces

• MPI (Message Passing Interface)
• OpenMP

4. Hybrid Programming with MPI and OpenMP
5. Case Study: Conjugate Gradient Method
6. Performance Profiling
7. Performance Prediction and Analysis

• Roofline Model
• Amdahl’s Law
• Gustafson-Barsis’s Law
• Karp-Flatt Metric

8. Key Metrics for Parallel Systems
9. Challenges in Achieving Ideal Speedup
10. Practical Exercise
11. Summary