🎉 Modern CUDA Learn Notes with PyTorch: CUDA Cores, Tensor Cores, fp32/tf32, fp16/bf16, fp8/int8, flash_attn, rope, sgemm, hgemm, sgemv, warp/block reduce, elementwise, softmax, layernorm, rmsnorm.

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.

Apply a function to each element in an array and assign the result to an element in an output array.

Standard library strided math functions.

Strided array math operations.

Apply a function to elements in two input arrays and assign the results to an output array.

Base strided.

Compute the absolute value.

Standard library strided array special math functions.

Standard library special math functions.

Apply a function to each element in an array and assign the result to an element in an output array, iterating from right to left.