Cross-platform FlashAttention-2 Triton implementation for Turing+ GPUs with custom configuration mode

FlashAttention for sliding window attention in Triton (fwd + bwd pass)

This repository contains multiple implementations of Flash Attention optimized with Triton kernels, showcasing progressive performance improvements through hardware-aware optimizations. The implementations range from basic block-wise processing to advanced techniques like FP8 quantization and prefetching

HRM-sMoE LLM training toolkit.

This repo represents my Nano-GPT speedrun playground, which started coding along Let's reproduce GPT-2 (124M), then moved into further improvements.

A 66M parameter decoder-only transformer language model implemented from scratch in PyTorch. Features a custom SentencePiece tokenizer, RoPE positional embeddings, SwiGLU feed-forward network, per-layer KV cache for efficient autoregressive inference, and a Svelte-based streaming chat interface.

ASR Pipeline (GLM-ASR) optimized using custom Triton kernels (achieving a 72.2% improvement in speed)

Research harness for evaluating query-time bounded elimination of reconstructable KV-cache witnesses in long-context transformer inference workloads. Related provisional filing: IN 202641062451.

CUDA kernels for LLM inference: FlashAttention forward, Tensor Core GEMM, and PyTorch bindings

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

Experimental GPT-2 scale (~124M param) LLM trained from scratch. Trained on 22B tokens od Cosmopedia Dataset. Includes full training pipeline, with SFT FineTuning and log analysis tools with backend and frontend and deployment

FlashAttention2 Analysis in Triton

An minimal CUDA implementation of FlashAttention v1 and v2

Pytorch implementation of the paper FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

A minimal, educational implementation of Ring Attention logic using custom OpenAI Triton kernels. Supports blockwise computation and online softmax merging.

A high-performance kernel implementation of multi-head attention using Triton. Focused on minimizing memory overhead and maximizing throughput for large-scale transformer layers. Includes clean-tensor layouts, head-grouping optimisations, and ready-to-benchmark code you can plug into custom models.

CUDA C++ FlashAttention reference implementation - O(N) memory, FP32/FP16, forward/backward