VindexLLM v0.1.0

[jarroddavis68]

VindexLLM — Liberating LLM inference

GPU-accelerated LLM inference written entirely in Delphi. Vulkan compute shaders handle the full transformer forward pass — no Python, no CUDA toolkit, no external runtimes. The only dependency is vulkan-1.dll, which ships with every modern GPU driver.

Highlights

• Full dense transformer inference — Gemma 3 4B forward pass across 34 layers, entirely on GPU with zero CPU fallbacks
• Vulkan compute — 37 GLSL 450 shaders compiled to SPIR-V and embedded as Windows resources. Runs on any Vulkan 1.0+ GPU (NVIDIA, AMD, Intel)
• QAT Q4_0 primary format — Quantization-Aware Training preserves quality at 4-bit precision, fitting the full stack on consumer GPUs with 4–6 GB VRAM
• TurboQuant TQ3 KV cache compression — custom 3-bit quantization using Walsh-Hadamard Transform with Lloyd-Max centroids, ~9× compression vs F32 with fused attention scoring
• Interactive multi-turn chat — TVdxConsoleChat with streaming output, ESC cancel, slash commands, and persistent conversation history
• SQLite-backed memory with RAG — SHA-256 dedup, semantic dedup via cosine similarity, and per-turn vector retrieval using EmbeddingGemma 300M
• Pure Delphi BPE tokenizer loaded directly from GGUF vocabulary — no external SentencePiece library
• Token sampling — temperature, top-K, top-P, min-P, repetition penalty, deterministic seeding (xoshiro256** PRNG)
• Memory-mapped model loading — GGUF files mapped via CreateFileMapping / MapViewOfFile, weights uploaded to VRAM through staging buffers
• Batched prefill — all prompt tokens processed in parallel via matmul shaders; autoregressive generation via matvec shaders
• Model abstraction layer with architecture registry for clean multi-architecture support

Vetted models:

Model	Purpose	Format	Download
gemma-3-4b-it-qat-q4_0	Inference	Q4_0	Download
embeddinggemma-300M-qat-Q4_0	Embeddings	Q4_0	Download

All vetted models are collected in the tinyBigGAMES Hugging Face collection.

Quick start:

LChat := TVdxConsoleChat.Create();
try
  LChat.ModelPath := 'gemma-3-4b-it-q4_0.gguf';
  LChat.EmbedderPath := 'embeddinggemma-300M-qat-Q4_0.gguf';
  LChat.MemoryDbPath := 'session.db';
  LChat.SystemPrompt := 'You are a helpful assistant.';
  LChat.Run();
finally
  LChat.Free();
end;

Performance (NVIDIA RTX 3060 12GB, Gemma 3 4B Q8_0):

Metric	Value
Prefill throughput	35.0 tok/s
Generation throughput	24.4 tok/s
Time to first token	457 ms

Requirements: Windows 10/11 x64. Any Vulkan 1.0+ GPU. 4 GB VRAM minimum (Q4_0). 16 GB RAM. Building from source requires Delphi 12+.

---

This discussion was created from the release VindexLLM v0.1.0.