tamingllms/notebooks/local.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(local)=\n",
    "# Local LLMs in Practice\n",
    "```{epigraph}\n",
    "Freedom is something that dies unless it's used.\n",
    "\n",
    "-- Hunter S. Thompson\n",
    "```\n",
    "```{contents}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Running Open Source LLMs locally versus depending on proprietary cloud-based models represents more than just a technical choice - it's a fundamental re-imagining of how we interact with AI technology, putting control back in the hands of users.\n",
    "\n",
    "Privacy concerns are a key driver for running LLMs locally. Individual users may want to process personal documents, photos, emails, and chat messages without sharing sensitive data with third parties. For enterprise use cases, organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.\n",
    "\n",
    "Cost considerations are another key driver. Organizations and individual consumers can better control expenses by matching model capabilities to their specific needs rather than paying for multiple cloud API subscriptions. For organizations with high-volume applications, this customization and control over costs becomes especially valuable compared to the often prohibitive per-request pricing of cloud solutions. For consumers, running multiple open source models locally eliminates the need to maintain separate subscriptions to access different model capabilities.\n",
    "\n",
    "Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical. Further, the emergence of Small Language Models (SLMs) has made edge deployment increasingly viable, enabling sophisticated language capabilities on resource-constrained devices like smartphones, tablets and IoT sensors. \n",
    "\n",
    "Running open source models locally also enables fine-grained optimization of resource usage and model characteristics based on target use case. Organizations and researchers can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation. \n",
    " \n",
    "However, local deployment introduces its own set of challenges and considerations. In this Chapter, we explore the landscape of local LLM deployment focused on Open Source models and tools. When choosing a local open source model, organizations must carefully evaluate several interconnected factors, from task suitability and performance requirements to resource constraints and licensing.\n",
    " \n",
    "We also cover key tools enabling local model serving and inference, including open source solutions such as LLama.cpp, Llamafile, and Ollama, along with user-friendly frontend interfaces that make local LLM usage more accessible. We conclude with a detailed case study, analyzing how different quantization approaches impact model performance in resource-constrained environments. This analysis reveals the critical tradeoffs between model size, inference speed, and output quality that practitioners must navigate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(local-model-selection)=\n",
    "## Choosing your Model\n",
    "\n",
    "The landscape of open source LLMs is rapidly evolving, with new models emerging by the day. While proprietary LLMs have garnered significant attention, open source LLMs are gaining traction due to their flexibility, customization options, and cost-effectiveness. \n",
    "\n",
    "It is important to observe long-term strategic considerations when choosing a model. These entails prioritization dimensions that may enable competitive advantage in the long-term, including:\n",
    "\n",
    "1. **Managed Services Support**: You may start experimenting locally with LLMs but eventually you will need to deployment options: either host models yourself or consider managed services. Cloud providers like AWS Bedrock, SambaNova and Together.ai can simplify deployment and management but model family support varies along with varying SLAs for model availability, support and model serving {cite}`artificialanalysis2024llmproviders`. One should evaluate the availability of managed services for your target model family. \n",
    "\n",
    "2. **Vendor Long-Term Viability**: Consider vendor's long-term strategy and transparency around future development. Evaluate factors like funding, market position, and development velocity to assess whether the vendor will remain a reliable partner. Further, transparency around long-term strategy and roadmap is a critical consideration when choosing a model vendor partner.\n",
    "\n",
    "3. **Single-Provider Lock-in**: Users and organizations should avoid the risk of lock-in by remaining flexible with your choice of LLM providers. Today's winning models are not guaranteed to be the same in the future.\n",
    "\n",
    "4. **Time-to-market and Customization**: As the same models are available to everyone, base capabilities are becoming commoditized. As a consequence, competitive advantage comes from the application layer. Hence, the ability to iterate fast while customizing to your specific domain becomes a critical strategic consideration when choosing a model.\n",
    "\n",
    "5. **Data Competitive Edge**: As the cost of (pre-trained) general intelligence decays rapidly, proprietary data becomes competitive advantage. Hence, the ability to add unique, custom, domain-specific datasets to base models is a critical consideration that will separate winners from losers.\n",
    "\n",
    "\n",
    "In this section, we aim to provide a comprehensive set of considerations to selecting the right open-source LLM for your specific needs, emphasizing the importance of aligning the LLM's capabilities with the intended task and considering resources constraints."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task Suitability\n",
    "\n",
    "When evaluating an open source LLM, task suitability is a critical first consideration. A model that performs well on general benchmarks may struggle with specific domain tasks. Understanding the intended use case helps narrow down model options based on their demonstrated strengths.\n",
    "\n",
    "**Task Categories**\n",
    "\n",
    "When determining which LLM task to prioritize, carefully consider your specific use case and end-user needs. Different applications require distinct model capabilities and optimizations. Common LLM Task Categories include:\n",
    "- **Text Summarization**: Condensing documents into concise summaries that capture key information.\n",
    "- **Question Answering**: Providing accurate responses by extracting relevant information from knowledge bases.\n",
    "- **Text Generation**: Creating high-quality content across formats, from documentation to creative writing.\n",
    "- **Code Generation**: Writing clean, documented code in multiple programming languages.\n",
    "- **Language Translation**: Converting text between languages while preserving meaning and nuance.\n",
    "- **Dialogue Systems**: Enabling natural conversations for customer support and interactive learning.\n",
    "- **Text Classification**: Categorizing and labeling text data for sentiment analysis, topic modeling, and content moderation.\n",
    "- **Named Entity Recognition**: Identifying and extracting specific entities from text, such as people, organizations, and locations.\n",
    "\n",
    "{numref}`task_number` shows the number models per task category available at Hugging Face as of December 22, 2024 {cite}`hf2024yearinreview`. Text generation is by far the most popular task category.\n",
    "\n",
    "```{figure} ../_static/local/task_number.png\n",
    "---\n",
    "name: task_number\n",
    "alt: Task Number\n",
    "scale: 40%\n",
    "align: center\n",
    "---\n",
    "Number of models per task category from Hugging Face as of December 22, 2024 {cite}`hf2024yearinreview`.\n",
    "```\n",
    "\n",
    "**Model Types**\n",
    " \n",
    "Open source LLMs can be broadly categorized into three main types as far as they level of customization is concerned, each with distinct characteristics and use cases (see {numref}`model_types`):\n",
    "\n",
    "- **Base Models**: These foundation models provide broad language understanding capabilities but typically require additional fine-tuning to excel at specific tasks. They serve as versatile starting points for customization. Examples: meta-llama/Llama-2-70b, Qwen/Qwen2.5-72B\n",
    "\n",
    "- **Instruction-Tuned Models**: Enhanced through fine-tuning on instruction-following datasets, these models excel at interpreting and executing explicit prompts and commands. They bridge the gap between general language capabilities and practical task execution. Chat models are a good example of this category. Examples: meta-llama/Llama-2-70b-chat-hf (Chat), Qwen/Qwen2.5-72B-Instruct\n",
    "\n",
    "- **Domain-Adapted Models**: Specialized for particular fields through targeted fine-tuning and/or preference-alignment on domain-specific data. Examples: Med-PaLM 2 for healthcare, BloombergGPT for finance.\n",
    "\n",
    "```{figure} ../_static/local/model_types.svg\n",
    "---\n",
    "name: model_types\n",
    "alt: Model Types\n",
    "scale: 60%\n",
    "align: center\n",
    "---\n",
    "Model Types.\n",
    "```\n",
    "\n",
    "\n",
    "The Llama 2 model family {cite}`touvron2023llama2openfoundation` illustrates these distinctions well. The base Llama 2, trained on 2 trillion tokens of public data, demonstrates general-purpose capabilities across text generation and translation tasks. Its chat-optimized instruction-tuned variant, Llama 2-Chat, underwent additional fine-tuning on over 1 million human-annotated conversational examples, making it particularly adept at natural dialogue.\n",
    "\n",
    "Benchmark results {cite}`meta2024llama2chat70b` in {numref}`llama2_benchmark` highlight the impact of model specialization. On the TruthfulQA {cite}`2021truthfulqa` and Toxigen {cite}`alnajjar2024toxigen` benchmarks measuring truthful and informative responses. We observe that the chat-optimized variants show substantially improved truthfulness. Similarly, on the ToxiGen benchmark measuring toxic content generation, Llama 2-Chat models demonstrate near-zero toxicity compared to base models' 21-26% rates.\n",
    "\n",
    "```{table} Benchmark results for Llama 2 family of models.\n",
    ":name: llama2_benchmark\n",
    ":align: center\n",
    "| Model | Size | TruthfulQA | Toxigen |\n",
    "|-------|------|------------|----------|\n",
    "| Llama 2 | 7B | 33.29 | 21.25 |\n",
    "| Llama 2 | 13B | 41.86 | 26.10 |\n",
    "| Llama 2 | 70B | 50.18 | 24.60 |\n",
    "| Llama-2-Chat | 7B | 57.04 | 0.00 |\n",
    "| Llama-2-Chat | 13B | 62.18 | 0.00 |\n",
    "| Llama-2-Chat | 70B | 64.14 | 0.01 |\n",
    "```\n",
    "\n",
    "While Llama family of models exhibits strong performance across general knowledge, instruction following, and specialized domains, purpose-built models may still outperform it in highly specific applications. Qwen/Qwen2.5-Coder-32B-Instruct {cite}`hui2024qwen2` is an example of a purpose-built model that demonstrates significant performance on the specific task of code generation.\n",
    "\n",
    "**Model Features**\n",
    "\n",
    "Model features can either enable or limit the feasibility of specific use cases. Understanding features of your candidate models is crucial for determining whether a model is suitable for your application. For example:\n",
    "\n",
    "- **Context Length**: The model's ability to process longer text sequences directly impacts task suitability. A legal contract analysis systems requiring the model to reason about a 5000-page document would be impractical with a model limited to 2,048 tokens, while models supporting 2M tokens could handle this task effectively without the need for other techniques e.g. context chunking.\n",
    "\n",
    "- **Output Control**: Some tasks require precise, factual and structured outputs while others allow more creative, unstructured generation. Models vary in their output reliability. Grammar constraints and other control mechanisms may be needed to ensure reliable outputs. See Chapter {ref}`structure` for more details.\n",
    "\n",
    "- **Caching**: Models that support caching can speed up inference at lower costs. This becomes particularly important for applications requiring cost-effective real-time responses.\n",
    "\n",
    "- **Multi-modal Capabilities**: Some applications fundamentally require multi-modal processing. A medical diagnosis assistant analyzing both patient records and X-ray images would be impossible to implement with a text-only model, necessitating a multi-modal model that can process both text and images coherently.\n",
    "\n",
    "- **Output Token Length**: The model's capacity to generate longer responses affects its suitability for content generation tasks. A model excelling at concise responses may struggle with long-form content creation like technical documentation or detailed analysis reports.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance & Cost\n",
    "\n",
    " General benchmarks are useful for comparing models across different standard tasks. Open Source models are becoming more competitive with proprietary models with LLama, Qwen, DeepSeek and Mistral model families being some of the most powerful open source models available today.\n",
    "\n",
    "Qwen model family {cite}`qwen2024qwen25technicalreport` emerged in 2024 as a model family achieving competitive performance with relatively smaller parameter counts compared to its competitors. The flagship Qwen2.5-72B-Instruct model demonstrates performance comparable to the much larger Llama-3-405B-Instruct while being about 5 times smaller. The models excel in specialized tasks like mathematics and coding, handle structured data effectively, and offer enhanced support for tool use and long-text generation as shown in {numref}`qwen_perf`.\n",
    "\n",
    "```{figure} ../_static/local/qwen_perf.png\n",
    "---\n",
    "name: qwen_perf\n",
    "alt: Qwen Performance\n",
    "scale: 40%\n",
    "align: center\n",
    "---\n",
    "Qwen Performance.\n",
    "```\n",
    "\n",
    "{numref}`perf_` shows a comparison including reference proprietary models such as GPT-40, Gemini 1.5 Pro and Claude 3.5 Sonnet. Leading models vary per domain but all top ranking models are proprietary. However, open source models do show competitive performance with Qwen and LLama models leading the pack, overall.\n",
    "\n",
    "```{figure} ../_static/local/perf_.png\n",
    "---\n",
    "name: perf_\n",
    "alt: Performance Comparison including proprietary models.\n",
    "scale: 40%\n",
    "align: center\n",
    "---\n",
    "Performance Comparison including proprietary models.\n",
    "```\n",
    "\n",
    "Also from China, DeepSeek-V3 {cite}`deepseek2024v3` represents a major breakthrough in open source language models, emerging as arguably the most capable open source large language model available as of the end of 2024. With 671 billion parameters and 37 billion active MoE (Mixture of Experts) parameters, it achieves performance on par with leading proprietary models like Claude 3.5 Sonnet and GPT 4o as shown in {numref}`deep`. The model demonstrates impressive cost efficiency metrics (see {numref}`deep2`), processing input tokens at $0.27 per million and output tokens at $1.1 per million, while maintaining a generation speed of 60 tokens per second (3x faster than DeepSeek-V2).\n",
    "\n",
    "What makes DeepSeek-V3 particularly remarkable is that these capabilities were achieved with a relatively modest training budget of just $5.5 million, used to train on 14.8 trillion tokens. This efficiency in training demonstrates the potential for open source models to compete with proprietary alternatives at a fraction of the cost. The model's release marks a significant milestone in the democratization of advanced AI capabilities, challenging the dominance of proprietary models within big tech. One should be cautious though as the model has not yet been battle-tested in the wild but this is an exciting development demonstrating the potential of open source models to compete with proprietary alternatives.\n",
    "\n",
    "```{figure} ../_static/local/deep.jpeg\n",
    "---\n",
    "name: deep\n",
    "alt: DeepSeek-V3\n",
    "scale: 65%\n",
    "align: center\n",
    "---\n",
    "DeepSeek-V3 Performance Comparison\n",
    "```\n",
    "\n",
    "```{figure} ../_static/local/deep2.jpeg\n",
    "---\n",
    "name: deep2\n",
    "alt: DeepSeek-V3 Cost Benefit Analysis\n",
    "scale: 65%\n",
    "align: center\n",
    "---\n",
    "DeepSeek-V3 Cost Benefit Analysis\n",
    "```\n",
    "\n",
    "While standard benchmarks provide valuable initial insights, they should be interpreted cautiously since models can be specifically optimized for these popular tests without necessarily performing well in target use cases. This necessitates developing custom evaluation frameworks with real-world validation - creating test datasets representing actual usage scenarios, defining metrics aligned with business objectives, and establishing clear baselines and improvement targets. Only through such rigorous testing can practitioners truly understand how well a model will perform in their specific context.\n",
    "\n",
    "In that way, after identifying candidate models, it's essential to rigorously evaluate their capabilities against unique use case requirements and constraints, as models that excel in standardized tests may struggle with the nuanced demands of real-world applications. Chapter {ref}`evals` explores this critical challenge in detail, providing frameworks and best practices for comprehensive model evaluation.\n",
    "\n",
    "Model quality performance should not be evaluated in isolation. It is important to also consider the cost of running the model once it's deployed as well as its computational performance. This depends on the model size, hardware, and the platform used (self-hosted vs. managed services). Key metrics include:\n",
    "\n",
    "- **Cost-Related**:\n",
    "  - **Cost Per Output Token (CPOT)**: This metric measures the cost of text generation.\n",
    "  - **Cost Per Input Token (CPIT)**: This metric measures the cost for input prompt processing.\n",
    "  - **Total Cost of Ownership (TCO)**: Consider the full lifecycle cost, including development, deployment, maintenance, infrastructure, and ongoing iteration.\n",
    "- **Time-Related**:\n",
    "  - **Time Per Output Token (TPOT)**: This metric measures the speed of text generation and is crucial for user experience, especially in interactive applications.\n",
    "  - **Time to First Token (TTFT)**: Essential for streaming applications like chatbots, as it measures how quickly the model begins generating a response.\n",
    "  - **Latency**: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.\n",
    "\n",
    "{numref}`p2` shows a comparison of quality now with the added dimension of cost. Quality is measured as an average of scores from MMLU, GPQA, Math & HumanEval benchmarks {cite}`artificialanalysis2024methodology`. Price is a blend of Cost Per Input Token plus Input & Cost Per Output Token (3:1 ratio). Reported numbers represent median across cloud providers {cite}`artificialanalysis2024providers` supporting these models. \n",
    "\n",
    "```{figure} ../_static/local/p2.png\n",
    "---\n",
    "name: p2\n",
    "alt: Performance Comparison including proprietary models.\n",
    "scale: 40%\n",
    "align: center\n",
    "---\n",
    "Performance Comparison including proprietary models.\n",
    "```\n",
    "\n",
    "We observe Qwen2.5 72B and Llama 3.3 70B offer the best value among Open Source models, providing high quality at a relatively affordable price comparable to GPT-4o mini, for instance. Meanwhile Nova Lite, Nova Micro, and Llama 3.1 8B demonstrate to be budget-friendly options catering to use cases where cost is a significant factor and some compromise on quality is acceptable. \n",
    "\n",
    "From {numref}`p1` we have evidence that output prices are higher than input prices. This reflects the greater computational resources typically required at inference time for output compared to processing input text (e.g. tokenization, encoding). We also observe a quite significant variation in pricing across different models. Prices range from a few cents per 1M tokens (e.g., Gemini 2.0 Flash, Nova Micro, Nova Lite) to several dollars per 1M tokens (e.g., Claude 3.5 Sonnet, GPT-4o). Mistral large 2 is the most expensive model at $2/$6 per 1M input/output tokens while Nova Micro family is the cheapest among Open Source options.\n",
    "\n",
    "\n",
    "\n",
    "```{figure} ../_static/local/p1.png\n",
    "---\n",
    "name: p1\n",
    "alt: Input and Output Prices\n",
    "scale: 40%\n",
    "align: center\n",
    "---\n",
    "Input and Output Prices Comparison.\n",
    "```\n",
    "\n",
    "Latency figures in {numref}`latency` put GPT-4o (Nov '24) as the best performing model but Llama, Nova Micro, Phi and Mistral model families all have options with latency of half a second or better beating Gemini and Claude models considered as well as GPT-4o mini.\n",
    "\n",
    "```{figure} ../_static/local/latency.png\n",
    "---\n",
    "name: latency\n",
    "alt: Latency Comparison\n",
    "scale: 40%\n",
    "align: center\n",
    "---\n",
    "Latency Comparison.\n",
    "```\n",
    "\n",
    "This analysis provides a framework for evaluating key performance considerations when selecting an LLM. While the specific figures for cost, latency, and quality change frequently (often daily) as providers update their offerings and pricing, the fundamental tradeoffs remain relevant. When evaluating model suitability for a specific use case, practitioners should carefully consider:\n",
    "\n",
    "- The balance between quality requirements and cost constraints\n",
    "- Latency requirements for the intended application\n",
    "- Total cost of ownership including both input and output token costs\n",
    "- Whether streaming capabilities are needed (TTFT becomes more critical)\n",
    "- Infrastructure and deployment costs\n",
    "\n",
    "Regular re-evaluation of these metrics is recommended as the landscape evolves rapidly. What represents the optimal choice today may change as new models are released and existing ones are updated.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Licensing\n",
    "\n",
    "When evaluating open-source LLMs, it's important to consider licensing and data usage policies. Some models may require attribution or commercial use licenses, while others may be more permissive. Additionally, ensure that the model's training data is compatible with your intended use case and complies with relevant data protection laws.\n",
    "\n",
    "The licensing landscape for LLMs spans from highly permissive to custom and restricted usage. {numref}`open_source_llms` provides a summary of the licensing terms for some of the most popular open source LLMs. We observe two types of licenses:\n",
    "- **Traditional Open Source**:\n",
    "  - Apache 2.0 (exemplified by Mistral AI's models) offers comprehensive commercial usage rights with minimal restrictions\n",
    "  - MIT License (used by Microsoft's Phi-3) provides similar freedoms with simpler terms\n",
    "\n",
    "- **Custom Commercial Licenses**:\n",
    "  - Meta's LLaMA 3 allows free usage for applications serving under 700 million users\n",
    "  - Alibaba's Qwen2.5 permits free deployment for services with fewer than 100 million users\n",
    "  - Both restrict using model outputs to train competing LLMs\n",
    "\n",
    "```{table} Open Source LLMs.\n",
    ":name: open_source_llms\n",
    ":align: center\n",
    "| Creator | LLM | License |\n",
    "|---------|-----|---------|\n",
    "| Meta AI | LLaMA 3 | Custom - Free if under 700M users, cannot use outputs to train other non-LLaMA LLMs |\n",
    "| Microsoft | Phi-3 | MIT |\n",
    "| Mistral AI | Mistral | Apache 2.0 |\n",
    "| Alibaba | Qwen2.5 | Custom - Free if under 100M users, cannot use outputs to train other non-Qwen LLMs |\n",
    "| Google | Gemma | Custom - Free with usage restrictions, models trained on outputs become Gemma derivatives |\n",
    "| DeepSeek | DeepSeek-V2 | Custom - Free with usage restrictions, models trained on outputs become DeepSeek derivatives |\n",
    "```\n",
    "\n",
    "When selecting an open-source LLM for deployment, practitioners must carefully evaluate licensing terms that align with intended usage (whether commercial, research, or other). While permissive licenses like Apache 2.0 and MIT allow broad usage rights, custom licenses may impose specific restrictions on commercial applications or model derivatives, making thorough license review essential for sustainable implementation.\n",
    "\n",
    "The training data sources for LLMs represent another critical consideration. Models vary significantly in their training data foundations - some leverage purely public datasets while others incorporate proprietary or restricted content with the added complexity that public data does not mean free data. These data choices fundamentally impact not only model capabilities but also legal and regulatory compliance.\n",
    "\n",
    "The legal landscape surrounding LLM training data has grown increasingly complex, particularly regarding copyright infringement concerns. The high-profile lawsuit between OpenAI and The New York Times {cite}`harvardlawreview2024nyt` serves as a pivotal example, where the Times claims its copyrighted materials were used without authorization to train language models. This litigation has far-reaching consequences for developers building LLM-powered applications. Should courts rule in favor of copyright holders, model providers may need to withdraw and retrain models containing protected content. These legal uncertainties introduce substantial complexity into LLM implementation strategies, demanding careful consideration during project planning phases.\n",
    "\n",
    "Recent LLM releases demonstrate varying levels of data transparency. For instance, Qwen2.5's approach {cite}`qwen2024qwen25technicalreport` illustrates common industry practices in both its achievements and limitations. On the training data scale front, Qwen2.5 does provide some transparency by discussing some training data methodology compared to previous versions such as expanding from 7 trillion to 18 trillion tokens, while implementing sophisticated quality filtering and carefully balancing domain representation through sampling adjustments.\n",
    "\n",
    "However, like many commercial LLMs, Qwen2.5 exhibits transparency limitations. The report provides incomplete disclosure of data sources and limited information about the proportions of different data types used in training. The preprocessing methodologies remain unclear, and there is minimal discussion of potential biases that may exist in the training data. \n",
    "\n",
    "Similarly, in the Llama 3 paper {cite}`grattafiori2024llama3herdmodels`, Meta AI does share some details about the pre-training corpus stating simply stating that it was around 15T multilingual tokens, compared to 1.8T tokens for Llama 2. The exact sources of data used for pre-training and post-training are not explicitly listed.\n",
    "\n",
    "These gaps in transparency reflect a broader industry challenge in balancing commercial interests with the need for openness and scientific reproducibility.\n",
    "\n",
    "A significant advancement in open-source language model training data is HuggingFace's release of the FineWeb datasets. In its first release {cite}`penedo2024finewebdatasetsdecantingweb`, FineWeb is made of a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. Additionally, data curation codebase and all of the models trained during our ablation experiments are made available. FineWeb is a fine example of an initiative that helps minimize the gap between proprietary and public knowledge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Community Support\n",
    "\n",
    "Community support plays a vital role in the open-source LLM ecosystem. Active communities contribute to model development, provide technical assistance, and share valuable resources. When evaluating open-source LLMs, the strength and engagement of the community should be a key consideration, as it directly impacts the model's long-term viability and practical utility.\n",
    "\n",
    "The popularity of different model families reflects their community adoption. In 2024, the Qwen and Llama families have emerged as clear favorites, with Qwen2.5-1.5B-Instruct alone representing 35% of total open source models downloads in 2024.\n",
    "\n",
    "```{figure} ../_static/local/downloads.png\n",
    "---\n",
    "name: downloads\n",
    "alt: Hugging Face Downloads\n",
    "scale: 30%\n",
    "align: center\n",
    "---\n",
    "Hugging Face Model Downloads in 2024 as of December 22 of the same year {cite}`hf2024yearinreview`.\n",
    "```\n",
    "\n",
    "Strong communities accelerate model innovation through collective effort. When developers and researchers collaborate on model development, they create a powerful ecosystem of continuous improvement. Through transparent sharing of findings, they enable rapid development of novel applications and specialized model variants for specific domains. This collaborative environment naturally leads to the establishment of best practices and frameworks that benefit the entire community. The success of this community-driven approach is evident in models like Qwen2.5-1.5B-Instruct, which has spawned 200+ derivative models through post-training adaptations {cite}`qwen25instruct2024`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Customization\n",
    "\n",
    "Model customization is an important consideration when selecting an open-source LLM. Adapting and fine-tuning to specific use cases can significantly impact practical utility and performance in production environments.\n",
    "\n",
    "Model providers increasingly offer streamlined fine-tuning services. For example, Mistral demonstrates an accessible approach to model customization.\n",
    "The code below shows Mistral's straightforward fine-tuning API. The example shows how to create and start a fine-tuning job with just a few lines of code. The fine-tuning job is configured with the base model \"open-mistral-7b\" and uses training and validation files from the Ultrachat dataset {cite}`hf2024ultrachat200k`. This API design makes it easy to experiment with model customization while maintaining control over the training process.\n",
    "\n",
    "```python\n",
    "# create a fine-tuning job\n",
    "created_jobs = client.fine_tuning.jobs.create(\n",
    "    model=\"open-mistral-7b\", \n",
    "    training_files=[{\"file_id\": ultrachat_chunk_train.id, \"weight\": 1}],\n",
    "    validation_files=[ultrachat_chunk_eval.id], \n",
    "    hyperparameters={\n",
    "        \"training_steps\": 10,\n",
    "        \"learning_rate\":0.0001\n",
    "    },\n",
    "    auto_start=False\n",
    ")\n",
    "\n",
    "# start a fine-tuning job\n",
    "client.fine_tuning.jobs.start(job_id = created_jobs.id)\n",
    "\n",
    "created_jobs\n",
    "```\n",
    "\n",
    "For more comprehensive customization needs, Hugging Face's Transformer Reinforcement Learning (TRL) toolkit provides robust capabilities for model adaptation. Built on the Transformers library, TRL supports {cite}`huggingface2024trl`:\n",
    "\n",
    "- Supervised Fine-Tuning (SFT)\n",
    "- Reward Modeling (RM)\n",
    "- Proximal Policy Optimization (PPO)\n",
    "- Direct Preference Optimization (DPO)\n",
    "\n",
    "In {ref}`alignment-case-study`, we will explore how to use TRL to fine-tune a model to align with user preferences.\n",
    "\n",
    "Successful model customization demands managing critical resources throughout the development lifecycle. This includes rigorous dataset preparation and validation to ensure high-quality training data, careful configuration of training infrastructure to optimize computational resources, systematic experimentation iterations while managing associated costs, comprehensive performance evaluation frameworks to measure improvements, and thoughtful deployment architecture planning to ensure smooth production integration. Of course, actual cost of storage and inference should be taken into consideration. {numref}`mistral_costs` shows as an example the cost of associated with fine-tuning Mistral models {cite}`mistraltechnology2024`.\n",
    "\n",
    "\n",
    "```{table} Mistral fine-tuning costs as of December 22, 2024.\n",
    ":name: mistral_costs\n",
    ":align: center\n",
    "| Model | One-off training (/M tokens) | Storage | Input (/M tokens) | Output (/M tokens) |\n",
    "|-------|----------------------------|---------|-------------------|-------------------|\n",
    "| Mistral NeMo | $1 | $2 per month per model | $0.15 | $0.15 |\n",
    "| Mistral Large 24.11 | $9 | $4 per month per model | $2 | $6 |\n",
    "| Mistral Small | $3 | $2 per month per model | $0.2 | $0.6 |\n",
    "| Codestral | $3 | $2 per month per model | $0.2 | $0.6 |\n",
    "```\n",
    "\n",
    "\n",
    "Small language models can serve as a lightweight alternative to customization compared to large models. Recent research has shown that smaller models can achieve competitive performance compared to larger models {cite}`zhao2024loraland310finetuned, hf2024scalingtesttime`. A noteworthy example is Hugging Face's SmolLM2 {cite}`allal2024SmolLM2`, a family of compact language models designed with several key advantages:\n",
    "1. Compact Sizes:\n",
    "- Available in three sizes: 135M, 360M, and 1.7B parameters\n",
    "- Small enough to run on-device and local hardware\n",
    "- Doesn't require expensive GPU resources\n",
    "\n",
    "2. Versatility:\n",
    "- Can perform a wide range of tasks despite small size\n",
    "- Supports text summarization, rewriting, and function calling\n",
    "- Can be used for multimodal applications (via SmolVLM)\n",
    "\n",
    "3. Easy Integration and Customization:\n",
    "- Supports multiple frameworks like llama.cpp, MLX, MLC, and transformers.js\n",
    "- Can be fine-tuned using TRL and PEFT for custom applications\n",
    "- Provides pre-training and fine-tuning scripts for customization\n",
    "- Includes synthetic data pipelines for creating custom training data\n",
    "\n",
    "These models address a crucial need in the AI ecosystem by making language models more accessible and practical for developers who need local, efficient solutions without compromising too much on capability. The provided tools and scripts for customization make it particularly valuable for developers who need to adapt the model for specific use cases or domains."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools for Local LLM Deployment\n",
    "\n",
    "Local LLM deployment tools generally fall into two categories: inference-focused tools that prioritize performance and programmability for technical users requiring production-grade deployments, and user interface (UI) tools that emphasize accessibility through graphical interfaces for non-technical users, trading some performance for ease of use and broader adoption. In the following sections we will explore some of these tools discussing their features, capabilities, and trade-offs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serving Models\n",
    "\n",
    "Serving an LLM model involves making it available for inference by setting up infrastructure to process requests and manage resources efficiently. This serving layer handles several key responsibilities, from loading model weights and managing compute resources to processing requests and optimizing performance. Let's examine the core components of model serving:\n",
    "\n",
    "1. **Model Loading and Initialization**\n",
    "- Loading the trained model weights and parameters into memory\n",
    "- Initializing any required runtime configurations and optimizations\n",
    "- Setting up inference pipelines and processing workflows\n",
    "\n",
    "2. **Resource Management** \n",
    "- Allocating and managing system memory (RAM/VRAM) for model weights\n",
    "- Handling computational resources like CPU/GPU efficiently\n",
    "- Implementing caching and batching strategies where appropriate\n",
    "\n",
    "3. **Request Processing and Inference**\n",
    "- Accepting input requests through defined interfaces\n",
    "- Converting input text into token vectors $\\mathbf{x} = [x_1, x_2, ..., x_n]$ through tokenization\n",
    "- Computing probability distributions $P(x_{n+1}|x_1, x_2, ..., x_n; θ)$ for next tokens\n",
    "- Performing matrix multiplications and attention computations\n",
    "- Sampling each new token from the calculated probability distribution\n",
    "- Post-processing and returning responses\n",
    "\n",
    "4. **Performance Optimization**\n",
    "- Implementing techniques like quantization to reduce memory usage\n",
    "- Optimizing inference speed through batching and caching\n",
    "- Managing concurrent requests and load balancing\n",
    "- Monitoring system resource utilization\n",
    "\n",
    "\n",
    "The serving layer acts as the bridge between the LLM and applications while working on top of a hardware stack as shown in {numref}`local_inference`. Getting this layer right is crucial for building locally-served reliable AI-powered applications, as it directly impacts the end-user experience in terms of response times, reliability, and resource efficiency. \n",
    "\n",
    "```{figure} ../_static/local/local_inference.svg\n",
    "---\n",
    "name: local_inference\n",
    "alt: Local Inference Server\n",
    "scale: 60%\n",
    "align: center\n",
    "---\n",
    "Local Inference Server.\n",
    "```\n",
    "\n",
    "Model inference can be performed on Open Source models using cloud solutions such as Groq, Cerebras Systems, and SambaNova Systems. Here, we limit our scope to Open Source solutions that enable inference on local machines which includes consumer hardware. We will cover the following:\n",
    "\n",
    "- **LLama.cpp**: A highly optimized C++ implementation for running LLMs on consumer hardware\n",
    "- **Llamafile**: A self-contained executable format by Mozilla for easy model distribution and deployment\n",
    "- **Ollama**: A tool that simplifies running and managing local LLMs with Docker-like commands\n",
    "\n",
    "Let's explore each of these options in detail.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LLama.cpp\n",
    "\n",
    "LLama.cpp {cite}`ggerganov2024llamacpp` is an MIT-licensed open source optimized implementation of the **LLama** model architecture designed to run efficiently on machines with limited memory.\n",
    "\n",
    "Originally developed by Georgi Gerganov and today counting with hundreds of contributors, this C/C++ LLama version provides a simplified interface and advanced features that allow language models to run locally without overwhelming systems. With the ability to run in resource-constrained environments, LLama.cpp makes powerful language models more accessible and practical for a variety of applications.\n",
    "\n",
    "In its \"Manifesto\" {cite}`ggerganov2023llamacppdiscussion`, the author highlights the significant potential in bringing AI from cloud to edge devices, emphasizing the importance of keeping development lightweight, experimental, and enjoyable rather than getting bogged down in complex engineering challenges. The author states a vision that emphasizes maintaining an exploratory, hacker-minded approach while building practical edge computing solutions highlighting the following core principles:\n",
    "\n",
    "- \"Will remain open-source\"\n",
    "- Focuses on simplicity and efficiency in codebase\n",
    "- Emphasizes quick prototyping over premature optimization\n",
    "- Aims to stay adaptable given rapid AI model improvements\n",
    "- Values practical experimentation over complex engineering\n",
    "\n",
    "LLama.cpp implementation characteristics include:\n",
    "\n",
    "1. **Memory Efficiency**: The main advantage of LLama.cpp is its ability to reduce memory requirements, allowing users to run large language models at the edge for instance offering ease of model quantization.\n",
    "\n",
    "2. **Computational Efficiency**: Besides reducing memory usage, LLama.cpp also focuses on improving execution efficiency, using specific C++ code optimizations to accelerate the process.\n",
    "\n",
    "3. **Ease of Implementation**: Although it's a lighter solution, LLama.cpp doesn't sacrifice result quality. It maintains the ability to generate texts and perform NLP tasks with high precision.\n",
    "\n",
    "**GGUF**\n",
    "\n",
    "GGUF (GPT-Generated Unified Format) {cite}`ggerganov2024ggufspec` is the latest model format used by LLama.cpp, replacing the older GGML format. It was designed specifically for efficient inference of large language models on consumer hardware. The key features that make GGUF particularly valuable include {cite}`ibm2024ggufversusggml`:\n",
    "\n",
    "- Improved quantization: GGUF supports multiple quantization levels to reduce model size while preserving performance. Common quantization schemes that are supported by GGUF include:\n",
    "    - 2-bit quantization: Offers the highest compression, significantly reducing model size and inference speed, though with a potential impact on accuracy.\n",
    "    - 4-bit quantization: Balances compression and accuracy, making it suitable for many practical applications.\n",
    "    - 8-bit quantization: Provides good accuracy with moderate compression, widely used in various applications.\n",
    "- Metadata support: The format includes standardized metadata about model architecture, tokenization, and other properties\n",
    "- Memory mapping: Enables efficient loading of large models by mapping them directly from disk rather than loading entirely into RAM\n",
    "- Architecture-specific optimizations: Takes advantage of CPU/GPU specific instructions for faster inference\n",
    "- Versioning support: Includes proper versioning to handle format evolution and backwards compatibility\n",
    "\n",
    "These capabilities make GGUF models significantly more practical for running LLMs locally compared to full-precision formats, often dramatically reducing memory requirements. Hugging Face hosts a growing collection of pre-converted GGUF models {cite}`huggingface2024ggufmodels` and provides a tool (ggml-org/gguf-my-repo) to convert existing models to GGUF format, making it easier for developers to access and deploy optimized versions of popular language models.\n",
    "\n",
    "\n",
    "**Setup**\n",
    "\n",
    "Please follow the instructions from the LLama.cpp [GitHub repository](https://github.com/ggerganov/llama.cpp) {cite}`ggerganov2024llamacpp` to install and compile the library.\n",
    "\n",
    "Here, we will compile the library from source on a Linux machine with 8 jobs in parallel for enhanced performance (add the `-j` argument to run multiple jobs in parallel). \n",
    "\n",
    "```bash\n",
    "sudo apt install cmake\n",
    "\n",
    "cmake -B build\n",
    "cmake --build build --config Release -j 8\n",
    "```\n",
    "\n",
    "Python bindings are available through `llama-cpp-python` package {cite}`betlen2024llamacpppython`.\n",
    "\n",
    "```bash\n",
    "pip install llama-cpp-python\n",
    "```\n",
    "\n",
    "**llama-cli**\n",
    "\n",
    "A comprehensive command line interface is available through `llama-cli` as demonstrated below, where we use the `-cnv` flag to run the model in a conversational mode. We will use `Qwen/Qwen2.5-0.5B-Instruct-GGUF` model. Download it from Hugging Face and place it in the `llamacpp/models` directory.\n",
    "\n",
    "```bash\n",
    "./build/bin/llama-cli -m ./models/qwen2.5-0.5b-instruct-q8_0.gguf -p \"You are a helpful assistant - Be succinct.\" -cnv\n",
    "```\n",
    "\n",
    "As a result, you can interact with the model in the terminal as a chatbot.\n",
    "\n",
    "```bash\n",
    "== Running in interactive mode. ==\n",
    " - Press Ctrl+C to interject at any time.\n",
    " - Press Return to return control to the AI.\n",
    " - To return control without starting a new line, end your input with '/'.\n",
    " - If you want to submit another line, end your input with '\\'.\n",
    "\n",
    "system\n",
    "You are a helpful assistant - Be succinct.\n",
    "\n",
    "> What is the meaning of life?\n",
    "The meaning of life is a philosophical question that has been debated and debated for thousands of years. Some people believe that the meaning of life is to seek personal fulfillment and happiness, while others believe that it is to find a purpose in life that aligns with one's values and beliefs. The answer may also vary depending on a person's cultural, religious, or personal background.\n",
    "\n",
    "> Are LLMs more helpful than dangerous?\n",
    "Yes, LLMs (Large Language Models) can be more helpful than dangerous in many cases. They are designed to assist with a wide range of tasks, from generating text to providing information. They can also be used to help with decision-making and problem-solving. However, like any tool, LLMs can be a tool of great power if not used responsibly and ethically. It is important to use LLMs for positive and beneficial purposes while being mindful of their potential to harm.\n",
    "\n",
    "> Bye bye.       \n",
    "Goodbye! If you have any other questions, feel free to ask.\n",
    "```\n",
    "\n",
    "**llama-server**\n",
    "\n",
    "`llama-server` is a server version of `llama-cli` that can be accessed via a web interface or API.\n",
    "\n",
    "```bash\n",
    "./build/bin/llama-server -m ./models/qwen2.5-0.5b-instruct-q8_0.gguf --port 8080\n",
    "```\n",
    "\n",
    "This will start a server on port 8080.\n",
    "```bash\n",
    "main: server is listening on http://127.0.0.1:8080 - starting the main loop\n",
    "```\n",
    "\n",
    "Now we can send a request as we would for any Cloud API but here instead send a request to our local server.\n",
    "```bash\n",
    "curl http://localhost:8080/v1/chat/completions \\\n",
    "-H \"Content-Type: application/json\" \\\n",
    "-H \"Authorization: Bearer no-key\" \\\n",
    "-d '{\n",
    "\"messages\": [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant - Be succinct.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"What is the meaning of life?\"\n",
    "    }\n",
    "  ]\n",
    "}'\n",
    "```\n",
    "\n",
    "We obtain a JSON response. As expected, assistant's response is in `content[0].message.content` following OpenAI's API format.\n",
    "\n",
    "```json\n",
    "{\n",
    "   \"choices\":[\n",
    "      {\n",
    "         \"finish_reason\":\"stop\",\n",
    "         \"index\":0,\n",
    "         \"message\":{\n",
    "            \"content\":\"The meaning of life is a question that has been debated throughout history. Some people believe it is to find happiness and purpose, while others believe it is to seek knowledge and knowledge. Ultimately, the meaning of life is a deeply personal and subjective question that cannot be answered universally.\",\n",
    "            \"role\":\"assistant\"\n",
    "         }\n",
    "      }\n",
    "   ],\n",
    "   \"created\":1734627879,\n",
    "   \"model\":\"gpt-3.5-turbo\",\n",
    "   \"object\":\"chat.completion\",\n",
    "   \"usage\":{\n",
    "      \"completion_tokens\":56,\n",
    "      \"prompt_tokens\":29,\n",
    "      \"total_tokens\":85\n",
    "   },\n",
    "   \"id\":\"chatcmpl-5Wl2TZJZDmzuPvxwP2GceDR8XbPsyHfm\",\n",
    "   \"timings\":{\n",
    "      \"prompt_n\":1,\n",
    "      \"prompt_ms\":48.132,\n",
    "      \"prompt_per_token_ms\":48.132,\n",
    "      \"prompt_per_second\":20.77619878666999,\n",
    "      \"predicted_n\":56,\n",
    "      \"predicted_ms\":1700.654,\n",
    "      \"predicted_per_token_ms\":30.36882142857143,\n",
    "      \"predicted_per_second\":32.92850867960208\n",
    "   }\n",
    "}\n",
    "```\n",
    "\n",
    "**Grammars**\n",
    "\n",
    "It is worth noting Llama.cpp provides a way to use grammars {cite}`ggerganov2024llamacppgrammars` to constrain the output of the model as demonstrated below. This is the same technique Ollama uses, a similar approach to Outlines' to generate structured outputs from LLMs. See Chapter {ref}`structure` for more details.\n",
    "\n",
    "```bash\n",
    "./build/bin/llama-cli -m ./models/qwen2.5-0.5b-instruct-q8_0.gguf --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'\n",
    "\n",
    "# {\"appointmentTime\": \"8pm\", \"appointmentDetails\": \"schedule a a call\"}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Python**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A handy Python binding {cite}`betlen2024llamacpppython` is available for LLama.cpp, which by default returns chat completions in OpenAI's API chat format as below. The package is very comprehensive supporting JSON Mode, function calling, multi-modal models and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_PATH = \"./models/qwen2.5-0.5b-instruct-q8_0.gguf\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_cpp import Llama\n",
    "llm = Llama(\n",
    "      model_path=MODEL_PATH\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = llm.create_chat_completion(\n",
    "      messages = [\n",
    "          {\"role\": \"system\", \"content\": \"You are a helpful assistant - Be succinct.\"},\n",
    "          {\n",
    "              \"role\": \"user\",\n",
    "              \"content\": \"What is the meaning of life?\"\n",
    "          }\n",
    "      ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The meaning of life is a philosophical question that has been debated by philosophers, scientists, and individuals throughout history. Some people believe that the meaning of life is to find happiness and fulfillment, while others believe that it is to seek knowledge and understanding of the universe. Ultimately, the meaning of life is a personal and subjective question that varies from person to person.'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response['choices'][0]['message']['content']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could have pulled our model directly from Hugging Face Hub:\n",
    "\n",
    "```python\n",
    "from llama_cpp import Llama\n",
    "llm = Llama.from_pretrained(\n",
    "    repo_id=\"Qwen/Qwen2-0.5B-Instruct-GGUF\",\n",
    "    verbose=False\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Llamafile\n",
    "\n",
    "\n",
    "Developed by Occupy Wall Street's former activist, Justine Tunney, Llamafile {cite}`mozilla2024llamafile` is an Appache 2.0 licensed open source tool that combines the power of LLama.cpp with **Cosmopolitan Libc**, a universal C standard library that allows creating portable executables compatible with multiple operating systems.\n",
    "\n",
    "In this way, Llamafile reduces all the complexity of LLMs to a single executable file (called a \"llamafile\") that runs locally without installation. Key advantages of Llamafile over plain Llama.cpp include:\n",
    "\n",
    "1. **Zero Installation/Configuration**\n",
    "- Llamafile: Single executable file that works immediately\n",
    "- Llama.cpp: Requires compilation, dependency management, and proper setup of your development environment\n",
    "\n",
    "2. **Cross-Platform Portability**\n",
    "- Llamafile: One binary works across Windows, macOS, and Linux without modification\n",
    "- Llama.cpp: Needs to be compiled separately for each operating system, managing platform-specific dependencies\n",
    "\n",
    "3. **Distribution Simplicity**\n",
    "- Llamafile: Share a single file that just works\n",
    "- Llama.cpp: Need to distribute source code or platform-specific binaries along with setup instructions\n",
    "\n",
    "Besides simplifying the use of LLMs, Llamafile delivers **durability** as model weights remain usable and reproducible over time, even as new formats and models are developed. In summary, Llamafile trades some optimization potential from LLama.cpp for improved ease of use and portability.\n",
    "\n",
    "\n",
    "A large collection of Llamafiles can be found on HuggingFace {cite}`huggingface2024llamafilemodels`. All you need to do is:\n",
    "\n",
    "1. Download a llamafile from HuggingFace\n",
    "2. Make the file executable\n",
    "3. Run the file\n",
    "\n",
    "Here's a simple bash script that shows all 3 setup steps for running TinyLlama-1.1B locally:\n",
    "\n",
    "```bash\n",
    "# Download a llamafile from HuggingFace\n",
    "wget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile\n",
    "\n",
    "# Make the file executable. On Windows, instead just rename the file to end in \".exe\".\n",
    "chmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile\n",
    "\n",
    "# Start the model server. Listens at http://localhost:8080 by default.\n",
    "./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser\n",
    "```\n",
    "\n",
    "As a result, a model server is running on http://localhost:8080. And we can use it as demonstrated in the previous section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ollama\n",
    "\n",
    "Ollama is a lightweight, MIT-licensed open-source tool for running LLMs locally. It provides a simple interface for interacting with a wide range of language models, including popular models like Llama 3.1 and Llama 3.2. Ollama is designed to be easy to install and use, making it a popular choice for developers who want to run LLMs locally without the need for extensive setup or configuration. Ollama's key advantages include:\n",
    "\n",
    "1. **Model Management**\n",
    "- Built-in model registry and easy downloading of popular models\n",
    "- Simple commands to list, remove, and switch between models\n",
    "- Handles model updates and versions automatically\n",
    "\n",
    "2. **API First Design**\n",
    "- Provides a REST API out of the box\n",
    "- Easy integration with applications and services\n",
    "- Built-in support for different programming languages\n",
    "\n",
    "3. **Container Support**\n",
    "- Native Docker integration\n",
    "- Easy deployment in containerized environments\n",
    "- Better resource isolation and management\n",
    "\n",
    "4. **User Experience**\n",
    "- More \"app-like\" experience with system tray integration\n",
    "- Simple CLI commands that feel familiar to developers\n",
    "- No need to deal with file permissions or executables\n",
    "\n",
    "Despite its advantages, Ollama comes with some trade-offs: it provides less low-level control compared to Llama.cpp, requires proper platform-specific installation unlike the portable Llamafile, and introduces additional resource overhead from running services that aren't present in bare Llama.cpp implementations.\n",
    "\n",
    "\n",
    "**Setup**\n",
    "\n",
    "First, install Ollama on your machine. You can do this through the terminal with the following command:\n",
    "\n",
    "```\n",
    "curl -sSfL https://ollama.com/download | sh\n",
    "```\n",
    "\n",
    "Or download the installer directly from https://ollama.com\n",
    "\n",
    "**Inference**\n",
    "\n",
    "After installation, you can download a pre-trained model. For example, to download the `qwen2:0.5b` model, run in terminal:\n",
    "\n",
    "```bash\n",
    "ollama run qwen2:0.5b\n",
    "```\n",
    "\n",
    "To see more details about the model, just run:\n",
    "\n",
    "```bash\n",
    "ollama show qwen2:0.5b\n",
    "```\n",
    "\n",
    "To stop the model server, run:\n",
    "\n",
    "```bash\n",
    "ollama stop qwen2:0.5b\n",
    "```\n",
    "\n",
    "To see all models you've downloaded:\n",
    "\n",
    "```bash\n",
    "ollama list\n",
    "```\n",
    "\n",
    "**Server**\n",
    "\n",
    "As in Llama.cpp and Llamafile, Ollama can be run as a server.\n",
    "\n",
    "```bash\n",
    "ollama serve\n",
    "```\n",
    "\n",
    "```bash\n",
    "ollama run qwen2:0.5b\n",
    "```\n",
    "\n",
    "And then we can send requests to the server.\n",
    "\n",
    "```bash\n",
    "curl http://localhost:11434/api/chat -d '{\n",
    "  \"model\": \"qwen2:0.5b\",\n",
    "  \"messages\": [\n",
    "    { \"role\": \"user\", \"content\": \"What is the meaning of life?\" }\n",
    "  ]\n",
    "}'\n",
    "```\n",
    "\n",
    "**Python**\n",
    "\n",
    "A Python binding is also available for Ollama.\n",
    "\n",
    "```bash\n",
    "pip install ollama\n",
    "```\n",
    "\n",
    "```python\n",
    "from ollama import chat\n",
    "from ollama import ChatResponse\n",
    "\n",
    "response: ChatResponse = chat(model='qwen2:0.5b', messages=[\n",
    "  {\n",
    "    'role': 'user',\n",
    "    'content': 'What is the meaning of life?',\n",
    "  },\n",
    "])\n",
    "print(response.message.content)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comparison\n",
    "\n",
    "Each solution offers distinct advantages and tradeoffs that make them suitable for different use cases. At a high-level, Ollama is the easiest to install and use and has become the most popular choice for your average use case, Llamafile is the easiest to distribute and a good choice when portability is a priority, and Llama.cpp is the most customizable and performant solution as summarized in {numref}`feature-comparison-local`.\n",
    "\n",
    "```{table} lama.cpp vs Ollama vs Llamafile Comparison\n",
    ":align: center\n",
    ":name: feature-comparison-local\n",
    "| Feature | Ollama | Llamafile | Llama.cpp |\n",
    "|---------|---------|-----------|-----------|\n",
    "| **Installation** | Package manager | No installation needed | Compilation / Package manager|\n",
    "| **Model Management** | Built-in registry | Manual download | Manual download |\n",
    "| **Containerization** | Native support | Possible with configuration | Possible with configuration |\n",
    "| **Portability** | Per-platform install | Single executable | Needs compilation |\n",
    "```\n",
    "\n",
    "Choose Ollama if you:\n",
    "- Want a user-friendly way to experiment with different models\n",
    "- Need API integration capabilities\n",
    "- Plan to use Docker in your workflow\n",
    "- Prefer a managed approach to model handling\n",
    " \n",
    "Choose Llamafile if you:\n",
    "- Need maximum portability\n",
    "- Want zero installation\n",
    "- Prefer a self-contained solution\n",
    " \n",
    "Choose Llama.cpp if you:\n",
    "- Need maximum performance\n",
    "- Want low-level control\n",
    "- Are building a custom solution\n",
    "\n",
    "\n",
    "### UI\n",
    "\n",
    "There is a growing number of UI tools for local LLM deployment that aim at providing a more user-friendly experience. Ranging from closed-source to open-source solutions across a range of features and capabilities. We will discuss LM Studio, Jan, and OpenWebUI.\n",
    "\n",
    "#### LM Studio\n",
    "\n",
    "LM Studio {cite}`lmstudio2024` is a closed-source GUI for running LLMs locally. In the context of local deployment, LM Studio positions itself as a more user-friendly, feature-rich solution compared to the other tools. It's particularly valuable for developers transitioning from cloud APIs to local deployment, and for users who prefer graphical interfaces over command-line tools. Key Features of LM Studio include:\n",
    "\n",
    "* **Model Parameter Customization**: Allows adjusting temperature, maximum tokens, frequency penalty, and other settings\n",
    "* **Chat History**: Enables saving prompts for later use\n",
    "* **Cross-platform**: Available on Linux, Mac, and Windows\n",
    "* **AI Chat and Playground**: Chat with LLMs and experiment with multiple models loaded simultaneously\n",
    "\n",
    "{numref}`lmstudio` and {numref}`lmstudio_server` show LM Studio's chat interface and server, respectively.\n",
    "\n",
    "```{figure} ../_static/local/lmstudio.png\n",
    "---\n",
    "name: lmstudio\n",
    "alt: LM Studio\n",
    "scale: 30%\n",
    "align: center\n",
    "---\n",
    "LM Studio Chat Interface.\n",
    "```\n",
    "\n",
    "```{figure} ../_static/local/lmstudio_server.png\n",
    "---\n",
    "name: lmstudio_server\n",
    "alt: LM Studio Server\n",
    "scale: 30%\n",
    "align: center\n",
    "---\n",
    "LM Studio Server.\n",
    "```\n",
    "\n",
    "One important feature of LM Studio is that it provides machine specification verification capabilities, checking computer specifications like GPU and memory to report compatible models therefore helping users choose the right model. It also includes a local inference server for developers that allows setting up a local HTTP server similar to OpenAI's API. Importantly, LM Studio's OpenAI API compatibility is a particularly strong feature for developers looking to move their applications from cloud to local deployment with minimal code changes.\n",
    "\n",
    "#### Jan\n",
    "\n",
    "Jan is an open source ChatGPT-alternative that runs local models. Its model's library contains popular LLMs like Llama, Gemma, Mistral, or Qwen. Key Features of Jan include:\n",
    "\n",
    "1. **User-Friendly Interface**: Run AI models with just a few clicks\n",
    "2. **Accessibility**: Intuitive platform for both beginners and experts\n",
    "3. **Local Server**: Local API Server with OpenAI-equivalent API\n",
    "4. **Model Hub Integration**: Easy access to various models with ease of import from LM Studio\n",
    "5. **Cross-Platform Support**: Works across different operating systems\n",
    "\n",
    "Jan has a default C++ inference server built on top of llama.cpp and provides an OpenAI-compatible API. Jan natively supports GGUF (through a llama.cpp engine) and TensorRT (through a TRT-LLM engine). HuggingFace models can be downloaded directly using the model’s ID or URL. User can optionally use cloud-based models (e.g. GPT, Claude models). {numref}`jan` shows Jan's chat interface.\n",
    "\n",
    "```{figure} ../_static/local/jan.png\n",
    "---\n",
    "name: jan\n",
    "alt: Jan\n",
    "scale: 50%\n",
    "align: center\n",
    "---\n",
    "Jan Chat Interface.\n",
    "```\n",
    "\n",
    "#### Open WebUI\n",
    "\n",
    "Open WebUI is an open-source web interface designed to enhance the local AI model experience, particularly for Ollama and OpenAI-compatible APIs. It aims to provide enterprise-grade features while maintaining user-friendliness. OpenWebUI's core features include:\n",
    "\n",
    "1. **Advanced User Interface**\n",
    "   - Full markdown and LaTeX support\n",
    "   - Voice and video call capabilities\n",
    "   - Mobile-friendly with PWA support\n",
    "   - Multi-model chat interface\n",
    "\n",
    "2. **Enterprise Features**\n",
    "   - Role-based access control\n",
    "   - User groups and permissions\n",
    "   - Usage monitoring\n",
    "   - Team collaboration tools\n",
    "\n",
    "3. **Advanced Capabilities**\n",
    "   - Local RAG (Retrieval Augmented Generation)\n",
    "   - Web search integration\n",
    "   - Image generation support\n",
    "   - Python function calling\n",
    "   - Document library\n",
    "   - Custom model building\n",
    "\n",
    "{numref}`openwebui` shows Open WebUI's chat interface.\n",
    "\n",
    "```{figure} ../_static/local/openwebui.png\n",
    "---\n",
    "name: openwebui\n",
    "alt: Open WebUI\n",
    "scale: 25%\n",
    "align: center\n",
    "---\n",
    "Open WebUI Chat Interface.\n",
    "```\n",
    "\n",
    "While Open WebUI offers advanced capabilities including RAG and multi-model support, these features require more system resources than simpler alternatives. Open WebUI is likely to be adopted by enterprise users who require advanced features and a more user-friendly interface.\n",
    "\n",
    "#### Comparison\n",
    "\n",
    "LM Studio excels at providing individual developers with a smooth transition from cloud APIs to local deployment, offering an intuitive interface and robust API compatibility, however it is closed-source. Jan focuses on simplicity and accessibility, making it ideal for personal use and basic deployments while maintaining open-source benefits. OpenWebUI makes additional features available to enterprise users and teams requiring advanced features like RAG, collaboration tools, and granular access controls, though this may come at the cost of increased complexity and resource requirements. We compare the three tools in {numref}`feature-comparison-ui`.\n",
    "\n",
    "```{table} LM Studio vs Jan vs OpenWebUI Comparison\n",
    ":align: center\n",
    ":name: feature-comparison-ui\n",
    "| Feature Category | LM Studio | Jan | OpenWebUI |\n",
    "|-----------------|------------|-----|-----------|\n",
    "| **Licensing** | Closed Source | Open Source | Open Source |\n",
    "| **Setup Complexity** | Medium | Easy | Complex |\n",
    "| **Resource Usage** | High | Medium | High |\n",
    "| **Target Users** | Individual/Developers | Individuals | Enterprise/Teams |\n",
    "| **UI Features** | - Full GUI<br>- Parameter tuning<br>- Chat history<br>- Model playground | - Simple GUI<br>- Basic parameter tuning<br>- Chat interface<br>- Model import | - Advanced GUI<br>- Full markdown/LaTeX<br>- Voice/video calls<br>- PWA support |\n",
    "| **Model Support** | - Multiple models<br>- Hardware verification<br>- Model compatibility check | - Multiple models<br>- Import from GPT4All/LM Studio<br>- Basic model management | - Multi-model chat<br>- Model builder<br>- Custom agents |\n",
    "| **API Features** | - OpenAI compatible<br>- Local inference server<br>- API documentation | - Basic OpenAI compatible<br>- Local API server | - Multiple API support<br>- Python function calling<br>- Advanced integrations |\n",
    "| **Enterprise Features** | Limited | None | - RBAC<br>- Team collaboration<br>- Usage monitoring |\n",
    "| **Advanced Features** | - Parameter visualization<br>- Performance metrics | - Basic chat<br>- Simple model switching | - RAG support<br>- Web search<br>- Document library<br>- Image generation |\n",
    "| **Best For** | - Individual developers<br>- API transition<br>- Local development | - Personal use<br>- Simple deployment<br>- Basic chat needs | - Enterprise use<br>- Team collaboration<br>- Advanced AI applications |\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: The Effect of Quantization on LLM Performance\n",
    "\n",
    "This case study examines how different quantization {cite}`hf2024quantization` levels affect the performance of language models running locally. Quantization is a crucial technique for reducing model size and memory footprint while enhancing inference speed, but it comes with potential tradeoffs in model quality. Understanding these tradeoffs is essential for practitioners deploying LLMs in resource-constrained environments.\n",
    "\n",
    "Using the Qwen 2.5 0.5B model as our baseline, we'll compare four variants:\n",
    "- The base fp16 model (no quantization)\n",
    "- Q2_K quantization (highest compression, lowest precision)\n",
    "- Q4_K quantization (balanced compression/precision)\n",
    "- Q6_K quantization (lowest compression, highest precision)\n",
    "\n",
    "The analysis will focus on three key types of metrics:\n",
    "- **Quality-based**:\n",
    "  1. Perplexity - to measure how well the model predicts text\n",
    "  2. KL divergence - to quantify differences in probability distributions against base model\n",
    "- **Resource/Performance-based**:\n",
    "  1. Prompt (tokens/second) - to assess impact in throughput\n",
    "  2. Text Generation (tokens/second) - to assess impact in text generation performance\n",
    "  3. Model Size (MiB) - to assess impact in memory footprint\n",
    "\n",
    "While we will focus on the Qwen 2.5 0.5B model, the same analysis can be applied to other models. These insights will help practitioners make informed decisions about quantization strategies based on their specific requirements for model performance and resource usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompts Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate the impact of quantization on model performance, we first need a set of prompts that will serve as input data for our experiments. We'll construct a dataset from WikiText-2 {cite}`salesforce2024wikitext`, which contains Wikipedia excerpts. \n",
    "\n",
    "In our experiments, we will use a total of `NUM_PROMPTS` prompts that vary in length from `MIN_PROMPT_LENGTH` to `MAX_PROMPT_LENGTH` tokens. Using a fixed set of prompts ensures consistent evaluation across model variants and enables direct comparison of metrics like perplexity and throughput.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_PROMPTS = 100\n",
    "MIN_PROMPT_LENGTH = 100\n",
    "MAX_PROMPT_LENGTH = 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets\n",
    "input_texts_raw = datasets.load_dataset(\"Salesforce/wikitext\", \"wikitext-2-raw-v1\", split=\"train\")[\"text\"]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_texts = [s for s in input_texts_raw if s!='' and len(s) > MIN_PROMPT_LENGTH and len(s) < MAX_PROMPT_LENGTH][:NUM_PROMPTS]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(input_texts)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(input_texts[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/local/prompts.txt', 'w') as f:\n",
    "    for text in input_texts:\n",
    "        # Escape any quotes in the text and wrap in quotes\n",
    "        escaped_text = text.replace('\"', '\\\\\"')\n",
    "        f.write(f'\"{escaped_text}\"\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantization\n",
    "\n",
    "We can quantize a model using the `llama-quantize` CLI. For instance, to quantize the Qwen 2.5 0.5B model to Q4_K, we can run the following command:\n",
    "```bash\n",
    "./llama-quantize -m ./models/qwen2.5-0.5b-instruct-fp16.gguf ./models/qwen2.5-0.5b-instruct-q8_0.gguf Q4_K\n",
    "```\n",
    "\n",
    "{numref}`quantization-levels` describes the key quantization levels used in this study {cite}`huggingface2024quantization`, where:\n",
    "- q is the quantized value\n",
    "- block_scale is the scaling factor for the block (with bit width in parentheses)\n",
    "- block_min is the block minimum value (with bit width in parentheses)\n",
    "\n",
    "```{table} Quantization Levels\n",
    ":align: center\n",
    ":name: quantization-levels\n",
    "| Quantization | Description | Bits per Weight | Formula |\n",
    "|--------------|-------------|-----------------|----------|\n",
    "| Q2_K | 2-bit quantization with 16 weights per block in 16-block superblocks | 2.5625 | w = q * block_scale(4-bit) + block_min(4-bit) |\n",
    "| Q4_K | 4-bit quantization with 32 weights per block in 8-block superblocks | 4.5 | w = q * block_scale(6-bit) + block_min(6-bit) |\n",
    "| Q6_K | 6-bit quantization with 16 weights per block in 16-block superblocks | 6.5625 | w = q * block_scale(8-bit) |\n",
    "```\n",
    "\n",
    "Each quantization level represents a different tradeoff between model size and accuracy. Q2_K provides the highest compression but potentially lower accuracy, while Q6_K maintains better accuracy at the cost of larger model size. The base model is 16-bit standard IEEE 754 half-precision floating-point number.\n",
    "\n",
    "### Benchmarking\n",
    "\n",
    "We will measure quantized model \"quality\" by means of perplexity and KL Divergence.\n",
    "\n",
    "**Perplexity**\n",
    "\n",
    "Perplexity is a common metric for evaluating language models that measures how well a model predicts a sample of text. Lower perplexity indicates better prediction (less \"perplexed\" by the text).\n",
    "\n",
    "Recall that for a sequence of N tokens, perplexity is defined as:\n",
    "\n",
    "$$ \\text{PPL(B, X)} = \\exp\\left(-\\frac{1}{N}\\sum_{i=1}^{N} \\log_2 P(x_i|x_{<i})\\right) $$\n",
    "\n",
    "where:\n",
    "- $P(x_i|x_{<i})$ is the probability the model $B$ with tokenized sequence $X$ assigns to token $x_i$ given the previous tokens $x_{<i}$\n",
    "- $N$ is the total number of tokens in the sequence\n",
    "\n",
    "To evaluate quantization quality, we first calculate perplexity scores for both the base model and quantized variants. We then compute the ratio of quantized to base perplexity and average it across all prompt samples as follows:\n",
    "\n",
    "$$ Avg PPL Ratio = \\frac{1}{N}\\sum_{i=1}^{N} \\frac{\\text{PPL}_i(Q)}{\\text{PPL}_i(\\text{base})} $$\n",
    "\n",
    "We also calculate the correlation between the log perplexities of the quantized and base models:\n",
    "\n",
    "$$ \\text{Corr}(\\ln(\\text{PPL}(Q)), \\ln(\\text{PPL}(\\text{base}))) $$\n",
    "\n",
    "\n",
    "These are two simple metrics to evaluate how much worse the quantized model performs on an intrinsic basis which we then can compare to the base model's perplexities.\n",
    "\n",
    "Arguably, KL Divergence is a better metric enabling direct comparison of relative performance instead of intrinsic performance.\n",
    "\n",
    "**KL Divergence**\n",
    "\n",
    "Recall that Kullback-Leibler (KL) Divergence (or Cross-Entropy) measures how one probability distribution differs from another reference distribution. For comparing logits between a base model (B) and quantized model (Q), we can calculate the KL divergence as follows:\n",
    "\n",
    "$$ D_{KL}(P||Q) = \\sum_{i} P(i) \\log \\frac{P(i)}{Q(i)} $$\n",
    "\n",
    "where:\n",
    "- $P(i)$ and $Q(i)$ are the softmax probabilities derived from the logits\n",
    "- The sum is taken over all tokens in the vocabulary\n",
    "\n",
    "**Implementation**\n",
    "\n",
    "We will use LLama.cpp's `llama-perplexity` CLI to calculate perplexity and KL divergence. The first step is to generate the logits for the base model, which will serve as the reference distribution. For instance, below we pass our input prompts (`prompts.txt`) and generate the logits for the base model `qwen2.5-0.5b-instruct-fp16.gguf` which will be saved in `logits.kld`.\n",
    "\n",
    "```bash\n",
    "./build/bin/llama-perplexity -m ./models/qwen2.5-0.5b-instruct-fp16.gguf --kl-divergence-base ../logits.kld -f ../prompts.txt\n",
    "```\n",
    "\n",
    "Next, we generate KL-Divergence and perplexity stats for quantized model `qwen2.5-0.5b-instruct-q2_k.gguf` against base model logits `logits.kld`.\n",
    "\n",
    "```bash\n",
    "./build/bin/llama-perplexity -m ./models/qwen2.5-0.5b-instruct-q2_k.gguf -f ../prompts.txt --kl-divergence-base ../logits.kld --kl-divergence &> ../q2_kresults.txt\n",
    "```\n",
    "\n",
    "We perform this process for each quantization level studied (Q2_K, Q4_K, Q6_K).\n",
    "\n",
    "\n",
    "### Results\n",
    "\n",
    "The KL divergence and perplexity results in {numref}`ppl1` and {numref}`ppl2` provide insights into model quality across different quantization levels. Q6 maintains near-perfect correlation (99.90%) with the base model and minimal KL divergence (0.004), indicating very close distribution matching. Q2's higher KL divergence (0.112) and lower correlation (98.31%) quantify its increased deviation from the base model's behavior.\n",
    " \n",
    "\n",
    "```{figure} ../_static/local/ppl2.png\n",
    "---\n",
    "name: ppl2\n",
    "alt: Perplexity\n",
    "scale: 50%\n",
    "align: center\n",
    "---\n",
    "KL Divergence results for Quantization Q2, Q4, and Q6 quantized models.\n",
    "```\n",
    "\n",
    "```{figure} ../_static/local/ppl1.png\n",
    "---\n",
    "name: ppl1\n",
    "alt: Perplexity\n",
    "scale: 50%\n",
    "align: center\n",
    "---\n",
    "Perplexity results for Quantization Q2, Q4, and Q6 quantized models.\n",
    "```\n",
    "\n",
    "From {numref}`quantization-benchmarks`, we observe that the Q2 model achieves the smallest size at 390 MiB \n",
    "(67% reduction from base) with prompt throughput of 81 tokens/s, but has the highest perplexity degradation at 10.36%. The Q4 model offers a better balance, with good size savings (60% reduction) and only 3.5% perplexity loss. Q6 comes closest to matching the base model's performance with just 0.93% perplexity degradation, while still providing 47% size reduction.\n",
    "\n",
    "\n",
    "\n",
    "```{table} Quantization Benchmarks\n",
    ":align: center\n",
    ":name: quantization-benchmarks\n",
    "| Model | Size (MiB) | Prompt Throughput (tokens/s) | PPL Ratio - 1 (%) | Correlation (%) | KL Divergence (Mean) |\n",
    "|-------|------------|----------------------|-------------------|-----------------|-------------------|\n",
    "| **Q2**    | 390.28     | 81.32               | 10.36 ± 0.78     | 98.31          | 0.112 ± 0.002     |\n",
    "| **Q4**    | 462.96     | 77.08               | 3.50 ± 0.40      | 99.50          | 0.030 ± 0.001     |\n",
    "| **Q6**    | 614.58     | 87.55               | 0.93 ± 0.18      | 99.90          | 0.004 ± 0.000     |\n",
    "| **Base**  | 1,170.00   | 94.39               | -                | -              | -                 |\n",
    "```\n",
    "\n",
    "Next, we benchmark text generation (inference) performance using `llama-bench` across all models:\n",
    "\n",
    "```bash\n",
    "./build/bin/llama-bench -r 10 -t 4 -m ./models/qwen2.5-0.5b-instruct-fp16.gguf -m ./models/qwen2.5-0.5b-instruct-q2_k.gguf -m ./models/qwen2.5-0.5b-instruct-q4_k_m.gguf -m ./models/qwen2.5-0.5b-instruct-q6_k.gguf\n",
    "```\n",
    "\n",
    "The benchmark parameters are:\n",
    "- `-r 10`: Run 10 iterations for each model\n",
    "- `-t 4`: Use 4 threads\n",
    "- `-m`: Specify model paths for base FP16 model and Q2, Q4, Q6 quantized versions\n",
    "\n",
    "This runs text generation on a default benchmark of 128 tokens generation length (configurable via `-g` parameter).\n",
    "\n",
    "Results in {numref}`tg` indicate the base model delivers text generation performance at 19.73 tokens/s, while the most aggressively quantized Q2 model (390.28 MiB) delivers the highest throughput at 42.62 tokens/s, representing a 2.16x speedup. This pattern continues across Q4 (462.96 MiB, 38.38 tokens/s) and Q6 (614.58 MiB, 35.43 tokens/s), which presents a 1.85x and 1.79x speedup, respectively.\n",
    "\n",
    "```{figure} ../_static/local/tg.png\n",
    "---\n",
    "name: tg\n",
    "alt: Text Generation Performance\n",
    "scale: 50%\n",
    "align: center\n",
    "---\n",
    "Text Generation Performance results for Quantization Q2, Q4, Q6 and base models.\n",
    "```\n",
    "\n",
    "\n",
    "Benchmarking was performed on Ubuntu 24.04 LTS for x86_64-linux-gnu on commodity hardware ({numref}`benchmarking-hardware`) with no dedicated GPU demonstrating the feasibility of running LLMs locally by nearly everyone with a personal computer thanks to LLama.cpp.\n",
    "\n",
    "```{table} Benchmarking Hardware\n",
    ":align: center\n",
    ":name: benchmarking-hardware\n",
    "| Device | Description |\n",
    "|--------|-------------|\n",
    "| processor |  Intel(R) Core(TM) i7-8550U CPU @ 1 |\n",
    "| memory | 15GiB System memory |\n",
    "| storage | Samsung SSD 970 EVO Plus 500GB |\n",
    "```\n",
    "\n",
    "### Takeaways\n",
    "\n",
    "The quantization analysis of the Qwen 2.5 0.5B model demonstrates a clear trade-off among model size, inference speed, and prediction quality. While the base model (1170 MiB) maintains the highest accuracy it operates at the lowest text generation and prompt throughput of 19.73 tokens/s and 94.39 tokens/s, respectively. In contrast, the Q2_K quantization achieves significant size reduction (67%) and the highest throughput (42.62 tokens/s), but exhibits the largest quality degradation with a 10.36% perplexity increase and lowest KL divergence among quantized models. Q4_K emerges as a compelling middle ground, offering substantial size reduction (60%) and strong text generation and prompt throughput performance (38.38 tokens/s and 77.08 tokens/s, respectively), while maintaining good model quality with only 3.5% perplexity degradation and middle-ground KL divergence level. \n",
    "\n",
    "These results, achieved on commodity CPU hardware, demonstrate that quantization can significantly improve inference speed and reduce model size while maintaining acceptable quality thresholds, making large language models more accessible for resource-constrained environments.\n",
    "\n",
    "It is important to note that these results are not meant to be exhaustive and are only meant to provide a general idea of the trade-offs involved in quantization. Targeted benchmarks should be performed for specific use cases and models to best reflect real-world performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running open source language models locally represents a compelling proposition in how we interact with AI technology. The transition from cloud-based to local deployment offers important advantages in terms of privacy, cost control, and customization flexibility, while introducing important technical considerations around resource management and performance optimization. The growing ecosystem of tools and frameworks, from low-level libraries like llama.cpp to user-friendly interfaces like LM Studio and Jan, has made local deployment increasingly accessible to both individual developers and organizations.\n",
    "\n",
    "Our case study demonstrated that quantization can significantly improve inference speed and reduce model size while maintaining acceptable quality thresholds, making large language models more accessible for resource-constrained environments. As demonstrated in our case study with the Qwen 2.5 0.5B model, practitioners can achieve significant reductions in model size and improvements in inference speed while maintaining acceptable performance levels. The Q4_K quantization scheme emerged as a particularly effective compromise, offering substantial size reduction (60%) and strong throughput while limiting quality degradation to just 3.5% in perplexity measures.\n",
    "\n",
    "Looking ahead, the continued development of open source models and deployment tools suggests a future where local AI deployment becomes increasingly viable and sophisticated. The success of open source models like Qwen and Llama, combined with improvements in local model serving and techniques couple with efficient small language models (SLMs), indicate that local deployment will likely play an increasingly important role in the AI landscape. However, practitioners must carefully evaluate their specific requirements across dimensions like task suitability, resource constraints, and performance needs when choosing between local and cloud-based deployment strategies.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa]\n",
    "\n",
    "[cc-by-nc-sa]: http://creativecommons.org/licenses/by-nc-sa/4.0/\n",
    "[cc-by-nc-sa-image]: https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png\n",
    "[cc-by-nc-sa-shield]: https://img.shields.io/badge/License-CC-BY--NC--SA-4.0-lightgrey.svg\n",
    "\n",
    "```\n",
    "@misc{tharsistpsouza2024tamingllms,\n",
    "  author = {Tharsis T. P. Souza},\n",
    "  title = {Taming LLMs: A Practical Guide to LLM Pitfalls with Open Source Software},\n",
    "  year = {2024},\n",
    "  chapter = {Local LLMs in Practice},\n",
    "  journal = {GitHub repository},\n",
    "  url = {https://github.com/souzatharsis/tamingLLMs)\n",
    "}\n",
    "```\n",
    "## References\n",
    "```{bibliography}\n",
    ":filter: docname in docnames\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}