6_InfiniteMixtureModel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Probabilistic Modelling using the Infinite Mixture Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many applications it is desirable to allow the model to adjust its complexity to the amount the data. Consider for example the task of assigning objects into clusters or groups. This task often involves the specification of the number of groups. However, often times it is not known beforehand how many groups exist. Moreover, in some applictions, e.g. modelling topics in text documents or grouping species, the number of examples per group is heavy tailed. This makes it impossible to predefine the number of groups and requiring the model to form new groups when data points from previously unseen groups are observed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A natural approach for such applications is the use of non-parametric models. This tutorial will introduce how to use the Dirichlet process in a mixture of infinitely many Gaussians using Turing. For further information on Bayesian nonparametrics and the Dirichlet process we refer to the [introduction by Zoubin Ghahramani](http://mlg.eng.cam.ac.uk/pub/pdf/Gha12.pdf) and the book \"Fundamentals of Nonparametric Bayesian Inference\" by Subhashis Ghosal and Aad van der Vaart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Turing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mixture Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before introducing infinite mixture models in Turing, we will briefly review the construction of finite mixture models. Subsequently, we will define how to use the [Chinese restaurant process](https://en.wikipedia.org/wiki/Chinese_restaurant_process) construction of a Dirichlet process for non-parametric clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Two-Component Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, consider the simple case of a mixture model with two Gaussian components with fixed covariance. \n",
    "The generative process of such a model can be written as:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\pi_1 &\\sim Beta(a, b) \\\\\n",
    "\\pi_2 &= 1-\\pi_1 \\\\\n",
    "\\mu_1 &\\sim Normal(\\mu_0, \\Sigma_0) \\\\\n",
    "\\mu_2 &\\sim Normal(\\mu_0, \\Sigma_0) \\\\\n",
    "z_i &\\sim Categorical(\\pi_1, \\pi_2) \\\\\n",
    "x_i &\\sim Normal(\\mu_{z_i}, \\Sigma)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "where $\\pi_1, \\pi_2$ are the mixing weights of the mixture model, i.e. $\\pi_1 + \\pi_2 = 1$, and $z_i$ is a latent assignment of the observation $x_i$ to a component (Gaussian).\n",
    "\n",
    "We can implement this model in Turing for 1D data as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "##two_model#362 (generic function with 2 methods)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@model two_model(x) = begin\n",
    "    \n",
    "    # Hyper-parameters\n",
    "    μ0 = 0.0\n",
    "    σ0 = 1.0\n",
    "    \n",
    "    # Draw weights.\n",
    "    π1 ~ Beta(1,1)\n",
    "    π2 = 1-π1\n",
    "    \n",
    "    # Draw locations of the components.\n",
    "    μ1 ~ Normal(μ0, σ0)\n",
    "    μ2 ~ Normal(μ0, σ0)\n",
    "    \n",
    "    # Draw latent assignment.\n",
    "    z ~ Categorical([π1, π2])\n",
    "    \n",
    "    # Draw observation from selected component.\n",
    "    if z == 1\n",
    "        x ~ Normal(μ1, 1.0)\n",
    "    else\n",
    "        x ~ Normal(μ2, 1.0)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finite Mixture Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we have more than two components, this model can elegantly be extend using a Dirichlet distribution as prior for the mixing weights $\\pi_1, \\dots, \\pi_K$. Note that the Dirichlet distribution is the multivariate generalization of the beta distribution. The resulting model can be written as:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "(\\pi_1, \\dots, \\pi_K) &\\sim Dirichlet(K, \\alpha) \\\\\n",
    "\\mu_k &\\sim Normal(\\mu_0, \\Sigma_0), \\;\\; \\forall k \\\\\n",
    "z &\\sim Categorical(\\pi_1, \\dots, \\pi_K) \\\\\n",
    "x &\\sim Normal(\\mu_z, \\Sigma) \n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "which resembles the model in the [Gaussian mixture model tutorial](https://turing.ml/dev/tutorials/1-gaussianmixturemodel/) with a slightly different notation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Infinite Mixture Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The question now arises, is there a generalization of a Dirichlet distribution for which the dimensionality $K$ is infinite, i.e. $K = \\infty$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But first, to implement an infinite Gaussian mixture model in Turing, we first need to load the `Turing.RandomMeasures` module. `RandomMeasures` contains a variety of tools useful in nonparametrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Turing.RandomMeasures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now will utilize the fact that one can integrate out the mixing weights in a Gaussian mixture model allowing us to arrive at the Chinese restaurant process construction. See Carl E. Rasmussen: [The Infinite Gaussian Mixture Model](https://www.seas.harvard.edu/courses/cs281/papers/rasmussen-1999a.pdf), NIPS (2000) for details.\n",
    "\n",
    "In fact, if the mixing weights are integrated out, the conditional prior for the latent variable $z$ is given by:\n",
    "\n",
    "$$ \n",
    "p(z_i = k \\mid z_{\\not i}, \\alpha) = \\frac{n_k + \\alpha/K}{N - 1 + \\alpha}\n",
    "$$\n",
    "\n",
    "where $z_{\\not i}$ are the latent assignments of all observations except observation $i$. Note that we use $n_k$ to denote the number of observations at component $k$ excluding observation $i$. The parameter $\\alpha$ is the concentration parameter of the Dirichlet distribution used as prior over the mixing weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Chinese Restaurant Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To obtain the Chinese restaurant process construction, we can now derive the conditional prior if $K \\rightarrow \\infty$.\n",
    "\n",
    "For $n_k > 0$ we obtain:\n",
    "\n",
    "$$\n",
    "p(z_i = k \\mid z_{\\not i}, \\alpha) = \\frac{n_k}{N - 1 + \\alpha}\n",
    "$$\n",
    "\n",
    "and for all infinitely many clusters that are empty (combined) we get:\n",
    "\n",
    "$$\n",
    "p(z_i = k \\mid z_{\\not i}, \\alpha) = \\frac{\\alpha}{N - 1 + \\alpha}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those equations show that the conditional prior for component assignments is proportional to the number of such observations, meaning that the Chinese restaurant process has a rich get richer property."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a better understanding of this property, we can plot the cluster choosen by for each new observation drawn from the conditional prior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concentration parameter.\n",
    "α = 10.0\n",
    "\n",
    "# Random measure, e.g. Dirichlet process.\n",
    "rpm = DirichletProcess(α)\n",
    "\n",
    "# Cluster assignments for each observation.\n",
    "z = Vector{Int}()\n",
    "\n",
    "# Maximum number of observations we observe.\n",
    "Nmax = 500\n",
    "\n",
    "for i in 1:Nmax\n",
    "    # Number of observations per cluster.\n",
    "    K = isempty(z) ? 0 : maximum(z)\n",
    "    nk = Vector{Int}(map(k -> sum(z .== k), 1:K))\n",
    "    \n",
    "    # Draw new assignment.\n",
    "    push!(z, rand(ChineseRestaurantProcess(rpm, nk)))\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Saved animation to \n",
      "│   fn = /home/cameron/code/TuringTutorials/tmp.gif\n",
      "└ @ Plots /home/cameron/.julia/packages/Plots/cc8wh/src/animation.jl:98\n"
     ]
    }
   ],
   "source": [
    "using Plots\n",
    "\n",
    "# Plot the cluster assignments over time \n",
    "@gif for i in 1:Nmax\n",
    "    scatter(collect(1:i), z[1:i], markersize = 2, xlabel = \"observation (i)\", ylabel = \"cluster (k)\", legend = false)\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![tmp](https://user-images.githubusercontent.com/422990/55284032-90cfa980-5323-11e9-8a99-f9315db170cb.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further, we can see that the number of clusters is logarithmic in the number of observations and data points. This is a side-effect of the \"rich get richer\" phenomenon, i.e. we expect large clusters and thus the number of clusters has to be smaller than the number of observations.\n",
    "\n",
    "$$ \n",
    "E[K \\mid N] \\approx \\alpha * log \\big(1 - \\frac{N}{\\alpha}\\big)\n",
    "$$\n",
    "\n",
    "We can see from the equation that the concetration parameter $\\alpha$ allows use to control the number of cluster formed a priori."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Turing we can implement an infinite Gaussian mixture model using the Chinese restaurant process construction of a Dirichlet process as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "##infiniteGMM#903 (generic function with 2 methods)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@model infiniteGMM(x) = begin\n",
    "    \n",
    "    # Hyper-parameters, i.e. concentration parameter and parameters of H.\n",
    "    α = 1.0\n",
    "    μ0 = 0.0\n",
    "    σ0 = 1.0\n",
    "    \n",
    "    # Define random measure, e.g. Dirichlet process.\n",
    "    rpm = DirichletProcess(α)\n",
    "    \n",
    "    # Define the base distribution, i.e. expected value of the Dirichlet process.\n",
    "    H = Normal(μ0, σ0)\n",
    "    \n",
    "    # Latent assignment.\n",
    "    z = tzeros(Int, length(x))\n",
    "        \n",
    "    # Locations of the infinitely many clusters.\n",
    "    μ = tzeros(Float64, 0)\n",
    "    \n",
    "    for i in 1:length(x)\n",
    "        \n",
    "        # Number of clusters.\n",
    "        K = maximum(z)\n",
    "        nk = Vector{Int}(map(k -> sum(z .== k), 1:K))\n",
    "\n",
    "        # Draw the latent assignment.\n",
    "        z[i] ~ ChineseRestaurantProcess(rpm, nk)\n",
    "        \n",
    "        # Create a new cluster?\n",
    "        if z[i] > K\n",
    "            push!(μ, 0.0)\n",
    "\n",
    "            # Draw location of new cluster.\n",
    "            μ[z[i]] ~ H\n",
    "        end\n",
    "                \n",
    "        # Draw observation.\n",
    "        x[i] ~ Normal(μ[z[i]], 1.0)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use Turing to infer the assignments of some data points. First, we will create some random data that comes from three clusters, with means of 0, -5, and 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Plots, Random\n",
    "\n",
    "# Generate some test data.\n",
    "Random.seed!(1)\n",
    "data = vcat(randn(10), randn(10) .- 5, randn(10) .+ 10)\n",
    "data .-= mean(data)\n",
    "data /= std(data);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll sample from our posterior using SMC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32mSampling: 100%|█████████████████████████████████████████| Time: 0:00:00\u001b[39m\n"
     ]
    }
   ],
   "source": [
    "# MCMC sampling\n",
    "Random.seed!(2)\n",
    "iterations = 1000\n",
    "model_fun = infiniteGMM(data);\n",
    "chain = sample(model_fun, SMC(), iterations);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can plot the number of clusters in each sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip4800\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip4800)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4801\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip4800)\" d=\"\n",
       "M175.611 1425.62 L2352.76 1425.62 L2352.76 47.2441 L175.611 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4802\">\n",
       "    <rect x=\"175\" y=\"47\" width=\"2178\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  235.172,1425.62 235.172,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  749.164,1425.62 749.164,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1263.16,1425.62 1263.16,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1777.15,1425.62 1777.15,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2291.14,1425.62 2291.14,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,1200.84 2352.76,1200.84 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,829.314 2352.76,829.314 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,457.784 2352.76,457.784 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,86.2547 2352.76,86.2547 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,1425.62 175.611,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  235.172,1425.62 235.172,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  749.164,1425.62 749.164,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1263.16,1425.62 1263.16,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1777.15,1425.62 1777.15,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2291.14,1425.62 2291.14,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,1200.84 201.736,1200.84 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,829.314 201.736,829.314 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,457.784 201.736,457.784 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,86.2547 201.736,86.2547 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 235.172, 1479.62)\" x=\"235.172\" y=\"1479.62\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 749.164, 1479.62)\" x=\"749.164\" y=\"1479.62\">250</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1263.16, 1479.62)\" x=\"1263.16\" y=\"1479.62\">500</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1777.15, 1479.62)\" x=\"1777.15\" y=\"1479.62\">750</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2291.14, 1479.62)\" x=\"2291.14\" y=\"1479.62\">1000</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 1218.34)\" x=\"151.611\" y=\"1218.34\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 846.814)\" x=\"151.611\" y=\"846.814\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 475.284)\" x=\"151.611\" y=\"475.284\">6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 103.755)\" x=\"151.611\" y=\"103.755\">8</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1264.18, 1559.48)\" x=\"1264.18\" y=\"1559.48\">Iteration</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">Number of clusters</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip4802)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  237.228,1015.08 239.284,1015.08 241.34,1015.08 243.396,829.314 245.452,829.314 247.508,829.314 249.564,1015.08 251.62,1015.08 253.676,829.314 255.732,1015.08 \n",
       "  257.788,1015.08 259.844,1015.08 261.899,1200.84 263.955,1200.84 266.011,643.549 268.067,643.549 270.123,643.549 272.179,1200.84 274.235,1200.84 276.291,1200.84 \n",
       "  278.347,1200.84 280.403,1200.84 282.459,1200.84 284.515,1200.84 286.571,1200.84 288.627,1200.84 290.683,1015.08 292.739,1200.84 294.795,1200.84 296.851,643.549 \n",
       "  298.907,643.549 300.963,457.784 303.019,643.549 305.075,643.549 307.131,643.549 309.187,643.549 311.243,829.314 313.299,1015.08 315.355,829.314 317.411,1200.84 \n",
       "  319.467,1015.08 321.523,1200.84 323.578,1200.84 325.634,1200.84 327.69,1200.84 329.746,1200.84 331.802,829.314 333.858,829.314 335.914,829.314 337.97,829.314 \n",
       "  340.026,829.314 342.082,829.314 344.138,829.314 346.194,829.314 348.25,829.314 350.306,643.549 352.362,643.549 354.418,457.784 356.474,829.314 358.53,829.314 \n",
       "  360.586,1015.08 362.642,1015.08 364.698,1015.08 366.754,1015.08 368.81,829.314 370.866,829.314 372.922,829.314 374.978,829.314 377.034,1015.08 379.09,1200.84 \n",
       "  381.146,1015.08 383.202,1015.08 385.257,829.314 387.313,829.314 389.369,829.314 391.425,829.314 393.481,1015.08 395.537,1015.08 397.593,1015.08 399.649,1015.08 \n",
       "  401.705,643.549 403.761,829.314 405.817,829.314 407.873,1386.61 409.929,1015.08 411.985,1200.84 414.041,829.314 416.097,643.549 418.153,829.314 420.209,829.314 \n",
       "  422.265,829.314 424.321,829.314 426.377,829.314 428.433,829.314 430.489,829.314 432.545,829.314 434.601,829.314 436.657,829.314 438.713,829.314 440.769,829.314 \n",
       "  442.825,829.314 444.881,457.784 446.936,272.019 448.992,457.784 451.048,457.784 453.104,457.784 455.16,643.549 457.216,829.314 459.272,829.314 461.328,829.314 \n",
       "  463.384,829.314 465.44,643.549 467.496,829.314 469.552,1200.84 471.608,1015.08 473.664,829.314 475.72,829.314 477.776,829.314 479.832,829.314 481.888,829.314 \n",
       "  483.944,457.784 486,457.784 488.056,457.784 490.112,457.784 492.168,457.784 494.224,272.019 496.28,272.019 498.336,272.019 500.392,86.2547 502.448,272.019 \n",
       "  504.504,457.784 506.56,457.784 508.615,457.784 510.671,457.784 512.727,272.019 514.783,272.019 516.839,272.019 518.895,457.784 520.951,457.784 523.007,457.784 \n",
       "  525.063,457.784 527.119,457.784 529.175,457.784 531.231,457.784 533.287,457.784 535.343,457.784 537.399,457.784 539.455,457.784 541.511,272.019 543.567,643.549 \n",
       "  545.623,1015.08 547.679,1015.08 549.735,1015.08 551.791,643.549 553.847,1015.08 555.903,829.314 557.959,643.549 560.015,643.549 562.071,643.549 564.127,643.549 \n",
       "  566.183,643.549 568.239,643.549 570.294,643.549 572.35,457.784 574.406,643.549 576.462,643.549 578.518,643.549 580.574,643.549 582.63,643.549 584.686,829.314 \n",
       "  586.742,643.549 588.798,643.549 590.854,643.549 592.91,643.549 594.966,829.314 597.022,829.314 599.078,829.314 601.134,829.314 603.19,829.314 605.246,829.314 \n",
       "  607.302,829.314 609.358,829.314 611.414,829.314 613.47,829.314 615.526,829.314 617.582,829.314 619.638,457.784 621.694,643.549 623.75,457.784 625.806,457.784 \n",
       "  627.862,457.784 629.918,1015.08 631.973,457.784 634.029,457.784 636.085,457.784 638.141,1015.08 640.197,1015.08 642.253,1015.08 644.309,1015.08 646.365,1015.08 \n",
       "  648.421,1015.08 650.477,1015.08 652.533,1200.84 654.589,1200.84 656.645,1200.84 658.701,1200.84 660.757,1015.08 662.813,829.314 664.869,1015.08 666.925,829.314 \n",
       "  668.981,1015.08 671.037,1015.08 673.093,1015.08 675.149,829.314 677.205,1200.84 679.261,1200.84 681.317,1200.84 683.373,1200.84 685.429,1200.84 687.485,1200.84 \n",
       "  689.541,1200.84 691.597,1386.61 693.652,1386.61 695.708,1200.84 697.764,1015.08 699.82,1200.84 701.876,1200.84 703.932,829.314 705.988,829.314 708.044,457.784 \n",
       "  710.1,643.549 712.156,643.549 714.212,643.549 716.268,643.549 718.324,643.549 720.38,643.549 722.436,643.549 724.492,643.549 726.548,643.549 728.604,643.549 \n",
       "  730.66,643.549 732.716,643.549 734.772,643.549 736.828,1015.08 738.884,1015.08 740.94,1015.08 742.996,829.314 745.052,1015.08 747.108,1200.84 749.164,1200.84 \n",
       "  751.22,1200.84 753.276,1200.84 755.331,829.314 757.387,829.314 759.443,829.314 761.499,643.549 763.555,829.314 765.611,829.314 767.667,829.314 769.723,829.314 \n",
       "  771.779,643.549 773.835,1015.08 775.891,1015.08 777.947,1015.08 780.003,1200.84 782.059,1200.84 784.115,1200.84 786.171,1200.84 788.227,643.549 790.283,643.549 \n",
       "  792.339,1200.84 794.395,1200.84 796.451,1200.84 798.507,829.314 800.563,829.314 802.619,457.784 804.675,829.314 806.731,643.549 808.787,829.314 810.843,829.314 \n",
       "  812.899,1015.08 814.955,1015.08 817.01,1015.08 819.066,1015.08 821.122,1015.08 823.178,1015.08 825.234,1015.08 827.29,1015.08 829.346,1200.84 831.402,1200.84 \n",
       "  833.458,643.549 835.514,643.549 837.57,1200.84 839.626,1015.08 841.682,1200.84 843.738,1200.84 845.794,1200.84 847.85,1015.08 849.906,1200.84 851.962,1015.08 \n",
       "  854.018,1015.08 856.074,1015.08 858.13,1015.08 860.186,1015.08 862.242,1015.08 864.298,1015.08 866.354,1015.08 868.41,1015.08 870.466,1015.08 872.522,1015.08 \n",
       "  874.578,1015.08 876.634,1015.08 878.689,643.549 880.745,829.314 882.801,829.314 884.857,829.314 886.913,829.314 888.969,829.314 891.025,1015.08 893.081,1015.08 \n",
       "  895.137,1015.08 897.193,1015.08 899.249,1015.08 901.305,1015.08 903.361,1015.08 905.417,829.314 907.473,1015.08 909.529,1015.08 911.585,1015.08 913.641,1015.08 \n",
       "  915.697,1015.08 917.753,1015.08 919.809,1015.08 921.865,1015.08 923.921,1015.08 925.977,1015.08 928.033,1015.08 930.089,1015.08 932.145,1015.08 934.201,1015.08 \n",
       "  936.257,1015.08 938.313,1015.08 940.368,1015.08 942.424,1015.08 944.48,1015.08 946.536,1015.08 948.592,1015.08 950.648,829.314 952.704,1015.08 954.76,1015.08 \n",
       "  956.816,643.549 958.872,643.549 960.928,829.314 962.984,457.784 965.04,457.784 967.096,643.549 969.152,643.549 971.208,643.549 973.264,643.549 975.32,829.314 \n",
       "  977.376,643.549 979.432,643.549 981.488,643.549 983.544,272.019 985.6,457.784 987.656,457.784 989.712,457.784 991.768,457.784 993.824,457.784 995.88,643.549 \n",
       "  997.936,643.549 999.992,1200.84 1002.05,1200.84 1004.1,1015.08 1006.16,1200.84 1008.22,1200.84 1010.27,1200.84 1012.33,1200.84 1014.38,1386.61 1016.44,1200.84 \n",
       "  1018.5,643.549 1020.55,643.549 1022.61,457.784 1024.66,829.314 1026.72,1200.84 1028.78,1015.08 1030.83,1200.84 1032.89,1200.84 1034.94,1015.08 1037,1015.08 \n",
       "  1039.05,1015.08 1041.11,1015.08 1043.17,1200.84 1045.22,1200.84 1047.28,643.549 1049.33,829.314 1051.39,829.314 1053.45,1015.08 1055.5,1015.08 1057.56,1015.08 \n",
       "  1059.61,829.314 1061.67,829.314 1063.73,643.549 1065.78,829.314 1067.84,1200.84 1069.89,1015.08 1071.95,1200.84 1074.01,1200.84 1076.06,1200.84 1078.12,1200.84 \n",
       "  1080.17,1015.08 1082.23,1015.08 1084.29,1015.08 1086.34,1015.08 1088.4,1015.08 1090.45,829.314 1092.51,1015.08 1094.57,1015.08 1096.62,1015.08 1098.68,1015.08 \n",
       "  1100.73,829.314 1102.79,1386.61 1104.85,1200.84 1106.9,1200.84 1108.96,1015.08 1111.01,829.314 1113.07,1200.84 1115.13,1200.84 1117.18,1200.84 1119.24,1200.84 \n",
       "  1121.29,1200.84 1123.35,1200.84 1125.41,1200.84 1127.46,1200.84 1129.52,1015.08 1131.57,1200.84 1133.63,1200.84 1135.69,1200.84 1137.74,1015.08 1139.8,1200.84 \n",
       "  1141.85,1200.84 1143.91,1200.84 1145.97,1200.84 1148.02,1200.84 1150.08,1200.84 1152.13,1200.84 1154.19,1200.84 1156.24,1200.84 1158.3,1200.84 1160.36,1200.84 \n",
       "  1162.41,1200.84 1164.47,1200.84 1166.52,1200.84 1168.58,1200.84 1170.64,1200.84 1172.69,1200.84 1174.75,1015.08 1176.8,1200.84 1178.86,1200.84 1180.92,1200.84 \n",
       "  1182.97,1200.84 1185.03,1200.84 1187.08,1200.84 1189.14,1200.84 1191.2,1200.84 1193.25,1200.84 1195.31,1200.84 1197.36,1200.84 1199.42,1200.84 1201.48,1200.84 \n",
       "  1203.53,1200.84 1205.59,1015.08 1207.64,1200.84 1209.7,1200.84 1211.76,1200.84 1213.81,1200.84 1215.87,1200.84 1217.92,1200.84 1219.98,1200.84 1222.04,1200.84 \n",
       "  1224.09,1200.84 1226.15,1200.84 1228.2,1200.84 1230.26,1200.84 1232.32,1015.08 1234.37,1200.84 1236.43,1015.08 1238.48,1015.08 1240.54,1015.08 1242.6,1200.84 \n",
       "  1244.65,1200.84 1246.71,1200.84 1248.76,1015.08 1250.82,829.314 1252.88,1200.84 1254.93,1200.84 1256.99,1015.08 1259.04,1015.08 1261.1,1200.84 1263.16,1200.84 \n",
       "  1265.21,1386.61 1267.27,1386.61 1269.32,1386.61 1271.38,1386.61 1273.44,1386.61 1275.49,1386.61 1277.55,1386.61 1279.6,1386.61 1281.66,1386.61 1283.71,1386.61 \n",
       "  1285.77,1386.61 1287.83,1386.61 1289.88,1015.08 1291.94,1015.08 1293.99,1015.08 1296.05,1015.08 1298.11,1015.08 1300.16,1015.08 1302.22,829.314 1304.27,829.314 \n",
       "  1306.33,829.314 1308.39,829.314 1310.44,1015.08 1312.5,1015.08 1314.55,829.314 1316.61,643.549 1318.67,643.549 1320.72,643.549 1322.78,643.549 1324.83,829.314 \n",
       "  1326.89,829.314 1328.95,1015.08 1331,643.549 1333.06,457.784 1335.11,457.784 1337.17,457.784 1339.23,1015.08 1341.28,1015.08 1343.34,829.314 1345.39,1200.84 \n",
       "  1347.45,1386.61 1349.51,1386.61 1351.56,1200.84 1353.62,1200.84 1355.67,1386.61 1357.73,1200.84 1359.79,1386.61 1361.84,1386.61 1363.9,1386.61 1365.95,1386.61 \n",
       "  1368.01,1386.61 1370.07,1386.61 1372.12,1386.61 1374.18,1386.61 1376.23,1386.61 1378.29,1015.08 1380.35,1200.84 1382.4,1200.84 1384.46,1200.84 1386.51,1386.61 \n",
       "  1388.57,1386.61 1390.63,1386.61 1392.68,1386.61 1394.74,457.784 1396.79,457.784 1398.85,643.549 1400.91,829.314 1402.96,829.314 1405.02,829.314 1407.07,1200.84 \n",
       "  1409.13,1200.84 1411.18,829.314 1413.24,829.314 1415.3,829.314 1417.35,829.314 1419.41,829.314 1421.46,1015.08 1423.52,829.314 1425.58,1200.84 1427.63,1015.08 \n",
       "  1429.69,829.314 1431.74,1015.08 1433.8,1015.08 1435.86,1015.08 1437.91,1015.08 1439.97,1015.08 1442.02,1015.08 1444.08,829.314 1446.14,829.314 1448.19,829.314 \n",
       "  1450.25,829.314 1452.3,829.314 1454.36,829.314 1456.42,829.314 1458.47,829.314 1460.53,829.314 1462.58,829.314 1464.64,829.314 1466.7,643.549 1468.75,643.549 \n",
       "  1470.81,643.549 1472.86,643.549 1474.92,643.549 1476.98,643.549 1479.03,457.784 1481.09,457.784 1483.14,457.784 1485.2,643.549 1487.26,643.549 1489.31,829.314 \n",
       "  1491.37,829.314 1493.42,829.314 1495.48,829.314 1497.54,829.314 1499.59,643.549 1501.65,829.314 1503.7,829.314 1505.76,829.314 1507.82,829.314 1509.87,829.314 \n",
       "  1511.93,829.314 1513.98,829.314 1516.04,829.314 1518.1,829.314 1520.15,829.314 1522.21,829.314 1524.26,829.314 1526.32,829.314 1528.37,457.784 1530.43,457.784 \n",
       "  1532.49,457.784 1534.54,457.784 1536.6,272.019 1538.65,829.314 1540.71,829.314 1542.77,829.314 1544.82,829.314 1546.88,829.314 1548.93,829.314 1550.99,829.314 \n",
       "  1553.05,829.314 1555.1,643.549 1557.16,829.314 1559.21,1200.84 1561.27,643.549 1563.33,829.314 1565.38,829.314 1567.44,829.314 1569.49,643.549 1571.55,1386.61 \n",
       "  1573.61,1200.84 1575.66,1200.84 1577.72,1200.84 1579.77,1200.84 1581.83,1200.84 1583.89,1015.08 1585.94,829.314 1588,829.314 1590.05,829.314 1592.11,1200.84 \n",
       "  1594.17,1200.84 1596.22,1200.84 1598.28,1200.84 1600.33,1200.84 1602.39,1200.84 1604.45,1200.84 1606.5,1200.84 1608.56,1200.84 1610.61,1386.61 1612.67,1386.61 \n",
       "  1614.73,1200.84 1616.78,1386.61 1618.84,1386.61 1620.89,829.314 1622.95,829.314 1625.01,829.314 1627.06,829.314 1629.12,643.549 1631.17,643.549 1633.23,829.314 \n",
       "  1635.29,829.314 1637.34,643.549 1639.4,1015.08 1641.45,829.314 1643.51,829.314 1645.57,643.549 1647.62,829.314 1649.68,829.314 1651.73,829.314 1653.79,1015.08 \n",
       "  1655.84,1015.08 1657.9,457.784 1659.96,457.784 1662.01,457.784 1664.07,457.784 1666.12,829.314 1668.18,643.549 1670.24,829.314 1672.29,643.549 1674.35,643.549 \n",
       "  1676.4,829.314 1678.46,829.314 1680.52,643.549 1682.57,829.314 1684.63,829.314 1686.68,643.549 1688.74,829.314 1690.8,643.549 1692.85,829.314 1694.91,829.314 \n",
       "  1696.96,829.314 1699.02,829.314 1701.08,829.314 1703.13,1386.61 1705.19,1386.61 1707.24,1015.08 1709.3,1015.08 1711.36,1015.08 1713.41,1015.08 1715.47,643.549 \n",
       "  1717.52,643.549 1719.58,643.549 1721.64,1015.08 1723.69,1015.08 1725.75,643.549 1727.8,829.314 1729.86,829.314 1731.92,829.314 1733.97,829.314 1736.03,1015.08 \n",
       "  1738.08,1015.08 1740.14,1015.08 1742.2,1015.08 1744.25,829.314 1746.31,643.549 1748.36,829.314 1750.42,643.549 1752.48,643.549 1754.53,643.549 1756.59,643.549 \n",
       "  1758.64,457.784 1760.7,829.314 1762.76,829.314 1764.81,829.314 1766.87,829.314 1768.92,643.549 1770.98,643.549 1773.03,643.549 1775.09,643.549 1777.15,829.314 \n",
       "  1779.2,1386.61 1781.26,1200.84 1783.31,1386.61 1785.37,643.549 1787.43,829.314 1789.48,1015.08 1791.54,829.314 1793.59,829.314 1795.65,829.314 1797.71,829.314 \n",
       "  1799.76,643.549 1801.82,643.549 1803.87,829.314 1805.93,829.314 1807.99,829.314 1810.04,829.314 1812.1,1015.08 1814.15,643.549 1816.21,643.549 1818.27,643.549 \n",
       "  1820.32,643.549 1822.38,643.549 1824.43,643.549 1826.49,643.549 1828.55,643.549 1830.6,457.784 1832.66,643.549 1834.71,457.784 1836.77,643.549 1838.83,643.549 \n",
       "  1840.88,643.549 1842.94,643.549 1844.99,643.549 1847.05,1015.08 1849.11,1200.84 1851.16,1015.08 1853.22,1015.08 1855.27,1015.08 1857.33,1015.08 1859.39,1015.08 \n",
       "  1861.44,829.314 1863.5,829.314 1865.55,829.314 1867.61,1015.08 1869.67,272.019 1871.72,272.019 1873.78,272.019 1875.83,457.784 1877.89,643.549 1879.95,829.314 \n",
       "  1882,1015.08 1884.06,1015.08 1886.11,829.314 1888.17,1200.84 1890.23,1200.84 1892.28,1200.84 1894.34,1200.84 1896.39,1200.84 1898.45,1200.84 1900.5,1015.08 \n",
       "  1902.56,1200.84 1904.62,1015.08 1906.67,1015.08 1908.73,1386.61 1910.78,1386.61 1912.84,1200.84 1914.9,1015.08 1916.95,1386.61 1919.01,1386.61 1921.06,1200.84 \n",
       "  1923.12,1200.84 1925.18,457.784 1927.23,272.019 1929.29,457.784 1931.34,272.019 1933.4,457.784 1935.46,1015.08 1937.51,1200.84 1939.57,1015.08 1941.62,1200.84 \n",
       "  1943.68,1386.61 1945.74,1386.61 1947.79,1386.61 1949.85,1200.84 1951.9,1200.84 1953.96,1015.08 1956.02,1200.84 1958.07,1200.84 1960.13,1015.08 1962.18,1015.08 \n",
       "  1964.24,1015.08 1966.3,1015.08 1968.35,1015.08 1970.41,1015.08 1972.46,1015.08 1974.52,1015.08 1976.58,1015.08 1978.63,1015.08 1980.69,1015.08 1982.74,829.314 \n",
       "  1984.8,829.314 1986.86,1015.08 1988.91,829.314 1990.97,829.314 1993.02,829.314 1995.08,1015.08 1997.14,829.314 1999.19,829.314 2001.25,1015.08 2003.3,1015.08 \n",
       "  2005.36,829.314 2007.42,1015.08 2009.47,829.314 2011.53,829.314 2013.58,1015.08 2015.64,1015.08 2017.7,1015.08 2019.75,1015.08 2021.81,1015.08 2023.86,1015.08 \n",
       "  2025.92,1015.08 2027.97,829.314 2030.03,829.314 2032.09,643.549 2034.14,1015.08 2036.2,1015.08 2038.25,1015.08 2040.31,1015.08 2042.37,1015.08 2044.42,1015.08 \n",
       "  2046.48,1015.08 2048.53,829.314 2050.59,829.314 2052.65,829.314 2054.7,829.314 2056.76,829.314 2058.81,829.314 2060.87,829.314 2062.93,829.314 2064.98,829.314 \n",
       "  2067.04,829.314 2069.09,829.314 2071.15,829.314 2073.21,643.549 2075.26,829.314 2077.32,829.314 2079.37,457.784 2081.43,457.784 2083.49,457.784 2085.54,457.784 \n",
       "  2087.6,457.784 2089.65,457.784 2091.71,272.019 2093.77,457.784 2095.82,457.784 2097.88,1015.08 2099.93,1015.08 2101.99,1015.08 2104.05,1015.08 2106.1,1200.84 \n",
       "  2108.16,1200.84 2110.21,1200.84 2112.27,1200.84 2114.33,1015.08 2116.38,1200.84 2118.44,1015.08 2120.49,829.314 2122.55,1015.08 2124.61,829.314 2126.66,1015.08 \n",
       "  2128.72,1015.08 2130.77,1015.08 2132.83,1015.08 2134.89,1015.08 2136.94,1015.08 2139,1015.08 2141.05,1015.08 2143.11,829.314 2145.16,829.314 2147.22,829.314 \n",
       "  2149.28,829.314 2151.33,643.549 2153.39,1015.08 2155.44,1015.08 2157.5,1015.08 2159.56,1015.08 2161.61,1015.08 2163.67,1015.08 2165.72,1015.08 2167.78,1200.84 \n",
       "  2169.84,1015.08 2171.89,643.549 2173.95,1200.84 2176,1200.84 2178.06,1200.84 2180.12,1200.84 2182.17,1200.84 2184.23,1200.84 2186.28,1200.84 2188.34,1200.84 \n",
       "  2190.4,1015.08 2192.45,1200.84 2194.51,1200.84 2196.56,1200.84 2198.62,1200.84 2200.68,1015.08 2202.73,1200.84 2204.79,1200.84 2206.84,1200.84 2208.9,1200.84 \n",
       "  2210.96,1200.84 2213.01,1200.84 2215.07,1200.84 2217.12,1200.84 2219.18,1200.84 2221.24,1200.84 2223.29,1200.84 2225.35,1200.84 2227.4,1200.84 2229.46,1200.84 \n",
       "  2231.52,1200.84 2233.57,1200.84 2235.63,1015.08 2237.68,1015.08 2239.74,1015.08 2241.8,1015.08 2243.85,1015.08 2245.91,1015.08 2247.96,1015.08 2250.02,1015.08 \n",
       "  2252.08,1200.84 2254.13,1200.84 2256.19,1015.08 2258.24,1015.08 2260.3,829.314 2262.36,829.314 2264.41,829.314 2266.47,829.314 2268.52,1200.84 2270.58,1200.84 \n",
       "  2272.63,1015.08 2274.69,643.549 2276.75,643.549 2278.8,643.549 2280.86,643.549 2282.91,829.314 2284.97,829.314 2287.03,829.314 2289.08,829.314 2291.14,1200.84 \n",
       "  \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip4800)\" d=\"\n",
       "M1874.89 251.724 L2280.76 251.724 L2280.76 130.764 L1874.89 130.764  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1874.89,251.724 2280.76,251.724 2280.76,130.764 1874.89,130.764 1874.89,251.724 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4800)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1898.89,191.244 2042.89,191.244 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4800)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2066.89, 208.744)\" x=\"2066.89\" y=\"208.744\">Chain 1</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract the number of clusters for each sample of the Markov chain.\n",
    "k = map(t -> length(unique(chain[:z].value[t,:,:])), 1:iterations);\n",
    "\n",
    "# Visualize the number of clusters.\n",
    "plot(k, xlabel = \"Iteration\", ylabel = \"Number of clusters\", label = \"Chain 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we visualize the histogram of the number of clusters sampled from our posterior, we observe that the model seems to prefer 3 clusters, which is the true number of clusters. Note that the number of clusters in a Dirichlet process mixture model is not limited a priori and will grow to infinity with probability one. However, if conditioned on data the posterior will concentrate on a finite number of clusters enforcing the resulting model to have a finite amount of clusters. It is, however, not given that the posterior of a Dirichlet process Gaussian mixture model converges to the true number of clusters, given that data comes from a finite mixture model. See Jeffrey Miller and Matthew Harrison: [A simple example of Dirichlet process mixture inconsitency for the number of components](https://arxiv.org/pdf/1301.2708.pdf) for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip5200\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5200)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5201\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5200)\" d=\"\n",
       "M167.279 1425.62 L2352.76 1425.62 L2352.76 47.2441 L167.279 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5202\">\n",
       "    <rect x=\"167\" y=\"47\" width=\"2186\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  546.826,1425.62 546.826,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1065.51,1425.62 1065.51,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1584.2,1425.62 1584.2,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2102.88,1425.62 2102.88,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  167.279,1386.61 2352.76,1386.61 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  167.279,1128.6 2352.76,1128.6 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  167.279,870.595 2352.76,870.595 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  167.279,612.588 2352.76,612.588 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  167.279,354.582 2352.76,354.582 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  167.279,96.575 2352.76,96.575 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,1425.62 167.279,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  546.826,1425.62 546.826,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1065.51,1425.62 1065.51,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1584.2,1425.62 1584.2,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2102.88,1425.62 2102.88,1409.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,1386.61 193.505,1386.61 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,1128.6 193.505,1128.6 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,870.595 193.505,870.595 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,612.588 193.505,612.588 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,354.582 193.505,354.582 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  167.279,96.575 193.505,96.575 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 546.826, 1479.62)\" x=\"546.826\" y=\"1479.62\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1065.51, 1479.62)\" x=\"1065.51\" y=\"1479.62\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1584.2, 1479.62)\" x=\"1584.2\" y=\"1479.62\">6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2102.88, 1479.62)\" x=\"2102.88\" y=\"1479.62\">8</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 143.279, 1404.11)\" x=\"143.279\" y=\"1404.11\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 143.279, 1146.1)\" x=\"143.279\" y=\"1146.1\">50</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 143.279, 888.095)\" x=\"143.279\" y=\"888.095\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 143.279, 630.088)\" x=\"143.279\" y=\"630.088\">150</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 143.279, 372.082)\" x=\"143.279\" y=\"372.082\">200</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 143.279, 114.075)\" x=\"143.279\" y=\"114.075\">250</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1260.02, 1559.48)\" x=\"1260.02\" y=\"1559.48\">Number of clusters</text>\n",
       "</g>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M287.484 1133.76 L287.484 1386.61 L417.155 1386.61 L417.155 1133.76 L287.484 1133.76 L287.484 1133.76  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  287.484,1133.76 287.484,1386.61 417.155,1386.61 417.155,1133.76 287.484,1133.76 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M417.155 1386.61 L417.155 1386.61 L546.826 1386.61 L546.826 1386.61 L417.155 1386.61 L417.155 1386.61  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  417.155,1386.61 417.155,1386.61 546.826,1386.61 417.155,1386.61 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M546.826 261.699 L546.826 1386.61 L676.498 1386.61 L676.498 261.699 L546.826 261.699 L546.826 261.699  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  546.826,261.699 546.826,1386.61 676.498,1386.61 676.498,261.699 546.826,261.699 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M676.498 1386.61 L676.498 1386.61 L806.169 1386.61 L806.169 1386.61 L676.498 1386.61 L676.498 1386.61  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  676.498,1386.61 676.498,1386.61 806.169,1386.61 676.498,1386.61 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M806.169 91.4148 L806.169 1386.61 L935.84 1386.61 L935.84 91.4148 L806.169 91.4148 L806.169 91.4148  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  806.169,91.4148 806.169,1386.61 935.84,1386.61 935.84,91.4148 806.169,91.4148 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M935.84 1386.61 L935.84 1386.61 L1065.51 1386.61 L1065.51 1386.61 L935.84 1386.61 L935.84 1386.61  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  935.84,1386.61 935.84,1386.61 1065.51,1386.61 935.84,1386.61 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1065.51 86.2547 L1065.51 1386.61 L1195.18 1386.61 L1195.18 86.2547 L1065.51 86.2547 L1065.51 86.2547  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1065.51,86.2547 1065.51,1386.61 1195.18,1386.61 1195.18,86.2547 1065.51,86.2547 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1195.18 1386.61 L1195.18 1386.61 L1324.85 1386.61 L1324.85 1386.61 L1195.18 1386.61 L1195.18 1386.61  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1195.18,1386.61 1195.18,1386.61 1324.85,1386.61 1195.18,1386.61 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1324.85 679.67 L1324.85 1386.61 L1454.52 1386.61 L1454.52 679.67 L1324.85 679.67 L1324.85 679.67  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1324.85,679.67 1324.85,1386.61 1454.52,1386.61 1454.52,679.67 1324.85,679.67 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1454.52 1386.61 L1454.52 1386.61 L1584.2 1386.61 L1584.2 1386.61 L1454.52 1386.61 L1454.52 1386.61  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1454.52,1386.61 1454.52,1386.61 1584.2,1386.61 1454.52,1386.61 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1584.2 999.598 L1584.2 1386.61 L1713.87 1386.61 L1713.87 999.598 L1584.2 999.598 L1584.2 999.598  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1584.2,999.598 1584.2,1386.61 1713.87,1386.61 1713.87,999.598 1584.2,999.598 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1713.87 1386.61 L1713.87 1386.61 L1843.54 1386.61 L1843.54 1386.61 L1713.87 1386.61 L1713.87 1386.61  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1713.87,1386.61 1713.87,1386.61 1843.54,1386.61 1713.87,1386.61 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1843.54 1298.89 L1843.54 1386.61 L1973.21 1386.61 L1973.21 1298.89 L1843.54 1298.89 L1843.54 1298.89  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1843.54,1298.89 1843.54,1386.61 1973.21,1386.61 1973.21,1298.89 1843.54,1298.89 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M1973.21 1386.61 L1973.21 1386.61 L2102.88 1386.61 L2102.88 1386.61 L1973.21 1386.61 L1973.21 1386.61  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1973.21,1386.61 1973.21,1386.61 2102.88,1386.61 1973.21,1386.61 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip5202)\" d=\"\n",
       "M2102.88 1381.45 L2102.88 1386.61 L2232.55 1386.61 L2232.55 1381.45 L2102.88 1381.45 L2102.88 1381.45  Z\n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5202)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2102.88,1381.45 2102.88,1386.61 2232.55,1386.61 2232.55,1381.45 2102.88,1381.45 \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "histogram(k, xlabel = \"Number of clusters\", legend = false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One issue with the Chinese restaurant process construction is that the number of latent parameters we need to sample scales with the number of observations. It may be desirable to use alternative constructions in certain cases. Alternative methods of constructing a Dirichlet process can be employed via the following representations:\n",
    "\n",
    "Size-Biased Sampling Process\n",
    "\n",
    "$$\n",
    "j_k \\sim Beta(1, \\alpha) * surplus\n",
    "$$\n",
    "\n",
    "Stick-Breaking Process\n",
    "$$\n",
    "v_k \\sim Beta(1, \\alpha)\n",
    "$$\n",
    "\n",
    "Chinese Restaurant Process\n",
    "$$\n",
    "p(z_n = k | z_{1:n-1}) \\propto \\begin{cases} \n",
    "        \\frac{m_k}{n-1+\\alpha}, \\text{ if } m_k > 0\\\\\\\n",
    "        \\frac{\\alpha}{n-1+\\alpha}\n",
    "    \\end{cases}\n",
    "$$\n",
    "\n",
    "For more details see [this article](https://www.stats.ox.ac.uk/~teh/research/npbayes/Teh2010a.pdf)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.0.4",
   "language": "julia",
   "name": "julia-1.0"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.4.0"
  },
  "nteract": {
   "version": "0.23.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}