Lecture5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zhHRr3VVOMFm"
   },
   "source": [
    "# Classical Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cWJD84-9OMFo"
   },
   "source": [
    "```{admonition} Hugh Cartwright\n",
    ":class: tip\n",
    "The tools of science are changing; artificial intelligence has spread to the laboratory.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vN4ra0MROMFp"
   },
   "source": [
    "<iframe class=\"speakerdeck-iframe\" frameborder=\"0\" src=\"https://speakerdeck.com/player/b098c15f50ce4a468a1c5eecd6de0f96\" title=\"Machine Learning for Materials (Lecture 5)\" allowfullscreen=\"true\" style=\"border: 0px; background-clip: padding-box; background-color: rgba(0, 0, 0, 0.1); margin: 0px; padding: 0px; border-radius: 6px; box-shadow: rgba(0, 0, 0, 0.2) 0px 5px 40px; width: 100%; height: auto; aspect-ratio: 560 / 420;\" data-ratio=\"1.3333333333333333\"></iframe>\n",
    "\n",
    "[Lecture slides](https://speakerdeck.com/aronwalsh/mlformaterials-lecture5-classical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IPrAgT4POMFp"
   },
   "source": [
    "## 🎲 Metal or insulator?\n",
    "\n",
    "Some decisions in life are difficult to make. We hope our experience informs a choice that is better than a random guess. The same is true for machine learning models.\n",
    "\n",
    "There are many situations where we want to classify materials according to their properties. One fundamental characteristic is whether a material is a metal or insulator. For this exercise, we can refer to these as class `0` and class `1` materials, respectively. \n",
    "\n",
    "Cu is clearly `0`, and MgO is `1`, but what about Tl<sub>2</sub>O<sub>3</sub> or Ni<sub>2</sub>Zn<sub>4</sub>?\n",
    "\n",
    "### Theoretical background\n",
    "\n",
    "Metals are characterised by their free electrons that facilitate the flow of electric current. This arises from a partially filled conduction band, allowing electrons to move easily when subjected to an electric field.\n",
    "\n",
    "Insulators are characterised by an occupied valence band and empty conduction band, impeding the flow of current. The absence of charge carriers in insulators hinders electrical conductivity, making them effective insulators of electricity. Understanding these fundamental differences between metals and insulators is crucial for designing and optimising electronic devices.\n",
    "\n",
    "In this practical, we can use the electronic band gap of a material as a simple descriptor of whether it is a metal (E$_g$ = 0) or an insulator (E$_g$ > 0).\n",
    "\n",
    "$$\n",
    "E_g = E^{conduction-band}_{minimum} - E^{valence-band}_{maximum}\n",
    "$$\n",
    "\n",
    "This classification is coarse as we are ignoring the intermediate regime of semiconductors and more exotic behaviour such as superconductivity.\n",
    "\n",
    "![image](./images/5_bands.png)\n",
    "\n",
    "## $k$-means clustering\n",
    "\n",
    "We'll start by generating some synthetic data for materials with their class labels. To make the analyisis faster and more illustrative, we perform a dimensionality reduction from a 10 D to 2 D feature space, and then cluster the data using $k$-means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CLEjvAiAOMFp",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Installation of libraries\n",
    "!pip install -U elementembeddings --quiet\n",
    "!pip install matminer --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import of modules\n",
    "import numpy as np  # Numerical operations\n",
    "import pandas as pd  # Data manipulation with DataFrames\n",
    "import matplotlib.pyplot as plt  # Plotting\n",
    "import seaborn as sns  # Statistical visualisation\n",
    "from sklearn.decomposition import PCA  # Principal component analysis\n",
    "from sklearn.cluster import KMeans  # k-means clustering\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix  # Metrics for model evaluation\n",
    "from sklearn.tree import DecisionTreeClassifier  # Decision tree classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Colab error solution</summary>\n",
    "If running the import module cell fails with an \"AttributeError\", click `Runtime` -> `Restart Session` and then simply rerun the cell.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pay attention to each step in the process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 622
    },
    "id": "ECkEUb8UOMFq",
    "outputId": "e97ee0bc-808b-4eb8-8950-66c533b68cbe",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Step 0: Set the number of clusters\n",
    "n_clusters = 0\n",
    "\n",
    "# Step 1: Generating sample data\n",
    "np.random.seed(42)\n",
    "num_materials = 200\n",
    "num_features = 10\n",
    "data = np.random.rand(num_materials, num_features)\n",
    "labels = np.random.randint(0, 2, num_materials)\n",
    "\n",
    "# Step 2: Reduce dimensions to 2 using principal component analysis (PCA)\n",
    "pca = PCA(n_components=2)\n",
    "reduced_data = pca.fit_transform(data)\n",
    "\n",
    "# Step 3: Cluster the data using k-means\n",
    "kmeans = KMeans(n_clusters=n_clusters, random_state=42)\n",
    "predicted_labels = kmeans.fit_predict(reduced_data)\n",
    "\n",
    "# Step 4: Create a plot to visualise the clusters and known labels\n",
    "plt.figure(figsize=(5, 4))\n",
    "\n",
    "# Plot the materials labeled as metal (label=1)\n",
    "plt.scatter(reduced_data[labels == 1, 0], reduced_data[labels == 1, 1], c='blue', label='Metal')\n",
    "\n",
    "# Plot the materials labeled as insulator (label=0)\n",
    "plt.scatter(reduced_data[labels == 0, 0], reduced_data[labels == 0, 1], c='red', label='Insulator')\n",
    "\n",
    "# Plot the cluster centers\n",
    "plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], c='green', s=100, label='Cluster centers')\n",
    "\n",
    "# Draw cluster boundaries\n",
    "h = 0.02  # step size for the meshgrid\n",
    "x_min, x_max = reduced_data[:, 0].min() - 1, reduced_data[:, 0].max() + 1\n",
    "y_min, y_max = reduced_data[:, 1].min() - 1, reduced_data[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.contourf(xx, yy, Z, alpha=0.5, cmap='viridis')\n",
    "\n",
    "plt.xlabel('Principal Component 1')\n",
    "plt.ylabel('Principal Component 2')\n",
    "plt.title('$k$-means clustering of artificial materials')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iyjuZJYmOMFq"
   },
   "source": [
    "<details>\n",
    "<summary> Code hint </summary>\n",
    "The algorithm fails for 0 clusters. \n",
    "Increase the value of `n_clusters` and look at the behaviour.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RWChaS_mOMFr"
   },
   "source": [
    "The cluster centres are shown by green dots. It doesn't do a great job, as we just generated this \"materials data\" from random numbers. There are no correlations for the algorithms to exploit. Nonetheless, this type of \"failed experiment\" is common in real research.\n",
    "\n",
    "Since we know the labels here, we can use quantify how bad the model is by calculating the classification accuracy. Is it better than flipping a coin? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XyShX2J-OMFr",
    "outputId": "c3c1425f-4354-4080-c42d-f025bedca416",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Step 5: Quantify classification accuracy\n",
    "accuracy = accuracy_score(labels, predicted_labels)\n",
    "conf_matrix = confusion_matrix(labels, predicted_labels)\n",
    "\n",
    "print(\"Accuracy:\", accuracy)\n",
    "print(\"\\nConfusion matrix:\")\n",
    "print(conf_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uSM_A4-ZOMFr"
   },
   "source": [
    "## Decision tree classifier\n",
    "\n",
    "Let's see if we can do better using a dedicated classifier. We will now train a decision tree classifier to tackle the same classification problem and visualise the decision boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 564
    },
    "id": "ZKbtozXuOMFr",
    "outputId": "1e23e4bb-e043-4c37-b74a-413a5e002bc1",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Step 0: Set the depth of the decision tree\n",
    "max_tree_depth = 0\n",
    "\n",
    "# Step 1: Train a decision tree classifier\n",
    "def train_decision_tree(depth, reduced_data, labels):\n",
    "    tree_classifier = DecisionTreeClassifier(max_depth=depth, random_state=42)\n",
    "    tree_classifier.fit(reduced_data, labels)\n",
    "    return tree_classifier\n",
    "\n",
    "tree_classifier = train_decision_tree(max_tree_depth, reduced_data, labels)\n",
    "predicted_labels = tree_classifier.predict(reduced_data)\n",
    "\n",
    "# Step 2: Create a plot to visualise the decision boundary of the decision tree\n",
    "plt.figure(figsize=(5, 4))\n",
    "\n",
    "# Plot the materials labeled as metal (label=1)\n",
    "plt.scatter(reduced_data[labels == 1, 0], reduced_data[labels == 1, 1], c='blue', label='Metal')\n",
    "\n",
    "# Plot the materials labeled as insulator (label=0)\n",
    "plt.scatter(reduced_data[labels == 0, 0], reduced_data[labels == 0, 1], c='red', label='Insulator')\n",
    "\n",
    "# Plot the decision boundary of the decision tree classifier\n",
    "h = 0.02  # step size for the meshgrid\n",
    "x_min, x_max = reduced_data[:, 0].min() - 1, reduced_data[:, 0].max() + 1\n",
    "y_min, y_max = reduced_data[:, 1].min() - 1, reduced_data[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "Z = tree_classifier.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.contourf(xx, yy, Z, alpha=0.5, cmap='viridis')\n",
    "\n",
    "plt.xlabel('Principal Component 1')\n",
    "plt.ylabel('Principal Component 2')\n",
    "plt.title(f'Decision tree (max depth={max_tree_depth}) for artificial materials')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SW0VbC_4OMFr"
   },
   "source": [
    "<details>\n",
    "<summary> Code hint </summary>\n",
    "With no nodes, you have made an indecisive tree 🥁.\n",
    "    \n",
    "Increase the value of `max_tree_depth` and look at the behaviour.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sOqtQymnOMFs"
   },
   "source": [
    "There should be more structure in the decision boundary due to the more complex model.\n",
    "\n",
    "$k$-means clustering provides a simple way to group materials based on similarity, yielding a clear linear decision boundary. This method works well when our data showcases distinct clusters. However, when facing more intricate and overlapping distributions, this will not capture the complexity of the underlying patterns.\n",
    "\n",
    "On the other hand, the decision tree classifier does better in handling non-linear separations. It constructs a boundary based on different feature thresholds, enabling it to capture fine-grained patterns. As always in ML, we must balance the trade-offs between simplicity and accuracy.\n",
    "\n",
    "Is the decision tree more accurate? Let's see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PucrDphBOMFs",
    "outputId": "09ad3aa5-1ee9-4597-e2ca-c38b5d85057b",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Step 3: Quantify classification accuracy\n",
    "accuracy = accuracy_score(labels, predicted_labels)\n",
    "conf_matrix = confusion_matrix(labels, predicted_labels)\n",
    "\n",
    "print(\"Accuracy:\", accuracy)\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(conf_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SNQRw2RhOMFs"
   },
   "source": [
    "If you chose a large tree depth, then the decision tree will approach a perfect accuracy of 1.0. It does this by memorising the training data but is unlikely to generalise well to new (unseen) data, i.e. overfitting. In contrast, the accuracy of $k$-means clustering is lower because it is an unsupervised algorithm designed for clustering, not classification. Its performance depends on the data structure and the presence of distinct clusters in that feature space.\n",
    "\n",
    "To obtain reliable results and a robust model, it is essential to split the data into training and testing sets, perform validation, and use other evaluation metrics to assess the model performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yAJreGhfOMFs",
    "tags": []
   },
   "source": [
    "## Real materials\n",
    "\n",
    "We can save time by using a pre-built dataset. We will return to [matminer](https://hackingmaterials.lbl.gov/matminer) that we used before and load `matbench_expt_is_metal`.\n",
    "\n",
    "### Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "import matminer\n",
    "from matminer.datasets.dataset_retrieval import load_dataset\n",
    "\n",
    "# Use matminer to download the dataset\n",
    "df = load_dataset('matbench_expt_is_metal')\n",
    "print(f'The full dataset contains {df.shape[0]} entries. \\n')\n",
    "\n",
    "print('The DataFrame is shown below:')\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sXq9bXwGOMFs"
   },
   "source": [
    "<details>\n",
    "<summary> Code hint </summary>\n",
    "To load a different dataset, you simply change the name in 'load_dataset()'.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y76d3NLhZADO"
   },
   "source": [
    "### Materials featurisation\n",
    "\n",
    "Revisiting concepts from Notebooks 3 and 4, featurising the chemical compositions is necessary to create a useful set of input vectors. This allows the presence (or absence) of an element (or element combination) to act as a feature that the classifier takes account for.\n",
    "\n",
    "We will use [ElementEmbeddings](https://wmd-group.github.io/ElementEmbeddings/0.4/) again to featurise the `composition` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 559
    },
    "id": "sTJg5-4yY9au",
    "outputId": "18f140fe-b6c1-41dc-b1d2-3adfc6c40e73",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Featurise the compositions \n",
    "from elementembeddings.composition import composition_featuriser\n",
    "onehot_df = composition_featuriser(df[\"composition\"], embedding=\"atomic\", stats=[\"sum\"])\n",
    "\n",
    "# Change the is_metal column to Boolean values\n",
    "onehot_df['is_metal'] = df['is_metal'].astype(int)\n",
    "onehot_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a DataFrame that is suitable for our clustering task!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 🚨 Exercise 5: Metallicity\n",
    "\n",
    "```{admonition} Coding exercises\n",
    ":class: note\n",
    "The exercises are designed to apply what you have learned with room for creativity. It is fine to discuss solutions with your classmates, but the actual code should not be directly copied.\n",
    "\n",
    "The completed notebooks are to be submitted at the end of class, but you can revist later, experiment with the code, and follow the further reading suggestions.\n",
    "```\n",
    "\n",
    "### Your details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Insert your values\n",
    "Name = \"No Name\" # Replace with your name\n",
    "CID = 123446 # Replace with your College ID (as a numeric value with no leading 0s)\n",
    "\n",
    "# Set a random seed using the CID value\n",
    "CID = int(CID)\n",
    "np.random.seed(CID)\n",
    "\n",
    "# Print the message\n",
    "print(\"This is the work of \" + Name + \" [CID: \" + str(CID) + \"]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4WAC3QJYOMFs",
    "tags": []
   },
   "source": [
    "### Tasks\n",
    "\n",
    "You will now apply the classification analysis tested above to real materials data. Remember that we have already defined the `onehot_df` DataFrame that contains materials features and the binary class (`is_metal`). You have one task to complete:\n",
    "\n",
    "1. Perform $k$-means clustering of the `matbench_expt_is_metal` dataset. The starting point is to extract appropriate x and Y values from `onehot_df`.\n",
    "\n",
    "```python \n",
    "# Process training data\n",
    "cols_to_drop = ['is_metal', 'formula']\n",
    "feature_cols = [col for col in list(onehot_df.columns) if col not in cols_to_drop]\n",
    "data = onehot_df[feature_cols].values\n",
    "labels = onehot_df['is_metal']\n",
    "```\n",
    "Do the resulting clusters map well onto the metal-insulator classes?\n",
    "\n",
    "*Self-study (optional)*  \n",
    "\n",
    "2. Perform decision tree classification on the dataset using cross-validation to make a robust model. Based on the performance, is it a useful model?\n",
    "  \n",
    "3. Predict if a new crystal that I discovered is metallic. Its composition is AlGaN$_2$. This will involve creating the feature vector for AlGaN$_2$ and then using your model predictively, e.g. `model.predict(AlGaN2)`. In reality, it should be an insulator (semiconductor) as it a mixture of GaN and AlN.\n",
    "\n",
    "<details>\n",
    "<summary> Task hint </summary>\n",
    "For task 4, you can featurise a new composition using a command such as `new_material = composition_featuriser([\"AlGaN2\"], embedding=\"atomic\", stats=[\"sum\"])`\n",
    "</details>\n",
    "\n",
    "```{admonition} Submission\n",
    ":class: note\n",
    "When your notebook is complete, click on the download icon on the top right, select `.pdf`, save the file and upload it to MyDepartment. If you are using Google Colab, you have to print to pdf.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Code block \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Comment block \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Code block \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Comment block \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 🌊 Dive deeper\n",
    "\n",
    "* _Level 1:_ Tackle Chapter 6 on Linear Two-Class Classification in [Machine Learning Refined](https://github.com/jermwatt/machine_learning_refined#what-is-new-in-the-second-edition).\n",
    "\n",
    "* _Level 2:_ Play [metal detection](http://palestrina.northwestern.edu/metal-detection/). Note, the website can be a little temperamental. \n",
    "\n",
    "* _Level 3:_ Dig deeper into the options for definitions decision trees and ensemble models in [scikit-learn](https://scikit-learn.org/stable/modules/tree.html)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}