minor

dimensionality-reduction/pca_angle.ipynb

@@ -0,0 +1,141 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Angles Between Two Vector Spaces\n",
+    "\n",
+    "**Setup**\n",
+    "\n",
+    "* Have big vector space $V \\in \\mathbb{R}^M$\n",
+    "* Have two subspaces $A \\in \\mathbb{R}^N$ and $B \\in \\mathbb{R}^L$\n",
+    "* The subspaces are given by orthonormal bases $\\{\\vec{a}_i\\}$ and $\\{\\vec{b}_j\\}$\n",
+    "\n",
+    "**Task**\n",
+    "\n",
+    "Find angle between the vector spaces\n",
+    "\n",
+    "**Solution**\n",
+    "\n",
+    " 1. Find an arbitrary vector in each vector space.\n",
+    "\n",
+    "$$\\vec{p}_A(\\vec{\\alpha}) = \\sum_i \\alpha_i\\vec{a}_i$$\n",
+    "$$\\vec{p}_B(\\vec{\\beta}) = \\sum_i \\beta_i \\vec{b}_i$$\n",
+    "\n",
+    " 2. Impose the constraint that the prefactors are normalized\n",
+    "\n",
+    "$$\\sum_i \\alpha_i^2 = 1$$ \n",
+    "$$\\sum_i \\beta_i^2 = 1$$\n",
+    "\n",
+    " 3. Define the cosine between the two vectors as their dot product\n",
+    "$$C = \\cos_{AB} = \\vec{p}_A \\cdot \\vec{p}_B = \\sum_{ij} M_{ij} \\alpha_i \\beta_j$$\n",
+    " Where $M_{ij} = \\vec{a}_i \\cdot \\vec{b}_j$\n",
+    " 4. Use Lagrange multipliers to search for extrema of $\\cos_{AB}$ subject to the normalization constraints.\n",
+    " \n",
+    "$$G = \\langle \\alpha | M | \\beta \\rangle + u (\\alpha^2 - 1) + v (\\beta^2 - 1)$$\n",
+    "\n",
+    "$$\\partial_\\vec{\\alpha} G = M \\vec{\\beta} + 2u\\vec{\\alpha} = 0$$\n",
+    "$$\\partial_\\vec{\\beta} G = M^T \\vec{\\alpha} + 2v\\vec{\\beta} = 0$$\n",
+    "\n",
+    "$$u = v = -\\frac{C}{2}$$\n",
+    "$$MM^T \\vec{\\alpha} = C^2 \\vec{\\alpha}$$\n",
+    "$$M^TM \\vec{\\beta} = C^2 \\vec{\\beta}$$\n",
+    "\n",
+    "**Result**\n",
+    "$$C^2 = \\cos^2_{AB} = \\mathrm{eval}(MM^T) = \\mathrm{eval}(M^TM)$$\n",
+    "$$\\vec{\\alpha} = \\mathrm{evec}(MM^T)$$\n",
+    "$$\\vec{\\beta} = \\mathrm{evec}(M^TM)$$\n",
+    "\n",
+    "**Conclusion**\n",
+    "* The relationship between two subspaces can be described by a set of angles.\n",
+    "* The number of angles is given by the size of the smaller of the two subsets. \n",
+    "* For each dimension of the smaller subset that is contained in the larger subset, there will be an angle of size zero\n",
+    "* For each dimension not fully contained in the larger set, there will be an angle determining the largest angle that dimension could form with the larger subset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 64,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Orthonormality error 2.220446049250313e-16\n",
+      "Orthonormality error 0.0\n",
+      "[0. 0.]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "def orthonormal(basis):\n",
+    "    B = np.array([b / np.linalg.norm(b) for b in basis])\n",
+    "    print(\"Orthonormality error\", np.linalg.norm(B.dot(B.T) - np.eye(len(B))))\n",
+    "    return B\n",
+    "\n",
+    "def angle_eval(b1, b2):\n",
+    "    if len(b1) > len(b2):\n",
+    "        return angle_eval(b2, b1)\n",
+    "    M = b1.dot(b2.T)\n",
+    "    return np.rad2deg(np.arccos(np.sqrt(np.linalg.eig(M.dot(M.T))[0])))\n",
+    "    #return np.linalg.eig(M.dot(M.T))[0]\n",
+    "\n",
+    "#Test1\n",
+    "B1 = orthonormal([[1,0,0,0], [0,0,0,1], [0,1,1,0]])\n",
+    "B2 = orthonormal([[1,0,0,0], [0,0,0,1]])\n",
+    "#B2 = orthonormal([[1,1,0,0], [1,-1,0,0], [0,0,1,1]])\n",
+    "\n",
+    "print(angle_eval(B1, B2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Application to PCA\n",
+    "\n",
+    "1. Have two conditions A, B, and some multivariate data of same shape gathered for each condition\n",
+    "2. For each dataset, extract the largest PCAs until explained variance reaches e.g. 90%\n",
+    "3. Compute angles between those PCAs\n",
+    "4. Make conclusions about how far the vector spaces lie\n",
+    "\n",
+    "### Problems\n",
+    "\n",
+    "1. How to reconcile multiple angles into 1 metric\n",
+    "2. How to include explained variances into the metric. Important are not only directions, but also extents into each direction. Naively multiplying each basis vector by variance or precision does not work, as the resulting metric is no longer an angle but something hard to interpret."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python (nest)",
+   "language": "python",
+   "name": "py36nest"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}