Update nlp_apps.ipynb (#764)

Author: antmarakis

nlp_apps.ipynb

@@ -15,7 +15,8 @@
    "source": [
     "## CONTENTS\n",
     "\n",
-    "* Language Recognition"
+    "* Language Recognition\n",
+    "* Author Recognition"
    ]
   },
   {
@@ -30,15 +31,13 @@
     "\n",
     "First we need to build our dataset. We will take as input text in English and in German and we will extract n-gram character models (in this case, *bigrams* for n=2). For English, we will use *Flatland* by Edwin Abbott and for German *Faust* by Goethe.\n",
     "\n",
-    "Let's build our text models for each language, which will hold the probability of each bigram occurring in the text."
+    "Let's build our text models for each language, which will hold the probability of each bigram occuring in the text."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 1,
-   "metadata": {
-    "collapsed": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "from utils import open_data\n",
@@ -67,9 +66,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "metadata": {
-    "collapsed": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "from learning import NaiveBayesLearner\n",
@@ -91,9 +88,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "metadata": {
-    "collapsed": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "def recognize(sentence, nBS, n):\n",
@@ -106,6 +101,8 @@
     "    for b, p in P_sentence.dictionary.items():\n",
     "        ngrams += [b]*p\n",
     "    \n",
+    "    print(ngrams)\n",
+    "    \n",
     "    return nBS(ngrams)"
    ]
   },
@@ -121,6 +118,13 @@
    "execution_count": 4,
    "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[(' ', 'i'), ('i', 'c'), ('c', 'h'), (' ', 'b'), ('b', 'i'), ('i', 'n'), ('i', 'n'), (' ', 'e'), ('e', 'i'), (' ', 'p'), ('p', 'l'), ('l', 'a'), ('a', 't'), ('t', 'z')]\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
@@ -141,6 +145,13 @@
    "execution_count": 5,
    "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[(' ', 't'), ('t', 'u'), ('u', 'r'), ('r', 't'), ('t', 'l'), ('l', 'e'), ('e', 's'), (' ', 'f'), ('f', 'l'), ('l', 'y'), (' ', 'h'), ('h', 'i'), ('i', 'g'), ('g', 'h')]\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
@@ -161,6 +172,13 @@
    "execution_count": 6,
    "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[(' ', 'd'), ('d', 'e'), ('e', 'r'), ('e', 'r'), (' ', 'p'), ('p', 'e'), ('e', 'l'), ('l', 'i'), ('i', 'k'), ('k', 'a'), ('a', 'n'), (' ', 'i'), ('i', 's'), ('s', 't'), (' ', 'h'), ('h', 'i'), ('i', 'e')]\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
@@ -181,6 +199,13 @@
    "execution_count": 7,
    "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[(' ', 'a'), ('a', 'n'), ('n', 'd'), (' ', 't'), (' ', 't'), ('t', 'h'), ('t', 'h'), ('h', 'u'), ('u', 's'), ('h', 'e'), (' ', 'w'), ('w', 'i'), ('i', 'z'), ('z', 'a'), ('a', 'r'), ('r', 'd'), (' ', 's'), ('s', 'p'), ('p', 'o'), ('o', 'k'), ('k', 'e')]\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
@@ -202,6 +227,150 @@
    "source": [
     "You can add more languages if you want, the algorithm works for as many as you like! Also, you can play around with *n*. Here we used 2, but other numbers work too (even though 2 suffices). The algorithm is not perfect, but it has high accuracy even for small samples like the ones we used. That is because English and German are very different languages. The closer together languages are (for example, Norwegian and Swedish share a lot of common ground) the lower the accuracy of the classifier."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## AUTHOR RECOGNITION\n",
+    "\n",
+    "Another similar application to language recognition is recognizing who is more likely to have written a sentence, given text written by them. Here we will try and predict text from Edwin Abbott and Jane Austen. They wrote *Flatland* and *Pride and Prejudice* respectively.\n",
+    "\n",
+    "We are optimistic we can determine who wrote what based on the fact that Abbott wrote his novella on much later date than Austen, which means there will be linguistic differences between the two works. Indeed, *Flatland* uses more modern and direct language while *Pride and Prejudice* is written in a more archaic tone containing more sophisticated wording.\n",
+    "\n",
+    "Similarly with Language Recognition, we will first import the two datasets. This time though we are not looking for connections between characters, since that wouldn't give that great results. Why? Because both authors use English and English follows a set of patterns, as we show earlier. Trying to determine authorship based on this patterns would not be very efficient.\n",
+    "\n",
+    "Instead, we will abstract our querying to a higher level. We will use words instead of characters. That way we can more accurately pick at the differences between their writing style and thus have a better chance at guessing the correct author.\n",
+    "\n",
+    "Let's go right ahead and import our data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from utils import open_data\n",
+    "from text import *\n",
+    "\n",
+    "flatland = open_data(\"EN-text/flatland.txt\").read()\n",
+    "wordseq = words(flatland)\n",
+    "\n",
+    "P_Abbott = UnigramWordModel(wordseq, 5)\n",
+    "\n",
+    "pride = open_data(\"EN-text/pride.txt\").read()\n",
+    "wordseq = words(pride)\n",
+    "\n",
+    "P_Austen = UnigramWordModel(wordseq, 5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This time we set the `default` parameter of the model to 5, instead of 0. If we leave it at 0, then when we get a sentence containing a word we have not seen from that particular author, the chance of that sentence coming from that author is exactly 0 (since to get the probability, we multiply all the separate probabilities; if one is 0 then the result is also 0). To avoid that, we tell the model to add 5 to the count of all the words that appear.\n",
+    "\n",
+    "Next we will build the Naive Bayes Classifier:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from learning import NaiveBayesLearner\n",
+    "\n",
+    "dist = {('Abbott', 1): P_Abbott, ('Austen', 1): P_Austen}\n",
+    "\n",
+    "nBS = NaiveBayesLearner(dist, simple=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that we have build our classifier, we will start classifying. First, we need to convert the given sentence to the format the classifier needs. That is, a list of words."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def recognize(sentence, nBS):\n",
+    "    sentence = sentence.lower()\n",
+    "    sentence_words = words(sentence)\n",
+    "    \n",
+    "    return nBS(sentence_words)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "First we will input a sentence that is something Abbott would write. Note the use of square and the simpler language."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Abbott'"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "recognize(\"the square is mad\", nBS)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The classifier correctly guessed Abbott.\n",
+    "\n",
+    "Next we will input a more sophisticated sentence, similar to the style of Austen."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Austen'"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "recognize(\"a most peculiar acquaintance\", nBS)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The classifier guessed correctly again.\n",
+    "\n",
+    "You can try more sentences on your own. Unfortunately though, since the datasets are pretty small, chances are the guesses will not always be correct."
+   ]
   }
  ],
  "metadata": {
@@ -220,7 +389,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.3"
+   "version": "3.6.3"
   }
  },
  "nbformat": 4,