add simple exercises

Author: piskvorky

1 - Streamed Corpora.ipynb

@@ -1,7 +1,7 @@
 {
  "metadata": {
   "name": "",
-  "signature": "sha256:00cb81ac4679ddb7ca14eda1511c747897a3d026bb9a633e1cb444a0a64fa06e"
+  "signature": "sha256:39725d0baebcf62b577927ded7ef2aa3f9d5a34b1b355b3ebc62bcdd8f53f0c9"
  },
  "nbformat": 3,
  "nbformat_minor": 0,
@@ -93,7 +93,7 @@
       "    # get information (metadata) about all files in the tarball\n",
       "    file_infos = [file_info for file_info in tf if file_info.isfile()]\n",
       "    \n",
-      "    # print one of them; for example, the last one\n",
+      "    # print one of them; for example, the first one\n",
       "    message = tf.extractfile(file_infos[0]).read()\n",
       "    print(message)"
      ],
@@ -107,14 +107,14 @@
      "source": [
       "This text is typical of real-world data. It contains a mix of relevant text, metadata (email headers), and downright noise. Even its relevant content is unstructured, with email addresses, people's names, quotations etc.\n",
       "\n",
-      "Most machine learning methods, topic modeling included, are only as good as the data you give it. At this point, we generally want to clean the data as much as possible. While the subsequent steps in the machine learning pipeline are more or less automated, handling the raw data should reflect the intended purpose of the application, its business logic, idiosyncracies, sanity check (aren't we accidentally receiving and parsing *image data instead of plain text*?). As always with automated processing it's [garbage in, garbage out](http://en.wikipedia.org/wiki/Garbage_in,_garbage_out)."
+      "Most machine learning methods, topic modeling included, are only as good as the data you give it. At this point, we generally want to clean the data as much as possible. While the subsequent steps in the machine learning pipeline are more or less automated, handling the raw data should reflect the intended purpose of the application, its business logic, idiosyncracies, sanity check (aren't we accidentally receiving and parsing *image data* instead of *plain text*?). As always with automated processing it's [garbage in, garbage out](http://en.wikipedia.org/wiki/Garbage_in,_garbage_out)."
      ]
     },
     {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
-      "As an example, let's write a function that aims to extract only the chunk of relevant text, ignoring the email headers:"
+      "As an example, let's write a function that aims to extract only the chunk of relevant text from each message, ignoring email headers:"
      ]
     },
     {
@@ -130,7 +130,7 @@
       "    message = gensim.utils.to_unicode(message, 'latin1').strip()\n",
       "    blocks = message.split(u'\\n\\n')\n",
       "    # skip email headers (first block) and footer (last block)\n",
-      "    content = u'\\n\\n'.join(blocks[1:-1])\n",
+      "    content = u'\\n\\n'.join(blocks[1:])\n",
       "    return content\n",
       "\n",
       "print process_message(message)"
@@ -145,23 +145,23 @@
      "source": [
       "Feel free to modify this function and test out other ideas for clean up. The flexibility Python gives you in processing text is superb -- [it'd be a crime](http://radimrehurek.com/2014/03/data-streaming-in-python-generators-iterators-iterables/) to hide the processing behind opaque APIs, exposing only one or two tweakable parameters.\n",
       "\n",
-      "There are a handful of handy Python libraries for text cleanup: [jusText](https://github.com/miso-belica/jusText) removes HTML boilerplate and extracting \"main text\". FIXME NLTK, Pattern and TextBlob for tokenization and generic NLP. Tokenization, sentence splitting, chunking, POS tagging."
+      "There are a handful of handy Python libraries for text cleanup: [jusText](https://github.com/miso-belica/jusText) removes HTML boilerplate and extracts \"main text\" of a web page. [NLTK](http://www.nltk.org/), [Pattern](http://www.clips.ua.ac.be/pattern) and [TextBlob](http://textblob.readthedocs.org/en/dev/) are good for tokenization, POS tagging, sentence splitting and generic NLP, with a nice Pythonic interface. None of them scales very well though, so keep the inputs small."
      ]
     },
     {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
-      "**Exercise (10 min)**: FIXME"
+      "**Exercise (5 min)**: Modify the `process_message` function to ignore message footers, too."
      ]
     },
     {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
-      "It's a good practice to inspect your data visually, at each point as it passes through your data processing pipeline. Simple printing (logging) a few arbitrary entries, ala UNIX `head`, does wonders for spotting unexpected bugs. *Oh, bad encoding! What is Chinese doing there, we were told all texts are English only? Do these rubbish tokens come from embedded images? How come everything's empty?* Taking a text with \"hey, let's tokenize it into a bag of words blindly, like they do in the tutorials, push it through this magical unicorn ML library and hope for the best\" is ill advised.\n",
+      "It's a good practice to inspect your data visually, at each point as it passes through your data processing pipeline. Simple printing (logging) a few arbitrary entries, ala UNIX `head`, does wonders for spotting unexpected bugs. *Oh, bad encoding! What is Chinese doing there, we were told all texts are English only? Do these rubbish tokens come from embedded images? How come everything's empty?* Taking a text with \"hey, let's tokenize it into a bag of words blindly, like they do in the tutorials, push it through this magical unicorn machine learning library and hope for the best\" is ill advised.\n",
       "\n",
-      "Another good practice is to keep internal strings as Unicode, and only encode/decode on IO (preferably using UTF8). As of Python 3.3, there is practically no memory penalty for using decoded Unicode ([PEP 393](http://legacy.python.org/dev/peps/pep-0393/))."
+      "Another good practice is to keep internal strings as Unicode, and only encode/decode on IO (preferably using UTF8). As of Python 3.3, there is practically no memory penalty for using Unicode over UTF8 byte strings ([PEP 393](http://legacy.python.org/dev/peps/pep-0393/))."
      ]
     },
     {
@@ -211,7 +211,7 @@
      "source": [
       "This uses the `process_message()` we wrote above, to process each message in turn. The messages are extracted on-the-fly, one after another, using a generator.\n",
       "\n",
-      "Such **data streaming** is a very important pattern: real data is typically too large to fit into RAM, and we don't need all of it in RAM at the same time anyway -- that's just wasteful. With streamed data, we can process arbitrarily large input, reading the data from a file on disk, shared network disk, or even more exotic remote network protocols."
+      "Such **data streaming** is a very important pattern: real data is typically too large to fit into RAM, and we don't need all of it in RAM at the same time anyway -- that's just wasteful. With streamed data, we can process arbitrarily large input, reading the data from a file on disk, SQL database, shared network disk, or even more exotic remote network protocols."
      ]
     },
     {
@@ -327,7 +327,7 @@
       "\n",
       "print(gensim.utils.lemmatize(\"worked\"))\n",
       "print(gensim.utils.lemmatize(\"working\"))\n",
-      "print(gensim.utils.lemmatize(\"The big fat cows jumped over a quick brown foxes.\"))"
+      "print(gensim.utils.lemmatize(\"I was working with a working class hero.\"))"
      ],
      "language": "python",
      "metadata": {},
@@ -367,7 +367,7 @@
      "cell_type": "markdown",
      "metadata": {},
      "source": [
-      "**Task (10 min)**: Modify `split_words()` to ignore (=not return) generic words, such as \"do\", \"then\", \"be\". These are called stopwords and we may want to remove them because some topic modeling algorithms are sensitive to their presence. An example of common stopwords for English is in `from gensim.parsing.preprocessing import STOPWORDS`."
+      "**Exercise (10 min)**: Modify `tokenize()` to ignore (=not return) generic words, such as \"do\", \"then\", \"be\", \"as\"... These are called stopwords and we may want to remove them because some topic modeling algorithms are sensitive to their presence. An example of common stopwords set for English is in `from gensim.parsing.preprocessing import STOPWORDS`."
      ]
     },
     {
@@ -386,7 +386,7 @@
       "\n",
       "[Named entity recognition (NER)](http://en.wikipedia.org/wiki/Named-entity_recognition) is the task of locating chunks of text that refer to people, locations, organizations etc.\n",
       "\n",
-      "Detecting collocations and named entities often has a significant business value: \"General Electric\" stays a single entity (token), rather than two words \"general\" and \"electric\". Same with \"Marathon Petroleum\", \"George Bush\" etc -- the model doesn't confuse its topics via words coming from unrelated entities, such as \"Korea\" and \"Carolina\" via \"North\"."
+      "Detecting collocations and named entities often has a significant business value: \"General Electric\" stays a single entity (token), rather than two words \"general\" and \"electric\". Same with \"Marathon Petroleum\", \"George Bush\" etc -- a topic model doesn't confuse its topics via words coming from unrelated entities, such as \"Korea\" and \"Carolina\" via \"North\"."
      ]
     },
     {
@@ -412,12 +412,12 @@
       "    tcf = TrigramCollocationFinder.from_words(words)\n",
       "    tcf.apply_freq_filter(min_freq)\n",
       "    trigrams = [' '.join(w) for w in tcf.nbest(TrigramAssocMeasures.chi_sq, top_n)]\n",
-      "    logging.info(\"%i trigrams found: %s...\" % (len(trigrams), list(trigrams)[:20]))\n",
+      "    logging.info(\"%i trigrams found: %s...\" % (len(trigrams), trigrams[:20]))\n",
       "\n",
       "    bcf = tcf.bigram_finder()\n",
       "    bcf.apply_freq_filter(min_freq)\n",
       "    bigrams = [' '.join(w) for w in bcf.nbest(BigramAssocMeasures.pmi, top_n)]\n",
-      "    logging.info(\"%i bigrams found: %s...\" % (len(bigrams), list(bigrams)[:20]))\n",
+      "    logging.info(\"%i bigrams found: %s...\" % (len(bigrams), bigrams[:20]))\n",
       "\n",
       "    pat_gram2 = re.compile('(%s)' % '|'.join(bigrams), re.UNICODE)\n",
       "    pat_gram3 = re.compile('(%s)' % '|'.join(trigrams), re.UNICODE)\n",
@@ -438,13 +438,21 @@
       "    def __init__(self, fname):\n",
       "        self.fname = fname\n",
       "        logging.info(\"collecting ngrams from %s\" % self.fname)\n",
+      "        # generator of documents; one element = list of words\n",
       "        documents = (self.split_words(text) for text in iter_20newsgroups(self.fname, log_every=1000))\n",
+      "        # generator: concatenate (chain) all words into a single sequence, lazily\n",
       "        words = itertools.chain.from_iterable(documents)\n",
       "        self.bigrams, self.trigrams = best_ngrams(words)\n",
       "\n",
-      "    def __iter__(self):\n",
-      "        for message in iter_20newsgroups(self.fname):\n",
-      "            yield self.tokenize(message)\n",
+      "    def split_words(self, text, stopwords=STOPWORDS):\n",
+      "        \"\"\"\n",
+      "        Break text into a list of single words. Ignore any token that falls into\n",
+      "        the `stopwords` set.\n",
+      "\n",
+      "        \"\"\"\n",
+      "        return [word\n",
+      "                for word in gensim.utils.tokenize(text, lower=True)\n",
+      "                if word not in STOPWORDS and len(word) > 3]\n",
       "\n",
       "    def tokenize(self, message):\n",
       "        \"\"\"\n",
@@ -459,15 +467,9 @@
       "        text = re.sub(self.bigrams, lambda match: match.group(0).replace(u' ', u'_'), text)\n",
       "        return text.split()\n",
       "\n",
-      "    def split_words(self, text, stopwords=STOPWORDS):\n",
-      "        \"\"\"\n",
-      "        Break text into a list of single words. Ignore any token that falls into\n",
-      "        the `stopwords` set.\n",
-      "\n",
-      "        \"\"\"\n",
-      "        return [word\n",
-      "                for word in gensim.utils.tokenize(text, lower=True)\n",
-      "                if word not in STOPWORDS and len(word) > 3]\n",
+      "    def __iter__(self):\n",
+      "        for message in iter_20newsgroups(self.fname):\n",
+      "            yield self.tokenize(message)\n",
       "\n",
       "%time collocations_corpus = Corpus20News_Collocations('./data/20news-bydate.tar.gz')\n",
       "print(list(itertools.islice(collocations_corpus, 2)))"
@@ -493,7 +495,7 @@
       "    \"\"\"Convenience fnc: return the first `n` elements of the stream, as plain list.\"\"\"\n",
       "    return list(itertools.islice(stream, n))\n",
       "\n",
-      "def best_phrases(document_stream, top_n=1000, prune_at=100000):\n",
+      "def best_phrases(document_stream, top_n=1000, prune_at=50000):\n",
       "    \"\"\"Return a set of `top_n` most common noun phrases.\"\"\"\n",
       "    np_counts = {}\n",
       "    for docno, doc in enumerate(document_stream):\n",
@@ -510,8 +512,8 @@
       "            # only consider multi-word NEs where each word contains at least one letter\n",
       "            if u' ' not in np:\n",
       "                continue\n",
-      "            # ignore phrases that contains too short/non-alphanetic words\n",
-      "            if all(len([ch for ch in word if ch.isalpha()]) > 2 for word in np.split()):\n",
+      "            # ignore phrases that contain too short/non-alphabetic words\n",
+      "            if all(word.isalpha() and len(word) > 2 for word in np.split()):\n",
       "                np_counts[np] = np_counts.get(np, 0) + 1\n",
       "\n",
       "    sorted_phrases = sorted(np_counts, key=lambda np: -np_counts[np])\n",
@@ -565,13 +567,6 @@
      "metadata": {},
      "outputs": []
     },
-    {
-     "cell_type": "markdown",
-     "metadata": {},
-     "source": [
-      "**Exercise**: FIXME"
-     ]
-    },
     {
      "cell_type": "heading",
      "level": 2,