semparse_homework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2016 term\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "All seven problems are required.\n",
    "\n",
    "* [Arithmetic domain](#Arithmetic-domain)\n",
    "  * [Question 1](#Question-1-[2-points])\n",
    "  * [Question 2](#Question-2-[2-points])\n",
    "  * [Question 3](#Question-3-[2-points])\n",
    "  \n",
    "* [Travel domain](#Travel-domain)  \n",
    "  * [Question 4](#Question-4-[2-points])\n",
    "  * [Question 5](#Question-5-[2-points])\n",
    "  \n",
    "* [GeoQuery domain](#GeoQuery-domain)  \n",
    "  * [Question 6](#Question-6-[2-points])\n",
    "  * [Question 7](#Question-7-[2-points])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure the `sys.path.append` value is the path to your local [SippyCup repository](https://github.com/wcmac/sippycup). (Alternatively, you can add SippyCup to your Python path.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../sippycup')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arithmetic domain\n",
    "\n",
    "SippyCup includes a module `arithmetic` with a class `ArithmeticDomain` that brings together\n",
    "the examples from unit 1. Here's an example of the sort of use you'll make of this domain\n",
    "for these homework problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from arithmetic import ArithmeticDomain\n",
    "from parsing import parse_to_pretty_string\n",
    "\n",
    "# Import the domain and make sure all is well:\n",
    "math_domain = ArithmeticDomain()\n",
    "\n",
    "# Core grammar:\n",
    "math_grammar = math_domain.grammar()\n",
    "\n",
    "# A few examples:\n",
    "parses = math_grammar.parse_input(\"minus two plus three\")\n",
    "for parse in parses:\n",
    "    print('\\nParse:\\n', parse_to_pretty_string(parse, show_sem=True))\n",
    "    print('Denotation:', math_domain.execute(parse.semantics))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a convenience function we'll use for seeing what the grammar is doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def display_examples(utterances, grammar=None, domain=None):\n",
    "    for utterance in utterances:        \n",
    "        print(\"=\"*70)\n",
    "        print(utterance)\n",
    "        parses = grammar.parse_input(utterance)\n",
    "        for parse in parses:\n",
    "            print('\\nParse:\\n', parse_to_pretty_string(parse, show_sem=True))\n",
    "            print('Denotation:', domain.execute(parse.semantics))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1 [2 points]\n",
    "\n",
    "Your task is to extend `ArithmeticDomain` to include the unary operators `squared` and `cubed`,  which form expressions like `three squared` and `nine cubed`. The following code will help you get started on this. \n",
    "\n",
    "__Submit__: your completion of this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from arithmetic import ArithmeticDomain\n",
    "from parsing import Rule, add_rule\n",
    "\n",
    "# Resort to the original grammar to avoid repeatedly adding the same\n",
    "# rules to the grammar during debugging, which multiplies the number\n",
    "# of parses without changing the set of parses produced:\n",
    "math_domain = ArithmeticDomain()\n",
    "math_grammar = math_domain.grammar()\n",
    "\n",
    "# Here's where your work should go:\n",
    "\n",
    "# Add rules to the grammar:\n",
    "\n",
    "# Extend domain.ops appropriately:\n",
    "\n",
    "# Make sure things are working:\n",
    "display_examples(('three squared', 'minus three squared', 'four cubed'), \n",
    "                 grammar=math_grammar, domain=math_domain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2 [2 points]\n",
    "\n",
    "Your task is to extend `ArithmeticDomain` to support numbers with decimals, so  that you can parse all expressions of the form `N point D` where `N` and `D` both denote `int`s. You can assume that both numbers are both spelled out as single words, as in \"one point twelve\" rather than \"one point one two\". (The grammar fragment has only \"one\", \"two\", \"three\", and \"four\" anyway.) \n",
    "\n",
    "__Submit__: your completion of the following code.\n",
    "\n",
    "__Important__: your grammar should not create spurious parses like `(two times three) point four`.  This means that you can't treat `point` like the other binary operators in your syntactic grammar. This will require you to add special rules to handle the internal structure of these decimal numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from arithmetic import ArithmeticDomain\n",
    "from parsing import Rule, add_rule\n",
    "\n",
    "# Clear out the grammar; remove this if you want your question 1 \n",
    "# extension to combine with these extensions:\n",
    "math_domain = ArithmeticDomain()\n",
    "math_grammar = math_domain.grammar()\n",
    "\n",
    "# Remember to add these rules to the grammar!\n",
    "integer_rules = [ \n",
    "    Rule('$I', 'one', 1), \n",
    "    Rule('$I', 'two', 2), \n",
    "    Rule('$I', 'three', 3), \n",
    "    Rule('$I', 'four', 4) ]\n",
    "\n",
    "tens_rules = [\n",
    "    Rule('$T', 'one', 1),\n",
    "    Rule('$T', 'two', 2),\n",
    "    Rule('$T', 'three', 3),\n",
    "    Rule('$T', 'four', 4) ]\n",
    "\n",
    "# Add the above rules to math_grammar:\n",
    "\n",
    "# Add rules to the grammar for using the above:\n",
    "\n",
    "# Extend domain.ops:\n",
    "\n",
    "# Make sure things are working:\n",
    "display_examples(('four point two', 'minus four point one', 'two minus four point one'),\n",
    "                 grammar=math_grammar, domain=math_domain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3 [2 points]\n",
    "\n",
    "Extend the grammar to support the multi-word expression `the average of` as  in `the average of one and four`. Your solution is required to treat each word in this multi-word expression as its own lexical item. Other than that, you have a lot of design freedom. Your solution can be limited to the case where the conjunction consists of just two numbers.\n",
    "\n",
    "__Submit__: your completion of this starter code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from arithmetic import ArithmeticDomain\n",
    "from parsing import Rule, add_rule\n",
    "import numpy as np\n",
    "\n",
    "math_domain = ArithmeticDomain()\n",
    "math_grammar = math_domain.grammar()\n",
    "\n",
    "# Add rules to the grammar:\n",
    "\n",
    "# Extend domain.ops:\n",
    "\n",
    "# Make sure things are working:\n",
    "display_examples(('the one', 'the average of one and four'),\n",
    "                 grammar=math_grammar, domain=math_domain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Travel domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an illustration of how parsing and interpretation work in this domain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from travel import TravelDomain\n",
    "\n",
    "travel_domain = TravelDomain()\n",
    "travel_grammar = travel_domain.grammar()\n",
    "\n",
    "display_examples(\n",
    "    (\"flight from Boston to San Francisco\",\n",
    "     \"directions from New York to Philadelphia\",\n",
    "     \"directions New York to Philadelphia\"),\n",
    "    grammar=travel_grammar,\n",
    "    domain=travel_domain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For these questions, we'll combine grammars with machine learning.\n",
    "Here's now to train and evaluate a model using the grammar \n",
    "that is included in `TravelDomain` along with a basic feature\n",
    "function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from travel import TravelDomain\n",
    "from scoring import Model\n",
    "from experiment import train_test\n",
    "from travel_examples import travel_train_examples, travel_test_examples\n",
    "from collections import defaultdict\n",
    "\n",
    "travel_domain = TravelDomain()\n",
    "travel_grammar = travel_domain.grammar()\n",
    "\n",
    "def basic_feature_function(parse):\n",
    "    \"\"\"Features for the rule used for the root node and its children\"\"\"\n",
    "    features = defaultdict(float)\n",
    "    features[str(parse.rule)] += 1.0\n",
    "    for child in parse.children:\n",
    "        features[str(child.rule)] += 1.0\n",
    "    return features\n",
    "        \n",
    "# This code evaluates the current grammar:    \n",
    "train_test(\n",
    "    model=Model(grammar=travel_grammar, feature_fn=basic_feature_function), \n",
    "    train_examples=travel_train_examples, \n",
    "    test_examples=travel_test_examples, \n",
    "    print_examples=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4 [2 points]\n",
    "\n",
    "With the default travel grammar, many of the errors on training examples occur because the origin  isn't marked by \"from\". You might have noticed that \"directions New York to Philadelphia\" is not handled properly in our opening example. Other examples include  \"transatlantic cruise southampton to tampa\",  \"fly boston to myrtle beach spirit airlines\", and  \"distance usa to peru\". \n",
    "\n",
    "__Your tasks__:\n",
    "\n",
    "0. Extend the grammar with a single rule to handle examples like these, and run another evaluation using this expanded grammar (submit your completion of the following starter code)\n",
    "\n",
    "0. In 1&ndash;2 sentences, summarize what happened to the post-training performance metrics when this rule was added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from travel import TravelDomain\n",
    "from parsing import Rule, add_rule\n",
    "from scoring import Model\n",
    "from experiment import train_test\n",
    "from travel_examples import travel_train_examples, travel_test_examples\n",
    "\n",
    "travel_domain = TravelDomain()\n",
    "travel_grammar = travel_domain.grammar()\n",
    "\n",
    "# Add your rule here:\n",
    "\n",
    "# This code evaluates the new grammar:\n",
    "train_test(\n",
    "    model=Model(grammar=travel_grammar, feature_fn=basic_feature_function), \n",
    "    train_examples=travel_train_examples, \n",
    "    test_examples=travel_test_examples, \n",
    "    print_examples=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5 [2 points]\n",
    "\n",
    "Your extended grammar for question 4 likely did some harm to the space of parses we \n",
    "consider. Consider how `number of parses` has changed. (If it's not intuitively clear why, \n",
    "check out some of the parses to see what's happening!)\n",
    "\n",
    "__Your task__: to try to make amends, expand the feature function to improve the \n",
    "ability of the optimizer to distinguish good parses from bad. You can write your\n",
    "own function and/or combine it with scoring functions that are available inside\n",
    "SippyCup. You should be able to achieve a gain in post-training train and\n",
    "test `semantics accuracy`. (Note: you should _not_ spend hours continually improving \n",
    "your score unless you are planning to develop this into a project. Any gain over\n",
    "the previous run will suffice here.) __Submit__: your completion of this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from parsing import Parse\n",
    "\n",
    "def expanded_feature_function(parse):\n",
    "    return {}\n",
    "\n",
    "# Evaluate the new grammar:\n",
    "train_test(\n",
    "    model=Model(grammar=travel_grammar, feature_fn=expanded_feature_function), \n",
    "    train_examples=travel_train_examples, \n",
    "    test_examples=travel_test_examples, \n",
    "    print_examples=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## GeoQuery domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a few simple examples from the GeoryQuery domain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from geoquery import GeoQueryDomain\n",
    "\n",
    "geo_domain = GeoQueryDomain()\n",
    "geo_grammar = geo_domain.grammar()\n",
    "\n",
    "display_examples(\n",
    "    (\"what is the biggest city in california ?\",\n",
    "     \"how many people live in new york ?\",\n",
    "     \"where is rochester ?\"),\n",
    "    grammar=geo_grammar,\n",
    "    domain=geo_domain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can train models just as we did for the travel domain, though we have to be more\n",
    "attentive to special features of semantic parsing in this domain. Here's a run with the\n",
    "default scoring model, metrics, etc.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from geoquery import GeoQueryDomain\n",
    "from scoring import Model\n",
    "from experiment import train_test\n",
    "\n",
    "geo_domain = GeoQueryDomain()\n",
    "geo_grammar = geo_domain.grammar()\n",
    "\n",
    "# We'll use this as our generic assessment interface for these questions:\n",
    "def special_geo_evaluate(grammar=None, feature_fn=geo_domain.features):\n",
    "    # Build the model by hand so that we can see all the pieces:\n",
    "    geo_mod = Model(\n",
    "        grammar=grammar, \n",
    "        feature_fn=feature_fn, \n",
    "        weights=geo_domain.weights(),\n",
    "        executor=geo_domain.execute)\n",
    "    # This can be done with less fuss using experiment.train_test_for_domain, \n",
    "    # but we want full access to the model, metrics, etc.\n",
    "    train_test(\n",
    "        model=geo_mod, \n",
    "        train_examples=geo_domain.train_examples(), \n",
    "        test_examples=geo_domain.test_examples(), \n",
    "        metrics=geo_domain.metrics(),\n",
    "        training_metric=geo_domain.training_metric(),    \n",
    "        seed=0,\n",
    "        print_examples=False)\n",
    "    \n",
    "special_geo_evaluate(grammar=geo_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6 [2 points]\n",
    "\n",
    "Two deficiencies of the current grammar: \n",
    "\n",
    "* The words \"where\" and \"is\" are treated as being of category `$Optional`, which means they are ignored. As a result, the grammar construes all questions of the form \"where is X\" as being about the identity of X!\n",
    "\n",
    "* Queries like \"how many people live in Florida\" are not handled correctly.\n",
    "\n",
    "__Your task__: Add grammar rules that address these problems and assess impact of the changes\n",
    "using the `train_test` based interface illustrated above. \n",
    "\n",
    "__Submit__: your expanded version of the starter code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from geoquery import GeoQueryDomain\n",
    "from parsing import Rule, add_rule\n",
    "\n",
    "geo_domain = GeoQueryDomain()\n",
    "geo_grammar = geo_domain.grammar()\n",
    "\n",
    "# Your rules go here:\n",
    "\n",
    "# Evaluation of the new grammar:\n",
    "special_geo_evaluate(grammar=geo_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7 [2 points]\n",
    "\n",
    "The success of the `empty_denotation` feature demonstrates the potential of denotation features. Can we go further?  Experiment with a feature or features that characterize the _size_ of the denotation (that is, the number of answers). This will involve extending `geo_domain.features` and running another assessment. __Submit__: your completion of the \n",
    "code below and 1&ndash;2 sentences saying how this feature seems to behave in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from geoquery import GeoQueryDomain\n",
    "\n",
    "def feature_function(parse):\n",
    "    # Bring in all the default features:\n",
    "    f = geo_domain.features(parse)\n",
    "    # Extend dictionary f with your new denotation-count feature\n",
    "    \n",
    "    return f\n",
    "\n",
    "# Evaluation of the new grammar:\n",
    "special_geo_evaluate(grammar=geo_grammar, feature_fn=feature_function)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}