GenePattern Python Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GenePattern Python Library Tutorial\n",
    "\n",
    "This is a short tutorial on how to use the [GenePattern Python library](http://www.broadinstitute.org/cancer/software/genepattern/programmers-guide#_Using_GenePattern_from_Python) in your Python scripts or in the [GenePattern Notebook](http://www.broadinstitute.org/cancer/software/genepattern/genepattern-notebooks) environment. In it we will develop a simple Python program that connects to a GenePattern Server, runs a module and loads the resulting files for further analysis. The included code can be executed directly from the notebook, so that you can try it out, modify it or create your own solutions. \n",
    "\n",
    "This tutorial assumes that the reader is familiar with [GenePattern](http://genepattern.org) and its basic associated concepts, such as modules, jobs and result files. If you wish to learn more about GenePattern, [click here](http://www.broadinstitute.org/cancer/software/genepattern/).\n",
    "\n",
    "### Installation\n",
    "\n",
    "If you are using this tutorial in the GenePattern Notebook environment, the GenePattern Python library will already be installed. If you're using this notebook outside of the GenePattern Notebook environment, you may need to install the library. [Installation instructions are available here.](http://www.broadinstitute.org/cancer/software/genepattern/programmers-guide#_Using_GenePattern_from_Python)\n",
    "\n",
    "\n",
    "### Compatibility\n",
    "\n",
    "The GenePattern Python library supports both Python 2.7 and Python 3.3+."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the Library\n",
    "\n",
    "The very first thing you will need to do is to import the GenePattern library into your script. This can be achieved by entering the following code shown below. All methods provided by the GenePattern library can then be accessed from the *gp* namespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to a GenePattern Server\n",
    "\n",
    "The next step in using the GenePattern Python library is to connect to an existing GenePattern server. This will require entering the URL of the server, as well as your username and password credentials. \n",
    "\n",
    "The code below connects to the public GenePattern server running at the Broad Institute. If you have not used this server before, you will first need to create an account. [Click here to register an account.](http://genepattern.broadinstitute.org/gp/pages/registerUser.jsf)\n",
    "\n",
    "Needless to say, you will need to change *myusername* and *mypassword* in the code below to your actual username and password."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a GenePattern server proxy instance\n",
    "gpserver = gp.GPServer('https://cloud.genepattern.org/gp','myusername', 'mypassword')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query for Available Modules \n",
    "\n",
    "If you are not aware of which modules are available on the GenePattern server, this can be programmatically explored by running the code shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get the list of tasks\n",
    "task_list = gpserver.get_task_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will return a list of GPTask objects. A GPTask object is a Python object which represents a specific module. In the case of the command above, the returned list will contain GPTask objects for every module the server makes available. These GPTask objects will provide the module name, LSID, a description and the version number of the module. More on GPTask objects is described in the next section.\n",
    "\n",
    "This list can be iterated over like any Python list to obtain information from each GPTask object. Below is a code example that will print the name of each module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for task in task_list:\n",
    "    print(task.get_name())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Referencing a Module\n",
    "\n",
    "If you already know the name or LSID of the module you want, you can also obtain a GPTask object for it directly. \n",
    "\n",
    "Throughout this tutorial we will use the PreprocessDataset module. This module is designed to perform some basic preprocessing steps on gene expression data. For example, it can apply a ceiling, floor and fold change filter. More on this module is [available here](http://www.broadinstitute.org/cancer/software/genepattern/modules/docs/PreprocessDataset/5).\n",
    "\n",
    "The example code below obtains a GPTask object for the PreprocessDataset module in two different ways: by name and by LSID. Either one of these methods works to obtain the reference, although specifying the LSID is more specific as it contains a specific version number for the module, whereas specifying by name always obtains the latest version of the module. See how GenePattern implements  [version numbers](https://www.genepattern.org/concepts/#_Version_Numbers) using Life Science Identifiers (LSIDs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Obtaining GPTask by module name\n",
    "module = gp.GPTask(gpserver, \"PreprocessDataset\")\n",
    "\n",
    "# Obtaining GPTask by LSID\n",
    "module = gp.GPTask(gpserver, \"urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00020\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Parameter Information\n",
    "\n",
    "At this point the GPTask object only contains some very basic data used to identify the module. Before you can use this module to run jobs, however, you will first need to load the full parameter data from the server. This can be accomplished by running the code shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the full parameter data\n",
    "module.param_load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is code example showing how to reference the information that was loaded for the module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Print the module name\n",
    "print( module.get_name() )\n",
    "\n",
    "# Print the module LSID\n",
    "print( module.get_lsid() )\n",
    "\n",
    "# Print the module version\n",
    "print( module.get_version() )\n",
    "\n",
    "# Print the description\n",
    "print( module.get_description() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After being loaded, each of the module's parameters will be represented by a GPTaskParam object. A list of each of the parameters can be obtained and explored using the example code below. Each GPTaskParam object contains a parameter name, type, description, whether it's optional and other metadata, as shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get the list of GPTaskParam objects\n",
    "params_list = module.get_parameters()\n",
    "\n",
    "for param in params_list:              # Loop through each parameter\n",
    "    print( param.get_name() )          # Print the parameter's name\n",
    "    print( param.get_type() )          # Print the parameter's type (text, number, file, etc.)\n",
    "    print( param.get_description() )   # Print the parameter's description\n",
    "    print( param.get_default_value() ) # Print the parameter's default value\n",
    "    print( param.is_optional() )       # Print whether the parameter is optional\n",
    "    print( '' )                        # Leave a blank line between printed parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some parameters come with a list of valid choices. These parameters can be identified by calling the is_choice_param() method. Additionally, choice parameters have a number of other methods available for working with the choice list. Calling these methods on a non-choice parameter will result in an error being thrown. An example is given below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Loop through each parameter\n",
    "for param in params_list:\n",
    "    if param.is_choice_param():        # If the parameter is a choice param\n",
    "        print( param.get_name() )      # Print the parameter's name\n",
    "        \n",
    "        choices = param.get_choices()  # Get a list of valid choices \n",
    "        for choice in choices:         # Print the label and value for each choice\n",
    "            print( choice['label'] + \" = \" + choice['value'] )\n",
    "            \n",
    "        # Print the default selected value for each choice\n",
    "        print( param.get_choice_selected_value() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Job Specification\n",
    "\n",
    "In order to run a GenePattern job from Python, you must first obtain a GPJobSpec object from the correct GPTask object and then set the appropriate parameters for the job. For many parameters their default values will suffice. For others, you will want to set a specific value.\n",
    "\n",
    "Below is code showing how to obtain a GPJobSpec object and how iterate over the parameters, setting them to their default values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create the GPJobSpec\n",
    "job_spec = module.make_job_spec()\n",
    "\n",
    "# Loop through all the parameters and set their default values\n",
    "for param in module.get_parameters():  \n",
    "    # If the parameter has a default value, set that value\n",
    "    if param.get_default_value() != None: \n",
    "        # Set the default value\n",
    "        job_spec.set_parameter( param.get_name(), param.get_default_value() )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To set a specific value for a parameter, the set_parameter() method should be called. In the code below you will set the *input.filename* parameter to point to a publicly available dataset. This data should suffice for the purposes of this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Attach the input file to the correct parameter\n",
    "job_spec.set_parameter(\"input.filename\", \"https://datasets.genepattern.org/data/all_aml/all_aml_test.gct\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data files can be uploaded by calling GPServer.upload_file(). This will return a GPFile object, and the parameter can be set to point to the URL of this object. An example for the PreprocessDataset module is shown below. The code has been commented out, however, as it will not be used in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Upload the input file\n",
    "# uploaded_file = gpserver.upload_file(\"file_name\", \"/path/to/the/file/on/the/file/system/file_name\")  \n",
    "\n",
    "# Attach the input file to the correct parameter\n",
    "# job_spec.set_parameter(\"input.filename\", 'uploaded_file.get_url()')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting Your First Job\n",
    "\n",
    "Once the GPJobSpec is ready, it can be used to launch a GenePattern job. This will return a GPJob object, representing the specific job that was just launched. A code example of how to do this is below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This will return the job object and continue execution even if the job isn't finished\n",
    "job = gpserver.run_job(job_spec, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why are we passing *False* in as a parameter, you ask? By default the run_job() method will halt code execution of your Python script until the job has finished running in GenePattern. For long running jobs, however, this may not be desirable. By optionally passing in *False* as a parameter, the method will return as soon as the job is submitted, allowing the Python program to continue.\n",
    "\n",
    "For the purposes of this tutorial, it is better that you do not have to wait. If you did want to submit the job and wait for it to complete, however, the code is below (albeit commented out)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#  This will halt execution until the job is complete\n",
    "# job = gpserver.run_job(job_spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Querying for Job Status\n",
    "\n",
    "When a GenePattern job is submitted, it passes through several states: pending, running and then either to complete or error. At any time after a job has been submitted, its status can be checked by calling *get_status_message()*. Similarly, its completion can be checked by calling *is_finished()*. Examples of both are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Prints a brief description of the job's current state\n",
    "print( job.get_status_message() )\n",
    "\n",
    "# Quaries the server and returns True if the job is complete, False otherwise\n",
    "print( job.is_finished() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, if at any point you decide that you just want to wait until the job is complete, you can always call *wait_until_done()*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "job.wait_until_done()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Output Files\n",
    "\n",
    "Once the job is complete, and assuming there were no errors, a list of its output files may be obtained by making the *get_output_files()* call shown below. \n",
    "\n",
    "This will return a list of GPFile objects, each containing methods to download or read the contents of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get a list of output files\n",
    "output_list = job.get_output_files()  \n",
    "\n",
    "for file in output_list:     # Loop through each output file\n",
    "    print( file.get_url() )  # Print the URL to the file\n",
    "    data = file.read()       # Read the data in the file "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the contents of a data file has been assigned to a variable, it may used in conjunction with other common Python libraries, such as matplotlib, pandas, numpy or scipy.\n",
    "\n",
    "The code below will print out the contents of the last output file assigned to the *data* variable in the previous code block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print ( data )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This concludes the tutorial on how to work with GenePattern using the GenePattern Python library. For more information, please see the [GenePattern Programmer's Guide](http://www.broadinstitute.org/cancer/software/genepattern/programmers-guide)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3.6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}