CSC219_Homework2_Pandas_NucCGCountMean.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.1"
    },
    "colab": {
      "provenance": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Get your workspace ready\n",
        "Always make a copy of lecture and sample notebooks in your own drive. You can follow along with the coding in lectures and try out the code on your own.\n",
        "For work that you need to submit, rename it \"CSC219_Summer2024_[hmk number]_Your First Name_Your Last Name\".\n",
        "Enable sharing to Anyone with a link to \"editor\" on your notebook settings. This will be necessary if you're submitting it for credit and/or to get feedback on your work.\n",
        "When you're done, if applicable submit a link of this colab notebook via specified method in the class site.\n",
        "Remember that you have reading resources and this book under Resources on the class canvas page: https://sfsu.instructure.com/courses/39198/pages/recommended-book?module_item_id=2511293"
      ],
      "metadata": {
        "id": "U4WDHqlhfw-F"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<h1> Homework 2: Nested dictionaries, Pandas, Nuc and CpG counts, Mean </h1>"
      ],
      "metadata": {
        "id": "lWvVludj0Ejb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **CSC 219 Homework 2**\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "ULdhzZOahZB9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''Due by July 29th, 2024, 11:59 PST'''\n",
        "# Share the link right now\n",
        "# when you are done, copy the link to Homework 2 on Canvas."
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "Ulluyi6giaJS",
        "outputId": "44ff20c8-4676-4317-fb08-1f5b1d884c03"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Due by July 29th, 2024, 11:59 PST'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Objectives\n",
        "In previous work we sourced nucleotide sequences for 10 species from FASTA files. The data was processed visualize using Numpy and some basic Pandas.\n",
        "<br>\n",
        "In this homework, you will use that same data set to practice using different data structures, how to manipulate them with Pandas tools into different formats to facilitate different visualization needs.\n",
        "<br>\n",
        "The code to source and process the data is provided below. Or you can use your solutions from previous work to arrive at the same data set to start this homework."
      ],
      "metadata": {
        "id": "pOcWuR36hH-t"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zyhtxdxzuote"
      },
      "source": [
        "# Expectations\n",
        "\n",
        "* Your functions, methods, and outputs might differ from the samples given and that is acceptable as long as they answer the prompts and or provide reasonable results that show you've learned the topic.\n",
        "  * Write pseudo code along with your code to indicate that you understand and have a plan -- even if the code isn't working.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Work:"
      ],
      "metadata": {
        "id": "o4WR11Jx6qKL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Provided code"
      ],
      "metadata": {
        "id": "Zo1imIVLiMtB"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "30yio_bxkUnI",
        "outputId": "d9a41949-3f35-4fec-ff73-5a4934a7dc17",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "\"\"\"\n",
        "Provided code block 1 - mount drive\n",
        "\"\"\"\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive/')\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\"\"\"\n",
        "Provided code block 2 - functions\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "import os\n",
        "from collections import defaultdict\n",
        "\n",
        "# make sure you change this part to your own directory\n",
        "path = '/content/drive/MyDrive/SFSU/CSC219/dataFiles/FASTA/'\n",
        "#FASTA files\n",
        "fileNameList = ['Sars_Cov2_NC_004718_3', 'ΗCoV_EMC_019843_3','Malaria']\n",
        "\n",
        "def getFilePathList(path, nameList):\n",
        "  '''\n",
        "    make file path variables, full directory path +  file name + extension\n",
        "  '''\n",
        "  filePathList = []\n",
        "  for f in nameList:\n",
        "    filePathList.append(path + f +'.fasta')\n",
        "  return filePathList\n",
        "\n",
        "def getFileHandle(filePath):\n",
        "  if not os.path.exists(filePath):\n",
        "    raise Exception(\"Path/file not found!\")\n",
        "  fileHandle = open(filePath)\n",
        "  return fileHandle\n",
        "\n",
        "def loadSequences(filePathList) -> dict:\n",
        "    seqDictionary = {}\n",
        "    for i,fp in enumerate(filePathList):\n",
        "      fileHandle = getFileHandle(fp)\n",
        "\n",
        "      linesList = fileHandle.readlines()[1:]\n",
        "      tempList = []\n",
        "      tempSeq = \"\"\n",
        "      for line in linesList:\n",
        "        cleanString = line.strip('\\n')\n",
        "        tempSeq += cleanString\n",
        "\n",
        "      seqDictionary[fileNameList[i]] = tempSeq\n",
        "    return seqDictionary\n",
        "\n",
        "def countNuc(DNA):\n",
        "  defDictionary = defaultdict(int)\n",
        "  for x in DNA:\n",
        "    defDictionary[x] +=1\n",
        "  return defDictionary\n",
        "\n",
        "filePathList = getFilePathList(path, fileNameList)\n",
        "sequencesDict = loadSequences(filePathList)"
      ],
      "metadata": {
        "id": "szHXCOAG0Qev"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prompt 1"
      ],
      "metadata": {
        "id": "bL6gOoX9tZuJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Getting the nucleotide count for each species** -\n",
        "We are going to use a nested dictionary data structure this time\n",
        "1. Built a nested dictionary, a dictionary of dictionaries to hold species' nuc counts. We covered nested dictionaries in Module 2A Data Structures and Objects, under Nested Dictionaries\n",
        "* Use the functions provided above or your own, from homework 1\n",
        "* the dictionary of dictionaries should look like this:\n",
        "          spNucCountDictionary = {\n",
        "            'species1': {'A': 1053, 'T': 1254, 'G': 705, 'C': 756},\n",
        "            'species2': {'A': 1021, 'T': 1384, 'G': 785, 'C': 872},\n",
        "            .\n",
        "            .\n",
        "            .\n",
        "          }\n",
        "\n"
      ],
      "metadata": {
        "id": "oggKj6K4rnhF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''Your code here'''"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "zK8xRJym1522",
        "outputId": "366d4b18-9962-4ee6-b998-28af87c9efd3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Your code here'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prompt 2\n"
      ],
      "metadata": {
        "id": "hGHgQLaWtxty"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "2. a. Create a Pandas DataFrame from this nested dictionary where, by default:\n",
        "  * the outer keys will become the dataframe columns,\n",
        "  * the inner keys will become the dataframe labeled index,\n",
        "  * and the values (of the inner dictionary) will be the dataframe columns\n",
        "  * see the sample DataFrame output below\n",
        "  * you can load a nested dictionary directly into a DataFrame as it is! That is, there's no need to pass any other argument besides the dictionary in the DataFrame constructor because the default settings are what we want\n",
        "* b.  Print out the dataframe, its columns, and its indexes. For the columns and indexes, add .to_list() at the end to get a nicer printout\n",
        " <br><br>\n",
        "-------\n",
        "\n",
        "![picture1](https://drive.google.com/uc?export=view&id=1l0qEURu3SDr9onzEbvJi_FP-kDpk53Ok)\n",
        "Figure 1 - Sample dataframe\n",
        " * species names are the columns\n",
        " * the nucs are the labels/index\n",
        " * the counts are the dataframe values (down each column)\n",
        "------\n"
      ],
      "metadata": {
        "id": "OEuaipsnt05Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# reference: module 2C\n",
        "'''Your code here'''"
      ],
      "metadata": {
        "id": "oRl4ZWm177WU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prompt 3"
      ],
      "metadata": {
        "id": "xEXEC6KzrJcK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "When the orientation of the data in the dataframe need to be changed then you can use the DataFrame.from_dict() function.\n",
        " * the orientation of the data is how the keys and values get mapped to the dataframe's columns, index, and rows\n",
        " * the options for the orient parameter is columns (the default), index, and tight\n",
        "          \n",
        "\n",
        "\n",
        "```\n",
        "        DataFrame.from_dict(data, orient='columns', dtype=None,columns=None)\n",
        "        # the default orientation is columns\n",
        "```\n",
        "\n",
        "\n",
        "3. Change the orientation of the data in the dataframe where\n",
        "  * the outer keys will become the dataframe labeled index,\n",
        "  * the inner keys will become the dataframe columns,\n",
        "  * and the values (of the inner dictionary) will be the dataframe rows\n",
        "  * see the sample DataFrame output below\n",
        "* a.  Do this by creating another Pandas DataFrame from the nested dictionary where with orient='index'\n",
        "* b. Print out the dataframe, its columns, and its indexes.\n",
        "For the columns and indexes, add .to_list() at the end to get a nicer printout\n",
        " <br><br>\n",
        "-------\n",
        "\n",
        "![picture1](https://drive.google.com/uc?export=view&id=1LDA50k3icuDjo5jVlB0ZDDzBqeE7Ez5g)\n",
        "<br>\n",
        "Figure 2 - Sample dataframe\n",
        " * species names are the labels/index\n",
        " * the nucs are the columns\n",
        " * the counts are the dataframe values (across each row)\n",
        "------\n",
        "  \n",
        "  \n"
      ],
      "metadata": {
        "id": "XbMP1TJ1rHMq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Reference: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.from_dict.html\n",
        "'''Your code here'''"
      ],
      "metadata": {
        "id": "Cn97nxh4kDZa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Prompt 4"
      ],
      "metadata": {
        "id": "ejvGuoukxg88"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "4. Plot a bar plot with the species on the x-axis and the count of nucleotides on the y-axis. Use a Pandas method to plot\n",
        " * in this case, which dataframe format would you use as the data source for the plot?"
      ],
      "metadata": {
        "id": "O-HwIdiqxkcM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''Your code here'''"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "1oT_-_Ifl0dx",
        "outputId": "223b0710-5f3f-413e-f3b0-9d251fc4b61b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Your code here'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## prompt 5 & 6"
      ],
      "metadata": {
        "id": "4IOwxEGjyWE-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the given code below before starting prompt 5 and 6 to obtain the CpG counts per 1000 bp for the species real sequences and for the randomly generated sequences"
      ],
      "metadata": {
        "id": "ZW28zIw7BfAF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\"\"\"\n",
        "Provided code block 4 - random and CpG sites\n",
        "\"\"\"\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive/')\n",
        "\n",
        "\n",
        "\n",
        "import random\n",
        "import numpy as np\n",
        "\n",
        "def randomSeqGenerator(sequencesDict) ->dict:\n",
        "  randomSequencesDict ={}\n",
        "  for name,seq in sequencesDict.items():\n",
        "    tempSeq = ''.join(random.sample(seq,len(seq)))\n",
        "    randomSequencesDict[name] = tempSeq\n",
        "  return randomSequencesDict\n",
        "\n",
        "randomSeqsDict = randomSeqGenerator(sequencesDict)\n",
        "# print(randomSeqsDict)\n",
        "\n",
        "def calCpGsites(sequencesDict, step=1000)->dict:\n",
        "  cgCountDict = {}\n",
        "  for name,seq in sequencesDict.items():\n",
        "    cgCountDict[name] = {}\n",
        "    listOfCGs = []\n",
        "    #iterate over dna sequence by step\n",
        "    for i in range(0, len(seq), step):\n",
        "      countPerStep=0\n",
        "      if len(seq) <= i+step:\n",
        "        countPerStep = seq[i:len(seq)].count(\"CG\")\n",
        "      else:\n",
        "        countPerStep = seq[i:i+step].count(\"CG\")\n",
        "      listOfCGs.append(countPerStep)\n",
        "\n",
        "    cgCountDict[name] = listOfCGs\n",
        "  return cgCountDict\n",
        "\n",
        "\n",
        "realCgCountPerStepDict =  calCpGsites(sequencesDict,1000)\n",
        "print(realCgCountPerStepDict)\n",
        "\n",
        "randomCgCountPerStepDict = calCpGsites(randomSeqsDict,1000)\n",
        "print(randomCgCountPerStepDict)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gh87wVBWro6q",
        "outputId": "6c07a4f9-5781-4b9f-dc62-a1a62f74a721"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'Sars_Cov2_NC_004718_3': [11, 12, 17, 12], 'ΗCoV_EMC_019843_3': [20, 17, 24, 19, 5], 'BatSarsCov_ FJ588692.1': [18, 20, 17, 15], 'CCR5': [7, 7, 13, 16, 7, 7, 0], 'HIV1': [22, 8, 1, 1, 5, 7, 7, 11, 17, 3], 'H5N8': [31], 'Malaria': [31, 33, 15], 'cat': [67, 17, 0], 'PolarBear_mRna': [60], 'HPV': [6]}\n",
            "{'Sars_Cov2_NC_004718_3': [31, 32, 44, 27], 'ΗCoV_EMC_019843_3': [30, 43, 43, 48, 1], 'BatSarsCov_ FJ588692.1': [50, 39, 47, 20], 'CCR5': [40, 44, 47, 43, 41, 49, 5], 'HIV1': [41, 48, 50, 43, 51, 31, 42, 45, 37, 5], 'H5N8': [59], 'Malaria': [90, 85, 59], 'cat': [74, 59, 0], 'PolarBear_mRna': [88], 'HPV': [8]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "5. Create a DataFrame from the dictionaries above, realCgCountPerStepDict and randomCgCountPerStepDict where\n",
        "* the dictionary keys are the dataframe indexes\n",
        "* the columns will be the default numbered indexes of the dictionary values, which are lists of CpG counts/1000 bp\n",
        "* see the sample dataframe below\n",
        " <br><br>\n",
        "-------\n",
        "\n",
        "![picture1](https://drive.google.com/uc?export=view&id=1rS7oOyGqqXZAvAKh3dq4bCodl42eL-J_)\n",
        "<br>\n",
        "Figure 3 - Sample dataframe\n",
        " * Dataframe automatically fills empty cells with Numpy.NaN to make the rows the same size\n",
        "------\n"
      ],
      "metadata": {
        "id": "RYWCuiIBCZZn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Think of what is required as index and what is required to be the column\n",
        "# compare with prompt 2\n",
        "'''Your code here'''"
      ],
      "metadata": {
        "id": "2QWm5vd4mIc2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "6. Calculate the mean of each species' CpG counts per 1000 bp - both real and randomly generated. Print them out\n",
        "  * Pandas is built on Numpy and it also has a mean(), and it has the option to calculate the mean with values down each row, and down each column for example. This is done with the parameter axis\n",
        "\n",
        "\n",
        "```\n",
        "    DataFrame.mean(axis=0, skipna=True, numeric_only=False, **kwargs)\n",
        "    # where axis specifies how the mean will be calculated, by index with axis=0, or by columns with axis=1\n",
        "```\n",
        "  * along which axis do you think the mean of each species CpG count/1000 bp should be calculated?\n",
        "\n"
      ],
      "metadata": {
        "id": "Iq176co0GRgw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# reference:\n",
        "# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mean.html\n",
        "# https://www.geeksforgeeks.org/python-pandas-dataframe-mean/\n",
        "\n",
        "'''Your code here'''"
      ],
      "metadata": {
        "id": "LgrI_z5Rm_y6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prompt 7"
      ],
      "metadata": {
        "id": "m7SHn6m8LRv9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "7. Plot the means of real and random CpG sites counts per 1000 bp, to visualize the significance of CpG sites analaysis.\n",
        " * plot a bar graph showing each species (x-axis) and the mean of CpG/1000pb (y-axis) for real and random versions. There will be 2 stacked bars for each species, the real and the random.\n",
        " * use Pandas plotting functions such as plot(kind='bar')\n",
        " * see sample plot below\n",
        " <br><br>\n",
        "-------\n",
        "\n",
        "![picture1](https://drive.google.com/uc?export=view&id=1yqa_EHN_mAIioa1cK7bNtfN6KNj7WTuL)\n",
        "<br>\n"
      ],
      "metadata": {
        "id": "lGTsOMyVYtN-"
      }
    }
  ]
}