ccf-api-usecase.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a398796e",
   "metadata": {},
   "source": [
    "### HRA API UseCase Documentation\n",
    "\n",
    "[Click here](https://github.com/hubmapconsortium/ccf-ui/blob/main/ccf-api-usage.ipynb) to view the installation and general documentation for ccf-api python module.\n",
    "\n",
    "In this notebook we will be demonstrating one particular usecase for which the CCF API Module can be utilized. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a25dd6d2",
   "metadata": {},
   "source": [
    "#### We first import all the necessary packages from the ccf_openapi_client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3ca29d53",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ccf_openapi_client\n",
    "from ccf_openapi_client.api import default_api"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c8bcc20",
   "metadata": {},
   "source": [
    "#### Import all other packages required to produce output as expected.\n",
    "The pandas module is used for a dataframe, which is used for displaying and aggregating. <br>\n",
    "The csv module is used for writing all the data into a csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e2a6846a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d626c0ef",
   "metadata": {},
   "source": [
    "#### Configuration\n",
    "You'll need to point the host in the configuration to the instance you'd like to work with. \n",
    "More Info [here](https://github.com/hubmapconsortium/ccf-ui/blob/main/ccf-api-usage.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "914a1924",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "configuration = ccf_openapi_client.Configuration(\n",
    "    host = \"https://apps.humanatlas.io/hra-api--staging/v1\"\n",
    ")\n",
    "api_client = ccf_openapi_client.ApiClient(configuration)\n",
    "\n",
    "api_instance = default_api.DefaultApi(api_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71827e1b",
   "metadata": {},
   "source": [
    "##### Check Database Status\n",
    "This is a optional step which can be used to reduce wait times for other methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "73bb3465",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Database ready!\n",
      " {'checkback': 3600000,\n",
      " 'load_time': 169445,\n",
      " 'message': 'Database successfully loaded',\n",
      " 'status': 'Ready'}\n"
     ]
    }
   ],
   "source": [
    "db_ready = False\n",
    "result = None\n",
    "while not db_ready:\n",
    "    result = api_instance.db_status()\n",
    "    if result['status'] == 'Ready':\n",
    "        db_ready = True\n",
    "    else:\n",
    "        print('Database not ready yet! Retrying...', result)\n",
    "        time.sleep(2)\n",
    "print('Database ready!\\n', result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bda78fb",
   "metadata": {},
   "source": [
    "### Our use case is using Kidney as the reference Organ.\n",
    "We are using the ontology link for kidney directly: http://purl.obolibrary.org/obo/UBERON_0002113"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee3ebe89",
   "metadata": {},
   "source": [
    "The scene method gives us information about the Anatomical Structures that the Kidney collides with. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "09e964f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sex = \"both\"\n",
    "ontology_terms = [\"http://purl.obolibrary.org/obo/UBERON_0002113\",]\n",
    "sceneResult = None\n",
    "try:\n",
    "    sceneResult = api_instance.scene(sex=sex, ontology_terms=ontology_terms)\n",
    "except ccf_openapi_client.ApiException as e:\n",
    "    print(\"Exception when calling DefaultApi->aggregate_results: %s\\n\" % e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c548879",
   "metadata": {},
   "source": [
    "We convert the sceneResult into a dictionary where the key is the entity_id and it contains an array of the Anatomical Structures that collide with Kidney."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f47ce87c",
   "metadata": {},
   "outputs": [],
   "source": [
    "sceneEntityASDict = {}\n",
    "for scene in sceneResult:\n",
    "    if 'entity_id' in scene:\n",
    "        if scene['entity_id'] not in sceneEntityASDict:\n",
    "            sceneEntityASDict[scene['entity_id']] = []\n",
    "        sceneEntityASDict[scene['entity_id']].extend(scene['ccf_annotations'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a911543",
   "metadata": {},
   "source": [
    "We then use a SPARQL query to get the AS label, CT Label and the CT iri using the AS iri's we obtained in the last step. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9b303ce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = '''PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n",
    "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n",
    "PREFIX ccf: <http://purl.org/ccf/>\n",
    "\n",
    "SELECT DISTINCT (STR(?asLabel) as ?as_label) (STR(?qlabel) as ?cell_label) ?as_iri ?cell_iri WHERE {\n",
    "  ?cell_iri ccf:ccf_located_in ?as_iri .\n",
    "  ?cell_iri rdfs:label ?qlabel .\n",
    "  ?as_iri rdfs:label ?asLabel .\n",
    "\n",
    "  FILTER (?as_iri in (%s))\n",
    "}'''\n",
    "\n",
    "purlIris = set()\n",
    "# Collecting all unique AS iri's\n",
    "for k in sceneEntityASDict.keys():\n",
    "    for v in sceneEntityASDict[k]:\n",
    "        purlIris.add(v)\n",
    "        \n",
    "\n",
    "purlString = \", \".join(\"<\" + s + \">\" for s in purlIris)\n",
    "queryResponse = None\n",
    "try:\n",
    "    queryResponse = api_instance.sparql(query=query % purlString, format='application/json')\n",
    "except ccf_openapi_client.ApiException as e:\n",
    "    print(\"Exception when calling DefaultApi->aggregate_results: %s\\n\" % e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc244f9",
   "metadata": {},
   "source": [
    "We now reverse the sceneEntityASDict to now have AS: entity_id relation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ea33611b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def reverseObject(input_object):\n",
    "    reversed_object = {}\n",
    "\n",
    "    for key, values in input_object.items():\n",
    "        for value in values:\n",
    "            if value not in reversed_object:\n",
    "                reversed_object[value] = set()\n",
    "            reversed_object[value].add(key)\n",
    "    return reversed_object\n",
    "\n",
    "ASEntityDict = reverseObject(sceneEntityASDict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f6e9b7d",
   "metadata": {},
   "source": [
    "Using the method `Tissue Blocks` we get donor label information for each entity_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e0e8542e",
   "metadata": {},
   "outputs": [],
   "source": [
    "sex = \"both\"\n",
    "ontology_terms = [\"http://purl.obolibrary.org/obo/UBERON_0002113\",]\n",
    "tissueBlockResult = None\n",
    "try:\n",
    "    tissueBlockResult = api_instance.tissue_blocks(sex=sex, ontology_terms=ontology_terms)\n",
    "except ccf_openapi_client.ApiException as e:\n",
    "    print(\"Exception when calling DefaultApi->aggregate_results: %s\\n\" % e)\n",
    "\n",
    "# Convert the TissueBlock Object to a dict\n",
    "tissueBlockResultList = list(map(lambda b: b.to_dict(), tissueBlockResult))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24714129",
   "metadata": {},
   "source": [
    "To make it easier to find the donor labels we convert the above result to a dictionary that related the entity_id to the donor_label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "8ea49ff9",
   "metadata": {},
   "outputs": [],
   "source": [
    "blockDonorLabel = {}\n",
    "for tissueBlock in tissueBlockResultList:\n",
    "    blockDonorLabel[tissueBlock['@id']] = tissueBlock['donor']['label']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d563e33e",
   "metadata": {},
   "source": [
    "Now we create the report, using data from the above methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bfa1a33f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Can't use set because - unhashable type: 'dict'\n",
    "mergedData = []\n",
    "for response in queryResponse:\n",
    "    entity_ids = ASEntityDict[response['as_iri']]\n",
    "    for e in entity_ids:\n",
    "        newResponse = response.copy()\n",
    "        newResponse['block_id'] = e\n",
    "        newResponse['donor_label'] = blockDonorLabel[e]\n",
    "        if newResponse not in mergedData:\n",
    "            mergedData.append(newResponse)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a347cc2c",
   "metadata": {},
   "source": [
    "This will save the output as a csv file. <br>\n",
    "You can change the name of the file in first line of the next block. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "60d340a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "fileName = \"exampleFileName\"\n",
    "import csv\n",
    "def save_to_csv(data, headers, filename):\n",
    "     with open(filename, 'w', newline='') as file:\n",
    "        writer = csv.writer(file)\n",
    "\n",
    "        # Write the headers\n",
    "        writer.writerow(headers)\n",
    "\n",
    "        # Write the data\n",
    "        for obj in data:\n",
    "            row = [obj.get(headers[header], '') for header in headers]\n",
    "            writer.writerow(row)\n",
    "\n",
    "header_mapping = {\n",
    "    'Block Id': 'block_id',\n",
    "    'AS': 'as_iri',\n",
    "    'AS Label': 'as_label',\n",
    "    'CT': 'cell_iri',\n",
    "    'CT Label': 'cell_label',\n",
    "    'Donor Label':'donor_label'\n",
    "}\n",
    "save_to_csv(mergedData, header_mapping, f'${fileName}.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e904d581",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cell_label</th>\n",
       "      <th>as_label</th>\n",
       "      <th>cell_iri</th>\n",
       "      <th>as_iri</th>\n",
       "      <th>block_id</th>\n",
       "      <th>donor_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"T cell\"</td>\n",
       "      <td>\"kidney\"</td>\n",
       "      <td>http://purl.obolibrary.org/obo/CL_0000084</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002113</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Male, Age 65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"T cell\"</td>\n",
       "      <td>\"kidney\"</td>\n",
       "      <td>http://purl.obolibrary.org/obo/CL_0000084</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002113</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Female, Age 57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"T cell\"</td>\n",
       "      <td>\"kidney\"</td>\n",
       "      <td>http://purl.obolibrary.org/obo/CL_0000084</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002113</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Male, Age 66, BMI 29.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"T cell\"</td>\n",
       "      <td>\"kidney\"</td>\n",
       "      <td>http://purl.obolibrary.org/obo/CL_0000084</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002113</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Male, Age 41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"T cell\"</td>\n",
       "      <td>\"kidney\"</td>\n",
       "      <td>http://purl.obolibrary.org/obo/CL_0000084</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002113</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Female, Age 45, BMI 22.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6106</th>\n",
       "      <td>\"capsule mesenchymal stromal cell\"</td>\n",
       "      <td>\"kidney capsule\"</td>\n",
       "      <td>https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002015</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Male, Age 66, BMI 31.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6107</th>\n",
       "      <td>\"capsule mesenchymal stromal cell\"</td>\n",
       "      <td>\"kidney capsule\"</td>\n",
       "      <td>https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002015</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Male, Age 28, BMI 19.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6108</th>\n",
       "      <td>\"capsule mesenchymal stromal cell\"</td>\n",
       "      <td>\"kidney capsule\"</td>\n",
       "      <td>https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002015</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Female, Age 55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6109</th>\n",
       "      <td>\"capsule mesenchymal stromal cell\"</td>\n",
       "      <td>\"kidney capsule\"</td>\n",
       "      <td>https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002015</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Female, Age 69, BMI 49.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6110</th>\n",
       "      <td>\"capsule mesenchymal stromal cell\"</td>\n",
       "      <td>\"kidney capsule\"</td>\n",
       "      <td>https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...</td>\n",
       "      <td>http://purl.obolibrary.org/obo/UBERON_0002015</td>\n",
       "      <td>https://entity.api.hubmapconsortium.org/entiti...</td>\n",
       "      <td>Male, Age 78</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6111 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                              cell_label          as_label  \\\n",
       "0                               \"T cell\"          \"kidney\"   \n",
       "1                               \"T cell\"          \"kidney\"   \n",
       "2                               \"T cell\"          \"kidney\"   \n",
       "3                               \"T cell\"          \"kidney\"   \n",
       "4                               \"T cell\"          \"kidney\"   \n",
       "...                                  ...               ...   \n",
       "6106  \"capsule mesenchymal stromal cell\"  \"kidney capsule\"   \n",
       "6107  \"capsule mesenchymal stromal cell\"  \"kidney capsule\"   \n",
       "6108  \"capsule mesenchymal stromal cell\"  \"kidney capsule\"   \n",
       "6109  \"capsule mesenchymal stromal cell\"  \"kidney capsule\"   \n",
       "6110  \"capsule mesenchymal stromal cell\"  \"kidney capsule\"   \n",
       "\n",
       "                                               cell_iri  \\\n",
       "0             http://purl.obolibrary.org/obo/CL_0000084   \n",
       "1             http://purl.obolibrary.org/obo/CL_0000084   \n",
       "2             http://purl.obolibrary.org/obo/CL_0000084   \n",
       "3             http://purl.obolibrary.org/obo/CL_0000084   \n",
       "4             http://purl.obolibrary.org/obo/CL_0000084   \n",
       "...                                                 ...   \n",
       "6106  https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...   \n",
       "6107  https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...   \n",
       "6108  https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...   \n",
       "6109  https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...   \n",
       "6110  https://purl.org/ccf/ASCTB-TEMP_capsule-mesenc...   \n",
       "\n",
       "                                             as_iri  \\\n",
       "0     http://purl.obolibrary.org/obo/UBERON_0002113   \n",
       "1     http://purl.obolibrary.org/obo/UBERON_0002113   \n",
       "2     http://purl.obolibrary.org/obo/UBERON_0002113   \n",
       "3     http://purl.obolibrary.org/obo/UBERON_0002113   \n",
       "4     http://purl.obolibrary.org/obo/UBERON_0002113   \n",
       "...                                             ...   \n",
       "6106  http://purl.obolibrary.org/obo/UBERON_0002015   \n",
       "6107  http://purl.obolibrary.org/obo/UBERON_0002015   \n",
       "6108  http://purl.obolibrary.org/obo/UBERON_0002015   \n",
       "6109  http://purl.obolibrary.org/obo/UBERON_0002015   \n",
       "6110  http://purl.obolibrary.org/obo/UBERON_0002015   \n",
       "\n",
       "                                               block_id  \\\n",
       "0     https://entity.api.hubmapconsortium.org/entiti...   \n",
       "1     https://entity.api.hubmapconsortium.org/entiti...   \n",
       "2     https://entity.api.hubmapconsortium.org/entiti...   \n",
       "3     https://entity.api.hubmapconsortium.org/entiti...   \n",
       "4     https://entity.api.hubmapconsortium.org/entiti...   \n",
       "...                                                 ...   \n",
       "6106  https://entity.api.hubmapconsortium.org/entiti...   \n",
       "6107  https://entity.api.hubmapconsortium.org/entiti...   \n",
       "6108  https://entity.api.hubmapconsortium.org/entiti...   \n",
       "6109  https://entity.api.hubmapconsortium.org/entiti...   \n",
       "6110  https://entity.api.hubmapconsortium.org/entiti...   \n",
       "\n",
       "                   donor_label  \n",
       "0                 Male, Age 65  \n",
       "1               Female, Age 57  \n",
       "2       Male, Age 66, BMI 29.1  \n",
       "3                 Male, Age 41  \n",
       "4     Female, Age 45, BMI 22.6  \n",
       "...                        ...  \n",
       "6106    Male, Age 66, BMI 31.4  \n",
       "6107    Male, Age 28, BMI 19.2  \n",
       "6108            Female, Age 55  \n",
       "6109  Female, Age 69, BMI 49.1  \n",
       "6110              Male, Age 78  \n",
       "\n",
       "[6111 rows x 6 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(mergedData)\n",
    "df.head(len(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "11389013",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUMMARY:\n",
      "Number of Blocks: 83,\n",
      "Number of Unique ASs identified: 8,\n",
      "Number of Unique CTs identified: 68\n"
     ]
    }
   ],
   "source": [
    "# Summary -- texttt\n",
    "AsCounts = df['as_iri'].value_counts()\n",
    "CtCounts = df['cell_iri'].value_counts()\n",
    "blockCounts = df['block_id'].value_counts()\n",
    "\n",
    "print(f'SUMMARY:\\n\\\n",
    "Number of Blocks: {len(blockCounts)},\\n\\\n",
    "Number of Unique ASs identified: {len(AsCounts)},\\n\\\n",
    "Number of Unique CTs identified: {len(CtCounts)}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}