notebooks/Ontology_Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "\n",
    "<img width='700' src=\"https://user-images.githubusercontent.com/8030363/108961534-b9a66980-7634-11eb-96e2-cc46589dcb8c.png\" style=\"vertical-align:middle\">\n",
    "\n",
    "## Pre-Knowledge Graph Build Ontology Cleaning\n",
    "***\n",
    "***\n",
    "\n",
    "**Author:** [TJCallahan](https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=callahantiff@gmail.com)  \n",
    "**GitHub Repository:** [PheKnowLator](https://github.com/callahantiff/PheKnowLator/wiki)  \n",
    "**Release:** **[v2.0.0](https://github.com/callahantiff/PheKnowLator/wiki/v2.0.0)**\n",
    "  \n",
    "\n",
    "## Purpose\n",
    "\n",
    "This notebook serves as a script to help prepare ontologies prior to be ingested into the knowledge graph build algorithm. This script performs the following steps:  \n",
    "1. [Clean Ontologies](#clean-ontologies)  \n",
    "2. [Merge Ontologies](#merge-ontologies)  \n",
    "\n",
    "## Assumptions and Dependencies  \n",
    "  \n",
    "**Assumptions:**   \n",
    "- Directory of Imported Ontologies has been populated ➞ `./resources/ontologies`     \n",
    "\n",
    "**Dependencies:**   \n",
    "- <u>Scripts</u>: This notebook utilizes several helper functions from the following scripts:  \n",
    "  - [utility scripts](https://github.com/callahantiff/PheKnowLator/blob/master/pkt_kg/utils)  \n",
    "  - [ontology_cleaning.py](https://github.com/callahantiff/PheKnowLator/blob/master/builds/ontology_cleaning.py) \n",
    "- <u>Software</u>: [OWLTools](https://github.com/owlcollab/owltools)  \n",
    "- <u>Data</u>: [`Merged_gene_rna_protein_identifiers.pkl`](https://storage.googleapis.com/pheknowlator/current_build/data/processed_data/Merged_gene_rna_protein_identifiers.pkl), which is automatically downloaded to the `./resources/ontologies` directory     \n",
    "\n",
    "<br>\n",
    "\n",
    "Details on the data utilized in this script can be found on the [Data Sources](https://github.com/callahantiff/PheKnowLator/wiki/v2-Data-Sources) Wiki. Data can be downloaded from [Zenodo](https://zenodo.org/doi/10.5281/zenodo.7030039). Please note that all build data are freely available and organized by release and build date. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "***\n",
    "### CLEAN ONTOLOGIES <a class=\"anchor\" id=\"clean-ontologies\"></a>\n",
    "***\n",
    "\n",
    "The ontology cleaning step includes the following error checks, each of which are explained below and each of which are applied to individual ontologies, the set of merged ontologies or both: (1) Value Errors, (2) Identifier Errors, (3) Duplicate and Obsolete Entities, (4) Punning Errors, and (5) Entity Normalization Errors.\n",
    "\n",
    "<br>\n",
    "\n",
    "### Value Errors  \n",
    "*** \n",
    "**Level:** `individual ontology`; `merged-ontology`    \n",
    "\n",
    "**Description:** This check utilizes the [`owlready2`](https://pypi.org/project/Owlready2/) library to read in each of the ontologies. This library is strict and will catch a wide variety of value errors. \n",
    "\n",
    "**Solution:** Parse the error message using the provided `ErrorType` and line number and repair it. For `ValueErrors` incorrectly typed input are re-typed.\n",
    "\n",
    "*Example Findings*  \n",
    "The [Cell Line Ontology](http://www.clo-ontology.org/) yield the following error message:\n",
    "\n",
    "```python\n",
    "ValueError: invalid literal for int() with base 10: '永生的乳腺衍生细胞系细胞'\n",
    "...\n",
    "OwlReadyOntologyParsingError: RDF/XML parsing error in file clo_with_imports.owl, line 10970, column 99.\n",
    "```\n",
    "\n",
    "This tells us that we need to repair the triple containing the Literal '永生的乳腺衍生细胞系细胞' by removing it and redefining it as a `string`, rather than an `int` as it is currently defined as. This is currently noted as an issue in the [Cell Line Ontology's](http://www.clo-ontology.org/) GitHub repo ([issue #48](https://github.com/CLO-ontology/CLO/issues/48)). \n",
    "\n",
    "<br>\n",
    "\n",
    "### Identifier Errors  \n",
    "*** \n",
    "**Level:** `individual ontology`; `merged-ontology`  \n",
    "\n",
    "**Description:** This check verifies consistency of identifier prefixes. For example, we want to find identifiers that are incorrectly formatted like occurrences of `PRO_XXXXXXX` which should be `PR_XXXXXXX`.\n",
    "\n",
    "**Solution:** Incorrectly formatted class identifiers are updated. This is a tricky task to do in an automated manner and is something that should be updated if any new ontologies are added to the `PheKnowLator` build. Currently, the code below checks and logs any hits, but only fixes the following known errors: Vaccine Ontology: `PRO` which should be `PR`.\n",
    "\n",
    "*Example Findings*  \n",
    "Running this check revealed mislabeling of `2` [pROtein Ontology](https://proconsortium.org/) identifiers in the [Vaccine Ontology](http://www.violinet.org/vaccineontology/) (see [this](https://github.com/vaccineontology/VO/issues/4) GitHub issue).\n",
    "\n",
    "<br>\n",
    "\n",
    "### Obsolete and/or Deprecated Entities\n",
    "*** \n",
    "**Level:** `individual ontology`; `merged-ontology`  \n",
    "\n",
    "**Description:** Verify that the ontology only contains current content.\n",
    "\n",
    "**Solution:** All obsolete classes and any triples that they participate in are removed from an ontologies.\n",
    "\n",
    "<br>\n",
    "\n",
    "### Normalization Errors  \n",
    "*** \n",
    "**Level:** `merged-ontology`\n",
    "\n",
    "These checks are performed at the individual- and merged-ontology levels. There are two types of checks that are performed:  \n",
    "\n",
    "<u>Normalize Existing Ontology Classes</u>  \n",
    "  - **Description:** Checks for inconsistencies in ontology classes that overlap with non-ontology entity identifiers (e.g. if HP includes `HGNC` identifiers, but PheKnowLator utilizes `Entrez` identifiers). \n",
    "\n",
    "  - **Solution:** While there are other types of identifiers, we currently focus primarily on resolving errors involving the genomic identifiers, since we have a master dictionary we can use (`Merged_gene_rna_protein_identifiers.pkl` -- which is generated during the data preporcessing steps of the build). This check can be updated in future iterations to include other types of identifiers, but given our detailed examination of the `v2.0.0` ontologies, these were the identifier types that needed repair.\n",
    "\n",
    "<u>Normalize Duplicate Ontology Concepts</u>  \n",
    "  - **Description:** Make sure that all classes that represent the same entity are connected to each other. For example, consider the following: the [Sequence Ontology](http://www.sequenceontology.org/), [ChEBI](https://www.ebi.ac.uk/chebi), and [PRotein Ontology](https://proconsortium.org/) all include terms for protein, but none of these classes are connected to each other.\n",
    "\n",
    "  - **Solution:** Choose a primary concept for all duplicate scenarios and make duplicate concepts an `RDFS:subClassOf` the primary concept. In the future, this check could be improved by leveraging [KBOOM](https://www.biorxiv.org/content/10.1101/048843v3).\n",
    "\n",
    "*Example Findings*  \n",
    "The follow classes occur in all of the ontologies used in the current build and have to be normalized so that there are not multiple versions of the same concept:  \n",
    "\n",
    "- Gene: [VO](http://purl.obolibrary.org/obo/OGG_0000000002)  \n",
    "  - <u>Solution</u>: Make the `VO` imported `OGG` class a subclass of the `SO` gene term  \n",
    "\n",
    "- Protein: [SO](http://purl.obolibrary.org/obo/SO_0000104), [PRO](http://purl.obolibrary.org/obo/PR_000000001), [ChEBI](http://purl.obolibrary.org/obo/CHEBI_36080) \n",
    "  - <u>Solution</u>: Make the `CHEBI` and `PRO` classes a subclass of the `SO` protein term  \n",
    "  \n",
    "- Disorder: [VO](http://purl.obolibrary.org/obo/OGMS_0000045)  \n",
    "  - <u>Solution</u>: Make the `VO` imported `OGMS` class a subclass of the `MONDO` disease term  \n",
    "\n",
    "- Antigen: [VO](http://purl.obolibrary.org/obo/OBI_1110034)  \n",
    "  - <u>Solution</u>: Make the `VO` imported OBI class a subclass of the `CHEBI` antigen term  \n",
    "\n",
    "- Gelatin: [VO]('http://purl.obolibrary.org/obo/VO_0003030') \n",
    "  - <u>Solution</u>: Make the `VO` class a subclass of the `CHEBI` gelatin term \n",
    "\n",
    "- Hormone: [VO](http://purl.obolibrary.org/obo/FMA_12278) \n",
    "  - <u>Solution</u>: Make the `VO` imported `FMA` class a subclass of the `CHEBI` hormone term\n",
    "\n",
    "<br>\n",
    "\n",
    "### Punning Errors \n",
    "*** \n",
    "**Level:** `individual ontology`; `merged-ontology`\n",
    "\n",
    "**Description:** [Punning](https://www.w3.org/2007/OWL/wiki/Punning) or redeclaration errors occur for a few different reasons, but the primary or most prevalent cause observed in the ontologies used in `PheKnowLator` is due to an `owl:ObjectProperty` being incorrectly redeclared as an `owl:AnnotationProperty` or an `owl:Class` also being defined as an `OWL:ObjectProperty`. \n",
    "\n",
    "**Solution:** Consistent with the solution described [here](https://github.com/oborel/obo-relations/issues/130), for `owl:ObjectProperty` redeclarations we remove all `owl:AnnotationProperty` declarations. For all `owl:Class` redeclarations, we remove all `owl:ObjectProperty` redeclarations. \n",
    "\n",
    "*Example Findings* \n",
    "The [Cell Line Ontology](http://www.clo-ontology.org/) had 7 object properties that were illegally redeclared and triggered punning errors. More details regarding these errors are shown below. \n",
    "\n",
    "```bash\n",
    "2020-12-03 20:57:15,616 ERROR (OWLOntologyManagerImpl:1138) Illegal redeclarations of entities: reuse of entity http://purl.obolibrary.org/obo/RO_0002091 in punning not allowed [Declaration(AnnotationProperty(<http://purl.obolibrary.org/obo/RO_0002091>)), Declaration(ObjectProperty(<http://purl.obolibrary.org/obo/RO_0002091>))]\n",
    "2020-12-03 20:57:15,619 ERROR (OWLOntologyManagerImpl:1138) Illegal redeclarations of entities: reuse of entity http://purl.obolibrary.org/obo/BFO_0000062 in punning not allowed [Declaration(AnnotationProperty(<http://purl.obolibrary.org/obo/BFO_0000062>)), Declaration(ObjectProperty(<http://purl.obolibrary.org/obo/BFO_0000062>))]\n",
    "2020-12-03 20:57:15,620 ERROR (OWLOntologyManagerImpl:1138) Illegal redeclarations of entities: reuse of entity http://purl.obolibrary.org/obo/BFO_0000063 in punning not allowed [Declaration(ObjectProperty(<http://purl.obolibrary.org/obo/BFO_0000063>)), Declaration(AnnotationProperty(<http://purl.obolibrary.org/obo/BFO_0000063>))]\n",
    "2020-12-03 20:57:15,620 ERROR (OWLOntologyManagerImpl:1138) Illegal redeclarations of entities: reuse of entity http://purl.obolibrary.org/obo/RO_0002222 in punning not allowed [Declaration(AnnotationProperty(<http://purl.obolibrary.org/obo/RO_0002222>)), Declaration(ObjectProperty(<http://purl.obolibrary.org/obo/RO_0002222>))]\n",
    "2020-12-03 20:57:15,620 ERROR (OWLOntologyManagerImpl:1138) Illegal redeclarations of entities: reuse of entity http://purl.obolibrary.org/obo/RO_0000087 in punning not allowed [Declaration(ObjectProperty(<http://purl.obolibrary.org/obo/RO_0000087>)), Declaration(AnnotationProperty(<http://purl.obolibrary.org/obo/RO_0000087>))]\n",
    "2020-12-03 20:57:15,620 ERROR (OWLOntologyManagerImpl:1138) Illegal redeclarations of entities: reuse of entity http://purl.obolibrary.org/obo/RO_0002161 in punning not allowed [Declaration(ObjectProperty(<http://purl.obolibrary.org/obo/RO_0002161>)), Declaration(AnnotationProperty(<http://purl.obolibrary.org/obo/RO_0002161>))]\n",
    "```\n",
    "\n",
    "From this message, we can see that we need to remove the following `owl:ObjectProperty` redeclared to `owl:AnnotationProperty`: `RO_0002091`, `BFO_0000062`, `BFO_0000063`, `RO_0002222`, `RO_0000087`, `RO_0002161`. There were also 2 classes (i.e. `CLO_0054407` and `CLO_0054409`) defined as being a `owl:Class` and an `owl:ObjectProperty`. This is currently noted as an issue in the Cell Line Ontology's GitHub repo [issue #43](https://github.com/CLO-ontology/CLO/issues/43))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "***  \n",
    "### Set-Up Environment\n",
    "***  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # uncomment and run to install any required modules from notebooks/requirements.txt\n",
    "# import sys\n",
    "# !{sys.executable} -m pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to ensure builds/*.py files and pkt_kg scripts can be reached from notebooks dir\n",
    "import sys\n",
    "sys.path.append('../')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Needed Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import needed libraries\n",
    "import datetime\n",
    "import glob\n",
    "import pickle\n",
    "import shutil\n",
    "\n",
    "from rdflib import Graph\n",
    "from tqdm import tqdm\n",
    "\n",
    "# import script containing helper functions\n",
    "from pkt_kg.utils import * \n",
    "from builds.ontology_cleaning import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set Global Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up environment variables\n",
    "write_location = '../resources/ontologies'\n",
    "knowledge_graphs_location = '../resources/knowledge_graphs'\n",
    "processed_data_location = '../resources/processed_data/'\n",
    "\n",
    "# set global namespaces\n",
    "schema = Namespace('http://www.w3.org/2001/XMLSchema#')\n",
    "obo = Namespace('http://purl.obolibrary.org/obo/')\n",
    "oboinowl = Namespace('http://www.geneontology.org/formats/oboInOwl#')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions needed for processing ontologies\n",
    "def logically_verifies_cleaned_ontologies(graph, temp_dir, file_location, owltools_location):\n",
    "    \"\"\"Logically verifies an ontology by running the ELK deductive logic reasoner. Before running the reasoner\n",
    "    the instantiated RDFLib object is saved locally.\n",
    "\n",
    "    Args:\n",
    "        graph: An RDFLib Graph object containing data.\n",
    "        temp_dir: A string specifying where where to read from and write to.\n",
    "        file_location: The name of the file to read and write to in the temp_dir directory.\n",
    "        owltools_location: A string specifying the location of OWLTOOLs (included in pkt_kg no need to download).\n",
    "    \n",
    "    Returns:\n",
    "        None.\n",
    "    \"\"\"\n",
    "\n",
    "    print('Logically Verifying Ontology')\n",
    "\n",
    "    # save graph in order to run reasoner\n",
    "    filename = temp_dir + '/' + file_location\n",
    "    graph.serialize(destination=filename, format='xml')\n",
    "    \n",
    "    # run reasoner\n",
    "    command = \"{} {} --reasoner {} --run-reasoner --assert-implied -o {}\"\n",
    "    return_code = os.system(command.format(owltools_location, filename, 'elk', filename))\n",
    "    if return_code != 0: raise ValueError('Reasoner Finished with Errors.')\n",
    "\n",
    "    return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "***\n",
    "### INDIVIDUAL ONTOLOGIES <a class=\"anchor\" id=\"individual-ontologies\"></a>\n",
    "***\n",
    "\n",
    "**Purpose:** This section focuses on cleaning the individual ontologies which consists of fixing: (1) Parsing Errors; (2) Identifier Errors; (3) Deprecated and Obsolete Classes; and (4) Punning Errors.\n",
    "\n",
    "\n",
    "**Inputs:** A directory (`write_location`) containing ontology files (`.owl`)\n",
    "\n",
    "**Outputs:** A directory (`write_location`) containing cleaned ontology files (`.owl`)  \n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "### ⚡ Important ⚡\n",
    "\n",
    "The `OWL API`, when running the [ELK reasoner](), seems to add back some of the errors that this script removes.\n",
    "\n",
    "- <u>Example 1</u>: In the Vaccine Ontology, we fix prefix errors where `\"PR\"` is recorded as `\"PRO\"`. If you save the ontology without running the reasoner and reload it, the fix remains. If you open it after running ELK, the fix has been reversed. \n",
    "\n",
    "\n",
    "- <u>Example 2</u>: When we create the human subset of the Protein Ontology we verify that it contains only a single large connected component. If you re-calculate the number of connected components after running ELK, there will be three components.  \n",
    "\n",
    "Luckily, the merged ontologies are not logically verified using a reasoner, thus the version used to build knowledge graphs remains free of these errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "# instantiate and set-up class\n",
    "ont_data = OntologyCleaner('', '', '', write_location)\n",
    "\n",
    "# updating ontology info dictionary\n",
    "ont_data.ontology_info = {k.split('/')[-1]: {} for k, v in ont_data.ontology_info.items()}\n",
    "\n",
    "# set owl tools location\n",
    "ont_data.owltools_location = '../pkt_kg/libs/owltools'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "# clean data\n",
    "for ont in ont_data.ontology_info.keys():\n",
    "    print('\\n#### Processing Ontology: {} ####'.format(ont.upper()))\n",
    "    ont_data.ont_file_location = ont\n",
    "    ont_data.ont_graph = Graph().parse(ont_data.temp_dir + '/' + ont_data.ont_file_location)\n",
    "    \n",
    "    # get starting statistics\n",
    "    ont_data.updates_ontology_reporter()\n",
    "    \n",
    "    # clean ontologies\n",
    "    ont_data.fixes_ontology_parsing_errors()\n",
    "    ont_data.fixes_identifier_errors()\n",
    "    ont_data.removes_deprecated_obsolete_entities()\n",
    "    ont_data.fixes_punning_errors()\n",
    "    \n",
    "    # run cleaned ontology through the elk reasoner\n",
    "    logically_verifies_cleaned_ontologies(ont_data.ont_graph,\n",
    "                                          ont_data.temp_dir,\n",
    "                                          ont_data.ont_file_location,\n",
    "                                          ont_data.owltools_location)\n",
    "\n",
    "    # verifies no errors caused during cleaning\n",
    "#     ontology_file_formatter(ont_data.temp_dir, '/' + ont_data.ont_file_location, ont_data.owltools_location)\n",
    "    \n",
    "    # read in cleaned, verified, and updated ontology containing inference\n",
    "    print('Reading in Cleaned Ontology -- Needed to Calculate Final Statistics')\n",
    "    ont_data.ont_graph = Graph().parse(ont_data.temp_dir + '/' + ont_data.ont_file_location)\n",
    "    \n",
    "    # get finishing statistics\n",
    "    ont_data.updates_ontology_reporter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "***\n",
    "### MERGED ONTOLOGIES <a class=\"anchor\" id=\"merge-ontologies\"></a>\n",
    "***\n",
    "\n",
    "**Purpose:** In this step, the [OWLTools](https://github.com/owlcollab/owltools) library is used to merge the directory of cleaned ontology files into a single ontology file. Then, the following cleaning steps are performed: (1) Identifier Errors; (2) Duplicate Classes; (3) Duplicate Class Concepts; and (4) Punning Errors.  \n",
    "\n",
    "**Inputs:** A directory of ontology files (`.owl`)\n",
    "\n",
    "**Outputs:** `PheKnowLator_MergedOntologies.owl`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Merge Clean Ontology Data')\n",
    "ont_data.ont_file_location = ont_data.merged_ontology_filename\n",
    "\n",
    "# reorder list of ontology files to prepare for merging\n",
    "onts = [ont_data.temp_dir + '/' + x for x in list(ont_data.ontology_info.keys())\n",
    "        if x != ont_data.merged_ontology_filename]\n",
    "\n",
    "# merge ontologies\n",
    "merges_ontologies(onts, ont_data.temp_dir + '/', ont_data.ont_file_location, ont_data.owltools_location)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load merged ontologies into RDF Lib Graph object\n",
    "print('Loading Merged Ontology Data')\n",
    "ont_data.ont_graph = Graph().parse(ont_data.temp_dir + '/' + ont_data.ont_file_location)\n",
    "\n",
    "# add merged ontology to dict\n",
    "ont_data.ontology_info[ont_data.ont_file_location] = {}\n",
    "\n",
    "# get stats on merged ontologies\n",
    "print(derives_graph_statistics(ont_data.ont_graph))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean Merged Ontologies\n",
    "🤔 *IMPORTANT*🤔  Please note there are a few decisions that can made be made at this point that you may want to consider. For our monthly `PheKnowLator` builds, we prefer to use Entrez gene identifiers. If you have run the [`Data_Preparation.ipynb`](https://github.com/callahantiff/PheKnowLator/blob/master/notebooks/Data_Preparation.ipynb) Jupyter Notebook without makeing updates, you would have also committed yourself to using this type of gene identifier. If you have not done with this and do not want to use Entrez gene, but rather prefer to use what the ontologies provide, please comment out `ont_data.normalizes_existing_classes()` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get starting statistics\n",
    "ont_data.updates_ontology_reporter()\n",
    "\n",
    "# clean merged ontologies\n",
    "ont_data.fixes_identifier_errors()\n",
    "ont_data.normalizes_duplicate_classes()\n",
    "ont_data.normalizes_existing_classes()\n",
    "ont_data.fixes_punning_errors()\n",
    "\n",
    "# get finishing statistics\n",
    "print(derives_graph_statistics(ont_data.ont_graph))\n",
    "ont_data.updates_ontology_reporter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output and Save Results\n",
    "The cleaned merged ontology file is saved to the `resources/knowledge_graphs` directory where it can be detected by the `PheKnowLator` algorithm during the build process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Save and Format Merged Ontology Data')\n",
    "ont_data.ont_graph.serialize(destination=knowledge_graphs_location + '/' + ont_data.ont_file_location, format='xml')\n",
    "ontology_file_formatter(knowledge_graphs_location, '/' + ont_data.ont_file_location, ont_data.owltools_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save Ontology Cleaning Results  \n",
    "To view the results of the ontology cleaning process print the `ont_data.ontology_info` dictionary. This dictionary is keyed by ontology filename and contains a separate dictionary for each ontology with descriptions of the results for each error check that is performed at the individual- and merged-ontology level. The results are also saved to `resources/ontologies/ontology_cleaning_report.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save output locally\n",
    "ont_order = sorted([x for x in ont_data.ontology_info.keys() if not x.startswith('Phe')]) + [ont_data.ont_file_location]\n",
    "with open(ont_data.temp_dir + '/ontology_cleaning_report.txt', 'w') as o:\n",
    "    o.write('=' * 50 + '\\n{}'.format('ONTOLOGY CLEANING REPORT'))\n",
    "    o.write('\\n{}\\n'.format(str(datetime.datetime.utcnow().strftime('%a %b %d %X UTC %Y'))) + '=' * 50 + '\\n\\n')\n",
    "    for key in ont_order:\n",
    "        o.write('\\n\\n\\nONTOLOGY: {}\\n'.format(key)); o.write('*' * (len(key) + 10) + '\\n\\n')\n",
    "        x = ont_data.ontology_info[key]\n",
    "        if 'Original GCS URL' in x.keys(): o.write('\\t- Original GCS URL: {}\\n'.format(x['Original GCS URL']))\n",
    "        if 'Processed GCS URL' in x: o.write('\\t- Processed GCS URL: {}\\n'.format(x['Processed GCS URL']))\n",
    "        o.write('\\t- Statistics Before Cleaning:\\n\\t\\t- {}\\n'.format(x['Starting Statistics']))\n",
    "        o.write('\\t- Statistics After Cleaning:\\n\\t\\t- {}\\n'.format(x['Final Statistics']))\n",
    "        if 'ValueErrors' in x.keys():\n",
    "            if isinstance( x['ValueErrors'], str): o.write('\\t- Value Errors (n=1):\\n\\t\\t- {}\\n'.format(x['ValueErrors']))\n",
    "            else:\n",
    "                for i in x['ValueErrors']: o.write('\\t\\t- {}\\n'.format(str(i)))\n",
    "        else: o.write('\\t- Value Errors: 0\\n')     \n",
    "        if x['IdentifierErrors'] != 'None':\n",
    "            o.write('\\t- Identifier Errors (n={}):\\n'.format(len(x['IdentifierErrors'].split(', '))))\n",
    "            for i in x['IdentifierErrors'].split(', '): o.write('\\t\\t- {}\\n'.format(str(i)))\n",
    "        else: o.write('\\t- Identifier Errors: 0\\n')\n",
    "        if 'PheKnowLator_MergedOntologies' not in key:\n",
    "            if x['Deprecated'] != 'None':\n",
    "                o.write('\\t- Deprecated Classes (n={}):\\n'.format(len(x['Deprecated'])))\n",
    "                for i in x['Deprecated']: o.write('\\t\\t- {}\\n'.format(str(i)))\n",
    "            else: o.write('\\t- Deprecated Classes: 0\\n') \n",
    "            if x['Obsolete'] != 'None':\n",
    "                o.write('\\t- Obsolete Classes (n={}):\\n'.format(len(x['Obsolete'])))\n",
    "                for i in x['Obsolete']: o.write('\\t\\t- {}\\n'.format(str(i)))\n",
    "            else: o.write('\\t- Obsolete Classes: 0\\n')\n",
    "        o.write('\\t- Punning Errors:\\n')\n",
    "        if x['PunningErrors - Classes'] != 'None':\n",
    "            o.write('\\t\\t- Classes (n={}):\\n'.format(len(x['PunningErrors - Classes'].split(', '))))\n",
    "            for i in x['PunningErrors - Classes'].split(', '): o.write('\\t\\t\\t- {}\\n'.format(i))\n",
    "        else: o.write('\\t\\t- Classes: 0\\n')\n",
    "        if x['PunningErrors - ObjectProperty'] != 'None':\n",
    "            o.write('\\t\\t- Object Properties (n={}):\\n'.format(len(x['PunningErrors - ObjectProperty'].split(', '))))\n",
    "            for i in x['PunningErrors - ObjectProperty'].split(', '): o.write('\\t\\t\\t- {}\\n'.format(i))\n",
    "        else: o.write('\\t\\t- Object Properties: 0\\n')\n",
    "        if 'Normalized - Duplicates' in x.keys():\n",
    "            o.write('\\t- Normalization:\\n')\n",
    "            if x['Normalized - Duplicates'] != 'None':\n",
    "                o.write('\\t\\t- Existing Entity Normalization (n={}):\\n'.format(len(x['Normalized - Duplicates'].split(', '))))\n",
    "                for i in x['Normalized - Duplicates'].split(', '): o.write('\\t\\t\\t- {}\\n'.format(i))\n",
    "            else: o.write('\\t\\t- Entity Normalization: 0\\n')\n",
    "            if x['Normalized - Gene IDs'] != 'None': o.write('\\t\\t- Normalized HGNC IDs: {}\\n'.format(x['Normalized - Gene IDs']))\n",
    "            if x['Normalized - NonOnt'] != 'None': o.write('\\t\\t- Other Classes that May Need Normalization: {}\\n'.format(x['Normalized - NonOnt']))\n",
    "            if x['Normalized - Dep'] != 'None':\n",
    "                o.write('\\t\\t- Deprecated Ontology HGNC Identifiers Needing Alignment (n={}):\\n'.format(len(x['Normalized - Dep'])))\n",
    "                for i in x['Normalized - Dep']: o.write('\\t\\t- {}\\n'.format(i))\n",
    "            else: o.write('\\t\\t- Deprecated Ontology HGNC Identifiers Needing Alignment: 0\\n')\n",
    "o.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean-Up Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove temp file in resources/ontologies\n",
    "os.remove(write_location + '/' + ont_data.ont_file_location)\n",
    "os.remove(write_location + '/Merged_gene_rna_protein_identifiers.pkl')\n",
    "\n",
    "# # remove logs directory\n",
    "# logs = glob.glob('..builds/logs/*.log')\n",
    "# shutil.rmtree('/'.join(logs[0].split('/')[:-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<br>\n",
    "\n",
    "***\n",
    "***\n",
    "\n",
    "```\n",
    "@misc{callahan_tj_2019_3401437,\n",
    "  author       = {Callahan, TJ},\n",
    "  title        = {PheKnowLator},\n",
    "  month        = mar,\n",
    "  year         = 2019,\n",
    "  doi          = {10.5281/zenodo.3401437},\n",
    "  url          = {https://doi.org/10.5281/zenodo.3401437}\n",
    "}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}