qcarchive_workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d60cc0ff",
   "metadata": {},
   "source": [
    "\n",
    "<center>\n",
    "<img src=\"./images/QCArchiveLogo_rectangle.png\">\n",
    "</center>\n",
    "\n",
    "# QCArchive Workshop\n",
    "\n",
    "This notebook shows you how to get started with QCArchive - a MolSSI software product that helps you run quantum chemistry calculations and archive the results. \n",
    "In most cases, QCArchive would be set up on a server for your reserach group or team, and access to the calculations and the results would be limited to whoever was granted access to your server. \n",
    "In this tutorial, we will be using the MolSSI QCArchive demonstration server, so that you can get a feel for what QCArchive can do.  \n",
    "\n",
    "QCArchive actually has multiple parts, including QCPortal, which is how you interact with the server. \n",
    "We will be using QCPortal and some of its features in this tutorial.\n",
    "\n",
    "## Installing QCPortal\n",
    "You have to intall QCPortal on your computer to use it to interact with the QCArchive Server. \n",
    "We maintain a QCPortal environment that you can install using `conda` that contains all the required components for QCArchive.\n",
    "To use this, you need to have `anaconda-client` installed in your base conda environment.  \n",
    "\n",
    "First install `anaconda-client` in your base environment, using this command. \n",
    "If it is already installed, it will just tell you it is already installed.\n",
    "\n",
    "```\n",
    "conda install anaconda-client -n base\n",
    "```\n",
    "\n",
    "Then create the qcarchive_demo enviorment and install all required software with this command\n",
    "```\n",
    "conda env create qcarchive/qcarchive_demo\n",
    "```\n",
    "\n",
    "Now activate your conda enviroment, and then restart this notebook.\n",
    "\n",
    "```\n",
    "conda activate qcarchive_demo\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17936106",
   "metadata": {},
   "outputs": [],
   "source": [
    "import qcportal as ptl\n",
    "from qcportal.molecules import Molecule\n",
    "\n",
    "import nglview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed540f79",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Create a client object and connect to the demo server\n",
    "\n",
    "The `PortalClient` is how you interact with the server, including querying records and submitting computations.\n",
    "\n",
    "The demo server allows for unauthenticated guest access, so no username/password is necessary for read access.\n",
    "However, you will need to log in to submit or modify computations. \n",
    "If you are taking a QCArchive workshop, you will be provided a username and password. \n",
    "If not, please contact the MolSSI (qcarchive@molssi.org) to get a demo username and password if you would like to try out the software. \n",
    "The demo server is intended for testing and demonstration use only, not any significant calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f8e54fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the way you would connect without authentication if you were only going to read information.\n",
    "#client = ptl.PortalClient(\"https://qcademo.molssi.org\")\n",
    "\n",
    "# This is how you long in with a username and password.\n",
    "client = ptl.PortalClient(\"https://qcademo.molssi.org\", username=\"qcademo\", password=\"molssi_qcademo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94cbe701",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Single-point energy calculations for one molecule\n",
    "\n",
    "### Specifying the molecule\n",
    "\n",
    "We are going to begin by creating one molecule and doing a singlepoint energy calculation.  \n",
    "\n",
    "The part of QCArchive that creates and manages molecules is called QCElemental.  There are several ways you can create a molecule.  If the molecule is simple enough, you can create it by directly specifiying the geometry with xyz coordinates.  This is hard unless it is a very simple molecule.\n",
    "\n",
    "This is a hydrogen molecule with a bond distance of 1.0 bohr."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2a92faa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Most direct way to make a molecule\n",
    "hydrogen = Molecule(symbols=['h', 'h'], geometry=[[0.0, 0.0, 0.0, 0.0, 0.0, 1.0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c0d190-1126-419c-b947-74d548ec6329",
   "metadata": {},
   "source": [
    "Another way is to specify an xyz file as a string and the generate the molecule from that string.  \n",
    "\n",
    "The triple quotes notation in python lets you specify multi-line strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f16987a4-8838-4d47-9adb-eea4a32a5ae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Another way to specify molecule is to create a string of an xyx file\n",
    "mol_xyz = \"\"\"\n",
    "  3\n",
    "\n",
    "  H   0.000000 1.000000 0.000000\n",
    "  H   1.000000 0.000000 0.000000\n",
    "  O   0.000000 0.000000 0.000000\n",
    "\"\"\"\n",
    "water = Molecule.from_data(mol_xyz)\n",
    "print(water)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67908200-39c0-4e5b-a53f-6e51a5838c1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another way is to have read in an xyz file that you already have.  \n",
    "water2 = Molecule.from_file('water.xyz')\n",
    "print(water2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2116c67a",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Choosing job options and submitting the computation\n",
    "\n",
    "To create our computations, there are different methods to add different types of computations. \n",
    "For example, for an energy caluation, the method is `add_singlepoints`. \n",
    "The methods for adding computations (like `add_singlepoints`) take a list of molecules and the details of how to run the computation. \n",
    "In this example, we only have one molecule for now.\n",
    "\n",
    "When we execute a method command, this sends it to the server, where it is queued for computation. \n",
    "This will eventually be picked up by a worker if there is a worker that can handle it (proper programs installed, etc).\n",
    "\n",
    "Here are the options you will specify for your computation:\n",
    "* **program** = use this program to run the computation\n",
    "* **driver** = the main type of computation to run (energy, gradient, properties, etc)\n",
    "* **method**/**basis** = the computational model to use\n",
    "\n",
    "All `add_` functions return two objects. \n",
    "The first is metadata about the addition, and second is a list of record ids - we will only have one because we only submitted one molecule.\n",
    "We can use these IDs later to retrieve the record, which will let us look at our results.\n",
    "\n",
    "QCArchive supports a lot of different QM calculations programs. \n",
    "The input to QCPortal is exacxtly the same no matter what QM program you are using. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79f0f6a4-b80a-4f62-9ca0-f4090d572985",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "qmprogram = 'psi4'\n",
    "calc_type = 'energy'\n",
    "calc_method = 'b3lyp'\n",
    "basis_set = 'def2-tzvp'\n",
    "\n",
    "\n",
    "# Note the first input to this function is the molecules that you want to calculate.  \n",
    "# Note that this must be a list, so we have\n",
    "# to use the square brackets, even if there is only one thing in the list.\n",
    "molecule_list = [water2]\n",
    "meta, record_ids = client.add_singlepoints(molecule_list, \n",
    "                                           program=qmprogram, \n",
    "                                           driver=calc_type, \n",
    "                                           method=calc_method, \n",
    "                                           basis=basis_set)\n",
    "\n",
    "#Print the list of record IDs that we will use to access the data later.\n",
    "# This will be a list of the same length as your molecule list\n",
    "print(record_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13d13e48-ed24-4d40-b42b-921f3e7ceca4",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Looking at the results\n",
    "Now we need to look at the results from our calculation.  To do this, we need to create a `record` of our calcluation results so that then we can pull out different information.  To create a record use `client.get_records` and input the record_id.  Remember that record_id is a list, so you have to use the list syntax.\n",
    "\n",
    "### Information from the calculation record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07baa1e4-a605-4c79-b2e8-9b059a0e391c",
   "metadata": {},
   "outputs": [],
   "source": [
    "water_record = client.get_records(record_ids[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "065bd931-2e50-4b23-8e96-6f6740e8c12b",
   "metadata": {},
   "source": [
    "This doesn't print anything, but now we have a `record` object that we can use to access different information about our calculation.  We can access a variety of information and the types of information we can access from the record depends on what kind of calculation we did. We will demonstate several kinds of information you can access from `record` in the new few cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75fdfb84-9f00-451e-86cd-c1281172c2be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can print the status of the calculation to see if it is finished running.\n",
    "print(water_record.status)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "325fc013-e195-459d-ba11-c7ca3247cd65",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Show the entire output file.  This is probably not what we are interested in, but you can do it.\n",
    "# print(water_record.stdout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b9217f7-20ef-4043-b589-b4c5e7071da8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can print the information about your molecule\n",
    "print(water_record.molecule)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c0b9acd-fb02-481c-85c6-17f7229e40e9",
   "metadata": {},
   "source": [
    "You can use this molecule information to visualize in nglview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a508ea9-8915-4623-a04c-0f4ccdb448ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(water2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1fa5936-0400-4a71-ab13-e0b66cf92e13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can print the details of your calculation.\n",
    "print(water_record.specification)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ceabd9e-2d9a-457e-a53f-26db894f1f52",
   "metadata": {},
   "source": [
    "Another thing you can access from the `record` is the provenance of your calculation.  This contains even more information than the specification, including the exact version of the software used to run the calculation and the walltime.  The syntax is slightly more complicated because you have to accesse the last element of the compute history and then the provenance from that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "500617d7-f46a-4133-a7c5-ef011229c663",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the information available in the provenance\n",
    "print(water_record.compute_history[-1].provenance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "328fac0b-af77-4926-af41-901edd9dd979",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access a particular peice of information from the provenance\n",
    "time_s = water_record.compute_history[-1].provenance.wall_time\n",
    "print(time_s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fb3f27d-30d7-474c-b24d-f7b13d366e0b",
   "metadata": {},
   "source": [
    "### Information from the calculation: molecular properties\n",
    "\n",
    "The things we have looked at from `record` so far are mostly details about the calculation itself and not really the results of the calculation.  Most of the information about the results (i.e. the properties of your molecule that you calculated) is stored in a python dictionary that you can access from `record`.  Remember that a python dictionary is a data structure where you look up information by a key (a word) instead of an index (a number).  First we extract the dictionary from the records and give it a name.  Then we can see all the keys, which are all the molecular properties we could access.  Then we can print or do any other operation with any of those properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8573dee5-4270-4b87-8b44-e948a56dd114",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extracting the dictionary from the record and giving it a name\n",
    "water_properties = water_record.properties\n",
    "# Looking at all the keys for the dictionary to see what information is stored\n",
    "print(water_properties.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d33a36a9-0ca3-4799-8dbf-88321b7b9a93",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Pick a property you want to look at and print it\n",
    "energy = water_properties['scf_total_energy']\n",
    "print(energy)\n",
    "# You could do this in one line if you wanted to like this\n",
    "print(water_properties['scf_total_energy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dac34b91-5a4a-41a9-be99-87847b1d659e",
   "metadata": {},
   "source": [
    "### Multiple calculations for one molecule\n",
    "We can iterate over multiple calculation types for one molecule (or many molecules, \n",
    "but we will get to that). Let's add some additional computational methods for our water calculation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "549bc2fd-2cdc-4cdd-8093-297c8212fe80",
   "metadata": {},
   "outputs": [],
   "source": [
    "qmprogram = 'psi4'\n",
    "calc_type = 'energy'\n",
    "#Change calc method to a list\n",
    "calc_method = ['b3lyp', 'm06-2x', 'pbe0']\n",
    "basis_set = 'def2-tzvp'\n",
    "# Don't forget list notation!\n",
    "molecule_list = [water2]\n",
    "\n",
    "#Set up empty lists to hold the meta data and record_ids\n",
    "record_ids = []\n",
    "\n",
    "for functional in calc_method:\n",
    "    meta, record_id = client.add_singlepoints(molecule_list, \n",
    "                                           program=qmprogram, \n",
    "                                           driver=calc_type, \n",
    "                                           method=functional, \n",
    "                                           basis=basis_set)\n",
    "    print(meta, record_id)\n",
    "    record_ids.extend(record_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3855f066-c58d-4b74-bb24-d907c314a9cf",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "1. Find the record IDs for your three calculations.\n",
    "2. Check to see if they are finished.\n",
    "3. Print the energy for each method in a neatly formatted print statement.\n",
    "\n",
    "### Extension Exercise\n",
    "Do a basis set convergence study. \n",
    "Increase the basis set size for one molecule. \n",
    "Look at the energies to see if the energy has converged with respect to basis set. \n",
    "Then look at the walltime to see how much more expensive the bigger calculations were.\n",
    "If you are not familiar with basis sets, you might try `cc-pvdz`, `cc-pvtz`, and `cc-pvqz` (ordered from smallest to largest basis set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79d0b06f-5c87-488a-9435-a2c0cdda02e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here\n",
    "\n",
    "\n",
    "\n",
    "    \n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80c52572-ceb2-480d-b2fa-18badee56779",
   "metadata": {},
   "source": [
    "## More complex calculations with QCSpecification\n",
    "\n",
    "If you are doing any calculation more complicated than a single-point calculation, then we need to use a QCSpecification. \n",
    "The QCSpecification is the details of the QM calculation that you want to run. \n",
    "The overall QCArchive ecosystem is structured in a hierarchical way, where a single point calculation is the simplest type. \n",
    "You can think of a geometry optimization as numerous single-point calculations, so the specification is the details of one of those single-point calculations.\n",
    "(Later, when we want to create a dataset by running the same kind of calculation for many molecules, we will also need to use a QCSpecification.) \n",
    "In this example, we will do a geometry optimization calculation, which will first require setting a specification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a954ebc-7a08-4f8b-9d10-52d456b0a7e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the QCSpecification module\n",
    "from qcportal.singlepoint import QCSpecification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca574ba2-5b73-4208-a73c-c6a9bea0b804",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up a water calculation with a stretched bond\n",
    "water_xyz = \"\"\"\n",
    "  3\n",
    "\n",
    "  H   0.000000 1.500000 0.000000\n",
    "  H   1.000000 0.000000 0.000000\n",
    "  O   0.000000 0.000000 0.000000\n",
    "\"\"\"\n",
    "\n",
    "water_stretched = Molecule.from_data(water_xyz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4339b8b0-8801-439e-b0f6-5e671339ce4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Visualize the molecule\n",
    "display(water_stretched)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa9e1add-235f-4379-83ca-ce7b42903ab4",
   "metadata": {},
   "source": [
    "Now we will set up our calculation details using QCSpecification.  Then when we use `add_optimization` to set up our calculation, we will give our specification instead of having to enter in all the computation details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "622a288e-34dc-405b-a1fe-b3d031b85e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the specification\n",
    "spec = QCSpecification(program=\"psi4\", \n",
    "                       driver=\"deferred\", \n",
    "                       method='hf', \n",
    "                       basis='sto-3g', \n",
    "                       keywords={'maxiter': 100})\n",
    "\n",
    "# Submit the geometry optimization\n",
    "meta, id = client.add_optimizations(initial_molecules=[water_stretched], \n",
    "                                    program=\"geometric\",\n",
    "                                    qc_specification=spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63247d3a-79aa-4fa4-afc2-35f9416b8ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the record for this calculation\n",
    "opt_record = client.get_records(id[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c26fe3-9ce8-45cf-bb3b-46baf6caf23f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# View the optimized geometry\n",
    "optimized_geometry = opt_record.final_molecule\n",
    "display(optimized_geometry)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f364c6e8-baec-4479-a2b6-4cacbbf765cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the optimized geometry to a file\n",
    "optimized_geometry.to_file('water_opt.xyz')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51757334-178a-40cf-93a4-8129991b9ffa",
   "metadata": {},
   "source": [
    "## Data Sets\n",
    "A data set is a set of one kind of calculation (single points, optimizations, etc.) for many different molecules (called entries in a QCArchive dataset).  Think of it like a table where the rows are the molecules and the columns are the different QM calculation methods (the specifications) you want to run. A cell within the table (intersection between a molecule and a specificiation) is a record that has the output for the molecule for that particular specification. \n",
    "\n",
    "|       | HF/sto-3g | B3LYP/def2-tzvp | MP2/cc-pvdz |\n",
    "| ---   | ---       | ---             | ---         |\n",
    "| water | 18263     | 18277           | 18295       |\n",
    "|methane| 19722     | 19642           | 19867       |\n",
    "|ethanol| 20212     | 20931           | 23210       |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73936ce7-6e20-42ee-8489-d5a57a1a62c5",
   "metadata": {},
   "source": [
    "### Creating a dataset\n",
    "In this example, we will create a singepoint energy data set for several elements.  Remember, a dataset can only contain one type of calculation.\n",
    "\n",
    "Note: Within your archive, you can not have more than one dataset with the same name.  Give your dataset in the example below a unique name.  Note that once you have run the cell, your dataset will exist, so if you try to run this cell again, it will tell you the dataset already exists.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51af59c6-c6b7-45a2-8217-58f2a972d583",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = client.add_dataset(\"singlepoint\",\n",
    "                        name=\"  \",\n",
    "                        description=\"Variety of calculations on single atoms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af58717-8d00-423b-a356-55aad18a687b",
   "metadata": {},
   "source": [
    "Now we will make a list of our favorite elements, and create a molecule object for each one.  We will specify that the coordinates for each atom is at the origin.  For each molecule, we will add it as an entry in our dataset using `ds.add_entry`.  We must give each entry a name, which is a string.  We will give each entry the name `X_atom` where X is the element's symbol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f00be9d-a3e3-4603-8cbb-27b3f6be9e2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a few of your favorite elements\n",
    "favorite_elements = [\"H\", \"C\", \"Si\", \"Ag\"]\n",
    "for element in favorite_elements:\n",
    "    mol = Molecule(symbols=[element], geometry=[0.0, 0.0, 0.0])\n",
    "    \n",
    "    # Creates an entry from the molecule. The entry contains the molecule and a name,\n",
    "    # but there are additional fields you can have as well\n",
    "    entry_name = element + \"_atom\"\n",
    "    ds.add_entry(name=entry_name, molecule=mol)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "282b5b36-9899-4d29-abf9-b52b7b591c91",
   "metadata": {},
   "source": [
    "We will now create two different specifications, and add them to the dataset. The first will be hf/sto-3g, and the second will be b3lyp/aug-cc-pvtz.\n",
    "\n",
    "In this example, we will use the same QM program, but you could use different specifications to compare programs, different functionals, basis sets, etc.\n",
    "\n",
    "For both calculations, we will increase the maximum number of SCF iterations to 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83f86204-2964-4bfa-bb1b-87848a221eec",
   "metadata": {},
   "outputs": [],
   "source": [
    "spec_1 = QCSpecification(\n",
    "            program=\"psi4\",\n",
    "            driver=\"energy\",\n",
    "            method=\"hf\",\n",
    "            basis=\"sto-3g\",\n",
    "            keywords={\"maxiter\": 100}\n",
    ")\n",
    "\n",
    "spec_2 = QCSpecification(\n",
    "            program=\"psi4\",\n",
    "            driver=\"properties\",\n",
    "            method=\"b3lyp\",\n",
    "            basis=\"aug-cc-pvtz\",\n",
    "            keywords={\"maxiter\": 100}\n",
    ")\n",
    "\n",
    "ds.add_specification(name=\"hf/sto-3g\", specification=spec_1)\n",
    "ds.add_specification(name=\"b3lyp/aug-cc-pvtz\", specification=spec_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "258b01b4-afe0-4a32-818c-f520bbd08106",
   "metadata": {},
   "source": [
    "### Submitting the computations and checking the status\n",
    "At this point, we have added specifications and entries,\n",
    "but have not submitted any calculations yet. We do that with\n",
    "the `submit()` function\n",
    "\n",
    "By default, this submits all calculations, but you could choose to only submit certain entries (particular molecules) or certain specifications (certain calculation types)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e290a42-ab68-4727-b0fa-040f29d75acb",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78dccab4-4fed-420c-969b-bb1e4a17d18b",
   "metadata": {},
   "source": [
    "We can check the status of our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe46db4e-1bb6-456e-b59f-402b67f3838f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The basic way to check the status is\n",
    "ds.status()\n",
    "# A better formatted print is\n",
    "ds.print_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41d8ef88-f7bd-4583-a633-56ab54e920c1",
   "metadata": {},
   "source": [
    "### Accessing results for calculations in a dataset\n",
    "If you want to access the results of one of these calculations, you need to create a record for the one you want to view.  Previously, we did this with the record id number, but now I don't know that record id for a particular calculation.  Instead, in a dataset, you can access the record by giving the entry name and the specification name, still using the `get_record` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb8cf532-e53a-44fc-a2ef-022b2b68e604",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a record for a particular entry/specification pair\n",
    "rec = ds.get_record('Ag_atom', 'hf/sto-3g')\n",
    "# I can print the record id for this entry in the data set\n",
    "print(rec.id)\n",
    "# I can print \n",
    "print(rec.properties['return_energy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa20f77b-ba15-4693-9aa7-73cabe5641b2",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "Create a new specification and add it to your data set.  You can also add new entries (additional atoms) if you wish.  Rerun the dataset and check the status.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a1fd787-f86d-41d7-b10b-54a614336f92",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0376c63-0532-454d-92f1-6c393890b6bb",
   "metadata": {},
   "source": [
    "### Extension Exercise\n",
    "For one of your entries (one of the atoms), compare the results of your new specification to the results for the b3lyp/aug-cc-pvtz calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60177ffd-be62-4451-bcb5-eb456ac93907",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940e0ccf-8a27-4035-b52d-fecaf50f39b3",
   "metadata": {},
   "source": [
    "### Accessing results for all the calculations in a dataset\n",
    "\n",
    "To get results for all the calclulations in a dataset, you can use the function `iterate_records` within a `for` loop.  This function iterates over three things: the entry, the specification, and the record.  You can give optional inputs to the `interate_records` function to only iterate over the calculations that are complete or only look at a particular specification, or other options.  \n",
    "\n",
    "In this example, we will print the entry name, the specification, and the final energy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03e2b3c2-ef4f-45ac-ad2b-b3233389be0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#e - entry \n",
    "#s - specification \n",
    "#r - record\n",
    "# We are only going to print the results for calculations that are complete.\n",
    "for e, s, r in ds.iterate_records(status='complete'):\n",
    "    # Get the energy from the record\n",
    "    # Remember it is in the properties dictionary that is accessed from the record\n",
    "    energy = r.properties['return_energy']\n",
    "    # Print this for everything in the dataset\n",
    "    print(e, s, energy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56c6f93a-a1a9-4ac5-b235-7181b3e1f92a",
   "metadata": {},
   "source": [
    "You could use formatted print statements to make this look better, or you could store these values in lists or arrays or any other type of Python data structure.\n",
    "\n",
    "If you look carefully, you notice that there are no results printed for the entry `Ag_atom` for the specification `b3lyp/aug-cc-pvtz`.  This is probably the calculation that had an error, therefore its status is not complete.  We will come back to this later."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a1a55d8-cd07-47f1-98b4-750e586a05c5",
   "metadata": {},
   "source": [
    "### Compiling results into a dataframe\n",
    "\n",
    "A more advanced way to analyze your results is to compile them into a dataframe, such that you could now use any of the querying and sorting capabilities of pandas.  To do this, we will use a function called `compile_values`.  The inputs of `compile_values` are a python callable (generally a function) and a string that gives a name for the data.  The output is a pandas dataframe. \n",
    "\n",
    "In general, you write a function that takes a record as an input and returns whatever data you want that you can access from the record.  Then you input that function into `compile_values` as the callable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "637857eb-81ef-43fb-b855-521c94062f36",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write a function to extract the data you want from the record\n",
    "def get_energy(record):\n",
    "    # Get the energy from the properties dictionary\n",
    "    energy = record.properties['return_energy']\n",
    "    return energy\n",
    "\n",
    "# Syntax: dataset_name.compile_values(callable, name_string)\n",
    "# The output is a dataframe\n",
    "energies_df = ds.compile_values(get_energy, 'total_energy')\n",
    "    \n",
    "# If you know what a lambda is you can do it this way    \n",
    "#df = ds.compile_values(lambda r: r.properties[\"return_energy\"], 'total_energy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9c37dc6-e7b0-4085-8381-f3b4562ca8bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the dataframe\n",
    "energies_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa634eee-9a22-478f-9352-8743868ca248",
   "metadata": {},
   "source": [
    "We again notice that the results for Ag_atom for b3lyp/aug-cc-pvtz is NaN.  This must be the calculation that had an error when we checked the status above.  A calculation that ends in an error still has a record.  Let's investigate the record for the failed calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f256847e-d820-4e98-a12f-fd2ac44c7850",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see why this is NaN\n",
    "# Get the record for the calculation that gave NaN results\n",
    "Ag_b3lyp_record = ds.get_record(\"Ag_atom\", \"b3lyp/aug-cc-pvtz\")\n",
    "# Print the status\n",
    "print(Ag_b3lyp_record.status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2cf276d-01b4-48be-a2af-96f59f401edd",
   "metadata": {},
   "source": [
    "Every record has a dictionary called `error`.  We can use the key `error_message` to access the error message from the dictionary.  It prints a lot of information, but generally, you can find the problem near the end of the message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5655c40-27ef-4c08-b37f-3ab61e884f75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# From the error, access the error dictionary and print the error message\n",
    "print(Ag_b3lyp_record.error['error_message'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c8da17e-ca3d-46bb-9f15-56bbba32a609",
   "metadata": {},
   "source": [
    "Perhaps unsurprinstly, there is no basis function for silver in the aug-cc-pvtz basis set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c66ab49a-cbaa-4c23-95d8-6a582babc142",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}