Merge pull request #54 from tee-lab/notebooks-update

Minor corrections and improvements in notebooks

notebooks/1_getting_started.ipynb

@@ -54,7 +54,10 @@
    "id": "e168e420-3b4e-46c3-a503-da93b380d51e",
    "metadata": {},
    "source": [
-    "This example uses a sample dataset, loaded using a helper function. For details about data formats and loading/saving data, see [Exporting Data](./5_exporting_data.ipynb)."
+    "This example uses a sample dataset, loaded using a helper function. The dataset contains net polarisation from a 1-dimensional agent-based simulation. For this model, the drift and diffusion functions are known -- the drift function is linear while the diffusion is quadratic. In this notebook, we will examine if PyDaddy can estimate the drift and diffusion correctly from this dataset.\n",
+    "\n",
+    "For more information about the model and the dataset, see [Jhawar and Guttal, 2020](https://doi.org/10.1098/rstb.2019.0381).\n",
+    "For details about data formats and loading/saving data, see [Exporting Data](./5_exporting_data.ipynb)."
    ]
   },
   {
@@ -122,7 +125,12 @@
    "id": "9022f125-37f4-4365-8f74-ae057887ffff",
    "metadata": {},
    "source": [
-    "In the above example, automatic model selection (`tune=True`) sucessfully found the correct threshold. If the data is too noisy, or if `order` is too high, automatic model selection can give poor results. In such cases, good results can be obtained with some manual intervention: see [Advanced Function Fitting](./3_advanced_function_fitting.ipynb) for more details."
+    "In the above example, automatic model selection (`tune=True`) found a reasonable linear function for drift.\n",
+    "For diffusion, it looks like there are insignificant $x$ and $x^3$ terms, which can be eliminated by a correct choice of threshold.\n",
+    "\n",
+    "If the data is too noisy, or if `order` is too high, automatic model selection can give poor results. In such cases, good results can be obtained with some manual intervention: see [Advanced Function Fitting](./3_advanced_function_fitting.ipynb) for more details.\n",
+    "\n",
+    "In cases like this, we can manually set the appropriate threshold using the `threshold` parameter."
    ]
   },
   {
@@ -145,6 +153,14 @@
     "ddsde.fit('G', order=2, threshold=0.01)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "89a113d0-465a-478c-b860-0762f9176df7",
+   "metadata": {},
+   "source": [
+    "The discovered drift and diffusion functions are consistent with the theoretical expectations for the model (see [Jhawar and Guttal, 2020](https://doi.org/10.1098/rstb.2019.0381))."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "f4895a95-de51-4c6a-9d6e-420cffba6249",
@@ -269,10 +285,22 @@
     "ddsde.model_diagnostics(oversample=10)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "e48c69ee-c511-4e40-90cd-5b525515ae8f",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "- To see PyDaddy in action on a 2-dimensional vector dataset, see [Getting Started with Vector Data](./2_getting_started_vector.ipynb).\n",
+    "- To learn more about how to manually fit the drift and diffusion functions, see [Advanced Function Fitting](./3_advanced_function_fitting.ipynb).\n",
+    "- To see PyDaddy in action on real world datasets, see [Example analysis: mesoscopic SDEs for fish schools](./7_example_fish_school.ipynb) and [Example analysis: SDEs for cancer cell migration](./8_example_cell_migration.ipynb)."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "890f678d-b7ca-4493-9f22-893dd646572f",
+   "id": "c37a2937-1ccb-49f1-9215-f9620392f3b2",
    "metadata": {},
    "outputs": [],
    "source": []

notebooks/2_getting_started_vector.ipynb

@@ -51,7 +51,11 @@
    "source": [
     "## Initializing the `pydaddy` object\n",
     "\n",
-    "Similar to the scalar analysis, we need to initialize a `pydaddy` object. In this case, `data` will be a two element list."
+    "Similar to the scalar analysis, we need to initialize a `pydaddy` object. In this case, `data` will be a two element list, with elements corresponding to coordinates.\n",
+    "\n",
+    "The example dataset used here the polarisation time series from a 2-dimensional agent-based simulation. For this model, the drift and diffusion functions are known -- the drift function is cubic while the diffusion is quadratic. In this notebook, we will examine if PyDaddy can estimate the drift and diffusion correctly from this dataset.\n",
+    "\n",
+    "For more information about the model and the dataset, see [Jhawar et. al., 2020](https://doi.org/10.1038/s41567-020-0787-y)."
    ]
   },
   {
@@ -177,6 +181,14 @@
     "**Note:** Since $G_{21}$ and $G_{12}$ are identical, fitting one will automatically assign the value for the other."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "abf26577-3a90-4312-9ade-44697cdb4b6d",
+   "metadata": {},
+   "source": [
+    "The discovered drift and diffusion functions are consistent with the theoretically expected SDE for the model. For more details, see [Jhawar et. al., 2020](https://doi.org/10.1038/s41567-020-0787-y)."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "87691411-2beb-4d6e-8d17-7b47319d2aaf",
@@ -261,12 +273,15 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "31b57925-92f7-441a-b195-aa688878613b",
+   "cell_type": "markdown",
+   "id": "f1cebe39-c8e7-4a3a-b398-37e20026f9cc",
    "metadata": {},
-   "outputs": [],
-   "source": []
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "- To learn more about how to manually fit the drift and diffusion functions, see [Advanced Function Fitting](./3_advanced_function_fitting.ipynb).\n",
+    "- To see PyDaddy in action on real world datasets, see [Example analysis: mesoscopic SDEs for fish schools](./7_example_fish_school.ipynb) and [Example analysis: SDEs for cancer cell migration](./8_example_cell_migration.ipynb)."
+   ]
   }
  ],
  "metadata": {

notebooks/4_sdes_from_simulated_timeseries.ipynb

@@ -39,29 +39,64 @@
    "id": "513e5005",
    "metadata": {},
    "source": [
-    "We will use the `sdeint` package to simulate a time series with specified drift and diffusion functions. The following cell contains some sample drift and diffusion functions. Uncomment a function of your choice, or define your own functions `f()` and `g()`."
+    "We will use the `sdeint` package to simulate a time series with specified drift and diffusion functions. The following cell contains some sample drift and diffusion functions. Uncomment a function of your choice, or define your own functions `f()` and `g()`.\n",
+    "\n",
+    "The example functions given below correspond to the functions in Figure 1 in the manuscript."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "588e71f1",
+   "id": "19e31598-8f7c-4c6f-a4f2-9c75a81f4448",
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Linear drift and constant diffusion (OU process)\n",
-    "# def f(x):\n",
-    "#     return -x\n",
+    "## ------ Model A ------\n",
     "\n",
-    "# def g(x):\n",
-    "#     return np.ones_like(x) * np.sqrt(2)\n",
+    "dt = 0.01\n",
+    "timepoints = 100000\n",
     "\n",
-    "# Cubic drift and quadratic diffusion\n",
     "def f(x):\n",
-    "    return x - x ** 3\n",
+    "    return -x\n",
     "\n",
     "def g(x):\n",
-    "    return 2 - x ** 2"
+    "    return np.ones_like(x) * np.sqrt(2)\n",
+    "\n",
+    "\n",
+    "## ------ Model B ------\n",
+    "\n",
+    "# dt = 0.001\n",
+    "# timepoints = 1000000\n",
+    "\n",
+    "# def f(x):\n",
+    "#     return x - x ** 3\n",
+    "\n",
+    "# def g(x):\n",
+    "#     return np.sqrt(2 * (1 + x ** 2))\n",
+    "\n",
+    "\n",
+    "## ------ Model C ------\n",
+    "\n",
+    "# dt = 0.01\n",
+    "# timepoints = 1000000\n",
+    "\n",
+    "# def f(x):\n",
+    "#     return 2 * x - 3 * x ** 3\n",
+    "    \n",
+    "# def g(x):\n",
+    "#     return np.ones_like(x) * 0.5\n",
+    "\n",
+    "\n",
+    "## ------ Model D ------\n",
+    "\n",
+    "# dt = 0.01\n",
+    "# timepoints = 1000000\n",
+    "\n",
+    "# def f(x):\n",
+    "#     return - x\n",
+    "\n",
+    "# def g(x):\n",
+    "#     return np.sqrt(2) * (1 - x ** 2)"
    ]
   },
   {
@@ -97,8 +132,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dt = 0.01            # Integration time-step\n",
-    "timepoints = 100000  # Number of time-points to simulate\n",
     "t = np.arange(0, dt * timepoints, step=dt)\n",
     "x0 = 0.0             # Initial condition\n",
     "\n",
@@ -186,7 +219,7 @@
    "id": "1a0b6a05",
    "metadata": {},
    "source": [
-    "**Note:** Automatic model selection may not always work, in which case some fine-tuning may be required. See the [Advanced Function Fitting](./3_advanced_function_fitting.ipynb) notebook for more details."
+    "**Note:** Automatic model selection may not always work, in which case some fine-tuning may be required. See the [Advanced Function Fitting](./3_advanced_function_fitting.ipynb) notebook for more details. If the above cell did not recover the correct function, adjust the threshold and order so that the correct function is discovered without overfitting."
    ]
   },
   {

notebooks/5_exporting_data.ipynb

@@ -35,17 +35,6 @@
     "`pydaddy` allows you to export data as a Pandas DataFrame, or save data into a CSV file."
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "6a2db6b2-aa3c-4a61-9951-78a9a349411d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import sys\n",
-    "sys.path.append('/Users/nabeel/Documents/Research/Code/pyddsde')"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,

notebooks/6_non_poly_function_fitting.ipynb

@@ -150,7 +150,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.6"
+   "version": "3.11.0"
   }
  },
  "nbformat": 4,

notebooks/7_example_fish_school.ipynb

@@ -15,7 +15,11 @@
    "id": "5abc6293-795b-437c-90c0-1f0f76b4b82f",
    "metadata": {},
    "source": [
-    "This notebook illustrates the use of PyDaddy to discover mesoscale SDEs for schooling fish. The notebook uses a dataset by [Jitesh et. al.](https://doi.org/10.1038/s41567-020-0787-y), which is also provided with PyDaddy as an example dataset."
+    "(This notebook assumes that you have gone through the [Getting Started](./1_getting_started.ipynb) and [Getting Started with Vector Data](./1_getting_started_vector.ipynb) notebooks.)\n",
+    "\n",
+    "This notebook illustrates the use of PyDaddy to discover mesoscale SDEs for schooling fish. The notebook uses a dataset by [Jitesh et. al.](https://doi.org/10.1038/s41567-020-0787-y), which is also provided with PyDaddy as an example dataset.\n",
+    "\n",
+    "The dataset contains a 2-dimensional time series fo the polarisation vector for a group of schooling fish."
    ]
   },
   {