Merge pull request #76 from CBroz1/main

Fixes for codebook deployment

Author: kabilar

CHANGELOG.md

@@ -3,6 +3,11 @@
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
 [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
+## [0.2.1] - 2022-07-22
++ Add - Mention CodeBook data directory in notebooks
++ Update - Remove directory assertion in notebooks for CodeBook deployment
++ Update - Move all export functions from `pipeline` to `export` script
+
 ## [0.2.0] - 2022-07-08
 
 + Add - Adopt black formatting into code base

notebooks/00-data-download-optional.ipynb

@@ -6,7 +6,9 @@
    "source": [
     "# Download example data\n",
     "\n",
-    "This workflow will need Ephys data collected from either SpikeGLX or OpenEphys and the output from kilosort2. We provided an example dataset to be downloaded to run through the pipeline. This notebook walks you through the process to download the dataset."
+    "This workflow will need Ephys data collected from either SpikeGLX or OpenEphys and the output from kilosort2. We provided an example dataset to be downloaded to run through the pipeline. This notebook walks you through the process to download the dataset.\n",
+    "\n",
+    "[CodeBook](codebook.datajoint.io) users can skip this step."
    ]
   },
   {
@@ -173,7 +175,7 @@
    "formats": "ipynb,py"
   },
   "kernelspec": {
-   "display_name": "Python 3.10.4 64-bit ('python3p10')",
+   "display_name": "Python 3.9.12 ('ele')",
    "language": "python",
    "name": "python3"
   },
@@ -187,11 +189,11 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.4"
+   "version": "3.9.12"
   },
   "vscode": {
    "interpreter": {
-    "hash": "ff52d424e56dd643d8b2ec122f40a2e279e94970100b4e6430cb9025a65ba4cf"
+    "hash": "61456c693db5d9aa6731701ec9a9b08ab88a172bee0780139a3679beb166da16"
    }
   }
  },

notebooks/01-configure.ipynb

@@ -35,11 +35,7 @@
    "outputs": [],
    "source": [
     "# change to the upper level folder\n",
-    "if os.path.basename(os.getcwd()) == \"notebooks\":\n",
-    "    os.chdir(\"..\")\n",
-    "assert os.path.basename(os.getcwd()) == \"workflow-array-ephys\", (\n",
-    "    \"Please move to the \" + \"workflow directory\"\n",
-    ")\n",
+    "if os.path.basename(os.getcwd()) == \"notebooks\": os.chdir(\"..\")\n",
     "import datajoint as dj"
    ]
   },
@@ -55,20 +51,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6820ef4c",
    "metadata": {},
    "outputs": [],
    "source": [
-    "import getpass"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dj.config[\"database.host\"] = \"{YOUR_HOST}\"\n",
+    "import getpass\n",
+    "dj.config[\"database.host\"] = \"{YOUR_HOST}\" # CodeBook users should omit this\n",
     "dj.config[\"database.user\"] = \"{YOUR_USERNAME}\"\n",
     "dj.config[\"database.password\"] = getpass.getpass()  # enter the password securily"
    ]
@@ -106,7 +93,7 @@
     "\n",
     "Giving a prefix to schema could help on the configuration of privilege settings. For example, if we set prefix `neuro_`, every schema created with the current workflow will start with `neuro_`, e.g. `neuro_lab`, `neuro_subject`, `neuro_ephys` etc.\n",
     "\n",
-    "The prefix could be configurated as follows in `dj.config`:"
+    "The prefix could be configurated in `dj.config` as follows. CodeBook users should keep their username as the prefix for schema for declaration permissions."
    ]
   },
   {
@@ -115,7 +102,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.config[\"custom\"] = {\"database.prefix\": \"neuro_\"}"
+    "username_as_prefix = dj.config[\"database.user\"] + \"_\"\n",
+    "dj.config[\"custom\"] = {\"database.prefix\": username_as_prefix}"
    ]
   },
   {
@@ -130,10 +118,10 @@
     "\n",
     "The root path typically **do not** contain information of subjects or sessions, all data from subjects/sessions should be subdirectories in the root path.\n",
     "\n",
-    "In the example dataset downloaded with [these instructions](00-data-download-optional.ipynb), `/tmp/test_data` will be the root\n",
+    "- In the example dataset downloaded with [these instructions](00-data-download-optional.ipynb), `/tmp/test_data` will be the root. \n",
+    "- For CodeBook users, the root is `/home/inbox/`\n",
     "\n",
     "```\n",
-    "/tmp/test_data/\n",
     "- subject6\n",
     "    - session1\n",
     "        - towersTask_g0_imec0\n",
@@ -151,7 +139,7 @@
     "# If there is only one root path.\n",
     "dj.config[\"custom\"][\"ephys_root_data_dir\"] = \"/tmp/test_data\"\n",
     "# If there are multiple possible root paths:\n",
-    "dj.config[\"custom\"][\"ephys_root_data_dir\"] = [\"/tmp/test_data1\", \"/tmp/test_data2\"]"
+    "dj.config[\"custom\"][\"ephys_root_data_dir\"] = [\"/tmp/test_data\", \"/home/inbox/\"]"
    ]
   },
   {
@@ -164,20 +152,13 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "id": "f7733450",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "[markdown]\n",
-    "# + In the database, every path for the ephys raw data is **relative to root path(s)**. The benefit is that the absolute path could be configured for each machine, and when data transfer happens, we just need to change the root directory in the config file.\n",
-    "#\n",
-    "# + The workflow supports **multiple root directories**. If there are multiple possible root directories, specify the `ephys_root_data_dir` as a list.\n",
-    "#\n",
-    "# + The root path(s) are **specific to each machine**, as the name of mounted drive could be different for different operating systems or machines.\n",
-    "#\n",
-    "# + In the context of the workflow, all the paths saved into the database or saved in the config file need to be in the **POSIX standards** (Unix/Linux), with `/`. The path conversion for machines of any operating system is taken care of inside the elements."
+    "+ In the database, every path for the ephys raw data is **relative to root path(s)** to allow for the absolute path to be configured for **each machine**. When transferring data, we just need to change the root directory in the config file.\n",
+    "\n",
+    "+ DataJoint Elements use `pathlib.Path()` to maintain path information in **POSIX standards** (Unix/Linux), with `/`. The path conversion for machines of any operating system is taken care of inside the elements."
    ]
   },
   {
@@ -269,7 +250,7 @@
    "formats": "ipynb,py_scripts//py"
   },
   "kernelspec": {
-   "display_name": "Python 3.10.4 64-bit ('python3p10')",
+   "display_name": "Python 3.9.12 ('ele')",
    "language": "python",
    "name": "python3"
   },
@@ -283,11 +264,11 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.4"
+   "version": "3.9.12"
   },
   "vscode": {
    "interpreter": {
-    "hash": "ff52d424e56dd643d8b2ec122f40a2e279e94970100b4e6430cb9025a65ba4cf"
+    "hash": "61456c693db5d9aa6731701ec9a9b08ab88a172bee0780139a3679beb166da16"
    }
   }
  },