models/__pipeline__.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "deluxe-force",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"10%\" alt=\"Naas\" src=\"https://landen.imgix.net/jtci2pxwjczr/assets/5ice39g4.png?w=160\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unlimited-internship",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# Naas - Create Pipeline\n",
    "<a href=\"https://app.naas.ai/user-redirect/naas/downloader?url=https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/Naas/Naas_Create_Pipeline.ipynb\" target=\"_parent\"><img src=\"https://naasai-public.s3.eu-west-3.amazonaws.com/Open_in_Naas_Lab.svg\"/></a><br><br><a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=&template=template-request.md&title=Tool+-+Action+of+the+notebook+\">Template request</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=Naas+-+Create+Pipeline:+Error+short+description\">Bug report</a> | <a href=\"https://app.naas.ai/user-redirect/naas/downloader?url=https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/Naas/Naas_Start_data_product.ipynb\" target=\"_parent\">Generate Data Product</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3097faf3-b2b9-41fd-8589-f7718b5f919a",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #naas #pipeline #jupyter #notebook #dataanalysis #workflow #streamline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opposite-guatemala",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Maxime Jublou](https://www.linkedin.com/in/maximejublou)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf2e0f26-7fdf-4351-9209-8bb54c5ef7e9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Description:** This notebook is a guide that teaches you how to create a notebook pipeline using naas."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "input_cell",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "import_cell",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "funny-neighbor",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "from naas.pipeline.pipeline import (\n",
    "    Pipeline,\n",
    "    DummyStep,\n",
    "    DummyErrorStep,\n",
    "    NotebookStep,\n",
    "    End,\n",
    "    ParallelStep,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b579d95-8da2-46aa-b8dd-843ae0d46964",
   "metadata": {},
   "source": [
    "### Setup variables\n",
    "- `pipeline_outputs_path`: Path of your pipeline executions. When the pipeline is run, a \"pipeline_executions\" folder will be created in your file system. Inside this folder, you will be able to access each pipeline executions. If you use NotebookStep, executed notebooks will be stored in this folder. This allows you to easily review and analyze the results of the pipeline, and to troubleshoot any issues that may have occurred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f67191c1-aaa8-4ad6-8793-eb563f72822c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_outputs_path = \"../outputs/pipeline_executions\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "model_cell",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e15d093c-0189-42c3-959f-64eed5530fd4",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup NotebookStep\n",
    "For demonstration purposes, we used the `DummyStep` to illustrate its functionality.\n",
    "\n",
    "In order to create a pipeline, you should used the `NotebookStep()` which has three parameters:\n",
    "- the name (string) of the step\n",
    "- the notebook path (string) for execution\n",
    "- the parameters (dictionary) that are injected through papermill in the first cell or after the cell labeled \"parameters.\"\n",
    "\n",
    "`NotebookStep(\"My Notebook\", \"my_notebook.ipynb\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa346868-d375-4547-b11b-79311e3f1fc3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "collection = DummyStep(\n",
    "    \"Collection\"\n",
    ")  # In this step, data can be collected from various sources such as databases, APIs, or file systems.\n",
    "cleaning = DummyStep(\n",
    "    \"Cleaning\"\n",
    ")  # Once the data is collected, it is often necessary to clean and preprocess it to remove any irrelevant or duplicate information. This step may involve tasks such as removing null values, correcting data formats, and standardizing column names.\n",
    "transformation1 = DummyStep(\n",
    "    \"Transformation 1\"\n",
    ")  # In this step, the data is transformed into the desired format, such as a flat file or a specific data model. This may involve tasks such as aggregating data, joining multiple tables, or calculating new fields.\n",
    "transformation2 = DummyStep(\n",
    "    \"Transformation 2\"\n",
    ")  # In this step, the data is transformed into the desired format, such as a flat file or a specific data model. This may involve tasks such as aggregating data, joining multiple tables, or calculating new fields.\n",
    "distribution = DummyStep(\n",
    "    \"Distribution\"\n",
    ")  # In this step, the data is loaded into its final destination, such as a data warehouse, a data lake, or a specific application."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "output_cell",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "display_cell",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Create Basic Pipeline\n",
    "- Link your notebook using this syntax: `>>`\n",
    "- Create ParallelStep using this syntax: `[step1, step2]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c14e4a38-75b1-475b-bc28-45db7effe45f",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline()\n",
    "\n",
    "(\n",
    "    pipeline\n",
    "    >> collection\n",
    "    >> cleaning\n",
    "    >> [transformation1, transformation2]\n",
    "    >> distribution\n",
    "    >> End()\n",
    ")\n",
    "\n",
    "pipeline.run(outputs_path=pipeline_outputs_path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "92ddbcf7c74813cc4c906ca6b7d04cc2590230b5fb16082b396de5b9872be0cf",
   "notebook_path": "Naas/Naas_Create_Pipeline.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}