bdc-06-spark-etl.ipynb

{
    "metadata": {
        "kernelspec": {
            "name": "SQL",
            "display_name": "SQL",
            "language": "sql"
        },
        "language_info": {
            "name": "sql",
            "version": ""
        }
    },
    "nbformat_minor": 2,
    "nbformat": 4,
    "cells": [
        {
            "cell_type": "markdown",
            "source": [
                "<img src=\"https://github.com/Microsoft/sqlworkshops/blob/master/graphics/solutions-microsoft-logo-small.png?raw=true\" alt=\"Microsoft\">\r\n",
                "<br>\r\n",
                "\r\n",
                "# **SQL Server 2019 big data cluster Tutorial**\r\n",
                "## **06 - Using Spark for ETL**\r\n",
                "\r\n",
                "In this tutorial you will learn how to work with Spark Jobs in a SQL Server big data cluster. \r\n",
                "\r\n",
                "Many times Spark is used to do transformations on data at large scale. In this Jupyter Notebook, you'll read a large text file into a Spark DataFrame, and then save out the top 10 examples as a table using SparkSQL.\r\n",
                "\r\n",
                "> Switch your kernel to PySpark and run print(\"hello\") or whatever you like to activate Spark context.  \r\n",
                ">\r\n",
                "> If it output error like this\r\n",
                "> ```\r\n",
                "> The code failed because of a fatal error:\r\n",
                ">   Error sending http request and maximum retry encountered..\r\n",
                "> ```\r\n",
                "> Please switch to another kernel and switch back, and run again."
            ],
            "metadata": {
                "azdata_cell_guid": "af570b3d-c696-4d29-9290-c5b5d042660f"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "# Test if SparkSession is available as 'spark'\r\n",
                "print('hello')"
            ],
            "metadata": {
                "azdata_cell_guid": "a5ac36e5-7869-4799-9e32-ff79478df18c",
                "tags": []
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "code",
            "source": [
                "# Read the product reviews CSV files into a spark data frame, print schema & top rows\r\n",
                "results = spark.read.option(\"inferSchema\", \"true\").csv('/product_review_data').toDF(\"Item_ID\", \"Review\")\r\n",
                "results.show()"
            ],
            "metadata": {
                "azdata_cell_guid": "7fe4f671-195f-4ce2-9d08-45c9ff175cb0",
                "tags": []
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "code",
            "source": [
                "# Save results as parquet file and create hive table\r\n",
                "results.write.format(\"parquet\").mode(\"overwrite\").saveAsTable(\"Top_Product_Reviews\")"
            ],
            "metadata": {
                "azdata_cell_guid": "59bdf34a-a397-4ed0-80ad-523768141073",
                "tags": []
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "code",
            "source": [
                "# Execute Spark SQL commands\r\n",
                "sqlDF = spark.sql(\"SELECT * FROM Top_Product_Reviews LIMIT 10\")\r\n",
                "sqlDF.show()"
            ],
            "metadata": {
                "azdata_cell_guid": "77ec4ac7-4a92-4bad-b170-a3e2568ae6f3",
                "tags": []
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Next Steps: Continue on to Working with Spark and Machine Learning**\r\n",
                "\r\n",
                "Now you're ready to open the Jupyter Notebook in this tutorial series - [bdc-07-spark-ml.ipynb](bdc-07-spark-ml.ipynb) - to learn how to create and work with Spark and Machine Learning."
            ],
            "metadata": {
                "azdata_cell_guid": "483977c9-a81c-4ec9-9290-40e4ba586e9e"
            }
        }
    ]
}