chore: update generate fabric doc (#2214)

* update generate fabric doc * update doc pipeline * update doc pipeline * update docgen * update doc pipeline * update doc pipeline * style --------- Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>
microsoft · May 21, 2024 · a8820fe · a8820fe
1 parent 5c37342
commit a8820fe
Show file tree

Hide file tree

Showing 9 changed files with 459 additions and 210 deletions.
diff --git a/docs/Explore Algorithms/AI Services/Multivariate Anomaly Detection.ipynb b/docs/Explore Algorithms/AI Services/Multivariate Anomaly Detection.ipynb
@@ -13,7 +13,12 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "alert",
+     "important"
+    ]
+   },
    "source": [
     "## Important\n",
     "Starting on the 20th of September, 2023 you won’t be able to create new Anomaly Detector resources. The Anomaly Detector service is being retired on the 1st of October, 2026."

diff --git a/docs/Explore Algorithms/AI Services/Overview.ipynb b/docs/Explore Algorithms/AI Services/Overview.ipynb
@@ -4,15 +4,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Azure AI Services"
+    "# Azure AI services"
    ]
   },
   {
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {
     "tags": [
-     "hide-synapse-internal"
+     "hide-synapse-internal",
+     "hide-azure"
     ]
    },
    "source": [
@@ -22,11 +23,57 @@
   {
    "cell_type": "markdown",
    "metadata": {},
+   "source": [
+    "Azure AI services help developers and organizations rapidly create intelligent, cutting-edge, market-ready, and responsible applications with out-of-the-box and pre-built and customizable APIs and models.\n",
+    "\n",
+    "SynapseML allows you to build powerful and highly scalable predictive and analytical models from various Spark data sources. Synapse Spark provide built-in SynapseML libraries including synapse.ml.services."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": [
+     "important",
+     "alert"
+    ]
+   },
    "source": [
     "## Important\n",
     "Starting on the 20th of September, 2023 you won’t be able to create new Anomaly Detector resources. The Anomaly Detector service is being retired on the 1st of October, 2026."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": [
+     "hide-synapse-internal",
+     "hide-azure"
+    ]
+   },
+   "source": [
+    "## Prerequisites on Azure Databricks\n",
+    "\n",
+    "1. Follow the steps in [Getting started](https://docs.microsoft.com/azure/services-services/big-data/getting-started) to set up your Azure Databricks and Azure AI services environment. This tutorial shows you how to install SynapseML and how to create your Spark cluster in Databricks.\n",
+    "1. After you create a new notebook in Azure Databricks, copy the **Shared code** below and paste into a new cell in your notebook.\n",
+    "1. Choose a service sample, below, and copy paste it into a second new cell in your notebook.\n",
+    "1. Replace any of the service subscription key placeholders with your own key.\n",
+    "1. Choose the run button (triangle icon) in the upper right corner of the cell, then select **Run Cell**.\n",
+    "1. View results in a table below the cell."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": [
+     "hide-synapse-internal"
+    ]
+   },
+   "source": [
+    "## Prerequisites on Azure Synapse Analytics\n",
+    "\n",
+    "The tutorial, [Pre-requisites for using Azure AI services in Azure Synapse](https://learn.microsoft.com/azure/synapse-analytics/machine-learning/tutorial-configure-cognitive-services-synapse), walks you through a couple steps you need to perform before using Azure AI services in Synapse Analytics.\n"
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",
@@ -54,7 +101,7 @@
     "- Group: divides a group of faces into disjoint groups based on similarity ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/face/GroupFaces.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.face.html#module-synapse.ml.services.face.GroupFaces))\n",
     "\n",
     "### Speech\n",
-    "[**Speech Services**](https://azure.microsoft.com/services/cognitive-services/speech-services/)\n",
+    "[**Speech Services**](https://azure.microsoft.com/products/ai-services/ai-speech)\n",
     "- Speech-to-text: transcribes audio streams ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/speech/SpeechToText.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.speech.html#module-synapse.ml.services.speech.SpeechToText))\n",
     "- Conversation Transcription: transcribes audio streams into live transcripts with identified speakers. ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/speech/ConversationTranscription.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.speech.html#module-synapse.ml.services.speech.ConversationTranscription))\n",
     "- Text to Speech: Converts text to realistic audio ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/speech/TextToSpeech.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.speech.html#module-synapse.ml.services.speech.TextToSpeech))\n",
@@ -70,7 +117,7 @@
     "\n",
     "\n",
     "### Translation\n",
-    "[**Translator**](https://azure.microsoft.com/services/cognitive-services/translator/)\n",
+    "[**Translator**](https://azure.microsoft.com/products/ai-services/translator)\n",
     "- Translate: Translates text. ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/translate/Translate.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.translate.html#module-synapse.ml.services.translate.Translate))\n",
     "- Transliterate: Converts text in one language from one script to another script. ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/translate/Transliterate.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.translate.html#module-synapse.ml.services.translate.Transliterate))\n",
     "- Detect: Identifies the language of a piece of text. ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/translate/Detect.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.translate.html#module-synapse.ml.services.translate.Detect))\n",
@@ -91,32 +138,13 @@
     "- List Custom Models: Get information about all custom models. ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/form/ListCustomModels.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.form.html#module-synapse.ml.services.form.ListCustomModels))\n",
     "\n",
     "### Decision\n",
-    "[**Anomaly Detector**](https://azure.microsoft.com/services/cognitive-services/anomaly-detector/)\n",
+    "[**Anomaly Detector**](https://azure.microsoft.com/products/ai-services/ai-anomaly-detector)\n",
     "- Anomaly status of latest point: generates a model using preceding points and determines whether the latest point is anomalous ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/anomaly/DetectLastAnomaly.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.anomaly.html#module-synapse.ml.services.anomaly.DetectLastAnomaly))\n",
     "- Find anomalies: generates a model using an entire series and finds anomalies in the series ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/anomaly/DetectAnomalies.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.anomaly.html#module-synapse.ml.services.anomaly.DetectAnomalies))\n",
     "\n",
     "### Search\n",
-    "- [Bing Image search](https://azure.microsoft.com/services/services-services/bing-image-search-api/) ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/bing/BingImageSearch.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.bing.html#module-synapse.ml.services.bing.BingImageSearch))\n",
-    "- [Azure Cognitive search](https://docs.microsoft.com/azure/search/search-what-is-azure-search) ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchWriter$.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.search.html#module-synapse.ml.services.search.AzureSearchWriter))"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {
-    "tags": [
-     "hide-synapse-internal"
-    ]
-   },
-   "source": [
-    "## Prerequisites\n",
-    "\n",
-    "1. Follow the steps in [Getting started](https://docs.microsoft.com/azure/services-services/big-data/getting-started) to set up your Azure Databricks and Azure AI services environment. This tutorial shows you how to install SynapseML and how to create your Spark cluster in Databricks.\n",
-    "1. After you create a new notebook in Azure Databricks, copy the **Shared code** below and paste into a new cell in your notebook.\n",
-    "1. Choose a service sample, below, and copy paste it into a second new cell in your notebook.\n",
-    "1. Replace any of the service subscription key placeholders with your own key.\n",
-    "1. Choose the run button (triangle icon) in the upper right corner of the cell, then select **Run Cell**.\n",
-    "1. View results in a table below the cell."
+    "- [**Bing Image search**](https://azure.microsoft.com/services/services-services/bing-image-search-api/) ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/bing/BingImageSearch.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.bing.html#module-synapse.ml.services.bing.BingImageSearch))\n",
+    "- [**Azure Cognitive search**](https://docs.microsoft.com/azure/search/search-what-is-azure-search) ([Scala](https://mmlspark.blob.core.windows.net/docs/1.0.4/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchWriter$.html), [Python](https://mmlspark.blob.core.windows.net/docs/1.0.4/pyspark/synapse.ml.services.search.html#module-synapse.ml.services.search.AzureSearchWriter))"
    ]
   },
   {
@@ -662,7 +690,7 @@
     ]
    },
    "source": [
-    "## Azure Cognitive search sample\n",
+    "## Azure AI search sample\n",
     "\n",
     "In this example, we show how you can enrich data using Cognitive Skills and write to an Azure Search Index using SynapseML."
    ]

diff --git a/docs/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.ipynb b/docs/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.ipynb
@@ -12,16 +12,20 @@
     }
    },
    "source": [
-    "# Recipe: Multivariate Anomaly Detection with Isolation Forest\n",
-    "This recipe shows how you can use SynapseML on Apache Spark for multivariate anomaly detection. Multivariate anomaly detection allows for the detection of anomalies among many variables or time series, taking into account all the inter-correlations and dependencies between the different variables. In this scenario, we use SynapseML to train an Isolation Forest model for multivariate anomaly detection, and we then use to the trained model to infer multivariate anomalies within a dataset containing synthetic measurements from three IoT sensors.\n",
+    "# Multivariate Anomaly Detection with Isolation Forest\n",
+    "This article shows how you can use SynapseML on Apache Spark for multivariate anomaly detection. Multivariate anomaly detection allows for the detection of anomalies among many variables or time series, taking into account all the inter-correlations and dependencies between the different variables. In this scenario, we use SynapseML to train an Isolation Forest model for multivariate anomaly detection, and we then use to the trained model to infer multivariate anomalies within a dataset containing synthetic measurements from three IoT sensors.\n",
     "\n",
     "To learn more about the Isolation Forest model please refer to the original paper by [Liu _et al._](https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf?q=isolation-forest)."
    ]
   },
   {
    "attachments": {},
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-synapse-internal"
+    ]
+   },
    "source": [
     "## Prerequisites\n",
     " - If running on Synapse, you'll need to [create an AML workspace and set up linked Service](../../Use%20with%20MLFlow/Overview.md) and add the following installation cell.\n",

diff --git a/docs/Explore Algorithms/Causal Inference/Quickstart - Measure Causal Effects.ipynb b/docs/Explore Algorithms/Causal Inference/Quickstart - Measure Causal Effects.ipynb
@@ -10,7 +10,7 @@
     }
    },
    "source": [
-    "# Startup Investment Attribution - Understand Outreach Effort's Effect\""
+    "# Startup Investment Attribution - Understand Outreach Effort's Effect"
    ]
   },
   {
@@ -120,7 +120,7 @@
     }
    },
    "source": [
-    "# Get Causal Effects with SynapseML DoubleMLEstimator"
+    "## Get Causal Effects with SynapseML DoubleMLEstimator"
    ]
   },
   {

diff --git a/docs/Explore Algorithms/Classification/Quickstart - SparkML vs SynapseML.ipynb b/docs/Explore Algorithms/Classification/Quickstart - SparkML vs SynapseML.ipynb
@@ -5,7 +5,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Classification - before and after SynapseML"
+    "# Classification - SparkML vs SynapseML"
    ]
   },
   {
@@ -157,12 +157,12 @@
     "## Classify using pyspark\n",
     "\n",
     "To choose the best LogisticRegression classifier using the `pyspark`\n",
-    "library, you need to *explicitly* perform the following steps:\n",
+    "library, we need to *explicitly* perform the following steps:\n",
     "\n",
     "1. Process the features:\n",
-    "   * Tokenize the text column\n",
-    "   * Hash the tokenized column into a vector using hashing\n",
-    "   * Merge the numeric features with the vector\n",
+    "   - Tokenize the text column\n",
+    "   - Hash the tokenized column into a vector using hashing\n",
+    "   - Merge the numeric features with the vector\n",
     "2. Process the label column: cast it into the proper type.\n",
     "3. Train multiple LogisticRegression algorithms on the `train` dataset\n",
     "   with different hyperparameters\n",

diff --git a/docs/Explore Algorithms/OpenAI/OpenAI.ipynb b/docs/Explore Algorithms/OpenAI/OpenAI.ipynb
@@ -322,7 +322,18 @@
    "source": [
     "### Generating Text Embeddings\n",
     "\n",
-    "In addition to completing text, we can also embed text for use in downstream algorithms or vector retrieval architectures. Creating embeddings allows you to search and retrieve documents from large collections and can be used when prompt engineering isn't sufficient for the task. For more information on using `OpenAIEmbedding`, see our [embedding guide](./Quickstart%20-%20OpenAI%20Embedding)."
+    "In addition to completing text, we can also embed text for use in downstream algorithms or vector retrieval architectures. Creating embeddings allows you to search and retrieve documents from large collections and can be used when prompt engineering isn't sufficient for the task."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": [
+     "hide-synapse-internal"
+    ]
+   },
+   "source": [
+    "For more information on using `OpenAIEmbedding` see our [embedding guide](./Quickstart%20-%20OpenAI%20Embedding)."
    ]
   },
   {