02 - Vertex AI AutoML/02c - Vertex AI > Pipelines - AutoML with clients (code) In automated pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3cebabe4",
   "metadata": {},
   "source": [
    "![ga4](https://www.google-analytics.com/collect?v=2&tid=G-6VDTYWLKX6&cid=1&en=page_view&sid=1&dl=statmike%2Fvertex-ai-mlops%2F02+-+Vertex+AI+AutoML&dt=02c+-+Vertex+AI+%3E+Pipelines+-+AutoML+with+clients+%28code%29+In+automated+pipeline.ipynb)\n",
    "\n",
    "# 02c - Vertex AI > Pipelines - AutoML with clients (code) In automated pipeline\n",
    "\n",
    "Use[ Kubeflow](https://www.kubeflow.org/) Pipelines running on [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) to orchestrate the process of training a custom model with AutoML Tabular and deploy it to a Vertex AI Endpoint for serving (online and batch) predictions and explanations.  This demonstrates how to automate the processes of (02a) or (02b) with pipeline orchestration.\n",
    "\n",
    "### Video Walkthrough of this notebook:\n",
    "Includes conversational walkthrough and more explanatory information than the notebook:\n",
    "<p align=\"center\" width=\"100%\"><center><a href=\"https://youtu.be/1gHJgY7AXAs\" target=\"_blank\" rel=\"noopener noreferrer\"><img src=\"../architectures/thumbnails/playbutton/02c.png\" width=\"40%\"></a></center></p>\n",
    "\n",
    "Notes Since Video:\n",
    "- update notebook on 8/31/2022 to:\n",
    "    - make the pipeline run with input parameters rather than using local variable in the pipeline definition\n",
    "    - update the model evaluation section to show update Python Client methods for metric retrieval\n",
    "\n",
    "### Prerequisites:\n",
    "-  01 -  BigQuery - Table Data Source\n",
    "\n",
    "### Resources:\n",
    "-  [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/build-pipeline#google-cloud-components) see aiplatform.PipelineJob\n",
    "-  [Python Client for Vertex AI](https://googleapis.dev/python/aiplatform/latest/aiplatform.html)\n",
    "-  [Kubeflow Pipelines Components for Google Cloud](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud)\n",
    "\n",
    "### Related Training:\n",
    "-  Codelab: [Vertex AI Pipelines Introduction](https://codelabs.developers.google.com/vertex-mlmd-pipelines#0)\n",
    "\n",
    "### Conceptual Flow & Workflow\n",
    "<p align=\"center\">\n",
    "  <img alt=\"Conceptual Flow\" src=\"../architectures/slides/02c_arch.png\" width=\"45%\">\n",
    "&nbsp; &nbsp; &nbsp; &nbsp;\n",
    "  <img alt=\"Workflow\" src=\"../architectures/slides/02c_console.png\" width=\"45%\">\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "229cef71",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b6b82e4",
   "metadata": {},
   "source": [
    "inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "63e4ffbd-4d77-4074-b84a-a9a10fe0b606",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b255bf32",
   "metadata": {},
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "DATANAME = 'fraud'\n",
    "NOTEBOOK = '02c'\n",
    "\n",
    "# Resources\n",
    "DEPLOY_COMPUTE = 'n1-standard-2'\n",
    "\n",
    "# Model Training\n",
    "VAR_TARGET = 'Class'\n",
    "VAR_OMIT = 'transaction_id' # add more variables to the string with space delimiters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00693f36",
   "metadata": {},
   "source": [
    "packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d1e6b18c-6deb-41dc-a632-8b305705eeec",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install -U google-cloud-pipeline-components -U -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f3bf8620",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:4: DeprecationWarning: The module `kfp.v2` is deprecated and will be removed in a futureversion. Please import directly from the `kfp` namespace, instead of `kfp.v2`.\n",
      "  after removing the cwd from sys.path.\n"
     ]
    }
   ],
   "source": [
    "from google.cloud import aiplatform\n",
    "from datetime import datetime\n",
    "import kfp\n",
    "from kfp.v2 import compiler\n",
    "#import kfp.v2.dsl as dsl\n",
    "#import google_cloud_pipeline_components as gcc_aip\n",
    "from google_cloud_pipeline_components.v1.dataset import TabularDatasetCreateOp\n",
    "from google_cloud_pipeline_components.v1.automl.training_job import AutoMLTabularTrainingJobRunOp\n",
    "from google_cloud_pipeline_components.v1.endpoint import EndpointCreateOp, ModelDeployOp\n",
    "\n",
    "from google.cloud import bigquery\n",
    "from google.protobuf import json_format\n",
    "from google.protobuf.struct_pb2 import Value\n",
    "import json\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b01a4860",
   "metadata": {},
   "source": [
    "clients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d0f88c50",
   "metadata": {},
   "outputs": [],
   "source": [
    "aiplatform.init(project=PROJECT_ID, location=REGION)\n",
    "bq = bigquery.Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d14fbbc",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f65b2223",
   "metadata": {},
   "outputs": [],
   "source": [
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "BUCKET = PROJECT_ID\n",
    "URI = f\"gs://{BUCKET}/{DATANAME}/models/{NOTEBOOK}\"\n",
    "DIR = f\"temp/{NOTEBOOK}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2c68630e-4fe3-473d-abb3-16d39fb782e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1026793852137-compute@developer.gserviceaccount.com'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SERVICE_ACCOUNT = !gcloud config list --format='value(core.account)' \n",
    "SERVICE_ACCOUNT = SERVICE_ACCOUNT[0]\n",
    "SERVICE_ACCOUNT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d35908-9b69-40be-97dc-fb4fb0279709",
   "metadata": {},
   "source": [
    "List the service accounts current roles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a9f46306-bd55-425a-951a-fd2753807b91",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROLE\n",
      "roles/bigquery.admin\n",
      "roles/owner\n",
      "roles/run.admin\n",
      "roles/storage.objectAdmin\n"
     ]
    }
   ],
   "source": [
    "!gcloud projects get-iam-policy $PROJECT_ID --filter=\"bindings.members:$SERVICE_ACCOUNT\" --format='table(bindings.role)' --flatten=\"bindings[].members\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8343a71-3484-4352-9753-ec045cb6dfc5",
   "metadata": {},
   "source": [
    ">Note: If the resulting list is missing [roles/storage.objectAdmin](https://cloud.google.com/storage/docs/access-control/iam-roles) then [revisit the setup notebook](../00%20-%20Setup/00%20-%20Environment%20Setup.ipynb#permissions) and add this permission to the service account with the provided instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f15187",
   "metadata": {},
   "source": [
    "environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f7383876",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {DIR}\n",
    "!mkdir -p {DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c08c3499",
   "metadata": {},
   "source": [
    "---\n",
    "## Pipeline (KFP) Definition\n",
    "- Flow\n",
    "    - Create Vertex AI Dataset from link to BigQuery table\n",
    "    - Create Vertex AI AutoML Tabular Training Job\n",
    "    - Create Endpoint and Depoy trained model\n",
    "    \n",
    "Use [AI Platform Pipeline Components](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-0.2.0/)\n",
    "- Specifically, [AutoMLTabularTrainingJobRunOp](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-0.2.0/google_cloud_pipeline_components.aiplatform.html#google_cloud_pipeline_components.aiplatform.AutoMLTabularTrainingJobRunOp)\n",
    "\n",
    "Define a Job:\n",
    "- Consider Weighting\n",
    "- Model Type\n",
    "- Optimization Objective"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "eb85abc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "@kfp.dsl.pipeline(\n",
    "    name = f'kfp-{NOTEBOOK}-{DATANAME}-{TIMESTAMP}',\n",
    "    pipeline_root = URI+'/'+str(TIMESTAMP)+'/kfp/'\n",
    ")\n",
    "def pipeline(\n",
    "    project: str,\n",
    "    dataname: str,\n",
    "    display_name: str,\n",
    "    deploy_machine: str,\n",
    "    bq_source: str,\n",
    "    var_target: str,\n",
    "    var_omit: str,\n",
    "    features: dict,\n",
    "    labels: dict \n",
    "):\n",
    "    \n",
    "    # dataset\n",
    "    dataset = TabularDatasetCreateOp(\n",
    "        project = project,\n",
    "        display_name = display_name,\n",
    "        bq_source = bq_source,\n",
    "        labels = labels\n",
    "    )\n",
    "    \n",
    "    # training\n",
    "    model = AutoMLTabularTrainingJobRunOp(\n",
    "        project = project,\n",
    "        display_name = display_name,\n",
    "        optimization_prediction_type = \"classification\",\n",
    "        optimization_objective = \"maximize-au-prc\",\n",
    "        budget_milli_node_hours = 1000,\n",
    "        disable_early_stopping=False,\n",
    "        column_specs = features,\n",
    "        dataset = dataset.outputs['dataset'],\n",
    "        target_column = var_target,\n",
    "        predefined_split_column_name = 'splits',\n",
    "        labels = labels\n",
    "    )\n",
    "    \n",
    "    # Endpoint: Creation\n",
    "    endpoint = EndpointCreateOp(\n",
    "        project = project,\n",
    "        display_name = display_name,\n",
    "        labels = labels\n",
    "    )\n",
    "    \n",
    "    # Endpoint: Deployment of Model\n",
    "    deployment = ModelDeployOp(\n",
    "        model = model.outputs[\"model\"],\n",
    "        endpoint = endpoint.outputs[\"endpoint\"],\n",
    "        dedicated_resources_min_replica_count = 1,\n",
    "        dedicated_resources_max_replica_count = 1,\n",
    "        traffic_split = {\"0\": 100},\n",
    "        dedicated_resources_machine_type= deploy_machine\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bd7cda6",
   "metadata": {},
   "source": [
    "---\n",
    "## Compile Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "db3e8ce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "compiler.Compiler().compile(\n",
    "    pipeline_func = pipeline,\n",
    "    package_path = f\"{DIR}/{NOTEBOOK}.json\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72d3a04e",
   "metadata": {},
   "source": [
    "Move compiled pipeline files to GCS Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "671d4cbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://temp/02c/02c.json [Content-Type=application/json]...\n",
      "/ [1 files][ 46.9 KiB/ 46.9 KiB]                                                \n",
      "Operation completed over 1 objects/46.9 KiB.                                     \n"
     ]
    }
   ],
   "source": [
    "!gsutil cp {DIR}/{NOTEBOOK}.json {URI}/{TIMESTAMP}/kfp/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45241ff1",
   "metadata": {},
   "source": [
    "---\n",
    "## Create Vertex AI Pipeline Job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fed3319-fac1-45bb-9483-e02cfd4e0e3e",
   "metadata": {},
   "source": [
    "Get features dictionary for the pipeline input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0ad629ff-3843-4e54-8a80-27bd7bb20cb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get feature names\n",
    "query = f\"SELECT * FROM {DATANAME}.INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '{DATANAME}_prepped'\"\n",
    "schema = bq.query(query).to_dataframe()\n",
    "OMIT = VAR_OMIT.split() + [VAR_TARGET, 'splits']\n",
    "features = schema[~schema.column_name.isin(OMIT)].column_name.tolist()\n",
    "features = dict.fromkeys(features, 'auto')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0adf1a54-05f1-48f9-99e5-3de1c8f40bd5",
   "metadata": {},
   "source": [
    "Run The pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f5dd3e4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = aiplatform.PipelineJob(\n",
    "    display_name = f'{NOTEBOOK}_{DATANAME}_{TIMESTAMP}',\n",
    "    template_path = f\"{URI}/{TIMESTAMP}/kfp/{NOTEBOOK}.json\",\n",
    "    parameter_values = {\n",
    "        \"project\" : PROJECT_ID,\n",
    "        \"dataname\" : DATANAME,\n",
    "        \"display_name\" : f'{NOTEBOOK}_{DATANAME}_{TIMESTAMP}',\n",
    "        \"deploy_machine\" : DEPLOY_COMPUTE,\n",
    "        \"bq_source\" : f'bq://{PROJECT_ID}.{DATANAME}.{DATANAME}_prepped',\n",
    "        \"var_target\" : VAR_TARGET,\n",
    "        \"var_omit\" : VAR_OMIT,\n",
    "        \"features\" : features,\n",
    "        \"labels\" : {'notebook': NOTEBOOK}       \n",
    "    },\n",
    "    labels = {'notebook': NOTEBOOK},\n",
    "    enable_caching=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ee521df",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/kfp-02c-fraud-20230825120638-20230825120849?project=1026793852137\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/kfp-02c-fraud-20230825120638-20230825120849 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "response = pipeline.run(\n",
    "    service_account = SERVICE_ACCOUNT\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e4c8e67-bceb-4629-a246-6a6f18a1c530",
   "metadata": {},
   "source": [
    "Visual Representation of the pipeline can be viewed in the colsole:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "bd262fe9-0693-446a-be88-779e15fb0809",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Pipeline as it runs here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/kfp-02c-fraud-20230825120638-20230825120849?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Pipeline as it runs here:\\nhttps://console.cloud.google.com/vertex-ai/locations/{REGION}/pipelines/runs/{pipeline.resource_name.split('/')[-1]}?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d982aa6-2999-47f6-92a0-01510050e681",
   "metadata": {},
   "source": [
    "Retrieve the pipeline information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "29ff7495",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pipeline_name</th>\n",
       "      <th>run_name</th>\n",
       "      <th>param.input:labels</th>\n",
       "      <th>param.input:project</th>\n",
       "      <th>param.input:display_name</th>\n",
       "      <th>param.input:bq_source</th>\n",
       "      <th>param.input:var_omit</th>\n",
       "      <th>param.input:var_target</th>\n",
       "      <th>param.input:deploy_machine</th>\n",
       "      <th>param.input:features</th>\n",
       "      <th>param.input:dataname</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>kfp-02c-fraud-20230825120638</td>\n",
       "      <td>kfp-02c-fraud-20230825120638-20230825120849</td>\n",
       "      <td>{'notebook': '02c'}</td>\n",
       "      <td>statmike-mlops-349915</td>\n",
       "      <td>02c_fraud_20230825120638</td>\n",
       "      <td>bq://statmike-mlops-349915.fraud.fraud_prepped</td>\n",
       "      <td>transaction_id</td>\n",
       "      <td>Class</td>\n",
       "      <td>n1-standard-2</td>\n",
       "      <td>{'V5': 'auto', 'V28': 'auto', 'Amount': 'auto'...</td>\n",
       "      <td>fraud</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  pipeline_name                                     run_name  \\\n",
       "0  kfp-02c-fraud-20230825120638  kfp-02c-fraud-20230825120638-20230825120849   \n",
       "\n",
       "    param.input:labels    param.input:project  param.input:display_name  \\\n",
       "0  {'notebook': '02c'}  statmike-mlops-349915  02c_fraud_20230825120638   \n",
       "\n",
       "                            param.input:bq_source param.input:var_omit  \\\n",
       "0  bq://statmike-mlops-349915.fraud.fraud_prepped       transaction_id   \n",
       "\n",
       "  param.input:var_target param.input:deploy_machine  \\\n",
       "0                  Class              n1-standard-2   \n",
       "\n",
       "                                param.input:features param.input:dataname  \n",
       "0  {'V5': 'auto', 'V28': 'auto', 'Amount': 'auto'...                fraud  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aiplatform.get_pipeline_df(pipeline = f'kfp-{NOTEBOOK}-{DATANAME}-{TIMESTAMP}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0523ffaa-9dfb-41cb-b7b5-6cc6ddae1613",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "## Evaluation\n",
    "While the model above was trained using AutoML with the API, it is still possible to review the evaluation metrics directly in the Google Cloud Console.  Just visit the Models section of Vertex AI service and select the model and it will present the evaluation metrics with many helpful visuals.\n",
    "\n",
    "It is also possible to retrieve the evaluation metrics for you model using the API.  This section shows how to use the API.\n",
    "\n",
    "<p align=\"center\" width=\"100%\"><center><a href=\"https://youtu.be/m_x3WBtr6y0\" target=\"_blank\" rel=\"noopener noreferrer\"><img src=\"../architectures/thumbnails/playbutton/02c.png\" width=\"40%\"></a></center></p>\n",
    "<p align=\"center\" width=\"100%\"><center>Part 2 Video</center></p>\n",
    "\n",
    "Notes Since Video:\n",
    "- update notebook on 8/31/2022 to:\n",
    "    - make the pipeline run with input parameters rather than using local variable in the pipeline definition\n",
    "    - update the model evaluation section to show update Python Client methods for metric retrieval\n",
    "\n",
    "For more information review [this page](https://cloud.google.com/vertex-ai/docs/training/evaluating-automl-models)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27b1c3f1",
   "metadata": {},
   "source": [
    "Get the Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b6634c0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "models = aiplatform.Model.list(filter=f'labels.notebook={NOTEBOOK}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "a4f7b61b-ad0f-487b-8dc6-ac8ed12e6afb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/models/3955644813528793088'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = models[0]\n",
    "model.resource_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71227e04-8bb0-46d5-862e-3fddfe0361ae",
   "metadata": {},
   "source": [
    "Retrives the aggregate model evalution metrics for the model as a whole.  \n",
    "\n",
    "Either:\n",
    "- First, use `model.list_model_evaluations()` to retrieve the evaluation id, then use `model.get_model_evaluation(evaluation_id = )` for the evaluation id\n",
    "- Or, use `.get_model_evaluation()` and it will retrieve the first model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b7b5d7e0-6ef5-4787-a5d0-953848318989",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation = model.get_model_evaluation().to_dict() # get first evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "35848c5d-d9af-4109-abd5-f093adba2acc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['name', 'metricsSchemaUri', 'metrics', 'createTime', 'sliceDimensions', 'modelExplanation'])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "1763abbd-d2f4-430a-9a03-9e5af413b82a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['confusionMatrix', 'confidenceMetrics', 'auPrc', 'auRoc', 'logLoss'])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation['metrics'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "a562dcf6-02af-4185-96be-5b8b77a5cfd7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9997687"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation['metrics']['auPrc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "45bd1c4b-00a3-4b2c-95a0-bb837307f8a0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'trueNegativeCount': '28387',\n",
       " 'falseNegativeCount': '5',\n",
       " 'falsePositiveRateAt1': 0.00042102308,\n",
       " 'falsePositiveRate': 0.0040348046,\n",
       " 'precision': 0.9959807,\n",
       " 'confidenceThreshold': 0.01,\n",
       " 'recall': 0.9998246,\n",
       " 'f1Score': 0.99789894,\n",
       " 'f1ScoreAt1': 0.99957895,\n",
       " 'truePositiveCount': '28497',\n",
       " 'confusionMatrix': {'annotationSpecs': [{'displayName': '0', 'id': '0'},\n",
       "   {'id': '1', 'displayName': '1'}],\n",
       "  'rows': [[28454.0, 1.0], [11.0, 36.0]]},\n",
       " 'f1ScoreMacro': 0.74111587,\n",
       " 'recallAt1': 0.99957895,\n",
       " 'falsePositiveCount': '115',\n",
       " 'precisionAt1': 0.99957895,\n",
       " 'f1ScoreMicro': 0.99789894}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation['metrics']['confidenceMetrics'][3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39aa9551-2e4a-4673-b0d5-1ddec1233e0e",
   "metadata": {},
   "source": [
    "Review several of the metrics included in the evaluation.  Also, compare these to the results in the console view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "bedd8a0e-cb68-4c45-bb58-c8622b9b0a96",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review this model in the console:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/models/3955644813528793088/versions/1/evaluations/5574753723182965420?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review this model in the console:\\nhttps://console.cloud.google.com/vertex-ai/locations/{REGION}/models/{model.name}/versions/{model.version_id}/evaluations/{evaluation['name'].split('/')[-1]}?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8dd6e04f-c0ec-4ad1-a8be-460cddcb0267",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9997687"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation['metrics']['auPrc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "edd9892b-eeac-4a64-aa9b-68a7a45a3e76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True Label =  0  has Predicted labels =  [28454.0, 1.0]\n",
      "True Label =  1  has Predicted labels =  [11.0, 36.0]\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(evaluation['metrics']['confusionMatrix']['annotationSpecs'])):\n",
    "    print('True Label = ', evaluation['metrics']['confusionMatrix']['annotationSpecs'][i]['displayName'], ' has Predicted labels = ', evaluation['metrics']['confusionMatrix']['rows'][i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2266afe6-be3f-4565-afd9-cb839682df2c",
   "metadata": {},
   "source": [
    "For models with labels you can retrieve the evaluation metrics for each slice of the model as well using the .gapic api version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "703ce330-8b74-44ff-a71f-f7670a1d924f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_client = aiplatform.gapic.ModelServiceClient(\n",
    "    client_options = {\n",
    "        'api_endpoint' : f'{REGION}-aiplatform.googleapis.com'\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "045a6f0d-ab29-4825-a29e-335f5a1727e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "slices = model_client.list_model_evaluation_slices(parent = evaluation['name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "4b7b5c5a-d43c-4c77-8806-fa92f6438e19",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label =  1 has auPrc =  0.86363876\n",
      "Label =  0 has auPrc =  0.9998231\n"
     ]
    }
   ],
   "source": [
    "for slice in slices:\n",
    "    print('Label = ', slice.slice_.value, 'has auPrc = ', slice.metrics['auPrc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3829782e",
   "metadata": {},
   "source": [
    "---\n",
    "## Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0269e25d",
   "metadata": {},
   "source": [
    "### Prepare a record for prediction: instance and parameters lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "5bda8542",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = bq.query(\n",
    "    query = f\"\"\"\n",
    "        SELECT * EXCEPT({VAR_TARGET}, splits, {VAR_OMIT})\n",
    "        FROM {DATANAME}.{DATANAME}_prepped\n",
    "        WHERE splits='TEST'\n",
    "        LIMIT 10\n",
    "    \"\"\"\n",
    ").to_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "bb5a77b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V20</th>\n",
       "      <th>V21</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35337</td>\n",
       "      <td>1.092844</td>\n",
       "      <td>-0.013230</td>\n",
       "      <td>1.359829</td>\n",
       "      <td>2.731537</td>\n",
       "      <td>-0.707357</td>\n",
       "      <td>0.873837</td>\n",
       "      <td>-0.796130</td>\n",
       "      <td>0.437707</td>\n",
       "      <td>0.396770</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.240428</td>\n",
       "      <td>0.037603</td>\n",
       "      <td>0.380026</td>\n",
       "      <td>-0.167647</td>\n",
       "      <td>0.027557</td>\n",
       "      <td>0.592115</td>\n",
       "      <td>0.219695</td>\n",
       "      <td>0.036970</td>\n",
       "      <td>0.010984</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60481</td>\n",
       "      <td>1.238973</td>\n",
       "      <td>0.035226</td>\n",
       "      <td>0.063003</td>\n",
       "      <td>0.641406</td>\n",
       "      <td>-0.260893</td>\n",
       "      <td>-0.580097</td>\n",
       "      <td>0.049938</td>\n",
       "      <td>-0.034733</td>\n",
       "      <td>0.405932</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.265080</td>\n",
       "      <td>-0.060003</td>\n",
       "      <td>-0.053585</td>\n",
       "      <td>-0.057718</td>\n",
       "      <td>0.104983</td>\n",
       "      <td>0.537987</td>\n",
       "      <td>0.589563</td>\n",
       "      <td>-0.046207</td>\n",
       "      <td>-0.006212</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>139587</td>\n",
       "      <td>1.870539</td>\n",
       "      <td>0.211079</td>\n",
       "      <td>0.224457</td>\n",
       "      <td>3.889486</td>\n",
       "      <td>-0.380177</td>\n",
       "      <td>0.249799</td>\n",
       "      <td>-0.577133</td>\n",
       "      <td>0.179189</td>\n",
       "      <td>-0.120462</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.374356</td>\n",
       "      <td>0.196006</td>\n",
       "      <td>0.656552</td>\n",
       "      <td>0.180776</td>\n",
       "      <td>-0.060226</td>\n",
       "      <td>-0.228979</td>\n",
       "      <td>0.080827</td>\n",
       "      <td>0.009868</td>\n",
       "      <td>-0.036997</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>162908</td>\n",
       "      <td>-3.368339</td>\n",
       "      <td>-1.980442</td>\n",
       "      <td>0.153645</td>\n",
       "      <td>-0.159795</td>\n",
       "      <td>3.847169</td>\n",
       "      <td>-3.516873</td>\n",
       "      <td>-1.209398</td>\n",
       "      <td>-0.292122</td>\n",
       "      <td>0.760543</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.923275</td>\n",
       "      <td>-0.545992</td>\n",
       "      <td>-0.252324</td>\n",
       "      <td>-1.171627</td>\n",
       "      <td>0.214333</td>\n",
       "      <td>-0.159652</td>\n",
       "      <td>-0.060883</td>\n",
       "      <td>1.294977</td>\n",
       "      <td>0.120503</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Time        V1        V2        V3        V4        V5        V6  \\\n",
       "0   35337  1.092844 -0.013230  1.359829  2.731537 -0.707357  0.873837   \n",
       "1   60481  1.238973  0.035226  0.063003  0.641406 -0.260893 -0.580097   \n",
       "2  139587  1.870539  0.211079  0.224457  3.889486 -0.380177  0.249799   \n",
       "3  162908 -3.368339 -1.980442  0.153645 -0.159795  3.847169 -3.516873   \n",
       "\n",
       "         V7        V8        V9  ...       V20       V21       V22       V23  \\\n",
       "0 -0.796130  0.437707  0.396770  ... -0.240428  0.037603  0.380026 -0.167647   \n",
       "1  0.049938 -0.034733  0.405932  ... -0.265080 -0.060003 -0.053585 -0.057718   \n",
       "2 -0.577133  0.179189 -0.120462  ... -0.374356  0.196006  0.656552  0.180776   \n",
       "3 -1.209398 -0.292122  0.760543  ... -0.923275 -0.545992 -0.252324 -1.171627   \n",
       "\n",
       "        V24       V25       V26       V27       V28  Amount  \n",
       "0  0.027557  0.592115  0.219695  0.036970  0.010984     0.0  \n",
       "1  0.104983  0.537987  0.589563 -0.046207 -0.006212     0.0  \n",
       "2 -0.060226 -0.228979  0.080827  0.009868 -0.036997     0.0  \n",
       "3  0.214333 -0.159652 -0.060883  1.294977  0.120503     0.0  \n",
       "\n",
       "[4 rows x 30 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "4c32ed6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Time': '35337',\n",
       " 'V1': 1.0928441854981998,\n",
       " 'V2': -0.0132303486713432,\n",
       " 'V3': 1.35982868199426,\n",
       " 'V4': 2.7315370965921004,\n",
       " 'V5': -0.707357349219652,\n",
       " 'V6': 0.8738370029866129,\n",
       " 'V7': -0.7961301510622031,\n",
       " 'V8': 0.437706509544851,\n",
       " 'V9': 0.39676985012996396,\n",
       " 'V10': 0.587438102569443,\n",
       " 'V11': -0.14979756231827498,\n",
       " 'V12': 0.29514781622888103,\n",
       " 'V13': -1.30382621882143,\n",
       " 'V14': -0.31782283120234495,\n",
       " 'V15': -2.03673231037199,\n",
       " 'V16': 0.376090905274179,\n",
       " 'V17': -0.30040350116459497,\n",
       " 'V18': 0.433799615590844,\n",
       " 'V19': -0.145082264348681,\n",
       " 'V20': -0.240427548108996,\n",
       " 'V21': 0.0376030733329398,\n",
       " 'V22': 0.38002620963091405,\n",
       " 'V23': -0.16764742731151097,\n",
       " 'V24': 0.0275573495476881,\n",
       " 'V25': 0.59211469704354,\n",
       " 'V26': 0.219695164116351,\n",
       " 'V27': 0.0369695108704894,\n",
       " 'V28': 0.010984441006191,\n",
       " 'Amount': 0.0}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred['Time'] = pred['Time'].astype(str)\n",
    "newobs = pred.to_dict(orient='records')\n",
    "newobs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2fde74e",
   "metadata": {},
   "source": [
    "Need to understand the format of variables that the predictions expect.  AutoML may convert the type of some variables. The following cells retrieve the model from the endpoint and its schemata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "d438383e",
   "metadata": {},
   "outputs": [],
   "source": [
    "instances = [json_format.ParseDict(newob, Value()) for newob in newobs]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec61f206",
   "metadata": {},
   "source": [
    "### Get Predictions: Python Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "116fefd9-e7c7-4971-8e6b-d630469f2330",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<google.cloud.aiplatform.models.Endpoint object at 0x7fe602bc6950> \n",
       " resource name: projects/1026793852137/locations/us-central1/endpoints/7339602954241966080]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aiplatform.Endpoint.list(filter=f'labels.notebook={NOTEBOOK}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "279cf479",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'02c_fraud_20230825120638'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint = aiplatform.Endpoint.list(filter=f'labels.notebook={NOTEBOOK}')[0]\n",
    "endpoint.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "c7c298d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'classes': ['0', '1'], 'scores': [0.9979369640350342, 0.002062972635030746]}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction = endpoint.predict(instances = instances) # or instances = newobs\n",
    "prediction.predictions[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "1323d595",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction.predictions[0]['classes'][np.argmax(prediction.predictions[0]['scores'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f68781f2",
   "metadata": {},
   "source": [
    "### Get Predictions: REST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "f6b3a5b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(f'{DIR}/request.json','w') as file:\n",
    "    file.write(json.dumps({\"instances\": [newobs[0]]}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "6e7b1b91",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"predictions\": [\n",
      "    {\n",
      "      \"scores\": [\n",
      "        0.99793696403503418,\n",
      "        0.002062972635030746\n",
      "      ],\n",
      "      \"classes\": [\n",
      "        \"0\",\n",
      "        \"1\"\n",
      "      ]\n",
      "    }\n",
      "  ],\n",
      "  \"deployedModelId\": \"8935011918330986496\",\n",
      "  \"model\": \"projects/1026793852137/locations/us-central1/models/3955644813528793088\",\n",
      "  \"modelDisplayName\": \"02c_fraud_20230825120638\",\n",
      "  \"modelVersionId\": \"1\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!curl -X POST \\\n",
    "-H \"Authorization: Bearer \"$(gcloud auth application-default print-access-token) \\\n",
    "-H \"Content-Type: application/json; charset=utf-8\" \\\n",
    "-d @{DIR}/request.json \\\n",
    "https://{REGION}-aiplatform.googleapis.com/v1/{endpoint.resource_name}:predict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b428c9d",
   "metadata": {},
   "source": [
    "### Get Predictions: gcloud (CLI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "d3523017",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using endpoint [https://us-central1-prediction-aiplatform.googleapis.com/]\n",
      "[{'classes': ['0', '1'], 'scores': [0.9979369640350342, 0.002062972635030746]}]\n"
     ]
    }
   ],
   "source": [
    "!gcloud beta ai endpoints predict {endpoint.name.rsplit('/',1)[-1]} --region={REGION} --json-request={DIR}/request.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d0d825b",
   "metadata": {},
   "source": [
    "---\n",
    "## Explanations\n",
    "Interpretation Guide\n",
    "- https://cloud.google.com/vertex-ai/docs/predictions/interpreting-results-automl#tabular"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80280fc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "explanation = endpoint.explain(instances = instances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b2a6912",
   "metadata": {},
   "outputs": [],
   "source": [
    "explanation.predictions[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "a862b22a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attribution:\n",
      "baseline output 0.9965742230415344\n",
      "instance output 0.9962950944900513\n",
      "output_index [0]\n",
      "output display value 0\n",
      "approximation error 0.013086364149574944\n"
     ]
    }
   ],
   "source": [
    "print(\"attribution:\")\n",
    "print(\"baseline output\",explanation.explanations[0].attributions[0].baseline_output_value)\n",
    "print(\"instance output\",explanation.explanations[0].attributions[0].instance_output_value)\n",
    "print(\"output_index\",explanation.explanations[0].attributions[0].output_index)\n",
    "print(\"output display value\",explanation.explanations[0].attributions[0].output_display_name)\n",
    "print(\"approximation error\",explanation.explanations[0].attributions[0].approximation_error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "6afdc889-9620-4e5c-adf9-5ecf2238ad75",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkQAAAIICAYAAACPevFNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyCklEQVR4nO3df7xdd13n+9d7gMRwsVbLURsLHhW8KEpqPFaUKWqjtxmJI16wPQy30A5MxHlwMfq4DPFh74yPuZeH1DrKBafjRA3QFtPcBhKwKS3cDse2Y0s8jaGlVmiLVWPQHqMdGhUU+rl/7LXp7uGck5P27LV39no9H4/9OHt/11r7fNbZ/Hjn+117fVJVSJIkddk/G3UBkiRJo2YgkiRJnWcgkiRJnWcgkiRJnWcgkiRJnWcgkiRJnff0URcwDM9+9rNrenp61GVIkqQ1ctddd/11VU0N6/0nMhBNT08zPz8/6jIkSdIaSfKnw3x/l8wkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnGYgkSVLnPX3UBUiSVmd658FRlyCt6KG3vWzUJTxpzhBJkqTOG4tAlGQuyYWLxnYkuSrJc5N8OMl9Sf4oyfSIypQkSRNqLAIRsAeYXTQ224xfDVxZVd8GnAc83HJtkiRpwo1LINoHbEuyHqCZBdoI/A3w9Kr6CEBVnaiqvx9ZlZIkaSKNRSCqquPAIWBrMzQL7AWeDzyS5P1J/jDJlUmeNqo6JUnSZBqLQNQYXDbrL5c9HTgf+D+A7wG+Gbh0qYOTbE8yn2R+YWFh+NVKkqSJMU6B6ACwJclmYENVHQaOAn9YVZ+uqi80+2xe6uCq2lVVM1U1MzU11VbNkiRpAoxNIKqqE8AcsJve7BDAHwBfnaSfcC4A/qj96iRJ0iQbm0DU2ANsAq4DqKov0lsuuyXJPUCA3xxdeZIkaRKN1Z2qq2o/vdAzOPYR4EWjqUiSJHXBuM0QSZIktW6sZogkScs7nftESePOGSJJktR5BiJJktR5LplJ0mlieufBUZegU+AS5+nFGSJJktR5YzFDlGQO+KWqunlgbAfwrcAJ4GX0wttHgJ+pqhpBmZIkaUKNywzRYB+zvn6D15fQuw/Rd9DrZ/YD7ZYmSZIm3bgEon3AtiTrAZJMAxuBfwS+AlgHrAeeAfzViGqUJEkTaiwCUVUdBw4BW5uhWWBvVd0BfBT4TPO4uaruG02VkiRpUo1FIGoMLpvNAnuSPA/4NuAc4BuAC5K8dKmDk2xPMp9kfmFhoZWCJUnSZBinQHQA2JJkM7Chqg4DPwHcWVUnquoE8CHgxUsdXFW7qmqmqmampqZaK1qSJJ3+xiYQNYFnDthNb7YI4M+AH0jy9CTPoHdBtUtmkiRpTY1NIGrsATYB1zWv9wEPAvcAHwc+XlW/O6LaJEnShBqL+xD1VdV+IAOvvwj81OgqkiRJXTBuM0SSJEmtG6sZIknS8uyNJQ2PM0SSJKnzDESSJKnzXDKTpNPE9M6Doy5horkk2W3OEEmSpM5rNRAlmUty4aKxHUmuSnJTkkeS3LBo+zcl+ViS+5PsTbKuzZolSdLka3uGaLBfWd9sM34lcMkSx1wB/FpVPR/4W+B1Q61QkiR1TtuBaB+wLcl6gCTTwEbg9qq6BXh0cOckAS5ojgN4D/DytoqVJEnd0GogqqrjwCFgazM0C+ytqlrmkLOAR6rqC83ro/S63kuSJK2ZUVxUPbhs1l8uW06WGFsyPCXZnmQ+yfzCwsJTLFGSJHXJKALRAWBLks3Ahqo6vMK+fw2cmaR/e4BzgGNL7VhVu6pqpqpmpqam1rRgSZI02VoPRFV1ApgDdrPy7BDNUtpHgVc2Q68FPjDM+iRJUveM6j5Ee4BNwHX9gSS3AdfTmz06OvD1/LcAP5fkAXrXFP1228VKkqTJNpI7VVfVfhZdH1RV5y+z76eB89qoS5IkdZN3qpYkSZ1nLzNJOk3Ya0saHmeIJElS5xmIJElS57lkJkmniemdB0ddwopc0tPpzBkiSZLUeWMRiJLMDdx3qD+2I8m7ktyV5EiSe5O8YVQ1SpKkyTUWgYgn9jfrmwXeDXx/VZ0LfC+wM8nGdkuTJEmTblwC0T5gW5L1AEmmgY3ArVX1+Waf9YxPvZIkaYKMRcCoquPAIWBrMzQL7K2qSvKcJHcDfw5cUVVLNneVJEl6ssYiEDUGl81mm9dU1Z9X1YuA5wGvTfJ1Sx2cZHuS+STzCwsLrRQsSZImwzgFogP0GrtuBjZU1eHBjc3M0L3Acj3PdlXVTFXNTE1NDb1YSZI0OcYmEFXVCWAO2E0zO5TknCQbmudfDbwE+OSoapQkSZNp3G7MuAd4P48vnX0b8J+SFBDgV6rqnlEVJ0mSJtNYBaKq2k8v+PRffwR40egqkiRJXTA2S2aSJEmjMlYzRJKk5dkrTBoeZ4gkSVLnOUMkSaeJtex272yT9ETOEEmSpM4zEEmSpM4bi0CUZC7JhYvGdiS5KskXkxxpHh8cVY2SJGlyjUUg4ol9zPr6/cz+oarObR7/sv3SJEnSpBuXQLQP2JZkPUCSaWAjcPsoi5IkSd0wFoGoqo4Dh4CtzdAssLeqCviKpov9nUlevtx72O1ekiQ9WWMRiBqDy2b95TKA51bVDPCvgLcn+ZalDrbbvSRJerLGKRAdALYk2QxsqKrDAFV1rPn5aWAO+K5RFShJkibT2ASiqjpBL/DsppkdSvLVA9cVPRt4CfBHo6pRkiRNpnG7U/Ue4P08vnT2bcB/TfIYvfD2tqoyEEmSpDU1VoGoqvYDGXj9+8B3jq4iSZLUBWMViCRJy7P/mDQ8Y3MNkSRJ0qgYiCRJUue5ZCZJp4npnQef1HEutUkn5wyRJEnqPAORJEnqvFYDUZK5JBcuGtuR5KokNyV5JMkNi7a/N8knk3wiye4kz2izZkmSNPnaniEa7FfW1+9bdiVwyRLHvBd4Ab37EW0AXj/MAiVJUve0HYj2AdsG2nFMAxuB26vqFuDRxQdU1Y3VAA4B57RYryRJ6oBWA1FVHacXarY2Q7PA3ibsrKhZKrsEuGmZ7duTzCeZX1hYWKuSJUlSB4ziourBZbP+ctlqXAXcWlW3LbWxqnZV1UxVzUxNTa1BmZIkqStGEYgOAFuSbAY2VNXhkx2Q5D8AU8DPDbk2SZLUQa3fmLGqTiSZA3azitmhJK8HLgS2VNVjQy5PkiR10KjuQ7QH2ARc1x9IchtwPb3Zo6MDX8//DeDrgDuSHEny71uvVpIkTbSRtO6oqv1AFo2dv8y+theRJElDZdiQpNOEPcmk4bF1hyRJ6jwDkSRJ6jyXzCTpNDG98+Ap7e8Sm7R6zhBJkqTOMxBJkqTOazUQJZkbuL9Qf2xHkhuT3JHk3iR3J7l4YPttzf2HjiQ5luRAmzVLkqTJ1/Y1RP0+ZjcPjM0CbwGOVdX9STYCdyW5uaoeGbw/UZL3AR9otWJJkjTx2l4y2wdsS7IeIMk0sJFe09b7AarqGPAwvd5lX5LkK4EL6PVCkyRJWjOtBqKqOg4cArY2Q7PA3qqq/j5JzgPWAQ8uOvwngFuq6rNLvXeS7Unmk8wvLCysffGSJGlijeKi6v6yGc3PLzV4TXI2cA1w2RKNXF/FCs1gq2pXVc1U1czU1NRyu0mSJH2ZUQSiA/QauG4GNlTVYYAkZwAHgcur6s7BA5KcBZzXbJckSVpTrQeiqjoBzAG7aWZ8kqwD9gNXV9X1Sxz2k8ANVfW5tuqUJEndMar7EO0BNgHXNa8vAl4KXDrwFftzB/Z/wtKaJEnSWhpJ646q2g9k4PW1wLUr7P+DLZQlSZI6yl5mknSasDeZNDy27pAkSZ1nIJIkSZ3nkpkknSamdy595xGX0qSnzhkiSZLUeQYiSZLUea0GoiRzSS5cNLYjyY1J7khyb5K7k1w8sP2NSR5IUkme3Wa9kiSpG9qeIRrsY9Y3C1wBvKaqXkiv8evbk5zZbP/vwA8Df9pWkZIkqVvaDkT7gG1J1gMkmQY2ArdW1f0AVXUMeBiYal7/YVU91HKdkiSpQ1oNRFV1HDhEbxYIerNDe6uq+vskOQ9YBzx4Ku+dZHuS+STzCwsLa1WyJEnqgFFcVD24bPaEHmVJzgauAS6rqsdO5U2raldVzVTVzNTU1JoVK0mSJt8oAtEBYEuSzcCGqjoMkOQM4CBweVXdOYK6JElSR7UeiKrqBDAH7KaZHUqyDtgPXF1V17ddkyRJ6rZR3YdoD7AJuK55fRHwUuDSJEeax7kASd6U5ChwDnB3kt8aRcGSJGlyjaR1R1XtBzLw+lrg2mX2fQfwjpZKkyRJHWQvM0k6TdizTBoeW3dIkqTOMxBJkqTOMxBJ0mlieudBpnceHHUZ0kQyEEmSpM4zEEmSpM5rNRAlmUty4aKxHUmuSnJTkkeS3LBo+wVJDif5RJL3JPGbcZIkaU21PUM02Mesr9/P7ErgksENSf4Z8B5gtqq+A/hT4LUt1ClJkjqk7UC0D9iWZD1AkmlgI3B7Vd0CPLpo/7OAz1fVp5rXHwFe0VKtkiSpI1oNRFV1HDgEbG2GZoG9VVXLHPLXwDOSzDSvXwk8Z6kdk2xPMp9kfmFhYS3LliRJE24UF1UPLpv1l8uW1ASlWeDXkhyiN4P0hWX23VVVM1U1MzU1tcYlS5KkSTaKC5QPAL+aZDOwoaoOr7RzVd0BnA+Q5H8BvnXoFUqSpE5pfYaoqk4Ac8BuVpgd6kvytc3P9cBbgN8YZn2SJKl7RnUfoj3AJuC6/kCS24DrgS1Jjg58Pf/NSe4D7gZ+t6r+W+vVSpKkiTaSe/pU1X4gi8bOX2bfNwNvbqMuSZLUTd7kUJJOEw+97WWjLkGaWLbukCRJnWcgkiRJnWcgkqTTxPTOg6MuQZpYBiJJktR5BiJJktR5rQaiJHMD9xfqj+1IclWSm5I8kuSGRduT5K1JPpXkviRvarNmSZI0+dr+2n2/j9nNA2Oz9O4ztA54JvBTi465lF5D1xdU1WP9O1dLkiStlbaXzPYB25o2HCSZBjYCt1fVLfSaty7208B/rKrHAKrq4ZZqlSRJHdFqIKqq48AhYGszNAvsbbraL+dbgIuTzCf5UJLnL7VTku3NPvMLCwtrW7gkSZpoo7iour9sRvPzZA1e1wOfq6oZ4DfpNYX9MlW1q6pmqmpmampqzYqVJEmTbxSB6AC9Bq6bgQ1Vdfgk+x8F3tc83w+8aIi1SZKkDmo9EFXVCWCO3kzPyWaHoBegLmie/wDwqaEUJkmSOmtU9yHaA2wCrusPJLkNuJ7e7NHRga/nvw14RZJ7gF8CXt92sZIkabKNpNt9Ve0Hsmjs/GX2fQSwxbMkSRoa71QtSaeJh97mvw2lYTEQSZKkzjMQSZKkzjMQSZKkzjMQSZKkzjMQSZKkzms1ECWZG7i/UH9sR5KrktyU5JEkNyzafluSI83jWJIDbdYsSZImX9v3Ier3Mbt5YGwWeDOwDngm8FODBwzenyjJ+4APDL9MSZLUJW0vme0DtiVZD5BkGtgI3F5VtwCPLndgkq+k18LjwPDLlCRJXdJqIKqq48AhYGszNAvsrapaxeE/AdxSVZ9damOS7Unmk8wvLCysTcGSJKkTRnFRdX/ZjObnahq8ArxqpX2raldVzVTVzNTU1FMsUZIkdckoAtEBeg1cNwMbqurwyQ5IchZwHnBwyLVJkqQOaj0QVdUJYA7Yzepnh34SuKGqPjesuiRJUneN6j5Ee4BNwHX9gSS3AdfTmz06uujr+aeytCZJknRK2v7aPQBVtR/IorHzl9mdqvrBYdckSZK6yztVS5KkzjMQSZKkzjMQSZKkzjMQSZKkzjMQSZKkzjMQSZKkzms1ECWZW3R/IZLsSHJjkjuS3Jvk7iQXD2zfkuRwkiNJbk/yvDZrliRJk6/tGaLBPmZ9s8AVwGuq6oX0Gr++PcmZzfb/Ary6qs4Ffge4vJ1SJUlSV7QdiPYB25KsB0gyDWwEbq2q+wGq6hjwMNDv0FrAGc3zrwKOtVmwJEmafK3eqbqqjic5RG8W6AP0Zof2VlX190lyHrAOeLAZej1wY5J/AD4LvHip906yHdgO8NznPndo5yBJkibPKC6qHlw2e0KPsiRnA9cAl1XVY83wzwI/WlXnAO8CfnWpN62qXVU1U1UzU1NTS+0iSZK0pFEEogP0GrhuBjZU1WGAJGcAB4HLq+rOZmwK2FRVH2uO3Qt8f/slS5KkSdZ6IKqqE8AcsJtmdijJOmA/cHVVXT+w+98CX5XkW5vXPwLc1161kiSpC0bS7Z5eEHo/jy+dXQS8FDgryaXN2KVVdSTJvwHel+QxegHpX7ddrCRJmmwZuJ55YszMzNT8/Pyoy5AkSWskyV1VNTOs9/dO1ZIkqfMMRJIkqfMMRJIkqfMMRJIkqfMMRJIkqfMMRJIkqfNaDURJ5pJcuGhsR5Ibk9yR5N4kdye5eIlj35nkRHvVSpKkrmj7xoz9PmY3D4zNAm8BjlXV/Uk2AnclubmqHgFIMgOc2XKtkiSpI9peMtsHbEuyHiDJNLARuLWq7geoqmPAw8BUs8/TgCuBf9dyrZIkqSNaDURVdRw4BGxthmaBvTVwu+wk5wHrgAeboTcCH6yqz6z03km2J5lPMr+wsLD2xUuSpIk1iouq+8tmND/39DckORu4Brisqh5rls9+Enjnyd60qnZV1UxVzUxNTQ2hbEmSNKlGEYgOAFuSbAY2VNVhgCRnAAeBy6vqzmbf7wKeBzyQ5CHgmUkeaL9kSZI0yVrvdl9VJ5LMAbtpZoeSrAP2A1dX1fUD+x4Evr7/OsmJqnpeuxVLkqRJN6r7EO0BNgHXNa8vAl4KXJrkSPM4d0S1SZKkjml9hgigqvYDGXh9LXDtKo571jDrkiRJ3eSdqiVJUucZiCRJUucZiCRJUucZiCRJUucZiCRJUueNS7f7q5LclOSRJDcs2v7GJA8kqSTPbrNeSZLUDW3PEA227ejrt++4ErhkiWP+O/DDwJ8OtzRJktRV49Lt/vaqugV4dPEBVfWHVfVQm0VKkqRuGbtu95IkSW0bq273T0WS7Unmk8wvLCysxVtKkqSOGJtu909VVe2qqpmqmpmamlqLt5QkSR3ReiCqqhPAHAPd7iVJkkZpXLrdk+Q24Hp6s0dH+1/PT/KmJEeBc4C7k/zWKAqWJEmTayy63Tdj5y+z7zuAd7RRlyRJ6ibvVC1JkjrPQCRJkjrPQCRJkjrPQCRJkjrPQCRJkjrPQCRJkjpvLAJRkrn+fYcGxnYkuSrJTUkeSXLDqOqTJEmTbSwCEU/sb9bX73N2JXBJ6xVJkqTOGJdAtA/YlmQ9QJJpYCNwe1XdAjw6wtokSdKEG4tAVFXHgUPA1mZoFthbVTW6qiRJUleMRSBqDC6b9ZfLVi3J9iTzSeYXFhbWvDhJkjS5xikQHaDX2HUzsKGqDp/KwVW1q6pmqmpmampqKAVKkqTJNDaBqKpOAHPAbk5xdkiSJOmpGJtA1NgDbAKu6w8kuQ24nt7s0dHFX8+XJEl6qp4+6gIGVdV+IIvGzh9ROZIkqSPGbYZIkiSpdQYiSZLUeQYiSZLUeQYiSZLUeQYiSZLUeQYiSZLUeWMRiJLMLb6/UJIdSa5KckWSTzSPi0dVoyRJmlxjEYh4Yh+zvlngr4DNwLnA9wJvTnJGu6VJkqRJNy6BaB+wLcl6gCTTwEbg74Hfq6ovVNXfAR8Hto6sSkmSNJHGIhBV1XHgEI+HnVlgL70A9C+SPDPJs4EfAp4zmiolSdKkGotA1BhcNpsF9lTVh4Ebgd9vtt8BfGGpg5NsTzKfZH5hYaGNeiVJ0oQYp0B0gF4D183Ahqo6DFBVb62qc6vqR+j1Obt/qYOraldVzVTVzNTUVGtFS5Kk09/YBKKqOgHMAbvpzQaR5GlJzmqevwh4EfDhUdUoSZIm01h1u6cXhN7P40tnzwBuSwLwWeB/q6oll8wkSZKerLEKRFW1n96yWP/154BvH11FkiSpC8ZmyUySJGlUDESSJKnzDESSJKnzDESSJKnzxuqiakkSTO88uOT4Q297WcuVSN3hDJEkSeq8VgNRkrkkFy4a25HkxiR3JLk3yd1JLh7YniRvTfKpJPcleVObNUuSpMnX9pJZv1/ZzQNjs8BbgGNVdX+SjcBdSW6uqkeAS+k1dH1BVT2W5GtbrlmSJE24tpfM9gHbkqwHSDINbARurar7AarqGPAw0G9I9tPAf6yqx5rtD7dcsyRJmnCtBqKqOg4cArY2Q7PA3qqq/j5JzgPWAQ82Q98CXNx0sv9Qkue3WbMkSZp8o7iour9sRvNzT39DkrOBa4DL+jNCwHrgc1U1A/wmveavXybJ9iY0zS8sLAyteEmSNHlGEYgOAFuSbAY2VNVhgCRnAAeBy6vqzoH9jwLva57vp9fx/stU1a6qmqmqmampqaV2kSRJWlLrgaiqTgBz9GZ69gAkWUcv7FxdVdcvOuQAcEHz/AeAT7VSqCRJ6oxR3YdoD7AJuK55fRHwUuDSJEeax7nNtrcBr0hyD/BLwOvbLlaSJE22kdypuqr2Axl4fS1w7TL7PgJ4e1ZJkjQ03qlakiR1nr3MJGnM2LNMap8zRJIkqfMMRJIkqfNcMpOkEZneefCU9ncpTRoeZ4gkSVLnjUUgSjKX5MJFYzuS3DdwX6IjST6X5OUjKlOSJE2osQhEPLG/Wd8ssL2qzq2qc+ndrfrvgQ+3XJskSZpw4xKI9gHbkqwHSDINbARuH9jnlcCHqurv2y9PkiRNsrEIRFV1HDgEbG2GZoG9VVUDu83S9D5bit3uJUnSkzUWgagxuGz2hPCT5GzgO4GblzvYbveSJOnJGqdAdADYkmQzsKGqDg9suwjYX1X/NJLKJEnSRBubQFRVJ4A5YDdfvjT2qiXGJEmS1sTYBKLGHmATcF1/oLnA+jnA742oJkmSNOHG6k7VVbUfyKKxh4BvGElBkiSpE8YqEElSl9iKQxof47ZkJkmS1DoDkSRJ6jyXzCRpiE61o/1KXGKThscZIkmS1HkGIkmS1HmtBqIkc0kuXDS2I8lVSW5K8kiSG5Y59p1JTrRTqSRJ6pK2Z4gG+5X19fuWXQlcstRBSWaAM4damSRJ6qy2A9E+YFuS9fClu1BvBG6vqluARxcfkORp9MLSv2uxTkmS1CGtBqKqOg4cArY2Q7PA3qqqFQ57I/DBqvrMSu+dZHuS+STzCwsLa1OwJEnqhFFcVD24bNZfLltSko3ATwLvPNmbVtWuqpqpqpmpqak1KVSSJHXDKALRAWBLks3Ahqo6vMK+3wU8D3ggyUPAM5M8MPwSJUlSl7R+Y8aqOpFkDtjNCrNDzb4Hga/vv05yoqqeN9wKJUlS14zqPkR7gE3Adf2BJLcB19ObPTq6+Ov5kiRJwzKS1h1VtR/IorHzV3Hcs4ZWlCRJ6ix7mUnSENl/TDo92LpDkiR1noFIkiR1nktmkvQUTO882NrvcvlNGh5niCRJUucZiCRJUucNLRAlOSvJkebxl0n+onl+IslVw/q9kiRJp2po1xA1jVzPBUjyi8CJqvqVYf0+SZKkJ6v1JbMkP5jkhub5LyZ5T5IPJ3koyf+a5JeT3JPkpiTPaPb77iS/l+SuJDcnObvtuiVJ0uQah2uIvgV4GfDjwLXAR6vqO4F/AF7WhKJ3Aq+squ+m1wPtrYvfJMn2JPNJ5hcWFtqrXpIknfbG4Wv3H6qqf0pyD/A04KZm/B5gGvifge8APpKEZp/PLH6TqtoF7AKYmZmp4ZctSZImxTgEos8DVNVjSf6pqvph5jF69QW4t6q+b1QFSpKkyTYOS2Yn80lgKsn3ASR5RpIXjrgmSZI0QcY+EFXVPwKvBK5I8nHgCPD9Iy1KkiRNlFaWzKrqFweezwFzi8eb189a5pgjwEuHWaMkSequcbiGSJJOW/YXkybD2C+ZSZIkDZuBSJIkdZ5LZpL0JEzvPNj673R5ThoeZ4gkSVLnGYgkSVLntRqIkswluXDR2I4kNya5I8m9Se5OcvHA9m9K8rEk9yfZm2RdmzVLkqTJ1/YM0R5gdtHYLHAF8JqqeiGwFXh7kjOb7VcAv1ZVzwf+FnhdS7VKkqSOaDsQ7QO2JVkPkGQa2AjcWlX3A1TVMeBheu06AlzQHAfwHuDlLdcsSZImXKuBqKqOA4fozQJBb3Zo70BDV5KcB6wDHgTOAh6pqi80m48C37DUeyfZnmQ+yfzCwsKwTkGSJE2gUVxUPbhsNtu8BiDJ2cA1wGVV9Ri9TveL1RJjVNWuqpqpqpmpqak1LlmSJE2yUQSiA8CWJJuBDVV1GCDJGcBB4PKqurPZ96+BM5P075d0DnCs5XolSdKEaz0QVdUJes1dd9PMDjXfHNsPXF1V1w/sW8BH6XW7B3gt8IE265UkSZNvVPch2gNsAq5rXl9Er5v9pUmONI9zm21vAX4uyQP0rin67baLlSRJk20krTuqaj8D1wdV1bXAtcvs+2ngvJZKkyRJHWQvM0l6EuwrJk0WW3dIkqTOMxBJkqTOc8lMklZheufBUZfgMp00RM4QSZKkzjMQSZKkzms1ECWZS3LhorEdSW5MckeSe5PcneTige2/neTjzfi+JM9qs2ZJkjT52p4hGuxj1jcLXAG8pqpeSK/x69uTnNls/9mq2lRVLwL+DHhjW8VKkqRuaDsQ7QO2JVkPkGQa2AjcWlX3A1TVMeBhYKp5/dlm3wAbWKa5qyRJ0pPVaiCqquPAIXqzQNCbHdrb9CwDIMl5wDrgwYGxdwF/CbwAeOdS751ke5L5JPMLCwtDOgNJkjSJRnFR9eCy2WzzGoAkZwPXAJdV1WP98aq6jN5M0n3AxSyhqnZV1UxVzUxNTQ2rdkmSNIFGEYgOAFuSbAY2VNVhgCRnAAeBy6vqzsUHVdUXgb3AK1qsVZIkdUDrgaiqTgBzwG6a2aEk64D9wNVVdX1/3/Q8r/8c+DHgj9uuWZIkTbZR3al6D/B+Hl86uwh4KXBWkkubsUuBu4H3NLNHAT4O/HSrlUqSpIk3kkBUVfvpBZz+62uBa5fZ/SWtFCVJkjrLXmaStAr2EZMmm607JElS5xmIJElS57lkJklLmN55cNQlfBmX7aThcYZIkiR1noFIkiR13lgEoiRzSS5cNLYjyY1J7khyb5K7kyzZtkOSJOmpGJdriPr9zW4eGJsF3gIcq6r7k2wE7kpyc1U9MoIaJUnShBqLGSJgH7AtyXqAJNP0mrneWlX3A1TVMeBhwM6tkiRpTY1FIKqq48AhYGszNAvsrarq75PkPGAd8OBS75Fke5L5JPMLCwvDLlmSJE2QsQhEjf6yGc3PPf0NSc4GrgEuq6rHljq4qnZV1UxVzUxNOYkkSZJWb5wC0QFgS5LNwIaqOgzQNHY9CFxeVXeOsD5JkjShxiYQVdUJYA7YTTM7lGQdsB+4uqquH111kiRpko1NIGrsATYB1zWvLwJeClya5EjzOHdUxUmSpMk0Ll+7B6Cq9gMZeH0tcO3oKpIkSV0wVoFIksaFfcOkbhm3JTNJkqTWGYgkSVLnuWQmScD0zoOjLuGkXMaThscZIkmS1HkGIkmS1HmtBqIkc0kuXDS2I8lVSW5K8kiSGxZt/+0kH09yd5J9SZ7VZs2SJGnytT1DNNivrK/ft+xK4JIljvnZqtpUVS8C/gx443BLlCRJXdN2INoHbEuyHiDJNLARuL2qbgEeXXxAVX222TfABqBaq1aSJHVCq4Goqo4Dh4CtzdAssLeqVgw5Sd4F/CXwAuCdy+yzPcl8kvmFhYU1rFqSJE26UVxUPbhs1l8uW1FVXUZvJuk+4OJl9tlVVTNVNTM1NbVWtUqSpA4YRSA6AGxJshnYUFWHV3NQVX0R2Au8Yoi1SZKkDmo9EFXVCWAO2M1JZofS87z+c+DHgD8edo2SJKlbRnWn6j3A+xn4xlmS2+hdI/SsJEeB1wEfAd6T5AwgwMeBn26/XEmSNMlGEoiqaj+9gDM4dv4yu79k+BVJkqQus5eZJGGfMKnrbN0hSZI6z0AkSZI6zyUzSZ00vfPgqEs4ZS7rScPjDJEkSeo8A5EkSeq8VgNRkrkkFy4a25HkqiQ3JXkkyQ2Ltr87yZ8kOdI8zm2zZkmSNPnavoao38fs5oGxWeDNwDrgmcBPLXHcm6tq3/DLkyRJXdT2ktk+YFuS9QBJpuk1bb29qm4BHm25HkmSpHYDUVUdBw4BW5uhWWBvVdVJDn1rkruT/Fo/TC2WZHuS+STzCwsLa1i1JEmadKO4qLq/bEbzc8UGr8DP0+tx9j3A1wBvWWqnqtpVVTNVNTM1NbVWtUqSpA4YRSA6AGxJshnYUFWHV9q5qj5TPZ8H3gWc10KNkiSpQ1oPRFV1ApgDdnPy2SGSnN38DPBy4BNDLE+SJHXQqO5UvQd4P48vnZHkNnpLY89KchR4XVXdDLw3yRQQ4AjwhvbLlSRJk2wkgaiq9tMLOINj5y+z7wWtFCVJkjrLXmaSOsm+YJIG2bpDkiR1noFIkiR1nktmkibW9M6Doy5hTbnMJw2PM0SSJKnzDESSJKnzWg1ESeaSXLhobEeSG5PckeTepmfZxQPb353kT5IcaR7ntlmzJEmafG1fQ9TvY3bzwNgsvf5kx6rq/iQbgbuS3FxVjzT7vLmq9rVbqiRJ6oq2l8z2Adv6HeuTTAMbgVur6n6AqjoGPAzYoVWSJLWi1UBUVceBQ8DWZmgW2FtV1d8nyXnAOuDBgUPf2iyl/Vo/TC2WZHuS+STzCwsLQzoDSZI0iUZxUXV/2Yzm55cavDaNXK8BLquqx5rhn6fX4+x7gK+ht7z2ZapqV1XNVNXM1JSTS5IkafVGEYgOAFuSbAY2VNVhgCRnAAeBy6vqzv7OVfWZ6vk88C7gvBHULEmSJljrgaiqTgBzwG6a2aEk64D9wNVVdf3g/s2sEUkCvBz4RIvlSpKkDhjVnar3AO/n8aWzi4CXAmclubQZu7SqjgDvTTIFBDgCvKHVSiVJ0sQbSSCqqv30Ak7/9bXAtcvse0FbdUmSpG6yl5mkiWXvL0mrZesOSZLUeQYiSZLUeS6ZSTrtTe88OOoSWuESoDQ8zhBJkqTOG5du91cluSnJI0luWLR9S5LDTaf725M8r82aJUnS5Gt7hmiwbUdfv33HlcAlSxzzX4BXV9W5wO8Alw+zQEmS1D3j0u3+9qq6BXh0iWMKOKN5/lXAsRbqlCRJHdLqRdVVdTxJv9v9B1ii2/0SXg/cmOQfgM8CLx5+pZIkqUvGqtv9Mn4W+NGqOodec9dfXWqnJNuTzCeZX1hYWLNiJUnS5BubbvdLaXqYbaqqjzVDe4HvX2rfqtpVVTNVNTM1NbXWNUuSpAk2Ft3uV/C3wFcl+dbm9Y8A9w2vOkmS1EXj0u2eJLcBLwCeleQo8LqqujnJvwHel+QxegHpX4+iYEmSNLnGott9M3b+Cvvub6MuSZLUTd6pWpIkdZ69zCSd9uzxJempcoZIkiR1noFIkiR1nktmksbC9M6Doy5h7Lk0KA2PM0SSJKnzWg1ESeaSXLhobEeSG5PckeTeJHcnuXhg+3uTfDLJJ5LsTvKMNmuWJEmTr+0ZosE+Zn2zwBXAa6rqhfQav749yZnN9vfSu2HjdwIb6DV7lSRJWjNtB6J9wLYk6wGSTAMbgVur6n6AqjoGPAxMNa9vrAZwCDin5ZolSdKEazUQVdVxeqFmazM0C+xtwg4ASc4D1gEPDh7bLJVdAtzUTrWSJKkrRnFR9eCy2SwDDV6TnA1cA1xWVY8tOu4qejNJty31pkm2J5lPMr+wsDCEsiVJ0qQaRSA6AGxJshnYUFWHAZKcARwELq+qOwcPSPIf6C2h/dxyb1pVu6pqpqpmpqamhla8JEmaPK3fh6iqTiSZA3bTzA4lWUevgevVVXX94P5JXg9cCGxZYtZIkiTpKRvVfYj2AJuA65rXFwEvBS5NcqR5nNts+w3g64A7mvF/33q1kiRpoo3kTtVVtR/IwOtrgWuX2de7aUuSpKHyTtWSJKnznH2RNBbs0yVplJwhkiRJnWcgkiRJneeSmaRTMr3z4KhL6CyXFaXhcYZIkiR13qoCUZKfSFJJXjDsglaoYUeSZ47q90uSpMm12hmiVwG383gPslHYARiIJEnSmjtpIEryLOAlwOtoAlGSH0zye0n+3ySfSvK2JK9OcijJPUm+pdnvG5PckuTu5udzm/F3J3nlwO84MfC+c0n2JfnjJO9Nz5uAjcBHk3x0zf8KkiSp01YzQ/Ry4Kaq+hTwN01TVui13vgZ4DuBS4BvrarzgN8C/vdmn1+n15/sRcB7gXes4vd9F73ZoG8Hvhl4SVW9AzgG/FBV/dAq3kOSJGnVVhOIXsXjPceua14D/EFVfaaqPg88CHy4Gb8HmG6efx/wO83za4B/vorfd6iqjjaNXI8MvNeKkmxPMp9kfmFhYTWHSJIkASf52n2Ss4ALgO9IUsDTgAJuBD4/sOtjA68fW+F9q/n5BZowliTAuoF9Bt/3iyer8UtvXLUL2AUwMzNTJ9ldkiTpS042Q/RKekte31hV01X1HOBPWN1MD8Dv8/iF2K+md2E2wEPAdzfPfxx4xire61HgK1f5eyVJklbtZIHoVcD+RWPvA/7VKt//TcBlSe6md53RzzTjvwn8QJJDwPcCf7eK99oFfMiLqiVJ0lpL1eStLs3MzNT8/Pyoy5AmkneqHh3vVK0uS3JXVc0M6/29U7UkSeo8e5lJOiXOUkiaRM4QSZKkzjMQSZKkznPJ7BR5QamkUXG5UhoeZ4gkSVLnjUUgahq6XrhobEeSq5rnZyT5iyS/PpoKJUnSJBuLQATs4fE7WvfNNuMA/xfwe61WJEmSOmNcAtE+YFuS9QBJpoGNwO1Jvhv4Oh5vHitJkrSmxiIQVdVx4BCwtRmaBfYCAf4T8OYRlSZJkjpgLAJRY3DZrL9c9m+BG6vqz092cJLtSeaTzC8sLAyxTEmSNGnGKRAdALYk2QxsqKrDwPcBb0zyEPArwGuSvG2pg6tqV1XNVNXM1NRUWzVLkqQJMDb3IaqqE0nmgN00F1NX1av725NcCsxU1c6RFChJkibWOM0QQS8IbQKuG3UhkiSpO8ZmhgigqvbTu5B6qW3vBt7dZj2SJKkbxm2GSJIkqXVjNUN0OrCXkCRJk8cZIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HkGIkmS1HmpqlHXsOaSLAB/Ouo6WvJs4K9HXcSIeO7d5Ll3k+feTYPn/o1VNTWsXzSRgahLksxX1cyo6xgFz91z7xrP3XPvmjbP3SUzSZLUeQYiSZLUeQai09+uURcwQp57N3nu3eS5d1Nr5+41RJIkqfOcIZIkSZ1nIBqhJF+T5CNJ7m9+fvUy+21N8skkDyTZuZrjk/x8s/8nk1zYjD0zycEkf5zk3iRvG9h/fZK9zTEfSzI9xFNv/dyb8bcm+fMkJxb9jkuTLCQ50jxeP4xzXk3ti/Zr49y78Ll/d5J7mm3vSJJmvJXPfblzGdiepq4HktydZHObf4dhGqNzn2vG+p/11w7zvFeqf2D7mp17krOSfDTJiSS/vuj3TPTnfpJzP7XPvap8jOgB/DKws3m+E7hiiX2eBjwIfDOwDvg48O0rHQ98e7PfeuCbmuOfBjwT+KFmn3XAbcC/aF7/W+A3muezwN5JOvdm24uBs4ETi37PpcCvT+rnfpJz78Lnfgj4PiDAhwb+Mz/0z32lcxnY50ebutJ8Th9r8+/QkXOfA2aGeb4jPvf/CfjnwBsW/2e6A5/7Sud+Sp+7M0Sj9ePAe5rn7wFevsQ+5wEPVNWnq+ofgeua41Y6/seB66rq81X1J8ADwHlV9fdV9VGA5r0OA+cs8V77gC1D/pdEq+cOUFV3VtVn1vg8noxxOveJ/tyTnA2cUVV3VO9/Ia9e5ncOy0rn0vfjwNXVcydwZlP36f53GItzH9K5nUyr515Vf1dVtwOfG/wFXfjclzv3J8NANFpf1/8/qebnUtN53wD8+cDro83YSsevdAwASc4Efgy4ZfExVfUF4H8AZz2Zk1qlkZ37Ml7RTN3uS/Kc1Z/GkzJO5z7pn/s3NM+Xei8Y/ue+ms9kpdrb+jsMw7ice9+7mmWT/7OFZaO2z32lOib9cz+ZVX/uT1/lG+pJSvL/AV+/xKZfWO1bLDF2sq8GrnhMkqcDe4B3VNWnn8LvWbmIMTz3ZfwusKeqPp/kDfT+9XHBKupbvojT59wn/XNf6b3W/HM/hbpWs09bf4dhGZdzB3h1Vf1Fkq8E3gdcQm+2ZFjaPvenUsdaG5dzh1P83A1EQ1ZVP7zctiR/leTsqvpMM1348BK7HQUG/+V6DnCseb7c8SsdA737OtxfVW9f4vccbQLTVwF/c9ITXMGYnvtSdR4fePmbwBUr7b8ap8u5M/mf+1EeXxZ+wnsN43Nfwmo+k+X2WbfCsWv2dxiicTl3quovmp+PJvkdekszwwxEbZ/7SnVM+ue+rFP93F0yG60PAq9tnr8W+MAS+/wB8Pwk35RkHb0LXz94kuM/CMym9w2ibwKeT+/COpL83/T+T2/HCrW8EvhvzZrzsLR+7stp/svV9y+B+07xXE7V2Jw7E/65N1PrjyZ5cTNd/pr+MS197iudS98Hgdc037x5MfA/mrpb+TsM0Vice5KnJ3k2QJJnANuATwzjhAe0fe5L6sjnvqQn9blXS1fd+1jyavyz6F3Dc3/z82ua8Y3AjQP7/SjwKXpX3//CyY5vtv1Cs/8nefxbNefQm368DzjSPF7fbPsK4Hp6FyIeAr55ks69Gf9lev8yeaz5+YvN+C8B99L7RsNHgRd06Ny78LnP0PsfwgeBX+fxG9K28rkvdS70vhHzhuZ5gP/cbL+HgW/FtPF3GPLnPfJzp/ctpLuAu5vP+/+h+fbZhJ37Q/Rmd0/Q++94/9tZXfjcv+zcn8zn7p2qJUlS57lkJkmSOs9AJEmSOs9AJEmSOs9AJEmSOs9AJEmSOs9AJEmSOs9AJEmSOs9AJEmSOu//B5RaS8f8Yd0fAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 648x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "features = []\n",
    "scores = []\n",
    "for k in explanation.explanations[0].attributions[0].feature_attributions:\n",
    "    features.append(k)\n",
    "    scores.append(explanation.explanations[0].attributions[0].feature_attributions[k])\n",
    "features = [x for _, x in sorted(zip(scores, features))]\n",
    "scores = sorted(scores)\n",
    "fig, ax = plt.subplots()\n",
    "fig.set_size_inches(9, 9)\n",
    "ax.barh(features, scores)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a12e906",
   "metadata": {},
   "source": [
    "---\n",
    "## Batch Predictions: BigQuery Source to BigQuery Destination, with Explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7145bae3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:Creating BatchPredictionJob\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob created. Resource name: projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976\n",
      "INFO:google.cloud.aiplatform.jobs:To use this BatchPredictionJob in another session:\n",
      "INFO:google.cloud.aiplatform.jobs:bpj = aiplatform.BatchPredictionJob('projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976')\n",
      "INFO:google.cloud.aiplatform.jobs:View Batch Prediction Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/batch-predictions/5522655591295614976?project=691911073727\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/691911073727/locations/us-central1/batchPredictionJobs/5522655591295614976 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "batch = aiplatform.BatchPredictionJob.create(\n",
    "    job_display_name = f'{NOTEBOOK}_{DATANAME}_{TIMESTAMP}',\n",
    "    model_name = endpoint.list_models()[0].model,\n",
    "    instances_format = \"bigquery\",\n",
    "    predictions_format = \"bigquery\",\n",
    "    bigquery_source = f'bq://{PROJECT_ID}.{DATANAME}.{DATANAME}_prepped',\n",
    "    bigquery_destination_prefix = f\"{PROJECT_ID}\",\n",
    "    generate_explanation=True,\n",
    "    labels = {'notebook':f'{NOTEBOOK}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4022b5bd",
   "metadata": {},
   "source": [
    "---\n",
    "## Remove Resources\n",
    "see notebook \"99 - Cleanup\""
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-11.m104",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-11:m104"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}