NOTEBOOK 1 (Hyperview-main-submission_eagleeyes  -- 0.2596 on Public LB).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## FISRT NOTEBOOK -- TEAM NOVICE"
      ],
      "metadata": {
        "id": "XEbfJDKW2rhy"
      },
      "id": "XEbfJDKW2rhy"
    },
    {
      "cell_type": "markdown",
      "id": "b3014c13",
      "metadata": {
        "id": "b3014c13"
      },
      "source": [
        "## 1. INTRODUCTION\n",
        "\n",
        "The objective of the AI4EO HYPERVIEW challenge is to predict agriculturally relevant soil parameters (K, Mg, P2O5, pH) from airborne hyperspectral images. We present a hybrid model fusing Random Forest and K-nearest neighbor regressors that exploit the average spectral\n",
        "reflectance, as well as derived features such as gradients, wavelet coefficients, and Fourier transforms. The solution is computationally lightweight and improves upon the challenge baseline by 21%."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c70ae309",
      "metadata": {
        "id": "c70ae309"
      },
      "source": [
        "## 2. SETUP\n",
        "\n",
        "In the following cell all modules are imported, that are needed in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2aad8734-02b8-4931-8de7-22a5da9dc009",
      "metadata": {
        "id": "2aad8734-02b8-4931-8de7-22a5da9dc009",
        "outputId": "f6cdcb0d-ddfe-4920-80e7-1d06b1207fac"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: PyWavelets in /home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages (1.2.0)\r\n",
            "Requirement already satisfied: numpy>=1.17.3 in /home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages (from PyWavelets) (1.25.2)\r\n"
          ]
        }
      ],
      "source": [
        "!pip install PyWavelets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c5b21bef",
      "metadata": {
        "id": "c5b21bef"
      },
      "outputs": [],
      "source": [
        "# GENERAL UTILITIES\n",
        "import os\n",
        "from glob import glob\n",
        "import pandas as pd\n",
        "from  tqdm.notebook import tqdm\n",
        "from matplotlib import pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "# MODEL DEVELOPMENT DEPENDENCIES\n",
        "import numpy as np\n",
        "import pywt\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.model_selection import KFold, StratifiedKFold\n",
        "from sklearn.multioutput import MultiOutputRegressor\n",
        "from sklearn.neighbors import KNeighborsRegressor\n",
        "from sklearn import preprocessing\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cc41efea-2a5c-4433-8b25-3b5edda6c5a4",
      "metadata": {
        "id": "cc41efea-2a5c-4433-8b25-3b5edda6c5a4"
      },
      "outputs": [],
      "source": [
        "# SOME CONSTANTS UTILIZED IN THE NOTEBOOK\n",
        "\n",
        "DEBUG = False\n",
        "AUGMENT_CONSTANT_RF=1\n",
        "AUGMENT_CONSTANT_KNN=6\n",
        "LABEL_NAMES = [\"P2O5\", \"K\", \"Mg\", \"pH\"]\n",
        "LABEL_MAXS = np.array([325.0, 625.0, 400.0, 7.8])\n",
        "#Y_BASE_FACT = np.array([121764.2 / 1731.0, 394876.1 / 1731.0, 275875.1 / 1731.0, 11747.67 / 1731.0]) / LABEL_MAXS\n",
        "\n",
        "COL_IX = [0, 1, 2, 3]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1fdc9158",
      "metadata": {
        "id": "1fdc9158"
      },
      "source": [
        "## 3. DATA LOADING\n",
        "\n",
        "In addition to loading the data in a standard way provided by original notebook, we make  pre-augmentation by loading randomly cropped 11x11 size fields, because the main performance loss is coming from this part of the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eff5c3dc",
      "metadata": {
        "id": "eff5c3dc"
      },
      "outputs": [],
      "source": [
        "\n",
        "def load_data(directory: str, gt_file_path: str, is_train=True, augment_constant: int = 0):\n",
        "    \"\"\"Load each cube, reduce its dimensionality and append to array.\n",
        "\n",
        "    Args:\n",
        "        directory (str): Directory to either train or test set\n",
        "        gt_file_path (str): File path for the ground truth labels (expected CVS file)\n",
        "        is_train (boolean): Binary flag for setting loader for Train (TRUE) or Test (FALSE)\n",
        "        augment_constant (int): number of augmentation steps to randomly crop from the larger agricultural fields\n",
        "    Returns:\n",
        "        [type]: Tuple of lists composed of raw field (data , mask) pairs,\n",
        "                and if exists: (augmented data, augmented mask) pairs, and ground truth labels\n",
        "    \"\"\"\n",
        "\n",
        "    datalist = []\n",
        "    masklist = []\n",
        "    aug_datalist = []\n",
        "    aug_masklist = []\n",
        "    aug_labellist = []\n",
        "\n",
        "    if is_train:\n",
        "        labels = load_gt(gt_file_path)\n",
        "\n",
        "    all_files = np.array(\n",
        "        sorted(\n",
        "            glob(os.path.join(directory, \"*.npz\")),\n",
        "            key=lambda x: int(os.path.basename(x).replace(\".npz\", \"\")),\n",
        "        )\n",
        "    )\n",
        "\n",
        "    if DEBUG:\n",
        "        all_files = all_files[:100]\n",
        "        if is_train:\n",
        "            labels = labels[:100]\n",
        "\n",
        "    for idx, file_name in tqdm(enumerate(all_files),total=len(all_files), desc=\"Loading {} data ..\"\n",
        "                               .format(\"training\" if is_train else \"test\")):\n",
        "       # We load the data into memory as provided in the example notebook of the challenge\n",
        "        with np.load(file_name) as npz:\n",
        "            mask = npz[\"mask\"]\n",
        "            data = npz[\"data\"]\n",
        "            datalist.append(data)\n",
        "            masklist.append(mask)\n",
        "\n",
        "    # for training data we make pre-augmentation by adding some randomly cropped samples\n",
        "    if is_train:\n",
        "        for i in range(augment_constant):\n",
        "            for idx, file_name in tqdm(enumerate(all_files),total=len(all_files), desc=\"Loading augmentation {} ..\".format(i+1)):\n",
        "                # print(file_name)\n",
        "                with np.load(file_name) as npz:\n",
        "                    flag = True\n",
        "                    mask = npz[\"mask\"]\n",
        "                    data = npz[\"data\"]\n",
        "                    ma = np.max(data, keepdims=True)\n",
        "                    sh = data.shape[1:]\n",
        "                    for i in range(10):\n",
        "                        # Repeating 11x11 cropping 10 times does not mean we use all croppings:\n",
        "                        # as seen in the Flag=False below at the end of the loop,\n",
        "                        # when we reach at the good crop (not coinciding to the masked area) we stop searching\n",
        "\n",
        "                        # Randomly cropping the fields with 11x11 size,\n",
        "                        # and adding some noise to the cropped samples\n",
        "                        edge = 11\n",
        "                        x = np.random.randint(sh[0] + 1 - edge)\n",
        "                        y = np.random.randint(sh[1] + 1 - edge)\n",
        "\n",
        "                        # get crops having meaningful pixels, not zeros\n",
        "                        if np.sum(mask[0, x : (x + edge), y : (y + edge)]) > 120:\n",
        "                            aug_data = (data[:, x : (x + edge), y : (y + edge)]\n",
        "                                        + np.random.uniform(-0.01, 0.01, (150, edge, edge)) * ma)\n",
        "                            aug_mask = mask[:, x : (x + edge), y : (y + edge)] | np.random.randint(0, 1, (150, edge, edge))\n",
        "\n",
        "                            flag = False #break the loop when you have a meaningful crop\n",
        "                            break\n",
        "\n",
        "                    # After having  11x11 croped sample, get another crop considering\n",
        "                    # the minimum edge length: (min_edge,min_edge)\n",
        "                    if flag:\n",
        "                        max_edge = np.max(sh)\n",
        "                        min_edge = np.min(sh)  # AUGMENT BY SHAPE\n",
        "                        edge = min_edge  # np.random.randint(16, min_edge)\n",
        "                        x = np.random.randint(sh[0] + 1 - edge)\n",
        "                        y = np.random.randint(sh[1] + 1 - edge)\n",
        "                        aug_data = (data[:, x : (x + edge), y : (y + edge)]\n",
        "                                    + np.random.uniform(-0.001, 0.001, (150, edge, edge)) * ma)\n",
        "                        aug_mask = mask[:, x : (x + edge), y : (y + edge)] | np.random.randint(0, 1, (150, edge, edge))\n",
        "\n",
        "                    aug_datalist.append(aug_data)\n",
        "                    aug_masklist.append(aug_mask)\n",
        "                    aug_labellist.append(\n",
        "                        labels[idx, :]\n",
        "                        + labels[idx, :] * np.random.uniform(-0.001, 0.001, 4)\n",
        "                    )\n",
        "\n",
        "    # do pre-augmentation only for training data\n",
        "    if is_train:\n",
        "        return (datalist,\n",
        "                masklist,\n",
        "                labels,\n",
        "                aug_datalist,\n",
        "                aug_masklist,\n",
        "                np.array(aug_labellist))\n",
        "    else:\n",
        "        return datalist, masklist\n",
        "\n",
        "\n",
        "def load_gt(file_path: str):\n",
        "    \"\"\"Load labels for train set from the ground truth file.\n",
        "    Args:\n",
        "        file_path (str): Path to the ground truth .csv file.\n",
        "    Returns:\n",
        "        [type]: 2D numpy array with soil properties levels\n",
        "    \"\"\"\n",
        "    gt_file = pd.read_csv(file_path)\n",
        "    labels = gt_file[[\"P\", \"K\", \"Mg\", \"pH\"]].values / LABEL_MAXS  # normalize ground-truth between 0-1\n",
        "\n",
        "    return labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b9d16282",
      "metadata": {
        "colab": {
          "referenced_widgets": [
            "21ebd8f761694793ba99a40b6e2c35ba",
            "bc3f255217074cf3bb5c44586c5dd3b3",
            "f23102ae6c51489e80edfd6ceabd33ae",
            "9879dd0185194279be7e106fee0a41bb",
            "4c091a6c64ed418a9c98110db1768587",
            "94a879ddb7c94a09b7d63620d682ea4f",
            "730f6292841445b9896b4c64687d3b2d",
            "17156c664cca45ff87e50f58fe6d6b0a"
          ]
        },
        "id": "b9d16282",
        "outputId": "cf543877-e66a-4941-9f38-c45c82d59bb3"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "21ebd8f761694793ba99a40b6e2c35ba",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading training data ..:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bc3f255217074cf3bb5c44586c5dd3b3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading augmentation 1 ..:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f23102ae6c51489e80edfd6ceabd33ae",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading augmentation 2 ..:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9879dd0185194279be7e106fee0a41bb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading augmentation 3 ..:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4c091a6c64ed418a9c98110db1768587",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading augmentation 4 ..:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "94a879ddb7c94a09b7d63620d682ea4f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading augmentation 5 ..:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "730f6292841445b9896b4c64687d3b2d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading augmentation 6 ..:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "17156c664cca45ff87e50f58fe6d6b0a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading test data ..:   0%|          | 0/1154 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train data size: 1732\n",
            "Train aug data size: 10392\n",
            "Test data size: 1154\n"
          ]
        }
      ],
      "source": [
        "# Please be sure that the directory and file locations are given correctly in your own system\n",
        "train_data_dir = \"train_data\"\n",
        "test_data_dir = \"test_data\"\n",
        "gt_data_path = \"train_gt.csv\"\n",
        "\n",
        "# Loading training raw data\n",
        "X_train, M_train, y_train, X_aug_train, M_aug_train, y_aug_train = load_data(train_data_dir,\n",
        "                                                                             gt_data_path,\n",
        "                                                                             is_train=True,\n",
        "                                                                             augment_constant=AUGMENT_CONSTANT_KNN)\n",
        "# Loading test raw data\n",
        "X_test, M_test = load_data(test_data_dir,\n",
        "                           gt_file_path=None,\n",
        "                           is_train=False)\n",
        "\n",
        "print(f\"Train data size: {len(X_train)}\")\n",
        "print(f\"Train aug data size: {len(X_aug_train)}\")\n",
        "print(f\"Test data size: {len(X_test)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fb1ddb38",
      "metadata": {
        "id": "fb1ddb38"
      },
      "source": [
        "## 4. PREPROCESSING THE LOADED DATA\n",
        "\n",
        "We have multiple preprocessing techniques to extract features before training in the function below:\n",
        "\n",
        "* Wavelet Transform\n",
        "* Fast Fourier Transform\n",
        "* Singular Value Decomposition\n",
        "* First, Second and Third Derivative of the Original Data\n",
        "* and some combinations of the methods above\n",
        "\n",
        "(For more information about the feature selection, you can look at the attached pre-print paper for ICIP2022 Conference)\n",
        "    \n",
        "    \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e91ec4d7",
      "metadata": {
        "id": "e91ec4d7"
      },
      "outputs": [],
      "source": [
        "def preprocess(data_list, mask_list, is_for_KNN=False):\n",
        "    \"\"\"Extract high-level features from the raw field data.\n",
        "\n",
        "    Args:\n",
        "        data_list: Directory to either train or test set\n",
        "        mask_list: File path for the ground truth labels (expected CVS file)\n",
        "        is_for_KNN: Binary flag for determining if the features are generated for KNN (TRUE) or Random Forest (FALSE)\n",
        "    Returns:\n",
        "        [type]: Tuple of lists composed of (features , field size) pairs for each field,\n",
        "                where field size will be used performance analysis.\n",
        "    \"\"\"\n",
        "\n",
        "    def _shape_pad(data):\n",
        "        # This sub-function makes padding to have square fields sizes.\n",
        "        # Not mandatory but eliminates the risk of calculation error in singular value decomposition,\n",
        "        # padding by warping also improves the performance slightly.\n",
        "        max_edge = np.max(image.shape[1:])\n",
        "        shape = (max_edge, max_edge)\n",
        "        padded = np.pad(data,((0, 0), (0, (shape[0] - data.shape[1])), (0, (shape[1] - data.shape[2]))),\"wrap\")\n",
        "        return padded\n",
        "\n",
        "    filtering = SpectralCurveFiltering()\n",
        "    w1 = pywt.Wavelet(\"sym3\")\n",
        "    w2 = pywt.Wavelet(\"dmey\")\n",
        "\n",
        "    processed_data = []\n",
        "    average_edge = []\n",
        "\n",
        "    for idx, (data, mask) in enumerate(\n",
        "        tqdm(\n",
        "            zip(data_list, mask_list),\n",
        "            total=len(data_list),\n",
        "            position=0,\n",
        "            leave=True,\n",
        "            desc=\"INFO: Preprocessing data ...\",\n",
        "        )\n",
        "    ):\n",
        "        data = data / 2210   # max-max=5419 mean-max=2210\n",
        "        m = 1 - mask.astype(int)\n",
        "        image = data * m\n",
        "\n",
        "        average_edge.append((image.shape[1] + image.shape[2]) / 2)\n",
        "        image = _shape_pad(image)\n",
        "\n",
        "        s = np.linalg.svd(image, full_matrices=False, compute_uv=False)\n",
        "        s0 = s[:, 0]\n",
        "        s1 = s[:, 1]\n",
        "        s2 = s[:, 2]\n",
        "        s3 = s[:, 3]\n",
        "        s4 = s[:, 4]\n",
        "        dXds1 = s0 / (s1 + np.finfo(float).eps)\n",
        "\n",
        "\n",
        "        data = np.ma.MaskedArray(data, mask)\n",
        "        arr = filtering(data)\n",
        "\n",
        "        cA0, cD0 = pywt.dwt(arr, wavelet=w2, mode=\"constant\")\n",
        "        cAx, cDx = pywt.dwt(cA0[12:92], wavelet=w2, mode=\"constant\")\n",
        "        cAy, cDy = pywt.dwt(cAx[15:55], wavelet=w2, mode=\"constant\")\n",
        "        cAz, cDz = pywt.dwt(cAy[15:35], wavelet=w2, mode=\"constant\")\n",
        "        cAw2 = np.concatenate((cA0[12:92], cAx[15:55], cAy[15:35], cAz[15:25]), -1)\n",
        "        cDw2 = np.concatenate((cD0[12:92], cDx[15:55], cDy[15:35], cDz[15:25]), -1)\n",
        "\n",
        "        cA0, cD0 = pywt.dwt(arr, wavelet=w1, mode=\"constant\")\n",
        "        cAx, cDx = pywt.dwt(cA0[1:-1], wavelet=w1, mode=\"constant\")\n",
        "        cAy, cDy = pywt.dwt(cAx[1:-1], wavelet=w1, mode=\"constant\")\n",
        "        cAz, cDz = pywt.dwt(cAy[1:-1], wavelet=w1, mode=\"constant\")\n",
        "        cAw1 = np.concatenate((cA0, cAx, cAy, cAz), -1)\n",
        "        cDw1 = np.concatenate((cD0, cDx, cDy, cDz), -1)\n",
        "\n",
        "        dXdl = np.gradient(arr, axis=0)\n",
        "        d2Xdl2 = np.gradient(dXdl, axis=0)\n",
        "        d3Xdl3 = np.gradient(d2Xdl2, axis=0)\n",
        "\n",
        "\n",
        "        fft = np.fft.fft(arr)\n",
        "        real = np.real(fft)\n",
        "        imag = np.imag(fft)\n",
        "        ffts = np.fft.fft(s0)\n",
        "        reals = np.real(ffts)\n",
        "        imags = np.imag(ffts)\n",
        "\n",
        "        # The best Feature combination for Random Forest based regression\n",
        "        out_rf = np.concatenate(\n",
        "            [\n",
        "                arr,\n",
        "                dXdl,\n",
        "                d2Xdl2,\n",
        "                d3Xdl3,\n",
        "                dXds1,\n",
        "                s0,\n",
        "                s1,\n",
        "                s2,\n",
        "                s3,\n",
        "                s4,\n",
        "                real,\n",
        "                imag,\n",
        "                reals,\n",
        "                imags,\n",
        "                cAw1,\n",
        "                cAw2,\n",
        "            ],\n",
        "            -1,\n",
        "        )\n",
        "\n",
        "        # The best Feature combination for KNN based regression\n",
        "        out_knn = np.concatenate(\n",
        "            [\n",
        "                arr,\n",
        "                dXdl,\n",
        "                d2Xdl2,\n",
        "                d3Xdl3,\n",
        "                s0,\n",
        "                s1,\n",
        "                s2,\n",
        "                s3,\n",
        "                s4,\n",
        "                real,\n",
        "                imag,\n",
        "\n",
        "            ],\n",
        "            -1,\n",
        "        )\n",
        "\n",
        "\n",
        "        if is_for_KNN:\n",
        "            processed_data.append(out_knn)\n",
        "        else:\n",
        "            processed_data.append(out_rf)\n",
        "\n",
        "    return np.array(processed_data), np.array(average_edge)\n",
        "\n",
        "\n",
        "\n",
        "class SpectralCurveFiltering: # Default class provided by the challenge organizers\n",
        "    \"\"\"\n",
        "    Create a histogram (a spectral curve) of a 3D cube, using the merge_function\n",
        "    to aggregate all pixels within one band. The return array will have\n",
        "    the shape of [CHANNELS_COUNT]\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, merge_function=np.mean):\n",
        "        self.merge_function = merge_function\n",
        "\n",
        "    def __call__(self, sample: np.ndarray):\n",
        "        return self.merge_function(sample, axis=(1, 2))\n",
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7fc492d7-a0b5-4090-92b4-013eb55f7676",
      "metadata": {
        "id": "7fc492d7-a0b5-4090-92b4-013eb55f7676"
      },
      "source": [
        "We are extractig features for training and test by pre-processing the data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5407d75e-cd48-401f-a9a4-dc8eba3215ae",
      "metadata": {
        "colab": {
          "referenced_widgets": [
            "0a5ba33bcf4f4056a02e303843680099",
            "d023a44079ba48de9db50d195a0994e7",
            "42dad07b6cd140b9a76f0c8df1576396",
            "5a2c24b1a0574d91be39bb0566138fd8",
            "acce7fe48dea49f1bea5ef193482a835",
            "52000433fc484770aaaa85ffcdcad303"
          ]
        },
        "id": "5407d75e-cd48-401f-a9a4-dc8eba3215ae",
        "outputId": "16f31982-85c9-4b03-fba2-b1e99641c146"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0a5ba33bcf4f4056a02e303843680099",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "INFO: Preprocessing data ...:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d023a44079ba48de9db50d195a0994e7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "INFO: Preprocessing data ...:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "42dad07b6cd140b9a76f0c8df1576396",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "INFO: Preprocessing data ...:   0%|          | 0/1154 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5a2c24b1a0574d91be39bb0566138fd8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "INFO: Preprocessing data ...:   0%|          | 0/1732 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "acce7fe48dea49f1bea5ef193482a835",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "INFO: Preprocessing data ...:   0%|          | 0/10392 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "52000433fc484770aaaa85ffcdcad303",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "INFO: Preprocessing data ...:   0%|          | 0/1154 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "\n",
        "\n",
        "# preprocessed data for random forest traninig and testing\n",
        "X_tr_processed_RF, avg_edge_train = preprocess(X_train, M_train, is_for_KNN=False)\n",
        "X_aug_processed_RF, avg_edge_train_aug_RF = preprocess(X_aug_train[:len(X_train)*AUGMENT_CONSTANT_RF], M_aug_train[:len(X_train)*AUGMENT_CONSTANT_RF], is_for_KNN=False)\n",
        "X_te_processed_RF, avg_edge_test = preprocess(X_test, M_test, is_for_KNN=False)\n",
        "\n",
        "# preprocessed data for KNN traninig and testing\n",
        "X_tr_processed_KNN, avg_edge_train = preprocess(X_train, M_train, is_for_KNN=True)\n",
        "X_aug_processed_KNN, avg_edge_train_aug_KNN = preprocess(X_aug_train, M_aug_train,is_for_KNN=True)\n",
        "X_te_processed_KNN, avg_edge_test = preprocess(X_test, M_test, is_for_KNN=True)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4aabb7ee",
      "metadata": {
        "id": "4aabb7ee"
      },
      "source": [
        "## 5. TRAINING THE MODEL\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "96c4735d-c366-41a7-899d-0a704fb44dae",
      "metadata": {
        "id": "96c4735d-c366-41a7-899d-0a704fb44dae"
      },
      "source": [
        "### 5.A. BASELINE REGRESSOR\n",
        "\n",
        "In the following cell the baseline regressor, which was given by the challenge is defined. This baseline regressor always predicts the mean for the soil parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9dba4731-6055-40cd-87ff-38785fe4e490",
      "metadata": {
        "id": "9dba4731-6055-40cd-87ff-38785fe4e490"
      },
      "outputs": [],
      "source": [
        "class BaselineRegressor:\n",
        "    \"\"\"\n",
        "    Baseline regressor, which calculates the mean value of the target from the training\n",
        "    data and returns it for each testing sample.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        self.mean = 0\n",
        "\n",
        "    def fit(self, X_train: np.ndarray, y_train: np.ndarray):\n",
        "        self.mean = np.mean(y_train, axis=0)\n",
        "        self.classes_count = y_train.shape[1]\n",
        "        return self\n",
        "\n",
        "    def predict(self, X_test: np.ndarray):\n",
        "        return np.full((len(X_test), self.classes_count), self.mean)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "967831ab-9851-4c83-85f7-8fde9ff7c13a",
      "metadata": {
        "id": "967831ab-9851-4c83-85f7-8fde9ff7c13a"
      },
      "source": [
        "### 5.B. TRAINING FOR LARGE FIELDS\n",
        "\n",
        "\n",
        "We are conducting seperate training for small field and large field samples because 11x11 pixel size fields performed poorly in a single training.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "953352b6-d0cc-4986-869e-11984f38d795",
      "metadata": {
        "id": "953352b6-d0cc-4986-869e-11984f38d795"
      },
      "source": [
        "#### 5.B.i. CROSS-VALIDATION WITH RANDOM FOREST REGRESSOR\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "38f3f43e",
      "metadata": {
        "id": "38f3f43e"
      },
      "outputs": [],
      "source": [
        "# Select set of labels\n",
        "\n",
        "y_train_col = y_train[:, COL_IX]\n",
        "y_aug_train_col = y_aug_train[:len(y_train_col)*AUGMENT_CONSTANT_RF, COL_IX]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1b494d33",
      "metadata": {
        "id": "1b494d33",
        "outputId": "2f535ae8-0c7c-4132-b7e2-bc65a41ba7f9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "5"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# 5-fold cross validation for training.\n",
        "\n",
        "kfold = KFold(shuffle=True, random_state=2022)\n",
        "kfold.get_n_splits(X_aug_train, y_aug_train_col)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "becb462f-5487-4cae-90ca-6105335ecc7d",
      "metadata": {
        "id": "becb462f-5487-4cae-90ca-6105335ecc7d"
      },
      "source": [
        "We start training with 5-fold cross validation, and preserve each trained model in the list. Later, we will use the list of the models for performance reporting or submission file generation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b974a4e5",
      "metadata": {
        "id": "b974a4e5",
        "outputId": "5ed5bf4c-9fad-4820-bf04-b0a091df0bf6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CROSS VALIDATION STEP: 0\n",
            "Prediction score: 0.12483527979031794\n",
            "CROSS VALIDATION STEP: 1\n",
            "Prediction score: 0.16989586170248464\n",
            "CROSS VALIDATION STEP: 2\n",
            "Prediction score: 0.15252493004994613\n",
            "CROSS VALIDATION STEP: 3\n",
            "Prediction score: 0.15429431279733952\n",
            "CROSS VALIDATION STEP: 4\n",
            "Prediction score: 0.1497631232363564\n"
          ]
        }
      ],
      "source": [
        "\n",
        "random_forest_models = []\n",
        "baseline_regressors = []\n",
        "\n",
        "y_hat_bl = []\n",
        "y_hat_rf = []\n",
        "y_v_list_rf = []\n",
        "edge_v_list_rf = []\n",
        "\n",
        "for idx, (ix_train, ix_valid) in enumerate(kfold.split(np.arange(0, len(y_train)), avg_edge_train.astype(int))):\n",
        "    print(\"CROSS VALIDATION STEP: {}\".format(idx))\n",
        "\n",
        "    # Merge original data with the augmented data on training set\n",
        "    X_t = np.concatenate((X_tr_processed_RF[ix_train], X_aug_processed_RF[ix_train]), axis=0)\n",
        "    y_t = np.concatenate((y_train_col[ix_train], y_aug_train_col[ix_train]), axis=0)\n",
        "\n",
        "    # Filter out Validation set\n",
        "    X_v = X_tr_processed_RF[ix_valid]\n",
        "    y_v =y_train_col[ix_valid]\n",
        "    y_v_list_rf.append(y_v)\n",
        "\n",
        "    #Field edge sizes will be used for performance analysis later\n",
        "    edge = avg_edge_train[ix_valid]\n",
        "    edge_v_list_rf.append(edge)\n",
        "\n",
        "    # baseline fiting\n",
        "    baseline = BaselineRegressor()\n",
        "    baseline.fit(X_t, y_t)\n",
        "    baseline_regressors.append(baseline)\n",
        "\n",
        "    # baseline predictions\n",
        "    y_b = baseline.predict(X_v)\n",
        "    y_hat_bl.append(y_b)\n",
        "\n",
        "    # random forest fitting\n",
        "    model = RandomForestRegressor(n_estimators=1000, n_jobs=-1)\n",
        "    model.fit(X_t, y_t)\n",
        "    random_forest_models.append(model)\n",
        "\n",
        "    # random forest predictions\n",
        "    y_hat = model.predict(X_v)\n",
        "    y_hat_rf.append(y_hat)\n",
        "\n",
        "    print('Prediction score: {}'.format(model.score(X_v, y_v)))\n",
        "\n",
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b234a272",
      "metadata": {
        "id": "b234a272"
      },
      "source": [
        "#### 5.B.ii. PERFORMANCE EVALUATION ON VALIDATION SET (FOR LARGE FIELDS)\n",
        "For the five trained models, the evaluation score is calculated on the validation set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3d941b92",
      "metadata": {
        "id": "3d941b92",
        "outputId": "936b3ae8-d918-4f1e-9435-98cc93d4ba26"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      1078.98\n",
            "Random Forest MSE: 1001.94 (-7.14 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      4633.81\n",
            "Random Forest MSE: 3715.33 (-19.82 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      1684.76\n",
            "Random Forest MSE: 1551.90 (-7.89 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.07\n",
            "Random Forest MSE: 0.06 (-15.65 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.8737570831202157\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      894.49\n",
            "Random Forest MSE: 789.82 (-11.70 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      3913.83\n",
            "Random Forest MSE: 3149.32 (-19.53 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      1653.14\n",
            "Random Forest MSE: 1393.37 (-15.71 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.06\n",
            "Random Forest MSE: 0.05 (-21.37 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.8292091266275969\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      861.49\n",
            "Random Forest MSE: 811.47 (-5.81 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      3673.56\n",
            "Random Forest MSE: 2982.23 (-18.82 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      1393.53\n",
            "Random Forest MSE: 1185.97 (-14.89 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.06\n",
            "Random Forest MSE: 0.05 (-22.85 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.8440768173155077\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      780.82\n",
            "Random Forest MSE: 682.62 (-12.58 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      3985.97\n",
            "Random Forest MSE: 3222.79 (-19.15 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      1579.19\n",
            "Random Forest MSE: 1342.59 (-14.98 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.07\n",
            "Random Forest MSE: 0.06 (-15.78 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.8437792574011327\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      734.85\n",
            "Random Forest MSE: 692.69 (-5.74 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      2962.62\n",
            "Random Forest MSE: 2302.35 (-22.29 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      1638.43\n",
            "Random Forest MSE: 1440.34 (-12.09 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.08\n",
            "Random Forest MSE: 0.06 (-20.81 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.8476825009419824\n",
            "\n",
            "\n",
            "OVERALL CV EVALUATION SCORE: 0.8477009570812871\n"
          ]
        }
      ],
      "source": [
        "scores = 0\n",
        "for y_hat, y_b, y_v, y_e in zip(y_hat_rf, y_hat_bl, y_v_list_rf, edge_v_list_rf):\n",
        "    score = 0\n",
        "    for i in COL_IX:\n",
        "        print('\\n')\n",
        "        print(\"Result for parameter: \", LABEL_NAMES[i])\n",
        "        mse_rf = mean_squared_error(y_v[:, i] * LABEL_MAXS[i], y_hat[:, i] * LABEL_MAXS[i])\n",
        "        mse_bl = mean_squared_error(y_v[:, i] * LABEL_MAXS[i], y_b[:, i] * LABEL_MAXS[i])\n",
        "        score += mse_rf / mse_bl\n",
        "        print(f\"Baseline MSE:      {mse_bl:.2f}\")\n",
        "        print(f\"Random Forest MSE: {mse_rf:.2f} ({1e2*(mse_rf - mse_bl)/mse_bl:+.2f} %)\")\n",
        "\n",
        "\n",
        "    print('\\n')\n",
        "    print(\"CV Evaluation score:\", score / 4)\n",
        "\n",
        "    scores += score\n",
        "\n",
        "print('\\n')\n",
        "print(\"OVERALL CV EVALUATION SCORE:\", scores / 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "539b5fb0",
      "metadata": {
        "id": "539b5fb0"
      },
      "source": [
        "Calculating the validation score for different field sizes, including patches of size 11x11."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c2b3d3fd",
      "metadata": {
        "id": "c2b3d3fd"
      },
      "outputs": [],
      "source": [
        "scores = 0\n",
        "out_table = []\n",
        "\n",
        "\n",
        "for y_h, y_b, y_v, y_e in zip(y_hat_rf, y_hat_bl, y_v_list_rf, edge_v_list_rf):\n",
        "    mse_rf = np.square(\n",
        "        np.subtract(y_h * LABEL_MAXS, y_v * LABEL_MAXS)\n",
        "    )\n",
        "    mse_bl = np.square(\n",
        "        np.subtract(y_v * LABEL_MAXS, y_b * LABEL_MAXS)\n",
        "    )\n",
        "    row = np.zeros((len(y_h), 9))\n",
        "    row[:, 8] = y_e\n",
        "    row[:, 0:4] = mse_rf\n",
        "    row[:, 4:8] = mse_bl\n",
        "    out_table.append(row)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b6dc28d1",
      "metadata": {
        "id": "b6dc28d1"
      },
      "outputs": [],
      "source": [
        "x = np.concatenate(out_table, 0)\n",
        "df = pd.DataFrame(\n",
        "    x,\n",
        "    columns=[\"P2O5\", \"K\", \"Mg\", \"pH\", \"P2O5_avg\", \"K_avg\", \"Mg_avg\", \"pH_avg\", \"Edge\"],\n",
        ")\n",
        "df.head(10)\n",
        "_, bin_edge = np.histogram(df.Edge.values, bins=4)\n",
        "\n",
        "# We have determined those bin edges after looking at the field size distribuiton\n",
        "bin_edge = [0, 11, 40, 50, 100, 110, 120, 130, 210]\n",
        "bin_edge_labels = [\n",
        "    \"0-11\",\n",
        "    \"11-40\",\n",
        "    \"40-50\",\n",
        "    \"50-100\",\n",
        "    \"100-110\",\n",
        "    \"110-120\",\n",
        "    \"120-130\",\n",
        "    \"130+\",\n",
        "]\n",
        "mse_per_edge = np.zeros((len(bin_edge) - 1, 6), dtype=object)\n",
        "\n",
        "for i in range(1, len(bin_edge)):\n",
        "    d_temp = df[(df.Edge <= bin_edge[i]) & (df.Edge > bin_edge[i - 1])]\n",
        "    mse_per_edge[i - 1, 0] = np.mean(d_temp.P2O5.values) / np.mean(\n",
        "        d_temp.P2O5_avg.values\n",
        "    )\n",
        "    mse_per_edge[i - 1, 1] = np.mean(d_temp.K.values) / np.mean(d_temp.K_avg.values)\n",
        "    mse_per_edge[i - 1, 2] = np.mean(d_temp.Mg.values) / np.mean(d_temp.Mg_avg.values)\n",
        "    mse_per_edge[i - 1, 3] = np.mean(d_temp.pH.values) / np.mean(d_temp.pH_avg.values)\n",
        "    mse_per_edge[i - 1, 5] = len(d_temp)\n",
        "    mse_per_edge[i - 1, 4] = bin_edge_labels[i - 1]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e5596d07",
      "metadata": {
        "id": "e5596d07"
      },
      "source": [
        "In the following cell, the mean validation score is demonstrated for each soil parameter and field size:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "51457529",
      "metadata": {
        "id": "51457529",
        "outputId": "84cfe853-7749-49c6-f3be-82a1f57343ed"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>P2O5</th>\n",
              "      <th>K</th>\n",
              "      <th>Mg</th>\n",
              "      <th>pH</th>\n",
              "      <th>Edge</th>\n",
              "      <th>Len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.057946</td>\n",
              "      <td>1.015635</td>\n",
              "      <td>1.01712</td>\n",
              "      <td>0.861278</td>\n",
              "      <td>0-11</td>\n",
              "      <td>650</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.487523</td>\n",
              "      <td>0.579514</td>\n",
              "      <td>0.531257</td>\n",
              "      <td>0.977514</td>\n",
              "      <td>11-40</td>\n",
              "      <td>94</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.722207</td>\n",
              "      <td>0.752374</td>\n",
              "      <td>0.415241</td>\n",
              "      <td>0.787324</td>\n",
              "      <td>40-50</td>\n",
              "      <td>326</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.687112</td>\n",
              "      <td>0.655311</td>\n",
              "      <td>0.624791</td>\n",
              "      <td>0.751544</td>\n",
              "      <td>50-100</td>\n",
              "      <td>138</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.917833</td>\n",
              "      <td>0.591326</td>\n",
              "      <td>0.388293</td>\n",
              "      <td>0.754186</td>\n",
              "      <td>100-110</td>\n",
              "      <td>113</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.887196</td>\n",
              "      <td>0.828525</td>\n",
              "      <td>0.60946</td>\n",
              "      <td>0.751809</td>\n",
              "      <td>110-120</td>\n",
              "      <td>118</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0.887643</td>\n",
              "      <td>0.763285</td>\n",
              "      <td>0.6605</td>\n",
              "      <td>0.661214</td>\n",
              "      <td>120-130</td>\n",
              "      <td>132</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0.802656</td>\n",
              "      <td>0.7658</td>\n",
              "      <td>0.844017</td>\n",
              "      <td>0.786326</td>\n",
              "      <td>130+</td>\n",
              "      <td>161</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       P2O5         K        Mg        pH     Edge  Len\n",
              "0  1.057946  1.015635   1.01712  0.861278     0-11  650\n",
              "1  0.487523  0.579514  0.531257  0.977514    11-40   94\n",
              "2  0.722207  0.752374  0.415241  0.787324    40-50  326\n",
              "3  0.687112  0.655311  0.624791  0.751544   50-100  138\n",
              "4  0.917833  0.591326  0.388293  0.754186  100-110  113\n",
              "5  0.887196  0.828525   0.60946  0.751809  110-120  118\n",
              "6  0.887643  0.763285    0.6605  0.661214  120-130  132\n",
              "7  0.802656    0.7658  0.844017  0.786326     130+  161"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "d_out = pd.DataFrame(mse_per_edge, columns=[\"P2O5\", \"K\", \"Mg\", \"pH\", \"Edge\", \"Len\"])\n",
        "d_out"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4ecf38df",
      "metadata": {
        "id": "4ecf38df"
      },
      "source": [
        "#### 5.B.iii. MAKE PREDICTIONS AND SUBMISSIONS ( FOR THE LARGE FIELDS > 11x11 PIXELS)\n",
        "Re-train the random forest model with the whole training and augmentation data without splitting them for cross-validation.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c19bdb71-5c7f-401f-a7bc-533c155f9fce",
      "metadata": {
        "id": "c19bdb71-5c7f-401f-a7bc-533c155f9fce"
      },
      "outputs": [],
      "source": [
        "\n",
        "random_forests = []\n",
        "model = RandomForestRegressor(n_estimators=1000, n_jobs=-1)\n",
        "\n",
        "X_t = np.concatenate((X_tr_processed_RF, X_aug_processed_RF), axis=0)\n",
        "y_t = np.concatenate((y_train_col, y_aug_train_col), axis=0)\n",
        "\n",
        "model.fit(X_t, y_t)\n",
        "\n",
        "random_forests.append(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a846f505-6d9e-4c08-b44a-1d2351b25e84",
      "metadata": {
        "id": "a846f505-6d9e-4c08-b44a-1d2351b25e84"
      },
      "source": [
        "In the test set, the patches with size 11x11 pixekls (small fields) are at the very beginning, we can therefore exclude them, by simply starting from hard-coded field index `432` or by dynamically filtering out data with smaller field sizes: (```X_te_processed_RF[avg_edge_test>11, :]```). Thus, we make predictions only for large fields with Random Forest Regressor:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3f82ccd9",
      "metadata": {
        "id": "3f82ccd9",
        "outputId": "d8901479-66d3-402a-852c-0a689ee8c0e8"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>P</th>\n",
              "      <th>K</th>\n",
              "      <th>Mg</th>\n",
              "      <th>pH</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>62.073528</td>\n",
              "      <td>269.879470</td>\n",
              "      <td>167.420540</td>\n",
              "      <td>6.828817</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>58.279397</td>\n",
              "      <td>280.515694</td>\n",
              "      <td>175.791144</td>\n",
              "      <td>6.752192</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>77.890719</td>\n",
              "      <td>279.263355</td>\n",
              "      <td>157.539417</td>\n",
              "      <td>6.883160</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>72.298301</td>\n",
              "      <td>271.406046</td>\n",
              "      <td>156.339853</td>\n",
              "      <td>6.838531</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>49.589958</td>\n",
              "      <td>252.538588</td>\n",
              "      <td>188.040150</td>\n",
              "      <td>6.755080</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>717</th>\n",
              "      <td>46.936877</td>\n",
              "      <td>169.610192</td>\n",
              "      <td>135.352855</td>\n",
              "      <td>6.628018</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>718</th>\n",
              "      <td>45.545826</td>\n",
              "      <td>169.880842</td>\n",
              "      <td>135.613945</td>\n",
              "      <td>6.610217</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>719</th>\n",
              "      <td>76.024305</td>\n",
              "      <td>239.529187</td>\n",
              "      <td>167.534027</td>\n",
              "      <td>6.643992</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>720</th>\n",
              "      <td>47.691755</td>\n",
              "      <td>182.878088</td>\n",
              "      <td>138.305905</td>\n",
              "      <td>6.612771</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>721</th>\n",
              "      <td>62.198148</td>\n",
              "      <td>216.640894</td>\n",
              "      <td>151.633768</td>\n",
              "      <td>6.632037</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>722 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "             P           K          Mg        pH\n",
              "0    62.073528  269.879470  167.420540  6.828817\n",
              "1    58.279397  280.515694  175.791144  6.752192\n",
              "2    77.890719  279.263355  157.539417  6.883160\n",
              "3    72.298301  271.406046  156.339853  6.838531\n",
              "4    49.589958  252.538588  188.040150  6.755080\n",
              "..         ...         ...         ...       ...\n",
              "717  46.936877  169.610192  135.352855  6.628018\n",
              "718  45.545826  169.880842  135.613945  6.610217\n",
              "719  76.024305  239.529187  167.534027  6.643992\n",
              "720  47.691755  182.878088  138.305905  6.612771\n",
              "721  62.198148  216.640894  151.633768  6.632037\n",
              "\n",
              "[722 rows x 4 columns]"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "predictions_large = []\n",
        "\n",
        "#make predictions on multiple models if there are more than 1 model and later mean the results over them\n",
        "for rf in random_forests:\n",
        "    pp = rf.predict(X_te_processed_RF[avg_edge_test>11, :]) #X_te_processed_RF[432:, :]\n",
        "    predictions_large.append(pp)\n",
        "\n",
        "predictions_large = np.asarray(predictions_large)\n",
        "predictions_large = np.mean(predictions_large, axis=0)\n",
        "predictions_large = predictions_large * LABEL_MAXS # predictions were in the range of [0,1] so that we neeed to scale them back.\n",
        "\n",
        "submission_df = pd.DataFrame(data=predictions_large, columns=[\"P\", \"K\", \"Mg\", \"pH\"])\n",
        "submission_df.to_csv(\"submission_large.csv\", index_label=\"sample_index\")\n",
        "submission_df"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5d0640c8",
      "metadata": {
        "id": "5d0640c8"
      },
      "source": [
        "### 5.C. TRAINING FOR SMALL FIELDS\n",
        "\n",
        "We are conducting separate training for small field and large field samples because 11x11 fields performed poorly in a single training. For the small patches we chose a KNN as model, because it showed better performance on this data. We again use a 5-fold cross validation."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9a223ff9-562e-439e-85cb-9d697ca8e3c1",
      "metadata": {
        "id": "9a223ff9-562e-439e-85cb-9d697ca8e3c1"
      },
      "source": [
        "#### 5.C.i. MERGE ORIGINAL AND AUGMENTED DATA\n",
        "\n",
        "In the following cell, we are merging original data with small field size (first 650 samples) and augmented data with randomly cropped fields from the large ones (sample indices larger than 650).\n",
        "\n",
        "TODO: file index is hard-coded since the field sizes were already checked.\n",
        "      For a real life scenario, a dynamic ranking of file indices by field size would be better.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cda2bbe1",
      "metadata": {
        "id": "cda2bbe1"
      },
      "outputs": [],
      "source": [
        "\n",
        "X_tr_processed_normalized_small = np.array(X_tr_processed_KNN[0:650, :], copy=True) #avg_edge_train<=11\n",
        "X_aug_processed_normalized_large = np.array(X_aug_processed_KNN[650:1732, :], copy=True) #avg_edge_train>11\n",
        "X_te_processed_normalized = np.array(X_te_processed_KNN, copy=True)\n",
        "\n",
        "for i in range(1, AUGMENT_CONSTANT_KNN):\n",
        "    X_aug_processed_normalized_large = np.concatenate(\n",
        "        [X_aug_processed_normalized_large,\n",
        "         X_aug_processed_KNN[650 + (i * 1732) : i * 1732 + 1732, :]],0,)\n",
        "\n",
        "y_train_small = y_train[0:650, :]\n",
        "avg_edge_train_small= avg_edge_train[0:650]\n",
        "y_aug_train_large = y_aug_train[650:1732, :]\n",
        "\n",
        "for i in range(1, AUGMENT_CONSTANT_KNN):\n",
        "    y_aug_train_large = np.concatenate(\n",
        "        [y_aug_train_large, y_aug_train[650 + (i * 1732) : (i * 1732) + 1732, :]], 0)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fd51dc53-f1e0-4aab-975d-b5f190acc5ed",
      "metadata": {
        "id": "fd51dc53-f1e0-4aab-975d-b5f190acc5ed"
      },
      "source": [
        "#### 5.C.ii. NORMALIZE DATA\n",
        "Data normalization improved the performance on KNN but not on Random Forest. That is why we conducted Robust Scaler here only for features to be used in KNN training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5c54f791",
      "metadata": {
        "id": "5c54f791"
      },
      "outputs": [],
      "source": [
        "# Feature normalization\n",
        "for i in range(int(X_tr_processed_normalized_small.shape[-1] / 150)):\n",
        "    scaler = preprocessing.RobustScaler()\n",
        "    scaler.fit(X_tr_processed_normalized_small[:, 150 * i : 150 * i + 150])\n",
        "\n",
        "    X_tr_processed_normalized_small[:, 150 * i : 150 * i + 150] = scaler.transform(\n",
        "        X_tr_processed_normalized_small[:, 150 * i : 150 * i + 150])\n",
        "\n",
        "    X_aug_processed_normalized_large[:, 150 * i : 150 * i + 150] = scaler.transform(\n",
        "        X_aug_processed_normalized_large[:, 150 * i : 150 * i + 150])\n",
        "\n",
        "    X_te_processed_normalized[:, 150 * i : 150 * i + 150] = scaler.transform(\n",
        "        X_te_processed_normalized[:, 150 * i : 150 * i + 150])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d694501a-ff8a-4377-af2f-d13ccd83392a",
      "metadata": {
        "id": "d694501a-ff8a-4377-af2f-d13ccd83392a"
      },
      "source": [
        "#### 5.C.iii. CROSS-VALIDATION WITH KNN REGRESSOR\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b4be3f00-6dac-4d80-bc91-44c8f47958e8",
      "metadata": {
        "id": "b4be3f00-6dac-4d80-bc91-44c8f47958e8"
      },
      "outputs": [],
      "source": [
        "# Select set of labels\n",
        "\n",
        "y_train_small_col = y_train_small[:, COL_IX]\n",
        "y_aug_train_large_col = y_aug_train_large[:, COL_IX]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bb935166",
      "metadata": {
        "id": "bb935166",
        "outputId": "e7997fb4-2b8d-47eb-cbb0-2b4473d5ce21"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "5"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# 5-fold cross validation for training.\n",
        "\n",
        "kfold = KFold(shuffle=True)\n",
        "kfold.get_n_splits(X_tr_processed_normalized_small, y_train_small)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "89251016",
      "metadata": {
        "id": "89251016",
        "outputId": "c9804f1f-f40b-40ae-e38c-7d250934cda9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CROSS VALIDATION STEP: \n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27c10>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27c10>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Prediction score: -0.033956877906187344\n",
            "CROSS VALIDATION STEP: \n",
            "Prediction score: -0.013176772046856355\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27c10>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27b80>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CROSS VALIDATION STEP: \n",
            "Prediction score: -0.024217940433127583\n",
            "CROSS VALIDATION STEP: \n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27940>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27940>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca279d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27ca0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca009d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca279d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca279d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00550>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27dc0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27d30>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27af0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Prediction score: -0.00795966083040045\n",
            "CROSS VALIDATION STEP: \n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00ca0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca00ca0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Prediction score: -0.009362415934441315\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      1398.82\n",
            "Random Forest MSE: 1454.08 (+3.95 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      2001.36\n",
            "Random Forest MSE: 1833.27 (-8.40 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      2694.59\n",
            "Random Forest MSE: 2646.68 (-1.78 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.08\n",
            "Random Forest MSE: 0.06 (-23.29 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.926209345948315\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      774.25\n",
            "Random Forest MSE: 785.49 (+1.45 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      2773.96\n",
            "Random Forest MSE: 2667.31 (-3.84 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      2767.05\n",
            "Random Forest MSE: 2732.16 (-1.26 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.08\n",
            "Random Forest MSE: 0.05 (-37.96 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.8959745664719774\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      1506.73\n",
            "Random Forest MSE: 1526.31 (+1.30 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      2282.00\n",
            "Random Forest MSE: 2163.22 (-5.20 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      3672.90\n",
            "Random Forest MSE: 3623.12 (-1.36 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.11\n",
            "Random Forest MSE: 0.08 (-27.82 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.917289017706194\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      1309.22\n",
            "Random Forest MSE: 1314.04 (+0.37 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      3424.35\n",
            "Random Forest MSE: 3316.75 (-3.14 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      2535.70\n",
            "Random Forest MSE: 2545.62 (+0.39 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.09\n",
            "Random Forest MSE: 0.06 (-32.49 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.9128203306643611\n",
            "\n",
            "\n",
            "Result for parameter:  P2O5\n",
            "Baseline MSE:      1282.56\n",
            "Random Forest MSE: 1266.22 (-1.27 %)\n",
            "\n",
            "\n",
            "Result for parameter:  K\n",
            "Baseline MSE:      4055.21\n",
            "Random Forest MSE: 3939.45 (-2.85 %)\n",
            "\n",
            "\n",
            "Result for parameter:  Mg\n",
            "Baseline MSE:      2670.96\n",
            "Random Forest MSE: 2739.69 (+2.57 %)\n",
            "\n",
            "\n",
            "Result for parameter:  pH\n",
            "Baseline MSE:      0.10\n",
            "Random Forest MSE: 0.07 (-24.04 %)\n",
            "\n",
            "\n",
            "CV Evaluation score: 0.9360121457945583\n",
            "\n",
            "\n",
            "OVERALL CV EVALUATION SCORE: 0.9176610813170812\n"
          ]
        }
      ],
      "source": [
        "#Train the model for small fields only\n",
        "\n",
        "\n",
        "KNN_models = []\n",
        "baseline_regressors = []\n",
        "\n",
        "y_hat_bl = []\n",
        "y_hat_knn = []\n",
        "y_v_list_knn = []\n",
        "edge_v_list_knn = []\n",
        "\n",
        "for idx, (ix_train, ix_valid) in enumerate(\n",
        "    kfold.split(np.arange(0, len(y_train_small)))\n",
        "):\n",
        "    print(\"CROSS VALIDATION STEP: \".format(str(idx)))\n",
        "\n",
        "    X_t = X_tr_processed_normalized_small[ix_train]\n",
        "    y_t = y_train_small_col[ix_train]\n",
        "\n",
        "    X_t = np.concatenate((X_tr_processed_normalized_small[ix_train], X_aug_processed_normalized_large), axis=0)\n",
        "    y_t = np.concatenate((y_train_small_col[ix_train], y_aug_train_large_col), axis=0)\n",
        "\n",
        "    X_v = X_tr_processed_normalized_small[ix_valid]\n",
        "    y_v = y_train_small_col[ix_valid]\n",
        "    y_v_list_knn.append(y_v)\n",
        "\n",
        "\n",
        "    edge = avg_edge_train_small[ix_valid]\n",
        "    edge_v_list_knn.append(edge)\n",
        "\n",
        "    # baseline training\n",
        "    baseline = BaselineRegressor()\n",
        "    baseline.fit(X_t, y_t)\n",
        "    baseline_regressors.append(baseline)\n",
        "\n",
        "    # baseline predictions\n",
        "    y_b = baseline.predict(X_v)\n",
        "    y_hat_bl.append(y_b)\n",
        "\n",
        "\n",
        "    # KNN training\n",
        "    model = MultiOutputRegressor(KNeighborsRegressor(n_neighbors=170, weights=\"distance\"))\n",
        "    model.fit(X_t, y_t)\n",
        "    KNN_models.append(model)\n",
        "\n",
        "\n",
        "    # KNN predictions\n",
        "    y_hat = model.predict(X_v)\n",
        "    y_hat_knn.append(y_hat)\n",
        "\n",
        "\n",
        "    print('Prediction score: {}'.format(model.score(X_v, y_v)))\n",
        "\n",
        "\n",
        "\n",
        "scores = 0\n",
        "for y_hat, y_b, y_v, y_e in zip(y_hat_knn, y_hat_bl, y_v_list_knn, edge_v_list_knn):\n",
        "    score = 0\n",
        "    for i in COL_IX:\n",
        "        print('\\n')\n",
        "        print(\"Result for parameter: \", LABEL_NAMES[i])\n",
        "        mse_knn = mean_squared_error(y_v[:, i] * LABEL_MAXS[i], y_hat[:, i] * LABEL_MAXS[i])\n",
        "        mse_bl = mean_squared_error(y_v[:, i] * LABEL_MAXS[i], y_b[:, i] * LABEL_MAXS[i])\n",
        "        score += mse_knn / mse_bl\n",
        "        print(f\"Baseline MSE:      {mse_bl:.2f}\")\n",
        "        print(f\"Random Forest MSE: {mse_knn:.2f} ({1e2*(mse_knn - mse_bl)/mse_bl:+.2f} %)\")\n",
        "\n",
        "\n",
        "    print('\\n')\n",
        "    print(\"CV Evaluation score:\", score / 4)\n",
        "\n",
        "    scores += score\n",
        "\n",
        "print('\\n')\n",
        "print(\"OVERALL CV EVALUATION SCORE:\", scores / 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "300d2d7a",
      "metadata": {
        "id": "300d2d7a"
      },
      "source": [
        "#### 5.C.iv. PERFORMANCE EVALUATION ON VALIDATION SET (FOR SMALL FIELDS)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2a2a7c8a",
      "metadata": {
        "id": "2a2a7c8a"
      },
      "outputs": [],
      "source": [
        "scores = 0\n",
        "out_table = []\n",
        "\n",
        "for y_h, y_b, y_v, y_e in zip(y_hat_knn, y_hat_bl, y_v_list_knn, edge_v_list_knn):\n",
        "    mse_knn = np.square(np.subtract(y_h * LABEL_MAXS, y_v * LABEL_MAXS))\n",
        "    mse_bl = np.square(np.subtract(y_v * LABEL_MAXS, y_b * LABEL_MAXS))\n",
        "    row = np.zeros((len(y_h), 9))\n",
        "    row[:, 8] = y_e\n",
        "    row[:, 0:4] = mse_knn\n",
        "    row[:, 4:8] = mse_bl\n",
        "    out_table.append(row)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d31b5ed4",
      "metadata": {
        "id": "d31b5ed4",
        "outputId": "0f407d98-a149-4554-87f3-fdee333cdd27"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.\n",
            "  return _methods._mean(a, axis=axis, dtype=dtype,\n",
            "/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide\n",
            "  ret = ret.dtype.type(ret / rcount)\n"
          ]
        }
      ],
      "source": [
        "x = np.concatenate(out_table, 0)\n",
        "df = pd.DataFrame(\n",
        "    x,\n",
        "    columns=[\"P2O5\", \"K\", \"Mg\", \"pH\", \"P2O5_avg\", \"K_avg\", \"Mg_avg\", \"pH_avg\", \"Edge\"],\n",
        ")\n",
        "df.head(10)\n",
        "_, bin_edge = np.histogram(df.Edge.values, bins=4)\n",
        "# print(bin_edge)\n",
        "bin_edge = [0, 11, 40, 50, 100, 110, 120, 130, 210]\n",
        "bin_edge_labels = [\n",
        "    \"0-11\",\n",
        "    \"11-40\",\n",
        "    \"40-50\",\n",
        "    \"50-100\",\n",
        "    \"100-110\",\n",
        "    \"110-120\",\n",
        "    \"120-130\",\n",
        "    \"130+\",\n",
        "]\n",
        "mse_per_edge = np.zeros((len(bin_edge) - 1, 6), dtype=object)\n",
        "\n",
        "for i in range(1, len(bin_edge)):\n",
        "    d_temp = df[(df.Edge <= bin_edge[i]) & (df.Edge > bin_edge[i - 1])]\n",
        "    mse_per_edge[i - 1, 0] = np.mean(d_temp.P2O5.values) / np.mean(\n",
        "        d_temp.P2O5_avg.values\n",
        "    )\n",
        "    mse_per_edge[i - 1, 1] = np.mean(d_temp.K.values) / np.mean(d_temp.K_avg.values)\n",
        "    mse_per_edge[i - 1, 2] = np.mean(d_temp.Mg.values) / np.mean(d_temp.Mg_avg.values)\n",
        "    mse_per_edge[i - 1, 3] = np.mean(d_temp.pH.values) / np.mean(d_temp.pH_avg.values)\n",
        "    mse_per_edge[i - 1, 5] = len(d_temp)\n",
        "    mse_per_edge[i - 1, 4] = bin_edge_labels[i - 1]\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2dd0777d-1d6b-4b7b-8720-99bb5afc9935",
      "metadata": {
        "id": "2dd0777d-1d6b-4b7b-8720-99bb5afc9935"
      },
      "source": [
        "In the following cell, the mean validation score is demonstrated for each soil parameter and field size:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "64afa461-69e1-4731-a3e6-45b28a76dc46",
      "metadata": {
        "id": "64afa461-69e1-4731-a3e6-45b28a76dc46",
        "outputId": "72161ac3-254b-497d-b3cc-00ce48ac1127"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>P2O5</th>\n",
              "      <th>K</th>\n",
              "      <th>Mg</th>\n",
              "      <th>pH</th>\n",
              "      <th>Edge</th>\n",
              "      <th>Len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.01189</td>\n",
              "      <td>0.957566</td>\n",
              "      <td>0.996239</td>\n",
              "      <td>0.710062</td>\n",
              "      <td>0-11</td>\n",
              "      <td>650</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>11-40</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>40-50</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>50-100</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>100-110</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>110-120</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>120-130</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>130+</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      P2O5         K        Mg        pH     Edge  Len\n",
              "0  1.01189  0.957566  0.996239  0.710062     0-11  650\n",
              "1      NaN       NaN       NaN       NaN    11-40    0\n",
              "2      NaN       NaN       NaN       NaN    40-50    0\n",
              "3      NaN       NaN       NaN       NaN   50-100    0\n",
              "4      NaN       NaN       NaN       NaN  100-110    0\n",
              "5      NaN       NaN       NaN       NaN  110-120    0\n",
              "6      NaN       NaN       NaN       NaN  120-130    0\n",
              "7      NaN       NaN       NaN       NaN     130+    0"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "d_out = pd.DataFrame(mse_per_edge, columns=[\"P2O5\", \"K\", \"Mg\", \"pH\", \"Edge\", \"Len\"])\n",
        "d_out"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "514663c7",
      "metadata": {
        "id": "514663c7"
      },
      "source": [
        "In the following cell the mean validation score is calculated for each soil parameter."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7bf20c4c",
      "metadata": {
        "id": "7bf20c4c"
      },
      "source": [
        "#### 5.C.v. MAKE SUBMISSIONS ( FOR THE SMALL FIELDS <= 11x11 PIXELS)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "12c162ae",
      "metadata": {
        "id": "12c162ae",
        "outputId": "47d10224-28f1-4c60-e497-9945101521a4"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7effcc2051f0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27ee0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7effcc2051f0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca278b0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7effcc2051f0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7effcc2051f0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27ee0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca278b0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7effcc2051f0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca278b0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27ee0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7effcc2051f0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27ee0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca278b0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca27ee0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca278b0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1c9f49d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca278b0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1c9f49d0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n",
            "Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7eff1ca278b0>\n",
            "Traceback (most recent call last):\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 400, in match_module_callback\n",
            "    self._make_module_from_path(filepath)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 515, in _make_module_from_path\n",
            "    module = module_class(filepath, prefix, user_api, internal_api)\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 606, in __init__\n",
            "    self.version = self.get_version()\n",
            "  File \"/home/ai4ex2022/anaconda3/envs/ai4ex/lib/python3.9/site-packages/threadpoolctl.py\", line 646, in get_version\n",
            "    config = get_config().split()\n",
            "AttributeError: 'NoneType' object has no attribute 'split'\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>P</th>\n",
              "      <th>K</th>\n",
              "      <th>Mg</th>\n",
              "      <th>pH</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>69.046753</td>\n",
              "      <td>217.210041</td>\n",
              "      <td>156.625057</td>\n",
              "      <td>6.876837</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>68.314687</td>\n",
              "      <td>219.122053</td>\n",
              "      <td>159.695079</td>\n",
              "      <td>6.869646</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>68.406225</td>\n",
              "      <td>218.932231</td>\n",
              "      <td>159.838505</td>\n",
              "      <td>6.868999</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>70.056213</td>\n",
              "      <td>219.971472</td>\n",
              "      <td>160.081745</td>\n",
              "      <td>6.875113</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>70.007512</td>\n",
              "      <td>219.658213</td>\n",
              "      <td>165.494298</td>\n",
              "      <td>6.915121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>427</th>\n",
              "      <td>70.749886</td>\n",
              "      <td>219.476535</td>\n",
              "      <td>165.619024</td>\n",
              "      <td>6.918288</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>428</th>\n",
              "      <td>69.396729</td>\n",
              "      <td>216.954871</td>\n",
              "      <td>157.758990</td>\n",
              "      <td>6.869399</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>429</th>\n",
              "      <td>69.869062</td>\n",
              "      <td>219.777196</td>\n",
              "      <td>159.857493</td>\n",
              "      <td>6.877047</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>430</th>\n",
              "      <td>72.610002</td>\n",
              "      <td>225.916411</td>\n",
              "      <td>163.957139</td>\n",
              "      <td>6.919657</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>431</th>\n",
              "      <td>69.236459</td>\n",
              "      <td>217.965035</td>\n",
              "      <td>158.721508</td>\n",
              "      <td>6.871160</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>432 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "             P           K          Mg        pH\n",
              "0    69.046753  217.210041  156.625057  6.876837\n",
              "1    68.314687  219.122053  159.695079  6.869646\n",
              "2    68.406225  218.932231  159.838505  6.868999\n",
              "3    70.056213  219.971472  160.081745  6.875113\n",
              "4    70.007512  219.658213  165.494298  6.915121\n",
              "..         ...         ...         ...       ...\n",
              "427  70.749886  219.476535  165.619024  6.918288\n",
              "428  69.396729  216.954871  157.758990  6.869399\n",
              "429  69.869062  219.777196  159.857493  6.877047\n",
              "430  72.610002  225.916411  163.957139  6.919657\n",
              "431  69.236459  217.965035  158.721508  6.871160\n",
              "\n",
              "[432 rows x 4 columns]"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#Generate the submission for the small fields\n",
        "\n",
        "predictions_small = []\n",
        "counter = 0\n",
        "for knn in KNN_models:\n",
        "    pp = knn.predict(X_te_processed_normalized[avg_edge_test<=11, :])\n",
        "    predictions_small.append(pp)\n",
        "\n",
        "predictions_small = np.asarray(predictions_small)\n",
        "\n",
        "\n",
        "predictions_small = np.mean(predictions_small, axis=0)\n",
        "predictions_small = predictions_small * LABEL_MAXS # predictions were in the range of [0,1] so that we neeed to scale them back.\n",
        "\n",
        "submission_df = pd.DataFrame(data=predictions_small, columns=[\"P\", \"K\", \"Mg\", \"pH\"])\n",
        "submission_df.to_csv(\"submission_small_fields.csv\", index_label=\"sample_index\")\n",
        "submission_df"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a75651c5",
      "metadata": {
        "id": "a75651c5"
      },
      "source": [
        "### 5.D. MERGE SMALL AND LARGE FIELD SUBMISSIONS ( FINAL SUBMISSION FILE)\n",
        "\n",
        "The final submission is the combined result of both models: The Random Forest for large patches and the KNN for small patches (11x11 pixels)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7f0c71bb",
      "metadata": {
        "id": "7f0c71bb",
        "outputId": "3503b71e-e523-4104-be41-41eeb4c19254"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>P</th>\n",
              "      <th>K</th>\n",
              "      <th>Mg</th>\n",
              "      <th>pH</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>69.046753</td>\n",
              "      <td>217.210041</td>\n",
              "      <td>156.625057</td>\n",
              "      <td>6.876837</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>68.314687</td>\n",
              "      <td>219.122053</td>\n",
              "      <td>159.695079</td>\n",
              "      <td>6.869646</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>68.406225</td>\n",
              "      <td>218.932231</td>\n",
              "      <td>159.838505</td>\n",
              "      <td>6.868999</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>70.056213</td>\n",
              "      <td>219.971472</td>\n",
              "      <td>160.081745</td>\n",
              "      <td>6.875113</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>70.007512</td>\n",
              "      <td>219.658213</td>\n",
              "      <td>165.494298</td>\n",
              "      <td>6.915121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1149</th>\n",
              "      <td>46.936877</td>\n",
              "      <td>169.610192</td>\n",
              "      <td>135.352855</td>\n",
              "      <td>6.628018</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1150</th>\n",
              "      <td>45.545826</td>\n",
              "      <td>169.880842</td>\n",
              "      <td>135.613945</td>\n",
              "      <td>6.610217</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1151</th>\n",
              "      <td>76.024305</td>\n",
              "      <td>239.529187</td>\n",
              "      <td>167.534027</td>\n",
              "      <td>6.643992</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1152</th>\n",
              "      <td>47.691755</td>\n",
              "      <td>182.878088</td>\n",
              "      <td>138.305905</td>\n",
              "      <td>6.612771</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1153</th>\n",
              "      <td>62.198148</td>\n",
              "      <td>216.640894</td>\n",
              "      <td>151.633768</td>\n",
              "      <td>6.632037</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1154 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "              P           K          Mg        pH\n",
              "0     69.046753  217.210041  156.625057  6.876837\n",
              "1     68.314687  219.122053  159.695079  6.869646\n",
              "2     68.406225  218.932231  159.838505  6.868999\n",
              "3     70.056213  219.971472  160.081745  6.875113\n",
              "4     70.007512  219.658213  165.494298  6.915121\n",
              "...         ...         ...         ...       ...\n",
              "1149  46.936877  169.610192  135.352855  6.628018\n",
              "1150  45.545826  169.880842  135.613945  6.610217\n",
              "1151  76.024305  239.529187  167.534027  6.643992\n",
              "1152  47.691755  182.878088  138.305905  6.612771\n",
              "1153  62.198148  216.640894  151.633768  6.632037\n",
              "\n",
              "[1154 rows x 4 columns]"
            ]
          },
          "execution_count": 27,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "submission_df = pd.DataFrame(data=np.concatenate([predictions_small , predictions_large],axis=0), columns=[\"P\", \"K\", \"Mg\", \"pH\"])\n",
        "submission_df.to_csv(\"submission_hybrid.csv\", index_label=\"sample_index\")\n",
        "submission_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fac09416",
      "metadata": {
        "id": "fac09416"
      },
      "outputs": [],
      "source": [
        "divisor = np.full(submission_df.shape ,np.array([ 70.30265589, 227.98851039, 159.28123557,   6.7827194 ]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2899fdf4",
      "metadata": {
        "id": "2899fdf4",
        "outputId": "8aaa3ff1-93a5-47aa-ea30-b361bb040393"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sample_index</th>\n",
              "      <th>Target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0_P</td>\n",
              "      <td>0.982136</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1_P</td>\n",
              "      <td>0.971723</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2_P</td>\n",
              "      <td>0.973025</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3_P</td>\n",
              "      <td>0.996495</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4_P</td>\n",
              "      <td>0.995802</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4611</th>\n",
              "      <td>1149_pH</td>\n",
              "      <td>0.977192</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4612</th>\n",
              "      <td>1150_pH</td>\n",
              "      <td>0.974567</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4613</th>\n",
              "      <td>1151_pH</td>\n",
              "      <td>0.979547</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4614</th>\n",
              "      <td>1152_pH</td>\n",
              "      <td>0.974944</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4615</th>\n",
              "      <td>1153_pH</td>\n",
              "      <td>0.977784</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>4616 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "     sample_index    Target\n",
              "0             0_P  0.982136\n",
              "1             1_P  0.971723\n",
              "2             2_P  0.973025\n",
              "3             3_P  0.996495\n",
              "4             4_P  0.995802\n",
              "...           ...       ...\n",
              "4611      1149_pH  0.977192\n",
              "4612      1150_pH  0.974567\n",
              "4613      1151_pH  0.979547\n",
              "4614      1152_pH  0.974944\n",
              "4615      1153_pH  0.977784\n",
              "\n",
              "[4616 rows x 2 columns]"
            ]
          },
          "execution_count": 46,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sub = (submission_df / divisor).unstack().reset_index().rename(columns={0: 'Target'})\n",
        "sub['sample_index'] = sub['level_1'].astype('str') + '_' + sub['level_0']\n",
        "sub = sub[['sample_index', 'Target']]\n",
        "sub"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "39ead86d",
      "metadata": {
        "id": "39ead86d"
      },
      "outputs": [],
      "source": [
        "sub.to_csv('baseline.csv', index=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "476a09db-5f0f-4a84-8f11-8d825d2cd1c0",
      "metadata": {
        "id": "476a09db-5f0f-4a84-8f11-8d825d2cd1c0"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "ai4ex",
      "language": "python",
      "name": "ai4ex"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    },
    "toc-autonumbering": false,
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}