examples/02_models_comparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RePlay recommender models comparison\n",
    "\n",
    "We will show the main RePlay functionality and compare performance of RePlay models on well-known MovieLens dataset. If you have not used RePlay before, start with 01_replay_basics.ipynb which introduces base concepts and describe main classes and functionality.\n",
    "\n",
    "### Dataset\n",
    "We will compare RePlay models on __MovieLens 1m__. \n",
    "\n",
    "### Dataset preprocessing: \n",
    "Ratings greater than or equal to 3 are considered as positive interactions.\n",
    "\n",
    "### Data split\n",
    "Dataset is split by date so that 20% of the last interactions as are placed in the test part. Cold items and users are dropped.\n",
    "\n",
    "### Predict:\n",
    "We will predict top-10 most relevant films for each user.\n",
    "\n",
    "### Metrics\n",
    "Quality metrics used:__ndcg@k, hitrate@k, map@k, mrr@k__ for k = 1, 5, 10\n",
    "Additional metrics used: __coverage@k__ and __surprisal@k__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install rs-datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-10T16:01:45.639135Z",
     "start_time": "2020-02-10T16:01:45.612577Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config Completer.use_jedi = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "from optuna.exceptions import ExperimentalWarning\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=ExperimentalWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type='text/css'>\n",
       ".datatable table.frame { margin-bottom: 0; }\n",
       ".datatable table.frame thead { border-bottom: none; }\n",
       ".datatable table.frame tr.coltypes td {  color: #FFFFFF;  line-height: 6px;  padding: 0 0.5em;}\n",
       ".datatable .bool    { background: #DDDD99; }\n",
       ".datatable .object  { background: #565656; }\n",
       ".datatable .int     { background: #5D9E5D; }\n",
       ".datatable .float   { background: #4040CC; }\n",
       ".datatable .str     { background: #CC4040; }\n",
       ".datatable .time    { background: #40CC40; }\n",
       ".datatable .row_index {  background: var(--jp-border-color3);  border-right: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  font-size: 9px;}\n",
       ".datatable .frame tbody td { text-align: left; }\n",
       ".datatable .frame tr.coltypes .row_index {  background: var(--jp-border-color0);}\n",
       ".datatable th:nth-child(2) { padding-left: 12px; }\n",
       ".datatable .hellipsis {  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .vellipsis {  background: var(--jp-layout-color0);  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .na {  color: var(--jp-cell-editor-border-color);  font-size: 80%;}\n",
       ".datatable .sp {  opacity: 0.25;}\n",
       ".datatable .footer { font-size: 9px; }\n",
       ".datatable .frame_dimensions {  background: var(--jp-border-color3);  border-top: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  display: inline-block;  opacity: 0.6;  padding: 1px 10px 1px 5px;}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import logging\n",
    "import time\n",
    "\n",
    "from pyspark.sql import functions as sf\n",
    "\n",
    "from replay.data import Dataset, FeatureHint, FeatureInfo, FeatureSchema, FeatureType\n",
    "from replay.data.dataset_utils import DatasetLabelEncoder\n",
    "from replay.metrics import Coverage, HitRate, MRR, MAP, NDCG, Surprisal, Experiment, OfflineMetrics\n",
    "from replay.models import (\n",
    "    ALSWrap, \n",
    "    ItemKNN,\n",
    "    SLIM, \n",
    "    PopRec,\n",
    "    RandomRec,\n",
    "    UCB,\n",
    "    Wilson, \n",
    "    Word2VecRec,\n",
    ")\n",
    "\n",
    "from replay.utils.session_handler import State\n",
    "from replay.splitters import TimeSplitter\n",
    "from replay.utils.spark_utils import convert2spark, get_log_info\n",
    "from rs_datasets import MovieLens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`State` object allows passing existing Spark session or create a new one, which will be used by the all RePlay modules.\n",
    "\n",
    "To create session with custom parameters ``spark.driver.memory`` and ``spark.sql.shuffle.partitions`` use function `get_spark_session` from `session_handler` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-10T15:59:09.227179Z",
     "start_time": "2020-02-10T15:59:06.427348Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "spark = State().session\n",
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.sparkContext.setLogLevel('ERROR')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "logger = logging.getLogger(\"replay\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = 10\n",
    "K_list_metrics = [1, 5, 10]\n",
    "BUDGET = 5\n",
    "SEED = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Preprocessing <a name='data-preparator'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.1 Data loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-10T15:59:42.041251Z",
     "start_time": "2020-02-10T15:59:09.230636Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratings\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  item_id  rating  timestamp\n",
       "0        1     1193       5  978300760\n",
       "1        1      661       3  978302109\n",
       "2        1      914       3  978301968"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "users\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id gender  age  occupation zip_code\n",
       "0        1      F    1          10    48067\n",
       "1        2      M   56          16    70072\n",
       "2        3      M   25          15    55117"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "items\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item_id                    title                        genres\n",
       "0        1         Toy Story (1995)   Animation|Children's|Comedy\n",
       "1        2           Jumanji (1995)  Adventure|Children's|Fantasy\n",
       "2        3  Grumpier Old Men (1995)                Comedy|Romance"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "data = MovieLens(\"1m\")\n",
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.2. Dataset preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### interactions preprocessing\n",
    "\n",
    "- converting to spark dataframe\n",
    "- separating positive and negative feedback interactions\n",
    "- spliting data for train/test and optimization dataframes\n",
    "- creating Dataset instances for experiments\n",
    "- encoding datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------+------+---------+\n",
      "|user_id|item_id|rating|timestamp|\n",
      "+-------+-------+------+---------+\n",
      "|      1|   1193|     5|978300760|\n",
      "|      1|    661|     3|978302109|\n",
      "|      1|    914|     3|978301968|\n",
      "|      1|   3408|     4|978300275|\n",
      "|      1|   2355|     5|978824291|\n",
      "+-------+-------+------+---------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "interactions_spark = convert2spark(data.ratings)\n",
    "interactions_spark.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "836478"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# will consider ratings >= 3 as positive feedback. A positive feedback is treated with rating = 1\n",
    "only_positives_interactions = interactions_spark.filter(sf.col('rating') >= 3).withColumn('rating', sf.lit(1))\n",
    "only_positives_interactions.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train info:\n",
      " total lines: 669181, total users: 5397, total items: 3569\n",
      "test info:\n",
      " total lines: 86542, total users: 1139, total items: 3279\n"
     ]
    }
   ],
   "source": [
    "# train/test split \n",
    "train_spl = TimeSplitter(\n",
    "    time_threshold=0.2,\n",
    "    drop_cold_items=True,\n",
    "    drop_cold_users=True,\n",
    "    query_column=\"user_id\",\n",
    ")\n",
    "train, test = train_spl.split(only_positives_interactions)\n",
    "print('train info:\\n', get_log_info(train, user_col=\"user_id\", item_col=\"item_id\"))\n",
    "print('test info:\\n', get_log_info(test, user_col=\"user_id\", item_col=\"item_id\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.is_cached"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(535343, 24241)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train/test split for hyperparameters selection\n",
    "opt_train, opt_val = train_spl.split(train)\n",
    "opt_train.count(), opt_val.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opt_train.is_cached"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "text/plain": [
       "798993"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# negative feedback will be used for Wilson and UCB models\n",
    "only_negatives_log = interactions_spark.filter(sf.col('rating') < 3).withColumn('rating', sf.lit(0.))\n",
    "test_start = test.agg(sf.min('timestamp')).first()[0]\n",
    "\n",
    "# train with both positive and negative feedback\n",
    "pos_neg_train=(train\n",
    "              .withColumn('rating', sf.lit(1.))\n",
    "              .union(only_negatives_log.filter(sf.col('timestamp') < test_start))\n",
    "             )\n",
    "pos_neg_train.cache()\n",
    "pos_neg_train.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_schema = FeatureSchema(\n",
    "    [\n",
    "        FeatureInfo(\n",
    "            column=\"user_id\",\n",
    "            feature_type=FeatureType.CATEGORICAL,\n",
    "            feature_hint=FeatureHint.QUERY_ID,\n",
    "        ),\n",
    "        FeatureInfo(\n",
    "            column=\"item_id\",\n",
    "            feature_type=FeatureType.CATEGORICAL,\n",
    "            feature_hint=FeatureHint.ITEM_ID,\n",
    "        ),\n",
    "        FeatureInfo(\n",
    "            column=\"rating\",\n",
    "            feature_type=FeatureType.NUMERICAL,\n",
    "            feature_hint=FeatureHint.RATING,\n",
    "        ),\n",
    "        FeatureInfo(\n",
    "            column=\"timestamp\",\n",
    "            feature_type=FeatureType.NUMERICAL,\n",
    "            feature_hint=FeatureHint.TIMESTAMP,\n",
    "        ),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_dataset = Dataset(\n",
    "    feature_schema=feature_schema,\n",
    "    interactions=interactions_spark,\n",
    ")\n",
    "\n",
    "train_dataset = Dataset(\n",
    "    feature_schema=feature_schema,\n",
    "    interactions=train,\n",
    ")\n",
    "\n",
    "test_dataset = Dataset(\n",
    "    feature_schema=feature_schema,\n",
    "    interactions=test,\n",
    ")\n",
    "\n",
    "train_neg_dataset = Dataset(\n",
    "    feature_schema=feature_schema,\n",
    "    interactions=pos_neg_train,\n",
    ")\n",
    "\n",
    "opt_train_dataset = Dataset(\n",
    "    feature_schema=feature_schema,\n",
    "    interactions=opt_train,\n",
    ")\n",
    "\n",
    "opt_val_dataset = Dataset(\n",
    "    feature_schema=feature_schema,\n",
    "    interactions=opt_val,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "encoder = DatasetLabelEncoder()\n",
    "encoder.fit(all_dataset)\n",
    "train_dataset = encoder.transform(train_dataset)\n",
    "train_neg_dataset = encoder.transform(train_neg_dataset)\n",
    "test_dataset = encoder.transform(test_dataset)\n",
    "opt_train_dataset = encoder.transform(opt_train_dataset)\n",
    "opt_val_dataset = encoder.transform(opt_val_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Metrics definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# experiment is used for metrics calculation\n",
    "e = Experiment(\n",
    "    [\n",
    "        MAP(K), \n",
    "        NDCG(K), \n",
    "        HitRate(K_list_metrics), \n",
    "        Coverage(K),\n",
    "        Surprisal(K),\n",
    "        MRR(K),\n",
    "    ],\n",
    "    test_dataset.interactions,\n",
    "    train_dataset.interactions,\n",
    "    query_column=train_dataset.feature_schema.query_id_column,\n",
    "    item_column=train_dataset.feature_schema.item_id_column,\n",
    "    rating_column=train_dataset.feature_schema.interactions_rating_column,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Models training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_predict_add_res(name, model, experiment, train_dataset, suffix=''):\n",
    "    \"\"\"\n",
    "    Run fit_predict for the `model`, measure time on fit_predict and evaluate metrics\n",
    "    \"\"\"\n",
    "    start_time=time.time()\n",
    "    \n",
    "    logs = {'dataset': train_dataset}\n",
    "    predict_params = {'k': K, 'queries': test_dataset.query_ids}\n",
    "    \n",
    "    if isinstance(model, (Wilson, UCB)):\n",
    "        logs['dataset'] = train_neg_dataset\n",
    "    \n",
    "    predict_params.update(logs)\n",
    "\n",
    "    model.fit(**logs)\n",
    "    fit_time = time.time() - start_time\n",
    "\n",
    "    pred=model.predict(**predict_params)\n",
    "    pred.cache()\n",
    "    pred.count()\n",
    "    predict_time = time.time() - start_time - fit_time\n",
    "\n",
    "    experiment.add_result(name + suffix, pred)\n",
    "    metric_time = time.time() - start_time - fit_time - predict_time\n",
    "    experiment.results.loc[name + suffix, 'fit_time'] = fit_time\n",
    "    experiment.results.loc[name + suffix, 'predict_time'] = predict_time\n",
    "    experiment.results.loc[name + suffix, 'metric_time'] = metric_time\n",
    "    experiment.results.loc[name + suffix, 'full_time'] = (fit_time + \n",
    "                                                          predict_time +\n",
    "                                                          metric_time)\n",
    "    pred.unpersist()\n",
    "    print(experiment.results[['NDCG@{}'.format(K), 'MRR@{}'.format(K), 'Coverage@{}'.format(K), 'fit_time']].sort_values('NDCG@{}'.format(K), ascending=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def full_pipeline(models, experiment, train_dataset, suffix='', budget=BUDGET):\n",
    "    \"\"\"\n",
    "    For each model:\n",
    "        -  if required: run hyperparameters search, set best params and save param values to `experiment`\n",
    "        - pass model to `fit_predict_add_res`        \n",
    "    \"\"\"\n",
    "    \n",
    "    for name, [model, params] in models.items():\n",
    "        model.logger.info(msg='{} started'.format(name))\n",
    "        if params != 'no_opt':\n",
    "            model.logger.info(msg='{} optimization started'.format(name))\n",
    "            best_params = model.optimize(opt_train_dataset, \n",
    "                                         opt_val_dataset, \n",
    "                                         param_borders=params, \n",
    "                                         k=K, \n",
    "                                         budget=budget)\n",
    "            logger.info(msg='best params for {} are: {}'.format(name, best_params))\n",
    "            model.set_params(**best_params)\n",
    "        \n",
    "        logger.info(msg='{} fit_predict started'.format(name))\n",
    "        fit_predict_add_res(name, model, experiment, train_dataset, suffix)\n",
    "        # here we call protected attribute to get all parameters set during model initialization\n",
    "        experiment.results.loc[name + suffix, 'params'] = str(model._init_args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Non-personalized models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "non_personalized_models = {\n",
    "    'Popular': [PopRec(), 'no_opt'], \n",
    "    'Random (uniform)': [RandomRec(seed=SEED, distribution='uniform'), 'no_opt'], \n",
    "    'Random (popularity-based)': [RandomRec(seed=SEED, distribution='popular_based'), {\"alpha\": [-0.5, 100]}],\n",
    "    'UCB': [UCB(exploration_coef=0.5), 'no_opt'],\n",
    "    'Wilson': [Wilson(), 'no_opt'],\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "10-Nov-23 17:23:20, replay, INFO: Popular started\n",
      "10-Nov-23 17:23:20, replay, INFO: Popular fit_predict started\n",
      "10-Nov-23 17:23:41, replay, INFO: Random (uniform) started                      \n",
      "10-Nov-23 17:23:41, replay, INFO: Random (uniform) fit_predict started\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          NDCG@10    MRR@10  Coverage@10  fit_time\n",
      "Popular  0.243711  0.390426     0.033903  3.411652\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "10-Nov-23 17:24:00, replay, INFO: Random (popularity-based) started             \n",
      "10-Nov-23 17:24:00, replay, INFO: Random (popularity-based) optimization started\n",
      "[I 2023-11-10 17:24:00,201] A new study created in memory with name: no-name-6b85ae22-5948-4082-85d0-6665ce7023ef\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   NDCG@10    MRR@10  Coverage@10  fit_time\n",
      "Popular           0.243711  0.390426     0.033903  3.411652\n",
      "Random (uniform)  0.026336  0.065413     0.954609  3.629176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/work/replay_gitlab/RePlay/replay/optimization/optuna_objective.py:77: FutureWarning: suggest_uniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use suggest_float instead.\n",
      "  res[param] = suggest_fn(param, low=low, high=high)\n",
      "[I 2023-11-10 17:24:12,623] Trial 0 finished with value: 0.07140950090839235 and parameters: {'distribution': 'popular_based', 'alpha': 0.0}. Best is trial 0 with value: 0.07140950090839235.\n",
      "[I 2023-11-10 17:24:23,765] Trial 1 finished with value: 0.05605109319473708 and parameters: {'distribution': 'popular_based', 'alpha': 85.86901011386752}. Best is trial 0 with value: 0.07140950090839235.\n",
      "[I 2023-11-10 17:24:34,597] Trial 2 finished with value: 0.05306280968531626 and parameters: {'distribution': 'popular_based', 'alpha': 65.19693619996353}. Best is trial 0 with value: 0.07140950090839235.\n",
      "[I 2023-11-10 17:24:45,317] Trial 3 finished with value: 0.052397564978934066 and parameters: {'distribution': 'popular_based', 'alpha': 99.70715640866017}. Best is trial 0 with value: 0.07140950090839235.\n",
      "[I 2023-11-10 17:24:56,990] Trial 4 finished with value: 0.06130793931641172 and parameters: {'distribution': 'popular_based', 'alpha': 74.04662750447105}. Best is trial 0 with value: 0.07140950090839235.\n",
      "10-Nov-23 17:24:56, replay, INFO: best params for Random (popularity-based) are: {'distribution': 'popular_based', 'alpha': 0.0}\n",
      "10-Nov-23 17:24:56, replay, INFO: Random (popularity-based) fit_predict started\n",
      "10-Nov-23 17:25:14, replay, INFO: UCB started                                   \n",
      "10-Nov-23 17:25:14, replay, INFO: UCB fit_predict started\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10  fit_time\n",
      "Popular                    0.243711  0.390426     0.033903  3.411652\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192  3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609  3.629176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "10-Nov-23 17:25:30, replay, INFO: Wilson started                                \n",
      "10-Nov-23 17:25:30, replay, INFO: Wilson fit_predict started\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10  fit_time\n",
      "Popular                    0.243711  0.390426     0.033903  3.411652\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192  3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609  3.629176\n",
      "UCB                        0.000458  0.001024     0.003082  3.415656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10  fit_time\n",
      "Popular                    0.243711  0.390426     0.033903  3.411652\n",
      "Wilson                     0.092121  0.180976     0.017092  3.453442\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192  3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609  3.629176\n",
      "UCB                        0.000458  0.001024     0.003082  3.415656\n",
      "CPU times: user 1.7 s, sys: 492 ms, total: 2.19 s\n",
      "Wall time: 2min 26s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "full_pipeline(non_personalized_models, e, train_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAP@10</th>\n",
       "      <th>NDCG@10</th>\n",
       "      <th>HitRate@1</th>\n",
       "      <th>HitRate@5</th>\n",
       "      <th>HitRate@10</th>\n",
       "      <th>Coverage@10</th>\n",
       "      <th>Surprisal@10</th>\n",
       "      <th>MRR@10</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>predict_time</th>\n",
       "      <th>metric_time</th>\n",
       "      <th>full_time</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Popular</th>\n",
       "      <td>0.157302</td>\n",
       "      <td>0.243711</td>\n",
       "      <td>0.284460</td>\n",
       "      <td>0.530290</td>\n",
       "      <td>0.645303</td>\n",
       "      <td>0.033903</td>\n",
       "      <td>0.118354</td>\n",
       "      <td>0.390426</td>\n",
       "      <td>3.411652</td>\n",
       "      <td>8.815796</td>\n",
       "      <td>8.258563</td>\n",
       "      <td>20.486010</td>\n",
       "      <td>{'use_rating': False, 'add_cold_items': True, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wilson</th>\n",
       "      <td>0.045002</td>\n",
       "      <td>0.092121</td>\n",
       "      <td>0.083406</td>\n",
       "      <td>0.345040</td>\n",
       "      <td>0.414399</td>\n",
       "      <td>0.017092</td>\n",
       "      <td>0.262190</td>\n",
       "      <td>0.180976</td>\n",
       "      <td>3.453442</td>\n",
       "      <td>6.875320</td>\n",
       "      <td>6.423975</td>\n",
       "      <td>16.752737</td>\n",
       "      <td>{'alpha': 0.05, 'add_cold_items': True, 'cold_...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random (popularity-based)</th>\n",
       "      <td>0.031447</td>\n",
       "      <td>0.074118</td>\n",
       "      <td>0.079017</td>\n",
       "      <td>0.269535</td>\n",
       "      <td>0.399473</td>\n",
       "      <td>0.635192</td>\n",
       "      <td>0.314159</td>\n",
       "      <td>0.160990</td>\n",
       "      <td>3.927062</td>\n",
       "      <td>6.529720</td>\n",
       "      <td>6.674805</td>\n",
       "      <td>17.131586</td>\n",
       "      <td>{'distribution': 'popular_based', 'alpha': 0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random (uniform)</th>\n",
       "      <td>0.009661</td>\n",
       "      <td>0.026336</td>\n",
       "      <td>0.032485</td>\n",
       "      <td>0.099210</td>\n",
       "      <td>0.181738</td>\n",
       "      <td>0.954609</td>\n",
       "      <td>0.537563</td>\n",
       "      <td>0.065413</td>\n",
       "      <td>3.629176</td>\n",
       "      <td>8.216730</td>\n",
       "      <td>7.108498</td>\n",
       "      <td>18.954404</td>\n",
       "      <td>{'distribution': 'uniform', 'alpha': 0.0, 'see...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UCB</th>\n",
       "      <td>0.000124</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000878</td>\n",
       "      <td>0.004390</td>\n",
       "      <td>0.003082</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.001024</td>\n",
       "      <td>3.415656</td>\n",
       "      <td>7.359273</td>\n",
       "      <td>5.900311</td>\n",
       "      <td>16.675241</td>\n",
       "      <td>{'exploration_coef': 0.5, 'sample': False, 'se...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             MAP@10   NDCG@10  HitRate@1  HitRate@5  \\\n",
       "Popular                    0.157302  0.243711   0.284460   0.530290   \n",
       "Wilson                     0.045002  0.092121   0.083406   0.345040   \n",
       "Random (popularity-based)  0.031447  0.074118   0.079017   0.269535   \n",
       "Random (uniform)           0.009661  0.026336   0.032485   0.099210   \n",
       "UCB                        0.000124  0.000458   0.000000   0.000878   \n",
       "\n",
       "                           HitRate@10  Coverage@10  Surprisal@10    MRR@10  \\\n",
       "Popular                      0.645303     0.033903      0.118354  0.390426   \n",
       "Wilson                       0.414399     0.017092      0.262190  0.180976   \n",
       "Random (popularity-based)    0.399473     0.635192      0.314159  0.160990   \n",
       "Random (uniform)             0.181738     0.954609      0.537563  0.065413   \n",
       "UCB                          0.004390     0.003082      1.000000  0.001024   \n",
       "\n",
       "                           fit_time  predict_time  metric_time  full_time  \\\n",
       "Popular                    3.411652      8.815796     8.258563  20.486010   \n",
       "Wilson                     3.453442      6.875320     6.423975  16.752737   \n",
       "Random (popularity-based)  3.927062      6.529720     6.674805  17.131586   \n",
       "Random (uniform)           3.629176      8.216730     7.108498  18.954404   \n",
       "UCB                        3.415656      7.359273     5.900311  16.675241   \n",
       "\n",
       "                                                                      params  \n",
       "Popular                    {'use_rating': False, 'add_cold_items': True, ...  \n",
       "Wilson                     {'alpha': 0.05, 'add_cold_items': True, 'cold_...  \n",
       "Random (popularity-based)  {'distribution': 'popular_based', 'alpha': 0.0...  \n",
       "Random (uniform)           {'distribution': 'uniform', 'alpha': 0.0, 'see...  \n",
       "UCB                        {'exploration_coef': 0.5, 'sample': False, 'se...  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.results.sort_values('NDCG@10', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "UCB is developed for iterative learning and improve its quality over time as the exploration reduces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "e.results.to_csv('res_21_rel_1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2  Personalized models without features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "common_models = {\n",
    "  'Implicit ALS': [ALSWrap(seed=SEED), None], \n",
    "  'Explicit ALS': [ALSWrap(seed=SEED, implicit_prefs=False), None], \n",
    "  'ItemKNN': [ItemKNN(), None], \n",
    "  'SLIM': [SLIM(seed=SEED), None],\n",
    "  'Word2Vec': [Word2VecRec(seed=SEED), None],\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "10-Nov-23 17:26:51, replay, INFO: Implicit ALS started\n",
      "10-Nov-23 17:26:51, replay, INFO: Implicit ALS optimization started\n",
      "[I 2023-11-10 17:26:51,161] A new study created in memory with name: no-name-346690a7-8179-4a82-8179-edbfdb8bf4a1\n",
      "/root/anaconda3/envs/replay/lib/python3.9/site-packages/pyspark/sql/context.py:125: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
      "  warnings.warn(\n",
      "[I 2023-11-10 17:27:08,366] Trial 0 finished with value: 0.2106789106218419 and parameters: {'rank': 10}. Best is trial 0 with value: 0.2106789106218419.\n",
      "[I 2023-11-10 17:27:24,184] Trial 1 finished with value: 0.1986859010576274 and parameters: {'rank': 26}. Best is trial 0 with value: 0.2106789106218419.\n",
      "[I 2023-11-10 17:27:41,154] Trial 2 finished with value: 0.1834988563514584 and parameters: {'rank': 40}. Best is trial 0 with value: 0.2106789106218419.\n",
      "[I 2023-11-10 17:28:51,126] Trial 3 finished with value: 0.16951885280473594 and parameters: {'rank': 168}. Best is trial 0 with value: 0.2106789106218419.\n",
      "[I 2023-11-10 17:29:08,679] Trial 4 finished with value: 0.18309653187459138 and parameters: {'rank': 44}. Best is trial 0 with value: 0.2106789106218419.\n",
      "10-Nov-23 17:29:08, replay, INFO: best params for Implicit ALS are: {'rank': 10}\n",
      "10-Nov-23 17:29:08, replay, INFO: Implicit ALS fit_predict started\n",
      "10-Nov-23 17:29:29, replay, INFO: Explicit ALS started                          \n",
      "10-Nov-23 17:29:29, replay, INFO: Explicit ALS optimization started\n",
      "[I 2023-11-10 17:29:29,410] A new study created in memory with name: no-name-5466a930-97d7-4516-a796-7fda85d4748e\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10  fit_time\n",
      "Implicit ALS               0.255083  0.412843     0.147380  4.430974\n",
      "Popular                    0.243711  0.390426     0.033903  3.411652\n",
      "Wilson                     0.092121  0.180976     0.017092  3.453442\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192  3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609  3.629176\n",
      "UCB                        0.000458  0.001024     0.003082  3.415656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/anaconda3/envs/replay/lib/python3.9/site-packages/pyspark/sql/context.py:125: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
      "  warnings.warn(\n",
      "[I 2023-11-10 17:29:43,046] Trial 0 finished with value: 0.01812132636739519 and parameters: {'rank': 10}. Best is trial 0 with value: 0.01812132636739519.\n",
      "[I 2023-11-10 17:30:18,482] Trial 1 finished with value: 0.01975373348436346 and parameters: {'rank': 107}. Best is trial 1 with value: 0.01975373348436346.\n",
      "[I 2023-11-10 17:30:36,546] Trial 2 finished with value: 0.02675199192979083 and parameters: {'rank': 46}. Best is trial 2 with value: 0.02675199192979083.\n",
      "[I 2023-11-10 17:30:50,656] Trial 3 finished with value: 0.014121230585317253 and parameters: {'rank': 9}. Best is trial 2 with value: 0.02675199192979083.\n",
      "[I 2023-11-10 17:31:24,553] Trial 4 finished with value: 0.014698300217981774 and parameters: {'rank': 102}. Best is trial 2 with value: 0.02675199192979083.\n",
      "10-Nov-23 17:31:24, replay, INFO: best params for Explicit ALS are: {'rank': 46}\n",
      "10-Nov-23 17:31:24, replay, INFO: Explicit ALS fit_predict started\n",
      "10-Nov-23 17:31:48, replay, INFO: ItemKNN started                               \n",
      "10-Nov-23 17:31:48, replay, INFO: ItemKNN optimization started\n",
      "[I 2023-11-10 17:31:48,566] A new study created in memory with name: no-name-9762b4f0-52a3-455f-baac-358aadcd5933\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10  fit_time\n",
      "Implicit ALS               0.255083  0.412843     0.147380  4.430974\n",
      "Popular                    0.243711  0.390426     0.033903  3.411652\n",
      "Wilson                     0.092121  0.180976     0.017092  3.453442\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192  3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609  3.629176\n",
      "Explicit ALS               0.019994  0.045540     0.431213  7.814560\n",
      "UCB                        0.000458  0.001024     0.003082  3.415656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2023-11-10 17:32:09,776] Trial 0 finished with value: 0.2081514555056189 and parameters: {'num_neighbours': 10, 'shrink': 0, 'weighting': None}. Best is trial 0 with value: 0.2081514555056189.\n",
      "[I 2023-11-10 17:32:18,049] Trial 1 finished with value: 0.21742351926120368 and parameters: {'num_neighbours': 27, 'shrink': 4, 'weighting': 'bm25'}. Best is trial 1 with value: 0.21742351926120368.\n",
      "[I 2023-11-10 17:32:26,907] Trial 2 finished with value: 0.22506007009967735 and parameters: {'num_neighbours': 84, 'shrink': 11, 'weighting': 'bm25'}. Best is trial 2 with value: 0.22506007009967735.\n",
      "[I 2023-11-10 17:32:35,270] Trial 3 finished with value: 0.23090776162732826 and parameters: {'num_neighbours': 73, 'shrink': 51, 'weighting': None}. Best is trial 3 with value: 0.23090776162732826.\n",
      "[I 2023-11-10 17:32:43,325] Trial 4 finished with value: 0.2285854011339672 and parameters: {'num_neighbours': 50, 'shrink': 66, 'weighting': None}. Best is trial 3 with value: 0.23090776162732826.\n",
      "10-Nov-23 17:32:43, replay, INFO: best params for ItemKNN are: {'num_neighbours': 73, 'shrink': 51, 'weighting': None}\n",
      "10-Nov-23 17:32:43, replay, INFO: ItemKNN fit_predict started\n",
      "10-Nov-23 17:33:14, replay, INFO: SLIM started                                  \n",
      "10-Nov-23 17:33:14, replay, INFO: SLIM optimization started\n",
      "[I 2023-11-10 17:33:14,427] A new study created in memory with name: no-name-7f54f9f5-9d7e-494f-a24f-184fd5427fbd\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10   fit_time\n",
      "ItemKNN                    0.257599  0.413689     0.058840  16.430850\n",
      "Implicit ALS               0.255083  0.412843     0.147380   4.430974\n",
      "Popular                    0.243711  0.390426     0.033903   3.411652\n",
      "Wilson                     0.092121  0.180976     0.017092   3.453442\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192   3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609   3.629176\n",
      "Explicit ALS               0.019994  0.045540     0.431213   7.814560\n",
      "UCB                        0.000458  0.001024     0.003082   3.415656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/work/replay_gitlab/RePlay/replay/optimization/optuna_objective.py:77: FutureWarning: suggest_loguniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use suggest_float(..., log=True) instead.\n",
      "  res[param] = suggest_fn(param, low=low, high=high)\n",
      "[I 2023-11-10 17:33:31,375] Trial 0 finished with value: 0.186897920331224 and parameters: {'beta': 0.01, 'lambda_': 0.01}. Best is trial 0 with value: 0.186897920331224.\n",
      "[I 2023-11-10 17:33:51,374] Trial 1 finished with value: 0.18130428970784088 and parameters: {'beta': 0.0023960955966723456, 'lambda_': 6.117746940780402e-05}. Best is trial 0 with value: 0.186897920331224.\n",
      "[I 2023-11-10 17:34:05,087] Trial 2 finished with value: 0.19167760516064122 and parameters: {'beta': 0.004819710134847662, 'lambda_': 0.03444850489984713}. Best is trial 2 with value: 0.19167760516064122.\n",
      "[I 2023-11-10 17:34:25,735] Trial 3 finished with value: 0.17566010185645373 and parameters: {'beta': 2.559865011138914e-05, 'lambda_': 0.00011950749449698255}. Best is trial 2 with value: 0.19167760516064122.\n",
      "[I 2023-11-10 17:34:43,953] Trial 4 finished with value: 0.18108660420466682 and parameters: {'beta': 0.0013705061741300481, 'lambda_': 0.0005686324375903997}. Best is trial 2 with value: 0.19167760516064122.\n",
      "10-Nov-23 17:34:43, replay, INFO: best params for SLIM are: {'beta': 0.004819710134847662, 'lambda_': 0.03444850489984713}\n",
      "10-Nov-23 17:34:43, replay, INFO: SLIM fit_predict started\n",
      "10-Nov-23 17:35:02, replay, INFO: Word2Vec started                              \n",
      "10-Nov-23 17:35:02, replay, INFO: Word2Vec optimization started\n",
      "[I 2023-11-10 17:35:02,794] A new study created in memory with name: no-name-781932c3-c815-4c1b-9ec3-d821b3f009ac\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10   fit_time\n",
      "ItemKNN                    0.257599  0.413689     0.058840  16.430850\n",
      "Implicit ALS               0.255083  0.412843     0.147380   4.430974\n",
      "Popular                    0.243711  0.390426     0.033903   3.411652\n",
      "SLIM                       0.237732  0.412639     0.085458   8.316423\n",
      "Wilson                     0.092121  0.180976     0.017092   3.453442\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192   3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609   3.629176\n",
      "Explicit ALS               0.019994  0.045540     0.431213   7.814560\n",
      "UCB                        0.000458  0.001024     0.003082   3.415656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/anaconda3/envs/replay/lib/python3.9/site-packages/pyspark/sql/context.py:125: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
      "  warnings.warn(\n",
      "[I 2023-11-10 17:35:37,137] Trial 0 finished with value: 0.1294242567130421 and parameters: {'rank': 100, 'window_size': 1, 'use_idf': False}. Best is trial 0 with value: 0.1294242567130421.\n",
      "[I 2023-11-10 17:36:21,890] Trial 1 finished with value: 0.03256363722289059 and parameters: {'rank': 87, 'window_size': 36, 'use_idf': True}. Best is trial 0 with value: 0.1294242567130421.\n",
      "[I 2023-11-10 17:37:20,820] Trial 2 finished with value: 0.03485106484524721 and parameters: {'rank': 196, 'window_size': 38, 'use_idf': True}. Best is trial 0 with value: 0.1294242567130421.\n",
      "[I 2023-11-10 17:38:16,795] Trial 3 finished with value: 0.04023855538139732 and parameters: {'rank': 128, 'window_size': 55, 'use_idf': False}. Best is trial 0 with value: 0.1294242567130421.\n",
      "[I 2023-11-10 17:39:27,691] Trial 4 finished with value: 0.03414244744501467 and parameters: {'rank': 156, 'window_size': 81, 'use_idf': True}. Best is trial 0 with value: 0.1294242567130421.\n",
      "10-Nov-23 17:39:27, replay, INFO: best params for Word2Vec are: {'rank': 100, 'window_size': 1, 'use_idf': False}\n",
      "10-Nov-23 17:39:27, replay, INFO: Word2Vec fit_predict started\n",
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            NDCG@10    MRR@10  Coverage@10   fit_time\n",
      "ItemKNN                    0.257599  0.413689     0.058840  16.430850\n",
      "Implicit ALS               0.255083  0.412843     0.147380   4.430974\n",
      "Popular                    0.243711  0.390426     0.033903   3.411652\n",
      "SLIM                       0.237732  0.412639     0.085458   8.316423\n",
      "Word2Vec                   0.138505  0.251616     0.152984   4.772598\n",
      "Wilson                     0.092121  0.180976     0.017092   3.453442\n",
      "Random (popularity-based)  0.074118  0.160990     0.635192   3.927062\n",
      "Random (uniform)           0.026336  0.065413     0.954609   3.629176\n",
      "Explicit ALS               0.019994  0.045540     0.431213   7.814560\n",
      "UCB                        0.000458  0.001024     0.003082   3.415656\n",
      "CPU times: user 9.24 s, sys: 2.14 s, total: 11.4 s\n",
      "Wall time: 13min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "full_pipeline(common_models, e, train_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coverage@10</th>\n",
       "      <th>HitRate@1</th>\n",
       "      <th>HitRate@5</th>\n",
       "      <th>HitRate@10</th>\n",
       "      <th>MAP@10</th>\n",
       "      <th>MRR@10</th>\n",
       "      <th>NDCG@10</th>\n",
       "      <th>Surprisal@10</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>predict_time</th>\n",
       "      <th>metric_time</th>\n",
       "      <th>full_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ItemKNN</th>\n",
       "      <td>0.058840</td>\n",
       "      <td>0.303775</td>\n",
       "      <td>0.555751</td>\n",
       "      <td>0.654083</td>\n",
       "      <td>0.167841</td>\n",
       "      <td>0.413689</td>\n",
       "      <td>0.257599</td>\n",
       "      <td>0.140874</td>\n",
       "      <td>16.430850</td>\n",
       "      <td>8.416453</td>\n",
       "      <td>6.230795</td>\n",
       "      <td>31.078098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Implicit ALS</th>\n",
       "      <td>0.147380</td>\n",
       "      <td>0.302019</td>\n",
       "      <td>0.566286</td>\n",
       "      <td>0.677788</td>\n",
       "      <td>0.162911</td>\n",
       "      <td>0.412843</td>\n",
       "      <td>0.255083</td>\n",
       "      <td>0.170293</td>\n",
       "      <td>4.430974</td>\n",
       "      <td>10.465904</td>\n",
       "      <td>5.813219</td>\n",
       "      <td>20.710098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Popular</th>\n",
       "      <td>0.033903</td>\n",
       "      <td>0.284460</td>\n",
       "      <td>0.530290</td>\n",
       "      <td>0.645303</td>\n",
       "      <td>0.157302</td>\n",
       "      <td>0.390426</td>\n",
       "      <td>0.243711</td>\n",
       "      <td>0.118354</td>\n",
       "      <td>3.411652</td>\n",
       "      <td>8.815796</td>\n",
       "      <td>8.258563</td>\n",
       "      <td>20.486010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SLIM</th>\n",
       "      <td>0.085458</td>\n",
       "      <td>0.298507</td>\n",
       "      <td>0.570676</td>\n",
       "      <td>0.669008</td>\n",
       "      <td>0.143271</td>\n",
       "      <td>0.412639</td>\n",
       "      <td>0.237732</td>\n",
       "      <td>0.154963</td>\n",
       "      <td>8.316423</td>\n",
       "      <td>5.008076</td>\n",
       "      <td>5.497450</td>\n",
       "      <td>18.821949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Word2Vec</th>\n",
       "      <td>0.152984</td>\n",
       "      <td>0.151010</td>\n",
       "      <td>0.385426</td>\n",
       "      <td>0.498683</td>\n",
       "      <td>0.073364</td>\n",
       "      <td>0.251616</td>\n",
       "      <td>0.138505</td>\n",
       "      <td>0.245530</td>\n",
       "      <td>4.772598</td>\n",
       "      <td>23.490503</td>\n",
       "      <td>5.852325</td>\n",
       "      <td>34.115426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wilson</th>\n",
       "      <td>0.017092</td>\n",
       "      <td>0.083406</td>\n",
       "      <td>0.345040</td>\n",
       "      <td>0.414399</td>\n",
       "      <td>0.045002</td>\n",
       "      <td>0.180976</td>\n",
       "      <td>0.092121</td>\n",
       "      <td>0.262190</td>\n",
       "      <td>3.453442</td>\n",
       "      <td>6.875320</td>\n",
       "      <td>6.423975</td>\n",
       "      <td>16.752737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random (popularity-based)</th>\n",
       "      <td>0.635192</td>\n",
       "      <td>0.079017</td>\n",
       "      <td>0.269535</td>\n",
       "      <td>0.399473</td>\n",
       "      <td>0.031447</td>\n",
       "      <td>0.160990</td>\n",
       "      <td>0.074118</td>\n",
       "      <td>0.314159</td>\n",
       "      <td>3.927062</td>\n",
       "      <td>6.529720</td>\n",
       "      <td>6.674805</td>\n",
       "      <td>17.131586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random (uniform)</th>\n",
       "      <td>0.954609</td>\n",
       "      <td>0.032485</td>\n",
       "      <td>0.099210</td>\n",
       "      <td>0.181738</td>\n",
       "      <td>0.009661</td>\n",
       "      <td>0.065413</td>\n",
       "      <td>0.026336</td>\n",
       "      <td>0.537563</td>\n",
       "      <td>3.629176</td>\n",
       "      <td>8.216730</td>\n",
       "      <td>7.108498</td>\n",
       "      <td>18.954404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Explicit ALS</th>\n",
       "      <td>0.431213</td>\n",
       "      <td>0.010536</td>\n",
       "      <td>0.086918</td>\n",
       "      <td>0.168569</td>\n",
       "      <td>0.006395</td>\n",
       "      <td>0.045540</td>\n",
       "      <td>0.019994</td>\n",
       "      <td>0.591741</td>\n",
       "      <td>7.814560</td>\n",
       "      <td>10.279030</td>\n",
       "      <td>5.894241</td>\n",
       "      <td>23.987831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UCB</th>\n",
       "      <td>0.003082</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000878</td>\n",
       "      <td>0.004390</td>\n",
       "      <td>0.000124</td>\n",
       "      <td>0.001024</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.415656</td>\n",
       "      <td>7.359273</td>\n",
       "      <td>5.900311</td>\n",
       "      <td>16.675241</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           Coverage@10  HitRate@1  HitRate@5  HitRate@10  \\\n",
       "ItemKNN                       0.058840   0.303775   0.555751    0.654083   \n",
       "Implicit ALS                  0.147380   0.302019   0.566286    0.677788   \n",
       "Popular                       0.033903   0.284460   0.530290    0.645303   \n",
       "SLIM                          0.085458   0.298507   0.570676    0.669008   \n",
       "Word2Vec                      0.152984   0.151010   0.385426    0.498683   \n",
       "Wilson                        0.017092   0.083406   0.345040    0.414399   \n",
       "Random (popularity-based)     0.635192   0.079017   0.269535    0.399473   \n",
       "Random (uniform)              0.954609   0.032485   0.099210    0.181738   \n",
       "Explicit ALS                  0.431213   0.010536   0.086918    0.168569   \n",
       "UCB                           0.003082   0.000000   0.000878    0.004390   \n",
       "\n",
       "                             MAP@10    MRR@10   NDCG@10  Surprisal@10  \\\n",
       "ItemKNN                    0.167841  0.413689  0.257599      0.140874   \n",
       "Implicit ALS               0.162911  0.412843  0.255083      0.170293   \n",
       "Popular                    0.157302  0.390426  0.243711      0.118354   \n",
       "SLIM                       0.143271  0.412639  0.237732      0.154963   \n",
       "Word2Vec                   0.073364  0.251616  0.138505      0.245530   \n",
       "Wilson                     0.045002  0.180976  0.092121      0.262190   \n",
       "Random (popularity-based)  0.031447  0.160990  0.074118      0.314159   \n",
       "Random (uniform)           0.009661  0.065413  0.026336      0.537563   \n",
       "Explicit ALS               0.006395  0.045540  0.019994      0.591741   \n",
       "UCB                        0.000124  0.001024  0.000458      1.000000   \n",
       "\n",
       "                            fit_time  predict_time  metric_time  full_time  \n",
       "ItemKNN                    16.430850      8.416453     6.230795  31.078098  \n",
       "Implicit ALS                4.430974     10.465904     5.813219  20.710098  \n",
       "Popular                     3.411652      8.815796     8.258563  20.486010  \n",
       "SLIM                        8.316423      5.008076     5.497450  18.821949  \n",
       "Word2Vec                    4.772598     23.490503     5.852325  34.115426  \n",
       "Wilson                      3.453442      6.875320     6.423975  16.752737  \n",
       "Random (popularity-based)   3.927062      6.529720     6.674805  17.131586  \n",
       "Random (uniform)            3.629176      8.216730     7.108498  18.954404  \n",
       "Explicit ALS                7.814560     10.279030     5.894241  23.987831  \n",
       "UCB                         3.415656      7.359273     5.900311  16.675241  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    e.results\n",
    "    .sort_values('NDCG@10', ascending=False)\n",
    "    [[\"Coverage@10\", \"HitRate@1\", \"HitRate@5\", \"HitRate@10\", \"MAP@10\", \"MRR@10\", \"NDCG@10\", \"Surprisal@10\", \"fit_time\", \"predict_time\", \"metric_time\", \"full_time\"]]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e.results.to_csv('res_22_rel_1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best results by quality and time were shown by the commonly-used models such as ItemKNN, ALS, Popular and SLIM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAP@10</th>\n",
       "      <th>NDCG@10</th>\n",
       "      <th>HitRate@1</th>\n",
       "      <th>HitRate@5</th>\n",
       "      <th>HitRate@10</th>\n",
       "      <th>Coverage@10</th>\n",
       "      <th>Surprisal@10</th>\n",
       "      <th>MRR@10</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>predict_time</th>\n",
       "      <th>metric_time</th>\n",
       "      <th>full_time</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ItemKNN</th>\n",
       "      <td>0.167841</td>\n",
       "      <td>0.257599</td>\n",
       "      <td>0.303775</td>\n",
       "      <td>0.555751</td>\n",
       "      <td>0.654083</td>\n",
       "      <td>0.058840</td>\n",
       "      <td>0.140874</td>\n",
       "      <td>0.413689</td>\n",
       "      <td>16.430850</td>\n",
       "      <td>8.416453</td>\n",
       "      <td>6.230795</td>\n",
       "      <td>31.078098</td>\n",
       "      <td>{'shrink': 51, 'use_rating': False, 'num_neigh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Implicit ALS</th>\n",
       "      <td>0.162911</td>\n",
       "      <td>0.255083</td>\n",
       "      <td>0.302019</td>\n",
       "      <td>0.566286</td>\n",
       "      <td>0.677788</td>\n",
       "      <td>0.147380</td>\n",
       "      <td>0.170293</td>\n",
       "      <td>0.412843</td>\n",
       "      <td>4.430974</td>\n",
       "      <td>10.465904</td>\n",
       "      <td>5.813219</td>\n",
       "      <td>20.710098</td>\n",
       "      <td>{'rank': 10, 'implicit_prefs': True, 'seed': 42}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Popular</th>\n",
       "      <td>0.157302</td>\n",
       "      <td>0.243711</td>\n",
       "      <td>0.284460</td>\n",
       "      <td>0.530290</td>\n",
       "      <td>0.645303</td>\n",
       "      <td>0.033903</td>\n",
       "      <td>0.118354</td>\n",
       "      <td>0.390426</td>\n",
       "      <td>3.411652</td>\n",
       "      <td>8.815796</td>\n",
       "      <td>8.258563</td>\n",
       "      <td>20.486010</td>\n",
       "      <td>{'use_rating': False, 'add_cold_items': True, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SLIM</th>\n",
       "      <td>0.143271</td>\n",
       "      <td>0.237732</td>\n",
       "      <td>0.298507</td>\n",
       "      <td>0.570676</td>\n",
       "      <td>0.669008</td>\n",
       "      <td>0.085458</td>\n",
       "      <td>0.154963</td>\n",
       "      <td>0.412639</td>\n",
       "      <td>8.316423</td>\n",
       "      <td>5.008076</td>\n",
       "      <td>5.497450</td>\n",
       "      <td>18.821949</td>\n",
       "      <td>{'beta': 0.004819710134847662, 'lambda_': 0.03...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Word2Vec</th>\n",
       "      <td>0.073364</td>\n",
       "      <td>0.138505</td>\n",
       "      <td>0.151010</td>\n",
       "      <td>0.385426</td>\n",
       "      <td>0.498683</td>\n",
       "      <td>0.152984</td>\n",
       "      <td>0.245530</td>\n",
       "      <td>0.251616</td>\n",
       "      <td>4.772598</td>\n",
       "      <td>23.490503</td>\n",
       "      <td>5.852325</td>\n",
       "      <td>34.115426</td>\n",
       "      <td>{'rank': 100, 'window_size': 1, 'use_idf': Fal...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                MAP@10   NDCG@10  HitRate@1  HitRate@5  HitRate@10  \\\n",
       "ItemKNN       0.167841  0.257599   0.303775   0.555751    0.654083   \n",
       "Implicit ALS  0.162911  0.255083   0.302019   0.566286    0.677788   \n",
       "Popular       0.157302  0.243711   0.284460   0.530290    0.645303   \n",
       "SLIM          0.143271  0.237732   0.298507   0.570676    0.669008   \n",
       "Word2Vec      0.073364  0.138505   0.151010   0.385426    0.498683   \n",
       "\n",
       "              Coverage@10  Surprisal@10    MRR@10   fit_time  predict_time  \\\n",
       "ItemKNN          0.058840      0.140874  0.413689  16.430850      8.416453   \n",
       "Implicit ALS     0.147380      0.170293  0.412843   4.430974     10.465904   \n",
       "Popular          0.033903      0.118354  0.390426   3.411652      8.815796   \n",
       "SLIM             0.085458      0.154963  0.412639   8.316423      5.008076   \n",
       "Word2Vec         0.152984      0.245530  0.251616   4.772598     23.490503   \n",
       "\n",
       "              metric_time  full_time  \\\n",
       "ItemKNN          6.230795  31.078098   \n",
       "Implicit ALS     5.813219  20.710098   \n",
       "Popular          8.258563  20.486010   \n",
       "SLIM             5.497450  18.821949   \n",
       "Word2Vec         5.852325  34.115426   \n",
       "\n",
       "                                                         params  \n",
       "ItemKNN       {'shrink': 51, 'use_rating': False, 'num_neigh...  \n",
       "Implicit ALS   {'rank': 10, 'implicit_prefs': True, 'seed': 42}  \n",
       "Popular       {'use_rating': False, 'add_cold_items': True, ...  \n",
       "SLIM          {'beta': 0.004819710134847662, 'lambda_': 0.03...  \n",
       "Word2Vec      {'rank': 100, 'window_size': 1, 'use_idf': Fal...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.results.sort_values('NDCG@10', ascending=False).head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "name": "movielens_nmf.ipynb",
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": [
     "null"
    ]
   }
  },
  "vscode": {
   "interpreter": {
    "hash": "2e7cb45cb9e942450ddf4e57ba99c929215192dab4b3e5d4b59b5877c1991cae"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}