Practice 02 - How to build a recommender.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2022/23\n",
    "\n",
    "### Practice 2 - Non personalized recommenders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We will use the Movielens 10 million dataset. We download it and uncompress the file we need.\n",
    "\n",
    "### In order to reuse it in the future, we will put all of that in a class that we can call easily"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "import zipfile, os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If file exists, skip the download\n",
    "data_file_path = \"Data_manager_split_datasets/Movielens10M/\"\n",
    "data_file_name = data_file_path + \"movielens_10m.zip\"\n",
    "\n",
    "# If directory does not exist, create\n",
    "if not os.path.exists(data_file_path):\n",
    "    os.makedirs(data_file_path)\n",
    "\n",
    "if not os.path.exists(data_file_name):\n",
    "    urlretrieve (\"http://files.grouplens.org/datasets/movielens/ml-10m.zip\", data_file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataFile = zipfile.ZipFile(data_file_path + \"movielens_10m.zip\")\n",
    "\n",
    "URM_path = dataFile.extract(\"ml-10M100K/ratings.dat\", path = data_file_path + \"decompressed/\")\n",
    "\n",
    "URM_file = open(URM_path, 'r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_io.TextIOWrapper"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(URM_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's take a look at the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "URM_all_dataframe = pd.read_csv(filepath_or_buffer=URM_path, \n",
    "                                sep=\"::\", \n",
    "                                header=None, \n",
    "                                dtype={0:int, 1:int, 2:float, 3:int},\n",
    "                                engine='python')\n",
    "\n",
    "URM_all_dataframe.columns = [\"UserID\", \"ItemID\", \"Interaction\", \"Timestamp\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>ItemID</th>\n",
       "      <th>Interaction</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>122</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838985046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>185</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>231</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>292</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>316</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>329</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>355</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838984474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>356</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>362</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838984885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>364</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983707</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  ItemID  Interaction  Timestamp\n",
       "0       1     122          5.0  838985046\n",
       "1       1     185          5.0  838983525\n",
       "2       1     231          5.0  838983392\n",
       "3       1     292          5.0  838983421\n",
       "4       1     316          5.0  838983392\n",
       "5       1     329          5.0  838983392\n",
       "6       1     355          5.0  838984474\n",
       "7       1     356          5.0  838983653\n",
       "8       1     362          5.0  838984885\n",
       "9       1     364          5.0  838983707"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "URM_all_dataframe.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of interactions is 10000054\n"
     ]
    }
   ],
   "source": [
    "print (\"The number of interactions is {}\".format(len(URM_all_dataframe)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can use this data to create a sparse matrix, notice that we have red UserID and ItemID as int\n",
    "### This is not always possible if the IDs are alphanumeric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we can extract the list of unique user id and item id and display some statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "userID_unique = URM_all_dataframe[\"UserID\"].unique()\n",
    "itemID_unique = URM_all_dataframe[\"ItemID\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of items\t 10677, Number of users\t 69878\n",
      "Max ID items\t 65133, Max Id users\t 71567\n",
      "\n"
     ]
    }
   ],
   "source": [
    "n_users = len(userID_unique)\n",
    "n_items = len(itemID_unique)\n",
    "n_interactions = len(URM_all_dataframe)\n",
    "\n",
    "print (\"Number of items\\t {}, Number of users\\t {}\".format(n_items, n_users))\n",
    "print (\"Max ID items\\t {}, Max Id users\\t {}\\n\".format(max(itemID_unique), max(userID_unique)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### See that the max ID of items and users is higher than the number of unique values -> empty profiles\n",
    "### We should remove empty indices, to do so we create a new mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapped_id, original_id = pd.factorize(URM_all_dataframe[\"UserID\"].unique())\n",
    "user_original_ID_to_index = pd.Series(mapped_id, index=original_id)\n",
    "\n",
    "mapped_id, original_id = pd.factorize(URM_all_dataframe[\"ItemID\"].unique())\n",
    "item_original_ID_to_index = pd.Series(mapped_id, index=original_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New index for item 292 is 3\n"
     ]
    }
   ],
   "source": [
    "original_item_ID = 292\n",
    "print(\"New index for item {} is {}\".format(original_item_ID, item_original_ID_to_index[original_item_ID]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We now replace the IDs in the dataframe and we are ready to use the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "URM_all_dataframe[\"UserID\"] = URM_all_dataframe[\"UserID\"].map(user_original_ID_to_index)\n",
    "URM_all_dataframe[\"ItemID\"] = URM_all_dataframe[\"ItemID\"].map(item_original_ID_to_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>ItemID</th>\n",
       "      <th>Interaction</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838985046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838984474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838984885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983707</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  ItemID  Interaction  Timestamp\n",
       "0       0       0          5.0  838985046\n",
       "1       0       1          5.0  838983525\n",
       "2       0       2          5.0  838983392\n",
       "3       0       3          5.0  838983421\n",
       "4       0       4          5.0  838983392\n",
       "5       0       5          5.0  838983392\n",
       "6       0       6          5.0  838984474\n",
       "7       0       7          5.0  838983653\n",
       "8       0       8          5.0  838984885\n",
       "9       0       9          5.0  838983707"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "URM_all_dataframe.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of items\t 10677, Number of users\t 69878\n",
      "Max ID items\t 10676, Max Id users\t 69877\n",
      "\n",
      "Average interactions per user 143.11\n",
      "Average interactions per item 936.60\n",
      "\n",
      "Sparsity 98.66 %\n"
     ]
    }
   ],
   "source": [
    "userID_unique = URM_all_dataframe[\"UserID\"].unique()\n",
    "itemID_unique = URM_all_dataframe[\"ItemID\"].unique()\n",
    "\n",
    "n_users = len(userID_unique)\n",
    "n_items = len(itemID_unique)\n",
    "n_interactions = len(URM_all_dataframe)\n",
    "\n",
    "print (\"Number of items\\t {}, Number of users\\t {}\".format(n_items, n_users))\n",
    "print (\"Max ID items\\t {}, Max Id users\\t {}\\n\".format(max(itemID_unique), max(userID_unique)))\n",
    "print (\"Average interactions per user {:.2f}\".format(n_interactions/n_users))\n",
    "print (\"Average interactions per item {:.2f}\\n\".format(n_interactions/n_items))\n",
    "\n",
    "print (\"Sparsity {:.2f} %\".format((1-float(n_interactions)/(n_items*n_users))*100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Rating distribution in time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAERCAYAAAB2CKBkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXBklEQVR4nO3df7RlZX3f8fdHfsRUQAgzJcoAg5aCJAUiExSbVAwsAqQrNJEsoCyJBBe1FZLU0kqStbRqs0xXNbVKKGGZKTVJwZBqhFWQJkTAVmgYKj+NkAmITqGLIUSwWpGBb/84e+rhzv2x79yzz6/9fq11Fvfsvc85383c+3zO8+y9n52qQpLUXy+bdAGSpMkyCCSp5wwCSeo5g0CSes4gkKSeMwgkqedmMgiSbE7yZJIHWmx7WJJbktyX5NYkG8ZRoyTNipkMAuBq4LSW234Y+GRVHQN8APhQV0VJ0iyaySCoqtuBp4eXJXltks8luTvJF5Ic1aw6Gril+fnzwJljLFWSpt5MBsESrgIuqarjgUuBK5rl9wJvbX7+GWDfJAdOoD5Jmkp7TrqAUUiyD/Am4LokOxd/X/PfS4HLk7wduB34X8COcdcoSdNqLoKAQc/mG1V13MIVVfU48LPw/wPjrVX1zHjLk6TpNRdDQ1X1LPBokp8DyMCxzc/rkuzcz18BNk+oTEmaSjMZBEmuAe4AjkyyLcmFwHnAhUnuBR7keweFTwIeSvIwcBDw6xMoWZKmVpyGWpL6bSZ7BJKk0Zm5g8Xr1q2rjRs3TroMSZopd99991NVtX6xdTMXBBs3bmTLli2TLkOSZkqSx5Za59CQJPWcQSBJPWcQSFLPGQSS1HMGgST13MydNSRJvfK9iTRfaoQXA9sjkKRptVQIrLRulewRSNI0GWED35ZBIEmTNoHGf5hDQ5I0SRMOAbBHIEmTMQUBsJM9Akkat1GEwAjPGrJHIEnjMqpewIjvI2OPQJLGYUpDAOwRSFK3pmwYaDEGgSR1ZS0hMMbbCDs0JEldmJEQAHsEkjRaMxQAO9kjkKRRmcEQgA6DIMnmJE8meWCJ9eclua95fDHJsV3VIkmdSnY/BKomGgLQbY/gauC0ZdY/Cry5qo4BPghc1WEtktSNGe0FDOvsGEFV3Z5k4zLrvzj09E5gQ1e1SNLIzUEA7DQtxwguBG5aamWSi5JsSbJl+/btYyxLkhYxRyEAUxAESd7CIAjes9Q2VXVVVW2qqk3r168fX3GSNGwtxwJgKkMAJnz6aJJjgE8Ap1fVX02yFkla1hwGwE4T6xEkORT4NPC2qnp4UnVI0ormOASgwx5BkmuAk4B1SbYB7wP2AqiqK4H3AgcCV2TwP3lHVW3qqh5J2i1rOS10RnR51tC5K6x/B/COrj5fktZkznsBw5xiQpKG9SgAdpr4WUOSNDV6GAJgj0CSBnpwLGApBoGkfutpL2CYQSCpn9Z657A5CQEwCCT1zQzcOnLcDAJJ/WAALMkgkDTfDIAVefqopPllCLRij0DS/BlFAEAvQgAMAknzxl7AqhkEkuaDAbDbDAJJs89rAtbEIJA0uwyAkTAIJM0mp4YYGYNA0myxFzByBoGk2WAAdMYLyiRNP0OgU/YIJE0vA2AsDAJJ08drAsbKIJA0PQyAiTAIJE2ecwNNlEEgaXIMgKlgEEiaDIeBpoZBIGm87AVMHYNA0ngYAFPLIJDULQNg6hkEkrphAMwMg0DS6Iyq8QcDYIwMAklrZwDMNINA0uqMstFfyBCYCINA0uK6bPAXMgAmyiCQ9FIGQO8YBJLG2/iDATBlDAKp7+wB9J5BIPXVuALAxn/qdXaryiSbkzyZ5IEl1h+V5I4kzyW5tKs6JC1iHCFQZQjMiC7vWXw1cNoy658GfhH4cIc1SBqWdBMCOxv94YdmRmdBUFW3M2jsl1r/ZFXdBTzfVQ2SGqMKgMUafBv9mddlj2BkklyUZEuSLdu3b590OdJsGWUAaC7NRBBU1VVVtamqNq1fv37S5UizYa29AL/x98ZMBIGkVVprL8DGv1c8fVSaJwaAdkNnQZDkGuAkYF2SbcD7gL0AqurKJD8IbAH2A15M8svA0VX1bFc1SXPL+/9qDToLgqo6d4X1/xvY0NXnS71hL0Br5NCQNKsMAI2IQSDNGoeBNGIGgTQrDAB1xCCQpp03gVfHDAJpmtkL0BgYBNK08mCwxsQgkKaNvQCNmUEgTRN7AZoAg0CaFmudIE7aTQaBNGkGgCZsxdlHk7wmyQ1JnmpuPfnZJK8ZR3HS3NvdEHB6aI1Qm2mo/xPwB8APAq8GrgOu6bIoae6t5V4BBoBGrE0QpKp+t6p2NI/fA/xNlHaXQ0GaMm2OEXw+yWXAtQwC4GzgvyT5AYCqWvK+xJKGGACaUm2C4Ozmv/9owfJfYBAMHi+QVmIIaIqtGARVdfg4CpHmktcFaAasGARJ9gB+Ctg4vH1V/WZ3ZUkzzgDQDGkzNHQD8B3gfuDFbsuR5oAhoBnTJgg2VNUxnVcizToDQDOqzemjNyU5tfNKpFlmCGiGtekR3Al8JsnLgOeBAFVV+3VamTQrPCNIM65NEHwEOBG4v8rfWuklDAHNgTZB8BfAA4aAtIBTRGhOtAmCJ4Bbk9wEPLdzoaePqtd2JwQMAE2pNkHwaPPYu3lI/WUvQHOozZXF7x9HIdLUsxegOdXmyuL1wL8Afgh4+c7lVfUTHdYlTRdDQHOszXUEvw98BTgceD/wVeCuDmuSposhoDnXJggOrKrfAZ6vqtuq6heAN3ZclzQdDAH1QJuDxc83/30iyU8BjwMbuitJmgIeFFaPtAmCf5XklcA/Az4O7Af8cpdFSRNlCKhn2gTBX1fVM8AzwFsAkvzdTquSJsEAUE+1OUbw8ZbLpNllCKjHluwRJDkReBOwPsm7h1btB+zRdWHSWDhXkLTs0NDewD7NNvsOLX8WOKvLoqSxMAQkYJkgqKrbgNuSXF1VjwE0U1HvU1XPjqtAaeS8d4D0Em2OEXwoyX5JXgF8GXgoyT9f6UVJNid5MskDS6xPko8l2ZrkviSvX2Xt0uqttRdgCGgOtQmCo5sewD8AbgQOBd7W4nVXA6cts/504IjmcRHw71u8p7R7EoeCpCW0CYK9kuzFIAg+W1XPAyv+VVTV7cDTy2xyJvDJGrgT2D/Jq1rUI7Wzs/G3FyAtq00Q/DaD+YVeAdye5DAGB4zX6mDg60PPtzXLdpHkoiRbkmzZvn37CD5ac2sUjf9OBoB6YsUgqKqPVdXBVXVG8+39MZoLy9Zosb/URf/yquqqqtpUVZvWr18/go/WXBpF4w/2AtQ7KwZBkoOS/E5zhzKSHA38/Ag+extwyNDzDQzmMZJWb5QhIPVMm6Ghq4GbgVc3zx9mNHMNXQ+c35w99Ebgmap6YgTvq74Z1TCQIaCeajPX0Lqq+oMkvwJQVTuSvLDSi5JcA5wErEuyDXgfsFfzHlcyOAPpDGAr8G3ggt3aA/WbxwKkNWsTBN9KciDN+P3Ob+8rvaiqzl1hfQHvalOktAsDQBqZNkHwbgbDOK9N8t+B9TjFhCbFq4KlkWtz8/r/meTNwJEMzvR5qLmWQBoPG3+pU21uXr8Hg7H8jc32pyahqn6z49rUF6M640fSbmkzNHQD8B3gfuDFbstR73QdAvYGpBW1CYINVXVM55Wof7oMAQNAaq3NdQQ3JTm180rUH6OaAmIphoC0Km16BHcCn2nuRfA8gwPGVVX7dVqZ5sM4x/8NAGm3tAmCjwAnAvc35/5LKzMApJnRJgj+AnjAEFAr4woAfx2lkWkTBE8AtzaTzj23c6Gnj2oXowwBG3ppbNoEwaPNY+/mIX1PFz0AQ0AaqzZXFr9/HIVohnjGjzRXlgyCJJdX1cVJbmCRG8ZU1U93WpmmjwEgzaXlegTnAxcDHx5TLZpGDv1Ic2+5IPhLgKq6bUy1aJoYAFJvLBcE65O8e6mVnjU0w8Y9yZsBIE215YJgD2AfFr/JvGbNpGb4NASkqbdcEDxRVR8YWyUanUlP62zjL82U5YLAnsAsmHSjv5ONvzSzlguCk8dWhXbPNISAASDNvCWDoKqeHmchmiE2/tJcaTPFhKbRJHoDBoA0lwwCDdjIS71lEPSZjb8kDIL+sNGXtASDYF7Z8Etqqc3N6zVtVjpQbAhIWgWDQJJ6ziCQpJ4zCCSp5wyCWTMN00pImisGwbzxQLGkVTIIZom9AUkdMAgkqecMAknquU6DIMlpSR5KsjXJZYusPyDJZ5Lcl+TPkvxwl/XMrKTdsJDHByTths6CIMkewG8BpwNHA+cmOXrBZr8K3FNVxwDnA/+uq3pmlscFJHWsyx7BCcDWqnqkqr4LXAucuWCbo4FbAKrqK8DGJAd1WNP8sjcgaTd1GQQHA18fer6tWTbsXuBnAZKcABwGbOiwJknSAl0GwWJjGgu/tv4GcECSe4BLgC8BO3Z5o+SiJFuSbNm+ffvIC5WkPutyGuptwCFDzzcAjw9vUFXPAhcAJAnwaPNgwXZXAVcBbNq0qT9jIG2PDzgsJGkNuuwR3AUckeTwJHsD5wDXD2+QZP9mHcA7gNubcFBbhoCkNeqsR1BVO5JcDNwM7AFsrqoHk7yzWX8l8Drgk0leAL4MXNhVPXPJEJA0Ap3eoayqbgRuXLDsyqGf7wCO6LIGSdLyvLJ4Wnn9gKQxMQhmlcNCkkbEIJhG9gYkjZFBIEk9ZxBMG3sDksbMIJhFHh+QNEIGwTSxNyBpAgyCaeF0EpImpNMLytSCvQBJE2aPYJJWGwL2BiR1wB7BJNgLkDRF7BGMmyEgacrYIxiXtQaAw0KSOmIQdM0AkDTlDIIujGr4xxCQNAYGwaiMcuzfAJA0RgbBWnRx4NcQkDRmBsHuMAAkzRGDoI2uT/k0BCRNkEGwlHGc728ASJoCBgGM/yIvA0DSFOlvENj4SxLQtyCw8ZekXfQnCMYVAjb+kmZMP4LAA7+StKT5D4KuQsCGX9KcmP8gGCUbf0lzyCBYio2+pJ4wCIbZ+EvqIYPAxl9Sz/X7VpWGgCT1PAgkST0IgqW+9dsbkCSgL8cIbPQlaUnz3yOQJC3LIJCknus0CJKcluShJFuTXLbI+lcmuSHJvUkeTHJBl/VIknbVWRAk2QP4LeB04Gjg3CRHL9jsXcCXq+pY4CTgI0n27qomSdKuuuwRnABsrapHquq7wLXAmQu2KWDfJAH2AZ4GdnRYkyRpgS6D4GDg60PPtzXLhl0OvA54HLgf+KWqenHhGyW5KMmWJFu2b9/eVb2S1EtdBsFi8z8vPI/zJ4F7gFcDxwGXJ9lvlxdVXVVVm6pq0/r160ddpyT1WpdBsA04ZOj5Bgbf/IddAHy6BrYCjwJHjbySU04Z3Jdg5+OUU0b+EZI0q7oMgruAI5Ic3hwAPge4fsE2XwNOBkhyEHAk8MhIqzjlFLjllpcuu+UWw0CSGp1dWVxVO5JcDNwM7AFsrqoHk7yzWX8l8EHg6iT3MxhKek9VPTXSQhaGwErLJalnOp1ioqpuBG5csOzKoZ8fB07tsgZJ0vK8sliSem7+g+Dkk1e3XJJ6Zv6D4E/+ZNdG/+STB8slST2ZhtpGX5KWNP89AknSsgwCSeo5g0CSes4gkKSeMwgkqedSM3Zj9yTbgcd28+XrgNFOYTH93Od+cJ/7YS37fFhVLTp988wFwVok2VJVmyZdxzi5z/3gPvdDV/vs0JAk9ZxBIEk917cguGrSBUyA+9wP7nM/dLLPvTpGIEnaVd96BJKkBQwCSeq5uQyCJKcleSjJ1iSXLbI+ST7WrL8vyesnUecotdjn85p9vS/JF5McO4k6R2mlfR7a7keTvJDkrHHW14U2+5zkpCT3JHkwyW3jrnHUWvxuvzLJDUnubfb5gknUOSpJNid5MskDS6wffftVVXP1YHB/5L8EXgPsDdwLHL1gmzOAmxjcJ/mNwP+YdN1j2Oc3AQc0P5/eh30e2u5PGdwy9axJ1z2Gf+f9gS8DhzbP/+ak6x7DPv8q8K+bn9cDTwN7T7r2Nezz3wNeDzywxPqRt1/z2CM4AdhaVY9U1XeBa4EzF2xzJvDJGrgT2D/Jq8Zd6AituM9V9cWq+uvm6Z3AhjHXOGpt/p0BLgH+M/DkOIvrSJt9/ofAp6vqawBVNev73WafC9g3SYB9GATBjvGWOTpVdTuDfVjKyNuveQyCg4GvDz3f1ixb7TazZLX7cyGDbxSzbMV9TnIw8DPAlWOsq0tt/p3/NnBAkluT3J3k/LFV1402+3w58DrgceB+4Jeq6sXxlDcRI2+/5vEOZVlk2cJzZNtsM0ta70+StzAIgh/rtKLutdnnjwLvqaoXBl8WZ16bfd4TOB44Gfh+4I4kd1bVw10X15E2+/yTwD3ATwCvBf44yReq6tmOa5uUkbdf8xgE24BDhp5vYPBNYbXbzJJW+5PkGOATwOlV9Vdjqq0rbfZ5E3BtEwLrgDOS7KiqPxpLhaPX9nf7qar6FvCtJLcDxwKzGgRt9vkC4DdqMIC+NcmjwFHAn42nxLEbefs1j0NDdwFHJDk8yd7AOcD1C7a5Hji/Ofr+RuCZqnpi3IWO0Ir7nORQ4NPA22b42+GwFfe5qg6vqo1VtRH4Q+CfzHAIQLvf7c8CP55kzyR/A3gD8OdjrnOU2uzz1xj0gEhyEHAk8MhYqxyvkbdfc9cjqKodSS4GbmZwxsHmqnowyTub9VcyOIPkDGAr8G0G3yhmVst9fi9wIHBF8w15R83wzI0t93mutNnnqvrzJJ8D7gNeBD5RVYuehjgLWv47fxC4Osn9DIZN3lNVMzs9dZJrgJOAdUm2Ae8D9oLu2i+nmJCknpvHoSFJ0ioYBJLUcwaBJPWcQSBJPWcQSNIUW2kSugXb/ttmwsF7kjyc5BttPsMg0NxK8mvNbJT3NX8Yb1jl69+e5NWrfM3Gxf5gh5cnOS7JGat5X/Xa1cBpbTasqn9aVcdV1XHAxxlcO7Qig0BzKcmJwN8HXl9VxwCn8NL5WVZ6/R7A24FVBUFLxzE4D1xa0WKT0CV5bZLPNfNJfSHJUYu89FzgmjafYRBoXr2KwVQLzwFU1VNV9ThAkpOTfCnJ/U23+/ua5V9N8t4k/43BH9Em4Peb3sT3Jzk+yW3NH9/NO2d8bJbfm+QO4F3LFdVcHfsB4Ozmfc9O8oqmjruaus5stn17kj/KYK79R5NcnOTdzTZ3JvmBjv7fafpdBVxSVccDlwJXDK9MchhwOIMp2FdkEGhe/VfgkGac9IokbwZI8nIGXe2zq+rvMLi6/h8Pve47VfVjVfV7wBbgvKabvYNBV/us5o9vM/DrzWv+A/CLVXXiSkU1Uym/F/hU04X/FPBrwJ9W1Y8CbwH+TZJXNC/5YQZTS5/QfN63q+pHgDuAWZ9ZVLshyT4M7i9yXZJ7gN9m8MVn2DnAH1bVC23ec+6mmJAAqur/JDke+HEGjeunMri71ZeAR4fmW/qPDL7Ff7R5/qkl3vJIBo3yHzdTdOwBPJHklcD+VbXzTmC/y+DGP6txKvDTSS5tnr8cOLT5+fNV9U3gm0meAW5olt8PHLPKz9F8eBnwjeYLylLOYYXe6TCDQHOr+TZ0K3BrMw/NzzOYrng531pieYAHF37rT7I/a5/CPMBbq+qhBe/9BuC5oUUvDj1/Ef9+e6mqnm2GCn+uqq7L4JvJMVV1L0CSI4EDGPQaW3FoSHMpyZFJjhhadBzwGPAVYGOSv9Usfxuw1H19vwns2/z8ELC+OQhNkr2S/FBVfQN4JsnO+zuc16K84feFwYRqlzR/0CT5kRbvoZ5oJqG7AzgyybYkFzL4Pbswyb3Ag7z0rm3nAtfWKiaS8xuF5tU+wMebb+w7GMzUeFFVfSeDm5tfl2RPBtMcLzVT6dXAlUn+L3AicBbwsWY4aE8Gw0kPMpj9cXOSbzNo1FfyeeCyZnz3Qwxmz/wocF8TBl9lcMaTRFWdu8SqRU8prap/udrPcPZRSeo5h4YkqecMAknqOYNAknrOIJCknjMIJKnnDAJJ6jmDQJJ67v8BxAzc0dUEHhIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as pyplot\n",
    "\n",
    "# Clone the list to avoid changing the ordering of the original data\n",
    "timestamp_sorted = URM_all_dataframe[\"Timestamp\"].values.copy()\n",
    "timestamp_sorted.sort()\n",
    "\n",
    "\n",
    "pyplot.plot(timestamp_sorted, 'ro')\n",
    "pyplot.ylabel('Timestamp ')\n",
    "pyplot.xlabel('Sorted Item')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To store the data we use a sparse matrix. We build it as a COO matrix and then change its format\n",
    "\n",
    "#### The COO constructor expects (data, (row, column))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<69878x10677 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 10000054 stored elements in COOrdinate format>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import scipy.sparse as sps\n",
    "\n",
    "URM_all = sps.coo_matrix((URM_all_dataframe[\"Interaction\"].values, \n",
    "                          (URM_all_dataframe[\"UserID\"].values, URM_all_dataframe[\"ItemID\"].values)))\n",
    "\n",
    "URM_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<69878x10677 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 10000054 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "URM_all.tocsr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We compute the item popularity as the number of interaction in each column\n",
    "\n",
    "### We can use the properties of sparse matrices in CSC format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2412, 14975, 17851, ...,     1,     1,     1], dtype=int32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "item_popularity = np.ediff1d(URM_all.tocsc().indptr)\n",
    "item_popularity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([    1,     1,     1, ..., 33668, 34457, 34864], dtype=int32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_popularity = np.sort(item_popularity)\n",
    "item_popularity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEGCAYAAABPdROvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgt0lEQVR4nO3df/RVdZ3v8eeLHyKaiOhXFwIKJqs1aCuU7xBOrXtNnGRcTdiMTRQluphLqZX9WPcmOavsdlsrp1vebAaL0kSlhMxJcmHloFndy4BfGhRBSeYK+hWuYpqSJgq87x/7c+Tw9XzPd5/f33O+r8dae5193nt/9v58BM+bz96f/dmKCMzMzKo1rNUVMDOz9uZEYmZmNXEiMTOzmjiRmJlZTZxIzMysJiNaXYFmO+6442Ly5MmtroaZWVvZsGHDsxHRVWrbkEskkydPpqenp9XVMDNrK5J29LfNl7bMzKwmTiRmZlYTJxIzM6uJE4mZmdWkYYlE0uGS1kt6UNJmSV9K8aslPSVpY1rOLyqzWNI2SVslnVcUnyFpU9p2nSSl+ChJK1J8naTJjWqPmZmV1sgeyV7gnIh4GzAdmCNpVtp2bURMT8tqAEnTgHnAacAcYImk4Wn/64FFwNS0zEnxhcDzEXEqcC1wTQPbY2bWnpYvh8mTYdiw7HP58roevmGJJDJ/TF9HpqXcVMNzgdsiYm9EPA5sA2ZKGg+MiYi1kU1VfDNwQVGZZWn9dmB2obdiZmZkSWPRItixAyKyz0WL6ppMGnqPRNJwSRuBZ4B7ImJd2vRxSQ9JulHSMSk2AXiyqHhvik1I633jh5SJiH3AC8CxJeqxSFKPpJ7du3fXp3FmZu3gqqvg5ZcPjb38chavk4YmkojYHxHTgYlkvYvTyS5TvZnsctcu4Otp91I9iSgTL1embz2WRkR3RHR3dZV8MNPMrDM98URl8So0ZdRWRPwB+CUwJyKeTgnmAPBdYGbarReYVFRsIrAzxSeWiB9SRtII4Gjguca0wsysDY0bV1m8Co0ctdUlaWxaHw2cCzya7nkUvA94OK2vAualkVhTyG6qr4+IXcAeSbPS/Y+LgDuLyixI6xcC94Zf+Whm1lSNnGtrPLAsjbwaBqyMiLsk3SJpOtklqO3ARwEiYrOklcAWYB9weUTsT8e6FLgJGA3cnRaAG4BbJG0j64nMa2B7zMzaz3P9XKTpL14FDbV/wHd3d4cnbTSzIWPy5GykVl8nnwzbt+c+jKQNEdFdapufbDcz62Tnn19ZvApOJGZmnWz16sriVXAiMTPrZJ0y/NfMzFrkpJMqi1fBicTMrJN95StwxBGHxo44IovXiROJmVknmz8fFiyA4WkO3OHDs+/z59ftFE4kZmadbPlyWLYM9qfH8vbvz763y6SNZmbWYu0+aaOZmbWYR22ZmVlNPGrLzMxq4lFbZmZWk/nzYenSbG4tKftcurSuo7YaOfuvmZkNBvPn1zVx9OUeiZlZp1u+PJsFeNiw7LOOQ3/BPRIzs862fDksWnRwCPCOHdl3qFsvxT0SM7NO5udIzMysJn6OxMzMauLnSMzMrCZ+jsTMzGrShOdIGpZIJB0uab2kByVtlvSlFB8n6R5Jj6XPY4rKLJa0TdJWSecVxWdI2pS2XSdJKT5K0ooUXydpcqPaY2bWtubPh+3b4cCB7LPOz5Q0skeyFzgnIt4GTAfmSJoFXAmsiYipwJr0HUnTgHnAacAcYImkNIE+1wOLgKlpmZPiC4HnI+JU4Frgmga2x8zMSmhYIonMH9PXkWkJYC6wLMWXARek9bnAbRGxNyIeB7YBMyWNB8ZExNqICODmPmUKx7odmF3orZiZWXM09B6JpOGSNgLPAPdExDrghIjYBZA+j0+7TwCeLCrem2IT0nrf+CFlImIf8AJwbIl6LJLUI6ln9+7ddWqdmZlBgxNJROyPiOnARLLexelldi/Vk4gy8XJl+tZjaUR0R0R3V1fXALU2M7NKNGXUVkT8Afgl2b2Np9PlKtLnM2m3XmBSUbGJwM4Un1gifkgZSSOAo4HnGtEGM7O21eC5tho5aqtL0ti0Pho4F3gUWAUsSLstAO5M66uAeWkk1hSym+rr0+WvPZJmpfsfF/UpUzjWhcC96T6KmZkBnHsufPjD2RxbEQfn2qpjMmnkpI3jgWVp5NUwYGVE3CVpLbBS0kLgCeD9ABGxWdJKYAuwD7g8ItLb6rkUuAkYDdydFoAbgFskbSPricxrYHvMzNrLZZfBmjVvjBfm2qrTMGANtX/Ad3d3R09PT6urYWbWeCNGwP79pbdJ2XMlOUnaEBHdpbb5yXYzs07VXxIBz7VlZmY18lxbZmZWk3aYa8vMzFrs5JMri1fJicTMrFM1YQp5cCIxM+tcTZhCHhr7HImZmbXa/Pl1Txx9uUdiZmY1cSIxM+tkl12WPZgoZZ+XXVb3U/jSlplZp7rsMrj++oPf9+8/+H3Jkrqdxj0SM7NOtXRpZfEqOZGYmXWq/qZIKTd1ShWcSMzMOtWwfn7i+4tXe5q6Hs3MzAaP0aMri1fJicTMrFO99FJl8So5kZiZdSpf2jIzs5r09+KqCl5olYcTiZmZ1cSJxMzMatKwRCJpkqT7JD0iabOkK1L8aklPSdqYlvOLyiyWtE3SVknnFcVnSNqUtl0nSSk+StKKFF8naXKj2mNm1nayn8r88So1skeyD/hsRPwZMAu4XNK0tO3aiJieltUAads84DRgDrBE0vC0//XAImBqWuak+ELg+Yg4FbgWuKaB7TEzay8RlcWr1LBEEhG7IuK3aX0P8AgwoUyRucBtEbE3Ih4HtgEzJY0HxkTE2ogI4GbggqIyy9L67cDsQm/FzGzIGz68sniVmnKPJF1yOgNYl0Ifl/SQpBslHZNiE4Ani4r1ptiEtN43fkiZiNgHvAAcW+L8iyT1SOrZvXt3fRplZjbYdcoUKZLeBPwY+FREvEh2merNwHRgF/D1wq4likeZeLkyhwYilkZEd0R0d3V1VdYAM7N21QnvbJc0kiyJLI+IOwAi4umI2B8RB4DvAjPT7r3ApKLiE4GdKT6xRPyQMpJGAEcDzzWmNWZmbeb88yuLV2nARCLpzZJGpfWzJX1S0tgc5QTcADwSEd8oio8v2u19wMNpfRUwL43EmkJ2U319ROwC9kialY55EXBnUZkFaf1C4N50H8XMzFavrixepTwvtvox0C3pVLLEsAr4ATBQSnsH8BFgk6SNKfZ54IOSppNdgtoOfBQgIjZLWglsIRvxdXlEFC7kXQrcBIwG7k4LqT63SNpG1hOZl6M9ZmZDw44dlcWrpIH+AS/ptxFxpqT/CrwSEd+S9O8RcUZda9Ik3d3d0dPT0+pqmJk13rBhpYf6ShVPkyJpQ0R0lzxNjvKvSfog2SWku1JsZEU1MDOz5htEz5FcApwFfCUiHk/3L26tay3MzKxtDXiPJCK2AJ8s+v448NVGVsrMzNrHgIlE0juAq4GT0/4CIiJOaWzVzMysHeQZtXUD8GlgA1DfxyHNzKzt5UkkL0TE3QPvZmZmQ1GeRHKfpK8BdwB7C8HChIxmZjZIHXlk6fezH3lkXU+TJ5G8PX0Wjx8O4Jy61sTMzOrr8MNLJ5LDD6/rafKM2npXXc9oZmbN8Vw/Uw/2F69Snrm2jpb0jcI07JK+LunoutbCzMzqb9y4yuJVyvNA4o3AHuDv0vIi8P261sLMzOrvlVcqi1cpzz2SN0fE3xZ9/1LRJIxmZjZYlbo/Ui5epTw9kj9JemfhS3pA8U91rYWZmbWtPD2SS4Fl6b6IyKZrv7iRlTIzszoYLMN/I2Ij8DZJY9L3F+taAzMza2v9JhJJH46IWyV9pk8cgOK3HpqZ2SDUpHsk5Xokhb7PUSW2+XW2ZmYGlEkkEfGdtPqvEfG/i7elG+5mZma5Rm19K2fMzMwGk3QrIne8Sv0mEklnSfos0CXpM0XL1cDwgQ4saZKk+yQ9ImmzpCtSfJykeyQ9lj6PKSqzWNI2SVslnVcUnyFpU9p2ndKNGkmjJK1I8XWSJlf/n8LMrMN87GOVxatUrkdyGPAmsstfRxUtLwIX5jj2PuCzEfFnwCzgcknTgCuBNRExFViTvpO2zQNOA+YASyQVEtb1wCJgalrmpPhC4PmIOBW4FrgmR73MzIaGJUtg9uxDY7NnZ/E6KneP5H7gfkk3RcSOSg8cEbuAXWl9j6RHgAnAXODstNsy4JfA51L8tojYCzwuaRswU9J2YExErAWQdDNwAXB3KnN1OtbtwD9JUkSd32xvZtaOli+HX//60Nivf53F58+v22ny3CP5nqSxhS+SjpH080pOki45nQGsA05ISaaQbI5Pu00Aniwq1ptiE9J63/ghZSJiH/ACcGwldTMz61hXXAGvvnpo7NVXs3gd5Ukkx0XEHwpfIuJ5Dv74D0jSm4AfA58a4GHGUnd/oky8XJm+dVhUmL149+7dA1XZzKwz/P73lcWrlCeRHJB0UuGLpJPJ+RyJpJFkSWR5RNyRwk9LGp+2jweeSfFeYFJR8YnAzhSfWCJ+SBlJI4CjyaZwOURELI2I7ojo7urqylN1MzPLKU8iuQr4jaRbJN0C/ApYPFChNLLqBuCRPk/BrwIWpPUFwJ1F8XlpJNYUspvq69Plrz2SZqVjXtSnTOFYFwL3+v6ImVlz5Zlr62eSziQbeSXg0xHxbI5jvwP4CLCpaNr5zwNfBVZKWgg8Abw/nWezpJXAFrIRX5dHxP5U7lLgJmA02U32u1P8BuCWdGP+ObJRX2Zm1kR5Zv8F2E92CepwYJokIuJX5QpExG8ofQ8DYHapYER8BfhKiXgPcHqJ+CukRGRmZkUuu6xppxowkUj6e+AKsnsTG8l6JmuBcxpaMzMzq97SpU07VZ57JFcAfw7siIh3kQ3j9dAnM7PBbP/+gfepkzyJ5JV0CQlJoyLiUeAtja2WmZk1TJ3n2spzj6Q3PZD4E+AeSc9zcPitmZm1mzrPtZVn1Nb70urVku4je1bjZ3WthZmZNU+z5toCkDQMeCgiTofX598yMzN7Xdl7JBFxAHiw+Ml2MzOzYnnukYwHNktaD7z+ot+IeG/DamVmZm0jTyL5UsNrYWZmbSvPzfb700SNUyPiXyUdQY43JJqZ2dAw4HMkkv4L2UujvpNCE8iGApuZmeV6IPFysgkYXwSIiMeo4H0kZmbW2fIkkr0R8forttJ7PzxVu5mZAfkSyf2SPg+MlvSXwI+Anza2WmZm1i7yJJIrySZp3AR8FFgdEVc1tFZmZtY28gz//UREfBP4biEg6YoUMzOzIS5Pj2RBidjFda6HmZm1qX57JJI+CHwImCJpVdGmo4DfN7piZmbWHspd2vo/wC7gOODrRfE9wEONrJSZmbWPfhNJROwAdgBnNa86ZmbWbvI82f43kh6T9IKkFyXtkfRijnI3SnpG0sNFsaslPSVpY1rOL9q2WNI2SVslnVcUnyFpU9p2nZS92kvSKEkrUnydpMkVt97MzGqW52b7PwLvjYijI2JMRBwVEWNylLsJmFMifm1ETE/LagBJ04B5wGmpzBJJhfm8rgcWAVPTUjjmQuD5iDgVuBa4JkedzMw63/LlTT1dnkTydEQ8UumBI+JXwHM5d58L3BYReyPicWAbMFPSeGBMRKyNiABuBi4oKrMsrd8OzC70VszMhrQrrmjq6fI8R9IjaQXZRI17C8GIuKPKc35c0kVAD/DZiHiebCLIfyvapzfFXkvrfeOkzydTXfZJegE4Fni27wklLSLr1XDSSX5Hl5l1uN83d2Btnh7JGOBl4N3AX6flPVWe73rgzcB0shFhhdFgpXoSUSZerswbgxFLI6I7Irq7uroqqrCZWUc59ti6HzLP+0guqdfJIuLpwrqk7wJ3pa+9wKSiXScCO1N8Yol4cZneNJHk0eS/lGZmNjR9s/6TkpR7IPFblJnlNyI+WenJJI2PiF3p6/uAwoiuVcAPJH0DOJHspvr6iNifRonNAtYBFwHfKiqzAFgLXAjcm+6jmJlZf+bPr/shy/VIemo5sKQfAmcDx0nqBb4InC1pOlmC2k42CSQRsVnSSmALsA+4PCL2p0NdSjYCbDRwd1oAbgBukbSNrCcyr5b6mplZdTTU/hHf3d0dPT015Ugzs8Gt3ADWKn/zJW2IiO5S2/LcbDczM+uXE4mZmdXEicTMzGoy4PBfSVOATwCTi/ePiPc2rlpmZtYu8jzZ/hOyEVI/BQ40tDZmZtZ28iSSVyLiuobXxMzM2lKeRPJNSV8EfsGhc239tmG1MjOz6jR55l/Il0jeCnwEOIeDl7YifTczs8HkYx9r+inzJJL3AadExKuNroyZmdXoj39s+inzDP99EBjb4HqYmVmjHXlkQw6bp0dyAvCopAc49B6Jh/+ambWT73ynIYfNk0i+2JAzm5lZczVg5l/I9z6S+xtyZjMz6wh5nmzfw8H3khwGjAReiogxjayYmZm1hzw9kqOKv0u6AJjZqAqZmVl7qXjSxoj4CX6GxMxs8DnssJacNs+lrb8p+joM6KbMK3jNzKxFXnutJafNM2rrr4vW95G9InduQ2pjZmZtJ889kkuaUREzM2ugadMaduh+E4mkL5QpFxHx5XIHlnQj8B7gmYg4PcXGASvI3m2yHfi7iHg+bVsMLAT2A5+MiJ+n+AzgJmA0sBq4IiJC0ijgZmAG8HvgAxGxvXxzzcyGqM2bG3bocjfbXyqxQPZj/7kcx74JmNMndiWwJiKmAmvSdyRNA+YBp6UySyQNT2WuBxYBU9NSOOZC4PmIOBW4FrgmR53MzKzO+k0kEfH1wgIsJesRXALcBpwy0IEj4lfAc33Cc4FlaX0ZcEFR/LaI2BsRjwPbgJmSxgNjImJtRARZD+SCEse6HZgtSQPVy8zM6qvs8F9J4yT9D+AhsstgZ0bE5yLimSrPd0JE7AJIn8en+ATgyaL9elNsQlrvGz+kTETsA14Aju2nHYsk9Ujq2b17d5VVNzMbxFr47+h+E4mkrwEPAHuAt0bE1YX7GQ1Q6r9AlImXK/PGYMTSiOiOiO6urq4qq2hmZqWU65F8FjgR+Adgp6QX07JH0otVnu/pdLmK9Fno2fQCk4r2mwjsTPGJJeKHlJE0AjiaN15KMzOzBit3j2RYRIyOiKMiYkzRclQN82ytAhak9QXAnUXxeZJGSZpCdlN9fbr8tUfSrHT/46I+ZQrHuhC4N91HMTOzYqNHN/TweR5IrIqkHwJnA8dJ6iWbjv6rwEpJC4EngPcDRMRmSSuBLWQPPV4eEfvToS7l4PDfu9MCcANwi6RtZD2ReY1qi5lZW3v55YYeXkPtH/Hd3d3R09PT6mqYmdXPEUfAn/7U//Y6/M5L2hAR3aW2VTxpo5mZDTLlkkgTOJGYmVlNnEjMzKwmTiRmZu1swoTy2xs4WWOBE4mZWTvbubP89gZO1ljgRGJmZjVxIjEza1eDZJ5aJxIzs07V4CfaC5xIzMw6VYOfaC9wIjEza0eD5LIWOJGYmXWmsWObdionEjOzTvR8o14f9UZOJGZm7Wb48FbX4BBOJGZm7ebAgfLbmzyruxOJmZnVxInEzKydHHFEq2vwBk4kZmbtZKB3j1x6aXPqUcSJxMyskyxZ0vRTOpGYmVlNWpJIJG2XtEnSRkk9KTZO0j2SHkufxxTtv1jSNklbJZ1XFJ+RjrNN0nXSIHrU08ys3gbpT1wreyTviojpRS+TvxJYExFTgTXpO5KmAfOA04A5wBJJhUHU1wOLgKlpmdPE+puZNc9hhw28T5OH/RYMpktbc4FlaX0ZcEFR/LaI2BsRjwPbgJmSxgNjImJtRARwc1EZM7PO8tprra5Bv1qVSAL4haQNkhal2AkRsQsgfR6f4hOAJ4vK9qbYhLTeN25m1lkGep0uwMiRja9HP0a06LzviIidko4H7pH0aJl9S10UjDLxNx4gS1aLAE466aRK62pm1loDvU4X4NVXG1+PfrSkRxIRO9PnM8C/ADOBp9PlKtLnM2n3XmBSUfGJwM4Un1giXup8SyOiOyK6u7q66tkUM7PGGqQ32Is1PZFIOlLSUYV14N3Aw8AqYEHabQFwZ1pfBcyTNErSFLKb6uvT5a89kmal0VoXFZUxM2t/eZNIi26yF7Ti0tYJwL+kkbojgB9ExM8kPQCslLQQeAJ4P0BEbJa0EtgC7AMuj4j96ViXAjcBo4G702Jm1v7aoCdSoGhxJmu27u7u6OnpaXU1zMz6V0kSadJvuKQNRY9rHGIwDf81Mxvali+vLIncemvj6lKBVo3aMjOzYpVeyjrxRJg/vzF1qZB7JGZmrVZpEhk5Ep56qjF1qYITiZlZK1VzU72Fz4yU4ktbZmatUO2orEE4QMqJxMysmWoZ1jsIkwj40paZWXNI1SeRadMGbRIB90jMzBqr1gcLB3ECKXCPxMys3s49t7YeSEEbJBFwj8TMrH7qOa1JmyQRcI/EzKx6xxxzsOdRryQye3ZbJRFwj8TMrDKNnEyxzRJIgROJmVk5zZiFt00TSIETiZlZQbOnbm/zBFLgRGJmQ0+r3/XRIQmkwInEzDpPqxNFKR2WPIp51JaZtY/iEVLllsFi7NgsgXRwEgH3SMys2QbTD30jdHjSKMU9EjPL5P3Xfq1Lp7n11oO9jiGYRMA9EhvqTjsNtmxpdS2s3QzRhNGftk8kkuYA3wSGA9+LiK/W/SSHHQavvVb3w5rZIDdy5KB7idRg1NaXtiQNB/4Z+CtgGvBBSdPqehInEbOhofjyVGFxEsmlrRMJMBPYFhH/NyJeBW4D5tb1DE4iZp2hVKIY4vc26qXdE8kE4Mmi770pdghJiyT1SOrZvXt30ypnZk0wUIJwomi4dk8kpYaAvOFvTEQsjYjuiOju6upqQrXMrCJ5k4ETxKDU7jfbe4FJRd8nAjvreoaRI315y4Yu/1BbDu2eSB4ApkqaAjwFzAM+VNczvPqqb7gPNf7xNKtIWyeSiNgn6ePAz8mG/94YEZvrfiKP3DAz61dbJxKAiFgNrG51PczMhqp2v9luZmYt5kRiZmY1cSIxM7OaOJGYmVlNFENsqKOk3cCOKosfBzxbx+oMRm5jZ3AbO8NgauPJEVHyie4hl0hqIaknIrpbXY9Gchs7g9vYGdqljb60ZWZmNXEiMTOzmjiRVGZpqyvQBG5jZ3AbO0NbtNH3SMzMrCbukZiZWU2cSMzMrCZOJDlJmiNpq6Rtkq5sdX3ykjRJ0n2SHpG0WdIVKT5O0j2SHkufxxSVWZzauVXSeUXxGZI2pW3XSSr1YrGWkTRc0r9Luit976g2Shor6XZJj6Y/z7M6sI2fTn9PH5b0Q0mHd0IbJd0o6RlJDxfF6tYuSaMkrUjxdZImN7WBEeFlgIVsivr/AE4BDgMeBKa1ul456z4eODOtHwX8DpgG/CNwZYpfCVyT1qel9o0CpqR2D0/b1gNnkb2Z8m7gr1rdvj5t/QzwA+Cu9L2j2ggsA/4+rR8GjO2kNpK9JvtxYHT6vhK4uBPaCPwn4Ezg4aJY3doFXAZ8O63PA1Y0tX2t/svTDkv6g/t50ffFwOJW16vKttwJ/CWwFRifYuOBraXaRvaul7PSPo8WxT8IfKfV7Smqz0RgDXAOBxNJx7QRGJN+ZNUn3kltnAA8CYwje8XFXcC7O6WNwOQ+iaRu7Srsk9ZHkD0Nr0a1pe/iS1v5FP6CF/SmWFtJ3d0zgHXACRGxCyB9Hp9266+tE9J63/hg8b+A/wYcKIp1UhtPAXYD30+X774n6Ug6qI0R8RTwP4EngF3ACxHxCzqojX3Us12vl4mIfcALwLENq3kfTiT5lLq+2lbjpiW9Cfgx8KmIeLHcriViUSbecpLeAzwTERvyFikRG9RtJPtX5pnA9RFxBvAS2eWQ/rRdG9M9grlkl3NOBI6U9OFyRUrEBnUbc6qmXS1tsxNJPr3ApKLvE4GdLapLxSSNJEsiyyPijhR+WtL4tH088EyK99fW3rTeNz4YvAN4r6TtwG3AOZJupbPa2Av0RsS69P12ssTSSW08F3g8InZHxGvAHcBf0FltLFbPdr1eRtII4GjguYbVvA8nknweAKZKmiLpMLKbWataXKdc0qiOG4BHIuIbRZtWAQvS+gKyeyeF+Lw0CmQKMBVYn7reeyTNSse8qKhMS0XE4oiYGBGTyf5s7o2ID9NZbfx/wJOS3pJCs4EtdFAbyS5pzZJ0RKrbbOAROquNxerZruJjXUj2/0DzemGtvgHVLgtwPtmIp/8Armp1fSqo9zvJurgPARvTcj7Z9dM1wGPpc1xRmatSO7dSNNoF6AYeTtv+iSbezKugvWdz8GZ7R7URmA70pD/LnwDHdGAbvwQ8mup3C9nIpbZvI/BDsvs+r5H1HhbWs13A4cCPgG1kI7tOaWb7PEWKmZnVxJe2zMysJk4kZmZWEycSMzOriROJmZnVxInEzMxq4kRi1g9JV6WZaB+StFHS2yssf7GkEyssM7l4hthScUnTJZ1fyXHNGmlEqytgNhhJOgt4D9nMyXslHUc2427e8sPJZq59mPo/VT2d7HmC1XU+rllV3CMxK2088GxE7AWIiGcjYieApNlp4sRN6T0To1J8u6QvSPoN2cys3cDy1JsZnd4lcb+kDZJ+XjQ9xgxJD0paC1xerlJpZoX/DnwgHfcDko5M9Xgg1Wtu2vdiST+R9FNJj0v6uKTPpH3+TdK4Bv23syHGicSstF8AkyT9TtISSf8ZQNLhwE3AByLirWS9+kuLyr0SEe+MiFvJnkKfHxHTgX3At4ALI2IGcCPwlVTm+8AnI+KsgSoVEa8CXyB738T0iFhB9hT0vRHx58C7gK+lmYEBTgc+BMxM53s5skkf15JNsWFWMycSsxIi4o/ADGAR2fTtKyRdDLyFbGLB36Vdl5G9tKhgRT+HfAvZj/o9kjYC/wBMlHQ0MDYi7k/73VJFdd8NXJmO+0uy6TJOStvui4g9EbGbbGrxn6b4JrL3Y5jVzPdIzPoREfvJfph/KWkT2aR4Gwco9lI/cQGb+/Y6JI2l9um+BfxtRGztc+y3A3uLQgeKvh/A//9bnbhHYlaCpLdImloUmg7sIJtQcLKkU1P8I8D9lLaH7PXGkE2+15Vu4iNppKTTIuIPwAuS3pn2m5+jesXHhezteJ9IM8Ii6YwcxzCrGycSs9LeBCyTtEXSQ2Tv0b46Il4BLgF+lHopB4Bv93OMm4Bvp0tOw8mm975G0oNkPZu/SPtdAvxzutn+pxx1uw+YVrjZDnwZGAk8lIYIf7nSxprVwrP/mplZTdwjMTOzmjiRmJlZTZxIzMysJk4kZmZWEycSMzOriROJmZnVxInEzMxq8v8B/PyuJJpK4jgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(item_popularity, 'ro')\n",
    "pyplot.ylabel('Num Interactions ')\n",
    "pyplot.xlabel('Sorted Item')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average per-item interactions over the whole dataset 936.60\n",
      "Average per-item interactions for the top 10% popular items 6479.52\n",
      "Average per-item interactions for the least 10% popular items 5.23\n",
      "Average per-item interactions for the median 10% popular items 136.45\n"
     ]
    }
   ],
   "source": [
    "ten_percent = int(n_items/10)\n",
    "\n",
    "print(\"Average per-item interactions over the whole dataset {:.2f}\".\n",
    "      format(item_popularity.mean()))\n",
    "\n",
    "print(\"Average per-item interactions for the top 10% popular items {:.2f}\".\n",
    "      format(item_popularity[-ten_percent:].mean()))\n",
    "\n",
    "print(\"Average per-item interactions for the least 10% popular items {:.2f}\".\n",
    "      format(item_popularity[:ten_percent].mean()))\n",
    "\n",
    "print(\"Average per-item interactions for the median 10% popular items {:.2f}\".\n",
    "      format(item_popularity[int(n_items*0.45):int(n_items*0.55)].mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of items with zero interactions 0\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of items with zero interactions {}\".\n",
    "      format(np.sum(item_popularity==0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We compute the user activity (profile length) as the number of interaction in each row\n",
    "\n",
    "### We can use the properties of sparse matrices in CSR format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeoklEQVR4nO3dfZQVd53n8feHhxCIIQmhk4NAgIxsdoiOMfQwycZ11ahBR0N0zG5HNJiJ2zOIGp05Z4Rh14czk3OyO6tr4gjakwc6m9YEH8O6RkXUeMxGSaMxhCQIDg/pBYFkNGAwJJDv/lG/HormdlP3ch+7P69z6lTV99av6tt94H676lf1K0UEZmZmJzKq0QmYmVlrcMEwM7NCXDDMzKwQFwwzMyvEBcPMzAoZ0+gEamXy5Mkxc+bMRqdhZtZSNmzY8FREtJX6bNgWjJkzZ9Lb29voNMzMWoqkHYN95ktSZmZWiAuGmZkV4oJhZmaFuGCYmVkhLhhmZlaIC4aZ2XDQ0wMzZ8KoUdm8p6fqhxi2t9WamY0YPT3Q2QkHD2brO3Zk6wALF1btMD7DMDNrdcuXHy0W/Q4ezOJV5IJhZtbqdu4sL14hFwwzs1Z33nnlxSvkgmFm1upuvBEmTDg2NmFCFq8iFwwzs1a3cCF0dcGMGSBl866uqnZ4g++SMjMbHhYurHqBGMhnGGZmVogLhpnZcFCHB/dqVjAkXSDp4dy0X9KHJU2StFbSljQ/K9dmmaStkjZLuiIXnytpY/rsFkmqVd5mZi2n/8G9HTsg4uiDe1UuGjUrGBGxOSIuioiLgLnAQeDrwFJgXUTMBtaldSTNATqAC4H5wApJo9PuVgKdwOw0za9V3mZmLWeYPbh3OfCriNgBLAC6U7wbuCotLwDujohDEbEN2ArMkzQFmBgRD0ZEAHfm2piZ2TB7cK8D+FJaPjcidgOk+TkpPhV4MtemL8WmpuWB8eNI6pTUK6l33759VUzfzKyJDZcH9ySdAlwJfPlEm5aIxRDx44MRXRHRHhHtbW0l32FuZjb8DKMH994M/Cwi9qT1PekyE2m+N8X7gOm5dtOAXSk+rUTczMygbg/u1aNgXMPRy1EAa4BFaXkRcG8u3iFpnKRZZJ3b69NlqwOSLkl3R12ba2NmZpAVh+3b4cUXs3kNHuKr6ZPekiYAbwT+Ihe+CVgt6XpgJ3A1QERskrQaeAw4DCyJiCOpzWJgFTAeuC9NZmZWR8puPBp+2tvbo7e3t9FpmJm1FEkbIqK91Gd+0tvMzApxwTAzs0JcMMzMrBAXDDMzK8QFw8zMCnHBMDOzQlwwzMysEBcMMzMrxAXDzMwKccEwM7NCXDDMzKwQFwwzMyvEBcPMzApxwTAzs0JcMMzMrBAXDDMzK8QFw8zMCnHBMDOzQmpaMCSdKekrkp6Q9LikSyVNkrRW0pY0Pyu3/TJJWyVtlnRFLj5X0sb02S2SVMu8zczseLU+w7gZ+HZE/FvglcDjwFJgXUTMBtaldSTNATqAC4H5wApJo9N+VgKdwOw0za9x3mZmNkDNCoakicBrgNsAIuL5iPgtsADoTpt1A1el5QXA3RFxKCK2AVuBeZKmABMj4sGICODOXBszM6uTWp5hnA/sA+6Q9HNJt0o6DTg3InYDpPk5afupwJO59n0pNjUtD4wfR1KnpF5Jvfv27avuT2NmNsLVsmCMAS4GVkbEq4BnSZefBlGqXyKGiB8fjOiKiPaIaG9rays3XzMzG0ItC0Yf0BcRP03rXyErIHvSZSbSfG9u++m59tOAXSk+rUTczMzqqGYFIyJ+DTwp6YIUuhx4DFgDLEqxRcC9aXkN0CFpnKRZZJ3b69NlqwOSLkl3R12ba2NmZnUypsb7/yDQI+kU4J+B68iK1GpJ1wM7gasBImKTpNVkReUwsCQijqT9LAZWAeOB+9JkZmZ1pOzGo+Gnvb09ent7G52GmVlLkbQhItpLfeYnvc3MrBAXDDMzK8QFw8zMCnHBMDOzQlwwzMysEBcMMzMrxAXDzMwKccEwM7NCXDDMzKwQFwwzMyvEBcPMzApxwTAzs0JcMMzMrBAXDDOz4aCnB2bOhFGjsnlPT9UPUev3YZiZWa319EBnJxw8mK3v2JGtAyxcWLXD+AzDzKzVLV9+tFj0O3gwi1fRCQuGpD+QNC4tv1bShySdWdUszMyscjt3lhevUJEzjK8CRyS9DLgNmAV8sapZmJlZ5c47r7x4hYoUjBcj4jDwduAzEfERYEqRnUvaLmmjpIcl9abYJElrJW1J87Ny2y+TtFXSZklX5OJz0362SrpFksr7Mc3MhrEbb4QJE46NTZiQxauoSMF4QdI1wCLgmyk2toxjvC4iLsq9I3YpsC4iZgPr0jqS5gAdwIXAfGCFpNGpzUqgE5idpvllHN/MbHhbuBC6umDGDJCyeVdXVTu8oVjBuA64FLgxIrZJmgXcdRLHXAB0p+Vu4Kpc/O6IOBQR24CtwDxJU4CJEfFgRARwZ66NmZlBVhy2b4cXX8zmVS4WUOC22oh4DPhQbn0bcFPB/QfwXUkBfCEiuoBzI2J32tduSeekbacCP8m17UuxF9LywPhxJHWSnYlwXpWv3ZmZjXQnLBiSLgM+AcxI2wuIiDi/wP4vi4hdqSislfTEUIcqEYsh4scHs4LUBdDe3l5yGzMzq0yRB/duAz4CbACOlLPziNiV5nslfR2YB+yRNCWdXUwB9qbN+4DpuebTgF0pPq1E3MzM6qhIH8YzEXFfROyNiKf7pxM1knSapNP7l4E3AY8Ca8g60Enze9PyGqBD0rjUTzIbWJ8uXx2QdEm6O+raXBszM6uTImcYP5D0D8DXgEP9wYj42QnanQt8Pd0BOwb4YkR8W9JDwGpJ1wM7gavT/jZJWg08BhwGlkRE/xnNYmAVMB64L01mZlZHym48GmID6QclwhERr69NStXR3t4evb29jU7DzKylSNqQewziGEXuknpd9VMyM7NWU2QsqTMkfVpSb5o+JemMeiRnZmbNo0in9+3AAeA/pmk/cEctkzIzs+ZTpNP7DyLiz3Lrn5T0cI3yMTOzJlXkDOP3kl7dv5Ie5Pt97VIyM7NmVKRgLAY+l0ae3QH8I/CXtU3LzMzK0gyvaI2Ih4FXSpqY1vdXPQszM6tcTw/8+Z/D889n6zt2ZOtQ1UEIB30OQ9K7I+IuSX9V6vOI+HTVsqgBP4dhZiPG5MnwdIkBOM4+G556qqxdVfocxmlpfnqJzzywn5lZsyhVLIaKV2jQghERX0iL34uIB/KfpY5vMzMbQYp0en+2YMzMzBrh7LPLi1do0DMMSZcC/w5oG9CPMREYXbqVmZnV3c03w3XXwQsvHI2NHZvFq2ioM4xTgJeQFZXTc9N+4J1VzcLMzCq3cCHcccex7/S+446qv6Z1qD6M+4H7Ja2KiB1VPaqZmVXXAw9AXx9EZPMHHqh6wSjSh3GrpDP7VySdJek7Vc3CzMwq9/73w8qVcCS9QujIkWz9/e+v6mGKFIzJEfHb/pWI+A1wTlWzMDOzynV1lRevUJGC8aKk8/pXJM3Az2GYmTWP/jOLovEKFRmtdjnwY0n3p/XXAJ1VzcLMzCo3enTp4jC6uje0nvAMIyK+DVwM3AOsBuZGROE+DEmjJf1c0jfT+iRJayVtSfOzctsuk7RV0mZJV+TicyVtTJ/dovSicDMzAzoH+Rt+sHiFilySAjgC7AWeAeZIek0Zx7gBeDy3vhRYFxGzgXVpHUlzgA7gQmA+sEJSf3lcSXZWMztN88s4vpnZ8LZiBSxefPSMYvTobH3FiqoepsgrWt8H/Aj4DvDJNP9EkZ1Lmgb8KXBrLrwA6E7L3cBVufjdEXEoIrYBW4F5kqYAEyPiwchGSrwz18bMzCArDocPZ7fVHj5c9WIBxc4wbgD+GNgREa8DXgXsK7j/zwB/A7yYi50bEbsB0rz/jqupwJO57fpSbGpaHhg/jqTO/neP79tXNEUzMyuiSMF4LiKeA5A0LiKeAC44USNJbwX2RsSGgrmU6peIIeLHByO6IqI9Itrb2toKHtbMzIoocpdUX3pw7xvAWkm/AXYVaHcZcKWktwCnAhMl3QXskTQlInany017+48DTM+1n5aO05eWB8bNzKyOitwl9faI+G1EfAL4r8BtFOhDiIhlETEtImaSdWZ/PyLeDawBFqXNFgH3puU1QIekcZJmkXVur0+XrQ5IuiTdHXVtro2ZmUHjX9EqaRTwSES8HP51fKmTdROwWtL1wE7g6rTvTZJWA48Bh4ElEdF/Y/FiYBUwHrgvTWZmBllx6OyEgwez9R07jt5SW49XtP7rBlIPsCwidlbtqHXgV7Sa2Ygxc2ZWJAaaMQO2by9rV5W+orXfFGCTpPXAs/3BiLiyrCzMzKw2ShWLoeIVKlIwPlnVI5qZWXXVaWiQExaMiLg/DTg4OyK+J2kCfuOemVnzqNPgg0We9P7PwFeAL6TQVLJbbM3MrBnMmFFevEJFHtxbQvZMxX6AiNiC34dhZtY8brwRJkw4NjZhQhavoiIF41BEPN+/ImkMfh+GmVnzWLgQFi06dvDBRYsa8orW+yX9LTBe0huBLwP/u6pZmJlZ5Xp6oLv72Fe0dndX/eG9IgVjKdlggxuBvwC+FRHLq5qFmZlVbvnyow/t9Tt4MItXUZHbaj8YETcD/9QfkHRDipmZWaPV6TmMImcYi0rE3lvVLMzMrHKDPW9Rr+cwJF0DvAuYJWlN7qPTgaermoWZmVWuTs9hDHVJ6v8Cu4HJwKdy8QPAI1XNwszMKjdjxuBjSVXRoAUjInYAO4BLq3pEMzOrrhtvhPe+N3s1a78xY+r/HIakd0jaIukZSfslHZC0v6pZmJlZ5R544NhiAdn6Aw9U9TBFhjffCrwtIh6v6pFrzMObm9mIMWoUlPoul+DFF8va1VDDmxe5S2pPqxULM7MRZbA//E9wQlCuIs9h9Eq6h2zAwUNH84ivVTUTMzNrakXOMCYCB4E3AW9L01tP1EjSqZLWS/qFpE2SPpnikyStTf0iayWdlWuzTNJWSZslXZGLz5W0MX12S3q3t5mZ1VGR92FcV+G+DwGvj4jfSRoL/FjSfcA7gHURcZOkpWRDj3xU0hygA7gQeCnwPUn/Jr3XeyXQCfwE+BYwH7/X28wsc/nlsG5d6XgVDfXg3mcZYlTaiPjQUDuOrDf9d2l1bJoCWAC8NsW7gR8CH03xuyPiELAtdbbPk7QdmBgRD6a87gSuwgXDzCyzdWt58QoNdYZx0rcYSRoNbABeBnwuIn4q6dyI2A0QEbsl9b9bYyrZGUS/vhR7IS0PjJuZGcDOneXFKzTUg3vdJ7vzdDnpIklnAl+X9PIhNi/VLxFDxI/fgdRJdumK8847r7xkzcxa1aRJ8HSJEZsmTarqYYp0ep+0iPgt2aWn+cAeSVMA0nxv2qwPmJ5rNg3YleLTSsRLHacrItojor2tra2aP4KZWfN67rny4hWqWcGQ1JbOLJA0HngD8ASwhqMj4C4C7k3La4AOSeMkzQJmA+vT5asDki5Jd0ddm2tjZmbPPltevEJFnsOo1BSgO/VjjAJWR8Q3JT0IrJZ0PbATuBogIjZJWg08BhwGlqRLWgCLgVXAeLLObnd4m5nVWZGhQWYBHwRmkiswEXFlTTM7SR4axMxGjKEeTSvzae+hhgYpcobxDeA2svd4lzcoiZmZDRtFCsZzEXFLzTMxM7OmVqRg3Czp48B3OXYsqZ/VLCszM2s6RQrGK4D3AK/n6CWpSOtmZjZCFCkYbwfOj4jna52MmZk1ryLPYfwCOLPGeZiZWZMrcoZxLvCEpIc4tg+jqW+rNTOz6ipSMD5e8yzMzKzpFXkfxv31SMTMzJrbCQuGpAMcHR32FLL3WjwbERNrmZiZmTWXImcYp+fXJV0FzKtVQmZm1pzKHq02Ir6Bn8EwMxtxilySekdudRTQzhCvbjUzs+GpyF1Sb8stHwa2k71/28zMRpAifRjX1SMRMzNrboMWDEkfG6JdRMTf1SAfMzNrUkOdYZR6t99pwPXA2YALhpnZCDJowYiIT/UvSzoduAG4Drgb+NRg7czMbHga8rZaSZMk/T3wCFlxuTgiPhoRe0+0Y0nTJf1A0uOSNkm6IbfPtZK2pPlZuTbLJG2VtFnSFbn4XEkb02e3SEO9j9DMzGph0IIh6R+Ah4ADwCsi4hMR8Zsy9n0Y+OuI+EPgEmCJpDnAUmBdRMwG1qV10mcdwIXAfGCFpNFpXyuBTmB2muaXkYeZmVXBUGcYfw28FPgvwC5J+9N0QNL+E+04Inb3v5UvIg4AjwNTyW7J7U6bdQNXpeUFwN0RcSgitgFbgXmSpgATI+LBiAjgzlwbMzOrk6H6MMp+CnwwkmYCrwJ+CpwbEbvTMXZLOidtNhX4Sa5ZX4q9kJYHxs3MrI6qVhQGI+klwFeBD0fEUGcmpfolYoh4qWN1SuqV1Ltv377ykzUzs0HVtGBIGktWLHoi4mspvCddZiLN+zvQ+4DpuebTgF0pPq1E/DgR0RUR7RHR3tbWVr0fxMzMalcw0p1MtwGPR8Sncx+tARal5UXAvbl4h6RxkmaRdW6vT5evDki6JO3z2lwbMzOrkyJjSVXqMuA9wEZJD6fY3wI3AaslXQ/sBK4GiIhNklYDj5HdYbUkIo6kdouBVcB44L40mZlZHSm78Wj4aW9vj97e3kanYWZWe0M9mlbmd7ykDRHRXuqzmnd6m5nZ8OCCYWZmhbhgmJm1sje8oW6HcsEwM2tl69bV7VAuGGZmVogLhpmZFeKCYWZmhbhgmJkNV1V+zs4Fw8zMCnHBMDOzQlwwzMysEBcMMzMrxAXDzMwKccEwM2tVQ41SWwMuGGZmVogLhpmZFeKCYWZmhbhgmJm1ohP1XyxeXPVD1qxgSLpd0l5Jj+ZikyStlbQlzc/KfbZM0lZJmyVdkYvPlbQxfXaLVOdeHjOzVrRiRdV3WcszjFXA/AGxpcC6iJgNrEvrSJoDdAAXpjYrJI1ObVYCncDsNA3cp5mZ1UHNCkZE/Aj4lwHhBUB3Wu4GrsrF746IQxGxDdgKzJM0BZgYEQ9GRAB35tqYmY1MDbrQUu8+jHMjYjdAmp+T4lOBJ3Pb9aXY1LQ8MG5mZnXWLJ3epcplDBEvvROpU1KvpN59+/ZVLTkzs5ZS5WHN+9W7YOxJl5lI870p3gdMz203DdiV4tNKxEuKiK6IaI+I9ra2tqombmbWFBp430+9C8YaYFFaXgTcm4t3SBonaRZZ5/b6dNnqgKRL0t1R1+bamJmNLA2+SXRMrXYs6UvAa4HJkvqAjwM3AaslXQ/sBK4GiIhNklYDjwGHgSURcSTtajHZHVfjgfvSZGZmpdTochSAooY7b6T29vbo7e1tdBpmZtVR9OziJL/TJW2IiPZSnzVLp7eZmQ2maLG4/PKapuGCYWbWzMrpt/je92qXBy4YZmbNq5xiUYfuhZp1epuZWYWadMg8Fwwzs2ZRaaGo081LLhhmZo12MmcUdbzT1QXDzKxRTvbSU50fi3DBMDOrp2r1TzTgGToXDDOzWqpFB3aDHrh2wTAzq6Za3+HUwNE5XDDMzCpR71tfm2AYJxcMM7PBNMPzEE1QKPq5YJjZyNQMxWAoTVQo+rlgmNnw0OwFoKgmLBT9XDDMrPGGy5d9pZq4SOS5YJjZ0Eb6l3kttEiBGMgFw6yR/GU8MrRogRjIw5tb5Xp6si88T5VPNrxElJ6GiZY5w5A0H7gZGA3cGhE31eAgVd+lmQ0zw6gAlKslzjAkjQY+B7wZmANcI2lOlQ9S1d2ZWQsa7AxhGJ4tVKIlCgYwD9gaEf8cEc8DdwMLGpyTmTWzsWOLFQAXg8JapWBMBZ7Mrfel2DEkdUrqldS7b9++uiVnZjVQ7pf9wOn55xv9Eww7rVIwSl0vOu7PgYjoioj2iGhva2urQ1pmI9TJfpn7r/2W1Cqd3n3A9Nz6NGBXg3Ixqy1/WVqTapWC8RAwW9Is4P8BHcC7qnqECHd8V4O/7MyGrZYoGBFxWNIHgO+Q3VZ7e0RsqsGBqr5LM7PhoiUKBkBEfAv4VqPzMDMbqVql09vMzBrMBcPMzApxwTAzs0JcMMzMrBDFML0zSNI+YEeFzScDT1UxnVpqpVzB+dZSK+UKrZVvK+UKJ5fvjIgo+eTzsC0YJ0NSb0S0NzqPIlopV3C+tdRKuUJr5dtKuULt8vUlKTMzK8QFw8zMCnHBKK2r0QmUoZVyBedbS62UK7RWvq2UK9QoX/dhmJlZIT7DMDOzQlwwzMysEBeMHEnzJW2WtFXS0jof+3ZJeyU9motNkrRW0pY0Pyv32bKU52ZJV+TicyVtTJ/dImVjtksaJ+meFP+ppJknket0ST+Q9LikTZJuaNZ8JZ0qab2kX6RcP9msuQ7Ie7Skn0v6ZrPnK2l7Os7DknqbOV9JZ0r6iqQn0r/fS5s41wvS77R/2i/pww3NNyI8Zf04o4FfAecDpwC/AObU8fivAS4GHs3F/juwNC0vBf5bWp6T8hsHzEp5j06frQcuJXtL4X3Am1P8/cDn03IHcM9J5DoFuDgtnw78MuXUdPmm/b4kLY8Ffgpc0oy5Dsj7r4AvAt9s5n8LaR/bgckDYk2ZL9ANvC8tnwKc2ay5Dsh7NPBrYEYj863Ll2ErTOmX+Z3c+jJgWZ1zmMmxBWMzMCUtTwE2l8qN7D0hl6ZtnsjFrwG+kN8mLY8hewpUVcr7XuCNzZ4vMAH4GfAnzZwr2Rsl1wGv52jBaOZ8t3N8wWi6fIGJwLaBbZsx1xK5vwl4oNH5+pLUUVOBJ3PrfSnWSOdGxG6AND8nxQfLdWpaHhg/pk1EHAaeAc4+2QTTKeyryP5yb8p80+Wdh4G9wNqIaNpck88AfwO8mIs1c74BfFfSBkmdTZzv+cA+4I50ue9WSac1aa4DdQBfSssNy9cF46hS72dt1nuOB8t1qJ+h6j+fpJcAXwU+HBH7h9p0kGPXJd+IOBIRF5H95T5P0suH2LyhuUp6K7A3IjYUbTLIsev5b+GyiLgYeDOwRNJrhti2kfmOIbvsuzIiXgU8S3ZJZzDN8LtF0inAlcCXT7TpIMeuWr4uGEf1AdNz69OAXQ3Kpd8eSVMA0nxvig+Wa19aHhg/po2kMcAZwL9UmpiksWTFoicivtbs+QJExG+BHwLzmzjXy4ArJW0H7gZeL+muJs6XiNiV5nuBrwPzmjTfPqAvnWECfIWsgDRjrnlvBn4WEXvSesPydcE46iFgtqRZqaJ3AGsanNMaYFFaXkTWV9Af70h3OMwCZgPr0+npAUmXpLsgrh3Qpn9f7wS+H+nCZbnSvm8DHo+ITzdzvpLaJJ2ZlscDbwCeaMZcASJiWURMi4iZZP8Gvx8R727WfCWdJun0/mWya+2PNmO+EfFr4ElJF6TQ5cBjzZjrANdw9HLUwGPUN9+T7YwZThPwFrI7fn4FLK/zsb8E7AZeIKv615NdS1wHbEnzSbntl6c8N5PueEjxdrL/sL8C/pGjT/OfSnZKu5XsjonzTyLXV5Odtj4CPJymtzRjvsAfAT9PuT4KfCzFmy7XErm/lqOd3k2ZL1m/wC/StKn//00T53sR0Jv+PXwDOKtZc037mwA8DZyRizUsXw8NYmZmhfiSlJmZFeKCYWZmhbhgmJlZIS4YZmZWiAuGmZkV4oJhI56k5cpGsn0kjQr6J2W2f6+kl5bZZqZyIxPn4q9VGqE2F1sl6Z3l7N+sFsY0OgGzRpJ0KfBWstF3D0maTDaKadH2o4H3kt3j3uiRAQYlaUxkYwWZVcxnGDbSTQGeiohDABHxVKShLiRdngap26jsfSXjUny7pI9J+jHZU7jtQE86Oxmf3j1wfxqM7zu5YRzmKnsvx4PAkkqSlXSTpMfS2dD/SLE2SV+V9FCaLkvxT0jqkvRd4M6T+zWZuWCYfReYLumXklZI+g+QvXgJWAX8p4h4BdnZ+OJcu+ci4tURcRfZk8MLIxvg8DDwWeCdETEXuB24MbW5A/hQRFxaSaKSJgFvBy6MiD8C/j59dDPwPyPij4E/A27NNZsLLIiId1VyTLM8X5KyES0ifidpLvDvgdcB9yh72+LPgW0R8cu0aTfZWcFn0vo9g+zyAuDlwNps2B5GA7slnQGcGRH3p+3+F9mgcselNFiqwH7gOeBWSf8H6O/reAMwJx0PYGL/+E7Amoj4/SD7NCuLC4aNeBFxhGwU2x9K2kg2GNvDJ2j27CBxAZsGnkWkARCLjMPzNNn4RnmTyC6bHZY0j2zQvA7gA2QvWRpF9hKcYwpDKiCD5WlWNl+SshFN2XuTZ+dCFwE7yEa0nSnpZSn+HuB+SjtA9qpayAZ9a0ud6UgaK+nCyIZWf0bSq9N2CwfZ1xbgpZL+MLWfAbwSeFjZ+0fOiIhvAR9OuUJ2We0DuZ/pIsxqwGcYNtK9BPhsOgM4TDZqZ2dEPCfpOuDL6T0BDwGfH2Qfq4DPS/o92Ssx3wncki5DjSG7jLUJuA64XdJBsldjHifdqfVusrfCnUo2evH7IuKZ1Hl+b4oL+Ehq9iHgc5IeScf7EfCXlf5CzAbj0WrNzKwQX5IyM7NCXDDMzKwQFwwzMyvEBcPMzApxwTAzs0JcMMzMrBAXDDMzK+T/A16W16trGQigAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "user_activity = np.ediff1d(URM_all.tocsr().indptr)\n",
    "user_activity = np.sort(user_activity)\n",
    "\n",
    "\n",
    "pyplot.plot(user_activity, 'ro')\n",
    "pyplot.ylabel('Num Interactions ')\n",
    "pyplot.xlabel('Sorted User')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In order to evaluate our recommender we have to define:\n",
    "* A splitting of the data in URM_train and URM_test\n",
    "* An evaluation metric\n",
    "* A functon computing the evaluation for each user\n",
    "\n",
    "### The splitting of the data is very important to ensure your algorithm is evaluated in a realistic scenario by using test it has never seen. We create two splits:\n",
    "#### - Train data: we will use this to train our model\n",
    "#### - Test data: we will use this to evaluate our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True, False, False, ..., False,  True,  True])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_test_split = 0.80\n",
    "\n",
    "n_interactions = URM_all.nnz\n",
    "\n",
    "\n",
    "train_mask = np.random.choice([True,False], n_interactions, p=[train_test_split, 1-train_test_split])\n",
    "train_mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<69878x10676 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 7998532 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "URM_train = sps.csr_matrix((URM_all.data[train_mask],\n",
    "                            (URM_all.row[train_mask], URM_all.col[train_mask])))\n",
    "\n",
    "URM_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<69878x10677 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 2001522 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_mask = np.logical_not(train_mask)\n",
    "\n",
    "URM_test = sps.csr_matrix((URM_all.data[test_mask],\n",
    "                            (URM_all.row[test_mask], URM_all.col[test_mask])))\n",
    "\n",
    "URM_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We call items in the test set 'relevant'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   5,   18,   20,   29,   30,   79,   83,  139,  166,  179,  235,\n",
       "        413, 1280, 1303, 1310, 1312, 1429, 1466, 2310, 2492])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_id = 124\n",
    "relevant_items = URM_test[user_id].indices\n",
    "relevant_items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Say that we have a recommendation list such as this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 241, 1622,   15,  857, 5823])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommended_items = np.array([241, 1622, 15, 857, 5823])\n",
    "recommended_items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, False, False])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_relevant = np.in1d(recommended_items, relevant_items, assume_unique=True)\n",
    "is_relevant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision: how many of the recommended items are relevant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def precision(recommended_items, relevant_items):\n",
    "    \n",
    "    is_relevant = np.in1d(recommended_items, relevant_items, assume_unique=True)\n",
    "    \n",
    "    precision_score = np.sum(is_relevant, dtype=np.float32) / len(is_relevant)\n",
    "    \n",
    "    return precision_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recall: how many of the relevant items I was able to recommend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recall(recommended_items, relevant_items):\n",
    "    \n",
    "    is_relevant = np.in1d(recommended_items, relevant_items, assume_unique=True)\n",
    "    \n",
    "    recall_score = np.sum(is_relevant, dtype=np.float32) / relevant_items.shape[0]\n",
    "    \n",
    "    return recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Average Precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def AP(recommended_items, relevant_items):\n",
    "   \n",
    "    is_relevant = np.in1d(recommended_items, relevant_items, assume_unique=True)\n",
    "    \n",
    "    # Cumulative sum: precision at 1, at 2, at 3 ...\n",
    "    p_at_k = is_relevant * np.cumsum(is_relevant, dtype=np.float32) / (1 + np.arange(is_relevant.shape[0]))\n",
    "    \n",
    "    ap_score = np.sum(p_at_k) / np.min([relevant_items.shape[0], is_relevant.shape[0]])\n",
    "\n",
    "    return ap_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now that we have the data, we can build our first recommender. We need two things:\n",
    "* a 'fit' function to train our model\n",
    "* a 'recommend' function that uses our model to recommend\n",
    "\n",
    "### Let's start with a random recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In a random recommend we don't have anything to learn from the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomRecommender(object):\n",
    "\n",
    "    def fit(self, URM_train):\n",
    "           \n",
    "        self.n_items = URM_train.shape[1]\n",
    "    \n",
    "    \n",
    "    def recommend(self, user_id, at=5):\n",
    "    \n",
    "        recommended_items = np.random.choice(self.n_items, at)\n",
    "\n",
    "        return recommended_items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9698  724 5900 2360 4826]\n",
      "[4754 5439 9629 7973 3819]\n",
      "[1171 5510 3819  760 9554]\n",
      "[8076 6261 8399  943 3476]\n",
      "[4895  905  307  390 2136]\n",
      "[5977 7352 2440 4525 3253]\n",
      "[9315 3572 6346 9759 8776]\n",
      "[6623 5896   81 7576 6647]\n",
      "[ 3392  9679  8316  7484 10076]\n",
      "[8383 3842 1655 2102 4722]\n"
     ]
    }
   ],
   "source": [
    "randomRecommender = RandomRecommender()\n",
    "randomRecommender.fit(URM_train)\n",
    "\n",
    "for user_id in range(10):\n",
    "    print(randomRecommender.recommend(user_id, at=5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Put all together in an evaluation function and let's test it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We pass as paramether the recommender class\n",
    "\n",
    "def evaluate_algorithm(URM_test, recommender_object, at=5):\n",
    "    \n",
    "    cumulative_precision = 0.0\n",
    "    cumulative_recall = 0.0\n",
    "    cumulative_AP = 0.0\n",
    "    \n",
    "    num_eval = 0\n",
    "\n",
    "\n",
    "    for user_id in range(URM_test.shape[0]):\n",
    "\n",
    "        relevant_items = URM_test.indices[URM_test.indptr[user_id]:URM_test.indptr[user_id+1]]\n",
    "        \n",
    "        if len(relevant_items)>0:\n",
    "            \n",
    "            recommended_items = recommender_object.recommend(user_id, at=at)\n",
    "            num_eval+=1\n",
    "\n",
    "            cumulative_precision += precision(recommended_items, relevant_items)\n",
    "            cumulative_recall += recall(recommended_items, relevant_items)\n",
    "            cumulative_AP += AP(recommended_items, relevant_items)\n",
    "            \n",
    "    cumulative_precision /= num_eval\n",
    "    cumulative_recall /= num_eval\n",
    "    MAP = cumulative_AP / num_eval\n",
    "    \n",
    "    print(\"Recommender results are: Precision = {:.4f}, Recall = {:.4f}, MAP = {:.4f}\".format(\n",
    "        cumulative_precision, cumulative_recall, MAP)) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommender results are: Precision = 0.0027, Recall = 0.0005, MAP = 0.0012\n"
     ]
    }
   ],
   "source": [
    "evaluate_algorithm(URM_test, randomRecommender)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So the code works. The recommendation quality however..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Top Popular recommender\n",
    "\n",
    "#### We recommend to all users the most popular items, that is those with the highest number of interactions\n",
    "#### In this case our model is the item popularity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TopPopRecommender(object):\n",
    "\n",
    "    def fit(self, URM_train):\n",
    "\n",
    "        item_popularity = np.ediff1d(URM_train.tocsc().indptr)\n",
    "\n",
    "        # We are not interested in sorting the popularity value,\n",
    "        # but to order the items according to it\n",
    "        self.popular_items = np.argsort(item_popularity)\n",
    "        self.popular_items = np.flip(self.popular_items, axis = 0)\n",
    "    \n",
    "    \n",
    "    def recommend(self, user_id, at=5):\n",
    "    \n",
    "        recommended_items = self.popular_items[0:at]\n",
    "\n",
    "        return recommended_items\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now train and test our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "topPopRecommender = TopPopRecommender()\n",
    "topPopRecommender.fit(URM_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n"
     ]
    }
   ],
   "source": [
    "for user_id in range(10):\n",
    "    print(topPopRecommender.recommend(user_id, at=5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommender results are: Precision = 0.0963, Recall = 0.0308, MAP = 0.0529\n"
     ]
    }
   ],
   "source": [
    "evaluate_algorithm(URM_test, topPopRecommender, at=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### That's better, but we can improve\n",
    "\n",
    "### Hint, remove items already seen by the user. We can either remove them from the recommended item list or we can set them to a score so low that it will cause them to end at the very bottom of all the available items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TopPopRecommender(object):\n",
    "\n",
    "    def fit(self, URM_train):\n",
    "        \n",
    "        self.URM_train = URM_train\n",
    "\n",
    "        item_popularity = np.ediff1d(URM_train.tocsc().indptr)\n",
    "\n",
    "        # We are not interested in sorting the popularity value,\n",
    "        # but to order the items according to it\n",
    "        self.popular_items = np.argsort(item_popularity)\n",
    "        self.popular_items = np.flip(self.popular_items, axis = 0)\n",
    "    \n",
    "    \n",
    "    def recommend(self, user_id, at=5, remove_seen=True):\n",
    "\n",
    "        if remove_seen:\n",
    "            seen_items = self.URM_train.indices[self.URM_train.indptr[user_id]:self.URM_train.indptr[user_id+1]]\n",
    "            \n",
    "            unseen_items_mask = np.in1d(self.popular_items, seen_items, assume_unique=True, invert = True)\n",
    "\n",
    "            unseen_items = self.popular_items[unseen_items_mask]\n",
    "\n",
    "            recommended_items = unseen_items[0:at]\n",
    "\n",
    "        else:\n",
    "            recommended_items = self.popular_items[0:at]\n",
    "            \n",
    "\n",
    "        return recommended_items\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1008    7  139 1293   22]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139 1293  175]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[1008    7  139   14 1293]\n",
      "[ 139   14 1293   22   19]\n"
     ]
    }
   ],
   "source": [
    "topPopRecommender_removeSeen = TopPopRecommender()\n",
    "topPopRecommender_removeSeen.fit(URM_train)\n",
    "\n",
    "for user_id in range(10):\n",
    "    print(topPopRecommender_removeSeen.recommend(user_id, at=5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommender results are: Precision = 0.1989, Recall = 0.0531, MAP = 0.1475\n"
     ]
    }
   ],
   "source": [
    "evaluate_algorithm(URM_test, topPopRecommender_removeSeen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Simple but effective. Always remove seen items if your purpose is to recommend \"new\" ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Global effects recommender\n",
    "\n",
    "#### We recommend to all users the highest rated items\n",
    "\n",
    "#### First we compute the average of all ratings, or global average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The global average is 3.51\n"
     ]
    }
   ],
   "source": [
    "globalAverage = np.mean(URM_train.data)\n",
    "\n",
    "print(\"The global average is {:.2f}\".format(globalAverage))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We subtract the bias to all ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.48756747 1.48756747 1.48756747 1.48756747 1.48756747 1.48756747\n",
      " 1.48756747 1.48756747 1.48756747 1.48756747]\n"
     ]
    }
   ],
   "source": [
    "URM_train_unbiased = URM_train.copy()\n",
    "\n",
    "URM_train_unbiased.data -= globalAverage\n",
    "\n",
    "print(URM_train_unbiased.data[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Then we compute the average rating for each item, or itemBias\n",
    "\n",
    "We cannot use the mean function because it would include also the zero values, which we want to exclude since they mean \"missing data\"\n",
    "\n",
    "The mean should be computed only on existing ratings. To avoid attributing high rating values to items rated by only a few users we can add a constant term to the denominator in order to penalize items with very few ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "lambda_item = 25\n",
    "lambda_user = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.64833751, -0.38417366, -0.58176363, ..., -0.01943945,\n",
       "       -0.05817048,  0.01875259])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This computes the mean of the column excluding the missing values\n",
    "col_nnz = np.ediff1d(sps.csc_matrix(URM_train_unbiased).indptr)\n",
    "\n",
    "item_mean_rating = URM_train_unbiased.sum(axis=0) / (col_nnz + lambda_item)\n",
    "item_mean_rating = np.asarray(item_mean_rating).ravel()  # converts 2-d matrix to 1-d array without anycopy\n",
    "item_mean_rating[col_nnz==0] = -np.inf\n",
    "\n",
    "item_mean_rating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZtUlEQVR4nO3dfbBddX3v8fcnJ4kkKZpEAuQJEsaMM4m2qOeiVKdVQQu5jhlbvUJBQG3PNRFvb53pbbhpta3jTJ+nUhE816ooqYDeIlGj4aGK7bRqTloSEiASUeBwaDk8SgmSkHz7x1obdnb3Xmc/rb3W3vvzmjlz9vqttdf6/pKwP/zWWvu3FBGYmZk1MqvoAszMrNwcFGZmlslBYWZmmRwUZmaWyUFhZmaZZhddQB5OOOGEWLVqVdFlmJn1jV27dj0SEUvqrRvIoFi1ahUTExNFl2Fm1jck3ddonU89mZlZJgeFmZllKjQoJH1W0sOS9jZYL0mXSzogaY+kV/e6RjOzYVf0iOLzwDkZ688F1qQ/Y8CVPajJzMyqFBoUEfFd4LGMTTYAX4jE94CFkpb2pjozM4PiRxQzWQ48ULU8mbb9F5LGJE1Impienu5JcWZmpbB1K6xaBbNmJb+3bu3q7sseFKrTVne624gYj4jRiBhdsqTurcBmZoNn61YYG4P77oOI5PfYWFfDouxBMQmsrFpeAUwVVIuZWfls2QIHDx7bdvBg0t4lZQ+KbcBF6d1PrwOejIiHii7KzKw07r+/tfY2FPrNbElfAt4InCBpEvgoMAcgIq4CtgPrgQPAQeC9xVRqZlZSixfDo4/Wb++SQoMiIs6fYX0AH+xROWZmVkfZTz2ZmVmWxxp8w6BRexscFGZm/WzBgtba2+CgMDPrZ08/3Vp7GxwUZmb9LOp+taxxexscFGZmlslBYWZmmRwUZmaWyUFhZmaZHBRmZv1q06aeHMZBYWbWr67szbPcHBRmZpbJQWFmZpkcFGZm/Wju3Oz111zTtUM5KMzM+tHhw9nrL7iga4dyUJiZWSYHhZlZv5F6ejgHhZnZoJkzp6u7c1CYmfWTZkYThw519ZAOCjOzfrFoUSGHdVCYmfWLJ56YeZt587p+WAeFmVk/aPYC9sGDXT+0g8LMrOyWL29uu1n5fKQXGhSSzpG0X9IBSZvrrH+jpCcl3Z7+fKSIOs3MCjU11dx2R47kcvjZuey1CZJGgCuAtwCTwE5J2yLizppN/yEi3tbzAs3MyqDZU05dfEZ2rSJHFGcAByLi3og4BFwLbCiwHjOz8li+vPmQ6PL3JmoVGRTLgQeqlifTtlpnStot6ZuS1jXamaQxSROSJqanp7tdq5lZ70jNn26Crn9volaRQVEvKmvHTv8CnBoRvwD8NfDVRjuLiPGIGI2I0SVLlnSvSjOzXmp1eo4cTzlVFBkUk8DKquUVwDERGhE/jYj/SF9vB+ZIOqF3JZqZ9Ugrp5oqcj7lVFFkUOwE1khaLWkucB6wrXoDSSdLyZ+cpDNI6n2055WameWp1VNNFTmfcqooLCgi4jngUmAHcBdwfUTsk/QBSR9IN3snsFfSbuBy4LyIHoyzzMx6Yf789maC3bixJ6ecKjSIn7ujo6MxMTFRdBlmZvXNnTvzg4cayekzW9KuiBitt66w71GYmQ2dTp8jUdD/2HsKDzOzvK1b17chAR5RmJnlqxtPoyv4EoFHFGZm3VYZQXRjFFGC68geUZiZdUs3n2VdgoCocFCYmXVqQAOiwqeezMzasXVrd04vVZTkNFM9HlGYmbWim6MHKG04VHNQmJnNpNvhAH0REBU+9WRmVk/ltFK3Q2Lhwr4KCXBQmJm9oFu3tdZTuQbx+OPd33fOfOrJzIbb/PnwzDP57b/PRg/1OCjMbPjkMWKoNQABUeFTT2Y2+KpvZc0zJCqnlwYoJMAjCjMbVL0YNQCsXQv79vXmWAVxUJjZYOhVMAAsWwYPPti74xXMQWFm/aeXoVBtwE4pNcvXKMys3DZtOvb6Qq9DYkCvO7TCIwozK4+iRgq1hjgU6nFQmFkxyhIKFQ6HhhwUZpavsgVChYOhaQ4KM+uOsgZCNYdDWwq9mC3pHEn7JR2QtLnOekm6PF2/R9Kri6jTzFK1F5WLusDcjOqL0EN+MbpThQWFpBHgCuBcYC1wvqS1NZudC6xJf8aAK3tapNkw6qcwqOZQyE2RI4ozgAMRcW9EHAKuBTbUbLMB+EIkvgcslLS014WaDZSsICh7GFR4tNBTRQbFcuCBquXJtK3VbQCQNCZpQtLE9PR0Vws16zv9HgTVHAqFKzIo6v2Lrf0X0Mw2SWPEeESMRsTokiVLOi7OrC8MShhAMmeSQ6GUirzraRJYWbW8AphqYxuzwdavH/yNzJoFR44UXYW1oMgRxU5gjaTVkuYC5wHbarbZBlyU3v30OuDJiHio14Wa5W6QThVVbNxYf4TgkOg7hY0oIuI5SZcCO4AR4LMRsU/SB9L1VwHbgfXAAeAg8N6i6jXr2Lp1cOedRVfRXUMwxbYV/IW7iNhOEgbVbVdVvQ7gg72uy6wjW7fChRcWXUV3+VrBUPM3s806kffzlnvFQWAZHBRmzern6wXgMLC2+XkUZvX060XlefPqX0B2SFgHPKIw65cQqJgzBw4dKroKGyIOChsu/RQKHgVYSTgobHD1Qyg4DKwPOChsMJQ9FBwI1sd8Mdv6z9lnl/tCsy8k24DxiMLKr2xBUM0hYEPAQWHlU9ZgcCjYkHJQWPHKGAwOBbPn+RqF9d6iReW6vuBrCmaZWhpRSFoErIyIPTnVY4OqDIEADgGzNsw4opD0HUkvlrQY2A18TtJf5l+a9bWi70ry09LMuqaZEcVLIuKnkn4D+FxEfFSSRxT2XxU5anAImOWmmWsUsyUtBf4H8PWc67F+U9SowSMFs55pJij+iOQpdAciYqek04B78i3LSq3X4VBvRlQz65kZgyIivhwRPx8Rm9LleyPi1/IvzUpj06beh0N1KBw82JtjmlldM16jkHQc8H5gHXBcpT0i3pdjXVa0Ik4lmVkpNXPq6YvAycCvALcBK4Cn8izKCtLLUYNPJZn1jWaC4mUR8fvA0xFxNfDfgVfmW5b1VBHhYGZ9o5mgOJz+fkLSK4CXAKtyq8h6pxcB4XAw63vNBMV4+o3s3we2AXcCf9rJQSUtlnSzpHvS34sabPcTSXdIul3SRCfHtCp5BkTtF93MrO81c9fTZyLi8Yi4LSJOi4gTI+KqDo+7Gbg1ItYAt6bLjbwpIk6PiNEOj2l5BcRZZ70QDPv2dX//Zlaohnc9SbowIq6R9OF66yOik2k8NgBvTF9fDXwH+N0O9mdZ8ggHjxbMhkbWiGJB+vv4Bj+dOCkiHgJIf5/YYLsAbpK0S9JY1g4ljUmakDQxPT3dYXkDotsjCJ9SMhtKDUcUEfHp9PcftrNjSbeQ3FZba0sLu3l9RExJOhG4WdLdEfHdehtGxDgwDjA6Ojrcn2TdDgczG2pZp56OA94NPA58Dfgd4JeAHwEfi4hHsnYcEWdn7PvfJS2NiIfSeaQebrCPqfT3w5JuAM4A6gaF0b2AmDMHDh3qzr7MrO9lnXr6AvBW4H0k1xBOBT5J8mW7z3d43G3Axenri4EbazeQtEDS8ZXXaS17Ozzu4Jk7t7unmCIcEmZ2jKwpPNZGxCskzQYmI+KX0/ZvSdrd4XH/GLhe0vuB+4F3AUhaBnwmItYDJwE3KPkAnA38bUR8q8PjDo758+GZZ7q3P59iMrMGsoLiEEBEPCdpqmbdkU4OGhGPAmfVaZ8C1qev7wV+oZPjDCxfgzCzHsoKihWSLgdU9Zp0eXnuldmxNm2CK6/szr5mzYIjHWW9mQ2RrKD4narXtd+K9reke6lbIwgHhJm1Iev22Kt7WYjV0a2AmDfPz3Qws7Y1M9eT9Vq37mJauNAP/jGzjs344CLrsW4EhC9Qm1kXOSjKwgFhZiXVzKNQVwMfInkGxfPbR8Tb8ytryHQaEsuWwYMPdqcWM7MazYwovgr8Dck0HkdzrWYYdRISHkGYWQ80ExQ/i4jLZ97MWtJJQKxd6+c+mFnPNBMUn5D0UeAm4NlKY0T8S25VDTqPIsysjzQTFK8E3gO8mRdOPUW6bK1YtAieeKK99zogzKwgzQTFO4DTIsJTinai3VGEL1SbWcGaCYrdwEIaPDPCmtBuSHgUYWYl0ExQnATcLWknx16j8O2xzXBImFmfayYoPpp7FYOqnZDYuBE+9anu12Jm1qYZgyIibpN0KrAmIm6RNB8Yyb+0PtdOSHgUYWYlNOOkgJJ+E/gK8Om0aTnJl/CsEYeEmQ2QZmaP/SDweuCnABFxD3BinkX1tVZDIsIhYWal1kxQPFt9a2z6DG1/stXTTkiYmZVcM0Fxm6T/C8yT9BbgyyTzPlk1h4SZDahmgmIzMA3cAfxPYHtEbMm1qn6zaFFr2zskzKyPNHN77Ici4hPA/6s0SPqttM2gtWk5HBJm1meaGVFcXKftkk4OKuldkvZJOippNGO7cyTtl3RA0uZOjpmbVk45OSTMrA81HFFIOh/4dWC1pG1Vq44HHu3wuHuBX+WFW27rHX8EuAJ4CzAJ7JS0LSLu7PDY3eOQMLMhkHXq6Z+Ah4ATgL+oan8K2NPJQSPiLgBlf9CeARyIiHvTba8FNgDlCAqHhJkNiYZBERH3AfcBZ/aunGMsBx6oWp4EXttoY0ljwBjAKaeckm9lrXBImFmfyzr19BT1vy8hICLixVk7lnQLcHKdVVsi4sYmaqv3v+wNP3UjYhwYBxgdHc3307nZ0cSyZbmWYWbWC1kjiuM72XFEnN3J+0lGECurllcAUx3us3ObNjW33axZfo6EmQ2EZu56KspOYI2k1ZLmAucB22Z4T/6uvLK57Y4cybcOM7MeKSQoJL1D0iTJ9Y9vSNqRti+TtB0gIp4DLgV2AHcB10fEviLqfV6zp5x8XcLMBkgzX7jruoi4AbihTvsUsL5qeTuwvYelNdbst683bsy3DjOzHivzqadyafbb137okJkNGAdFM5q9gO1TTmY2gBwUzWj2AraZ2QByUMxk69bmtvNowswGlINiJhdeOPM2DgkzG2AOCjMzy+SgyDIyMvM2Hk2Y2YBzUGQ5erToCszMCueg6IRHE2Y2BBwUjaxbV3QFZmal4KBo5M4Zno80y390ZjYc/GnXLs8Oa2ZDwkFhZmaZHBT1zPRt7IULe1KGmVkZOCjqec97stc//nhv6jAzKwEHRT2+7dXM7HkOCjMzy+SgqDXTsyfmzetNHWZmJeGgqDXTsycOHuxNHWZmJeGgMDOzTA4KMzPL5KBohaftMLMhVMgnn6R3Sdon6aik0YztfiLpDkm3S5roZY11edoOMxtCsws67l7gV4FPN7HtmyLikZzrMTOzBgoJioi4C0BSEYc3M7MWlP2kewA3SdolaSxrQ0ljkiYkTUxPT7d3tOXL23ufmdkAy21EIekW4OQ6q7ZExI1N7ub1ETEl6UTgZkl3R8R3620YEePAOMDo6Gh7c3BMTbX1NjOzQZZbUETE2V3Yx1T6+2FJNwBnAHWDwszM8lHaU0+SFkg6vvIaeCvJRXAzM+uhom6PfYekSeBM4BuSdqTtyyRtTzc7CfhHSbuBHwDfiIhvFVEvABs3FnZoM7MiKQZwSu3R0dGYmGjjaxdZd2EN4J+TmVmFpF0RUfd7baU99dRzvuPJzKwuB0WF73gyM6vLQWFmZpkcFM1YuLDoCszMCuOgaMbjjxddgZlZYRwUZmaWyUFhZmaZHBQAZ3c824iZ2cByUADcemvRFZiZlZaDwszMMjkoZrJsWdEVmJkVykExkwcfLLoCM7NCOSjMzCyTg8LMzDI5KMzMLJODwszMMjkozMwsk4PCzMwyOSjMzCyTgwJg7tzW2s3MhoiDAuDQodbazcyGiIPCzMwyFRIUkv5M0t2S9ki6QdLCBtudI2m/pAOSNudW0Etf2lq7mdkQKWpEcTPwioj4eeCHwGW1G0gaAa4AzgXWAudLWptLNZ/4BIyMHNs2MpK0m5kNuUKCIiJuiojn0sXvASvqbHYGcCAi7o2IQ8C1wIbcipo1K3vZzGxIleHT8H3AN+u0LwceqFqeTNvqkjQmaULSxPT0dGsVbNkChw8f23b4cNJuZjbkZue1Y0m3ACfXWbUlIm5Mt9kCPAdsrbeLOm3R6HgRMQ6MA4yOjjbcrq7772+t3cxsiOQWFBGR+SBqSRcDbwPOioh6H+yTwMqq5RXAVPcqrHLKKXDfffXbzcyGXFF3PZ0D/C7w9og42GCzncAaSaslzQXOA7blUtD69a21m5kNkaKuUXwSOB64WdLtkq4CkLRM0naA9GL3pcAO4C7g+ojYl0s127e31m5mNkRyO/WUJSJe1qB9ClhftbwdyP/Tut5pp6x2M7MhUoa7nszMrMQcFGZmlslBYWZmmRwUAAsWtNZuZjZEHBQAF13UWruZ2RBxUIBvjzUzy+CgAE/hYWaWwUEBsHhxa+1mZkPEQWFmZpkcFACPPdZau5nZEHFQQONZYj17rJmZgwKAj38c5s8/tm3+/KTdzGzIOSgALrgAxsfh1FNBSn6PjyftZmZDrpDZY0vpggscDGZmdXhEYWZmmRwUZmaWyUFhZmaZHBRmZpbJQWFmZpkUEUXX0HWSpoF2Hnh9AvBIl8spG/dxcAxDP93H3jk1IpbUWzGQQdEuSRMRMVp0HXlyHwfHMPTTfSwHn3oyM7NMDgozM8vkoDjWeNEF9ID7ODiGoZ/uYwn4GoWZmWXyiMLMzDI5KMzMLJODApB0jqT9kg5I2lx0Pa2QtFLStyXdJWmfpN9K2xdLulnSPenvRVXvuSzt635Jv1LV/hpJd6TrLpekIvrUiKQRSf8q6evp8iD2caGkr0i6O/07PXPQ+inpt9N/q3slfUnScf3eR0mflfSwpL1VbV3rk6QXSboubf++pFU97WBEDPUPMAL8CDgNmAvsBtYWXVcL9S8FXp2+Ph74IbAW+FNgc9q+GfiT9PXatI8vAlanfR9J1/0AOBMQ8E3g3KL7V9PXDwN/C3w9XR7EPl4N/Eb6ei6wcJD6CSwHfgzMS5evBy7p9z4CvwS8Gthb1da1PgGbgKvS1+cB1/W0f0X/wyn6J/1L2VG1fBlwWdF1ddCfG4G3APuBpWnbUmB/vf4BO9I/g6XA3VXt5wOfLro/VfWsAG4F3swLQTFofXxx+iGqmvaB6WcaFA8Ai0meh/N14K2D0EdgVU1QdK1PlW3S17NJvsmtvPpS++NTTy/8w62YTNv6TjocfRXwfeCkiHgIIP19YrpZo/4uT1/XtpfFXwH/Bzha1TZofTwNmAY+l55i+4ykBQxQPyPiQeDPgfuBh4AnI+ImBqiPVbrZp+ffExHPAU8CL82t8hoOimSIV6vv7hmW9HPA/wf+d0T8NGvTOm2R0V44SW8DHo6IXc2+pU5bqfuYmk1y+uLKiHgV8DTJKYtG+q6f6Xn6DSSnXJYBCyRdmPWWOm2l7mMT2ulTof11UCSpvbJqeQUwVVAtbZE0hyQktkbE36XN/y5pabp+KfBw2t6ov5Pp69r2Mng98HZJPwGuBd4s6RoGq4+Q1DcZEd9Pl79CEhyD1M+zgR9HxHREHAb+DvhFBquPFd3s0/PvkTQbeAnwWG6V13BQwE5gjaTVkuaSXCjaVnBNTUvvivgb4K6I+MuqVduAi9PXF5Ncu6i0n5feRbEaWAP8IB0aPyXpdek+L6p6T6Ei4rKIWBERq0j+fv4+Ii5kgPoIEBH/Bjwg6eVp01nAnQxWP+8HXidpflrbWcBdDFYfK7rZp+p9vZPkv4HejaCKvPhTlh9gPcndQj8CthRdT4u1v4FkCLoHuD39WU9y/vJW4J709+Kq92xJ+7qfqjtFgFFgb7ruk/TwYlkL/X0jL1zMHrg+AqcDE+nf51eBRYPWT+APgbvT+r5IcvdPX/cR+BLJNZfDJP/3//5u9gk4DvgycIDkzqjTetk/T+FhZmaZfOrJzMwyOSjMzCyTg8LMzDI5KMzMLJODwszMMjkobKhJ2pLOZLpH0u2SXtvi+y+RtKzF96yqnmW0Xruk0yWtb2W/ZnmZXXQBZkWRdCbwNpLZd5+VdALJjK3Nvn+EZObTvXT/W8Gnk9xTv73L+zVrmUcUNsyWAo9ExLMAEfFIREwBSDornZjvjvRZAy9K238i6SOS/pFkds9RYGs6GpmXPk/gNkm7JO2omsLhNZJ2S/pn4INZRaUzBPwR8O50v++WtCCtY2da14Z020skfVXS1yT9WNKlkj6cbvM9SYtz+rOzIeKgsGF2E7BS0g8lfUrSLwNIOg74PPDuiHglych7Y9X7fhYRb4iIa0i+RX1BRJwOPAf8NfDOiHgN8Fng4+l7Pgf8r4g4c6aiIuIQ8BGSZw6cHhHXkXyT9+8j4r8BbwL+LJ1ZFuAVwK8DZ6THOxjJpIL/TDINhFlHHBQ2tCLiP4DXAGMk03tfJ+kS4OUkE9f9MN30apIH01Rc12CXLyf50L5Z0u3A7wErJL0EWBgRt6XbfbGNct8KbE73+x2SKR1OSdd9OyKeiohpkumnv5a230HyjASzjvgahQ21iDhC8sH7HUl3kEy8dvsMb3u6QbuAfbWjBkkL6XxKaAG/FhH7a/b9WuDZqqajVctH8X/j1gUeUdjQkvRySWuqmk4H7iOZsG6VpJel7e8BbqO+p0geQQvJBG9L0ovkSJojaV1EPAE8KekN6XYXNFFe9X4hecLZh9JZRZH0qib2YdYVDgobZj8HXC3pTkl7SJ5l/AcR8TPgvcCX01HGUeCqBvv4PHBVekpohGQK6D+RtJtkZPKL6XbvBa5IL2Y/00Rt3wbWVi5mAx8D5gB70ltoP9ZqZ83a5dljzcwsk0cUZmaWyUFhZmaZHBRmZpbJQWFmZpkcFGZmlslBYWZmmRwUZmaW6T8BE7d2CBsU8wQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "item_mean_rating = np.array(item_mean_rating).squeeze()\n",
    "item_mean_rating = np.sort(item_mean_rating[item_mean_rating!=0])\n",
    "\n",
    "pyplot.plot(item_mean_rating, 'ro')\n",
    "pyplot.ylabel('Item Bias')\n",
    "pyplot.xlabel('Sorted Item')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### And the average rating for each user, or userBias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.91542613, -0.17391898,  0.25592998, ...,  0.3731392 ,\n",
       "        0.27841988, -0.43311858])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This computes the mean of the row excluding the missing values\n",
    "row_nnz = np.ediff1d(sps.csr_matrix(URM_train_unbiased).indptr)\n",
    "\n",
    "# finally, let's compute the bias\n",
    "user_mean_rating = URM_train_unbiased.sum(axis=1).ravel() / (row_nnz + lambda_user)\n",
    "user_mean_rating = np.asarray(user_mean_rating).ravel()\n",
    "user_mean_rating[row_nnz==0] = -np.inf\n",
    "        \n",
    "user_mean_rating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEICAYAAACuxNj9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAabUlEQVR4nO3df7BcZZ3n8ffHm0QBEYbNFUJCEqxJuRtcQOiNULgumOgmFGPWGWYmTFBGx7olyqo7VTsDmypnrFqqZsctZ0TQeMtFYMgAOjOR7BAMIa4y7CxDbjRAIgQySOBW0FzQCWpACH73j/O0dG66+/S9t7vP6e7Pq6rrnn7O0+d8cwn9yXN+PEcRgZmZWTOvK7oAMzMrP4eFmZnlcliYmVkuh4WZmeVyWJiZWS6HhZmZ5So0LCTdKOmApF0N1l8o6aCknen16W7XaGZmMKvg/d8EXA/c0qTPP0TEJVPZ6Ny5c2Px4sUzKMvMbLDs2LHjuYgYbrS+0LCIiPskLW73dhcvXszY2Fi7N2tm1rck7Wu2vhfOWZwv6SFJd0s6o1EnSSOSxiSNTUxMdLM+M7O+V/aw+C6wKCLOAr4AfKNRx4gYjYhKRFSGhxuOpMzMbBpKHRYR8UJE/CwtbwZmS5pbcFlmZgOn1GEh6RRJSsvLyOp9vtiqzMwGT6EnuCXdBlwIzJU0DvwJMBsgItYDlwJXSjoMvAisCU+Ta2bWdUVfDXVZzvrryS6tNTOzRjZsgHXr4OmnYeFCuPZaWLu2rbso+j4LMzObiQ0b4MMfhpdfzt7v25e9h7YGRqnPWZiZWY5PfvK1oKh6+eWsvY0cFmZmvez5Btf8NGqfJoeFmZnlcliYmVkuh4WZmeVyWJiZWS6HhZmZ5XJYmJlZLoeFmZnlcliYmVkuh4WZmeVyWJiZWS6HhZmZ5XJYmJlZLoeFmZnlcliYmVmuQsNC0o2SDkja1WC9JF0naa+khyWd0+0azcys+JHFTcDKJutXAUvSawT4UhdqMjPrDfPnd21XhYZFRNwH/LhJl9XALZF5ADhR0rzuVGdmVnL793dtV0WPLPLMB56peT+e2szMrIvKHhaq0xZ1O0ojksYkjU1MTHS4LDOzwVL2sBgHTqt5vwCoO+6KiNGIqEREZXh4uCvFmZkNirKHxSbgg+mqqPOAgxHxbNFFmZmVXtQ9CDNts9q6tSmSdBtwITBX0jjwJ8BsgIhYD2wGLgb2AoeADxVTqZlZyajeUfrOKTQsIuKynPUBfLxL5ZiZWQNlPwxlZmaTdXlUAQ4LMzNrgcPCzKyXrFiR3+fWW9u+W4eFmVkv2bYtv8/atW3frcPCzKxXFHCuosphYWbWC1oNijbfX1HlsDAzK7sCRxRVDgszszKbSlB0aFQBDgszs/IqwYiiqtA7uM3MrI7phEQHRxXgkYWZWbmUMCjAIwszs3Io0SGnehwWZmZFmmlIdGFUAQ4LM7Ni9EhIVDkszMy6pV2HmrocFOCwMDPrvHaejyggKMBhYWbWGe0+YV1QSFQ5LMzM2qGTVzMVHBRQ8H0WklZK2iNpr6Sr66y/UNJBSTvT69NF1GlmdhTpyFcnRJQiKKDAkYWkIeAG4D3AOLBd0qaI+P6krv8QEZd0vUAzs8m6dS9ESQKiVpEji2XA3oh4MiJeBm4HVhdYj5nZkboxeqhVopHEZEWGxXzgmZr346ltsvMlPSTpbklnNNqYpBFJY5LGJiYm2l2rmQ2CbocDvBYQJQ2JqiLDot5/icm/re8CiyLiLOALwDcabSwiRiOiEhGV4eHh9lVpZv1pcjB0e7qNHgiIWkWGxThwWs37BcD+2g4R8UJE/CwtbwZmS5rbvRLNrG8UGQxVPTKKqKfIsNgOLJF0uqQ5wBpgU20HSadI2X9VScvI6n2+65WaWW8petRQVRsOPRgQtQq7GioiDku6CtgCDAE3RsRuSR9N69cDlwJXSjoMvAisiejx37iZtU/ZZmrt468n9eN3b6VSibGxsaLLMLN2KlswVPXJd6ikHRFRabTed3CbWbmUNRSq+iQcpsphYWbFKXswwMCGw2QOCzPrvF4IBXAwNOGwMLP26JVAqOVwaJnDwsymphdDARwMM+SwMLOj9WoggEOhQxwWZoOsl0MBHAxd5LAw63e9HgjgUCgBh4VZP+iHQKhyMJSSw8KsF/RTGFQ5FHqKw8KsTPotFBwIfcNhYdZt/RYI4FAYAA4Ls05wIFifcViYTVc/BgI4FKwuh4VZMw4EM8BhYZbpx1BwIFgbOSxscPRbIDgMrIuKfAY3klZK2iNpr6Sr66yXpOvS+oclnVNEndZj6j1/uZeDYvJznPvgec7WewoLC0lDwA3AKmApcJmkpZO6rQKWpNcI8KWuFmnl1CgMejkUGgWCQ8FKosiRxTJgb0Q8GREvA7cDqyf1WQ3cEpkHgBMlzet2oVaQfgoDcCBYTysyLOYDz9S8H09tU+0DgKQRSWOSxiYmJtpaqHVYP4WCA8H6VJFhUe/bYPL/Ua30yRojRiOiEhGV4eHhGRdnHdAvoeBAsAFU5NVQ48BpNe8XAPun0cfKqBdDYDJ/+Zv9SpEji+3AEkmnS5oDrAE2TeqzCfhguirqPOBgRDzb7UItRy+PFjxKMGtJYSOLiDgs6SpgCzAE3BgRuyV9NK1fD2wGLgb2AoeADxVVr9FbITCZv/zNZqTQm/IiYjNZINS2ra9ZDuDj3a7L6L1gcBiYdZTv4LbeCgaHglkhcs9ZSLpA0nFp+XJJn5O0qPOlWcf0wvkFn0cwK5VWTnB/CTgk6Szgj4B9wC0drcraq8zh4FAw6wmthMXhdO5gNfD5iPg8cHxny7IZKWs4OBTMelYr5yx+Kuka4HLgXWlOp9mdLcumpEyBUOUgMOsrrYwsfhf4BfAHEfFDsuk2PtvRqixfWUYOPoxkNhByRxYpID5X8/5pfM6i+4oOBXAImA2wVq6GOk/Sdkk/k/SypFclHexGcQOvyNGDRwtmVqOVw1DXA5cBTwDHAB8hew6FdUJRAeFgMLMmWropLyL2ShqKiFeBr0r6xw7XNXiKCAczsxa1EhaH0kR/OyX9OfAscFxnyxoQ3QwIh4OZzUArh6E+QDbR31XAz8mmDP+tThbV97pxmMmHlcysjVq5GmpfWnwR+Exny+lz3QgIM7MOaBgWkr4WEb8j6RHqPJ0uIs7saGX9pFMh4XAwsy5pNrL4ZPp5STcK6UudCAkHhJkVoOE5i+oT6SJiX/VFds7i6ZpDU1bPihXtDQqfezCzgjUMi3Qz3rcl/Z2kt0vaBewCfiRpZfdK7DESbNvWnm05IMysJJodhroe+G/ACcC3gFUR8YCkfw3cBnxzujuVdBJwB7AYeAr4nYj4SZ1+TwE/BV4lm/22Mt19dkU7RhMOBzMroWaXzs6KiHsi4uvADyPiAYCIeKwN+70a2BYRS4Bt6X0jF0XE2X0fFB5FmFmJNQuLX9Ysvzhp3Uy/1VYDN6flm4H/NMPtFWem90w4JMysBzQ7DHWWpBcAAcekZdL7N8xwvyfXnEB/VtKbG/QL4B5JAXw5IkYbbVDSCDACsHDhwhmW16KZhoSZWY9oGBYRMTSTDUu6Fzilzqp1U9jMBRGxP4XJVkmPRcR99TqmIBkFqFQqnf8mnm5QOCTMrAe1NJHgdETEikbrJP1I0rw0qpgHHGiwjf3p5wFJG4FlQN2w6KrpBIVDwsx6WCtzQ3XCJuCKtHwFcOfkDpKOk3R8dRl4L9mlu8VyUJjZACoqLP4MeI+kJ4D3pPdIOlXS5tTnZOB+SQ8BDwJ3RcS0L9dtixUNB0uNOSjMrA80PQwlaQjY0uyQ0nRExPPA8jrt+4GL0/KTwFnt3O+MTfVmOweFmfWJpiOL9LCjQ5JO6FI95TWVw0++HNbM+kwrJ7hfAh6RtJVsbigAIuITHauqbKYaFGZmfaaVsLgrvSyPg8LM+lQrDz+6WdIxwMKI2NOFmsql1VGFg8LM+lju1VCSfgPYSZo4UNLZkjZ1uK5y6OYzss3MSqyVS2f/lOxmuH8BiIidwOkdq6gXeVRhZn2ulbA4HBEHJ7X1/7fjsce21s9BYWYDoJUT3Lsk/R4wJGkJ8AngHztbVgm8OHmiXTOzwdXKyOI/A2cAvyB76NELwKc6WFPv8KjCzAZEK1dDHSKbKXZduqP7uIh4qeOVFamVE9sOCjMbIK1cDfXXkt6UJvPbDeyR9F87X5qZmZVFK4ehlkbEC2RPs9sMLAQ+0MmiSs+jCjMbMK2ExWxJs8nC4s6IeIV+vhrK91aYmR2llbD4MvAUcBxwn6RFZCe5zcxsQOSGRURcFxHzI+LiiAjgaeCizpdWUj4EZWYDqOHVUJL+cFJTAM8B90fEDzpalZmZlUqzkcXxk15vAirA3ZLWdKG27vP5CjOzuhqOLCLiM/XaJZ0E3Avc3qmiSsuHoMxsQE35GdwR8WNgRv8El/TbknZL+qWkSpN+KyXtkbRX0tUz2aeZmU3flMNC0ruBn8xwv7uA3wTua7KfIeAGYBWwFLhM0tIZ7tfMzKah2QnuRzj6foqTgP3AB2ey04h4NO2jWbdlwN6IeDL1vR1YDXx/Jvs2M7OpazY31CWT3gfwfET8vF7nDpgPPFPzfhx4R6POkkaAEYCFCxe2vxqfrzCzAdbsBPe+mWxY0r3AKXVWrYuIO1vZRL2yGnWOiFFgFKBSqfib3cysjVp5nsW0RMSKGW5iHDit5v0CskNgneHLZs3MGpryCe4u2g4skXS6pDnAGmAwnv1tZlYyhYSFpPdLGgfOB+6StCW1nyppM0BEHAauArYAjwJfi4jdRdRrZjboOnYYqpmI2AhsrNO+H7i45v1msmnRzcysQGU+DFUey5cXXYGZWaEcFq24996iKzAzK5TDwszMcjkszMwsl8PCzMxyOSzMzCyXw8LMzHI5LADmzCm6AjOzUnNYALzyStEVmJmVmsPCzMxyOSzMzCyXwyLPlVcWXYGZWeEcFnm++MWiKzAzK5zDwszMcjkszMwsl8PCzMxyOSzMzCxXUY9V/W1JuyX9UlKlSb+nJD0iaaeksW7WaGZmrynksarALuA3gS+30PeiiHiuw/WYmVkTRT2D+1EASUXs3szMpqjs5ywCuEfSDkkjzTpKGpE0JmlsYmKiS+WZmQ2Gjo0sJN0LnFJn1bqIuLPFzVwQEfslvRnYKumxiLivXseIGAVGASqVSkyraDMzq6tjYRERK9qwjf3p5wFJG4FlQN2wMDOzzintYShJx0k6vroMvJfsxLiZmXVZUZfOvl/SOHA+cJekLan9VEmbU7eTgfslPQQ8CNwVEd8sol4zs0FX1NVQG4GNddr3Axen5SeBs7pcmpmZ1VHaw1BmZlYeDgszM8vlsDAzs1wOCzMzy+WwMDOzXA4LMzPL5bAwM7NcDgszM8vlsDAzs1wOCzMzy+WwMDOzXA4LMzPL5bAwM7NcDgszM8vlsDAzs1wOCzMzy+WwMDOzXEU9VvWzkh6T9LCkjZJObNBvpaQ9kvZKurrLZZqZWVLUyGIr8LaIOBN4HLhmcgdJQ8ANwCpgKXCZpKVdrdLMzICCwiIi7omIw+ntA8CCOt2WAXsj4smIeBm4HVjdrRrNzOw1ZThn8WHg7jrt84Fnat6Pp7a6JI1IGpM0NjEx0eYSzcwGW8fCQtK9knbVea2u6bMOOAxsqLeJOm3RaH8RMRoRlYioDA8PT63YoaGptZuZDZhZndpwRKxotl7SFcAlwPKIqBcC48BpNe8XAPvbV2GNm2+Gyy+v325mZoVdDbUS+GPgfRFxqEG37cASSadLmgOsATZ1pKC1a2H58iPbli/P2s3MrLBzFtcDxwNbJe2UtB5A0qmSNgOkE+BXAVuAR4GvRcTujlTzsY/Btm1Htm3blrWbmRmqfwSot1UqlRgbG2v9A7NmwauvHt0+NASHDx/dbmbWZyTtiIhKo/VluBqqePWColm7mdmAcViYmVkuh4WZmeVyWAAsWjS1djOzAeOwALj2Wjj22CPbjj02azczM4cFkN1PMTqajSSk7OfoqO+zMDNLOnYHd89Zu9bhYGbWgEcWZmaWy2FRtWEDLF4Mr3td9nNDvbkNzcwGkw9DQRYMIyNwKE1TtW9f9h58aMrMDI8sMuvWvRYUVYcOZe1mZuawAODpp6fWbmY2YBwWAAsXTq3dzGzAOCwgu/luzpwj2+bM8U15ZmaJw6Jq8lTtfTh1u5nZdDksIDuR/corR7a98opPcJuZJQ4L8AluM7McRT2D+7OSHpP0sKSNkk5s0O8pSY+kR69O4dF3U+QT3GZmTRU1stgKvC0izgQeB65p0veiiDi72eP+ZsyzzpqZNVVIWETEPRFRfbj1A8CCIur4Fc86a2bWlKLgq34k/W/gjoi4tc66HwA/AQL4ckSMNtnOCDACsHDhwnP37dvXoYrNzPqPpB3NjuB0bG4oSfcCp9RZtS4i7kx91gGHgUaz9l0QEfslvRnYKumxiLivXscUJKMAlUrF172ambVRx8IiIlY0Wy/pCuASYHk0GN5ExP7084CkjcAyoG5YmJlZ5xR1NdRK4I+B90XEoQZ9jpN0fHUZeC+wq3tVmplZVVFXQ10PHE92aGmnpPUAkk6VtDn1ORm4X9JDwIPAXRHxzWLKNTMbbIU8zyIifr1B+37g4rT8JHBWN+syM7P6Cr8aqhMkTQDTvRxqLvBcG8vppF6qFXqr3l6qFVxvJ/VSrTD9ehdFxHCjlX0ZFjMhaayjNwC2US/VCr1Vby/VCq63k3qpVuhcvZ4byszMcjkszMwsl8PiaA3vEi+hXqoVeqveXqoVXG8n9VKt0KF6fc7CzMxyeWRhZma5HBZmZpbLYZFIWilpj6S9kq7u8r5vlHRA0q6atpMkbZX0RPr5azXrrkl17pH0H2vaz00Pi9or6TpJSu2vl3RHav8nSYtnUOtpkv6PpEcl7Zb0ybLWK+kNkh6U9FCq9TNlrXVS3UOSvifp78tcr+o8nKystabtnSjpb5Q9eO1RSeeXtV5Jb02/1+rrBUmfKrTeiBj4FzAE/DPwFmAO8BCwtIv7fxdwDrCrpu3PgavT8tXA/0jLS1N9rwdOT3UPpXUPAucDAu4GVqX2jwHr0/Iasinhp1vrPOCctHw82cOrlpax3rTdN6bl2cA/AeeVsdZJdf8h8NfA35f878JTwNxJbaWsNW3jZuAjaXkOcGKZ662pewj4IbCoyHq78mVY9lf6RW6peX8NcE2Xa1jMkWGxB5iXlucBe+rVBmxJ9c8DHqtpv4zsGSC/6pOWZ5Hd3ak21X0n8J6y1wscC3wXeEeZayV7ENg24N28FhalrJf6YVHWWt8E/GDy58ta76Qa3wv836Lr9WGozHzgmZr346mtSCdHxLMA6eebU3ujWuen5cntR3wmsicUHgT+1UwLTMPWt5P9i72U9aZDOjuBA8DWiChtrclfAn8E/LKmraz1BnCPpB3KHj5W5lrfAkwAX02H+L6ibDbrstZbaw1wW1ourF6HRUZ12sp6TXGjWpv9Gdr+55P0RuBvgU9FxAvNujbYd1fqjYhXI+Jssn+xL5P0tibdC61V0iXAgYjY0epHGuy7W38XLoiIc4BVwMclvatJ36JrnUV2qPdLEfF24Odkh3EaKbrebIPSHOB9wNfzujbYd9vqdVhkxoHTat4vAPYXVEvVjyTNA0g/D6T2RrWOc+SzzGv/DL/6jKRZwAnAj6dbmKTZZEGxISL+ruz1AkTEvwDfBlaWuNYLgPdJegq4HXi3pFvLWm/UPJwM2Ej2cLJS1pq2NZ5GlgB/QxYeZa23ahXw3Yj4UXpfWL0Oi8x2YImk01OSrwE2FVzTJuCKtHwF2bmBavuadCXD6cAS4ME0JP2ppPPS1Q4fnPSZ6rYuBb4V6UDlVKVt/y/g0Yj4XJnrlTQs6cS0fAywAnisjLUCRMQ1EbEgIhaT/R38VkRcXsZ61fjhZKWrFSAifgg8I+mtqWk58P2y1lvjMl47BDV5H92td6YnX/rlRfYcjcfJriJY1+V93wY8C7xClvZ/QHbscBvwRPp5Uk3/danOPaQrG1J7hex/2H8me8BU9Q79N5ANY/eSXRnxlhnU+k6yoerDwM70uriM9QJnAt9Lte4CPp3aS1drndov5LUT3KWrl+wcwEPptbv6/0wZa63Zz9nAWPr78A3g10pe77HA88AJNW2F1evpPszMLJcPQ5mZWS6HhZmZ5XJYmJlZLoeFmZnlcliYmVkuh4UNNEnrlM1I+3Ca3fMdU/z870s6dYqfWayaGYZr2i9Ummm2pu0mSZdOZftmnTCr6ALMiiLpfOASsll0fyFpLtlspK1+fgj4fbJr2Iu+478hSbMim/vHbNo8srBBNg94LiJ+ARARz0WawkLS8jTh3CPKnjfy+tT+lKRPS7qf7O7aCrAhjUqOSc8O+E6aXG9LzdQM5yp7rsb/Az4+nWIl/Zmk76dR0P9MbcOS/lbS9vS6ILX/qaRRSfcAt8zs12TmsLDBdg9wmqTHJX1R0n+A7KFJwE3A70bEvyUbgV9Z87mXIuKdEXEr2R3BayObrPAw8AXg0og4F7gRuDZ95qvAJyLi/OkUKukk4P3AGRFxJvDf06rPA38REf8O+C3gKzUfOxdYHRG/N519mtXyYSgbWBHxM0nnAv8euAi4Q9lTEr8H/CAiHk9dbyYbDfxlen9Hg02+FXgbsDWbhoch4FlJJwAnRsR3Ur+/Ipsg7qiSGpUKvAC8BHxF0l1A9dzGCmBp2h/Am6pzNgGbIuLFBts0mxKHhQ20iHiVbDbab0t6hGxitZ05H/t5g3YBuyePHtJkhq3Mq/M82XxFtU4iO1R2WNIysgnw1gBXkT0g6XVkD7A5IhRSeDSq02zKfBjKBpay5xwvqWk6G9hHNjPtYkm/nto/AHyH+n5K9nhZyCZwG04nzpE0W9IZkU2PflDSO1O/tQ229QRwqqR/kz6/CDgL2Kns+SEnRMRm4FOpVsgOpV1V82c6G7MO8MjCBtkbgS+kf/kfJpt9cyQiXpL0IeDraZ7/7cD6Btu4CVgv6UWyx1heClyXDj3NIjt0tRv4EHCjpENkj7M8Sroi63Kyp7m9gWwW4o9ExMF0ovzO1C7gv6SPfQK4QdLDaX/3AR+d7i/ErBHPOmtmZrl8GMrMzHI5LMzMLJfDwszMcjkszMwsl8PCzMxyOSzMzCyXw8LMzHL9f/ZkHx50Y5WVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "user_mean_rating = np.array(user_mean_rating).squeeze()\n",
    "user_mean_rating = np.sort(user_mean_rating[user_mean_rating!=0.0])\n",
    "\n",
    "pyplot.plot(user_mean_rating, 'ro')\n",
    "pyplot.ylabel('User Bias')\n",
    "pyplot.xlabel('Sorted User')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now we can sort the items by their itemBias and use the same recommendation principle as in TopPop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GlobalEffectsRecommender(object):\n",
    "\n",
    "    def fit(self, URM_train, lambda_user=10, lambda_item=25):\n",
    "        \n",
    "        self.URM_train = URM_train\n",
    "\n",
    "        # 1) global average\n",
    "        self.mu = self.URM_train.data.sum(dtype=np.float32) / self.URM_train.nnz\n",
    "\n",
    "        # 2) item average bias\n",
    "        # compute the number of non-zero elements for each column\n",
    "        col_nnz = np.ediff1d(sps.csc_matrix(self.URM_train).indptr)\n",
    "\n",
    "        URM_train_unbiased = self.URM_train.copy()\n",
    "        URM_train_unbiased.data -= self.mu\n",
    "        \n",
    "        item_bias = URM_train_unbiased.sum(axis=0) / (col_nnz + lambda_item)\n",
    "        item_bias = np.asarray(item_bias).ravel()  # converts 2-d matrix to 1-d array without anycopy\n",
    "        item_bias[col_nnz==0] = -np.inf \n",
    "        \n",
    "        self.bestRatedItems = np.argsort(item_bias)\n",
    "        self.bestRatedItems = np.flip(self.bestRatedItems, axis = 0)\n",
    "        \n",
    "    \n",
    "    def recommend(self, user_id, at=5, remove_seen=True):\n",
    "\n",
    "        if remove_seen:\n",
    "            seen_items = self.URM_train.indices[self.URM_train.indptr[user_id]:self.URM_train.indptr[user_id+1]]\n",
    "            \n",
    "            unseen_items_mask = np.in1d(self.bestRatedItems, seen_items, assume_unique=True, invert = True)\n",
    "\n",
    "            unseen_items = self.bestRatedItems[unseen_items_mask]\n",
    "\n",
    "            recommended_items = unseen_items[0:at]\n",
    "\n",
    "        else:\n",
    "            recommended_items = self.bestRatedItems[0:at]\n",
    "            \n",
    "\n",
    "        return recommended_items\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommender results are: Precision = 0.0802, Recall = 0.0197, MAP = 0.0573\n"
     ]
    }
   ],
   "source": [
    "globalEffectsRecommender = GlobalEffectsRecommender()\n",
    "globalEffectsRecommender.fit(URM_train)\n",
    "\n",
    "evaluate_algorithm(URM_test, globalEffectsRecommender)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GlobalEffects has a worse result than TopPop even though we are using \"more\" information.\n",
    "\n",
    "### Let's see how TopPop accuracy changes by using only highly rated items to compute the popularity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removing ratings <= 0\n",
      "Recommender results are: Precision = 0.1989, Recall = 0.0531, MAP = 0.1475\n",
      "Removing ratings <= 1\n",
      "Recommender results are: Precision = 0.1956, Recall = 0.0526, MAP = 0.1436\n",
      "Removing ratings <= 2\n",
      "Recommender results are: Precision = 0.1887, Recall = 0.0516, MAP = 0.1361\n",
      "Removing ratings <= 3\n",
      "Recommender results are: Precision = 0.1614, Recall = 0.0460, MAP = 0.1084\n",
      "Removing ratings <= 4\n",
      "Recommender results are: Precision = 0.1243, Recall = 0.0382, MAP = 0.0748\n"
     ]
    }
   ],
   "source": [
    "for rating_threshold in range(0, 5):\n",
    "    \n",
    "    print(\"Removing ratings <= {}\".format(rating_threshold))\n",
    "    \n",
    "    URM_train_filtered = URM_train.copy()\n",
    "    URM_train_filtered.data[URM_train.data <= rating_threshold] = 0\n",
    "    \n",
    "    URM_train_filtered.eliminate_zeros()\n",
    "    \n",
    "    topPopRecommender = TopPopRecommender()\n",
    "    topPopRecommender.fit(URM_train_filtered)\n",
    "    \n",
    "    evaluate_algorithm(URM_test, topPopRecommender)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question:\n",
    "\n",
    "#### Why is GlobalEffect performing worse than TopPop even if we are taking into account more information about the interaction?\n",
    "\n",
    "#### Why TopPop getting worse if we remove interactions with low rating value?\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    "### Sometimes ratings are not really more informative than interactions. The community has been moving away from explicit ratings for a decade now, preferring implicit interactions that are easier to collect and more reliable.\n",
    "\n",
    "### Generally speaking, \"more information\" is not necessarily better if noisy or hard to model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}