Practice extra 3 - Hybrid model with LightFM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2017/18\n",
    "\n",
    "### Practice 10 - Hybrid model with LightFM\n",
    "\n",
    "##### The following notebook is available at [LightFM Github repo](https://github.com/lyst/lightfm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Item cold-start: recommending StackExchange questions\n",
    "In this example we'll use the StackExchange dataset to explore recommendations under item-cold start. Data dumps from the StackExchange network are available at https://archive.org/details/stackexchange, and we'll use one of them --- for stats.stackexchange.com --- here.\n",
    "\n",
    "The consists of users answering questions: in the user-item interaction matrix, each user is a row, and each question is a column. Based on which users answered which questions in the training set, we'll try to recommend new questions in the training set.\n",
    "\n",
    "Let's start by loading the data. We'll use the `datasets` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from lightfm.datasets import fetch_stackexchange\n",
    "\n",
    "data = fetch_stackexchange('crossvalidated',\n",
    "                           test_set_fraction=0.1,\n",
    "                           indicator_features=False,\n",
    "                           tag_features=True)\n",
    "\n",
    "train = data['train']\n",
    "test = data['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset has 3221 users and 72360 items, with 4307 interactions in the test and 57830 interactions in the training set.\n"
     ]
    }
   ],
   "source": [
    "print('The dataset has %s users and %s items, '\n",
    "      'with %s interactions in the test and %s interactions in the training set.'\n",
    "      % (train.shape[0], train.shape[1], test.getnnz(), train.getnnz()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training and test set are divided chronologically: the test set contains the 10% of interactions that happened after the 90% in the training set. This means that many of the questions in the test set have no interactions. This is an accurate description of a questions answering system: it is most important to recommend questions that have not yet been answered to the expert users who can answer them.\n",
    "\n",
    "## A pure collaborative filtering model\n",
    "\n",
    "This is clearly a cold-start scenario, and so we can expect a traditional collaborative filtering model to do very poorly. Let's check if that's the case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 436 ms, sys: 8 ms, total: 444 ms\n",
      "Wall time: 248 ms\n"
     ]
    }
   ],
   "source": [
    "# Import the model\n",
    "from lightfm import LightFM\n",
    "\n",
    "# Set the number of threads; you can increase this\n",
    "# ify you have more physical cores available.\n",
    "NUM_THREADS = 2\n",
    "NUM_COMPONENTS = 30\n",
    "NUM_EPOCHS = 3\n",
    "ITEM_ALPHA = 1e-6\n",
    "\n",
    "# Let's fit a WARP model: these generally have the best performance.\n",
    "model = LightFM(loss='warp',\n",
    "                item_alpha=ITEM_ALPHA,\n",
    "               no_components=NUM_COMPONENTS)\n",
    "\n",
    "# Run 3 epochs and time it.\n",
    "%time model = model.fit(train, epochs=NUM_EPOCHS, num_threads=NUM_THREADS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a means of sanity checking, let's calculate the model's AUC on the training set first. If it's reasonably high, we can be sure that the model is not doing anything stupid and is fitting the training data well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collaborative filtering train AUC: 0.889321\n"
     ]
    }
   ],
   "source": [
    "# Import the evaluation routines\n",
    "from lightfm.evaluation import auc_score\n",
    "\n",
    "# Compute and print the AUC score\n",
    "train_auc = auc_score(model, train, num_threads=NUM_THREADS).mean()\n",
    "print('Collaborative filtering train AUC: %s' % train_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic, the model is fitting the training set well. But what about the test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collaborative filtering test AUC: 0.354165\n"
     ]
    }
   ],
   "source": [
    "# We pass in the train interactions to exclude them from predictions.\n",
    "# This is to simulate a recommender system where we do not\n",
    "# re-recommend things the user has already interacted with in the train\n",
    "# set.\n",
    "test_auc = auc_score(model, test, train_interactions=train, num_threads=NUM_THREADS).mean()\n",
    "print('Collaborative filtering test AUC: %s' % test_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is terrible: we do worse than random! This is not very surprising: as there is no training data for the majority of the test questions, the model cannot compute reasonable representations of the test set items. \n",
    "\n",
    "The fact that we score them lower than other items (AUC < 0.5) is due to estimated per-item biases, which can be confirmed by setting them to zero and re-evaluating the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collaborative filtering test AUC: 0.508637\n"
     ]
    }
   ],
   "source": [
    "# Set biases to zero\n",
    "model.item_biases *= 0.0\n",
    "\n",
    "test_auc = auc_score(model, test, train_interactions=train, num_threads=NUM_THREADS).mean()\n",
    "print('Collaborative filtering test AUC: %s' % test_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A hybrid model\n",
    "\n",
    "We can do much better by employing LightFM's hybrid model capabilities. The StackExchange data comes with content information in the form of tags users apply to their questions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1246 distinct tags, with values like ['bayesian', 'prior', 'elicitation'].\n"
     ]
    }
   ],
   "source": [
    "item_features = data['item_features']\n",
    "tag_labels = data['item_feature_labels']\n",
    "\n",
    "print('There are %s distinct tags, with values like %s.' % (item_features.shape[1], tag_labels[:3].tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<72360x1246 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 198963 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3221x72360 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 57830 stored elements in COOrdinate format>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use these features (instead of an identity feature matrix like in a pure CF model) to estimate a model which will generalize better to unseen examples: it will simply use its representations of item features to infer representations of previously unseen questions.\n",
    "\n",
    "Let's go ahead and fit a model of this type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define a new model instance\n",
    "model = LightFM(loss='warp',\n",
    "                item_alpha=ITEM_ALPHA,\n",
    "                no_components=NUM_COMPONENTS)\n",
    "\n",
    "# Fit the hybrid model. Note that this time, we pass\n",
    "# in the item features matrix.\n",
    "model = model.fit(train,\n",
    "                item_features=item_features,\n",
    "                epochs=NUM_EPOCHS,\n",
    "                num_threads=NUM_THREADS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, let's sanity check the model on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hybrid training set AUC: 0.848155\n"
     ]
    }
   ],
   "source": [
    "# Don't forget the pass in the item features again!\n",
    "train_auc = auc_score(model,\n",
    "                      train,\n",
    "                      item_features=item_features,\n",
    "                      num_threads=NUM_THREADS).mean()\n",
    "print('Hybrid training set AUC: %s' % train_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the training set AUC is lower than in a pure CF model. This is fine: by using a lower-rank item feature matrix, we have effectively regularized the model, giving it less freedom to fit the training data.\n",
    "\n",
    "Despite this the model does much better on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hybrid test set AUC: 0.702735\n"
     ]
    }
   ],
   "source": [
    "test_auc = auc_score(model,\n",
    "                    test,\n",
    "                    train_interactions=train,\n",
    "                    item_features=item_features,\n",
    "                    num_threads=NUM_THREADS).mean()\n",
    "print('Hybrid test set AUC: %s' % test_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is as expected: because items in the test set share tags with items in the training set, we can provide better test set recommendations by using the tag representations learned from training.\n",
    "\n",
    "## Bonus: tag embeddings\n",
    "\n",
    "One of the nice properties of the hybrid model is that the estimated tag embeddings capture semantic characteristics of the tags. Like the word2vec model, we can use this property to explore semantic tag similarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most similar tags for bayesian: ['mcmc' 'prior' 'r-project']\n",
      "Most similar tags for regression: ['multiple-regression' 'explanatory-models' 'bic']\n",
      "Most similar tags for survival: ['cox-model' 'kaplan-meier' 'epidemiology']\n"
     ]
    }
   ],
   "source": [
    "def get_similar_tags(model, tag_id):\n",
    "    # Define similarity as the cosine of the angle\n",
    "    # between the tag latent vectors\n",
    "    \n",
    "    # Normalize the vectors to unit length\n",
    "    tag_embeddings = (model.item_embeddings.T\n",
    "                      / np.linalg.norm(model.item_embeddings, axis=1)).T\n",
    "    \n",
    "    query_embedding = tag_embeddings[tag_id]\n",
    "    similarity = np.dot(tag_embeddings, query_embedding)\n",
    "    most_similar = np.argsort(-similarity)[1:4]\n",
    "    \n",
    "    return most_similar\n",
    "\n",
    "\n",
    "for tag in (u'bayesian', u'regression', u'survival'):\n",
    "    tag_id = tag_labels.tolist().index(tag)\n",
    "    print('Most similar tags for %s: %s' % (tag_labels[tag_id],\n",
    "                                            tag_labels[get_similar_tags(model, tag_id)]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}