{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Collaborative based reccomenders\n---\n_Project by Qijun Jin, Johhny Nuñez and Marcos Plaza._","metadata":{}},{"cell_type":"markdown","source":"### The goal\n\nThis first practice of the course, is based on using the data from this [kaggle](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/overview) competition to obtain a **product recommendation** based on customer transaction data, as well as other metadata.\n\nTo achieve this goal, we have relied on **collaborative methods**. The hypothesis is: _**\"Similar users tend to like similar items\"**_. More precisely, this is the definition of Neighborhood-based methods (that were among the earliest algorithms developed for collaborative filtering. The previous sentece is valid for a user-based system). So, for this problem as we have a lot of users, we have focused our efforts on proposing a **item-based model**.\n\nFirst of all let's check the data to gain some insights:","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport os\nfrom tqdm.notebook import tqdm\n\n# load data\noriginal_df_customers=pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/customers.csv')\noriginal_df_items=pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/articles.csv')\noriginal_df_customers_items=pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')","metadata":{"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Information of the customers","metadata":{}},{"cell_type":"code","source":"original_df_customers.head(10)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Information of the items","metadata":{}},{"cell_type":"code","source":"original_df_items.head(10)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Information of the transactions: Which customers have purchased certain items?","metadata":{}},{"cell_type":"code","source":"original_df_customers_items.head(10)","metadata":{"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is the data set on which we will focus our implementation. As we can see, there are different metadata that mainly tell us which customer bought a certain item. On the other hand, the number of transactions is huge, so it will not be feasible to use the whole dataset at a computational level. \n\nTherefore, we must force a good enough limit to give a good recommendation. We thought it was a good idea to **take into account the most recent transactions**, since products such as clothing tend to renew and change over time and seasons. We will discard all products before to the following date: ``2020-08-31``.\n","metadata":{}},{"cell_type":"code","source":"d = original_df_customers_items.copy()\n\noriginal_df_customers_items = d[d['t_dat'] > '2020-08-31']\n\ncounts_df = original_df_customers_items.groupby(['t_dat', 'customer_id', 'article_id', 'price', 'sales_channel_id']).size()\ncounts_df = counts_df.to_frame()\ncounts_df.reset_index(inplace=True)\n\nsmall_counts = counts_df.rename(columns={0: 'count'})","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_counts","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Implementation","metadata":{}},{"cell_type":"markdown","source":"The first thing we have to do is to build a matrix that will serve, in some way, as an indication of each person's preferences. For this purpose, we will construct a matrix $m$×$n$, where $m$ is the number of customers and $n$ the number of items or articles. In every entry $(i,j)$ of this matrix we will have the number of times that customer have bought this article.","metadata":{}},{"cell_type":"code","source":"from scipy.sparse import csr_matrix, dok_matrix\nfrom pandas.api.types import CategoricalDtype\n\ndef to_dense(array):\n    \"\"\"\n    Accepta una csr_matrix, dok_matrix o matrix i la converteix en una \n    np.array normal, densa.\n    \n    :param array: Array a convertir\n    :return: np.array densa, sense cap dimensió de tamany 1\n    \"\"\"\n    try:\n        array = array.todense()\n    except:\n        pass\n    \n    return np.array(array).squeeze()\n    \ndef build_counts_table(df):\n    \"\"\"\n    Retorna una csr_matrix on les columnes són els `items`, les files `customer_id` i els valors\n    el nombre de vegades que un usuari ha escoltat un `item`\n    \n    :param df: DataFrame original després de creuar-lo\n    :return: Una tupla constistent de:\n        * La csr_matrix descrita\n        * Els indexos corresponents a cada fila (el customerID de la fila `i` corresponent a l'element `i` d'aquesta array)\n        * Les columnes corresponents a cada columna (el article_id de la columna `j` correspon a l'element `j` d'aquesta array)\n    \"\"\"\n    # Ids, sense repeticions i ordenats\n    customer_ids = CategoricalDtype(sorted(df.customer_id.unique()), ordered=True)\n    item_ids = CategoricalDtype(sorted(df.article_id.unique()), ordered=True)\n\n    # Conversió a csr\n    row = df.customer_id.astype(customer_ids).cat.codes\n    col = df.article_id.astype(item_ids).cat.codes\n    sparse_matrix = csr_matrix((df[\"count\"], (row, col)), \\\n                           shape=(customer_ids.categories.size, item_ids.categories.size))\n\n    return sparse_matrix, customer_ids, item_ids","metadata":{"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Top 10 customers & Top 10 items","metadata":{}},{"cell_type":"code","source":"def top_active_customers(counts, indexes, columns, n):\n    \"\"\"\n    Exemple: Retorna els ids dels n usuaris que més reproduccions han acumulat\n    \n    :param counts, indexes, columns: Tupla retornada per `build_counts_table`\n    :param n: Quanitat d'usuaris\n    :return: Llista, tupla o pd.Series de customerID dels n usuaris\n    \"\"\"\n    # Operate with the sparse matrix, convert to dense the result (as it has much fewer entries)\n    sums = to_dense(counts.sum(axis=1))\n    # Get indices\n    indices = sums.argsort()\n    return indexes.categories[indices[-n:]]\n\ndef top_bought_articles(counts, indexes, columns, n):\n    \"\"\"\n    Exemple: Retorna els ids dels n itemes més escoltats\n    \n    :param counts, indexes, columns: Tupla retornada per `build_counts_table`\n    :param n: Quanitat d'itemes\n    :return: Llista, tupla o pd.Series de itemID dels n itemes\n    \"\"\"\n    # Operate with the sparse matrix, convert to dense the result (as it has much fewer entries)\n    sums = to_dense(counts.sum(axis=0))\n    # Get indices\n    indices = sums.argsort()\n    return columns.categories[indices[-n:]]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"counts, indexes, columns = build_counts_table(small_counts)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is the top 10 customers who have made the most purchases. On the other hand we have the 10 most purchased products (within the restrictions that have been applied).","metadata":{}},{"cell_type":"code","source":"top_customers = top_active_customers(counts, indexes, columns, 10)\ntop_customers","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_items = top_bought_articles(counts, indexes, columns, 10)\ntop_items","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At the same time, we have set another limit to further reduce the dimensionality of our data. We will **keep those top items and customers**.","metadata":{}},{"cell_type":"code","source":"counts, indexes, columns = build_counts_table(small_counts)\nprint(counts.shape)\ntop_customers = top_active_customers(counts, indexes, columns, 5000)\ntop_items = top_bought_articles(counts, indexes, columns, 5000)\n\n\ns = small_counts.copy()\nprint(\"Total: \", len(s))\ns = s[s.article_id.isin(top_items)]\nprint(\"Filter top articles: \", len(s))\ns = s[s.customer_id.isin(top_customers)]\nprint(\"Filter top customers: \", len(s))\n# s = s[:10000]\nprint(\"Total: \", len(s))\ns","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"s = s.drop(s.columns[[0, 3, 4]], axis=1)\ns","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"counts, indexes, columns = build_counts_table(s) # build the counts matrix","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Compute similarities","metadata":{}},{"cell_type":"markdown","source":"To make collaborative recommendation there are two options, to make a user-based recommender or an item-based recommender. As we said in the introduction, we wanted to give an approach to an **item-based system**:\n+ On an item-based approach, we will consider the matrix N×M (items×users), in order to recommend, you will have to base your recommendation on the similarities between the items.","metadata":{}},{"cell_type":"markdown","source":"To compute the product-based similarity matrix, we only need to transpose the ``counts`` matrix above (``counts.T``). This will give us a triangular matrix of N x M based on items.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics.pairwise import pairwise_distances\n\ndef similarity_matrix(similarity_function, counts):\n    if similarity_function is None:\n        x = to_dense(counts)\n        y = to_dense(counts.T)\n        \n        matrix_prod = np.dot(x,y)\n        diagonal = np.diag(matrix_prod)\n        inversa = 1 / diagonal\n        inversa[np.isinf(inversa)] = 0\n        inv_mag = np.sqrt(inversa)\n        \n        cosine = matrix_prod * inv_mag\n        cosine = cosine.T * inv_mag\n        np.fill_diagonal(cosine, 0)\n        \n        return cosine\n    \n    else: \n        print(\"matrix\")\n        x = pd.DataFrame.sparse.from_spmatrix(counts)\n        matrix = pairwise_distances(X = x, metric = similarity_function, n_jobs = -1)\n        del x\n        matrix = csr_matrix(matrix)\n                \n        return matrix","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\n\ntry:\n    with open('similarities.pkl', 'rb') as fp:\n        similarities = pickle.load(fp)\nexcept:\n    similarities = similarity_matrix(similarity_function=\"correlation\", counts=counts.T)\n        \n    with open('similarities.pkl', 'wb') as fp:\n        pickle.dump(similarities, fp, pickle.HIGHEST_PROTOCOL)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Make a prediction","metadata":{}},{"cell_type":"markdown","source":"To make a collaborative recommendation, we need a function that gives us a value of how good the recommendation would be. In our case, as it is item-based, **the score** for a user $u$ i item $i$ is:\n$$pred(u, i) = \\hat{r}_{u,i} = \\frac{\\sum_{j\\neq i,r_{u,j}>0} sim(i, j)\\cdot r_{u,j}}{\\sum_{j\\neq i,r_{u,j}>0} sim(i, j)}$$","metadata":{}},{"cell_type":"code","source":"def score(counts, indexes, columns, similarities, customer, item):\n    customer_ = indexes.categories.get_loc(customer)\n    item_ = columns.categories.get_loc(item)\n    \n    rui = to_dense(counts[:,customer_])\n    \n    sim = np.triu(to_dense(similarities[item_]), k=1)\n    sim = sim[item_]\n    \n    numerador = np.sum(rui* sim)\n    denominador = np.sum(sim[rui > 0])\n    \n    if denominador != 0 and numerador != 0:\n        return numerador/denominador\n    else:\n        return 0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(score(counts.T, indexes, columns, similarities, '01959be607170cc2f092ee8fd13eda251b13cde70ef38dd37e37dcdadfde3b9e',781758057)) #111,123","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def score_mean(counts, indexes, columns, similarities, customer, item):\n    customer_ = indexes.categories.get_loc(customer)\n    item_ = columns.categories.get_loc(item)\n    ru = np.mean(counts[:,item_])\n    ri = to_dense(counts.mean(axis=1))\n    rpi = to_dense(counts[:,customer_])\n    sim = np.triu(to_dense(similarities[item_]), k=1)\n    sim = sim[item_]\n    diferencia = np.subtract(rpi,ri)[rpi>0]\n    numerador = np.sum(diferencia *sim[rpi > 0])\n    denominador = np.sum(sim[rpi > 0])\n    if numerador !=0 and denominador !=0:\n        return ru + (numerador/denominador)\n    else:\n        return ru","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(score_mean(counts.T, indexes, columns, similarities, '01959be607170cc2f092ee8fd13eda251b13cde70ef38dd37e37dcdadfde3b9e',781758057)) \n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from queue import PriorityQueue\nfrom tqdm.notebook import trange, tqdm\nimport heapq as hq\nimport operator\n\ndef recommend_n_items(counts, indexes, columns, similarities, customer, N):\n    minheap = []\n    customer_ = indexes.categories.get_loc(customer)\n    for item in tqdm(columns.categories.tolist()):\n        item_ = columns.categories.get_loc(item)\n        if counts[customer_,item_] == 0:\n            valor = score(counts, indexes, columns, similarities, customer, item)\n            hq.heappush(minheap, (-valor, item))\n\n    res = []\n\n    for i in range(N):\n        res.append(hq.heappop(minheap)[1])\n\n    return res","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(recommend_n_items(counts.T, indexes, columns, similarities, '01959be607170cc2f092ee8fd13eda251b13cde70ef38dd37e37dcdadfde3b9e',12)) ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from queue import PriorityQueue\nimport heapq as hq\nimport operator\n\ndef recommend_n_items_mean(counts, indexes, columns, similarities, customer, N):\n    minheap = []\n    customer_ = indexes.categories.get_loc(customer)\n    for item in tqdm(columns.categories.tolist()):\n        item_ = columns.categories.get_loc(item)\n        if counts[customer_,item_] == 0:\n            valor = score_mean(counts, indexes, columns, similarities, customer, item)\n            hq.heappush(minheap, (-valor, item))\n\n    res = []\n\n    for i in range(N):\n        res.append(hq.heappop(minheap)[1])\n\n    return res","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(recommend_n_items_mean(counts.T, indexes, columns, similarities, '01959be607170cc2f092ee8fd13eda251b13cde70ef38dd37e37dcdadfde3b9e',12))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Convert(string):\n    li = list(string.split(\" \"))\n    return li","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generate the ``submission.csv`` file","metadata":{}},{"cell_type":"code","source":"kaggle_df_customer_articles = pd.read_csv('./data/sample_submission.csv')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kaggle_df_customer_articles.customer_id.unique()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    from tqdm.notebook import trange, tqdm\nexcept:\n    def tqdm(a, _, *__): return a\n    \n# Fetch kaggle public data and merge\nkaggle_df_customer_articles = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv')\n    \n# Obtain counts\npred_counts, pred_indexes, pred_columns = build_counts_table(s)\n    \n# Similarity\ntry:\n    with open('pred_similarities.pkl', 'rb') as fp:\n        pred_similarities = pickle.load(fp)\nexcept:\n    pred_similarities = similarity_matrix(similarity_function=\"correlation\", counts=pred_counts.T)\n        \n    with open('pred_similarities.pkl', 'wb') as fp:\n        pickle.dump(pred_similarities, fp, pickle.HIGHEST_PROTOCOL)\n    \nresults = pd.DataFrame(columns=['customer_id', 'prediction'])\ntop_n_items = top_bought_articles(pred_counts, pred_indexes, pred_columns, 12)\ntop_n_items = ''.join(['0'+str(i)+' ' for i in top_n_items])[:-1]\n\nfor idx, customer in enumerate(tqdm(kaggle_df_customer_articles.customer_id.unique())):\n    try:\n        article_ids = recommend_n_items_mean(pred_counts.T, pred_indexes, pred_columns, pred_similarities, customer,12)\n        article_ids = Convert(article_ids)\n        article_ids = ''.join(['0'+str(i)+' ' for i in article_ids])[:-1]\n    except:\n        article_ids = top_n_items\n    results.loc[idx] = (customer, ''.join(article_ids))\n        \nresults.to_csv('submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]}]}