{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# H&M Consumer Analytics-RFM Segmentation and Collaborative Filtering","metadata":{"id":"gGziO4HliSQ7"}},{"cell_type":"markdown","source":"### We have applied consumer analytics on H&M Data set\n\nhttps://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendation\n\n***Overview:\n\n\n* Overview of Data\n* Handling Missing value treatment / Feature Engineering\n \n Find the EDA-\n https://github.com/techanalyst84/customer-analytics-project/blob/main/h-m-sales-and-customers-deep-analysis_final.ipynb \n \nSolution Approach\n##RFM\n##Recommender Algorithm ##Singular Value Decomposition ##\n\nThe idea is to classify the customers based on sale transactions info over the last 1 year, into segments on basis of RFM(recency of purchase, frequency of purchase, and Montetory value they bring to the table). \nThen to build a base recommender model based on Top RFM segments as training data and based on ML concepts like matrix factorization, Cosine similarity and Singular Value decomposition.\nFurther to use this model to come up with predictions/recommendations of products/articles for the entire customer list. \nTo begin off this is good with Score of 0.0068, as the model is based on popular items in the recent past(amongst the popular customers); however, one can improvise upon this by building models specific to the customer's age group.\n\nThank you for your time !!\n\nPlease dont forget to upvote, if you find this informative for your work!!","metadata":{"id":"ZzY1c7H5iSRC"}},{"cell_type":"code","source":"## https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/overview -Score of Score: 0.0068-Please upvote!!\n## Version Update- Made changes to train the model based on purchases in last year on Top 5 RFM segment and then come up with recommendation for entire customers\n## Score of 0.0068, with much better run time. This model can be further improvised to include customer age into account..Wait for the next update!!\nimport numpy as np \nimport pandas as pd\n \n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.style.use('seaborn-white')\nsns.set_style(\"whitegrid\")\nsns.despine()\nplt.rc(\"figure\", autolayout=True)\nplt.rc(\"axes\", labelweight=\"bold\", labelsize=\"large\", titleweight=\"bold\", titlesize=14, titlepad=10)\n\nimport matplotlib as mpl\n\nmpl.rcParams['axes.spines.left'] = False\nmpl.rcParams['axes.spines.right'] = False\nmpl.rcParams['axes.spines.top'] = False\nmpl.rcParams['axes.spines.bottom'] = False\nplt.rcParams[\"font.weight\"] = \"bold\"\nplt.rcParams[\"axes.labelweight\"] = \"bold\"","metadata":{"id":"Y9qNgocBiSRS","outputId":"dbfba3d3-2932-48bb-c985-190486aea568","execution":{"iopub.status.busy":"2022-05-06T15:55:10.309076Z","iopub.execute_input":"2022-05-06T15:55:10.31097Z","iopub.status.idle":"2022-05-06T15:55:11.442022Z","shell.execute_reply.started":"2022-05-06T15:55:10.310866Z","shell.execute_reply":"2022-05-06T15:55:11.441209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data import","metadata":{"id":"Nxp_FoTOiSRU"}},{"cell_type":"code","source":"\n#articles = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\", \n #                      encoding=\"ISO-8859-1\", header=0)\n#customers = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\",\n #                       encoding=\"ISO-8859-1\", header=0)\ntransactions =  pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\",\n                           encoding=\"ISO-8859-1\",dtype={'article_id':str}, header=0).drop_duplicates()","metadata":{"id":"UdyjcsSBiSRV","execution":{"iopub.status.busy":"2022-05-06T15:55:11.443629Z","iopub.execute_input":"2022-05-06T15:55:11.443872Z","iopub.status.idle":"2022-05-06T15:57:04.977351Z","shell.execute_reply.started":"2022-05-06T15:55:11.443846Z","shell.execute_reply":"2022-05-06T15:57:04.976165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.head(7)","metadata":{"id":"fthC0ivViSR6","outputId":"75a21c87-c05e-4023-96ec-ff17f4b9fb9b","execution":{"iopub.status.busy":"2022-05-06T15:57:04.979218Z","iopub.execute_input":"2022-05-06T15:57:04.979504Z","iopub.status.idle":"2022-05-06T15:57:05.00038Z","shell.execute_reply.started":"2022-05-06T15:57:04.979474Z","shell.execute_reply":"2022-05-06T15:57:04.999396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# RFM Analysis\n","metadata":{"id":"iK6RuC7piSSb"}},{"cell_type":"code","source":"# import required libraries for clustering\nimport sklearn\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics import silhouette_score\nfrom scipy.cluster.hierarchy import linkage\nfrom scipy.cluster.hierarchy import dendrogram\nfrom scipy.cluster.hierarchy import cut_tree","metadata":{"id":"0Mt2lGg6iSSb","execution":{"iopub.status.busy":"2022-05-06T15:57:05.002794Z","iopub.execute_input":"2022-05-06T15:57:05.003224Z","iopub.status.idle":"2022-05-06T15:57:05.418973Z","shell.execute_reply.started":"2022-05-06T15:57:05.003197Z","shell.execute_reply":"2022-05-06T15:57:05.41813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" \n\ntransactions['InvoiceDate'] = pd.to_datetime(transactions['t_dat'],format='%Y-%m-%d')\ntransactions=transactions[[\"InvoiceDate\", \"customer_id\", \"article_id\", \"price\",\"sales_channel_id\"]].drop_duplicates()","metadata":{"id":"QQKqhkkJiSSb","execution":{"iopub.status.busy":"2022-05-06T15:57:05.420085Z","iopub.execute_input":"2022-05-06T15:57:05.420292Z","iopub.status.idle":"2022-05-06T15:57:38.692776Z","shell.execute_reply.started":"2022-05-06T15:57:05.420262Z","shell.execute_reply":"2022-05-06T15:57:38.691937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.shape","metadata":{"id":"aF90clwKiSSb","execution":{"iopub.status.busy":"2022-05-06T15:57:38.694092Z","iopub.execute_input":"2022-05-06T15:57:38.694386Z","iopub.status.idle":"2022-05-06T15:57:38.701346Z","shell.execute_reply.started":"2022-05-06T15:57:38.694329Z","shell.execute_reply":"2022-05-06T15:57:38.700421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# checking df's missing value's attribution in %\ndf_null = round(100*(transactions.isnull().sum())/len(transactions), 2)\ndf_null","metadata":{"id":"EeOT6J7NiSSb","outputId":"aa2d3f51-9d31-4389-fd3b-2324a1dfe7dd","execution":{"iopub.status.busy":"2022-05-06T15:57:38.70307Z","iopub.execute_input":"2022-05-06T15:57:38.703528Z","iopub.status.idle":"2022-05-06T15:57:44.687551Z","shell.execute_reply.started":"2022-05-06T15:57:38.703489Z","shell.execute_reply":"2022-05-06T15:57:44.686749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# checking df's missing value's attribution in %\ndf_null = round(100*(transactions.isna().sum())/len(transactions), 2)\ndf_null","metadata":{"id":"pLU8uCLyiSSc","outputId":"d0f4aefe-2b3e-4a98-b05c-d976dd89b03d","execution":{"iopub.status.busy":"2022-05-06T15:57:44.688675Z","iopub.execute_input":"2022-05-06T15:57:44.688905Z","iopub.status.idle":"2022-05-06T15:57:50.57533Z","shell.execute_reply.started":"2022-05-06T15:57:44.688869Z","shell.execute_reply":"2022-05-06T15:57:50.574159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##Generate Invoice ID as combination of Customer id and Transaction Date.\n#transactions['_ID'] = transactions['customer_id']  + transactions['InvoiceDate'].astype(str) \n\n#transactions['Invoice_id'] = pd.factorize(transactions['_ID'])[0]\n","metadata":{"id":"9Hg-XvvTiSSc","execution":{"iopub.status.busy":"2022-05-06T15:57:50.576556Z","iopub.execute_input":"2022-05-06T15:57:50.57678Z","iopub.status.idle":"2022-05-06T15:57:50.580221Z","shell.execute_reply.started":"2022-05-06T15:57:50.576753Z","shell.execute_reply":"2022-05-06T15:57:50.579668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.head()","metadata":{"id":"YXLJTz4AiSSc","outputId":"49fd7437-a2d2-43a1-d2b2-64f1e25c72c0","execution":{"iopub.status.busy":"2022-05-06T15:57:50.582528Z","iopub.execute_input":"2022-05-06T15:57:50.582704Z","iopub.status.idle":"2022-05-06T15:57:50.600156Z","shell.execute_reply.started":"2022-05-06T15:57:50.58268Z","shell.execute_reply":"2022-05-06T15:57:50.599512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import datetime as dt","metadata":{"id":"A_iSC30ciSSc","execution":{"iopub.status.busy":"2022-05-06T15:57:50.601494Z","iopub.execute_input":"2022-05-06T15:57:50.602351Z","iopub.status.idle":"2022-05-06T15:57:50.610881Z","shell.execute_reply.started":"2022-05-06T15:57:50.602305Z","shell.execute_reply":"2022-05-06T15:57:50.610417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_date = dt.datetime(2020,3,1)\n\n# Filter transactions by date\ntransactions[\"t_dat\"] = pd.to_datetime(transactions[\"InvoiceDate\"])\ntransactions = transactions.loc[transactions[\"t_dat\"] >= start_date]","metadata":{"execution":{"iopub.status.busy":"2022-05-06T15:57:50.61207Z","iopub.execute_input":"2022-05-06T15:57:50.612417Z","iopub.status.idle":"2022-05-06T15:57:52.11231Z","shell.execute_reply.started":"2022-05-06T15:57:50.612383Z","shell.execute_reply":"2022-05-06T15:57:52.111418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#analysis_date = max(transactions['InvoiceDate']) + dt.timedelta(days= 1)\nanalysis_date=dt.datetime(2020,9,23)\nprint((analysis_date).date())","metadata":{"id":"80tJbpvtiSSd","outputId":"5a970194-3aef-4982-86d3-d81b7da699d3","execution":{"iopub.status.busy":"2022-05-06T15:57:52.113584Z","iopub.execute_input":"2022-05-06T15:57:52.113867Z","iopub.status.idle":"2022-05-06T15:57:52.12014Z","shell.execute_reply.started":"2022-05-06T15:57:52.11383Z","shell.execute_reply":"2022-05-06T15:57:52.119169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions['date']=transactions['InvoiceDate']","metadata":{"execution":{"iopub.status.busy":"2022-05-06T15:57:52.12148Z","iopub.execute_input":"2022-05-06T15:57:52.121643Z","iopub.status.idle":"2022-05-06T15:57:52.145541Z","shell.execute_reply.started":"2022-05-06T15:57:52.121621Z","shell.execute_reply":"2022-05-06T15:57:52.145001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rfm = transactions.groupby('customer_id').agg({\n    'InvoiceDate': lambda x: (analysis_date - x.max()).days,\n    'date': 'count',\n    'price': 'sum'})\n#rfm.head()\nrfm.columns=[\"Recency\",\"Frequency\",\"Monetary\"]\nrfm = rfm[rfm[\"Monetary\"] > 0]\n \n #https://www.kaggle.com/code/kanberburak/rfm-analysis/notebook","metadata":{"id":"AEKQ9VhdiSSd","outputId":"dcfe0e46-e912-4834-f2dd-158486b80d25","scrolled":true,"execution":{"iopub.status.busy":"2022-05-06T15:57:52.146702Z","iopub.execute_input":"2022-05-06T15:57:52.147085Z","iopub.status.idle":"2022-05-06T15:59:55.351457Z","shell.execute_reply.started":"2022-05-06T15:57:52.147061Z","shell.execute_reply":"2022-05-06T15:59:55.350651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" transactions.head(1)","metadata":{"execution":{"iopub.status.busy":"2022-05-06T15:59:55.352571Z","iopub.execute_input":"2022-05-06T15:59:55.352765Z","iopub.status.idle":"2022-05-06T15:59:55.366194Z","shell.execute_reply.started":"2022-05-06T15:59:55.352739Z","shell.execute_reply":"2022-05-06T15:59:55.364765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Date from customer's last purchase.The nearest date gets 5 and the furthest date gets 1.\nrfm[\"recency_score\"] = pd.qcut(rfm['Recency'], 5, labels=[5, 4, 3, 2, 1])\n# Total number of purchases.The least frequency gets 1 and the maximum frequency gets 5.\nrfm[\"frequency_score\"] = pd.qcut(rfm[\"Frequency\"].rank(method=\"first\"), 5, labels=[1, 2, 3, 4, 5])\n#Total spend by the customer.The least money gets 1, the most money gets 5.\nrfm[\"monetary_score\"]= pd.qcut(rfm[\"Monetary\"],5,labels=[1,2,3,4,5])\nrfm.head()","metadata":{"id":"pnRGfaemiSSe","outputId":"aa7c0df6-73e0-4cb7-bdaa-6019298e1c3f","execution":{"iopub.status.busy":"2022-05-06T15:59:55.367847Z","iopub.execute_input":"2022-05-06T15:59:55.368085Z","iopub.status.idle":"2022-05-06T15:59:55.769664Z","shell.execute_reply.started":"2022-05-06T15:59:55.368056Z","shell.execute_reply":"2022-05-06T15:59:55.76794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#RFM - The value of 2 different variables that were formed was recorded as a RFM_SCORE\nrfm[\"RFM_SCORE\"] = (rfm[\"recency_score\"].astype(str) + rfm[\"frequency_score\"].astype(str))","metadata":{"id":"a4vyxGgEiSSe","execution":{"iopub.status.busy":"2022-05-06T15:59:55.77149Z","iopub.execute_input":"2022-05-06T15:59:55.771762Z","iopub.status.idle":"2022-05-06T15:59:56.189223Z","shell.execute_reply.started":"2022-05-06T15:59:55.771725Z","shell.execute_reply":"2022-05-06T15:59:56.188436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seg_map = {\n    r'[1-2][1-2]': 'hibernating',\n    r'[1-2][3-4]': 'at_Risk',\n    r'[1-2]5': 'cant_loose',\n    r'3[1-2]': 'about_to_sleep',\n    r'33': 'need_attention',\n    r'[3-4][4-5]': 'loyal_customers',\n    r'41': 'promising',\n    r'51': 'new_customers',\n    r'[4-5][2-3]': 'potential_loyalists',\n    r'5[4-5]': 'champions'\n}\nrfm['segment'] = rfm['RFM_SCORE'].replace(seg_map, regex=True)\nrfm.head()","metadata":{"id":"Ro2LwGlxiSSe","outputId":"49c3de02-fd99-4ac3-c035-02722e7db4d1","execution":{"iopub.status.busy":"2022-05-06T15:59:56.190208Z","iopub.execute_input":"2022-05-06T15:59:56.190449Z","iopub.status.idle":"2022-05-06T16:00:17.072004Z","shell.execute_reply.started":"2022-05-06T15:59:56.190417Z","shell.execute_reply":"2022-05-06T16:00:17.07108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rfm[[\"segment\", \"Recency\",\"Frequency\",\"Monetary\"]].groupby(\"segment\").agg([\"mean\",\"count\",\"max\"]).round()","metadata":{"id":"9ttdWHb3iSSe","outputId":"1c5914c8-6af9-4344-f980-4ec268636b3e","execution":{"iopub.status.busy":"2022-05-06T16:00:17.072991Z","iopub.execute_input":"2022-05-06T16:00:17.073173Z","iopub.status.idle":"2022-05-06T16:00:17.338726Z","shell.execute_reply.started":"2022-05-06T16:00:17.073149Z","shell.execute_reply":"2022-05-06T16:00:17.337844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.express as px\n","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:17.339763Z","iopub.execute_input":"2022-05-06T16:00:17.339949Z","iopub.status.idle":"2022-05-06T16:00:18.883187Z","shell.execute_reply.started":"2022-05-06T16:00:17.339924Z","shell.execute_reply":"2022-05-06T16:00:18.882583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" \nx = rfm.segment.value_counts()\nfig = px.treemap(x, path=[x.index], values=x)\nfig.update_layout(title_text='Distribution of the RFM Segments', title_x=0.5,\n                  title_font=dict(size=20))\nfig.update_traces(textinfo=\"label+value+percent root\")\nfig.show()","metadata":{"id":"PxWJfAXYiSSe","outputId":"a401b730-69e5-4111-92df-ce217d745761","execution":{"iopub.status.busy":"2022-05-06T16:00:18.884277Z","iopub.execute_input":"2022-05-06T16:00:18.884905Z","iopub.status.idle":"2022-05-06T16:00:20.009158Z","shell.execute_reply.started":"2022-05-06T16:00:18.884873Z","shell.execute_reply":"2022-05-06T16:00:20.008666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Recommend Items Frequently Purchased Together\n","metadata":{"id":"H2OHNccQiSSk"}},{"cell_type":"markdown","source":"Item-Item Based Collaborative Filtering\n\n* Objective-To produce recommendations of Items for Hibernating customer-User 5(from RFM) for their upcoming purchase. \n\nStep 1- Matrix Factorization.\nThe Entries in table are based on time-adjusted count of # of purchase by user- item A bought 5 times on the first day of the train period is inferior to item B bought 4 times on the last day of the train period. This is done by weighted down exponentially by day of purchase\n\nStep 2- See Recommendation as optimization problem, Rating Prediction-make good Recommendation or prediction.\nQuantify Goodness using RMSE:\nLower RMSE =>better recommendation.\nWant to make good recommendation on items that user has not yet seen or purchase before(example – Hibernating customer-User 5 from RFM). Purely based on Popularity of the item. How we do?\n\nLet’s build a system such that it works well on known (User, Product) rating/purchase counts. And hope the system will also predict well the unknown ratings.\n\nDone by optimization method– Epoch. Then use this system to predict/recommend items unknown users\n\nUse Latent Factor Model like SVD to Dimension Reduction, handling nulls.\nNow this is can be assumed as vector space in 2D\nAnd we can calculate the distant of two point using Cosine-get the nearest neighbor.\n\nStep 6-Use this system/model to predict hibernating users-User 5 recommendation.\n\nConcept based on -\n\nhttps://www.youtube.com/watch?v=E8aMcwmqsTg \n\n\n\nhttps://www.analyticsvidhya.com/blog/2021/07/recommendation-system-understanding-the-basic-concepts/#:~:text=A%20recommendation%20system%20is%20a,suggests%20relevant%20items%20to%20users.","metadata":{"id":"aCwziCbEiSSk"}},{"cell_type":"markdown","source":"![](https://github.com/techanalyst84/customer-analytics-project/blob/main/Recommendator%201.jpg?raw=true)","metadata":{}},{"cell_type":"markdown","source":"![](https://github.com/techanalyst84/customer-analytics-project/blob/main/Recommendator%202.JPG?raw=true)","metadata":{}},{"cell_type":"markdown","source":"![](https://github.com/techanalyst84/customer-analytics-project/blob/main/Recommendator%203.JPG?raw=true)","metadata":{"id":"194hOcuxiSSl"}},{"cell_type":"code","source":" ","metadata":{"id":"MVhR-kVGiSSp"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Item-Based Collaborative Filtering -using Probabilistic Matrix Factorization\n\n","metadata":{"id":"tQcnd5_eiSSp"}},{"cell_type":"markdown","source":"**Preparing the data** \nWe need to restrict the data respect to a minimum transaction date. In that way, we reduce the dimensionality of the problem and we get rid of transactions that are not important in terms of the time decaying popularity.\n\nAlso, we are getting rid of articles that have not been bought enough. (Minimum 10 purchases are required)\n\n\nhttps://www.kaggle.com/code/luisrodri97/item-based-collaborative-filtering","metadata":{"id":"G5IenImeiSSp"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport datetime\nfrom tqdm import tqdm","metadata":{"executionInfo":{"elapsed":12,"status":"aborted","timestamp":1651256340248,"user":{"displayName":"Suneeth Sreedharan","userId":"09929491400446734699"},"user_tz":240},"id":"zH6MK-OKiSS_","execution":{"iopub.status.busy":"2022-05-06T16:00:20.010205Z","iopub.execute_input":"2022-05-06T16:00:20.010499Z","iopub.status.idle":"2022-05-06T16:00:20.013753Z","shell.execute_reply.started":"2022-05-06T16:00:20.010472Z","shell.execute_reply":"2022-05-06T16:00:20.013279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rfm=rfm.reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:20.014926Z","iopub.execute_input":"2022-05-06T16:00:20.015201Z","iopub.status.idle":"2022-05-06T16:00:20.088432Z","shell.execute_reply.started":"2022-05-06T16:00:20.015176Z","shell.execute_reply":"2022-05-06T16:00:20.087944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" transactions.head(1)","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:20.089687Z","iopub.execute_input":"2022-05-06T16:00:20.09062Z","iopub.status.idle":"2022-05-06T16:00:20.101796Z","shell.execute_reply.started":"2022-05-06T16:00:20.090585Z","shell.execute_reply":"2022-05-06T16:00:20.101185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions=pd.merge(transactions,rfm[[\"customer_id\",\"segment\"]],how='inner',on='customer_id')\ntraining_segment = ['champions', 'potential_loyalists', 'new_customers','promising','loyal_customers']\ntransactions = transactions[transactions['segment'].isin(training_segment)]\ntransactions=transactions.drop('segment', axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:20.103456Z","iopub.execute_input":"2022-05-06T16:00:20.104471Z","iopub.status.idle":"2022-05-06T16:00:32.007518Z","shell.execute_reply.started":"2022-05-06T16:00:20.104439Z","shell.execute_reply":"2022-05-06T16:00:32.006696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_date = datetime.datetime(2020,9,1)\n# Filter transactions by date\ntransactions[\"t_dat\"] = pd.to_datetime(transactions[\"InvoiceDate\"])\ntransactions = transactions.loc[transactions[\"InvoiceDate\"] >= start_date]","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:32.00881Z","iopub.execute_input":"2022-05-06T16:00:32.009009Z","iopub.status.idle":"2022-05-06T16:00:32.544335Z","shell.execute_reply.started":"2022-05-06T16:00:32.008981Z","shell.execute_reply":"2022-05-06T16:00:32.543419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# Filter transactions by number of an article has been bought\narticle_bought_count = transactions[['article_id', 'InvoiceDate']].groupby('article_id').count().reset_index().rename(columns={'InvoiceDate': 'count'})\nmost_bought_articles = article_bought_count[article_bought_count['count']>10]['article_id'].values\ntransactions = transactions[transactions['article_id'].isin(most_bought_articles)]\ntransactions[\"bought\"]=1 ","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:32.548087Z","iopub.execute_input":"2022-05-06T16:00:32.548325Z","iopub.status.idle":"2022-05-06T16:00:33.03433Z","shell.execute_reply.started":"2022-05-06T16:00:32.548295Z","shell.execute_reply":"2022-05-06T16:00:33.033155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"Due to the big amount of items, we can not consider the whole matrix in order to train. Therefore, we need to generate some negative samples: transactions that have never occured.\n\n","metadata":{}},{"cell_type":"code","source":"# Generate negative samples\nnp.random.seed(0)\n\nnegative_samples = pd.DataFrame({\n    'article_id': np.random.choice(transactions.article_id.unique(), transactions.shape[0]),\n    'customer_id': np.random.choice(transactions.customer_id.unique(), transactions.shape[0]),\n    'bought': np.zeros(transactions.shape[0])\n})","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:33.035735Z","iopub.execute_input":"2022-05-06T16:00:33.035951Z","iopub.status.idle":"2022-05-06T16:00:33.588994Z","shell.execute_reply.started":"2022-05-06T16:00:33.035923Z","shell.execute_reply":"2022-05-06T16:00:33.588419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"Model will be based on recommendations computed through the time decaying popularity and the most similar items to those items bought the most times by each user. Similarity among items is computed through cosine distance.\n\n","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics.pairwise import cosine_similarity\n\n\nclass ItemBased_RecSys:\n    ''' Collaborative filtering using a custom sim(u,u'). '''\n\n    def __init__(self, positive_transactions, negative_transactions, num_components=10):\n        ''' Constructor '''\n        self.positive_transactions = positive_transactions\n        self.transactions = pd.concat([positive_transactions, negative_transactions])\n        self.customers = self.transactions.customer_id.values\n        self.articles = self.transactions.article_id.values\n        self.bought = self.transactions.bought.values\n        self.num_components = num_components\n\n        self.customer_id2index = {c: i for i, c in enumerate(np.unique(self.customers))}\n        self.article_id2index = {a: i for i, a in enumerate(np.unique(self.articles))}\n        \n    def __sdg__(self):\n        for idx in tqdm(self.training_indices):\n            # Get the current sample\n            customer_id = self.customers[idx]\n            article_id = self.articles[idx]\n            bought = self.bought[idx]\n\n            # Get the index of the user and the article\n            customer_index = self.customer_id2index[customer_id]\n            article_index = self.article_id2index[article_id]\n\n            # Compute the prediction and the error\n            prediction = self.predict_single(customer_index, article_index)\n            error = (bought - prediction) # error\n            \n            # Update latent factors in terms of the learning rate and the observed error\n            self.customers_latent_matrix[customer_index] += self.learning_rate * \\\n                                    (error * self.articles_latent_matrix[article_index] - \\\n                                     self.lmbda * self.customers_latent_matrix[customer_index])\n            self.articles_latent_matrix[article_index] += self.learning_rate * \\\n                                    (error * self.customers_latent_matrix[customer_index] - \\\n                                     self.lmbda * self.articles_latent_matrix[article_index])\n                \n                \n    def fit(self, n_epochs=10, learning_rate=0.001, lmbda=0.1):\n        ''' Compute the matrix factorization R = P x Q '''\n        self.learning_rate = learning_rate\n        self.lmbda = lmbda\n        n_samples = self.transactions.shape[0]\n        \n        # Initialize latent matrices\n        self.customers_latent_matrix = np.random.normal(scale=1., size=(len(np.unique(self.customers)), self.num_components))\n        self.articles_latent_matrix = np.random.normal(scale=1., size=(len(np.unique(self.articles)), self.num_components))\n\n        for epoch in range(n_epochs):\n            print('Epoch: {}'.format(epoch))\n            self.training_indices = np.arange(n_samples)\n            \n            # Shuffle training samples and follow stochastic gradient descent\n            np.random.shuffle(self.training_indices)\n            self.__sdg__()\n\n    def predict_single(self, customer_index, article_index):\n        ''' Make a prediction for an specific user and article '''\n        prediction = np.dot(self.customers_latent_matrix[customer_index], self.articles_latent_matrix[article_index])\n        prediction = np.clip(prediction, 0, 1)\n        \n        return prediction\n\n    def default_recommendation(self):\n        ''' Calculate time decaying popularity '''\n        # Calculate time decaying popularity. This leads to items bought more recently having more weight in the popularity list.\n        # In simple words, item A bought 5 times on the first day of the train period is inferior than item B bought 4 times on the last day of the train period.\n        self.positive_transactions['pop_factor'] = self.positive_transactions['t_dat'].apply(lambda x: 1/(datetime.datetime(2020,9,23) - x).days)\n        transactions_by_article = self.positive_transactions[['article_id', 'pop_factor']].groupby('article_id').sum().reset_index()\n        return transactions_by_article.sort_values(by='pop_factor', ascending=False)['article_id'].values[:12]\n\n\n    def predict(self, customers):\n        ''' Make recommendations '''\n        recommendations = []\n        self.articles_latent_matrix[np.isnan(self.articles_latent_matrix)] = 0\n        # Compute similarity matrix (cosine)\n        similarity_matrix = cosine_similarity(self.articles_latent_matrix, self.articles_latent_matrix, dense_output=False)\n\n        # Convert similarity matrix into a matrix containing the 12 most similar items' index for each item\n        similarity_matrix = np.argsort(similarity_matrix, axis=1)\n        similarity_matrix = similarity_matrix[:, -12:]\n\n        # Get default recommendation (time decay popularity)\n        default_recommendation = self.default_recommendation()\n\n        # Group articles by user and articles to compute the number of times each article has been bought by each user\n        transactions_by_customer = self.positive_transactions[['customer_id', 'article_id', 'bought']].groupby(['customer_id', 'article_id']).count().reset_index()\n        most_bought_article = transactions_by_customer.loc[transactions_by_customer.groupby('customer_id').bought.idxmax()]['article_id'].values\n\n        # Make predictions\n        for customer in tqdm(customers):\n            try:\n                rec_aux1 = []\n                rec_aux2 = []\n                aux = []\n\n                # Retrieve the most bought article by customer\n                user_most_bought_article_id = most_bought_article[self.customer_id2index[customer]]\n\n                # Using the similarity matrix, get the 6 most similar articles\n                rec_aux1 = self.articles[similarity_matrix[self.article_id2index[user_most_bought_article_id]]]\n                # Return the half of the default recommendation\n                rec_aux2 = default_recommendation\n\n                # Merge half of both recommendation lists\n                for rec_idx in range(6):\n                    aux.append(rec_aux2[rec_idx])\n                    aux.append(rec_aux1[rec_idx])\n\n                recommendations.append(' '.join(aux))\n            except:\n                # Return the default recommendation\n                recommendations.append(' '.join(default_recommendation))\n        \n        return pd.DataFrame({\n            'customer_id': customers,\n            'prediction': recommendations,\n        })","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:33.590139Z","iopub.execute_input":"2022-05-06T16:00:33.591485Z","iopub.status.idle":"2022-05-06T16:00:33.623296Z","shell.execute_reply.started":"2022-05-06T16:00:33.591454Z","shell.execute_reply":"2022-05-06T16:00:33.622692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"Define your hyperparameters and fit the model. Take into account that there are more customizable parameters in the data processing section.\n\n","metadata":{}},{"cell_type":"code","source":"rec = ItemBased_RecSys(transactions, negative_samples, num_components=1000)\nrec.fit(n_epochs=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:00:33.624434Z","iopub.execute_input":"2022-05-06T16:00:33.625413Z","iopub.status.idle":"2022-05-06T16:02:02.965197Z","shell.execute_reply.started":"2022-05-06T16:00:33.625325Z","shell.execute_reply":"2022-05-06T16:02:02.964138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv'\n                       ,encoding=\"ISO-8859-1\", dtype={'article_id':str},header=0  ).customer_id.unique()","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:02:02.966287Z","iopub.execute_input":"2022-05-06T16:02:02.966695Z","iopub.status.idle":"2022-05-06T16:02:09.578204Z","shell.execute_reply.started":"2022-05-06T16:02:02.966636Z","shell.execute_reply":"2022-05-06T16:02:09.577101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"recommendations = rec.predict(customers)","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:02:09.579573Z","iopub.execute_input":"2022-05-06T16:02:09.5798Z","iopub.status.idle":"2022-05-06T16:02:48.278352Z","shell.execute_reply.started":"2022-05-06T16:02:09.579771Z","shell.execute_reply":"2022-05-06T16:02:48.277148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"recommendations.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-06T16:02:48.279732Z","iopub.execute_input":"2022-05-06T16:02:48.279969Z","iopub.status.idle":"2022-05-06T16:02:48.290458Z","shell.execute_reply.started":"2022-05-06T16:02:48.279939Z","shell.execute_reply":"2022-05-06T16:02:48.2893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ","metadata":{"execution":{"iopub.status.busy":"2022-05-02T19:56:20.751491Z","iopub.execute_input":"2022-05-02T19:56:20.751821Z","iopub.status.idle":"2022-05-02T19:56:35.348237Z","shell.execute_reply.started":"2022-05-02T19:56:20.751773Z","shell.execute_reply":"2022-05-02T19:56:35.346845Z"}}}]}