{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <span style='color:#8A0808'>🥼H&M fast EDA and Memory reduction🎈</span>","metadata":{}},{"cell_type":"markdown","source":"# <span style='color:#8A0808'>📚Introduction</span>\n\n## <span style='color:#4A0404'>🎯Goal</span>\n\nIn this competition, H&M Group invites you to develop product recommendations based on data from previous transactions, as well as from customer and product meta data.\n\n## <span style='color:#4A0404'>💾Data</span>\n\nFor this challenge you are given the purchase history of customers across time, along with supporting metadata. Your challenge is to predict what articles each customer will purchase in the 7-day period immediately after the training data ends. Customer who did not make any purchase during that time are excluded from the scoring.\n\n**Files**\n* images/ - a folder of images corresponding to each article_id; images are placed in subfolders starting with the first three digits of the article_id; note, not all article_id values have a corresponding image.\n* articles.csv - detailed metadata for each article_id available for purchase\n* customers.csv - metadata for each customer_id in dataset\n* sample_submission.csv - a sample submission file in the correct format\n* transactions_train.csv - the training data, consisting of the purchases each customer for each date, as well as additional information. Duplicate rows correspond to multiple purchases of the same item. Your task is to predict the article_ids each customer will purchase during the 7-day period immediately after the training data period.\n\n**NOTE**: You must make predictions for all customer_id values found in the sample submission. All customers who made purchases during the test period are scored, regardless of whether they had purchase history in the training data.\n\n## <span style='color:#4A0404'>🔑Metric</span>\n\nSubmissions are evaluated according to the Mean Average Precision @ 12 (MAP@12):\n\n## <span style='color:blue'>$MAP@12=\\frac{1}{U}\\sum_{u=1}^U \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k)$</span>\n\nwhere $U$ is the number of customers, $P(k)$ is the precision at cutoff $k$, $n$ is the number predictions per customer, and $rel(k)$ is an indicator function equaling 1 if the item at rank  is a relevant (correct) label, zero otherwise.\n\n**Notes**:\n\nYou will be making purchase predictions for all customer_id values provided, regardless of whether these customers made purchases in the training data.\nCustomer that did not make any purchase during test period are excluded from the scoring.\nThere is never a penalty for using the full 12 predictions for a customer that ordered fewer than 12 items; thus, it's advantageous to make 12 predictions for each customer.","metadata":{}},{"cell_type":"markdown","source":"# <span style='color:#8A0808'>⚡Fast EDA</span>","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n\nimport os","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Articles","metadata":{}},{"cell_type":"markdown","source":"There are 105542 articles","metadata":{}},{"cell_type":"code","source":"articles = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/articles.csv')\nprint(f'articles shape {articles.shape}:\\n{articles.loc[0,:]}')","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:09:05.3797Z","iopub.execute_input":"2022-02-07T22:09:05.380206Z","iopub.status.idle":"2022-02-07T22:09:06.092804Z","shell.execute_reply.started":"2022-02-07T22:09:05.380171Z","shell.execute_reply":"2022-02-07T22:09:06.091815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.info()","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:18:40.430594Z","iopub.execute_input":"2022-02-07T22:18:40.431087Z","iopub.status.idle":"2022-02-07T22:18:40.624878Z","shell.execute_reply.started":"2022-02-07T22:18:40.431048Z","shell.execute_reply":"2022-02-07T22:18:40.623773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Articles are grouped into 762 groups. Just 86 article groups have images","metadata":{}},{"cell_type":"code","source":"print('Number of article groups:', articles.article_id.apply(lambda x: str(x)[:3]).nunique())\nprint('Number of image groups:', len(os.listdir('../input/h-and-m-personalized-fashion-recommendations/images')))","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:40:14.51429Z","iopub.execute_input":"2022-02-07T22:40:14.515179Z","iopub.status.idle":"2022-02-07T22:40:14.605935Z","shell.execute_reply.started":"2022-02-07T22:40:14.515135Z","shell.execute_reply":"2022-02-07T22:40:14.605227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show some articles in a same group","metadata":{}},{"cell_type":"code","source":"article_group = '010'\nimpath = f'/kaggle/input/h-and-m-personalized-fashion-recommendations/images/{article_group}'\n\nplt.figure(figsize=(10,10))\nfor idx, file in enumerate(os.listdir(impath)):\n    plt.subplot(2,2,idx+1)\n    plt.imshow(mpimg.imread(f'{impath}/{file}'))","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:46:36.195385Z","iopub.execute_input":"2022-02-07T22:46:36.195742Z","iopub.status.idle":"2022-02-07T22:46:37.654326Z","shell.execute_reply.started":"2022-02-07T22:46:36.195709Z","shell.execute_reply":"2022-02-07T22:46:37.653308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Why the articles below are in a same group?","metadata":{}},{"cell_type":"code","source":"article_group = '039'\nimpath = f'/kaggle/input/h-and-m-personalized-fashion-recommendations/images/{article_group}'\n\nplt.figure(figsize=(10,10))\nfor idx, file in enumerate(os.listdir(impath)):\n    plt.subplot(2,2,idx+1)\n    plt.imshow(mpimg.imread(f'{impath}/{file}'))\n    if idx>2: break","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:47:43.793329Z","iopub.execute_input":"2022-02-07T22:47:43.793695Z","iopub.status.idle":"2022-02-07T22:47:45.715225Z","shell.execute_reply.started":"2022-02-07T22:47:43.79366Z","shell.execute_reply":"2022-02-07T22:47:45.714627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Customers","metadata":{}},{"cell_type":"markdown","source":"There are 1371980 customers","metadata":{}},{"cell_type":"code","source":"customers = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/customers.csv')\nprint(f'customers shape {customers.shape}:\\n{customers.loc[0,:]}')","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:10:10.422691Z","iopub.execute_input":"2022-02-07T22:10:10.423674Z","iopub.status.idle":"2022-02-07T22:10:14.26021Z","shell.execute_reply.started":"2022-02-07T22:10:10.423617Z","shell.execute_reply":"2022-02-07T22:10:14.259252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Customer age has a bi-modal distribution (young/old)","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,5))\ncustomers.age.hist(bins=100);","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:50:48.862759Z","iopub.execute_input":"2022-02-07T22:50:48.863407Z","iopub.status.idle":"2022-02-07T22:50:49.312056Z","shell.execute_reply.started":"2022-02-07T22:50:48.863354Z","shell.execute_reply":"2022-02-07T22:50:49.311491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Transactions","metadata":{}},{"cell_type":"markdown","source":"There are 31788324 transactions in the train set","metadata":{}},{"cell_type":"code","source":"%%time\ntransactions = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv',\n                           dtype={'article_id': str},\n                           low_memory=True)\nprint(f'transactions shape {transactions.shape}:\\n{transactions.loc[0,:]}')","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:11:25.467017Z","iopub.execute_input":"2022-02-07T22:11:25.46808Z","iopub.status.idle":"2022-02-07T22:12:34.455716Z","shell.execute_reply.started":"2022-02-07T22:11:25.468019Z","shell.execute_reply":"2022-02-07T22:12:34.454743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 104547 articles in the train set (over the total number of articles of 105542) = 99.06%","metadata":{}},{"cell_type":"code","source":"transactions.article_id.nunique()/105542*100","metadata":{"execution":{"iopub.status.busy":"2022-02-07T23:02:54.310272Z","iopub.execute_input":"2022-02-07T23:02:54.311253Z","iopub.status.idle":"2022-02-07T23:02:54.662769Z","shell.execute_reply.started":"2022-02-07T23:02:54.311197Z","shell.execute_reply":"2022-02-07T23:02:54.661613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 1362281 customers in the train set (over the total number of customers of 1371980) = 99.29%","metadata":{}},{"cell_type":"code","source":"transactions.customer_id.nunique()/1371980*100","metadata":{"execution":{"iopub.status.busy":"2022-02-07T23:02:18.681215Z","iopub.execute_input":"2022-02-07T23:02:18.681796Z","iopub.status.idle":"2022-02-07T23:02:27.879507Z","shell.execute_reply.started":"2022-02-07T23:02:18.681746Z","shell.execute_reply":"2022-02-07T23:02:27.878711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are two sale channels","metadata":{}},{"cell_type":"code","source":"transactions.sales_channel_id.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:51:38.515784Z","iopub.execute_input":"2022-02-07T22:51:38.516172Z","iopub.status.idle":"2022-02-07T22:51:38.710265Z","shell.execute_reply.started":"2022-02-07T22:51:38.516139Z","shell.execute_reply":"2022-02-07T22:51:38.709328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 69.48% and 93.16% articles in the first and second sale channels, respectively.","metadata":{}},{"cell_type":"code","source":"print('Number of articles in the first sale channel:', transactions.article_id[transactions.sales_channel_id==1].nunique()/105542*100)\nprint('Number of articles in the second sale channel:', transactions.article_id[transactions.sales_channel_id==2].nunique()/105542*100)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T23:05:34.166289Z","iopub.execute_input":"2022-02-07T23:05:34.167216Z","iopub.status.idle":"2022-02-07T23:05:35.195671Z","shell.execute_reply.started":"2022-02-07T23:05:34.167167Z","shell.execute_reply":"2022-02-07T23:05:35.194837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 53.73% and 80.79% customers in the first and second sale channels, respectively. A large amount of customers buy articles via both channels.","metadata":{}},{"cell_type":"code","source":"print('Number of customers in the first sale channel:', transactions.customer_id[transactions.sales_channel_id==1].nunique()/1371980*100)\nprint('Number of customers in the second sale channel:', transactions.customer_id[transactions.sales_channel_id==2].nunique()/1371980*100)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T23:06:41.621527Z","iopub.execute_input":"2022-02-07T23:06:41.6219Z","iopub.status.idle":"2022-02-07T23:06:52.237943Z","shell.execute_reply.started":"2022-02-07T23:06:41.621859Z","shell.execute_reply":"2022-02-07T23:06:52.23685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prices distribution (prices are scaled)","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,5))\ntransactions.price.hist(bins=100);\nplt.xlim(0,0.2)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:53:02.957925Z","iopub.execute_input":"2022-02-07T22:53:02.958295Z","iopub.status.idle":"2022-02-07T22:53:04.294502Z","shell.execute_reply.started":"2022-02-07T22:53:02.95825Z","shell.execute_reply":"2022-02-07T22:53:04.293682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv')\nprint(f'submission shape {submission.shape}:\\n{submission.loc[0,:]}')","metadata":{"execution":{"iopub.status.busy":"2022-02-07T22:14:24.839113Z","iopub.execute_input":"2022-02-07T22:14:24.839601Z","iopub.status.idle":"2022-02-07T22:14:29.682785Z","shell.execute_reply.started":"2022-02-07T22:14:24.839567Z","shell.execute_reply":"2022-02-07T22:14:29.681918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span style='color#A80808'>🎈Memory reduction</span>","metadata":{}},{"cell_type":"markdown","source":"Convert csv to pickle, parquet, feather to gain some memory space","metadata":{}},{"cell_type":"code","source":"articles.to_pickle('articles.pkl')\ncustomers.to_pickle('customers.pkl')\ntransactions.to_pickle('transactions_train.pkl')\nsubmission.to_pickle('sample_submission.pkl')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.to_parquet('articles.parquet')\ncustomers.to_parquet('customers.parquet')\ntransactions.to_parquet('transactions_train.parquet')\nsubmission.to_parquet('sample_submission.parquet')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.to_feather('articles.feather')\ncustomers.to_feather('customers.feather')\ntransactions.to_feather('transactions_train.feather')\nsubmission.to_feather('sample_submission.feather')","metadata":{},"execution_count":null,"outputs":[]}]}