{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Popular Items Benchmark\n\nI am having issues downloading files so this notebook runs the benchmark on it's own and then submits.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"**Imports**","metadata":{}},{"cell_type":"code","source":"from tqdm import tqdm\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2022-12-16T19:50:02.556363Z","iopub.execute_input":"2022-12-16T19:50:02.556819Z","iopub.status.idle":"2022-12-16T19:50:02.562498Z","shell.execute_reply.started":"2022-12-16T19:50:02.556783Z","shell.execute_reply":"2022-12-16T19:50:02.561278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read the Data\n\nWe are using [Radek Osmulski]'s(https://www.kaggle.com/radek1)'s memory optimized [Parquet dataset](https://www.kaggle.com/datasets/radek1/otto-full-optimized-memory-footprint) for the data loading.","metadata":{}},{"cell_type":"code","source":"train = pd.read_parquet('../input/otto-full-optimized-memory-footprint/train.parquet')\ntest = pd.read_parquet('../input/otto-full-optimized-memory-footprint/test.parquet')","metadata":{"execution":{"iopub.status.busy":"2022-12-16T19:50:08.030794Z","iopub.execute_input":"2022-12-16T19:50:08.031385Z","iopub.status.idle":"2022-12-16T19:50:26.272679Z","shell.execute_reply.started":"2022-12-16T19:50:08.031337Z","shell.execute_reply":"2022-12-16T19:50:26.271247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_data = pd.concat([train, test])","metadata":{"execution":{"iopub.status.busy":"2022-12-16T19:53:37.603450Z","iopub.execute_input":"2022-12-16T19:53:37.603924Z","iopub.status.idle":"2022-12-16T19:53:40.853281Z","shell.execute_reply.started":"2022-12-16T19:53:37.603882Z","shell.execute_reply":"2022-12-16T19:53:40.852044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Determine the articles which were most often clicked, added to cart, and ordered.","metadata":{}},{"cell_type":"code","source":"clicks_counts = full_data[full_data['type'] == 0]['aid'].value_counts()\ncarts_counts = full_data[full_data['type'] == 1]['aid'].value_counts()\norders_counts = full_data[full_data['type'] == 2]['aid'].value_counts()\n\nclicks_list = list(clicks_counts[:20].index)\ncarts_list = list(carts_counts[:20].index)\norders_list = list(orders_counts[:20].index)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T19:53:40.855118Z","iopub.execute_input":"2022-12-16T19:53:40.855721Z","iopub.status.idle":"2022-12-16T19:53:56.949883Z","shell.execute_reply.started":"2022-12-16T19:53:40.855680Z","shell.execute_reply":"2022-12-16T19:53:56.948681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Read the test sessions in","metadata":{}},{"cell_type":"code","source":"test_sessions = pd.read_json('/kaggle/input/otto-recommender-system/test.jsonl', lines=True, chunksize=100000)\n\ntest_session_ids = []\nfor chunk in tqdm(test_sessions):\n    for session_id, events in chunk.values:\n        test_session_ids.append(session_id)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T19:54:24.497814Z","iopub.execute_input":"2022-12-16T19:54:24.498262Z","iopub.status.idle":"2022-12-16T19:54:54.840306Z","shell.execute_reply.started":"2022-12-16T19:54:24.498223Z","shell.execute_reply":"2022-12-16T19:54:54.838973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Write the predictions to CSV","metadata":{"execution":{"iopub.status.busy":"2022-12-16T19:53:30.820119Z","iopub.execute_input":"2022-12-16T19:53:30.821030Z","iopub.status.idle":"2022-12-16T19:53:30.830758Z","shell.execute_reply.started":"2022-12-16T19:53:30.820981Z","shell.execute_reply":"2022-12-16T19:53:30.828540Z"}}},{"cell_type":"code","source":"SUBMISSION_FILENAME = 'submission.csv'\n\nwith open(SUBMISSION_FILENAME, 'w') as file:\n    file.write(\"session_type,labels\\n\")\n    for session_id in tqdm(test_session_ids):\n        file.write(\"%d_clicks,\" % session_id \\\n                   + \" \".join([str(c) for c in clicks_list]) + \"\\n\")\n        file.write(\"%d_carts,\" % session_id \\\n                   + \" \".join([str(c) for c in carts_list]) + \"\\n\")\n        file.write(\"%d_orders,\" % session_id \\\n                   + \" \".join([str(c) for c in orders_list]) + \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-12-16T19:56:13.475908Z","iopub.execute_input":"2022-12-16T19:56:13.476417Z","iopub.status.idle":"2022-12-16T19:56:44.052902Z","shell.execute_reply.started":"2022-12-16T19:56:13.476379Z","shell.execute_reply":"2022-12-16T19:56:44.051688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This benchmark is garbage! Score of 0.007... RIP!","metadata":{}},{"cell_type":"markdown","source":"## Hypothesis About Low Score\n\nI was expecting this to score higher, but I think I have an explaination for why it doesn't: The highest weighted category is orders, which is unlikely to be correct due to the following order counts statistics.","metadata":{}},{"cell_type":"code","source":"print(\"%d articles ordered\" % len(orders_counts))\nprint(\"Mean # of orders: %.4f\" % orders_counts.mean())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T20:08:13.610640Z","iopub.execute_input":"2022-12-16T20:08:13.611761Z","iopub.status.idle":"2022-12-16T20:08:13.618842Z","shell.execute_reply.started":"2022-12-16T20:08:13.611719Z","shell.execute_reply":"2022-12-16T20:08:13.617846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nn_orders = 500\nplt.bar(range(len(orders_counts[:n_orders])), orders_counts[:n_orders])\nplt.title(\"%d most ordered items.\" % n_orders)\nplt.ylabel(\"Number of orders\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T20:09:35.555040Z","iopub.execute_input":"2022-12-16T20:09:35.556755Z","iopub.status.idle":"2022-12-16T20:09:36.595522Z","shell.execute_reply.started":"2022-12-16T20:09:35.556686Z","shell.execute_reply":"2022-12-16T20:09:36.594073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Because we are making predictions for some 1671803 users, each of whom only makes a few orders on average, it is unlikely that they will order one of the most frequently ordered items.","metadata":{}}]}