{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook is an attempt to heuristically find promoted products and the corresponding day/week of promotion.\n\nIts largerly based on this: [nice notebook](https://www.kaggle.com/code/adaubas/otto-interesting-times-series-eda-on-products).\n\nHope you enjoy.","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we will display time series EDA for products which have the most clicks/carts/orders. <br>\nWe will use RAPIDS cuDF to process dataframes and seaborn to display EDA. <br>\n\nMany products doesn't exhibit regular pattern.<br>\nA product with a lot of clicks/carts/orders one week, will not have necessary a lot of clicks/carts/orders next week. <br>\n\nLet's have a look to AID n°485256 below : almost 97 000 sessions with a click on this product day 235, an no click all other days...<br>\n\nSo there are many outliers : we should build models and make predictions only for products which have some regular patterns, otherwise we will overfit. <br>\nDo you agree ?<br>\n\nThere is a Kaggle discussion about those strange product time series [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368373).<br>","metadata":{}},{"cell_type":"markdown","source":"# Libraries & Train data","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\npd.set_option('max_columns', 100)\npd.set_option('max_rows', 200)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cudf, cupy\nprint('Using RAPIDS version',cudf.__version__)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-11-24T22:05:35.296007Z","iopub.execute_input":"2022-11-24T22:05:35.296728Z","iopub.status.idle":"2022-11-24T22:05:38.12497Z","shell.execute_reply.started":"2022-11-24T22:05:35.296688Z","shell.execute_reply":"2022-11-24T22:05:38.12383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain = cudf.read_parquet('../input/otto-full-optimized-memory-footprint/train.parquet')\ntrain.ts = cudf.to_datetime((train.ts + 2*60*60) * 1e9)\nprint('Train min date and max date are:', train.ts.min(),'and', train.ts.max())","metadata":{"execution":{"iopub.status.busy":"2022-11-24T22:05:38.126958Z","iopub.execute_input":"2022-11-24T22:05:38.127277Z","iopub.status.idle":"2022-11-24T22:05:59.795353Z","shell.execute_reply.started":"2022-11-24T22:05:38.127247Z","shell.execute_reply":"2022-11-24T22:05:59.793522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"dayofyear\"] = (train.ts.dt.dayofyear).astype(np.int16)\ntrain[\"weekofyear\"] = ((train[\"dayofyear\"]-3)//7).astype(np.int8)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-24T22:05:59.796607Z","iopub.execute_input":"2022-11-24T22:05:59.796953Z","iopub.status.idle":"2022-11-24T22:05:59.909749Z","shell.execute_reply.started":"2022-11-24T22:05:59.796923Z","shell.execute_reply":"2022-11-24T22:05:59.908636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"weekofyear\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-11-24T22:05:59.91269Z","iopub.execute_input":"2022-11-24T22:05:59.913044Z","iopub.status.idle":"2022-11-24T22:06:00.092504Z","shell.execute_reply.started":"2022-11-24T22:05:59.913008Z","shell.execute_reply":"2022-11-24T22:06:00.091457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Session with AID by week, day and type\n\nWe want to know in how many sessions, an AID is, so I'm using .count()","metadata":{}},{"cell_type":"code","source":"aid_per_week = train.groupby([\"aid\", \"weekofyear\", \"type\"])[\"session\"].count().reset_index()\naid_per_day = train.groupby([\"aid\", \"dayofyear\", 'type'])[\"session\"].count().reset_index()\ndisplay(aid_per_week.head(10))\ndisplay(aid_per_day.head(10))","metadata":{"execution":{"iopub.status.busy":"2022-11-24T22:06:00.094267Z","iopub.execute_input":"2022-11-24T22:06:00.094623Z","iopub.status.idle":"2022-11-24T22:06:00.893949Z","shell.execute_reply.started":"2022-11-24T22:06:00.094586Z","shell.execute_reply":"2022-11-24T22:06:00.893058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# An attempt to detect promoted items\n\nI calculate the std in session for the dataframe <code>aid_per_day</code>. The larger the std, the more probable we have a promoted item. \n\nWe can use that information for example to introduce a weight in co visitation matrix for the promoted day!\n\nAlso we can use <code>aid_per_week</code> and in similar way find promoted weeks!","metadata":{}},{"cell_type":"code","source":"std_in_session = aid_per_day.loc[aid_per_day.type==2].groupby(\"aid\").agg({\"session\":\"std\"}).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T13:12:28.398516Z","iopub.execute_input":"2022-12-16T13:12:28.398994Z","iopub.status.idle":"2022-12-16T13:12:28.470463Z","shell.execute_reply.started":"2022-12-16T13:12:28.398926Z","shell.execute_reply":"2022-12-16T13:12:28.469192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"std_in_session.fillna(-1).sort_values(\"session\").tail(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.options.display.max_rows = 999","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aid_per_day.loc[(aid_per_day[\"aid\"]==876493) & (aid_per_day[\"type\"]==2)].sort_values([\"aid\",\"dayofyear\",\"type\"])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aid_per_day.loc[(aid_per_day[\"aid\"]==1603001) & (aid_per_day[\"type\"]==2)].sort_values([\"aid\",\"dayofyear\",\"type\"])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aid_per_day.loc[(aid_per_day[\"aid\"]==1083665) & (aid_per_day[\"type\"]==2)].sort_values([\"aid\",\"dayofyear\",\"type\"])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aid_per_day.loc[(aid_per_day[\"aid\"]==80222) & (aid_per_day[\"type\"]==2)].sort_values([\"aid\",\"dayofyear\",\"type\"])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aid_per_day.loc[(aid_per_day[\"aid\"]==1406660) & (aid_per_day[\"type\"]==2)].sort_values([\"aid\",\"dayofyear\",\"type\"])","metadata":{},"execution_count":null,"outputs":[]}]}