{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"The purpose of this notebook is to become familiar with datasets.\n\n1. The data set used for analysis has a minimized size, more about it [here](https://www.kaggle.com/datasets/radek1/otto-full-optimized-memory-footprint).","metadata":{}},{"cell_type":"code","source":"!nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2023-01-08T14:43:03.537538Z","iopub.execute_input":"2023-01-08T14:43:03.537915Z","iopub.status.idle":"2023-01-08T14:43:04.766705Z","shell.execute_reply.started":"2023-01-08T14:43:03.537881Z","shell.execute_reply":"2023-01-08T14:43:04.765552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Imports \n# Data processing tools\nimport cudf\nimport pandas as pd\nimport numpy as np\nimport torch\n\n# Visualization tools\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython.display import display, Markdown, HTML, Image\n\n# System\nimport os\nimport gc\nimport warnings\n\n\n\n# Settings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\npd.set_option('display.float_format', lambda x: '%.3f' % x)\n\n# Colors settings\nsns.set()\nmain_color = '#E44747'\nsecond_color = '#FC8963'\nthird_color = '#E5F5C5'\nhighlight_color = '#A74859'\ndark_color = \"#79416A\"\n\nevent_type_map = {0:\"clicks\", 1:\"carts\", 2:\"orders\"}","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:20.709995Z","iopub.execute_input":"2023-01-08T13:48:20.711233Z","iopub.status.idle":"2023-01-08T13:48:24.205457Z","shell.execute_reply.started":"2023-01-08T13:48:20.711110Z","shell.execute_reply":"2023-01-08T13:48:24.204009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Custom function for displaying the output\ndef __display(s):\n '''\n The function for displaying the output as markdown\n '''\n display(Markdown(s))\n\ndef side_by_side(*dfs):\n '''\n The function for display two pd.DataFrames side by side\n '''\n html = '
'\n for df in dfs:\n html += '
'\n html += df.to_html()\n html += '
'\n html += '
'\n display(HTML(html))","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:24.207616Z","iopub.execute_input":"2023-01-08T13:48:24.208526Z","iopub.status.idle":"2023-01-08T13:48:24.216562Z","shell.execute_reply.started":"2023-01-08T13:48:24.208486Z","shell.execute_reply":"2023-01-08T13:48:24.214887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Path to data folder\ndata_path = '/kaggle/input/otto-full-optimized-memory-footprint'","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:24.218309Z","iopub.execute_input":"2023-01-08T13:48:24.218966Z","iopub.status.idle":"2023-01-08T13:48:24.226243Z","shell.execute_reply.started":"2023-01-08T13:48:24.218930Z","shell.execute_reply":"2023-01-08T13:48:24.225245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# Daset overview:","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/otto-img/img/Otto_Main_Info.png\")","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-01-08T13:48:24.229210Z","iopub.execute_input":"2023-01-08T13:48:24.229846Z","iopub.status.idle":"2023-01-08T13:48:24.252329Z","shell.execute_reply.started":"2023-01-08T13:48:24.229812Z","shell.execute_reply":"2023-01-08T13:48:24.251404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Original **dataset link**: https://github.com/otto-de/recsys-dataset\n\n\n> The `OTTO` session dataset is a large-scale dataset intended for multi-objective recommendation research. We collected the data from anonymized behavior logs of the [OTTO](https://otto.de/) webshop and the app. The mission of this dataset is to serve as a benchmark for session-based recommendations and foster research in the multi-objective and session-based recommender systems area. \n\n## Key Features\n* **12M** real-world anonymized user **sessions**\n* **220M** events, consiting of *clicks*, *carts* and *orders*\n* **1.8M** *unique articles* in the catalogue\n* Ready to use data in *.jsonl format*\n* Evaluation metrics for multi-objective optimization","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# Loading Train & Test Datasets","metadata":{}},{"cell_type":"code","source":"# Loading train & test datasets\ntrain = cudf.read_parquet(os.path.join(data_path,'train.parquet'))\ntest = cudf.read_parquet(os.path.join(data_path,'test.parquet'))\n__display(\"**TRAIN** & **TEST**\")\nside_by_side(train.head().to_pandas(), test.head().to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:24.253133Z","iopub.execute_input":"2023-01-08T13:48:24.253434Z","iopub.status.idle":"2023-01-08T13:48:50.115898Z","shell.execute_reply.started":"2023-01-08T13:48:24.253406Z","shell.execute_reply":"2023-01-08T13:48:50.114942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# General info","metadata":{}},{"cell_type":"markdown","source":"## Features\n* `session` - the unique session id (single user)\n* `events` - the time ordered sequence of events in the session\n * `aid` - the article id (product code) of the associated event\n * `ts` - the Unix timestamp of the event\n * `type` - the event type, i.e., whether a product\n * 0 - was clicked, \n * 1 - added to the user's cart,\n * 2 - ordered during the session","metadata":{}},{"cell_type":"markdown","source":"## Number of unique `session`","metadata":{}},{"cell_type":"code","source":"__display(f\"**TRAIN:** Number of unique sessions - **{train.session.nunique()}**\")\n__display(f\"**TEST:** Number of unique sessions - **{test.session.nunique()}**\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:50.118056Z","iopub.execute_input":"2023-01-08T13:48:50.118681Z","iopub.status.idle":"2023-01-08T13:48:50.243128Z","shell.execute_reply.started":"2023-01-08T13:48:50.118645Z","shell.execute_reply":"2023-01-08T13:48:50.242125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Number of `events`","metadata":{}},{"cell_type":"code","source":"__display(f\"**TRAIN:** shape: **{train.shape[0]}**\")\n__display(f\"**TEST:** shape: **{test.shape[0]}**\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:50.244545Z","iopub.execute_input":"2023-01-08T13:48:50.244876Z","iopub.status.idle":"2023-01-08T13:48:50.256986Z","shell.execute_reply.started":"2023-01-08T13:48:50.244843Z","shell.execute_reply":"2023-01-08T13:48:50.256035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Number of unique `products`","metadata":{}},{"cell_type":"code","source":"__display(f\"**TRAIN:** Number of unique product - **{train.aid.nunique()}**\")\n__display(f\"**TEST:** Number of unique product - **{test.aid.nunique()}**\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:50.258410Z","iopub.execute_input":"2023-01-08T13:48:50.258924Z","iopub.status.idle":"2023-01-08T13:48:50.431438Z","shell.execute_reply.started":"2023-01-08T13:48:50.258888Z","shell.execute_reply":"2023-01-08T13:48:50.430397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Summary:\n* The `traian` dataset contains:\n 1. **~12.9M** unique `sessions`\n 2. **~217M** `events`\n 3. **~1.85M** unique `products`\n* When `test` dataset contains:\n 1. **~1.67M** unique `sessions`\n 2. **~6.9M** `events`\n 3. **~783K** unique `products`\n * But this info we already have known from the documentation of the dataset. So let's dive into the datasets and explore each feature.","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# `Sessions` x `TS`","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/otto-img/img/session_time.png\")","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-01-08T13:48:50.432981Z","iopub.execute_input":"2023-01-08T13:48:50.433607Z","iopub.status.idle":"2023-01-08T13:48:50.455523Z","shell.execute_reply.started":"2023-01-08T13:48:50.433570Z","shell.execute_reply":"2023-01-08T13:48:50.453779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The focus of this section would be on the **timestamps** of the events. \n\nThe main **goal** :\n\n> is to explore the time when the customers (`sessions`) are the most active and to investing does there exist any explicit patterns when the users make more `orders`, `clicks`, or `add to their carts`.\n\n","metadata":{}},{"cell_type":"markdown","source":"## The timeline of the datasets\n* The **earliest** and the **latest** session in the datasets","metadata":{}},{"cell_type":"code","source":"# Function for converting the delta between two datetimes into the number of days, hours and minutes\ndef days_hour_min(time_delta):\n return time_delta.item().days, time_delta.item().seconds//3600, (time_delta.item().seconds//60)%60","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:50.460040Z","iopub.execute_input":"2023-01-08T13:48:50.460683Z","iopub.status.idle":"2023-01-08T13:48:50.465631Z","shell.execute_reply.started":"2023-01-08T13:48:50.460649Z","shell.execute_reply":"2023-01-08T13:48:50.464823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_min_date = cudf.to_datetime(train.ts.min(), unit='s')\ntrain_max_date = cudf.to_datetime(train.ts.max(), unit='s')\ntest_min_date = cudf.to_datetime(test.ts.min(), unit='s')\ntest_max_date = cudf.to_datetime(test.ts.max(), unit='s')\n__display(\"---\")\n__display(f\"**Train**: First session time: {train_min_date}\")\n__display(f\"**Train**: Last session time: {train_max_date}\")\n__display(\"**Train**: Amount of days {}, hours {}, minutes {}\".format(*days_hour_min(train_max_date-train_min_date)))\n__display(\"---\")\n__display(f\"**Test**: First session time: {test_min_date}\")\n__display(f\"**Test**: Last session time: {test_max_date}\")\n__display(\"**Test**: Amount of days {}, hours {}, minutes {}\".format(*days_hour_min(test_max_date-test_min_date)))\n__display(\"---\")\ndel train_min_date, train_max_date, test_min_date, test_max_date\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:50.467298Z","iopub.execute_input":"2023-01-08T13:48:50.468259Z","iopub.status.idle":"2023-01-08T13:48:51.655670Z","shell.execute_reply.started":"2023-01-08T13:48:50.468126Z","shell.execute_reply":"2023-01-08T13:48:51.654713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There is no time overlapping between datasets (What also could be proven by the [train/test split](https://github.com/otto-de/recsys-dataset#traintest-split) information from the documentation )","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/otto-img/img/train_test_split.png\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:51.657107Z","iopub.execute_input":"2023-01-08T13:48:51.657571Z","iopub.status.idle":"2023-01-08T13:48:51.669572Z","shell.execute_reply.started":"2023-01-08T13:48:51.657533Z","shell.execute_reply":"2023-01-08T13:48:51.668723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Users activity by day\n* **Q1:** What day of the week has the most events?\n * **Q1.1** What day of the week has the most `clicks`, `add to cart`, and `orders`?\n* **Q2:** When do users make more `orders`?\n","metadata":{}},{"cell_type":"code","source":"# Create `day`, `day of the week` feature from `ts`\ntrain_date_tmp = cudf.to_datetime(train.ts, unit='s')\ntrain['day'] = train_date_tmp.dt.day\ntrain['day_of_week'] = train_date_tmp.dt.dayofweek\n\n\ntest_date_tmp = cudf.to_datetime(test.ts, unit='s')\ntest['day'] = test_date_tmp.dt.day\ntest['day_of_week'] = test_date_tmp.dt.dayofweek\n\ndel train_date_tmp, test_date_tmp\ntorch.cuda.empty_cache()\n\n__display(\"**TRAIN** & **TEST**\")\nside_by_side(train.head().to_pandas(), test.head().to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:51.671958Z","iopub.execute_input":"2023-01-08T13:48:51.672609Z","iopub.status.idle":"2023-01-08T13:48:51.749155Z","shell.execute_reply.started":"2023-01-08T13:48:51.672574Z","shell.execute_reply":"2023-01-08T13:48:51.748119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create tmp dataframe with the day and the day number of the week info \ntrain_day_map_day_of_week = train.groupby('day').day_of_week.first().reset_index().to_pandas().set_index('day')\ntest_day_map_day_of_week = test.groupby('day').day_of_week.first().reset_index().to_pandas().set_index('day')","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:51.750809Z","iopub.execute_input":"2023-01-08T13:48:51.751168Z","iopub.status.idle":"2023-01-08T13:48:51.986900Z","shell.execute_reply.started":"2023-01-08T13:48:51.751135Z","shell.execute_reply":"2023-01-08T13:48:51.985946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the right order of the days and create a color palette for each day. Highlight Sundays\n# TRAIN\ntrain_date_order = [31, *range(1,29)]\ntrain_date_colors = [ main_color if day != 6 else highlight_color for day in train_day_map_day_of_week.loc[train_date_order,'day_of_week']]\n\n# TEST\ntest_date_order = [*range(28,32), *range(1,5)]\ntest_date_colors = [ second_color if day != 6 else highlight_color for day in test_day_map_day_of_week.loc[test_date_order,'day_of_week']]","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:51.988503Z","iopub.execute_input":"2023-01-08T13:48:51.988862Z","iopub.status.idle":"2023-01-08T13:48:51.999186Z","shell.execute_reply.started":"2023-01-08T13:48:51.988827Z","shell.execute_reply":"2023-01-08T13:48:51.996900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Q1:** What day of the week has the most events?","metadata":{}},{"cell_type":"code","source":"# Create the barplot of the amount of event by day\ntrain_date_activity = train.day.value_counts().reset_index().to_pandas().rename(columns={\"index\":\"day\", \"day\":\"count\"})\ntest_date_activity = test.day.value_counts().reset_index().to_pandas().rename(columns={\"index\":\"day\", \"day\":\"count\"})\n\n# Subplots figure\nfig, axs = plt.subplots(1,2, figsize=(20,10),sharey=True,gridspec_kw={'width_ratios': [3, 1]})\n\n# plot train\naxs[0].set_title('Train Activity', weight=\"bold\", size=20)\nsns.barplot(data=train_date_activity, x='day', y='count', palette=train_date_colors, ax=axs[0], order=train_date_order)\n# plot test\naxs[1].set_title('Test Activity', weight=\"bold\", size=20)\nsns.barplot(data=test_date_activity, x='day', y='count', palette=test_date_colors, ax=axs[1], order = test_date_order)\n\nfor ax in axs:\n ax.set_ylabel(\"number of events\", size=15, weight='bold')\n ax.set_xlabel(\"Day\", size=15, weight='bold')\n\ndel fig, axs, ax\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:52.000572Z","iopub.execute_input":"2023-01-08T13:48:52.001544Z","iopub.status.idle":"2023-01-08T13:48:52.988925Z","shell.execute_reply.started":"2023-01-08T13:48:52.001506Z","shell.execute_reply":"2023-01-08T13:48:52.987956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### **A1:** \nFrom the chart above, it appears that **Sunday** has the highest user activity.\n> **But does this mean that most products are sold on Sunday?**\n\n\nLet's check it out!!!","metadata":{}},{"cell_type":"markdown","source":"### **Q1.1** What day of the week has the most `clicks`, `add to cart` and `orders`?","metadata":{}},{"cell_type":"code","source":"# Create dataframe with the day and the amount of each event type.\ntrain_day_events_type_count = train.groupby(['day','type']).agg({'aid':'count'}).reset_index().rename(columns={\"aid\":\"count\"}).sort_values(by=['day','type'], ignore_index=True).to_pandas()\ntest_day_events_type_count = test.groupby(['day','type']).agg({'aid':'count'}).reset_index().rename(columns={\"aid\":\"count\"}).sort_values(by=['day','type'], ignore_index=True).to_pandas()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:52.990617Z","iopub.execute_input":"2023-01-08T13:48:52.991304Z","iopub.status.idle":"2023-01-08T13:48:53.192810Z","shell.execute_reply.started":"2023-01-08T13:48:52.991266Z","shell.execute_reply":"2023-01-08T13:48:53.191860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the barplot, day, and the amount of each event type.\nfig, axs = plt.subplots(3,2, figsize=(20,10),sharey='row',gridspec_kw={'width_ratios': [3, 1]})\nfor i, ax in enumerate(axs):\n # plot train\n sns.barplot(data=train_day_events_type_count.loc[train_day_events_type_count.type==i], x='day', y='count', palette=train_date_colors, ax=ax[0], order=train_date_order)\n # plot test\n sns.barplot(data=test_day_events_type_count.loc[test_day_events_type_count.type==i], x='day', y='count', palette=test_date_colors, ax=ax[1], order = test_date_order)\n \n if i ==0:\n ax[0].set_title('Train Activity', weight=\"bold\", size=20)\n ax[1].set_title('Test Activity', weight=\"bold\", size=20)\n\n for a in ax:\n a.set_ylabel(f\"number of {event_type_map[i]}\", size=11, weight='bold')\n if i!=2:\n a.set_xlabel(\"\")\n else:\n a.set_xlabel(\"Day\", size=15, weight='bold')\ndel fig, axs, ax,a\ngc.collect()\ntorch.cuda.empty_cache()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:53.194057Z","iopub.execute_input":"2023-01-08T13:48:53.194417Z","iopub.status.idle":"2023-01-08T13:48:54.960224Z","shell.execute_reply.started":"2023-01-08T13:48:53.194383Z","shell.execute_reply":"2023-01-08T13:48:54.959127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **A1.1**\nFrom the above graphs, we further observe that on Sunday has the highest activity of various types, but it is still not known **whether it is true that Sunday has the most sales**? Since the graphs above just show the number of events.\n* Therefore, in order to see which day the most orders occur, we should take the percentage of order events from the total number of events for this day.","metadata":{}},{"cell_type":"markdown","source":"### **Q2:** When do users make more orders?","metadata":{}},{"cell_type":"code","source":"# Count the percent of each event type for the day\ntrain_day_events_type_count['percent']= train_day_events_type_count.apply(lambda x: x['count']/train_date_activity.loc[train_date_activity.day==x['day'],'count'].item(),axis=1).to_numpy()\ntest_day_events_type_count['percent']= test_day_events_type_count.apply(lambda x: x['count']/test_date_activity.loc[test_date_activity.day==x['day'],'count'].item(),axis=1).to_numpy()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:54.961646Z","iopub.execute_input":"2023-01-08T13:48:54.962121Z","iopub.status.idle":"2023-01-08T13:48:55.011263Z","shell.execute_reply.started":"2023-01-08T13:48:54.962047Z","shell.execute_reply":"2023-01-08T13:48:55.010356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axs = plt.subplots(3,2, figsize=(20,10),sharey='row',gridspec_kw={'width_ratios': [3, 1]})\nfor i, ax in enumerate(axs):\n # plot train\n sns.barplot(data=train_day_events_type_count.loc[train_day_events_type_count.type==i], x='day', y='percent', palette=train_date_colors, ax=ax[0], order=train_date_order)\n # plot test\n sns.barplot(data=test_day_events_type_count.loc[test_day_events_type_count.type==i], x='day', y='percent', palette=test_date_colors, ax=ax[1], order = test_date_order)\n \n if i ==0:\n ax[0].set_title('Train Activity', weight=\"bold\", size=20)\n ax[1].set_title('Test Activity', weight=\"bold\", size=20)\n\n for a in ax:\n a.set_ylabel(f\"percent of {event_type_map[i]}\", size=11, weight='bold')\n if i!=2:\n a.set_xlabel(\"\")\n else:\n a.set_xlabel(\"Day\", size=15, weight='bold')\ndel fig, axs, ax,a\ngc.collect()\ntorch.cuda.empty_cache()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:55.012902Z","iopub.execute_input":"2023-01-08T13:48:55.013317Z","iopub.status.idle":"2023-01-08T13:48:56.781902Z","shell.execute_reply.started":"2023-01-08T13:48:55.013282Z","shell.execute_reply":"2023-01-08T13:48:56.779223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. From the above chart, we can't see any strongly marked day of the week, that would have more **clicks** over the other days. On the contrary, we can be convinced that the number of **clicks** in relation to the total number of events on this day is `approximately equal on all days of the week` and is kept at **~80%**.\n2. The situation for the amount of `add to cart` events is the same as for `click` events. There is no strongly marked day of the week, that would have more `add to cart` events than on other days. The number of `add to cart` events is **~8%** of all events.\n3. As for the `order` events, we can notice that there is only *one day* in the entire timeline that has a higher percentage of orders, and that is **August 9, 2022**. On other days we can still see that the percentage of `clicks` is **~2%**.\n\n> Let's dive a little bit deeper and explore the amount of `orders` versus the number of `clicked` and `add to cart` events.","metadata":{}},{"cell_type":"markdown","source":"#### The number of `order` events versus the number of `click` and `add to cart` events in that day","metadata":{}},{"cell_type":"code","source":"# Count the percent of order versus the number of clicks and adds to cart for each day\ntrain_day_order_click_cart = pd.DataFrame(train_day_events_type_count.day.unique(), columns=['day'] )\ntest_day_order_click_cart = pd.DataFrame(test_day_events_type_count.day.unique(), columns=['day'] )\n\ntrain_day_order_click_cart['orders/clicks'] = train_day_order_click_cart.apply(lambda x: \\\n (train_day_events_type_count.loc[(train_day_events_type_count.day==x[\"day\"]) & ((train_day_events_type_count.type==2)),'count'].item()/\n train_day_events_type_count.loc[(train_day_events_type_count.day==x[\"day\"]) & ((train_day_events_type_count.type==0)),'count'].item())*100,\n axis=1\n ).to_numpy()\ntrain_day_order_click_cart['orders/cart'] = train_day_order_click_cart.apply(lambda x:\\\n (train_day_events_type_count.loc[(train_day_events_type_count.day==x[\"day\"]) & ((train_day_events_type_count.type==2)),'count'].item()/\n train_day_events_type_count.loc[(train_day_events_type_count.day==x[\"day\"]) & ((train_day_events_type_count.type==1)),'count'].item())*100,\n axis=1\n ).to_numpy()\n\ntest_day_order_click_cart['orders/clicks'] = test_day_order_click_cart.apply(lambda x:\\\n (test_day_events_type_count.loc[(test_day_events_type_count.day==x[\"day\"]) & ((test_day_events_type_count.type==2)),'count'].item()/\n test_day_events_type_count.loc[(test_day_events_type_count.day==x[\"day\"]) & ((test_day_events_type_count.type==0)),'count'].item())*100,\n axis=1\n ).to_numpy()\ntest_day_order_click_cart['orders/cart'] = test_day_order_click_cart.apply(lambda x:\\\n (test_day_events_type_count.loc[(test_day_events_type_count.day==x[\"day\"]) & ((test_day_events_type_count.type==2)),'count'].item()/\n test_day_events_type_count.loc[(test_day_events_type_count.day==x[\"day\"]) & ((test_day_events_type_count.type==1)),'count'].item())*100,\n axis=1\n ).to_numpy()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:56.783578Z","iopub.execute_input":"2023-01-08T13:48:56.783953Z","iopub.status.idle":"2023-01-08T13:48:56.888102Z","shell.execute_reply.started":"2023-01-08T13:48:56.783916Z","shell.execute_reply":"2023-01-08T13:48:56.887108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axs = plt.subplots(2,2, figsize=(20,10),sharey='row',gridspec_kw={'width_ratios': [3, 1]})\n# Plot percent of order over the clicks\n# train\nsns.barplot(data=train_day_order_click_cart, x='day', y='orders/clicks', palette=train_date_colors, ax=axs[0][0], order=train_date_order)\n# test\nsns.barplot(data=test_day_order_click_cart, x='day', y='orders/clicks', palette=test_date_colors, ax=axs[0][1], order = test_date_order)\naxs[0][0].set_title('Train Activity', weight=\"bold\", size=20)\naxs[0][1].set_title('Test Activity', weight=\"bold\", size=20)\n\n\n# Plot percent of order over the adds to cart\n# train\nsns.barplot(data=train_day_order_click_cart, x='day', y='orders/cart', palette=train_date_colors, ax=axs[1][0], order=train_date_order)\n# test\nsns.barplot(data=test_day_order_click_cart, x='day', y='orders/cart', palette=test_date_colors, ax=axs[1][1], order = test_date_order)\n\ndel fig, axs\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:56.889563Z","iopub.execute_input":"2023-01-08T13:48:56.889884Z","iopub.status.idle":"2023-01-08T13:48:58.273965Z","shell.execute_reply.started":"2023-01-08T13:48:56.889859Z","shell.execute_reply":"2023-01-08T13:48:58.273001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.drop(columns=['day','day_of_week'], inplace=True)\ntest.drop(columns=['day','day_of_week'], inplace=True)\ndel train_date_activity, train_day_map_day_of_week, train_day_order_click_cart, train_day_events_type_count,\\\n test_date_activity, test_day_map_day_of_week, test_day_order_click_cart, test_day_events_type_count,\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:58.275546Z","iopub.execute_input":"2023-01-08T13:48:58.275904Z","iopub.status.idle":"2023-01-08T13:48:58.526321Z","shell.execute_reply.started":"2023-01-08T13:48:58.275869Z","shell.execute_reply":"2023-01-08T13:48:58.525337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A2:\nThe charts above have a similar distribution, comparing the distribution of the `percentage of orders` chart for the day.\n> Thus, we can sum up that there isn't any explicit day when people would order more products than usual.\n\nAnd the only anomalous day is **August 9, 2022**.\n\n\n**PS:**\n> But what about the **time**, maybe there is some kind of pattern and explicit **hours when customers are more active than usual**? \n\nLet's explore it!","metadata":{}},{"cell_type":"markdown","source":"## Users activity by hour\n* **Q1:** When customers are the most active ?\n* **Q2:** When customers orders the most?\n","metadata":{}},{"cell_type":"code","source":"# Create the color palette for hours plots\nhour_color_pallet = [third_color if i<12 else second_color for i in range(24)]","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:58.530982Z","iopub.execute_input":"2023-01-08T13:48:58.533202Z","iopub.status.idle":"2023-01-08T13:48:58.539435Z","shell.execute_reply.started":"2023-01-08T13:48:58.533157Z","shell.execute_reply":"2023-01-08T13:48:58.538363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create hour feature\ntrain_date_tmp = cudf.to_datetime(train.ts, unit='s')\ntrain['hour'] = train_date_tmp.dt.hour\n\ntest_date_tmp = cudf.to_datetime(test.ts, unit='s')\ntest['hour'] = test_date_tmp.dt.hour\n\ndel train_date_tmp, test_date_tmp\ntorch.cuda.empty_cache()\n\n__display(\"**TRAIN** & **TEST**\")\nside_by_side(train.head().to_pandas(), test.head().to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:58.544596Z","iopub.execute_input":"2023-01-08T13:48:58.547135Z","iopub.status.idle":"2023-01-08T13:48:58.604629Z","shell.execute_reply.started":"2023-01-08T13:48:58.547100Z","shell.execute_reply":"2023-01-08T13:48:58.603824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Q1:** When customers are the most active ?","metadata":{}},{"cell_type":"code","source":"# Create the barplot of the number of events by each hour in dataset\ntrain_hour_activity = train.hour.value_counts().reset_index().to_pandas().rename(columns={\"index\":\"hour\", \"hour\":\"count\"})\ntest_hour_activity = test.hour.value_counts().reset_index().to_pandas().rename(columns={\"index\":\"hour\", \"hour\":\"count\"})\n\n# Subplots figure\nfig, axs = plt.subplots(2,2, figsize=(20,10), gridspec_kw={'width_ratios': [1, 1], 'height_ratios':[3,1]}, sharex='col')\n\n# plot train\naxs[0][0].set_title('Train Activity', weight=\"bold\", size=20)\nsns.barplot(data=train_hour_activity, x='hour', y='count', ax=axs[0][0], palette=hour_color_pallet)\naxs[0][0].set_ylabel(\"number of events\", size=15, weight='bold')\naxs[0][0].set_xlabel(\"Hour\", size=15, weight='bold')\naxs[0][0].tick_params(labelbottom=True)\nsns.boxplot(data=train.to_pandas(), x='hour', ax=axs[1][0], notch=False, showcaps=True, color=main_color)\n\n# plot test\naxs[0][1].set_title('Test Activity', weight=\"bold\", size=20)\nsns.barplot(data=test_hour_activity, x='hour', y='count', ax=axs[0][1], palette=hour_color_pallet)\naxs[0][1].set_ylabel(\"number of events\", size=15, weight='bold')\naxs[0][1].set_xlabel(\"Hour\", size=15, weight='bold')\naxs[0][1].tick_params(labelbottom=True)\nsns.boxplot(data=test.to_pandas(), x='hour', ax=axs[1][1], notch=False, showcaps=True, color=main_color)\n\ndel fig, axs\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:48:58.608504Z","iopub.execute_input":"2023-01-08T13:48:58.610633Z","iopub.status.idle":"2023-01-08T13:49:08.530813Z","shell.execute_reply.started":"2023-01-08T13:48:58.610598Z","shell.execute_reply":"2023-01-08T13:49:08.529700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The users activity distribution is similar for `train` and `test` dataset. \n> But when the percentage of ordered products are the greatest? ","metadata":{}},{"cell_type":"markdown","source":"### **Q2:** When customers orders the most?","metadata":{}},{"cell_type":"code","source":"train_hour_type_count_events = train.groupby(['hour','type']).agg({'aid':'count'}).reset_index().rename(columns={\"aid\":\"count\"}).sort_values(by=['hour','type'], ignore_index=True).to_pandas()\ntest_hour_type_count_events = test.groupby(['hour','type']).agg({'aid':'count'}).reset_index().rename(columns={\"aid\":\"count\"}).sort_values(by=['hour','type'], ignore_index=True).to_pandas()\n\ntrain_hour_type_count_events['percent']= train_hour_type_count_events.apply(lambda x: x['count']/train_hour_activity.loc[train_hour_activity.hour==x['hour'],'count'].item(),axis=1).to_numpy()\ntest_hour_type_count_events['percent']= test_hour_type_count_events.apply(lambda x: x['count']/test_hour_activity.loc[test_hour_activity.hour==x['hour'],'count'].item(),axis=1).to_numpy()\n","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:08.532527Z","iopub.execute_input":"2023-01-08T13:49:08.533211Z","iopub.status.idle":"2023-01-08T13:49:08.777667Z","shell.execute_reply.started":"2023-01-08T13:49:08.533174Z","shell.execute_reply":"2023-01-08T13:49:08.776744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Subplots figure\nfig, axs = plt.subplots(3,2, figsize=(20,10),sharey='row',gridspec_kw={'width_ratios': [1, 1]})\nfor i, ax in enumerate(axs):\n # plot train\n sns.barplot(data=train_hour_type_count_events.loc[train_hour_type_count_events.type==i], x='hour', y='percent', palette=hour_color_pallet, ax=ax[0])\n # plot test\n sns.barplot(data=train_hour_type_count_events.loc[train_hour_type_count_events.type==i], x='hour', y='percent', palette=hour_color_pallet, ax=ax[1])\n \n if i ==0:\n ax[0].set_title('Train Hours Activity', weight=\"bold\", size=20)\n ax[1].set_title('Test Hours Activity', weight=\"bold\", size=20)\n\n for a in ax:\n a.set_ylabel(f\"percent of {event_type_map[i]}\", size=11, weight='bold')\n if i!=2:\n a.set_xlabel(\"\")\n else:\n a.set_xlabel(\"Hour\", size=15, weight='bold')\ndel fig, axs, ax,a\ngc.collect()\ntorch.cuda.empty_cache()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:08.785534Z","iopub.execute_input":"2023-01-08T13:49:08.785808Z","iopub.status.idle":"2023-01-08T13:49:11.465738Z","shell.execute_reply.started":"2023-01-08T13:49:08.785783Z","shell.execute_reply":"2023-01-08T13:49:11.464785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the chart above we can't see any highlighting hours when the users would view more products or add them to carts and then buy.\n\n> What about the part of the day, maybe there is some **part of the day (morning, noon, night)** when users are more active than usual?","metadata":{}},{"cell_type":"markdown","source":"#### Users activity by **parts of the day**","metadata":{}},{"cell_type":"code","source":"# Create hour - part of the day dataframe\nmorning = [*range(6,10)]\nnoon = [*range(10,14)]\nafternoon = [*range(14,18)]\nevening = [*range(18,22)]\nnight = [22,23,*range(0,6)]\ndef get_part_of_the_day(time):\n if time in morning:\n return 'morning'\n elif time in noon:\n return 'noon'\n elif time in afternoon:\n return 'afternoon'\n elif time in evening:\n return 'evening'\n elif time in night:\n return 'night'\nhour_part_of_day_df = pd.DataFrame([*range(24)], columns=['hour'])\nhour_part_of_day_df['part_of_the_day'] = hour_part_of_day_df.hour.apply(lambda x: get_part_of_the_day(x))","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:11.467933Z","iopub.execute_input":"2023-01-08T13:49:11.469201Z","iopub.status.idle":"2023-01-08T13:49:11.479586Z","shell.execute_reply.started":"2023-01-08T13:49:11.469147Z","shell.execute_reply":"2023-01-08T13:49:11.478542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a color palette for part of the day\npart_of_day_color = []\npart_of_day_order = [*morning,*noon,*afternoon,*evening,*night]\nfor i in part_of_day_order:\n if i in morning:\n part_of_day_color.append(third_color)\n elif i in noon:\n part_of_day_color.append(second_color)\n elif i in afternoon:\n part_of_day_color.append(main_color)\n elif i in evening:\n part_of_day_color.append(highlight_color)\n elif i in night:\n part_of_day_color.append(dark_color)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:11.480918Z","iopub.execute_input":"2023-01-08T13:49:11.481605Z","iopub.status.idle":"2023-01-08T13:49:11.493852Z","shell.execute_reply.started":"2023-01-08T13:49:11.481569Z","shell.execute_reply":"2023-01-08T13:49:11.492855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Subplots figure\nfig, axs = plt.subplots(3,2, figsize=(20,10),sharey='row',gridspec_kw={'width_ratios': [1, 1]})\nfor i, ax in enumerate(axs):\n # plot train\n sns.barplot(data=train_hour_type_count_events.loc[train_hour_type_count_events.type==i], x='hour', y='percent', palette=part_of_day_color, ax=ax[0], order=part_of_day_order)\n # plot test\n sns.barplot(data=train_hour_type_count_events.loc[train_hour_type_count_events.type==i], x='hour', y='percent', palette=part_of_day_color, ax=ax[1], order=part_of_day_order)\n \n if i ==0:\n ax[0].set_title('Train Hours Activity', weight=\"bold\", size=20)\n ax[1].set_title('Test Hours Activity', weight=\"bold\", size=20)\n\n for a in ax:\n a.set_ylabel(f\"percent of {event_type_map[i]}\", size=11, weight='bold')\n if i!=2:\n a.set_xlabel(\"\")\n else:\n a.set_xlabel(\"Hour\", size=15, weight='bold')\ndel fig, axs, ax,a\ngc.collect()\ntorch.cuda.empty_cache()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:11.495428Z","iopub.execute_input":"2023-01-08T13:49:11.495803Z","iopub.status.idle":"2023-01-08T13:49:13.456814Z","shell.execute_reply.started":"2023-01-08T13:49:11.495762Z","shell.execute_reply":"2023-01-08T13:49:13.454409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create dataframe with number of events for the part of the day.\ntrain_part_of_the_day_activity_count = train_hour_type_count_events.merge(hour_part_of_day_df, on='hour').groupby(['part_of_the_day','type']).agg({\"count\":['sum']})\ntrain_part_of_the_day_activity_count.columns = ['count']\ntrain_part_of_the_day_activity_count = train_part_of_the_day_activity_count.reset_index()\ntrain_part_of_the_day_activity_count['percent'] = train_part_of_the_day_activity_count.apply(lambda x: x['count']/train_part_of_the_day_activity_count.loc[train_part_of_the_day_activity_count.part_of_the_day==x['part_of_the_day'],'count'].sum(), axis=1)\n\ntest_part_of_the_day_activity_count = test_hour_type_count_events.merge(hour_part_of_day_df, on='hour').groupby(['part_of_the_day','type']).agg({\"count\":['sum']})\ntest_part_of_the_day_activity_count.columns = ['count']\ntest_part_of_the_day_activity_count = test_part_of_the_day_activity_count.reset_index()\ntest_part_of_the_day_activity_count['percent'] = test_part_of_the_day_activity_count.apply(lambda x: x['count']/test_part_of_the_day_activity_count.loc[test_part_of_the_day_activity_count.part_of_the_day==x['part_of_the_day'],'count'].sum(), axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:13.458896Z","iopub.execute_input":"2023-01-08T13:49:13.459977Z","iopub.status.idle":"2023-01-08T13:49:13.505195Z","shell.execute_reply.started":"2023-01-08T13:49:13.459938Z","shell.execute_reply":"2023-01-08T13:49:13.504307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"part_of_day_order_str = ['morning', 'noon', \"afternoon\", \"evening\", \"night\"]\ncolors_part_of_the_day = [third_color,second_color, main_color, highlight_color, dark_color]\n# Subplots figure\nfig, axs = plt.subplots(3,2, figsize=(20,10),gridspec_kw={'width_ratios': [1, 1]})\nfor i, ax in enumerate(axs):\n # plot train\n sns.barplot(data=train_part_of_the_day_activity_count.loc[train_part_of_the_day_activity_count.type==i], x='part_of_the_day', y='count', palette=colors_part_of_the_day, ax=ax[0], order=part_of_day_order_str)\n # plot test\n sns.barplot(data=test_part_of_the_day_activity_count.loc[test_part_of_the_day_activity_count.type==i], x='part_of_the_day', y='count', palette=colors_part_of_the_day, ax=ax[1], order=part_of_day_order_str)\n \n if i ==0:\n ax[0].set_title('Train Part Of the Day Activity', weight=\"bold\", size=20)\n ax[1].set_title('Test Part Of the Day Activity', weight=\"bold\", size=20)\n\n for a in ax:\n a.set_ylabel(f\"number of {event_type_map[i]}\", size=11, weight='bold')\n if i!=2:\n a.set_xlabel(\"\")\n else:\n a.set_xlabel(\"Part of the day\", size=15, weight='bold')\ndel fig, axs, ax,a\ngc.collect()\ntorch.cuda.empty_cache()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:13.506499Z","iopub.execute_input":"2023-01-08T13:49:13.506868Z","iopub.status.idle":"2023-01-08T13:49:14.590101Z","shell.execute_reply.started":"2023-01-08T13:49:13.506832Z","shell.execute_reply":"2023-01-08T13:49:14.589030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Subplots figure\nfig, axs = plt.subplots(3,2, figsize=(20,10),sharey='row',gridspec_kw={'width_ratios': [1, 1]})\nfor i, ax in enumerate(axs):\n # plot train\n sns.barplot(data=train_part_of_the_day_activity_count.loc[train_part_of_the_day_activity_count.type==i], x='part_of_the_day', y='percent', palette=colors_part_of_the_day, ax=ax[0], order=part_of_day_order_str)\n # plot test\n sns.barplot(data=test_part_of_the_day_activity_count.loc[test_part_of_the_day_activity_count.type==i], x='part_of_the_day', y='percent', palette=colors_part_of_the_day, ax=ax[1], order=part_of_day_order_str)\n \n if i ==0:\n ax[0].set_title('Train Part Of the Day Activity', weight=\"bold\", size=20)\n ax[1].set_title('Test Part Of the Day Activity', weight=\"bold\", size=20)\n\n for a in ax:\n a.set_ylabel(f\"percent of {event_type_map[i]}\", size=11, weight='bold')\n if i!=2:\n a.set_xlabel(\"\")\n else:\n a.set_xlabel(\"Part of the day\", size=15, weight='bold')\ndel fig, axs, ax,a\ngc.collect()\ntorch.cuda.empty_cache()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:14.591752Z","iopub.execute_input":"2023-01-08T13:49:14.592931Z","iopub.status.idle":"2023-01-08T13:49:15.583886Z","shell.execute_reply.started":"2023-01-08T13:49:14.592887Z","shell.execute_reply":"2023-01-08T13:49:15.582970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **A1:**\nFrom the charts above, you can see that users are most active `in the evening from 17:00. until 8 pm.`\n\nBut this fact doesn't mean that customers buy most in the evening.\n### **A2:**\nFrom the charts of the **percentage** of `clicks`, `add to the cart` and `orders`, we can see that the **percentage** of clicks is the same for different times and is equal to ~80%.\n\nBut more `add to the cart` and `orders` are made in the `first part of the day`.","metadata":{}},{"cell_type":"code","source":"# delete objects\ndel train_hour_activity, train_hour_type_count_events, train_part_of_the_day_activity_count , \\\n test_hour_activity, test_hour_type_count_events, test_part_of_the_day_activity_count \ntrain.drop(columns=['hour'], inplace=True)\ntest.drop(columns=['hour'], inplace=True)\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:15.585362Z","iopub.execute_input":"2023-01-08T13:49:15.586358Z","iopub.status.idle":"2023-01-08T13:49:15.761586Z","shell.execute_reply.started":"2023-01-08T13:49:15.586321Z","shell.execute_reply":"2023-01-08T13:49:15.760345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The time duration of `sessions`\n* **Q1:** What is the longest session time and the shortest in the datasets?\n* **Q2:** Whats is the distribution of the sessions time duration?","metadata":{}},{"cell_type":"code","source":"# Create the min and max timestamp features for the session.\n# TRAIN\ntrain_session_duration = train.groupby('session').agg({'ts':['min','max']})\ntrain_session_duration.columns = ['_'.join(col) for col in train_session_duration.columns.values]\ntrain_session_duration.reset_index(inplace=True)\n# TEST\ntest_session_duration = test.groupby('session').agg({'ts':['min','max']})\ntest_session_duration.columns = ['_'.join(col) for col in test_session_duration.columns.values]\ntest_session_duration.reset_index(inplace=True)\n\n\nside_by_side(train_session_duration.head().to_pandas(), test_session_duration.head().to_pandas())\n\n","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:15.763163Z","iopub.execute_input":"2023-01-08T13:49:15.763638Z","iopub.status.idle":"2023-01-08T13:49:15.938940Z","shell.execute_reply.started":"2023-01-08T13:49:15.763592Z","shell.execute_reply":"2023-01-08T13:49:15.937988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Count the duration of the session (max ts - min ts)\ntrain_session_duration['duration_s'] = (train_session_duration['ts_max'] - train_session_duration['ts_min'])\ntest_session_duration['duration_s'] = (test_session_duration['ts_max'] - test_session_duration['ts_min'])\n\nside_by_side(train_session_duration.head().to_pandas(), test_session_duration.head().to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:15.940372Z","iopub.execute_input":"2023-01-08T13:49:15.940714Z","iopub.status.idle":"2023-01-08T13:49:15.963647Z","shell.execute_reply.started":"2023-01-08T13:49:15.940671Z","shell.execute_reply":"2023-01-08T13:49:15.962840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Q1:** What is the **longest** session time and the **shortest** in the datasets?","metadata":{}},{"cell_type":"code","source":"__display(f\"**Train**: Longest session time: `{train_session_duration.duration_s.max()/3600}` hours or {pd.to_timedelta(train_session_duration.duration_s.max(), unit='s')}\")\n__display(f\"**Train**: Shortes session time: {train_session_duration.duration_s.min()} hours\")\n\n__display(f\"**Test**: Longest session time: `{test_session_duration.duration_s.max()/3600}` hours or {pd.to_timedelta(test_session_duration.duration_s.max(), unit='s')}\")\n__display(f\"**Test**: Shortes session time: {test_session_duration.duration_s.min()} hours\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:15.965993Z","iopub.execute_input":"2023-01-08T13:49:15.966576Z","iopub.status.idle":"2023-01-08T13:49:16.052667Z","shell.execute_reply.started":"2023-01-08T13:49:15.966541Z","shell.execute_reply":"2023-01-08T13:49:16.051664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **A1:**\nThe **longest** session duration for both datasets is equal to the entire timelines of the datasets.\n\nHowever,  the **shortest** sessions in both datasets did not last even seconds and their duration is **0**.\n\n> To understand more sessions duration, let's examine its distribution and analyze the *mean* and other session duration *statistics*.\n","metadata":{}},{"cell_type":"markdown","source":"### **Q2:** Whats is the distribution of the sessions time duration?","metadata":{}},{"cell_type":"code","source":"# Display session duration distribution in hours and days TRAIN\ntrain_sessions_duration_in_hours = train_session_duration.duration_s/(3600)\ntrain_sessions_duration_in_days = train_sessions_duration_in_hours/24\n\ntrain_hours_duration_stat = train_sessions_duration_in_hours.describe()\ntrain_days_duration_stat = train_sessions_duration_in_days.describe()\n\nf, (a0, a1) = plt.subplots(2, 1, gridspec_kw={'height_ratios': [1, 1]}, figsize=(24, 15))\na0.set_title(\"TRAIN: Session duration in hours\", weight=\"bold\", size=20)\na0.set_xlabel(\"Hours\", weight=\"bold\", size=12)\nsns.histplot(train_sessions_duration_in_hours.to_pandas()[::50], bins=200, ax=a0, color=main_color)\na0.set_xlim(xmin=-10)\nfor i in ['mean', '25%', '50%', '75%']:\n a0.axvline(x=train_hours_duration_stat[i], color=highlight_color)\n a0.text(s=f\"{i}\\n{train_hours_duration_stat[i]}\" ,x=train_hours_duration_stat[i]+1, y=a0.get_ylim()[1]/2, weight='bold')\n\na1.set_title(\"TRAIN: Session duration in days\", weight=\"bold\", size=20)\na1.set_xlabel(\"Days\", weight=\"bold\", size=12)\nsns.histplot(train_sessions_duration_in_days.to_pandas()[::50], bins=56, ax=a1, color=main_color)\na1.set_xlim(xmin=-1)\nfor i in ['mean', '25%', '50%', '75%']:\n a1.axvline(x=train_days_duration_stat[i], color=highlight_color)\n a1.text(s=f\"{i}\\n{train_days_duration_stat[i]}\" ,x=train_days_duration_stat[i], y=a1.get_ylim()[1]/2, weight='bold')","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:16.054242Z","iopub.execute_input":"2023-01-08T13:49:16.054580Z","iopub.status.idle":"2023-01-08T13:49:18.035091Z","shell.execute_reply.started":"2023-01-08T13:49:16.054547Z","shell.execute_reply":"2023-01-08T13:49:18.034133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display session duration distribution in hours and days TEST\ntest_sessions_duration_in_hours = test_session_duration.duration_s/(3600)\ntest_sessions_duration_in_days = test_sessions_duration_in_hours/24\n\ntest_hours_duration_stat = test_sessions_duration_in_hours.describe()\ntest_days_duration_stat = test_sessions_duration_in_days.describe()\n\nf, (a0, a1) = plt.subplots(2, 1, gridspec_kw={'height_ratios': [1, 1]}, figsize=(24, 15))\na0.set_title(\"TEST: Session duration in hours\", weight=\"bold\", size=20)\na0.set_xlabel(\"Hours\", weight=\"bold\", size=12)\nsns.histplot(test_sessions_duration_in_hours.to_pandas(), bins=60, ax=a0, color=main_color)\na0.set_xlim(xmin=-2)\nfor i in ['50%']:\n a0.axvline(x=test_hours_duration_stat[i], color=highlight_color)\n a0.text(s=f\"{i}\\n{test_hours_duration_stat[i]}\" ,x=test_hours_duration_stat[i]+1, y=a0.get_ylim()[1]/2, weight='bold')\n\na1.set_title(\"TEST: Session duration in days\", weight=\"bold\", size=20)\na1.set_xlabel(\"Days\", weight=\"bold\", size=12)\nsns.histplot(test_sessions_duration_in_days.to_pandas(), bins=14, ax=a1, color=main_color)\na1.set_xlim(xmin=-0.5)\nfor i in ['50%']:\n a1.axvline(x=test_days_duration_stat[i], color=highlight_color)\n a1.text(s=f\"{i}\\n{test_days_duration_stat[i]}\" ,x=test_days_duration_stat[i], y=a1.get_ylim()[1]/2, weight='bold')","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:18.036727Z","iopub.execute_input":"2023-01-08T13:49:18.037374Z","iopub.status.idle":"2023-01-08T13:49:21.239924Z","shell.execute_reply.started":"2023-01-08T13:49:18.037335Z","shell.execute_reply":"2023-01-08T13:49:21.239009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transform discibe() objects\n# TRAIN\ntrain_hours_duration_stat = train_hours_duration_stat.reset_index().rename(columns={\"index\":\"stats\",\"duration_s\":'hours'})\ntrain_days_duration_stat = train_days_duration_stat.reset_index().rename(columns={\"index\":\"stats\",\"duration_s\":'days'})\n# TEST\ntest_hours_duration_stat = test_hours_duration_stat.reset_index().rename(columns={\"index\":\"stats\",\"duration_s\":'hours'})\ntest_days_duration_stat = test_days_duration_stat.reset_index().rename(columns={\"index\":\"stats\",\"duration_s\":'days'})","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:21.241630Z","iopub.execute_input":"2023-01-08T13:49:21.242011Z","iopub.status.idle":"2023-01-08T13:49:21.254432Z","shell.execute_reply.started":"2023-01-08T13:49:21.241973Z","shell.execute_reply":"2023-01-08T13:49:21.253414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"__display(\"## **TRAIN:** VS **TEST**: Duration Statistics\")\nside_by_side(train_hours_duration_stat.merge(train_days_duration_stat, on=['stats']).iloc[1:].to_pandas(), test_hours_duration_stat.merge(test_days_duration_stat, on=['stats']).iloc[1:].to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:21.255956Z","iopub.execute_input":"2023-01-08T13:49:21.256896Z","iopub.status.idle":"2023-01-08T13:49:21.287601Z","shell.execute_reply.started":"2023-01-08T13:49:21.256861Z","shell.execute_reply":"2023-01-08T13:49:21.286622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# deleting object\ndel test_days_duration_stat, test_hours_duration_stat, test_sessions_duration_in_days, test_sessions_duration_in_hours,\\\n train_days_duration_stat, train_hours_duration_stat, train_sessions_duration_in_days, train_sessions_duration_in_hours, test_session_duration, train_session_duration\n\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:21.288985Z","iopub.execute_input":"2023-01-08T13:49:21.289340Z","iopub.status.idle":"2023-01-08T13:49:21.483931Z","shell.execute_reply.started":"2023-01-08T13:49:21.289306Z","shell.execute_reply":"2023-01-08T13:49:21.482849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **A2:**\nThe statistics above shows that **50%** of all sessions have duration less than: \n* 2.1 days (51.5 hours) for **train** dataset and \n* 0.008 hour for **test** dataset\n---","metadata":{}},{"cell_type":"markdown","source":"# `Session` x `Events`","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/otto-img/img/events tittle.png\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:21.485470Z","iopub.execute_input":"2023-01-08T13:49:21.485826Z","iopub.status.idle":"2023-01-08T13:49:21.499476Z","shell.execute_reply.started":"2023-01-08T13:49:21.485791Z","shell.execute_reply":"2023-01-08T13:49:21.498596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The goal of this section is to explore the events in the sessions, analyze the number of it and familiarize with the `event types`. \n\n1. **Q1:** What is the number of events per session?\n2. **Q2:** Event types:\n * **Q2.1:** The number of the event types for datasets?\n * **Q2.2:** What are the statistics of each event type per session?\n\n ","metadata":{}},{"cell_type":"markdown","source":"## **Q1:** What is the number of events per session?","metadata":{}},{"cell_type":"code","source":"# Display number of events per session\ntrain_session_events_count = train['session'].value_counts()\ntrain_events_count_stats = train_session_events_count.describe()\nf, (a0, a1) = plt.subplots(2, 1, gridspec_kw={'height_ratios': [3, 1]}, figsize=(24, 15))\nsns.histplot(train_session_events_count.values.get()[::50], bins=500, kde=True, ax=a0, color=main_color)\na0.set_xlim(xmin=-10)\nfor i in ['75%', 'max']:\n a0.axvline(x=train_events_count_stats[i], color=highlight_color)\n a0.text(s=f\"{i}\\n{train_events_count_stats[i]}\" ,x=train_events_count_stats[i]+1, y=a0.get_ylim()[1]/2, weight='bold')\n \nsns.boxplot(x=train_session_events_count.values.get(), ax=a1, notch=False, showcaps=True, color=second_color)\nplt.suptitle(\"TRAIN: number of events per sessions\", weight=\"bold\", size=25)\nplt.show()\n\n# TEST\ntest_session_events_count = test['session'].value_counts()\ntest_events_count_stats = test_session_events_count.describe()\nf, (a0, a1) = plt.subplots(2, 1, gridspec_kw={'height_ratios': [3, 1]}, figsize=(24, 15))\nsns.histplot(test_session_events_count.values.get()[::50], bins=100, kde=True, ax=a0, color=main_color)\nfor i in ['75%', 'max']:\n a0.axvline(x=test_events_count_stats[i], color=highlight_color)\n a0.text(s=f\"{i}\\n{test_events_count_stats[i]}\" ,x=test_events_count_stats[i]+1, y=a0.get_ylim()[1]/2, weight='bold')\nsns.boxplot(x=test_session_events_count.values.get(), ax=a1, notch=False, showcaps=True, color=second_color)\nplt.suptitle(\"TEST: number of events per sessions\", weight=\"bold\", size=25)\nplt.show()\n\ndel train_session_events_count, test_session_events_count\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:21.501041Z","iopub.execute_input":"2023-01-08T13:49:21.501470Z","iopub.status.idle":"2023-01-08T13:49:28.662915Z","shell.execute_reply.started":"2023-01-08T13:49:21.501437Z","shell.execute_reply":"2023-01-08T13:49:28.661994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display number of events statistic\n__display(\"### **TRAIN** VS **TEST**: Number of events\")\nside_by_side(\n train_events_count_stats.reset_index().rename(columns={\"index\":\"statistics\",\"session\":\"number of events\"}).iloc[1:].to_pandas(),\n test_events_count_stats.reset_index().rename(columns={\"index\":\"statistics\",\"session\":\"number of events\"}).iloc[1:].to_pandas()\n )","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:28.664591Z","iopub.execute_input":"2023-01-08T13:49:28.664965Z","iopub.status.idle":"2023-01-08T13:49:28.687541Z","shell.execute_reply.started":"2023-01-08T13:49:28.664929Z","shell.execute_reply":"2023-01-08T13:49:28.686618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **A1:**\n\n1. For **Train** datasets :\n * The *average* number of event per session is **~16-17**\n * but **75%** of sessions have less than *15* events \n\n2. For **Test** datasets:\n * The *average* number of event per session is **~4**\n * but **75%** of sessions have less than *4* events ","metadata":{}},{"cell_type":"markdown","source":"## **Q2:** Event types","metadata":{}},{"cell_type":"markdown","source":"### **Q2.1:** The number of the event types for datasets?","metadata":{}},{"cell_type":"code","source":"# Distribution of event types\n\ntrain_type_count = train.type.value_counts().reset_index().rename({\"index\":'event',\"type\":\"count\"}, axis=1)\ntrain_type_count['percent'] = (train_type_count['count']/train.shape[0])*100\n\ntest_type_count = test.type.value_counts().reset_index().rename({\"index\":'event',\"type\":\"count\"}, axis=1)\ntest_type_count['percent'] = (test_type_count['count']/test.shape[0])*100\n\n__display(\"**TRAIN** & **TEST**\")\nside_by_side(\n train_type_count.to_pandas(),\n test_type_count.to_pandas()\n )","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:28.688738Z","iopub.execute_input":"2023-01-08T13:49:28.689083Z","iopub.status.idle":"2023-01-08T13:49:28.976574Z","shell.execute_reply.started":"2023-01-08T13:49:28.689033Z","shell.execute_reply":"2023-01-08T13:49:28.975553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **A2.1:**\n`Event types` distribution for both datasets are similar.\n* for `train` / `test` dataset:\n * **89.851%** / **90.827%** of all events are `clicks`.\n * **7.796%** / **8.227** of all events are `carts`.\n * **2.35** / **0.945** of all events are `orders`.","metadata":{}},{"cell_type":"markdown","source":"### **Q2.2:** What are the statistics of each event type per session?\n\n`Session` and the **number** of `clicked`, added to `cart` and `ordered` product:","metadata":{}},{"cell_type":"code","source":"# Create dataset with info about the number of clicked, added to cart and ordered for each session\n# TRAIN\ntrain_session_events_type_count = train.groupby(['session', 'type']).agg({'aid':[\"count\"]}).sort_index()\ntrain_session_events_type_count.columns = train_session_events_type_count.columns.droplevel()\ntrain_session_events_type_count = train_session_events_type_count.unstack(1)\ntrain_session_events_type_count.columns = train_session_events_type_count.columns.map('{0[0]}_{0[1]}'.format) \ntrain_session_events_type_count.fillna(value=0, inplace=True)\ntrain_session_events_type_count['events_sum'] = train_session_events_type_count.sum(axis=1)\n\n# # Create feature descibing the percentage of clicked, added to cart and ordered product for each session\ntrain_session_events_type_count['percent_of_clicked'] = (train_session_events_type_count['count_0']/train_session_events_type_count['events_sum'])*100\ntrain_session_events_type_count['percent_of_added_to_cart'] = (train_session_events_type_count['count_1']/train_session_events_type_count['events_sum'])*100\ntrain_session_events_type_count['percent_of_ordered'] = (train_session_events_type_count['count_2']/train_session_events_type_count['events_sum'])*100\n\n__display(\"### TRAIN:\")\ndisplay(train_session_events_type_count)\n\n\n# TEST\ntest_session_events_type_count = test.groupby(['session', 'type']).agg({'aid':[\"count\"]}).sort_index()\ntest_session_events_type_count.columns = test_session_events_type_count.columns.droplevel()\ntest_session_events_type_count = test_session_events_type_count.unstack(1)\ntest_session_events_type_count.columns = test_session_events_type_count.columns.map('{0[0]}_{0[1]}'.format) \ntest_session_events_type_count.fillna(value=0, inplace=True)\ntest_session_events_type_count['events_sum'] = test_session_events_type_count.sum(axis=1)\n\n# Create feature descibing the percentage of clicked, added to cart and ordered product for each session\ntest_session_events_type_count['percent_of_clicked'] = (test_session_events_type_count['count_0']/test_session_events_type_count['events_sum'])*100\ntest_session_events_type_count['percent_of_added_to_cart'] = (test_session_events_type_count['count_1']/test_session_events_type_count['events_sum'])*100\ntest_session_events_type_count['percent_of_ordered'] = (test_session_events_type_count['count_2']/test_session_events_type_count['events_sum'])*100\n\n__display(\"### TEST:\")\ndisplay(test_session_events_type_count)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:28.978018Z","iopub.execute_input":"2023-01-08T13:49:28.978615Z","iopub.status.idle":"2023-01-08T13:49:30.632824Z","shell.execute_reply.started":"2023-01-08T13:49:28.978577Z","shell.execute_reply":"2023-01-08T13:49:30.631848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the statistic for the number of each event type per session\ntrain_session_events_type_count_stat = train_session_events_type_count.iloc[:,:3].describe()\ntest_session_events_type_count_stat = test_session_events_type_count.iloc[:,:3].describe()\n\n\n__display(\"### **TRAIN** VS **TEST**: Number of each event type in session\")\nside_by_side(train_session_events_type_count_stat.iloc[1:].to_pandas(), test_session_events_type_count_stat.iloc[1:].to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:30.637106Z","iopub.execute_input":"2023-01-08T13:49:30.639340Z","iopub.status.idle":"2023-01-08T13:49:31.111507Z","shell.execute_reply.started":"2023-01-08T13:49:30.639302Z","shell.execute_reply":"2023-01-08T13:49:31.110374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the statistic for the percent of each event type per session\ntrain_session_events_type_count_percent_stat = train_session_events_type_count.iloc[:,4:].describe()\ntest_session_events_type_count_percent_stat = test_session_events_type_count.iloc[:,4:].describe()\n\n\n__display(\"### **TRAIN** VS **TEST**: Percent of each event type in session\")\nside_by_side(train_session_events_type_count_percent_stat.iloc[1:].to_pandas(), test_session_events_type_count_percent_stat.iloc[1:].to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:31.115890Z","iopub.execute_input":"2023-01-08T13:49:31.118319Z","iopub.status.idle":"2023-01-08T13:49:31.471967Z","shell.execute_reply.started":"2023-01-08T13:49:31.118273Z","shell.execute_reply":"2023-01-08T13:49:31.469078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the number of clicks, add to cart and order event in sessios\nfig, axs = plt.subplots(3,2, figsize=(20,30))\nfor i, ax in enumerate(axs):\n# TRAIN\n sns.histplot(train_session_events_type_count[f'count_{i}'].to_pandas(), bins=300, ax=ax[0], color=main_color)\n ax[0].set_title(f\"TRAIN: Number of {event_type_map[i]} in sessions\", weight=\"bold\", size=20)\n ax[0].axvline(x=train_session_events_type_count_stat.loc[\"50%\", f'count_{i}'], color=highlight_color)\n ax[0].text(s=f\"50%\\n{train_session_events_type_count_stat.loc['50%', f'count_{i}']}\",\\\n x=train_session_events_type_count_stat.loc[\"50%\", f'count_{i}']+0.2,\\\n y=ax[0].get_ylim()[1]/2,\n weight=\"bold\"\n )\n# TEST\n sns.histplot(test_session_events_type_count[f'count_{i}'].to_pandas(), bins=300, ax=ax[1], color=main_color)\n ax[1].set_title(f\"TEST: Number of {event_type_map[i]} in sessions\", weight=\"bold\", size=20)\n ax[1].axvline(x=test_session_events_type_count_stat.loc[\"50%\", f'count_{i}'], color=highlight_color)\n ax[1].text(s=f\"50%\\n{test_session_events_type_count_stat.loc['50%', f'count_{i}']}\",\\\n x=test_session_events_type_count_stat.loc[\"50%\", f'count_{i}']+0.2,\\\n y=ax[1].get_ylim()[1]/2,\n weight=\"bold\"\n ) ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:49:31.473370Z","iopub.execute_input":"2023-01-08T13:49:31.473761Z","iopub.status.idle":"2023-01-08T13:50:04.421909Z","shell.execute_reply.started":"2023-01-08T13:49:31.473717Z","shell.execute_reply":"2023-01-08T13:50:04.420986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_number_of_sessions = train_session_events_type_count.shape[0]\ntrain_number_of_sesson_with_clicks = train_session_events_type_count.loc[train_session_events_type_count.count_0>0].shape[0]\ntrain_number_of_sesson_with_carts = train_session_events_type_count.loc[train_session_events_type_count.count_1>0].shape[0]\ntrain_number_of_sesson_with_orders = train_session_events_type_count.loc[train_session_events_type_count.count_2>0].shape[0]\n\ntest_number_of_sessions = test_session_events_type_count.shape[0]\ntest_number_of_sesson_with_clicks = test_session_events_type_count.loc[test_session_events_type_count.count_0>0].shape[0]\ntest_number_of_sesson_with_carts = test_session_events_type_count.loc[test_session_events_type_count.count_1>0].shape[0]\ntest_number_of_sesson_with_orders = test_session_events_type_count.loc[test_session_events_type_count.count_2>0].shape[0]\n\n__display(f\"**Train:** Number of session with `clicks` - {train_number_of_sesson_with_clicks} or **{(train_number_of_sesson_with_clicks/train_number_of_sessions)*100:.2f}** percents\")\n__display(f\"**Train:** Number of session with `adds to cart` - {train_number_of_sesson_with_carts} or **{(train_number_of_sesson_with_carts/train_number_of_sessions)*100:.2f}** percents\")\n__display(f\"**Train:** Number of session with `orders` - {train_number_of_sesson_with_orders} or **{(train_number_of_sesson_with_orders/train_number_of_sessions)*100:.2f}** percents\")\n__display(\"---\")\n__display(f\"**Test:** Number of session with `clicks` - {test_number_of_sesson_with_clicks} or **{(test_number_of_sesson_with_clicks/test_number_of_sessions)*100:.2f}** percents\")\n__display(f\"**Test:** Number of session with `adds to cart` - {test_number_of_sesson_with_carts} or **{(test_number_of_sesson_with_carts/test_number_of_sessions)*100:.2f}** percents\")\n__display(f\"**Test:** Number of session with `orders` - {test_number_of_sesson_with_orders} or **{(test_number_of_sesson_with_orders/test_number_of_sessions)*100:.2f}** percents\")\n","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:50:04.423586Z","iopub.execute_input":"2023-01-08T13:50:04.423956Z","iopub.status.idle":"2023-01-08T13:50:04.515698Z","shell.execute_reply.started":"2023-01-08T13:50:04.423919Z","shell.execute_reply":"2023-01-08T13:50:04.514838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_session_events_type_count, test_session_events_type_count\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T13:50:04.517008Z","iopub.execute_input":"2023-01-08T13:50:04.517451Z","iopub.status.idle":"2023-01-08T13:50:04.739104Z","shell.execute_reply.started":"2023-01-08T13:50:04.517417Z","shell.execute_reply":"2023-01-08T13:50:04.738004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **A2.2:**\n* For `train` dataset:\n * **100%** of sessions have some `clicks`,\n * when only **~30%** of then have been added something to `cart`,\n * and just **~13%** have been made `orders`.\n\n\n* For `test` dataset:\n * **99.90%** of sessions have some `clicks`,\n * when only **~14.5%** of then have been added something to `cart`,\n * and just **~2%** have been made `orders`.\n\n\n\n1. `Clicks` events:\n * For `train` dataset:\n * the **average** number of this type events is about **15**,\n * when the **75%** of all sessions have less than **14**,\n * and **50%** have even less than **5**.\n * For `test` dataset:\n * the **average** number of this type events is about **4**,\n * when the **75%** of all sessions have less than **4**,\n * and **50%** have less than **2**.\n\n2. `Carts` events:\n * For `train` dataset:\n * the **average** number of this type events is just about **1**,\n * when the **75%** of all sessions have even less than **1**,\n * For `test` dataset:\n * the **average** number of this type events is even less than **1**,\n * and most of all session **don't have** any `add to cart` events.\n3. `Orders` events:\n * **Most** of the sessions in both datasets **have no orders**, so the average number of orders per session is less than 1 event.","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# `Sessions` x `Event Type`\nThe section focus on analyzing the event types. \n\nCombinations of event types that appear during sessions in datasets were investigated.","metadata":{}},{"cell_type":"code","source":"# DataFrame with info about made events during the session:\n # Feature 1 - list of events.\n # Feature 2 - list of unique event types.\n # Feature 3 - number of unique event types.\ntrain_session_event = train.groupby('session').agg({'type':['collect','unique','nunique']})\ntrain_session_event.columns = ['_'.join(col) for col in train_session_event.columns.values]\ntrain_session_event.type_unique = cudf.Series(train_session_event[\"type_unique\"].to_pandas().astype(\"str\"))\ntrain_session_event.reset_index(inplace=True)\n__display(\"**TRAIN:**\")\ndisplay(train_session_event)\n\ntest_session_event = test.groupby('session').agg({'type':['collect','unique','nunique']})\ntest_session_event.columns = ['_'.join(col) for col in test_session_event.columns.values]\ntest_session_event.type_unique = cudf.Series(test_session_event[\"type_unique\"].to_pandas().astype(\"str\"))\ntest_session_event.reset_index(inplace=True)\n__display(\"**TEST:**\")\ndisplay(test_session_event)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T14:49:00.629653Z","iopub.execute_input":"2023-01-08T14:49:00.630026Z","iopub.status.idle":"2023-01-08T14:56:23.122104Z","shell.execute_reply.started":"2023-01-08T14:49:00.629992Z","shell.execute_reply":"2023-01-08T14:56:23.121030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Number of unique `event types` per `session`","metadata":{}},{"cell_type":"code","source":"__display('### **TRAIN** vs **TEST:** Number of unique `events` per `session`')\n\nside_by_side(train_session_event[\"type_nunique\"].value_counts(normalize=True).reset_index().rename({\"index\":\"n unique events\", 'type_nunique':\"percent\"},axis=1).to_pandas(),\n test_session_event[\"type_nunique\"].value_counts(normalize=True).reset_index().rename({\"index\":\"n unique events\", 'type_nunique':\"percent\"},axis=1).to_pandas()\n)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:01:31.394968Z","iopub.execute_input":"2023-01-08T15:01:31.395473Z","iopub.status.idle":"2023-01-08T15:01:31.461801Z","shell.execute_reply.started":"2023-01-08T15:01:31.395431Z","shell.execute_reply":"2023-01-08T15:01:31.460770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `Unique Event Types` in sessions","metadata":{}},{"cell_type":"code","source":"train_event_list_count = train_session_event['type_unique'].value_counts(\n).reset_index().rename({'index': \"event\", \"type_unique\": \"count\"}, axis=1)\ntrain_event_list_count['percent'] = (\n train_event_list_count['count']/train_session_event.shape[0])*100\ntest_event_list_count = test_session_event['type_unique'].value_counts(\n).reset_index().rename({'index': \"event\", \"type_unique\": \"count\"}, axis=1)\ntest_event_list_count['percent'] = (\n test_event_list_count['count']/test_session_event.shape[0])*100\n\n__display('### **TRAIN** vs **TEST:** Distribution of unique event types in sessions')\nside_by_side(train_event_list_count.to_pandas(),\n test_event_list_count.to_pandas())\n \ndel train_event_list_count, test_event_list_count\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:02:39.836651Z","iopub.execute_input":"2023-01-08T15:02:39.837096Z","iopub.status.idle":"2023-01-08T15:02:40.764593Z","shell.execute_reply.started":"2023-01-08T15:02:39.837037Z","shell.execute_reply":"2023-01-08T15:02:40.763464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the `train` dataset:\n* **~70%** of all sessions have just 1 event type:\n * it's just `click` events.\n* **~17.5%** have 2 event type:\n * most of them session with `clicks` & `add to cart` events - 17.2%, \n * but also apears session with `clicks` & `orders` events - 0.28%.\n* **12%** have 3 event type.\n\nFor the `test` dataset:\n* **~85%** of all sessions have just 1 event type:\n * most of session just `clicks` - 85%,\n * but also apears sessions with just `add to cart` events - ~0.063%\n * and sessions with just `orders` events - 0.028%\n* **~12%** have 2 event type:\n * most of them session with `clicks` & `add to cart` events - 12.492%, \n * also apears session with `clicks` & `orders` events - 0.146%,\n * even exist sessions with `add to cart` & `orders` event - 0.005%.\n* **2%** have 3 event type.\n\n> to understand more the distribution of event types, let's explore the `session` with **1** and **2** **unique events** and their event types **combinations**.\n","metadata":{}},{"cell_type":"markdown","source":"## `sessions` that have `1` unique event type","metadata":{}},{"cell_type":"code","source":"# Select session with only 1 unique event type\ntrain_session_event_one_unique = train_session_event.iloc[train_session_event['type_nunique']==1]\ndisplay(train_session_event_one_unique)\ntest_session_event_one_unique = test_session_event.iloc[test_session_event['type_nunique']==1]\ndisplay(test_session_event_one_unique)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:01.279762Z","iopub.execute_input":"2023-01-08T15:04:01.280166Z","iopub.status.idle":"2023-01-08T15:04:01.465300Z","shell.execute_reply.started":"2023-01-08T15:04:01.280131Z","shell.execute_reply":"2023-01-08T15:04:01.464130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"__display(\"**TRAIN**: list of unique event types for session that have only one unique event type\")\ndisplay(train_session_event_one_unique['type_unique'].astype('str').unique())\n__display(\"**TEST**: list of unique event types for session that have only one unique event type\")\ndisplay(test_session_event_one_unique['type_unique'].astype('str').unique())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:01.467317Z","iopub.execute_input":"2023-01-08T15:04:01.467784Z","iopub.status.idle":"2023-01-08T15:04:01.535625Z","shell.execute_reply.started":"2023-01-08T15:04:01.467742Z","shell.execute_reply":"2023-01-08T15:04:01.534380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"__display(\"## TEST: Distribution of event types for sessions with only `1` unique event type\")\ntmp = test_session_event_one_unique['type_unique'].astype('str').value_counts().reset_index().rename({'index':\"event\",\"type_unique\":\"count\"}, axis=1)\ntmp['percent'] = (tmp['count']/test_session_event_one_unique.shape[0])*100\ndisplay(tmp)\ndel tmp\ngc.collect()","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2023-01-08T15:04:01.537345Z","iopub.execute_input":"2023-01-08T15:04:01.538153Z","iopub.status.idle":"2023-01-08T15:04:02.181550Z","shell.execute_reply.started":"2023-01-08T15:04:01.538079Z","shell.execute_reply":"2023-01-08T15:04:02.180503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `sessions` that have `2` unique event type","metadata":{}},{"cell_type":"code","source":"train_session_event_two_unique = train_session_event.iloc[train_session_event['type_nunique']==2].reset_index(drop=True)\ndisplay(train_session_event_two_unique)\n\ntest_session_event_two_unique = test_session_event.iloc[test_session_event['type_nunique']==2].reset_index(drop=True)\ndisplay(test_session_event_two_unique)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:02.184531Z","iopub.execute_input":"2023-01-08T15:04:02.185190Z","iopub.status.idle":"2023-01-08T15:04:02.363817Z","shell.execute_reply.started":"2023-01-08T15:04:02.185153Z","shell.execute_reply":"2023-01-08T15:04:02.362594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tmp_train = train_session_event_two_unique['type_unique'].value_counts().reset_index().rename({'index':\"event\",\"type_unique\":\"count\"}, axis=1)\ntmp_train['percent'] = (tmp_train['count']/train_session_event_two_unique.shape[0])*100\ntmp_test = test_session_event_two_unique['type_unique'].value_counts().reset_index().rename({'index':\"event\",\"type_unique\":\"count\"}, axis=1)\ntmp_test['percent'] = (tmp_test['count']/test_session_event_two_unique.shape[0])*100\n__display(\"### **TRAIN** & **TEST :** Distribution of event types for sessions with `2` unique event type:\")\nside_by_side(tmp_train.to_pandas(), tmp_test.to_pandas())\n\ndel tmp_train,tmp_test\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:02.365961Z","iopub.execute_input":"2023-01-08T15:04:02.366430Z","iopub.status.idle":"2023-01-08T15:04:03.007709Z","shell.execute_reply.started":"2023-01-08T15:04:02.366387Z","shell.execute_reply":"2023-01-08T15:04:03.006522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_session_event_one_unique, test_session_event_one_unique ,train_session_event_two_unique, test_session_event_two_unique, train_session_event, test_session_event\ngc.collect()\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:03.009560Z","iopub.execute_input":"2023-01-08T15:04:03.009938Z","iopub.status.idle":"2023-01-08T15:04:03.646441Z","shell.execute_reply.started":"2023-01-08T15:04:03.009902Z","shell.execute_reply":"2023-01-08T15:04:03.645328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the investigation of sessions with `1` and `2` **unique event types**, we could notice that in the `test` dataset appears the sessions with **only** `add to cart` or `order` event types or the sessions with the combinations of this type of events. What might seem weird at first glance, but the documentation answers a similar topic question and explains the existence of such records.\n\n> [How can a session start with an order or a cart?](https://github.com/otto-de/recsys-dataset#how-can-a-session-start-with-an-order-or-a-cart)\n> * This can happen if the ordered item was already in the customer's cart before the data extraction period started. Similarly, a wishlist in our shop can lead to cart additions without a previous click.\n\n ","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# AID (Products)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/otto-img/img/aids.png\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:03.647937Z","iopub.execute_input":"2023-01-08T15:04:03.649232Z","iopub.status.idle":"2023-01-08T15:04:03.675480Z","shell.execute_reply.started":"2023-01-08T15:04:03.649178Z","shell.execute_reply":"2023-01-08T15:04:03.674437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Number of unique products","metadata":{}},{"cell_type":"code","source":"__display(f\"**TRAIN:** Number of unique product - **{train.aid.nunique()}**\")\n__display(f\"**TEST:** Number of unique product - **{test.aid.nunique()}**\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:03.676947Z","iopub.execute_input":"2023-01-08T15:04:03.677386Z","iopub.status.idle":"2023-01-08T15:04:03.819942Z","shell.execute_reply.started":"2023-01-08T15:04:03.677352Z","shell.execute_reply":"2023-01-08T15:04:03.818936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_of_common_product_in_train_tests = len(set(train[\"aid\"].to_pandas()).intersection(set(test[\"aid\"].to_pandas())))\n__display(f\"**Number of common** products in train and test datasets - {count_of_common_product_in_train_tests}\")\n__display(f\"**Number of different** product in test set - {test.aid.nunique()-count_of_common_product_in_train_tests}\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:03.821895Z","iopub.execute_input":"2023-01-08T15:04:03.822301Z","iopub.status.idle":"2023-01-08T15:04:51.077718Z","shell.execute_reply.started":"2023-01-08T15:04:03.822262Z","shell.execute_reply":"2023-01-08T15:04:51.076637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Overview:\n* `train` dataset contains ~**1.85M** of unique product ids.\n* `test` dataset doesn't contain any new product ids.","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"## AIDs Statistics","metadata":{}},{"cell_type":"code","source":"TOP_N = 5","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:51.082036Z","iopub.execute_input":"2023-01-08T15:04:51.082358Z","iopub.status.idle":"2023-01-08T15:04:51.087200Z","shell.execute_reply.started":"2023-01-08T15:04:51.082331Z","shell.execute_reply":"2023-01-08T15:04:51.086255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to color columns values\ndef color_columns(column, color_map):\n if column.name in color_map.keys():\n return [color_map[column.name] for i in range(len(column.values))]\n else:\n return [None for i in range(len(column.values))]","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:51.088587Z","iopub.execute_input":"2023-01-08T15:04:51.089349Z","iopub.status.idle":"2023-01-08T15:04:51.099806Z","shell.execute_reply.started":"2023-01-08T15:04:51.089314Z","shell.execute_reply":"2023-01-08T15:04:51.099030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the dataframe with amount of each event type for aid (product id) and some additional feature:\n# events_sum\tpercent_of_clicked\tpercent_of_add_to_cart\tpercent_of_orders\tcarts/clicks\torders/clicks\torders/carts\n\n# TRAIN\ntrain_aid_event_type_count = train.groupby(['aid','type']).agg({'session':['count']}).sort_index()\ntrain_aid_event_type_count.columns = train_aid_event_type_count.columns.droplevel()\ntrain_aid_event_type_count = train_aid_event_type_count.unstack(1)\ntrain_aid_event_type_count.columns = train_aid_event_type_count.columns.map('{0[0]}_{0[1]}'.format) \ntrain_aid_event_type_count.fillna(value=0, inplace=True)\ntrain_aid_event_type_count['events_sum'] = train_aid_event_type_count.sum(axis=1)\n\ntrain_aid_event_type_count['percent_of_clicked'] = (train_aid_event_type_count.count_0/train_aid_event_type_count.events_sum)*100\ntrain_aid_event_type_count['percent_of_add_to_cart'] = (train_aid_event_type_count.count_1/train_aid_event_type_count.events_sum)*100\ntrain_aid_event_type_count['percent_of_orders'] = (train_aid_event_type_count.count_2/train_aid_event_type_count.events_sum)*100\n\ntrain_aid_event_type_count['carts/clicks'] = (train_aid_event_type_count.count_1/train_aid_event_type_count.count_0)*100\ntrain_aid_event_type_count['orders/clicks'] = (train_aid_event_type_count.count_2/train_aid_event_type_count.count_0)*100\ntrain_aid_event_type_count['orders/carts'] = (train_aid_event_type_count.count_2/train_aid_event_type_count.count_1)*100\ntrain_aid_event_type_count.fillna(value=0, inplace=True)\n\n__display(\"**TRAIN**\")\ndisplay(train_aid_event_type_count)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:51.100965Z","iopub.execute_input":"2023-01-08T15:04:51.102079Z","iopub.status.idle":"2023-01-08T15:04:51.664110Z","shell.execute_reply.started":"2023-01-08T15:04:51.101984Z","shell.execute_reply":"2023-01-08T15:04:51.663053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the dataframe with amount of each event type for aid (product id) and some additional feature:\n# events_sum\tpercent_of_clicked\tpercent_of_add_to_cart\tpercent_of_orders\tcarts/clicks\torders/clicks\torders/carts\n\n# TEST\ntest_aid_event_type_count = test.groupby(['aid','type']).agg({'session':['count']}).sort_index()\ntest_aid_event_type_count.columns = test_aid_event_type_count.columns.droplevel()\ntest_aid_event_type_count = test_aid_event_type_count.unstack(1)\ntest_aid_event_type_count.columns = test_aid_event_type_count.columns.map('{0[0]}_{0[1]}'.format) \ntest_aid_event_type_count.fillna(value=0, inplace=True)\ntest_aid_event_type_count['events_sum'] = test_aid_event_type_count.sum(axis=1)\n\ntest_aid_event_type_count['percent_of_clicked'] = (test_aid_event_type_count.count_0/test_aid_event_type_count.events_sum)*100\ntest_aid_event_type_count['percent_of_add_to_cart'] = (test_aid_event_type_count.count_1/test_aid_event_type_count.events_sum)*100\ntest_aid_event_type_count['percent_of_orders'] = (test_aid_event_type_count.count_2/test_aid_event_type_count.events_sum)*100\n\ntest_aid_event_type_count['carts/clicks'] = (test_aid_event_type_count.count_1/test_aid_event_type_count.count_0)*100\ntest_aid_event_type_count['orders/clicks'] = (test_aid_event_type_count.count_2/test_aid_event_type_count.count_0)*100\ntest_aid_event_type_count['orders/carts'] = (test_aid_event_type_count.count_2/test_aid_event_type_count.count_1)*100\ntest_aid_event_type_count.fillna(value=0, inplace=True)\n\n__display(\"**TEST**\")\ndisplay(test_aid_event_type_count)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:51.665581Z","iopub.execute_input":"2023-01-08T15:04:51.665997Z","iopub.status.idle":"2023-01-08T15:04:51.895444Z","shell.execute_reply.started":"2023-01-08T15:04:51.665953Z","shell.execute_reply":"2023-01-08T15:04:51.894192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Percent of each event type for `aid`","metadata":{}},{"cell_type":"code","source":"__display(\"### **TRAIN** vs **TEST**: Percent of each event type for aid\")\ntrain_percent_stat = train_aid_event_type_count.loc[:,['percent_of_clicked','percent_of_add_to_cart', 'percent_of_orders']].describe().reset_index().loc[1:].to_pandas()\ntest_percent_stat = test_aid_event_type_count.loc[:,['percent_of_clicked','percent_of_add_to_cart', 'percent_of_orders']].describe().reset_index().loc[1:].to_pandas()\nside_by_side(train_percent_stat, test_percent_stat)\n","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:51.900712Z","iopub.execute_input":"2023-01-08T15:04:51.901357Z","iopub.status.idle":"2023-01-08T15:04:52.106247Z","shell.execute_reply.started":"2023-01-08T15:04:51.901313Z","shell.execute_reply":"2023-01-08T15:04:52.102721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* In **average** each `aid` has **90%** of `clicks`, **~7%** of `add to cart` and just **~2%** of `orders`.","metadata":{}},{"cell_type":"markdown","source":"## Top 5 aids by the number of all events","metadata":{}},{"cell_type":"code","source":"color_map = {\"events_sum\": f\"color: {main_color};\"}\n__display(f\"### **TRAIN:** Top {TOP_N} aids by the number of all events\")\ndisplay(train_aid_event_type_count.sort_values(by='events_sum', ascending=False)[:TOP_N].reset_index().to_pandas().style.apply(color_columns, color_map=color_map).bar(subset=[\"events_sum\"], color=third_color))\n__display(f\"### **TEST:** Top {TOP_N} aids by the number of all events\")\ndisplay(test_aid_event_type_count.sort_values(by='events_sum', ascending=False)[:TOP_N].reset_index().to_pandas().style.apply(color_columns, color_map=color_map).bar(subset=[\"events_sum\"], color=third_color))","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:52.107825Z","iopub.execute_input":"2023-01-08T15:04:52.113774Z","iopub.status.idle":"2023-01-08T15:04:52.430603Z","shell.execute_reply.started":"2023-01-08T15:04:52.113728Z","shell.execute_reply":"2023-01-08T15:04:52.429201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Clicks","metadata":{}},{"cell_type":"markdown","source":"### Top 5 **most** clickable aid","metadata":{}},{"cell_type":"code","source":"# Function to color common train and test aids records\ndef color_common_aids(val, common_aids_set):\n color = main_color if val in common_aids_set else 'black'\n return 'color: %s' % color","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:52.437010Z","iopub.execute_input":"2023-01-08T15:04:52.437988Z","iopub.status.idle":"2023-01-08T15:04:52.449396Z","shell.execute_reply.started":"2023-01-08T15:04:52.437948Z","shell.execute_reply":"2023-01-08T15:04:52.447701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create most clickable \n# TRAIN\ntrain_most_clickable = train_aid_event_type_count.sort_values(by='count_0', ascending=False).loc[:,['count_0']][:TOP_N].reset_index().to_pandas()\ntrain_most_clickable['percent_of_all_clicks'] = (train_most_clickable['count_0']/train_aid_event_type_count.count_0.sum())*100\ntrain_top_clickable_aids = train_most_clickable.aid\n\n# TEST\ntest_most_clickable = test_aid_event_type_count.sort_values(by='count_0', ascending=False).loc[:,['count_0']][:TOP_N].reset_index().to_pandas()\ntest_most_clickable['percent_of_all_clicks'] = (test_most_clickable['count_0']/test_aid_event_type_count.count_0.sum())*100\ntest_top_clickable_aids = test_most_clickable.aid\n\ncommon_aids = set(train_top_clickable_aids).intersection(set(test_top_clickable_aids))\n\ntrain_most_clickable = train_most_clickable.style.applymap(color_common_aids, common_aids_set = common_aids, subset=['aid'])\ntest_most_clickable = test_most_clickable.style.applymap(color_common_aids, common_aids_set = common_aids, subset=['aid'])\n\n__display(f\"### **TRAIN** VS **TEST**: Top {TOP_N} **most clickable** AIDs\")\nside_by_side(train_most_clickable, test_most_clickable)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:52.454393Z","iopub.execute_input":"2023-01-08T15:04:52.457773Z","iopub.status.idle":"2023-01-08T15:04:52.582587Z","shell.execute_reply.started":"2023-01-08T15:04:52.457737Z","shell.execute_reply":"2023-01-08T15:04:52.581639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Top 5 **less** clickable aid","metadata":{}},{"cell_type":"code","source":"# Create most clickable \n# TRAIN\ntrain_less_clickable = train_aid_event_type_count.sort_values(by='count_0', ascending=True).loc[:,['count_0']][:TOP_N].reset_index().to_pandas()\ntrain_less_clickable['percent_of_all_clicks'] = (train_less_clickable['count_0']/train_aid_event_type_count.count_0.sum())*100\ntrain_less_clickable_aids = train_less_clickable.aid\n# TEST\ntest_less_clickable = test_aid_event_type_count.sort_values(by='count_0', ascending=True).loc[:,['count_0']][:TOP_N].reset_index().to_pandas()\ntest_less_clickable['percent_of_all_clicks'] = (test_less_clickable['count_0']/test_aid_event_type_count.count_0.sum())*100\ntest_less_clickable_aids = test_less_clickable.aid\n\ncommon_aids = set(train_less_clickable_aids).intersection(set(test_less_clickable_aids))\n\ntrain_less_clickable = train_less_clickable.style.applymap(color_common_aids, common_aids_set = common_aids, subset=['aid'])\ntest_less_clickable = test_less_clickable.style.applymap(color_common_aids, common_aids_set = common_aids, subset=['aid'])\n\n__display(f\"### **TRAIN** VS **TEST**: Top {TOP_N} **less clickable** AIDs\")\nside_by_side(train_less_clickable, test_less_clickable)","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:52.584048Z","iopub.execute_input":"2023-01-08T15:04:52.584591Z","iopub.status.idle":"2023-01-08T15:04:52.712820Z","shell.execute_reply.started":"2023-01-08T15:04:52.584558Z","shell.execute_reply":"2023-01-08T15:04:52.711514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Summarize:\n* For train and test datasets exist common **most clickable** aids, 3 out of 5 most clickable are similar. But we can't say the same for **less clickable** aid. In train datasets, less clickable aids have some clicks while the test aids don't have any clicks. To understand more which amount of clicks per aid is ok, let's see the number of clicks per aid.","metadata":{}},{"cell_type":"markdown","source":"### Number of clicks per AID","metadata":{}},{"cell_type":"code","source":"# Display number of clicks per aid\ntrain_aid_clicks_stats = train_aid_event_type_count.count_0.describe()\n\nf, a0 = plt.subplots(1, 1, figsize=(24, 15))\nsns.histplot(train_aid_event_type_count.count_0.values.get()[::25], bins=500, kde=True, ax=a0, color=main_color)\na0.set_xlim(xmin=-10)\nfor i in ['50%']:\n a0.axvline(x=train_aid_clicks_stats[i], color=highlight_color)\n a0.text(s=f\"{i}\\n{train_aid_clicks_stats[i]}\" ,x=train_aid_clicks_stats[i]+1, y=a0.get_ylim()[1]/2, weight='bold')\nplt.suptitle(\"TRAIN: number of clicks per AID\", weight=\"bold\", size=25)\nplt.show()\n\n# TEST\ntest_aid_clicks_stats = test_aid_event_type_count.count_0.describe()\n\nf, a0 = plt.subplots(1, 1, figsize=(24, 15))\nsns.histplot(test_aid_event_type_count.count_0.values.get()[::25], bins=500, kde=True, ax=a0, color=main_color)\na0.set_xlim(xmin=-10)\nfor i in ['50%']:\n a0.axvline(x=test_aid_clicks_stats[i], color=highlight_color)\n a0.text(s=f\"{i}\\n{test_aid_clicks_stats[i]}\" ,x=test_aid_clicks_stats[i]+1, y=a0.get_ylim()[1]/2, weight='bold')\n\nplt.suptitle(\"TEST: number of clicks per AID\", weight=\"bold\", size=25)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:52.718189Z","iopub.execute_input":"2023-01-08T15:04:52.726329Z","iopub.status.idle":"2023-01-08T15:04:55.762427Z","shell.execute_reply.started":"2023-01-08T15:04:52.726288Z","shell.execute_reply":"2023-01-08T15:04:55.761332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"__display(\"### **TRAIN** VS **TEST**: Number of clicks per AID\")\nside_by_side(train_aid_clicks_stats.reset_index().iloc[1:].to_pandas(), test_aid_clicks_stats.reset_index().iloc[1:].to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:55.763999Z","iopub.execute_input":"2023-01-08T15:04:55.764627Z","iopub.status.idle":"2023-01-08T15:04:55.785587Z","shell.execute_reply.started":"2023-01-08T15:04:55.764587Z","shell.execute_reply":"2023-01-08T15:04:55.784723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Summarize:\n* As we can see from the charts and statistics above 50% of all aids have less than 18 clicks in train and less than 2 clicks in test datasets. ","metadata":{}},{"cell_type":"markdown","source":"## The number of clicks for **top clickable products** by day\n","metadata":{}},{"cell_type":"code","source":"# TRAIN\ntrain_aid_copy = train.loc[train.type==0,['aid']].to_pandas()\ntrain_top_clickable_aids_index_mask = train_aid_copy.loc[train_aid_copy.aid.isin(train_top_clickable_aids.to_list())].index\ntrain_top_clickable = train.iloc[train_top_clickable_aids_index_mask]\n\ndel train_aid_copy, train_top_clickable_aids_index_mask\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:55.787102Z","iopub.execute_input":"2023-01-08T15:04:55.787553Z","iopub.status.idle":"2023-01-08T15:04:59.613678Z","shell.execute_reply.started":"2023-01-08T15:04:55.787517Z","shell.execute_reply":"2023-01-08T15:04:59.612491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TEST\ntest_aid_copy = test.loc[ test.type==0,['aid']].to_pandas()\ntest_top_clickable_aids_index_mask = test_aid_copy.loc[test_aid_copy.aid.isin(test_top_clickable_aids.to_list())].index\ntest_top_clickable = test.iloc[test_top_clickable_aids_index_mask]\n\ndel test_aid_copy, test_top_clickable_aids_index_mask\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:04:59.615245Z","iopub.execute_input":"2023-01-08T15:04:59.615725Z","iopub.status.idle":"2023-01-08T15:05:00.250094Z","shell.execute_reply.started":"2023-01-08T15:04:59.615687Z","shell.execute_reply":"2023-01-08T15:05:00.249048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create `day`, `day of the week` feature from `ts` column\ntrain_date_tmp = cudf.to_datetime(train_top_clickable.ts, unit='s')\ntrain_top_clickable['day'] = train_date_tmp.dt.day\ntrain_top_clickable['day_of_week'] = train_date_tmp.dt.dayofweek\n\n\ntest_date_tmp = cudf.to_datetime(test_top_clickable.ts, unit='s')\ntest_top_clickable['day'] = test_date_tmp.dt.day\ntest_top_clickable['day_of_week'] = test_date_tmp.dt.dayofweek\n\ndel train_date_tmp, test_date_tmp\ntorch.cuda.empty_cache()\n\n__display(\"**TRAIN** & **TEST**\")\nside_by_side(train_top_clickable.head().to_pandas(), train_top_clickable.head().to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:00.251629Z","iopub.execute_input":"2023-01-08T15:05:00.252000Z","iopub.status.idle":"2023-01-08T15:05:00.280501Z","shell.execute_reply.started":"2023-01-08T15:05:00.251964Z","shell.execute_reply":"2023-01-08T15:05:00.279446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the barplot of the amount of events per each day in datasets\n# Subplots figure\nfig, axs = plt.subplots(len(train_top_clickable_aids),1, figsize=(20,15), gridspec_kw={'hspace': 0.5})\n\nfig.suptitle(\"TRAIN: Number of Clicks for Top clickable aids by day\", weight='bold')\nfor i, aid in enumerate(train_top_clickable_aids.to_list()):\n train_date_activity = train_top_clickable.loc[train_top_clickable.aid==aid, 'day'].value_counts().reset_index().to_pandas().rename(columns={\"index\":\"day\", \"day\":\"count\"})\n # plot train\n sns.barplot(data=train_date_activity, x='day', y='count', palette=train_date_colors, ax=axs[i], order=train_date_order)\n axs[i].set_title(f\"{aid}\", weight='bold', size=11)\n axs[i].set_ylabel(\"number of clicks\")\n\n if i != len(train_top_clickable_aids)-1:\n axs[i].set_xlabel(\"\")\n else:\n axs[i].set_xlabel(\"Day\", size=15, weight='bold')","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:00.282109Z","iopub.execute_input":"2023-01-08T15:05:00.282484Z","iopub.status.idle":"2023-01-08T15:05:02.201563Z","shell.execute_reply.started":"2023-01-08T15:05:00.282447Z","shell.execute_reply":"2023-01-08T15:05:02.200383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Subplots figure\nfig, axs = plt.subplots(len(test_top_clickable_aids),1, figsize=(20,15), gridspec_kw={'hspace': 0.5})\n\nfig.suptitle(\"TEST: Number of Clicks for Top clickable aids by day\", weight='bold')\nfor i, aid in enumerate(test_top_clickable_aids.to_list()):\n test_date_activity = test_top_clickable.loc[test_top_clickable.aid==aid, 'day'].value_counts().reset_index().to_pandas().rename(columns={\"index\":\"day\", \"day\":\"count\"})\n # plot train\n sns.barplot(data=test_date_activity, x='day', y='count', palette=test_date_colors, ax=axs[i], order=test_date_order)\n axs[i].set_title(f\"{aid}\", weight='bold', size=11)\n axs[i].set_ylabel(\"number of clicks\")\n\n if i != len(train_top_clickable_aids)-1:\n axs[i].set_xlabel(\"\")\n else:\n axs[i].set_xlabel(\"Day\", size=15, weight='bold')","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:02.203263Z","iopub.execute_input":"2023-01-08T15:05:02.203654Z","iopub.status.idle":"2023-01-08T15:05:03.269910Z","shell.execute_reply.started":"2023-01-08T15:05:02.203618Z","shell.execute_reply":"2023-01-08T15:05:03.268930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Summary:\nFrom the charts above, we can't notice any strange days when the number of clicks would be higher for most of the aids. And only 1 product out of 5 in both datasets, has only one day when it had some clicks, and the value of this `aid` is **485256**. For the train dataset, it has clicked on 23.08, for the test it's 30.08, and both of these dates are **Tuesdays**. \n\n> To understand more the `aid` **485256**, let's explore the number of all event types for this aid by day.","metadata":{}},{"cell_type":"markdown","source":"### AID **485256** number of event by day","metadata":{}},{"cell_type":"code","source":"# TRAIN\ntrain_485256 = train.loc[train.aid==485256]\ntrain_date_tmp = cudf.to_datetime(train_485256.ts, unit='s')\ntrain_485256['day'] = train_date_tmp.dt.day\ntrain_485256['day_of_week'] = train_date_tmp.dt.dayofweek\n\n__display(\"**TRAIN**\")\ndisplay(train_485256)\n\n# TEST\ntest_485256 = test.loc[test.aid==485256]\ntest_date_tmp = cudf.to_datetime(test_485256.ts, unit='s')\ntest_485256['day'] = test_date_tmp.dt.day\ntest_485256['day_of_week'] = test_date_tmp.dt.dayofweek\n\n__display(\"**TEST**\")\ndisplay(test_485256)\ndel train_date_tmp, test_date_tmp\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:03.271366Z","iopub.execute_input":"2023-01-08T15:05:03.271746Z","iopub.status.idle":"2023-01-08T15:05:04.025922Z","shell.execute_reply.started":"2023-01-08T15:05:03.271711Z","shell.execute_reply":"2023-01-08T15:05:04.024998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the barplot the number of each event type for aid 485256\nfig, axs = plt.subplots(3,2, figsize=(20,10),sharey='row',gridspec_kw={'width_ratios': [3, 1]})\n\ntrain_485256_day_events_type_count = train_485256.groupby(['day','type']).agg({'aid':'count'}).reset_index().rename(columns={\"aid\":\"count\"}).sort_values(by=['day','type'], ignore_index=True).to_pandas()\ntest_485256_day_events_type_count = test_485256.groupby(['day','type']).agg({'aid':'count'}).reset_index().rename(columns={\"aid\":\"count\"}).sort_values(by=['day','type'], ignore_index=True).to_pandas()\n\nfor i, ax in enumerate(axs):\n # plot train\n sns.barplot(data=train_485256_day_events_type_count.loc[train_485256_day_events_type_count.type==i], x='day', y='count', palette=train_date_colors, ax=ax[0], order=train_date_order)\n # plot test\n sns.barplot(data=test_485256_day_events_type_count.loc[test_485256_day_events_type_count.type==i], x='day', y='count', palette=test_date_colors, ax=ax[1], order = test_date_order)\n \n if i ==0:\n ax[0].set_title('Train Activity', weight=\"bold\", size=20)\n ax[1].set_title('Test Activity', weight=\"bold\", size=20)\n\n for a in ax:\n a.set_ylabel(f\"number of {event_type_map[i]}\", size=11, weight='bold')\n a.set_ylim(0)\n if i!=2:\n a.set_xlabel(\"\")\n else:\n a.set_xlabel(\"Day\", size=15, weight='bold')\n\ndel fig, axs, ax,a\ngc.collect()\ntorch.cuda.empty_cache()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:04.027556Z","iopub.execute_input":"2023-01-08T15:05:04.027936Z","iopub.status.idle":"2023-01-08T15:05:06.292130Z","shell.execute_reply.started":"2023-01-08T15:05:04.027899Z","shell.execute_reply":"2023-01-08T15:05:06.291003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"__display(f\"**TRAIN:** aid **485256** event days: {train_485256.day.unique().values}\")\n__display(f\"**TRAIN:** aid **485256** events: {train_485256.type.unique().values}\")\n\n__display(f\"**TEST:** aid **485256** event days: {test_485256.day.unique().values}\")\n__display(f\"**TEST:** aid **485256** events: {test_485256.type.unique().values}\")","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:06.298015Z","iopub.execute_input":"2023-01-08T15:05:06.300104Z","iopub.status.idle":"2023-01-08T15:05:06.318219Z","shell.execute_reply.started":"2023-01-08T15:05:06.300053Z","shell.execute_reply":"2023-01-08T15:05:06.317193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_485256, test_485256, train_485256_day_events_type_count, test_485256_day_events_type_count\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:06.319806Z","iopub.execute_input":"2023-01-08T15:05:06.320166Z","iopub.status.idle":"2023-01-08T15:05:06.887715Z","shell.execute_reply.started":"2023-01-08T15:05:06.320132Z","shell.execute_reply":"2023-01-08T15:05:06.886449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### AID **485256** summery:\nThe aid just has events on the days that were found early (train - 23.08, test - 30.08), so there are no other days when the aid would be clicked, added to cart or ordered. Also, the aid was **only** `clicked` and `added to carts` but there are `no orders`.","metadata":{}},{"cell_type":"markdown","source":"## Orders","metadata":{}},{"cell_type":"markdown","source":"### Top 5 **most ordered** aids\n* **Q1:** How to find the most ordered aids?\n * **A1:** Looking for the aids with the highest number of orders wouldn't show us the most ordered aids. Because in theory the aids with the highest amount of clicks would also have the highest amount of orders, but it doesn't mean that the aid is the most ordered.\n * **A2:** Looking for the aids with the highest orders percentage also would be wrong. Because from the official documentation known, that exist sessions which contain just add-to-cart or order events, so what means that it's also possible to have the aids with only add-to-cart or orders. Or the amount of such type events would be greater than the number of click events.\n> The topic still open, **How to find the aids that have the highest chance to be ordered?**","metadata":{}},{"cell_type":"markdown","source":"#### **A1:** **most ordered** by number of orders","metadata":{}},{"cell_type":"code","source":"color_map = {\"count_2\": f\"color: {main_color};\", \"percent_of_orders\": f\"color: {second_color};\"}\n__display(f\"### **TRAIN:** Top {TOP_N} **most ordered** AIDs\")\ndisplay(train_aid_event_type_count.sort_values(by='count_2', ascending=False)[:TOP_N].reset_index().to_pandas().style.apply(color_columns, color_map=color_map).bar(subset=[\"count_2\", \"percent_of_orders\"], color=third_color))\n__display(f\"### **TEST:** Top {TOP_N} **most ordered** AIDs\")\ndisplay(test_aid_event_type_count.sort_values(by='count_2', ascending=False)[:TOP_N].reset_index().to_pandas().style.apply(color_columns, color_map=color_map).bar(subset=[\"count_2\", \"percent_of_orders\"], color=third_color))","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:06.889732Z","iopub.execute_input":"2023-01-08T15:05:06.890315Z","iopub.status.idle":"2023-01-08T15:05:06.970119Z","shell.execute_reply.started":"2023-01-08T15:05:06.890271Z","shell.execute_reply":"2023-01-08T15:05:06.968930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **A2: most ordered** by percent of orders","metadata":{}},{"cell_type":"code","source":"color_map = {\"percent_of_orders\": f\"color: {main_color};\", \"count_2\": f\"color: {second_color};\", \"count_1\": f\"color: {second_color};\", \"count_0\": f\"color: {second_color};\"}\n__display(f\"### **TRAIN:** Top {TOP_N} **most ordered** AIDs\")\ndisplay(train_aid_event_type_count.sort_values(by='percent_of_orders', ascending=False)[:TOP_N].reset_index().to_pandas().style.apply(color_columns, color_map=color_map).bar(subset=[\"count_2\", \"percent_of_orders\"], color=third_color))\n__display(f\"### **TEST:** Top {TOP_N} **most ordered** AIDs\")\ndisplay(test_aid_event_type_count.sort_values(by='percent_of_orders', ascending=False)[:TOP_N].reset_index().to_pandas().style.apply(color_columns, color_map=color_map).bar(subset=[\"count_2\", \"percent_of_orders\"], color=third_color))","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:06.972088Z","iopub.execute_input":"2023-01-08T15:05:06.972792Z","iopub.status.idle":"2023-01-08T15:05:07.061145Z","shell.execute_reply.started":"2023-01-08T15:05:06.972750Z","shell.execute_reply":"2023-01-08T15:05:07.059779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **A3:** real **most ordered** aids","metadata":{}},{"cell_type":"code","source":"# ?","metadata":{"execution":{"iopub.status.busy":"2023-01-08T15:05:07.063347Z","iopub.execute_input":"2023-01-08T15:05:07.063811Z","iopub.status.idle":"2023-01-08T15:05:07.070956Z","shell.execute_reply.started":"2023-01-08T15:05:07.063768Z","shell.execute_reply":"2023-01-08T15:05:07.069530Z"},"trusted":true},"execution_count":null,"outputs":[]}]}