{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### SEEING IS KNOWING\n\nI plotted the session data by event type using a plotly timeline and gantt chart. Since each user action is just a snapshot in time, thus I used the `ts` as the start time of the action and added a short span of time to create a 'finish time' of the action. Hope this visualization will provide insights to your work!\n\nI will be grateful if you can give an upvote to this notebook 🙏 ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom matplotlib import pyplot as plt\nimport plotly.express as px\nimport plotly.figure_factory as ff\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-11-05T15:51:36.065511Z","iopub.execute_input":"2022-11-05T15:51:36.067449Z","iopub.status.idle":"2022-11-05T15:51:37.464835Z","shell.execute_reply.started":"2022-11-05T15:51:36.067306Z","shell.execute_reply":"2022-11-05T15:51:37.461170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us read in the `train` and `test` datasets.","metadata":{}},{"cell_type":"code","source":"train = pd.read_parquet('../input/otto-full-optimized-memory-footprint/train.parquet')\ntest = pd.read_parquet('../input/otto-full-optimized-memory-footprint/test.parquet')","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:51:37.471244Z","iopub.execute_input":"2022-11-05T15:51:37.471593Z","iopub.status.idle":"2022-11-05T15:51:50.314454Z","shell.execute_reply.started":"2022-11-05T15:51:37.471562Z","shell.execute_reply":"2022-11-05T15:51:50.313337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install pickle5\n\nimport pickle5 as pickle\n\nwith open('../input/otto-full-optimized-memory-footprint/id2type.pkl', \"rb\") as fh:\n    id2type = pickle.load(fh)","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:51:50.315489Z","iopub.execute_input":"2022-11-05T15:51:50.315858Z","iopub.status.idle":"2022-11-05T15:52:02.459305Z","shell.execute_reply.started":"2022-11-05T15:51:50.315826Z","shell.execute_reply":"2022-11-05T15:52:02.457657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:52:02.463908Z","iopub.execute_input":"2022-11-05T15:52:02.464339Z","iopub.status.idle":"2022-11-05T15:52:02.476575Z","shell.execute_reply.started":"2022-11-05T15:52:02.464303Z","shell.execute_reply":"2022-11-05T15:52:02.474959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:52:02.478240Z","iopub.execute_input":"2022-11-05T15:52:02.478652Z","iopub.status.idle":"2022-11-05T15:52:02.501236Z","shell.execute_reply.started":"2022-11-05T15:52:02.478617Z","shell.execute_reply":"2022-11-05T15:52:02.499658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualization","metadata":{}},{"cell_type":"code","source":"# this is the additional time (in minutes) added to action timestamp. This will affect the width of the bar in the chart.\n# Since you can hover and view the relevant data in the chart, you can ignore the 'Finish time' in the data. \n# The 'Start time' is the true timestamp of the user actions.\n\n# if you set this value too small, the bar will not be immediately visible unless you zoom in. \n# If you set this too large, the bars may overlapped with other nearby bars.\n# The tradeoff is: the smaller the value, the more accurate the data but less conveniently visible. Vice versa.\nadditional_mins = 60\n\n# choose the session id to wish to visualize\nsession_id = 2","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:55:43.714451Z","iopub.execute_input":"2022-11-05T15:55:43.714942Z","iopub.status.idle":"2022-11-05T15:55:43.721714Z","shell.execute_reply.started":"2022-11-05T15:55:43.714908Z","shell.execute_reply":"2022-11-05T15:55:43.720199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_df = train[train.session==session_id]\nsession_df['ts1'] = session_df['ts'] + 60*1000*additional_mins\nsession_df = session_df.astype({\"ts\": \"datetime64[ms]\", \"ts1\": \"datetime64[ms]\"})\n# the px.create_gantt function requires the dataframe to have columns names 'Start', 'Finish' and 'Task'\nsession_df = session_df.rename(columns={'ts':'Start','type':'Task','ts1':'Finish'})\nsession_df['Task'] = session_df['Task'].replace({0:'clicks', 1:'carts', 2:'orders'})\n\nsession_df","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:55:44.236298Z","iopub.execute_input":"2022-11-05T15:55:44.236757Z","iopub.status.idle":"2022-11-05T15:55:44.465706Z","shell.execute_reply.started":"2022-11-05T15:55:44.236722Z","shell.execute_reply":"2022-11-05T15:55:44.464411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Using px.timeline\nZoom in to view more data.","metadata":{"execution":{"iopub.status.busy":"2022-11-05T11:26:51.787691Z","iopub.execute_input":"2022-11-05T11:26:51.788926Z","iopub.status.idle":"2022-11-05T11:26:51.795251Z","shell.execute_reply.started":"2022-11-05T11:26:51.788856Z","shell.execute_reply":"2022-11-05T11:26:51.793669Z"}}},{"cell_type":"code","source":"px.timeline(\n    data_frame=session_df,\n    x_start='Start',\n    x_end='Finish',\n    y='Task',\n    color='Task',\n    hover_data=['aid'],\n    category_orders={'Task':['clicks', 'carts', 'orders']},\n    range_x=[session_df.Start.min(), session_df.Start.max()],\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:55:46.116086Z","iopub.execute_input":"2022-11-05T15:55:46.116574Z","iopub.status.idle":"2022-11-05T15:55:46.528428Z","shell.execute_reply.started":"2022-11-05T15:55:46.116528Z","shell.execute_reply":"2022-11-05T15:55:46.527520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Session 2 is an interesting example. The user put an item s/he didn't click before into the carts!","metadata":{}},{"cell_type":"markdown","source":"#### Using the create_gantt function","metadata":{}},{"cell_type":"code","source":"colors = {'clicks': 'rgb(220, 0, 0)',\n          'carts': 'rgb(3, 3, 252)',\n          'orders': 'rgb(59, 6, 28)'}\n\nfig = ff.create_gantt(session_df,colors=colors,  index_col='Task', show_colorbar=True,\n                      group_tasks=True, data='aid')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:52:03.716511Z","iopub.execute_input":"2022-11-05T15:52:03.716933Z","iopub.status.idle":"2022-11-05T15:52:03.781791Z","shell.execute_reply.started":"2022-11-05T15:52:03.716897Z","shell.execute_reply":"2022-11-05T15:52:03.780828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualizing multiple user sessions","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:07:33.670185Z","iopub.status.idle":"2022-11-05T15:07:33.670622Z","shell.execute_reply.started":"2022-11-05T15:07:33.670421Z","shell.execute_reply":"2022-11-05T15:07:33.670440Z"}}},{"cell_type":"code","source":"multiple_sessions = [1, 2, 3]","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:52:03.783259Z","iopub.execute_input":"2022-11-05T15:52:03.783656Z","iopub.status.idle":"2022-11-05T15:52:03.789465Z","shell.execute_reply.started":"2022-11-05T15:52:03.783620Z","shell.execute_reply":"2022-11-05T15:52:03.788076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"multi_session_df = train[train.session.isin(multiple_sessions)]\nmulti_session_df['ts1'] = multi_session_df['ts'] + 60*1000*additional_mins\nmulti_session_df = multi_session_df.astype({\"ts\": \"datetime64[ms]\", \"ts1\": \"datetime64[ms]\"})\n# the px.create_gantt function requires the dataframe to have columns names 'Start', 'Finish' and 'Task'\nmulti_session_df = multi_session_df.rename(columns={'ts':'Start','type':'Task','ts1':'Finish'})\nmulti_session_df['Task'] = multi_session_df['Task'].replace({0:'clicks', 1:'carts', 2:'orders'})\n\nmulti_session_df","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:52:03.791014Z","iopub.execute_input":"2022-11-05T15:52:03.791404Z","iopub.status.idle":"2022-11-05T15:52:05.027107Z","shell.execute_reply.started":"2022-11-05T15:52:03.791342Z","shell.execute_reply":"2022-11-05T15:52:05.025855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"px.timeline(\n    data_frame=multi_session_df,\n    x_start='Start',\n    x_end='Finish',\n    y='Task',\n    color='Task',\n    hover_data=['aid'],\n    category_orders={'Task':['clicks', 'carts', 'orders']},\n    range_x=[multi_session_df.Start.min(), multi_session_df.Start.max()],\n    facet_row='session',\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-05T15:56:49.337119Z","iopub.execute_input":"2022-11-05T15:56:49.337660Z","iopub.status.idle":"2022-11-05T15:56:49.583781Z","shell.execute_reply.started":"2022-11-05T15:56:49.337616Z","shell.execute_reply":"2022-11-05T15:56:49.582191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [WIP] Other interesting visualizations","metadata":{}},{"cell_type":"markdown","source":"### Reference\nThis work is built on the [full dataset](https://www.kaggle.com/code/radek1/eda-an-overview-of-the-full-dataset/notebook) by RADEK OSMULSKI.","metadata":{}}]}