{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":38760,"databundleVersionId":4493939,"sourceType":"competition"},{"sourceId":10626606,"sourceType":"datasetVersion","datasetId":6579497}],"dockerImageVersionId":30839,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n End-to-End Recommendation System Guide\n
\n\n\n
![\"image\"](\"https://plus.unsplash.com/premium_photo-1684179639963-e141ce2f8074?q=80&w=3570&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D\")
\n
Import Python Libraries
","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport json\nimport os\nimport glob\nimport dask.dataframe as dd","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:00.004038Z","iopub.execute_input":"2025-02-02T22:19:00.004537Z","iopub.status.idle":"2025-02-02T22:19:01.347447Z","shell.execute_reply.started":"2025-02-02T22:19:00.004497Z","shell.execute_reply":"2025-02-02T22:19:01.346361Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# β³ Load the Dataset
","metadata":{}},{"cell_type":"markdown","source":"## π Loading Train JSON Files & Converting to Parquet\n\nThis process ensures efficient handling of the large JSONL file by converting it to Parquet for faster access and reduced memory usage.\n\n**π Steps Taken:**\n\n1. **Preview the file:** Read and display only a few lines.\n1. **Check the structure:** Identify the keys and their types.\n1. **Read JSONL File in Chunks:** Use chunking or stream processing if the file is too large.\n - The dataset is too large to fit in RAM, so I used line-by-line processing to load it efficiently.\n - Used Pythonβs json module to read each line as a dictionary.\n\n1. **Extract Relevant Fields:**\n - Each record contains a **session ID** and a list of event dictionaries (aid, ts, type).\n - Flattened nested structures to create a structured DataFrame.\n\n1. **Batch Processing & Conversion:**\n - Used Pandas with chunking to process data in memory-efficient batches.\n - Converted each batch into a Pandas DataFrame and saved it in Parquet format.\n - ***Saved each chunk separately as Parquet files in the Kaggle output folder (/kaggle/working/).***\n\n\n1. **Why Parquet?**\n - Better compression β Reduces storage size.\n - Faster read/write speeds compared to CSV or JSON.\n - Optimized for large-scale analytics (columnar storage).\n\n1. **Merge All Parquet Files into a Single DataFrame:**\n - Read all the generated Parquet files from the output directory.\n - Concatenated them into a single DataFrame for analysis.\n - **Saved the final merged DataFrame as a single Parquet file in the Kaggle output folder.**\n\n1. **Download & Upload to Kaggle Datasets:**\n - Manually downloaded the final Parquet file to my local machine.\n - Created a Kaggle Dataset, uploaded the file, and imported it into my notebook as input.\n\n\n**π Final Note:**\n\nI executed this data processing pipeline once to generate the final Parquet file. The complete code is attached below for reference.\n\nAfter successfully creating the merged Parquet dataset, I commented out the code to prevent it from running on every notebook execution. \n\nNow, I simply load the final dataset from my uploaded Kaggle Dataset as input instead of re-processing the raw JSONL files. π","metadata":{}},{"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:06.219922Z","iopub.execute_input":"2025-02-02T22:19:06.220569Z","iopub.status.idle":"2025-02-02T22:19:06.234494Z","shell.execute_reply.started":"2025-02-02T22:19:06.220532Z","shell.execute_reply":"2025-02-02T22:19:06.232932Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Read and display a few lines to understand the structure\nfile_path = \"/kaggle/input/otto-recommender-system/train.jsonl\"\n# with open(file_path, \"r\", encoding=\"utf-8\") as f:\n# for _ in range(5): # Read first 5 lines\n# print(json.loads(f.readline()))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:06.555836Z","iopub.execute_input":"2025-02-02T22:19:06.556219Z","iopub.status.idle":"2025-02-02T22:19:06.560893Z","shell.execute_reply.started":"2025-02-02T22:19:06.556190Z","shell.execute_reply":"2025-02-02T22:19:06.559737Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Function to process the file in chunks\ndef process_jsonl_in_chunks(file_path, chunk_size=100000):\n \"\"\"Reads large JSONL file efficiently in chunks and converts it to a DataFrame.\"\"\"\n data = []\n batch = 0\n\n with open(file_path, \"r\", encoding=\"utf-8\") as f:\n for line in f:\n record = json.loads(line) # Load each JSON object\n\n # Extract session ID and expand the events\n session_id = record[\"session\"]\n for event in record[\"events\"]:\n event[\"session\"] = session_id # Add session ID to each event\n data.append(event)\n\n # Process in chunks\n if len(data) >= chunk_size:\n df = pd.DataFrame(data)\n df.to_parquet(f\"output_chunk_{batch}.parquet\", index=False) # Save to Parquet\n print(f\"Processed {batch * chunk_size} rows...\")\n data = [] # Reset list\n batch += 1\n\n # Process the remaining data\n if data:\n df = pd.DataFrame(data)\n df.to_parquet(f\"output_chunk_{batch}.parquet\", index=False)\n print(f\"Final batch processed: {batch * chunk_size} rows.\")\n\n print(\"Processing complete. Data saved as Parquet files.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:07.055864Z","iopub.execute_input":"2025-02-02T22:19:07.056321Z","iopub.status.idle":"2025-02-02T22:19:07.064359Z","shell.execute_reply.started":"2025-02-02T22:19:07.056287Z","shell.execute_reply":"2025-02-02T22:19:07.063116Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Run processing function\n#process_jsonl_in_chunks(file_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:07.626902Z","iopub.execute_input":"2025-02-02T22:19:07.627282Z","iopub.status.idle":"2025-02-02T22:19:07.631899Z","shell.execute_reply.started":"2025-02-02T22:19:07.627252Z","shell.execute_reply":"2025-02-02T22:19:07.630497Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# files = glob.glob(\"output_chunk_*.parquet\")\n# df = pd.concat([pd.read_parquet(f) for f in files], ignore_index=True)\n# df.to_parquet(\"final_output.parquet\", index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:09.184831Z","iopub.execute_input":"2025-02-02T22:19:09.185225Z","iopub.status.idle":"2025-02-02T22:19:09.189807Z","shell.execute_reply.started":"2025-02-02T22:19:09.185190Z","shell.execute_reply":"2025-02-02T22:19:09.188362Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load Parquet file\nfile_path = \"/kaggle/input/otto-parquet-full-training-file/final_output.parquet\"\ndf = dd.read_parquet(file_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:09.580599Z","iopub.execute_input":"2025-02-02T22:19:09.581019Z","iopub.status.idle":"2025-02-02T22:19:09.603904Z","shell.execute_reply.started":"2025-02-02T22:19:09.580988Z","shell.execute_reply":"2025-02-02T22:19:09.602777Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Load the Test Dataset\n","metadata":{}},{"cell_type":"code","source":"# Path to the test.jsonl file\nfile_path = \"/kaggle/input/otto-recommender-system/test.jsonl\"\n\n# Step 1: Read the JSONL file directly\nwith open(file_path, \"r\", encoding=\"utf-8\") as f:\n data = [json.loads(line) for line in f]\n\n# Step 2: Expand the nested 'events' column\nflattened_data = []\nfor record in data:\n session_id = record[\"session\"]\n for event in record[\"events\"]:\n event[\"session\"] = session_id # Add session ID to each event\n flattened_data.append(event)\n\n# Step 3: Convert to DataFrame\ntest_df = pd.DataFrame(flattened_data)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:11.005153Z","iopub.execute_input":"2025-02-02T22:19:11.005536Z","iopub.status.idle":"2025-02-02T22:19:48.279795Z","shell.execute_reply.started":"2025-02-02T22:19:11.005508Z","shell.execute_reply":"2025-02-02T22:19:48.278715Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_df.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:48.281109Z","iopub.execute_input":"2025-02-02T22:19:48.281392Z","iopub.status.idle":"2025-02-02T22:19:48.289052Z","shell.execute_reply.started":"2025-02-02T22:19:48.281368Z","shell.execute_reply":"2025-02-02T22:19:48.287964Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:48.291099Z","iopub.execute_input":"2025-02-02T22:19:48.291493Z","iopub.status.idle":"2025-02-02T22:19:48.326373Z","shell.execute_reply.started":"2025-02-02T22:19:48.291465Z","shell.execute_reply":"2025-02-02T22:19:48.325351Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# π Exploratory Data Analysis
\n\nHere are some ideas for the Exploratory Data Analysis (EDA), tailored to our multi-objective recommender system competition:\n\n1. **Data Overview & Basic Statistics:**\n - Check the number of unique sessions and total events. β
\n - Identify the distribution of event types (clicks, cart additions, orders). β
\n - Examine the time range covered in the dataset (earliest & latest timestamps). β
\n - Identify missing values or anomalies. β
\n\n2. **Session-Level Analysis:**\n - Analyze the distribution of session lengths (number of events per session). β
\n - Identify sessions with only one event vs. sessions with multiple interactions. β
\n - Study how session lengths correlate with conversions (do longer sessions result in more orders?).\n\n3. **Event Type Distribution:**\n - Compare the proportions of clicks, cart additions, and orders. β
\n - Examine how many sessions contain only clicks vs. those that proceed to cart and order. β
\n - Identify patterns in event sequences (e.g., common paths from click β cart β order).\n\n4. **Time-Based Patterns:**\n - Explore the distribution of timestamps (e.g., peak activity hours). β
\n - Analyze session duration (time difference between first and last event). β
\n - Study time gaps between events within a session.\n\n5. **Product (Article ID) Analysis:**\n - Find the most frequently clicked, added-to-cart, and ordered products. β
\n - Determine whether certain products have a high conversion rate (click β order). β
\n - Identify whether some products are frequently abandoned (added to cart but not ordered).\n\n6. **User Behavior Analysis:**\n - Identify common sequences of interactions (e.g., do users always click before adding to cart?).\n - Examine whether users revisit the same product multiple times before purchasing.\n - Look for session drop-off points (e.g., when do users stop engaging?).\n\n7. **Reordering Behavior:**\n - Check if users purchase the same product multiple times in a session.\n - Analyze products that are frequently added to cart multiple times before ordering.\n\n8. **Popularity Trends:**\n - Identify products with seasonal or time-based spikes in engagement.\n - Examine whether certain products have high engagement but low conversion.\n\n9. **Feature Engineering Insights:**\n - Explore the impact of past interactions on future events within the same session.\n - Identify whether session length, time gaps, or product popularity can be useful predictors.\n - Study if certain event patterns predict the likelihood of an order.","metadata":{}},{"cell_type":"markdown","source":"## 1. Data Overview and Basic Statistics\n","metadata":{}},{"cell_type":"code","source":"df.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:48.327911Z","iopub.execute_input":"2025-02-02T22:19:48.328230Z","iopub.status.idle":"2025-02-02T22:19:54.145784Z","shell.execute_reply.started":"2025-02-02T22:19:48.328203Z","shell.execute_reply":"2025-02-02T22:19:54.144491Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Convert Unix timestamp to datetime (milliseconds assumed)\ndf[\"datetime\"] = dd.to_datetime(df[\"ts\"], unit=\"ms\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:54.147058Z","iopub.execute_input":"2025-02-02T22:19:54.148314Z","iopub.status.idle":"2025-02-02T22:19:54.159736Z","shell.execute_reply.started":"2025-02-02T22:19:54.148273Z","shell.execute_reply":"2025-02-02T22:19:54.158437Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Convert Unix timestamp to datetime (milliseconds assumed)\ntest_df[\"datetime\"] = pd.to_datetime(test_df[\"ts\"], unit=\"ms\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:54.160936Z","iopub.execute_input":"2025-02-02T22:19:54.161246Z","iopub.status.idle":"2025-02-02T22:19:54.823602Z","shell.execute_reply.started":"2025-02-02T22:19:54.161220Z","shell.execute_reply":"2025-02-02T22:19:54.821762Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"nb_unique_sessions = df['session'].nunique().compute()\nprint(f\"The number of unique sessions: {nb_unique_sessions}\")\nprint(f\"The number of total events: {df.shape[0].compute()}\")\nprint(f\"Earliest Timestamp in the test set: {df['datetime'].min().compute()}\")\nprint(f\"Latest Timestamp in the test set: {df['datetime'].max().compute()}\")\nprint(f\"The number of unique article ids (product codes): {df['aid'].nunique().compute()}\")\n\n# Check for missing values (NaNs) in each column\nmissing_values = df.isnull().sum().compute()\nprint('Check for missing values')\nprint('-------------------------------------')\nmissing_values","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:19:54.824935Z","iopub.execute_input":"2025-02-02T22:19:54.825551Z","iopub.status.idle":"2025-02-02T22:21:27.566286Z","shell.execute_reply.started":"2025-02-02T22:19:54.825488Z","shell.execute_reply":"2025-02-02T22:21:27.565213Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Key Features:**\n\n- 12M real-world anonymized user sessions\n- 220M events, consiting of clicks, carts and orders\n- 1.8M unique articles in the catalogue","metadata":{}},{"cell_type":"code","source":"# Count occurrences of each event type\nevent_counts = df[\"type\"].value_counts().compute().reset_index()\nevent_counts.columns = [\"Event Type\", \"Count\"]\n\n# Calculate percentage\nevent_counts[\"Percentage %\"] = (event_counts[\"Count\"] / event_counts[\"Count\"].sum()) * 100","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:21:27.569749Z","iopub.execute_input":"2025-02-02T22:21:27.570086Z","iopub.status.idle":"2025-02-02T22:21:34.653526Z","shell.execute_reply.started":"2025-02-02T22:21:27.570049Z","shell.execute_reply":"2025-02-02T22:21:34.652157Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"event_counts.style.set_table_styles(\n [\n {\"selector\": \"th\", \"props\": [(\"background-color\", \"#016FD0\"), (\"color\", \"white\"), (\"font-weight\", \"bold\")]}\n ]\n).hide(axis=\"index\").background_gradient(subset=[\"Percentage %\"], cmap=\"Blues\") # Blue-Green gradient","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:21:34.654886Z","iopub.execute_input":"2025-02-02T22:21:34.655200Z","iopub.status.idle":"2025-02-02T22:21:34.709516Z","shell.execute_reply.started":"2025-02-02T22:21:34.655173Z","shell.execute_reply":"2025-02-02T22:21:34.708299Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1.1 Basic Stats of the Test Dataset","metadata":{}},{"cell_type":"code","source":"nb_unique_sessions_test = test_df['session'].nunique()\nprint(f\"The number of unique sessions: {nb_unique_sessions_test}\")\nprint(f\"The number of total events: {test_df.shape[0]}\")\nprint(f\"Earliest Timestamp in the training set: {test_df['datetime'].min()}\")\nprint(f\"Latest Timestamp in the training set: {test_df['datetime'].max()}\")\nprint(f\"The number of unique article ids (product codes): {test_df['aid'].nunique()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:21:34.710737Z","iopub.execute_input":"2025-02-02T22:21:34.711155Z","iopub.status.idle":"2025-02-02T22:21:35.098912Z","shell.execute_reply.started":"2025-02-02T22:21:34.711114Z","shell.execute_reply":"2025-02-02T22:21:35.097827Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 1. Overlap in Session IDs\ntrain_sessions = set(df[\"session\"].unique().compute())\ntest_sessions = set(test_df[\"session\"].unique())\n\nsession_overlap = train_sessions.intersection(test_sessions)\nnum_session_overlap = len(session_overlap)\n\n# 2. Overlap in Article IDs (AIDs)\ntrain_aids = set(df[\"aid\"].unique().compute())\ntest_aids = set(test_df[\"aid\"].unique())\n\naid_overlap = train_aids.intersection(test_aids)\nnum_aid_overlap = len(aid_overlap)\n\n# 3. Display Results\nprint(f\"Total overlapping session IDs: {num_session_overlap}\")\nprint(f\"Total overlapping article IDs (AIDs): {num_aid_overlap}\")\n\n# Percentage overlap for context\nsession_overlap_percentage = (num_session_overlap / len(test_sessions)) * 100\naid_overlap_percentage = (num_aid_overlap / len(test_aids)) * 100\n\nprint(f\"Session ID Overlap Percentage: {session_overlap_percentage:.2f}%\")\nprint(f\"AID Overlap Percentage: {aid_overlap_percentage:.2f}%\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:21:35.100128Z","iopub.execute_input":"2025-02-02T22:21:35.100447Z","iopub.status.idle":"2025-02-02T22:21:55.960047Z","shell.execute_reply.started":"2025-02-02T22:21:35.100421Z","shell.execute_reply":"2025-02-02T22:21:55.958798Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Session-Level Analysis","metadata":{}},{"cell_type":"code","source":"# Step 1: Group by session and count the number of events per session\nsession_lengths = df.groupby(\"session\").size().compute().reset_index(name=\"num_events\")\n\n# Step 2: Display the distribution of session lengths\n# Basic descriptive statistics\nsession_length_stats = session_lengths[\"num_events\"].describe()\n\nsession_length_stats","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:40:49.424583Z","iopub.execute_input":"2025-02-02T22:40:49.425060Z","iopub.status.idle":"2025-02-02T22:41:01.211107Z","shell.execute_reply.started":"2025-02-02T22:40:49.425030Z","shell.execute_reply":"2025-02-02T22:41:01.209751Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"session_length_stats.iloc[1]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:22:07.320153Z","iopub.execute_input":"2025-02-02T22:22:07.320492Z","iopub.status.idle":"2025-02-02T22:22:07.327442Z","shell.execute_reply.started":"2025-02-02T22:22:07.320464Z","shell.execute_reply":"2025-02-02T22:22:07.326195Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"session_length_stats.iloc[7]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:22:19.091597Z","iopub.execute_input":"2025-02-02T22:22:19.092109Z","iopub.status.idle":"2025-02-02T22:22:19.100029Z","shell.execute_reply.started":"2025-02-02T22:22:19.092063Z","shell.execute_reply":"2025-02-02T22:22:19.098762Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Event Type Distribution","metadata":{}},{"cell_type":"code","source":"# Define event types to keep\nevent_types_to_keep = [\"carts\", \"orders\"]\n\n# Step 1: Identify sessions that contain \"cart\" or \"order\"\nsession_filter = df[df[\"type\"].isin(event_types_to_keep)][\"session\"].unique().compute()\n\n# Step 2: Filter the original DataFrame to keep only these sessions\nfiltered_df = df[df[\"session\"].isin(session_filter)]\n\n# Define event type mappings\nfiltered_df[\"is_click\"] = (filtered_df[\"type\"] == \"clicks\").astype(\"int8\")\nfiltered_df[\"is_cart\"] = (filtered_df[\"type\"] == \"carts\").astype(\"int8\")\nfiltered_df[\"is_order\"] = (filtered_df[\"type\"] == \"orders\").astype(\"int8\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:22:28.028819Z","iopub.execute_input":"2025-02-02T22:22:28.029217Z","iopub.status.idle":"2025-02-02T22:22:38.584184Z","shell.execute_reply.started":"2025-02-02T22:22:28.029182Z","shell.execute_reply":"2025-02-02T22:22:38.583173Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"filtered_df.head()","metadata":{"trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-02-02T22:22:42.204341Z","iopub.execute_input":"2025-02-02T22:22:42.204726Z","iopub.status.idle":"2025-02-02T22:22:47.257060Z","shell.execute_reply.started":"2025-02-02T22:22:42.204695Z","shell.execute_reply":"2025-02-02T22:22:47.255762Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"num_rows = filtered_df.shape[0].compute() # Compute row count\nnum_columns = len(filtered_df.columns) # Column count\n\nprint(f\"Shape of DataFrame: ({num_rows}, {num_columns})\")\nprint(f\"The number of unique sessions that have at least one cart or order: {filtered_df['session'].nunique().compute()}\")","metadata":{"trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-02-02T22:22:47.258372Z","iopub.execute_input":"2025-02-02T22:22:47.258723Z","iopub.status.idle":"2025-02-02T22:23:32.504961Z","shell.execute_reply.started":"2025-02-02T22:22:47.258691Z","shell.execute_reply":"2025-02-02T22:23:32.503689Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Step 1: Group by session and sum up the click, cart, and order counts\n\nsession_event_counts = (\n filtered_df.groupby(\"session\")\n .agg(\n num_events=(\"is_click\", \"size\"), # Count total events per session\n total_clicks=(\"is_click\", \"sum\"), # Sum of clicks\n total_carts=(\"is_cart\", \"sum\"), # Sum of carts\n total_orders=(\"is_order\", \"sum\") # Sum of orders\n )\n .compute()\n .reset_index()\n)\n\n# Rename columns for clarity\nsession_event_counts.columns = [\"session_id\", \"num_events\", \"num_clicks\", \"num_carts\", \"num_orders\"]","metadata":{"trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-02-02T22:41:30.976960Z","iopub.execute_input":"2025-02-02T22:41:30.977384Z","iopub.status.idle":"2025-02-02T22:41:53.179800Z","shell.execute_reply.started":"2025-02-02T22:41:30.977349Z","shell.execute_reply":"2025-02-02T22:41:53.178531Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"session_event_counts.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:41:53.181761Z","iopub.execute_input":"2025-02-02T22:41:53.182080Z","iopub.status.idle":"2025-02-02T22:41:53.193365Z","shell.execute_reply.started":"2025-02-02T22:41:53.182054Z","shell.execute_reply":"2025-02-02T22:41:53.192111Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"session_event_counts[\"num_events\"].describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:48:11.777467Z","iopub.execute_input":"2025-02-02T22:48:11.777916Z","iopub.status.idle":"2025-02-02T22:48:11.913610Z","shell.execute_reply.started":"2025-02-02T22:48:11.777882Z","shell.execute_reply":"2025-02-02T22:48:11.911952Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"total_sessions = session_event_counts.shape[0]\nclicks_carts = session_event_counts[(session_event_counts['num_carts'] > 0) & (session_event_counts['num_orders'] == 0) ].shape[0]\nclicks_carts_orders = session_event_counts[(session_event_counts['num_carts'] > 0) & (session_event_counts['num_orders'] > 0) ].shape[0]\nclicks_ordrers = session_event_counts[(session_event_counts['num_orders'] > 0) & (session_event_counts['num_carts'] == 0) ].shape[0]\n\nprint(f\"Number of unique sessions that contain at least one extra event except for clicks: {total_sessions} - ~{round((total_sessions / nb_unique_sessions) * 100)}% of the total number of unique sessions in the dataset.\")\n\nprint(\"-------------------------------------------------------------------------------------------------------------------------------------\")\nprint(\"The percentages below are calculated based on the number of unique sessions that contain at least one extra event except for clicks!\")\nprint(f\"Number of unique sessions that contain clicks and carts events: {clicks_carts} - ~{round((clicks_carts / total_sessions) * 100)}%\")\nprint(f\"Number of unique sessions that contain clicks, carts and orders events: {clicks_carts_orders} - ~{round((clicks_carts_orders / total_sessions) * 100)}%\")\nprint(f\"Number of unique sessions that contain clicks and orders events: {clicks_ordrers} - ~{round((clicks_ordrers / total_sessions) * 100)}%\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:23:55.614919Z","iopub.execute_input":"2025-02-02T22:23:55.615361Z","iopub.status.idle":"2025-02-02T22:23:55.790068Z","shell.execute_reply.started":"2025-02-02T22:23:55.615316Z","shell.execute_reply":"2025-02-02T22:23:55.789003Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"only_clicks = df[~df['session'].isin(session_filter)]['session'].nunique().compute()\nprint(f\"Number of unique sessions that contain ONLY clicks events: {only_clicks} ~{round((only_clicks / nb_unique_sessions) * 100)}% of the total number of unique sessions in the dataset.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:23:55.791032Z","iopub.execute_input":"2025-02-02T22:23:55.791388Z","iopub.status.idle":"2025-02-02T22:24:05.095743Z","shell.execute_reply.started":"2025-02-02T22:23:55.791358Z","shell.execute_reply":"2025-02-02T22:24:05.094318Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"only_clicks_df = df[~df['session'].isin(session_filter)]\nonly_clicks_stats = (\n only_clicks_df.groupby(\"session\")\n .agg(\n num_events=(\"session\", \"size\"), # Count total events per session\n )\n .compute()\n .reset_index()\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:46:36.184908Z","iopub.execute_input":"2025-02-02T22:46:36.185764Z","iopub.status.idle":"2025-02-02T22:46:49.643263Z","shell.execute_reply.started":"2025-02-02T22:46:36.185482Z","shell.execute_reply":"2025-02-02T22:46:49.641560Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"only_clicks_stats[\"num_events\"].describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-02T22:48:06.655088Z","iopub.execute_input":"2025-02-02T22:48:06.655491Z","iopub.status.idle":"2025-02-02T22:48:06.985857Z","shell.execute_reply.started":"2025-02-02T22:48:06.655458Z","shell.execute_reply":"2025-02-02T22:48:06.984747Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Time-Based Patterns","metadata":{"_kg_hide-input":false}},{"cell_type":"code","source":"# Extract time-based features\ndf[\"hour\"] = df[\"datetime\"].dt.hour\n\n# Extract the weekday name\ndf[\"weekday\"] = df[\"datetime\"].dt.strftime(\"%A\") # Full weekday name (e.g., \"Monday\")\n\n# Extract day of the week as a number (0 = Monday, 6 = Sunday)\ndf[\"weekday_number\"] = df[\"datetime\"].dt.weekday\n\n# Extract the month name\ndf[\"month\"] = df[\"datetime\"].dt.month # Full month name (e.g., \"August\")","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Compute distribution of timestamps (e.g., peak activity hours)\nhourly_activity = df[\"hour\"].value_counts().compute().reset_index()\nhourly_activity.columns = [\"Hour\", \"Event Count\"]\nhourly_activity[\"Percentage %\"] = (hourly_activity[\"Event Count\"] / hourly_activity[\"Event Count\"].sum()) * 100\nhourly_activity = hourly_activity.sort_values(\"Hour\", ascending=True)","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"hourly_activity.style.set_table_styles(\n [\n {\"selector\": \"th\", \"props\": [(\"background-color\", \"#016FD0\"), (\"color\", \"white\"), (\"font-weight\", \"bold\")]}\n ]\n).hide(axis=\"index\").background_gradient(subset=[\"Percentage %\"], cmap=\"Blues\") # Blue-Green gradient","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Compute distribution of timestamps (e.g., peak activity hours)\ndaily_activity = df[\"weekday_number\"].value_counts().compute().reset_index()\ndaily_activity.columns = [\"Day\", \"Event Count\"]\ndaily_activity[\"Percentage %\"] = (daily_activity[\"Event Count\"] / daily_activity[\"Event Count\"].sum()) * 100\ndaily_activity = daily_activity.sort_values(\"Day\", ascending=True)\n\n# Mapping of day numbers to weekday names\nday_mapping = {\n 0: \"Monday\",\n 1: \"Tuesday\",\n 2: \"Wednesday\",\n 3: \"Thursday\",\n 4: \"Friday\",\n 5: \"Saturday\",\n 6: \"Sunday\"\n}\n\n# Assuming df is your DataFrame\ndaily_activity[\"Weekday\"] = daily_activity[\"Day\"].map(day_mapping)","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"daily_activity.style.set_table_styles(\n [\n {\"selector\": \"th\", \"props\": [(\"background-color\", \"#016FD0\"), (\"color\", \"white\"), (\"font-weight\", \"bold\")]}\n ]\n).hide(axis=\"index\").background_gradient(subset=[\"Percentage %\"], cmap=\"Blues\") # Blue-Green gradient","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Compute session start and end timestamps\nsession_times = df.groupby(\"session\")[\"datetime\"].agg([\"min\", \"max\"]).compute()\nsession_times[\"session_duration\"] = (session_times[\"max\"] - session_times[\"min\"]).dt.total_seconds()\n# Add session duration in minutes\nsession_times[\"session_duration_minutes\"] = session_times[\"session_duration\"] / 60\nsession_times[\"session_duration_hours\"] = session_times[\"session_duration\"] / 3600","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"session_times.head()","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Compute descriptive statistics\nsession_duration_stats = session_times[\"session_duration_hours\"].describe().reset_index()\nsession_duration_stats.columns = [\"statistic\", \"value\"]\nsession_duration_stats","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Product (Article ID) Analysis","metadata":{"execution":{"iopub.status.busy":"2025-01-31T15:11:58.541702Z","iopub.execute_input":"2025-01-31T15:11:58.542324Z","iopub.status.idle":"2025-01-31T15:11:58.560965Z","shell.execute_reply.started":"2025-01-31T15:11:58.542278Z","shell.execute_reply":"2025-01-31T15:11:58.559085Z"},"_kg_hide-input":false}},{"cell_type":"code","source":"df[\"is_click\"] = (df[\"type\"] == \"clicks\").astype(\"int8\")\ndf[\"is_cart\"] = (df[\"type\"] == \"carts\").astype(\"int8\")\ndf[\"is_order\"] = (df[\"type\"] == \"orders\").astype(\"int8\")","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Step 1: Count clicks, carts, and orders per product (aid)\nproduct_event_counts = df.groupby(\"aid\")[[\"is_click\", \"is_cart\", \"is_order\"]].sum().compute().reset_index()\n\n# Rename columns for clarity\nproduct_event_counts.columns = [\"aid\", \"num_clicks\", \"num_carts\", \"num_orders\"]","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Compute total number of clicks, carts, and orders\ntotal_clicks = product_event_counts[\"num_clicks\"].sum()\ntotal_carts = product_event_counts[\"num_carts\"].sum()\ntotal_orders = product_event_counts[\"num_orders\"].sum()","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Step 2: Calculate Conversion Rate (orders / clicks)\nproduct_event_counts[\"conversion_rate\"] = product_event_counts[\"num_orders\"] / product_event_counts[\"num_clicks\"]\nproduct_event_counts[\"conversion_rate\"] = product_event_counts[\"conversion_rate\"].fillna(0) # Handle division by zero\n\n# Add percentage columns\nproduct_event_counts[\"click_percentage\"] = (product_event_counts[\"num_clicks\"] / total_clicks) * 100\nproduct_event_counts[\"cart_percentage\"] = (product_event_counts[\"num_carts\"] / total_carts) * 100\nproduct_event_counts[\"order_percentage\"] = (product_event_counts[\"num_orders\"] / total_orders) * 100","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"product_event_counts = product_event_counts.sort_values(\"conversion_rate\", ascending=False)\n\nproduct_event_counts.head(20).style.set_table_styles(\n [\n {\"selector\": \"th\", \"props\": [(\"background-color\", \"#016FD0\"), (\"color\", \"white\"), (\"font-weight\", \"bold\")]}\n ]\n).hide(axis=\"index\").background_gradient(subset=[\"conversion_rate\"], cmap=\"Blues\") # Blue-Green gradient","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### π Pareto Analysis Insight\n\nIn the dataset, I performed a Pareto analysis to identify the distribution of orders across products. The results revealed that **7% of the products** account for **80% of the total orders,** confirming the Pareto Principle (80/20 rule), **where a small percentage of products drive the majority of sales.**\n\nπ Key Insight:\n\n**This indicates a high concentration of demand for a small subset of products.** Focusing on these high-performing products can:\n\n1. Optimize recommendation models by prioritizing popular items.\n2. Improve inventory management and marketing strategies targeting best-sellers.\n3. Highlight the importance of product popularity in driving conversion rates.","metadata":{}},{"cell_type":"code","source":"# Step 1: Sort products by the number of orders in descending order\nproduct_event_counts = product_event_counts.sort_values(\"num_orders\", ascending=False)\nproduct_event_counts[\"cumulative_orders\"] = product_event_counts[\"num_orders\"].cumsum() / product_event_counts[\"num_orders\"].sum()\nproduct_event_counts","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"(product_event_counts[\nproduct_event_counts['cumulative_orders'] <= 0.8\n].shape[0] / product_event_counts.shape[0]) * 100\n\n","metadata":{"trusted":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# π― Understanding the Goal of the Competition
\n\n## Problem Statement and Context\n\n***From the authors of the competition:*** [Kaggle: Machine Learning competition for the best Multi-Objective Recommender System](https://www.otto.de/jobs/en/technology/techblog/blogpost/otto-machine-learning-competition.php)\n\nMajor online retailers such as OTTO offer their customers millions of products to explore and purchase. However, finding the right product from such a vast selection without a little guidance can be exhausting! So we work to guide our customers to those products that best match their interests and motivations using **personalised recommendations**. For this reason we want to enhance our ability to forecast in real time which products each customer will want to see, add to their cart and order at any given moment of their visit.\n\nEven though an active research community focusing on recommender systems has established itself over the last couple of years, **there's still a lack of large-scale user-interaction datasets available in the e-commerce domain.** As a result, newly published models risk offering insufficient scalability when applied to retailers the size of OTTO. To tackle this problem and support further research in the area of session-based recommendations, we decided to publish a large-scale dataset that we gathered from anonymised behaviour logs generated in our webshop and shopping app. \n\nIn our target to ensure the popularity of our dataset, we quickly realised that combining it with a fun competition might well be the best way to get thousands of research teams interested in our data! We therefore decided to launch this competition on the popular data-science platform Kaggle, providing $30,000 in prize money for the three best submissions. Our competition kicks off on 01 November 2022 and we're warmly inviting everybody who's interested in applying machine learning to real-world problems to join in the fun here. The challenge will run for three months, ending on 31 January 2023. To help you get started, we also provide a [GitHub repository](https://github.com/otto-de/recsys-dataset) containing a complete dataset description and evaluation scripts.\n\nOnline shoppers have their pick of millions of products from large retailers. While such variety may be impressive, having so many options to explore can be overwhelming, resulting in shoppers leaving with empty carts. This neither benefits shoppers seeking to make a purchase nor retailers that missed out on sales. This is one reason online retailers rely on recommender systems to guide shoppers to products that best match their interests and motivations. Using data science to enhance retailers' ability to predict which products each customer actually wants to see, add to their cart, and order at any given moment of their visit in real-time could improve your customer experience the next time you shop online with your favorite retailer.\n\nCurrent recommender systems consist of various models with different approaches, ranging **from simple matrix factorization to a transformer-type deep neural network.** However, **no single model** exists that **can simultaneously optimize multiple objectives.** \n\nWith more than 10 million products from over 19,000 brands, OTTO is the largest German online shop. OTTO is a member of the Hamburg-based, multi-national Otto Group, which also subsidizes Crate & Barrel (USA) and 3 Suisses (France).\n\nYour work will help online retailers select more relevant items from a vast range to recommend to their customers based on their real-time behavior. Improving recommendations will ensure navigating through seemingly endless options is more effortless and engaging for shoppers.\n\n\n## Goal of the Competition\n\nThe task we're encouraging our participants to solve is to build a **multi-objective recommendation model** to optimise both \n\n- the **click-through** and \n- **purchase rates** of the recommended articles.\n\n**Most current state-of-the-art models only optimise for CTR,** so we hope this multi-objective task will serve as an exciting challenge for the ML community. The **goal** of this competition is to predict **e-commerce clicks**, **cart additions**, and **orders**. You'll build a multi-objective recommender system **based on previous events in a user session.** In this competition, youβll build a single entry to predict:\n\n1. **click-through**, \n1. **add-to-cart**, and\n2. **conversion rates** based on previous same-session events.\n\nπ― Target for Each Event Type\n\n1. **Clicks:**\n - Ground Truth: **Only one product** (the **next product clicked**).\n - Prediction: You can predict up to 20 products, ranked by likelihood of being clicked.\n\n1. **Carts:**\n - Ground Truth: **All products that were added to the cart after the last timestamp in the session.**\n - Prediction: Predict up to 20 products likely to be added to the cart.\n\n1. **Orders:**\n - Ground Truth: **All products that were ordered after the last timestamp.**\n - Prediction: Predict up to 20 products likely to be ordered.\n\n---------------\n- Data recorder over a 5 week period\n- There is no overlap between the train and the test data (referring to Session ID)\n- For each `session` in the **test data**, your task it to predict the `aid` values **for each type** that occur after the last timestamp `ts` the test session. In other words, the test data contains sessions truncated by timestamp, and you are to predict what occurs after the point of truncation.\n - For **clicks** there is **only a single ground truth value for each session**, which is the **next aid clicked during the session** (although you can still predict up to 20 aid values).\n - The **ground truth** for **carts** and **orders** contains **all aid values that were added to a cart and ordered respectively *during the session.***\n\n\n
![\"image\"](\"https://github.com/otto-de/recsys-dataset/blob/main/.readme/ground_truth.png?raw=true\")
\n