{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84493,"databundleVersionId":9871156,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Table of Contents","metadata":{}},{"cell_type":"markdown","source":"* [Load Data](#chapter1)\n* [Weight](#chapter1)","metadata":{"execution":{"iopub.status.busy":"2024-11-06T19:20:38.986445Z","iopub.execute_input":"2024-11-06T19:20:38.987143Z","iopub.status.idle":"2024-11-06T19:20:38.995458Z","shell.execute_reply.started":"2024-11-06T19:20:38.987100Z","shell.execute_reply":"2024-11-06T19:20:38.993441Z"}}},{"cell_type":"markdown","source":"# TLDR","metadata":{}},{"cell_type":"markdown","source":"- Total days\n    - 1700 across 10 partitions, 170 days per partition\n- Features:\n    - 80% of missing vaules are at the beginning of the day, time_id 1-67\n    - Most features have missing value everyday. Feature_8 is an outlider, only has extrememly large number of null values on specific days, spikes observed\n    - Features with high correlations are from the same tag\n    - Features from the same tag group tend to have similar distribution\n \n- Symbol_id:\n    - The number of symbol_id increases\n    - Some symbols are not available at the beginning of the dataset\n \n- Weight:\n    - The sum of weight increases across the 1700 days\n    - All weights are positive values, ranging from 0.149 - 10.24\n \n- resp_6\n    - correlation between resp_6 and other responders: 8 7 > 3 > 5 6\n    - correlation between resp_6 and features: weak\n \n","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\n\n# plotting \nfrom pandas.plotting import lag_plot\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport missingno as msno\nimport plotly.express as px\nimport plotly.graph_objects as go\ncolorMap = sns.light_palette(\"blue\", as_cmap=True)\nfrom statsmodels.tsa.seasonal import seasonal_decompose\n\nimport os\nimport gc","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:59:18.321949Z","iopub.execute_input":"2024-11-08T00:59:18.322423Z","iopub.status.idle":"2024-11-08T00:59:22.154814Z","shell.execute_reply.started":"2024-11-08T00:59:18.322371Z","shell.execute_reply":"2024-11-08T00:59:22.153470Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# configs\npd.set_option('display.max_columns', None) # we want to display all columns in this notebook\npd.set_option('display.max_rows', 100) # increase number of displayed rows","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:59:24.826519Z","iopub.execute_input":"2024-11-08T00:59:24.827251Z","iopub.status.idle":"2024-11-08T00:59:24.836388Z","shell.execute_reply.started":"2024-11-08T00:59:24.827203Z","shell.execute_reply":"2024-11-08T00:59:24.834774Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 1. Load Data <a class=\"anchor\"  id=\"chapter1\"></a>","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:59:27.812917Z","iopub.execute_input":"2024-11-08T00:59:27.813820Z","iopub.status.idle":"2024-11-08T00:59:27.868753Z","shell.execute_reply.started":"2024-11-08T00:59:27.813762Z","shell.execute_reply":"2024-11-08T00:59:27.867048Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n# Define the base path for the dataset\nbase_path = '/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/'\n\n# Can't concat all partitions together, as it would kill the kernel\ntrain_file_0 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=0/part-0.parquet\")\ntrain_file_1 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=1/part-0.parquet\")\ntrain_file_2 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=2/part-0.parquet\")\ntrain_file_3 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=3/part-0.parquet\")\ntrain_file_4 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=4/part-0.parquet\")\ntrain_file_5 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=5/part-0.parquet\")\ntrain_file_6 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=6/part-0.parquet\")\ntrain_file_7 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=7/part-0.parquet\")\ntrain_file_8 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=8/part-0.parquet\")\ntrain_file_9 = pd.read_parquet(\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=9/part-0.parquet\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:59:30.594773Z","iopub.execute_input":"2024-11-08T00:59:30.595299Z","iopub.status.idle":"2024-11-08T01:00:50.654368Z","shell.execute_reply.started":"2024-11-08T00:59:30.595237Z","shell.execute_reply":"2024-11-08T01:00:50.650268Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Each partition file has 170 days\n# all partition has 170 * 10=1700 days /365 = 4.7 years\ntrain_file_0['date_id'].unique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:53:06.199287Z","iopub.execute_input":"2024-11-06T21:53:06.199814Z","iopub.status.idle":"2024-11-06T21:53:06.230869Z","shell.execute_reply.started":"2024-11-06T21:53:06.199767Z","shell.execute_reply":"2024-11-06T21:53:06.229567Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_file_0.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:53:08.794723Z","iopub.execute_input":"2024-11-06T21:53:08.795232Z","iopub.status.idle":"2024-11-06T21:53:08.874206Z","shell.execute_reply.started":"2024-11-06T21:53:08.795184Z","shell.execute_reply":"2024-11-06T21:53:08.873015Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Weight <a class=\"anchor\"  id=\"chapter2\"></a>","metadata":{}},{"cell_type":"code","source":"# concat weights together, to find out trend in weights\n\n# List to store the weight column from each DataFrame\nweight_columns = []\n\n# Loop through each DataFrame by directly using the variable names\nfor i in range(10):\n    # Access each train_file_i DataFrame dynamically and extract the 'weight' column\n    df = globals()[f'train_file_{i}']\n    weight_columns.append(df[['weight', 'date_id', 'time_id', 'symbol_id', \n                              'responder_0', 'responder_1', 'responder_2',\n                              'responder_3', 'responder_4', 'responder_5',\n                              'responder_6', 'responder_7', 'responder_8',]]) \n\n# Concatenate all weight columns into a single DataFrame\nfinal_weight_df = pd.concat(weight_columns, ignore_index=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T01:00:57.110053Z","iopub.execute_input":"2024-11-08T01:00:57.110680Z","iopub.status.idle":"2024-11-08T01:01:02.366681Z","shell.execute_reply.started":"2024-11-08T01:00:57.110621Z","shell.execute_reply":"2024-11-08T01:01:02.364913Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"final_weight_df.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:53:40.610058Z","iopub.execute_input":"2024-11-06T21:53:40.610603Z","iopub.status.idle":"2024-11-06T21:53:40.619131Z","shell.execute_reply.started":"2024-11-06T21:53:40.610531Z","shell.execute_reply":"2024-11-06T21:53:40.617850Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"final_weight_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:53:42.378811Z","iopub.execute_input":"2024-11-06T21:53:42.379325Z","iopub.status.idle":"2024-11-06T21:53:42.398524Z","shell.execute_reply.started":"2024-11-06T21:53:42.379281Z","shell.execute_reply":"2024-11-06T21:53:42.397185Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"percent_zeros = (100/final_weight_df.shape[0])*((final_weight_df.weight.values < 0).sum())\npercent_zeros","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:53:44.567217Z","iopub.execute_input":"2024-11-06T21:53:44.567677Z","iopub.status.idle":"2024-11-06T21:53:44.733968Z","shell.execute_reply.started":"2024-11-06T21:53:44.567633Z","shell.execute_reply":"2024-11-06T21:53:44.732603Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"final_weight_df['weight'].describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:53:46.955477Z","iopub.execute_input":"2024-11-06T21:53:46.956761Z","iopub.status.idle":"2024-11-06T21:53:49.275676Z","shell.execute_reply.started":"2024-11-06T21:53:46.956706Z","shell.execute_reply":"2024-11-06T21:53:49.274406Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Trend of Weight Overall","metadata":{}},{"cell_type":"markdown","source":"## Sum of weight by date_id","metadata":{}},{"cell_type":"code","source":"sum_weight_per_day = final_weight_df.groupby('date_id')['weight'].sum().reset_index()\n\nplt.figure(figsize=(14, 7))\nplt.plot(sum_weight_per_day['date_id'], sum_weight_per_day['weight'], color='blue', alpha=0.7)\nplt.title('Sum Weight per Day')\nplt.xlabel('Date ID')\nplt.ylabel('Mean Weight')\nplt.xticks(rotation=45)\nplt.grid()\nplt.tight_layout()  # Adjust the layout to prevent clipping of labels\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:53:56.149042Z","iopub.execute_input":"2024-11-06T21:53:56.149515Z","iopub.status.idle":"2024-11-06T21:53:56.638238Z","shell.execute_reply.started":"2024-11-06T21:53:56.149469Z","shell.execute_reply":"2024-11-06T21:53:56.636839Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Sum of weight by date_id - Time Series Analysis","metadata":{}},{"cell_type":"code","source":"decomposition_sum = seasonal_decompose(sum_weight_per_day['weight'], \n                                   model='additive', \n                                   period=20) # suppose a week (trading days)) = 5 days,\n                                                # a month = 20 days\n\n# Plot the decomposed components\nplt.figure(figsize=(12, 10))\nplt.subplot(4, 1, 1)\nplt.plot(sum_weight_per_day['weight'], label='Original')\nplt.legend(loc='upper left')\nplt.subplot(4, 1, 2)\nplt.plot(decomposition_sum.trend, label='Trend')\nplt.legend(loc='upper left')\nplt.subplot(4, 1, 3)\nplt.plot(decomposition_sum.seasonal, label='Seasonal')\nplt.legend(loc='upper left')\nplt.subplot(4, 1, 4)\nplt.plot(decomposition_sum.resid, label='Residual')\nplt.legend(loc='upper left')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:30:20.531897Z","iopub.execute_input":"2024-11-08T00:30:20.532461Z","iopub.status.idle":"2024-11-08T00:30:21.567065Z","shell.execute_reply.started":"2024-11-08T00:30:20.532411Z","shell.execute_reply":"2024-11-08T00:30:21.565490Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Mean of weigiht by date_id","metadata":{}},{"cell_type":"code","source":"mean_weight_per_day = final_weight_df.groupby('date_id')['weight'].mean().reset_index()\n\nplt.figure(figsize=(14, 7))\nplt.plot(mean_weight_per_day['date_id'], mean_weight_per_day['weight'], color='blue', alpha=0.7)\nplt.title('Mean Weight per Day')\nplt.xlabel('Date ID')\nplt.ylabel('Mean Weight')\nplt.xticks(rotation=45)\nplt.grid()\nplt.tight_layout()  # Adjust the layout to prevent clipping of labels\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:54:05.675700Z","iopub.execute_input":"2024-11-06T21:54:05.676206Z","iopub.status.idle":"2024-11-06T21:54:06.120890Z","shell.execute_reply.started":"2024-11-06T21:54:05.676161Z","shell.execute_reply":"2024-11-06T21:54:06.119630Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Mean of weight by date_id - Time Series Analysis","metadata":{}},{"cell_type":"code","source":"# Perform seasonal decomposition\ndecomposition_mean = seasonal_decompose(mean_weight_per_day['weight'], \n                                   model='additive', \n                                   period=20) # 5 or 20\n\n# Plot the decomposed components\nplt.figure(figsize=(12, 10))\nplt.subplot(4, 1, 1)\nplt.plot(mean_weight_per_day['weight'], label='Original')\nplt.legend(loc='upper left')\nplt.subplot(4, 1, 2)\nplt.plot(decomposition_mean.trend, label='Trend')\nplt.legend(loc='upper left')\nplt.subplot(4, 1, 3)\nplt.plot(decomposition_mean.seasonal, label='Seasonal')\nplt.legend(loc='upper left')\nplt.subplot(4, 1, 4)\nplt.plot(decomposition_mean.resid, label='Residual')\nplt.legend(loc='upper left')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:31:07.600811Z","iopub.execute_input":"2024-11-08T00:31:07.601401Z","iopub.status.idle":"2024-11-08T00:31:08.634841Z","shell.execute_reply.started":"2024-11-08T00:31:07.601328Z","shell.execute_reply":"2024-11-08T00:31:08.633414Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Original Series (Top Panel): The series fluctuates significantly, with visible peaks and valleys. There appears to be an overall trend where weights vary but generally decrease until around day 1000, after which there is a steady increase and subsequent decline.\n\n- Trend Component (Second Panel): Pretty much same as the original.\n  \n- Seasonal Component (Third Panel): This component reveals a cyclical pattern with minor fluctuations around the baseline. The amplitude is relatively low (around ±0.02), meaning that the seasonal influence is subtle compared to the trend and overall fluctuations.\n\n- Residual Component (Bottom Panel): The residuals are somewhat volatile, particularly in the earlier part of the series. This noise could represent random fluctuations or other irregular influences that aren’t captured by the trend or seasonality.","metadata":{}},{"cell_type":"markdown","source":"## Trend of Weight by symbol_id","metadata":{}},{"cell_type":"code","source":"# Group by symbol_id and date_id to get the sum of weights for each symbol on each day\nweights_per_symbol = final_weight_df.groupby(['symbol_id', 'date_id'])['weight'].sum().reset_index()\nweights_per_symbol","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T23:52:45.770988Z","iopub.execute_input":"2024-11-07T23:52:45.771511Z","iopub.status.idle":"2024-11-07T23:52:52.347428Z","shell.execute_reply.started":"2024-11-07T23:52:45.771469Z","shell.execute_reply":"2024-11-07T23:52:52.346113Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Trend of Weight for each Symbol_id by date_id","metadata":{}},{"cell_type":"code","source":"# Get unique symbol IDs\nunique_symbols = weights_per_symbol['symbol_id'].unique()\n\n# Define the number of symbols per subplot and the number of subplots\nsymbols_per_plot = 5\nnum_plots = (len(unique_symbols) + symbols_per_plot - 1) // symbols_per_plot  # Calculate number of plots needed\n\n# Set up the figure with a 2x2 grid for 4 subplots\nfig, axs = plt.subplots(4, 2, figsize=(18, 20))\naxs = axs.flatten()  # Flatten to easily iterate over axes\n\n# Loop over each subplot and plot 10 symbols per subplot\nfor i in range(num_plots):\n    start = i * symbols_per_plot\n    end = start + symbols_per_plot\n    symbols_subset = unique_symbols[start:end]\n    \n    # Plot each symbol's data in the corresponding subplot\n    for symbol_id in symbols_subset:\n        symbol_data = weights_per_symbol[weights_per_symbol['symbol_id'] == symbol_id]\n        axs[i].plot(symbol_data['date_id'], symbol_data['weight'], label=f'Symbol {symbol_id}', alpha=0.7)\n    \n    # Set titles and labels for each subplot\n    axs[i].set_title(f\"Sum of Weights per Day for Symbols {start + 1} to {min(end, len(unique_symbols))}\")\n    axs[i].set_xlabel(\"Date ID\")\n    axs[i].set_ylabel(\"Sum of Weights\")\n    axs[i].legend(loc='upper right', bbox_to_anchor=(1.15, 1))  # Adjust legend to avoid overlap\n    axs[i].grid(True)\n\n# Adjust layout to prevent overlapping\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:54:52.992806Z","iopub.execute_input":"2024-11-06T21:54:52.994241Z","iopub.status.idle":"2024-11-06T21:54:56.159135Z","shell.execute_reply.started":"2024-11-06T21:54:52.994186Z","shell.execute_reply":"2024-11-06T21:54:56.157406Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Meaning some symbol_ids start to have weights later in the dataset\n\nA more detailed graph for each symbol_id over date_id: https://www.kaggle.com/code/ravi20076/janestreet2024-eda-v1?scriptVersionId=203721491&cellId=21","metadata":{}},{"cell_type":"markdown","source":"## Days with missing value for each symbol_id","metadata":{}},{"cell_type":"code","source":"# Define the full range of dates and symbols\nall_dates = weights_per_symbol['date_id'].unique()\nall_symbols = weights_per_symbol['symbol_id'].unique()\n\n# Create a complete DataFrame with all combinations of date_id and symbol_id\nall_combinations = pd.MultiIndex.from_product([all_symbols, all_dates], names=['symbol_id', 'date_id'])\ncomplete_df = pd.DataFrame(index=all_combinations).reset_index()\n\n# Merge with the actual data to find missing entries\nmerged_df = complete_df.merge(weights_per_symbol, on=['symbol_id', 'date_id'], how='left')\n\n# Create a missing indicator: 1 if weight is missing, 0 if it's present\nmerged_df['missing'] = merged_df['weight'].isnull().astype(int)\n\n# Pivot the DataFrame to get symbols as rows and dates as columns\nmissing_matrix = merged_df.pivot(index='symbol_id', columns='date_id', values='missing')\n\n# Plot the heatmap\nplt.figure(figsize=(18, 10))\nsns.heatmap(missing_matrix, cmap='YlGnBu', cbar_kws={'label': 'Missing Indicator (1 = Missing)'})\nplt.title(\"Days with Missing Data for Each Symbol\")\nplt.xlabel(\"Date ID\")\nplt.ylabel(\"Symbol ID\")\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:55:05.428081Z","iopub.execute_input":"2024-11-06T21:55:05.428521Z","iopub.status.idle":"2024-11-06T21:55:06.876669Z","shell.execute_reply.started":"2024-11-06T21:55:05.428483Z","shell.execute_reply":"2024-11-06T21:55:06.875188Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Conclusion:\n\nSo new symbol_ids are added half way across the 1700 days\n\nA more detailed analysis: Number of symbols traded over 1700 days: https://www.kaggle.com/code/shiyili/js2024-rmf-understanding-the-data?scriptVersionId=201826986&cellId=8","metadata":{}},{"cell_type":"markdown","source":"## Correlations between symbols based on weights","metadata":{}},{"cell_type":"code","source":"# Group by symbol_id and sum weights for each symbol\nsymbol_weight_sum = final_weight_df.groupby(['symbol_id'])['weight'].sum().reset_index()\n\n# Calculate the total weight across all symbols\ntotal_weight = symbol_weight_sum['weight'].sum()\n\n# Add a new column for the percentage of weight for each symbol\nsymbol_weight_sum['weight_percentage'] = (symbol_weight_sum['weight'] / total_weight) * 100\n\n# print(symbol_weight_sum)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:56:10.975334Z","iopub.execute_input":"2024-11-06T21:56:10.975850Z","iopub.status.idle":"2024-11-06T21:56:12.075022Z","shell.execute_reply.started":"2024-11-06T21:56:10.975804Z","shell.execute_reply":"2024-11-06T21:56:12.073549Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"final_weight_df_agg = final_weight_df.groupby(['date_id', 'symbol_id'])['weight'].sum().reset_index()\n\n# Now pivot the data to have symbols as columns and date_id as the index\nweights_pivot = final_weight_df_agg.pivot(index='date_id', columns='symbol_id', values='weight')\n\n# Calculate the correlation matrix\ncorrelation_matrix = weights_pivot.corr()\n\n# Plot the correlation matrix as a heatmap\nplt.figure(figsize=(18, 14))\nsns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0, annot_kws={\"size\": 8})\nplt.title(\"Correlation Matrix of Symbols Based on Weight\")\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:56:18.374759Z","iopub.execute_input":"2024-11-06T21:56:18.375295Z","iopub.status.idle":"2024-11-06T21:56:25.999127Z","shell.execute_reply.started":"2024-11-06T21:56:18.375249Z","shell.execute_reply":"2024-11-06T21:56:25.997849Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Features","metadata":{}},{"cell_type":"markdown","source":"## Types of Value for each Feature (Discrete, Continuous, Categorical)","metadata":{}},{"cell_type":"markdown","source":"* Conclusion\n    * Continuous: feature 00-08, 12-78\n    * Discrete: 9-11 - Only 3 features in tag_1\n    * Categorical: None","metadata":{}},{"cell_type":"code","source":"features = [col for col in train_file_9.columns if col.startswith('feature_')]\nfor feature in features:\n    print(feature, \"s type is: \", train_file_9[feature].dtype)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:56:31.230739Z","iopub.execute_input":"2024-11-06T21:56:31.231283Z","iopub.status.idle":"2024-11-06T21:56:31.246329Z","shell.execute_reply.started":"2024-11-06T21:56:31.231231Z","shell.execute_reply":"2024-11-06T21:56:31.244759Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Feature Distribution","metadata":{}},{"cell_type":"markdown","source":"- Conclusion:\n  - Features from the same tag seem to have similar distribution","metadata":{}},{"cell_type":"code","source":"# Plot distributions for each feature\n# so that we can know whether the values are discrete, continuous, categorial\nfig, axes = plt.subplots(nrows=10, ncols=8, figsize=(20, 25))\naxes = axes.flatten()\n\nfor i, col in enumerate(train_file_9[features]):\n    ax = axes[i]\n    train_file_9[col].hist(bins=30, ax=ax)\n    ax.set_title(col)\n    ax.set_xticks([])  # Optional: Hide xticks for readability\n    ax.set_yticks([])  # Optional: Hide yticks for readability\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:56:38.298797Z","iopub.execute_input":"2024-11-06T21:56:38.299282Z","iopub.status.idle":"2024-11-06T21:56:58.964036Z","shell.execute_reply.started":"2024-11-06T21:56:38.299234Z","shell.execute_reply":"2024-11-06T21:56:58.962195Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Correlations\n\n- Conclusion\n    - Features from the same tag group have high correlation.\n    - feature 73-78 - tag_8\n    - feature 67-72, feature 12-14 - tag_9\n    - feature 62-64 feature 15-17 - tag_6\n    - feature 39-44 high correlation - tag_4, tag_5\n    - feature 39-50 mid high correlation - tag_3\n","metadata":{}},{"cell_type":"code","source":"# Set up the figure size for a large heatmap\nplt.figure(figsize=(25, 20))\n\n# Generate the heatmap for the entire dataset\nsns.heatmap(train_file_9.corr(), annot=False, cmap='coolwarm', linewidths=0.5)\n\n# Add title\nplt.title('Heatmap of Correlations for responder0')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T22:16:18.173589Z","iopub.execute_input":"2024-11-07T22:16:18.174159Z","iopub.status.idle":"2024-11-07T22:18:56.362779Z","shell.execute_reply.started":"2024-11-07T22:16:18.174108Z","shell.execute_reply":"2024-11-07T22:18:56.361245Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# Patterns on Missing value","metadata":{}},{"cell_type":"markdown","source":"## Total Missing Values by feature_id","metadata":{}},{"cell_type":"code","source":"df_dict = {\n    'df1': train_file_0,\n    'df2': train_file_1,\n    'df3': train_file_2,\n    'df4': train_file_3,\n    'df5': train_file_4,\n    'df6': train_file_5,\n    'df7': train_file_6,\n    'df8': train_file_7,\n    'df9': train_file_8,\n    'df10': train_file_9,\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:57:04.984822Z","iopub.execute_input":"2024-11-06T21:57:04.985316Z","iopub.status.idle":"2024-11-06T21:57:04.992053Z","shell.execute_reply.started":"2024-11-06T21:57:04.985269Z","shell.execute_reply":"2024-11-06T21:57:04.990502Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Calculate the number of null values for each column in each dataframe\nnull_counts = {name: df.isnull().sum() for name, df in df_dict.items()}\n\n# Convert the results into a single DataFrame for better visualization\nnull_counts_df = pd.DataFrame(null_counts)\nnull_counts_df.loc['Total_Null_Values'] = null_counts_df.sum()\nnull_counts_df['Total_Per_Feature'] = null_counts_df.sum(axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:57:06.904603Z","iopub.execute_input":"2024-11-06T21:57:06.905107Z","iopub.status.idle":"2024-11-06T21:57:13.293206Z","shell.execute_reply.started":"2024-11-06T21:57:06.905059Z","shell.execute_reply":"2024-11-06T21:57:13.291992Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Add percentage of null values for each feature\ntotal_null_values = null_counts_df['Total_Per_Feature'].iloc[4:83].sum()\nnull_counts_df['Percentage_Null_Values'] = (null_counts_df['Total_Per_Feature'] / total_null_values) * 100","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:57:15.828123Z","iopub.execute_input":"2024-11-06T21:57:15.828619Z","iopub.status.idle":"2024-11-06T21:57:15.836845Z","shell.execute_reply.started":"2024-11-06T21:57:15.828558Z","shell.execute_reply":"2024-11-06T21:57:15.835427Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"null_counts_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:57:18.791740Z","iopub.execute_input":"2024-11-06T21:57:18.792323Z","iopub.status.idle":"2024-11-06T21:57:18.833476Z","shell.execute_reply.started":"2024-11-06T21:57:18.792267Z","shell.execute_reply":"2024-11-06T21:57:18.831870Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Number of null values decreased significantly across time.\n\nAnother analysis on sum(null_values) over 1700 days, same conclusion: https://www.kaggle.com/code/motono0223/eda-v2-jane-street-real-time-market-forecasting\n","metadata":{}},{"cell_type":"markdown","source":"## Total Missing Value by feature_id - Tags","metadata":{}},{"cell_type":"code","source":"feature_tags = pd.read_csv(\"/kaggle/input/jane-street-real-time-market-data-forecasting/features.csv\" ,index_col=0)\n# convert to binary\nfeature_tags = feature_tags*1\nfeature_tags['Total_Null_Per_feature'] = null_counts_df['Total_Per_Feature'].iloc[4:83]\nfeature_tags['Percentage_Null_Values'] = null_counts_df['Percentage_Null_Values'].iloc[4:83]\n\nfeature_tags.style.background_gradient(cmap='Blues')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T19:53:14.109722Z","iopub.execute_input":"2024-11-06T19:53:14.110177Z","iopub.status.idle":"2024-11-06T19:53:14.366510Z","shell.execute_reply.started":"2024-11-06T19:53:14.110136Z","shell.execute_reply":"2024-11-06T19:53:14.365226Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Feature 73-78: All in tag_8, tends to miss values during specific time intervals as the number of missing value in each df is the same\n- Feautre 0-4: Tag 2","metadata":{}},{"cell_type":"markdown","source":"## Daily Patterns of Missing Values","metadata":{}},{"cell_type":"markdown","source":"Values are not missing randomly.\n\nDaily patterns of missing value from day 1530 - 1536\n- All 7 days have missing value at the beginning of the day (about 80% of missing valules are under time_id=1~67 ref:https://www.kaggle.com/code/aymanallawi/basic-insights-on-missing-values)\n- Some days have missing values in the middle of the day\n- Missing values at the end of the day","metadata":{}},{"cell_type":"code","source":"# Calculate the number of missing values for each row\ntrain_file_9['n_missing_vals'] = train_file_9.isna().sum(axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T20:40:17.100925Z","iopub.execute_input":"2024-11-07T20:40:17.102703Z","iopub.status.idle":"2024-11-07T20:40:20.448804Z","shell.execute_reply.started":"2024-11-07T20:40:17.102627Z","shell.execute_reply":"2024-11-07T20:40:20.446397Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create subplots for the specified date_id range\nf, ax = plt.subplots(nrows=1, ncols=7, figsize=(28, 8))\n\n# Loop through the specified date_id range (1530 to 1536)\nfor i in range(7):\n    day = 1530 + i\n    # Filter and plot for each specific date_id in the range\n    train_file_9[train_file_9.date_id == day].n_missing_vals.reset_index(drop=True).plot(ax=ax[i])\n    ax[i].set_title(f'Day {day}')\n    ax[i].set_ylim([-1, 80])\n\n# Set overall title\nplt.suptitle('Daily Patterns of Missing Values for date_id 1530 to 1536', fontsize=16)\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T20:40:23.845415Z","iopub.execute_input":"2024-11-07T20:40:23.845903Z","iopub.status.idle":"2024-11-07T20:40:25.582020Z","shell.execute_reply.started":"2024-11-07T20:40:23.845858Z","shell.execute_reply":"2024-11-07T20:40:25.580600Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Just another visualisation picking day 1654. The y-axis is time, x-axis is 79 features","metadata":{}},{"cell_type":"code","source":"day_1654 =  train_file_9[train_file_9['date_id'] == 1654].iloc[:, 4:83]\nmsno.matrix(day_1654, color=(0.35, 0.35, 0.75));\n# Y-asix is all the date_id and time_id, X-Axis are columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T20:53:46.480895Z","iopub.execute_input":"2024-11-07T20:53:46.481328Z","iopub.status.idle":"2024-11-07T20:53:47.897694Z","shell.execute_reply.started":"2024-11-07T20:53:46.481287Z","shell.execute_reply":"2024-11-07T20:53:47.896163Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* Basically, feature 21 26 27 31 has null values at specific time regularly during a day\n* Other features have null values at the beginning of a day","metadata":{}},{"cell_type":"markdown","source":"## Missing Values for Selected Features","metadata":{}},{"cell_type":"markdown","source":"We are interested in whether feaetures with relatively high number of null values have missing values every day","metadata":{}},{"cell_type":"code","source":"# Select specific features to analyze for null trends\nselected_features = [\n    'feature_15', 'feature_17', 'feature_21', 'feature_26', 'feature_27', \n    'feature_31', 'feature_32', 'feature_33', 'feature_39', 'feature_41', 'feature_42', \n    'feature_44', 'feature_45', 'feature_46', 'feature_50', 'feature_52', 'feature_53', \n    'feature_55', 'feature_58', 'feature_65', 'feature_66', 'feature_73', 'feature_74', \n    'feature_75', 'feature_76', 'feature_77', 'feature_78'\n]\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T21:50:05.271778Z","iopub.execute_input":"2024-11-07T21:50:05.272238Z","iopub.status.idle":"2024-11-07T21:50:05.279061Z","shell.execute_reply.started":"2024-11-07T21:50:05.272186Z","shell.execute_reply":"2024-11-07T21:50:05.277697Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Merge the last 3 partitions for analysis","metadata":{}},{"cell_type":"code","source":"df7_selected = train_file_7[selected_features + ['date_id', 'time_id', 'symbol_id']]\ndf8_selected = train_file_8[selected_features + ['date_id', 'time_id', 'symbol_id']]\ndf9_selected = train_file_9[selected_features + ['date_id', 'time_id', 'symbol_id']]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T21:50:07.165350Z","iopub.execute_input":"2024-11-07T21:50:07.165838Z","iopub.status.idle":"2024-11-07T21:50:09.156274Z","shell.execute_reply.started":"2024-11-07T21:50:07.165793Z","shell.execute_reply":"2024-11-07T21:50:09.154302Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"concatenated_df789 = pd.concat([df7_selected, df8_selected, df9_selected], ignore_index=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T21:50:10.892727Z","iopub.execute_input":"2024-11-07T21:50:10.893192Z","iopub.status.idle":"2024-11-07T21:50:12.910433Z","shell.execute_reply.started":"2024-11-07T21:50:10.893148Z","shell.execute_reply":"2024-11-07T21:50:12.908556Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Calculate the number of null values for each feature by date_id\nnull_trends = concatenated_df789.groupby('date_id')[selected_features].apply(lambda x: x.isnull().sum())\n\n# Plot the trends\nplt.figure(figsize=(14, 8))\nfor feature in selected_features:\n    plt.plot(null_trends.index, null_trends[feature], label=feature)\n\nplt.xlabel('Date ID')\nplt.ylabel('Number of Null Values')\nplt.title('Trend of Null Values for Selected Features over Time')\nplt.legend()\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T22:19:20.388985Z","iopub.execute_input":"2024-11-06T22:19:20.389486Z","iopub.status.idle":"2024-11-06T22:19:25.837950Z","shell.execute_reply.started":"2024-11-06T22:19:20.389445Z","shell.execute_reply":"2024-11-06T22:19:25.836571Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We're interested to see whether the features have missing values at specific time intervals during the day.\n\nAnswer: Missing value at the beginning of the day","metadata":{}},{"cell_type":"code","source":"selected_days = [1654]\n# Filter data for the selected days\nfiltered_data = concatenated_df789[concatenated_df789['date_id'].isin(selected_days)]\n\n# Check for missing values by date_id and time_id for each selected feature\nmissing_intervals = filtered_data.groupby(['date_id', 'time_id'])[selected_features].apply(lambda x: x.isnull().sum())\n\n# Plot missing values for each feature across the time intervals of the selected days\nfig, axes = plt.subplots(len(selected_features), 1, figsize=(14, 20), sharex=True)\n\nfor i, feature in enumerate(selected_features):\n    ax = axes[i]\n    for day in selected_days:\n        day_data = missing_intervals.loc[day][feature]\n        ax.plot(day_data.index, day_data.values, label=f'Day {day}')\n    \n    ax.set_title(f\"Missing Values for {feature} by Time Interval (time_id)\")\n    ax.set_ylabel('Missing Count')\n    ax.legend()\n\nplt.xlabel('Time ID')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T22:26:45.524705Z","iopub.execute_input":"2024-11-06T22:26:45.525298Z","iopub.status.idle":"2024-11-06T22:26:52.949260Z","shell.execute_reply.started":"2024-11-06T22:26:45.525249Z","shell.execute_reply":"2024-11-06T22:26:52.947713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Outlier - feature_08","metadata":{}},{"cell_type":"markdown","source":"Feature 8 is an outlier tends to have spikes in the number of null values on specific days","metadata":{}},{"cell_type":"code","source":"# Calculate the number of null values for each feature by date_id\nnull_trends = train_file_9.groupby('date_id')['feature_08'].apply(lambda x: x.isnull().sum())\n\n# Plot the trends\nplt.figure(figsize=(14, 8))\nplt.plot(null_trends.index, null_trends, label='feature_08')\n\nplt.xlabel('Date ID')\nplt.ylabel('Number of Null Values')\nplt.title('Trend of Null Values for Selected Features over Time')\nplt.legend()\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T21:54:41.235512Z","iopub.execute_input":"2024-11-07T21:54:41.236001Z","iopub.status.idle":"2024-11-07T21:54:41.950115Z","shell.execute_reply.started":"2024-11-07T21:54:41.235958Z","shell.execute_reply":"2024-11-07T21:54:41.948308Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# List of data files\ntrain_files = [train_file_0, train_file_1, train_file_2, train_file_3, train_file_4,\n               train_file_5, train_file_6, train_file_7, train_file_8, train_file_9]\n\n# Set up the plot\nplt.figure(figsize=(14, 8))\n\n# Loop through each file and plot null trends for feature_08\nfor i, train_file in enumerate(train_files):\n    # Calculate null values for feature_08 by date_id for each file\n    null_trends = train_file.groupby('date_id')['feature_08'].apply(lambda x: x.isnull().sum())\n    \n    # Plot the null trend with a label for each file\n    plt.plot(null_trends.index, null_trends, label=f'train_file_{i} feature_08')\n\n# Labels and title\nplt.xlabel('Date ID')\nplt.ylabel('Number of Null Values')\nplt.title('Trend of Null Values for Feature 08 over Time across train_file_0 to train_file_9')\nplt.legend()\nplt.grid(True)\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T21:54:19.166746Z","iopub.execute_input":"2024-11-07T21:54:19.167252Z","iopub.status.idle":"2024-11-07T21:54:22.226716Z","shell.execute_reply.started":"2024-11-07T21:54:19.167205Z","shell.execute_reply":"2024-11-07T21:54:22.225141Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"## Missing Values by Symbol_id","metadata":{}},{"cell_type":"markdown","source":"Let's pick features that has high number of missing values and break it down by symbol_id.\nWe're interested in the patterns for symbols.\nMissing values for symbol might mean that some assets were not available during specific dates","metadata":{}},{"cell_type":"markdown","source":"So for feature 21 26 27, symbol_36 has not data since","metadata":{}},{"cell_type":"code","source":"# Filter the data for symbol_id=1 and select feature_42\nfiltered_data = concatenated_df789[(concatenated_df789['symbol_id'] == 9)]\n\n# Group by date_id and calculate the count of missing values for feature_42\nmissing_trend = filtered_data.groupby('date_id')['feature_31'].apply(lambda x: x.isnull().sum())\n\n# Plot the trend of missing values over date_id\nplt.figure(figsize=(10, 6))\nmissing_trend.plot( linestyle='-', color='b')\nplt.title(\"Trend of Missing Values for Feature 21 (symbol_id=1) Over Date ID\")\nplt.xlabel(\"Date ID\")\nplt.ylabel(\"Number of Missing Values\")\nplt.xticks(rotation=45)\nplt.grid(True)\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T22:27:32.724779Z","iopub.execute_input":"2024-11-06T22:27:32.725421Z","iopub.status.idle":"2024-11-06T22:27:33.371560Z","shell.execute_reply.started":"2024-11-06T22:27:32.725373Z","shell.execute_reply":"2024-11-06T22:27:33.370124Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Filter data for symbol_id=1, date_id=1625, and select feature_42\nfiltered_data = concatenated_df789[(concatenated_df789['symbol_id'] == 0) & (train_file_9['date_id'] == 1625)]\n\n# Group by time_id and calculate the count of missing values for feature_42\nmissing_trend_time_id = filtered_data.groupby('time_id')['feature_32'].apply(lambda x: x.isnull().sum())\n\n# Plot the trend of missing values over time_id within the day\nplt.figure(figsize=(10, 6))\nmissing_trend_time_id.plot(linestyle='-', color='b')\nplt.title(\"Trend of Missing Values for Feature 42 (symbol_id=1, date_id=1625) Over Time ID\")\nplt.xlabel(\"Time ID\")\nplt.ylabel(\"Number of Missing Values\")\nplt.xticks(rotation=45)\nplt.grid(True)\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T22:27:53.792926Z","iopub.execute_input":"2024-11-06T22:27:53.793414Z","iopub.status.idle":"2024-11-06T22:27:58.566458Z","shell.execute_reply.started":"2024-11-06T22:27:53.793370Z","shell.execute_reply":"2024-11-06T22:27:58.564950Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"null_values_by_symbol_id = concatenated_df789.groupby('symbol_id')[selected_features].apply(lambda x: x.isnull().sum())\nnull_values_by_symbol_id.loc['Total_Null_Values'] = null_values_by_symbol_id.sum()\nnull_values_by_symbol_id['Total_Null_By_Symbol'] = null_values_by_symbol_id.sum(axis=1)\n\nnull_values_by_symbol_id","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T20:51:18.975435Z","iopub.execute_input":"2024-11-06T20:51:18.975934Z","iopub.status.idle":"2024-11-06T20:51:28.769061Z","shell.execute_reply.started":"2024-11-06T20:51:18.975891Z","shell.execute_reply":"2024-11-06T20:51:28.767953Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Calculate the number of null values for each symbol_id and date_id across the selected features\nmissing_values_by_symbol_date = concatenated_df789.groupby(['date_id', 'symbol_id']).apply(lambda x: x.isnull().sum().sum()).reset_index(name='missing_value_count')\n\n# Pivot the data so that each symbol_id becomes a column, and date_id is the index\npivot_table = missing_values_by_symbol_date.pivot(index='date_id', columns='symbol_id', values='missing_value_count').fillna(0)\n\n# Set up the plot\nplt.figure(figsize=(18, 10))\n\n# Plot each symbol's trend with distinct styles for readability\nfor symbol in pivot_table.columns:\n    plt.plot(pivot_table.index, pivot_table[symbol], label=symbol, linestyle='-', linewidth=1)\n\n# Configure labels and title\nplt.xlabel(\"Date ID\")\nplt.ylabel(\"Number of Missing Values\")\nplt.title(\"Trend of Missing Values Across Date ID for Each Symbol\")\n\n# Position the legend outside of the plot for readability\nplt.legend(title=\"Symbol ID\", bbox_to_anchor=(1.05, 1), loc='upper left', ncol=2, fontsize='small')\nplt.xticks(rotation=45)\nplt.grid(True)\nplt.tight_layout(rect=[0, 0, 0.85, 1])  # Adjust layout to fit the legend\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:14:38.846232Z","iopub.execute_input":"2024-11-06T21:14:38.846805Z","iopub.status.idle":"2024-11-06T21:14:55.516770Z","shell.execute_reply.started":"2024-11-06T21:14:38.846761Z","shell.execute_reply":"2024-11-06T21:14:55.515448Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's pick symbol_1, and break down by feature","metadata":{}},{"cell_type":"code","source":"# Filter the data for `symbol_id=1`\nfiltered_data = concatenated_df789[concatenated_df789['symbol_id'] == 1]\n\n\n# Calculate the number of missing values for each feature across date_id for symbol_id=1\nmissing_values = filtered_data.groupby('date_id')[selected_features].apply(lambda x: x.isnull().sum())\n\n# Set up the plot\nplt.figure(figsize=(15, 10))\n\n# Plot each feature's missing values over date_id\nfor feature in selected_features:\n    plt.plot(missing_values.index, missing_values[feature], label=feature, linewidth=1)\n\n# Configure the plot for readability\nplt.xlabel(\"Date ID\")\nplt.ylabel(\"Number of Missing Values\")\nplt.title(\"Trend of Missing Values for Each Feature Over Date ID (symbol_id=1)\")\nplt.xticks(rotation=45)\nplt.legend(title=\"Features\", bbox_to_anchor=(1.05, 1), loc='upper left', ncol=2, fontsize='small')\nplt.grid(True)\nplt.tight_layout(rect=[0, 0, 0.85, 1])  # Reserve space for the legend\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-06T21:19:34.161142Z","iopub.execute_input":"2024-11-06T21:19:34.162063Z","iopub.status.idle":"2024-11-06T21:19:35.398333Z","shell.execute_reply.started":"2024-11-06T21:19:34.162010Z","shell.execute_reply":"2024-11-06T21:19:35.397005Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ❇️Thoughts on filling missing values","metadata":{}},{"cell_type":"markdown","source":"- In the final submission: forward filling first + fill with 0 for remaining NaN : https://www.kaggle.com/code/gogo827jz/optimise-speed-of-filling-nan-function\n  - fast Filling-NaN Function: https://www.kaggle.com/code/gogo827jz/optimise-speed-of-filling-nan-function\n\n- Another notebook about different ways of filling NaN values: https://www.kaggle.com/code/iamleonie/utility-function-and-patterns-in-missing-values/notebook#Handling-of-Missing-Values-in-Time-Series\n  - FIll NaN with Outlier (-999): shouldn't be done with linear models, only with tree-based models like XGBoost\n  - Fill NaN with Mean Value\n  - Fill NaN with last value with .ffill()\n  - Fill NaN with Linearly Interpoloated Value with .interpolate() - not feasble\n\n \n- KNN imputation or other imputation methods?","metadata":{}},{"cell_type":"markdown","source":"## Feature mean by date_id","metadata":{}},{"cell_type":"markdown","source":"Some features are cyclic\n\nUseful reference from this discussion:\nhttps://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/542985#3030230\n\nAll the feature means over date_id: https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/542985#3032744","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T20:34:53.422809Z","iopub.execute_input":"2024-11-07T20:34:53.423263Z","iopub.status.idle":"2024-11-07T20:34:55.777408Z","shell.execute_reply.started":"2024-11-07T20:34:53.423218Z","shell.execute_reply":"2024-11-07T20:34:55.776176Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Responders","metadata":{}},{"cell_type":"code","source":"\n# Define the responder columns\nresponder_columns = [f'responder_{i}' for i in range(9)]\n\n# Group by date_id and calculate the mean for each responder column\nmean_responders = final_weight_df.groupby('date_id')[responder_columns].mean()\n\n# Plot the mean values of each responder across date_id\nplt.figure(figsize=(14, 8))\nfor responder in responder_columns:\n    plt.plot(mean_responders.index, mean_responders[responder], label=responder)\n\n# Labels and title\nplt.xlabel('Date ID')\nplt.ylabel('Mean Responder Value')\nplt.title('Mean of Each Responder Across Date ID')\nplt.legend(loc='upper right')\nplt.grid(True)\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:28:05.128845Z","iopub.execute_input":"2024-11-08T00:28:05.130067Z","iopub.status.idle":"2024-11-08T00:28:08.312055Z","shell.execute_reply.started":"2024-11-08T00:28:05.129939Z","shell.execute_reply":"2024-11-08T00:28:08.310324Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Define the responder columns\nresponder_columns = [f'responder_{i}' for i in range(9)]\n\n# Group by date_id and calculate the mean for each responder column\nmean_responders = final_weight_df.groupby('date_id')[responder_columns].mean()\n\n# Calculate the cumulative sum of each responder's mean across date_id\ncumsum_responders = mean_responders.cumsum()\n\n# Plot the cumulative sum of each responder's mean across date_id\nplt.figure(figsize=(14, 8))\nfor responder in responder_columns:\n    plt.plot(cumsum_responders.index, cumsum_responders[responder], label=responder)\n\n# Labels and title\nplt.xlabel('Date ID')\nplt.ylabel('Cumulative Sum of Mean Responder Value')\nplt.title('Cumulative Sum of Mean of Each Responder Across Date ID')\nplt.legend(loc='upper left')\nplt.grid(True)\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:24:55.361511Z","iopub.execute_input":"2024-11-08T00:24:55.362090Z","iopub.status.idle":"2024-11-08T00:24:58.612564Z","shell.execute_reply.started":"2024-11-08T00:24:55.362039Z","shell.execute_reply":"2024-11-08T00:24:58.611231Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Pick 5 days 1530-1534\n\nPlot mean of responders of different symbols","metadata":{}},{"cell_type":"code","source":"# Define the responder columns\nresponder_columns = [f'responder_{i}' for i in range(9)]\n\n# Filter data for date_id from 1530 to 1536\nfiltered_data = final_weight_df[(final_weight_df['date_id'] >= 1530) & (final_weight_df['date_id'] <= 1534)]\n\n# Group by time_id and sum across symbol_id for each responder\nsummed_data = filtered_data.groupby('time_id')[responder_columns].mean()\n\n# Set up the plot grid with 3 rows and 3 columns (one plot for each responder)\nfig, axs = plt.subplots(3, 3, figsize=(18, 12))\naxs = axs.flatten()  # Flatten to iterate over\n\n# Plot each responder in its own subplot\nfor i, responder in enumerate(responder_columns):\n    axs[i].plot(summed_data.index, summed_data[responder], label=responder, alpha=0.7)\n    axs[i].set_title(f'Trend of {responder} Across Time ID')\n    axs[i].set_xlabel('Time ID')\n    axs[i].set_ylabel('Responder Value')\n    axs[i].grid(True)\n    axs[i].legend()\n\n# Adjust layout to prevent overlapping\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:46:52.855074Z","iopub.execute_input":"2024-11-08T00:46:52.856560Z","iopub.status.idle":"2024-11-08T00:46:55.370703Z","shell.execute_reply.started":"2024-11-08T00:46:52.856489Z","shell.execute_reply":"2024-11-08T00:46:55.369255Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Plot std of responders (aggregated by sbymbols)","metadata":{}},{"cell_type":"code","source":"# Define the responder columns\nresponder_columns = [f'responder_{i}' for i in range(9)]\n\n# Filter data for date_id from 1530 to 1536\nfiltered_data = final_weight_df[(final_weight_df['date_id'] >= 1530) & (final_weight_df['date_id'] <= 1534)]\n\n# Group by time_id and sum across symbol_id for each responder\nsummed_data = filtered_data.groupby('time_id')[responder_columns].std()\n\n# Set up the plot grid with 3 rows and 3 columns (one plot for each responder)\nfig, axs = plt.subplots(3, 3, figsize=(18, 12))\naxs = axs.flatten()  # Flatten to iterate over\n\n# Plot each responder in its own subplot\nfor i, responder in enumerate(responder_columns):\n    axs[i].plot(summed_data.index, summed_data[responder], label=responder, alpha=0.7)\n    axs[i].set_title(f'Trend of {responder} Across Time ID')\n    axs[i].set_xlabel('Time ID')\n    axs[i].set_ylabel('Responder Value')\n    axs[i].grid(True)\n    axs[i].legend()\n\n# Adjust layout to prevent overlapping\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-08T00:46:35.883071Z","iopub.execute_input":"2024-11-08T00:46:35.883608Z","iopub.status.idle":"2024-11-08T00:46:38.468779Z","shell.execute_reply.started":"2024-11-08T00:46:35.883558Z","shell.execute_reply":"2024-11-08T00:46:38.467213Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"rolling mean of resp_6?","metadata":{}},{"cell_type":"markdown","source":"## resp breakdown by symbol_id?","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Correlation: resp_6 vs feature","metadata":{}},{"cell_type":"markdown","source":"Weak relationships are observed for all features with respect to resp_6 \n\nThe strongest correlation is feature_6 = -0.09\n\n\n- This notebook dropped all the features that has a absolute correlation < 0.05 with resp_6\nhttps://www.kaggle.com/code/malakafaqahmad/features-engineering-on-janestreet\n\n    - And this notebook used this approach for feature engineering: https://www.kaggle.com/code/dasbro/janestreet-lgbm-dataload-kfold-baseline#Feature-Reduction-and-Acknowledgments\n \n- This notebook breakdown feature vs resp_6 correlation by symbol_id, the correlation is still weak: https://www.kaggle.com/code/ravi20076/janestreet2024-eda-v1?scriptVersionId=203721491&cellId=33\n      ","metadata":{}},{"cell_type":"markdown","source":"## Feature importance","metadata":{}},{"cell_type":"markdown","source":"## Correlation: resp_6 vs other resps","metadata":{}},{"cell_type":"markdown","source":"- Conclusion\n    - resp_8, 7 3 > resp_5, 6\n    - This notebook breakdowns each correlation by symbol: https://www.kaggle.com/code/ravi20076/janestreet2024-eda-v1?scriptVersionId=203721491&cellId=41","metadata":{}}]}