Welcome to competition, hosted by Jane Street, where we'll build a model using real-world data derived from production systems, which offers a glimpse into the daily challenges of successful trading. This challenge highlights the difficulties in modeling financial markets, including fat-tailed distributions, non-stationary time series, and sudden shifts in market behavior
","metadata":{}},{"cell_type":"markdown","source":"# Additional notebook:\n\n[**JS Time series analysis: EDA + Intraday Structures**](https://www.kaggle.com/code/allegich/js-time-series-analysis-eda-intraday-structures)\n\nThis notebook provides a deeper dive into the intraday data structure.\n","metadata":{}},{"cell_type":"markdown","source":"\n
Table of Contents
\n
DATA LOADING AND PREPROCESSING\n
TIME SERIES ANALYSIS AND EDA\n
MODELLING\n
SUB TO SERVER \n","metadata":{}},{"cell_type":"markdown","source":"#
DATA LOADING AND PREPROCESSING
","metadata":{}},{"cell_type":"markdown","source":"
GO BACK","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv, pd.read_parquet )\nimport polars as pl\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom matplotlib.ticker import MaxNLocator, FormatStrFormatter, PercentFormatter\n\nimport os, gc\nfrom tqdm.auto import tqdm\nimport pickle # module to serialize and deserialize objects\nimport re # for Regular expression operations \n\nimport tensorflow as tf\nfrom tensorflow.keras import layers, models\nfrom tensorflow.keras.optimizers import Adam\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom pytorch_lightning import (LightningDataModule, LightningModule, Trainer)\nfrom pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint, Timer\n\nfrom sklearn.metrics import r2_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.ensemble import VotingRegressor\n\nimport lightgbm as lgb\nfrom lightgbm import LGBMRegressor\n\nfrom xgboost import XGBRegressor\nfrom catboost import CatBoostRegressor\n\nimport warnings\nwarnings.filterwarnings('ignore')\npd.options.display.max_columns = None\n\nimport kaggle_evaluation.jane_street_inference_server","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:51:11.309852Z","iopub.execute_input":"2025-03-07T10:51:11.310071Z","iopub.status.idle":"2025-03-07T10:51:38.643644Z","shell.execute_reply.started":"2025-03-07T10:51:11.310037Z","shell.execute_reply":"2025-03-07T10:51:38.642698Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"gridColor = 'lightgrey'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:08.500382Z","iopub.execute_input":"2025-03-07T10:52:08.500703Z","iopub.status.idle":"2025-03-07T10:52:08.504093Z","shell.execute_reply.started":"2025-03-07T10:52:08.500676Z","shell.execute_reply":"2025-03-07T10:52:08.503263Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\npath = \"/kaggle/input/jane-street-real-time-market-data-forecasting\"\nsamples = [] \n\n# Load a data from each file:\nr = range(2)\nfor i in r:\n file_path = f\"{path}/train.parquet/partition_id={i}/part-0.parquet\"\n part = pd.read_parquet(file_path)\n samples.append(part)\n \nsample_df = pd.concat(samples, ignore_index=True) # Concatenate all samples into one DataFrame if needed\n\nsample_df.round(1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:10.042151Z","iopub.execute_input":"2025-03-07T10:52:10.042472Z","iopub.status.idle":"2025-03-07T10:52:20.713589Z","shell.execute_reply.started":"2025-03-07T10:52:10.042448Z","shell.execute_reply":"2025-03-07T10:52:20.712697Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see that two files (0 and 1) has a total of 4 748 457 rows. \nI have used pandas to load it and it took almost 9 sec. Th ere are a total of 339 days (about one years of trading data). ","metadata":{}},{"cell_type":"markdown","source":"#
TIME SERIES ANALYSIS AND EDA
","metadata":{}},{"cell_type":"markdown","source":"
GO BACK","metadata":{}},{"cell_type":"markdown","source":"Let us take a look at the target values over time (for `symbol_id`=1)","metadata":{}},{"cell_type":"code","source":"train =sample_df\ntrain['N']=train.index.values \ntrain['id']=train.index.values \n\nxx= sample_df[(sample_df.symbol_id==1)] ['id']\nyy=sample_df[ (sample_df.symbol_id==1)]['responder_6']\n\nplt.figure(figsize=(16, 5))\nplt.plot(xx,yy, color = 'black', linewidth =0.05)\nplt.suptitle('Returns, responder_6', weight='bold', fontsize=16)\nplt.xlabel(\"Time\", fontsize=12)\nplt.ylabel(\"Returns\", fontsize=12)\nplt.grid(color = gridColor , linewidth=0.8)\nplt.axhline(0, color='red', linestyle='-', linewidth=1.2)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:40:34.209452Z","iopub.execute_input":"2025-03-07T10:40:34.209748Z","iopub.status.idle":"2025-03-07T10:40:35.256912Z","shell.execute_reply.started":"2025-03-07T10:40:34.209722Z","shell.execute_reply":"2025-03-07T10:40:35.255747Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let us take a look at the cumulative values of response over time","metadata":{}},{"cell_type":"code","source":"#for symbol_id=1\nplt.figure(figsize=(14, 4))\nplt.plot(xx,yy.cumsum(), color = 'black', linewidth =0.6)\nplt.suptitle('Cumulative responder_6', weight='bold', fontsize=16)\nplt.xlabel(\"Time\", fontsize=12)\nplt.ylabel(\"Cumulative res\", fontsize=12)\nplt.yticks(np.arange(-500,1000,250))\n#plt.xticks(np.arange(0,170,10))\nplt.grid(color = gridColor)\n#plt.grid(color = 'lightblue')\nplt.axhline(0, color='red', linestyle='-', linewidth=0.7)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:40:35.258518Z","iopub.execute_input":"2025-03-07T10:40:35.258826Z","iopub.status.idle":"2025-03-07T10:40:35.491439Z","shell.execute_reply.started":"2025-03-07T10:40:35.258800Z","shell.execute_reply":"2025-03-07T10:40:35.490316Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now let's compare this responder (6) with other responders","metadata":{}},{"cell_type":"code","source":"# for symbol_id == 0\nplt.figure(figsize=(18, 7))\npredictor_cols = [col for col in sample_df.columns if 'responder' in col]\nfor i in predictor_cols: \n if i == 'responder_6': \n c='red'\n lw=2.5\n plt.plot((sample_df[sample_df.symbol_id == 0].groupby(['date_id'])[i].mean()).cumsum(), linewidth = lw, color = c)\n else: \n lw=1\n plt.plot((sample_df[sample_df.symbol_id == 0].groupby(['date_id'])[i].mean()).cumsum(), linewidth = lw)\n\nplt.xlabel('Trade days')\nplt.ylabel('Cumulative response')\nplt.title('Response time series over trade days \\n Responder 6 (red) and other responders', weight='bold')\nplt.grid(visible=True, color = gridColor, linewidth = 0.7)\nplt.axhline(0, color='blue', linestyle='-', linewidth=1)\nplt.legend(predictor_cols)\nsns.despine()\n#plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:40:35.492715Z","iopub.execute_input":"2025-03-07T10:40:35.493130Z","iopub.status.idle":"2025-03-07T10:40:37.779806Z","shell.execute_reply.started":"2025-03-07T10:40:35.493088Z","shell.execute_reply":"2025-03-07T10:40:37.778698Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- We can see that `resp6` (red) most closely follows `resp0` and `resp3`\n\nLet's build a correlation matrix and see it numerically.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(6, 6))\nresponders = pd.read_csv(f\"{path}/responders.csv\")\nmatrix = responders[[ f\"tag_{no}\" for no in range(0,5,1) ] ].T.corr()\nsns.heatmap(matrix, square=True, cmap=\"coolwarm\", alpha =0.9, vmin=-1, vmax=1, center= 0, linewidths=0.5, \n linecolor='white', annot=True, fmt='.2f')\nplt.xlabel(\"Responder_0 - Responder_8\")\nplt.ylabel(\"Responder_0 - Responder_8\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:40:37.780767Z","iopub.execute_input":"2025-03-07T10:40:37.781042Z","iopub.status.idle":"2025-03-07T10:40:38.213052Z","shell.execute_reply.started":"2025-03-07T10:40:37.781019Z","shell.execute_reply":"2025-03-07T10:40:38.211794Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let us take a look at the returns and cumulative daily returns, and disribution of returns for all responders","metadata":{}},{"cell_type":"code","source":"df_train=sample_df\ns_id = 0 # Change params to take a look at other symbols\nres_columns = [col for col in df_train.columns if re.match(\"responder_\", col)]\nrow = 9\nj = 0\n\nfig, axs = plt.subplots(figsize=(18, 4*row))\nfor i in range(1, 3 * len(res_columns) + 1, 3):\n xx= sample_df[(sample_df.symbol_id==s_id)] ['N']\n yy=sample_df[ (sample_df.symbol_id==s_id)][f'responder_{j}']\n c='black'\n if j == 6: c='red'\n \n ax1 = plt.subplot(9, 3, i)\n ax1.plot( xx,yy.cumsum() , color = c, linewidth =0.8 )\n plt.axhline(0, color='blue', linestyle='-', linewidth=0.9)\n plt.grid(color =gridColor )\n \n ax2 = plt.subplot(9, 3, i+1)\n #by_date = df_symbolX.groupby([\"date_id\"])\n ax2.plot(xx,yy , color = c, linewidth =0.05)\n plt.axhline(0, color='blue', linestyle='-', linewidth=1.2)\n ax2.set_title(f\"responder_{j}\", fontsize = 14)\n plt.grid(color = gridColor)\n \n ax3 = plt.subplot(9, 3, i+2)\n b=1000\n ax3.hist(yy, bins=b, color = c,density=True, histtype=\"step\" )\n ax3.hist(yy, bins=b, color = 'lightgrey',density=True)\n plt.grid(color = gridColor)\n ax3.set_ylim([0, 3.5])\n ax3.set_xlim([-2.5, 2.5])\n \n j = j + 1\n \nfig.patch.set_linewidth(3)\nfig.patch.set_edgecolor('#000000')\nfig.patch.set_facecolor('#eeeeee') \nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:40:38.214331Z","iopub.execute_input":"2025-03-07T10:40:38.214623Z","iopub.status.idle":"2025-03-07T10:40:59.866434Z","shell.execute_reply.started":"2025-03-07T10:40:38.214596Z","shell.execute_reply":"2025-03-07T10:40:59.865281Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see that responders have different behavior and distributions.\n\nLet us now study the behavior of `responder 6` for different `symbol_id`","metadata":{}},{"cell_type":"code","source":"res_columns = [col for col in df_train.columns if re.match(\"responder_\", col)]\nrow=10\nfig, axs = plt.subplots(figsize=(18, 5*row))\nb=300\nj = 0\nfor i in range(1, 3 * row + 1, 3):\n xx= sample_df[(sample_df.symbol_id==j)] ['N']\n yy= sample_df[(sample_df.symbol_id==j)]['responder_6']\n c='black'\n \n ax1 = plt.subplot(row, 3, i)\n ax1.plot( xx,yy.cumsum() , color = c, linewidth =0.8 )\n plt.axhline(0, color='red', linestyle='-', linewidth=0.7)\n plt.grid(color = gridColor)\n plt.xlabel('Time')\n \n ax2 = plt.subplot(row, 3, i+1)\n ax2.plot(xx,yy , color = c, linewidth =0.05)\n plt.axhline(0, color='red', linestyle='-', linewidth=0.7)\n ax2.set_title(f\"symbol_id={j}\", fontsize = '14')\n plt.grid(color = gridColor)\n plt.xlabel('Time')\n \n ax3 = plt.subplot(row, 3, i+2)\n ax3.hist(yy, bins=b, color = c, density=True, histtype=\"step\" )\n ax3.hist(yy, bins=b, color = 'lightgrey',density=True)\n plt.grid(color = gridColor)\n ax3.set_xlim([-2.5, 2.5])\n ax3.set_ylim([0, 1.5])\n plt.xlabel('Time')\n \n j = j + 1\n \nfig.patch.set_linewidth(3)\nfig.patch.set_edgecolor('#000000')\nfig.patch.set_facecolor('#eeeeee') \nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:40:59.867613Z","iopub.execute_input":"2025-03-07T10:40:59.867950Z","iopub.status.idle":"2025-03-07T10:41:11.610688Z","shell.execute_reply.started":"2025-03-07T10:40:59.867922Z","shell.execute_reply":"2025-03-07T10:41:11.609434Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- We see that the behavior and distribution of one `responder 6` is different for different `symbol_id`\n\nNow let's study the data in more detail and then continue diving into time series analysis\n","metadata":{}},{"cell_type":"markdown","source":"## Files and variables overview","metadata":{}},{"cell_type":"markdown","source":"### Features.csv\nfeatures.csv - metadata pertaining to the anonymized features\n\n#### Features have many missing values.","metadata":{}},{"cell_type":"code","source":"df_train = sample_df\nplt.figure(figsize=(20, 3)) # Plot missing values\nplt.bar(x=df_train.isna().sum().index, height=df_train.isna().sum().values, color=\"red\", label='missing') # analog: using missingno\nplt.xticks(rotation=90)\nplt.title(f'Missing values over the {len(df_train)} samples which have a target')\nplt.grid()\nplt.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:11.612024Z","iopub.execute_input":"2025-03-07T10:41:11.612397Z","iopub.status.idle":"2025-03-07T10:41:14.532194Z","shell.execute_reply.started":"2025-03-07T10:41:11.612364Z","shell.execute_reply":"2025-03-07T10:41:14.531054Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Some columns are not very useful in our sample (either Null or show the partition number).","metadata":{}},{"cell_type":"markdown","source":"#### Structure of features:","metadata":{}},{"cell_type":"code","source":"features = pd.read_csv(f\"{path}/features.csv\")\nfeatures","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:14.533186Z","iopub.execute_input":"2025-03-07T10:41:14.533475Z","iopub.status.idle":"2025-03-07T10:41:14.565368Z","shell.execute_reply.started":"2025-03-07T10:41:14.533450Z","shell.execute_reply":"2025-03-07T10:41:14.564422Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Tags visualizing:","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(18, 6))\nplt.imshow(features.iloc[:, 1:].T.values, cmap=\"gray_r\")\nplt.xlabel(\"feature_00 - feature_78\")\nplt.ylabel(\"tag_0 - tag_16\")\nplt.yticks(np.arange(17))\nplt.xticks(np.arange(79))\nplt.grid(color = 'lightgrey')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:14.568701Z","iopub.execute_input":"2025-03-07T10:41:14.569025Z","iopub.status.idle":"2025-03-07T10:41:15.175771Z","shell.execute_reply.started":"2025-03-07T10:41:14.568999Z","shell.execute_reply":"2025-03-07T10:41:15.174625Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Correlation matrix between feature_XX and feature_YY","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(11, 11))\nmatrix = features[[ f\"tag_{no}\" for no in range(0,17,1) ] ].T.corr()\nsns.heatmap(matrix, square=True, cmap=\"coolwarm\", alpha =0.9, vmin=-1, vmax=1, center= 0, linewidths=0.5, linecolor='white')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:15.177594Z","iopub.execute_input":"2025-03-07T10:41:15.177937Z","iopub.status.idle":"2025-03-07T10:41:15.881180Z","shell.execute_reply.started":"2025-03-07T10:41:15.177907Z","shell.execute_reply":"2025-03-07T10:41:15.879965Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Responders.csv\nresponders.csv - metadata pertaining to the anonymized responders\n#### Structure of responders:","metadata":{}},{"cell_type":"code","source":"responders = pd.read_csv(f\"{path}/responders.csv\")\nresponders","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:15.882221Z","iopub.execute_input":"2025-03-07T10:41:15.882509Z","iopub.status.idle":"2025-03-07T10:41:15.896918Z","shell.execute_reply.started":"2025-03-07T10:41:15.882484Z","shell.execute_reply":"2025-03-07T10:41:15.895788Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Weights\n#### Basic stats:","metadata":{}},{"cell_type":"code","source":"sample_df['weight'].describe().round(1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:15.898478Z","iopub.execute_input":"2025-03-07T10:41:15.898783Z","iopub.status.idle":"2025-03-07T10:41:16.036534Z","shell.execute_reply.started":"2025-03-07T10:41:15.898759Z","shell.execute_reply":"2025-03-07T10:41:16.035429Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,3))\nplt.hist(sample_df['weight'], bins=30, color='grey', edgecolor = 'white',density=True )\nplt.title('Distribution of weights')\nplt.grid(color = 'lightgrey', linewidth=0.5)\nplt.axvline(1.7, color='red', linestyle='-', linewidth=0.7)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:16.037693Z","iopub.execute_input":"2025-03-07T10:41:16.038106Z","iopub.status.idle":"2025-03-07T10:41:17.314391Z","shell.execute_reply.started":"2025-03-07T10:41:16.038064Z","shell.execute_reply":"2025-03-07T10:41:17.313452Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Sample submission.csv\nsample_submission.csv - This file illustrates the format of the predictions your model should make.","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv(f\"{path}/sample_submission.csv\")\nprint( f\"shape = {sub.shape}\" )\nsub.head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:17.315434Z","iopub.execute_input":"2025-03-07T10:41:17.315818Z","iopub.status.idle":"2025-03-07T10:41:17.333201Z","shell.execute_reply.started":"2025-03-07T10:41:17.315790Z","shell.execute_reply":"2025-03-07T10:41:17.332248Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Train.parquet\n\n- **train.parquet** - The training set, contains historical data and returns. For convenience, the training set has been partitioned into ten parts.\n - `date_id` and `time_id` - Integer values that are ordinally sorted, providing a chronological structure to the data, although the actual time intervals between `time_id` values may vary.\n - `symbol_id` - Identifies a unique financial instrument.\n - `weight` - The weighting used for calculating the scoring function.\n - `feature_{00...78}` - Anonymized market data.\n - `responder_{0...8}` - Anonymized responders clipped between -5 and 5. The `responder_6` field is what you are trying to predict.\n \n \nEach row in the `{train/test}.parquet` dataset corresponds to a unique combination of a symbol (identified by `symbol_id`) and a timestamp (represented by `date_id` and `time_id`). You will be provided with multiple responders, with `responder_6` being the only responder used for scoring. The date_id column is an integer which represents the day of the event, while `time_id` represents a time ordering. It's important to note that the real time differences between each time_id are not guaranteed to be consistent.\n\n- The `symbol_id` column contains encrypted identifiers. Each `symbol_id` is not guaranteed to appear in all `time_id` and `date_id` combinations.\n- Additionally, new `symbol_id` values **may appear in future** test sets.est sets.","metadata":{}},{"cell_type":"markdown","source":"## Responders: analysis, statistics and distributions","metadata":{}},{"cell_type":"code","source":"col =[]\nfor i in range(9):\n col.append(f\"responder_{i}\") \n\nsample_df[col].describe().round(1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:17.334207Z","iopub.execute_input":"2025-03-07T10:41:17.334471Z","iopub.status.idle":"2025-03-07T10:41:19.136073Z","shell.execute_reply.started":"2025-03-07T10:41:17.334448Z","shell.execute_reply":"2025-03-07T10:41:19.135067Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Interesting fact:\n- The values of all variables are strictly within the range of `[-5, 5]`","metadata":{}},{"cell_type":"markdown","source":"#### Responders-Responder distributions\nLet's dive deeper into the relationships between respondents and plot mutual distributions between 'reps 6' and other responders","metadata":{}},{"cell_type":"code","source":"numerical_features=[]\nnumerical_features=sample_df.filter(regex='^responder_').columns.tolist() # Separate responders\nnumerical_features.remove('responder_6')\n\ngs=600\nk=1;\ncol = 3\nrow = 3\nfig, axs = plt.subplots(row, col, figsize=(5*col, 5*row))\n\nfor i in numerical_features:\n \n plt.subplot(col,row, k)\n plt.hexbin(sample_df[i], sample_df['responder_6'], gridsize=gs, cmap='CMRmap', bins='log', alpha = 0.2)\n plt.xlabel(f'{i}', fontsize = 12)\n plt.ylabel('responder_6', fontsize = 12)\n plt.tick_params(axis='x', labelsize=6)\n plt.tick_params(axis='y', labelsize=6)\n k=k+1\nfig.patch.set_linewidth(3)\nfig.patch.set_edgecolor('#000000')\nfig.patch.set_facecolor('#eeeeee') \n\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:19.137155Z","iopub.execute_input":"2025-03-07T10:41:19.137405Z","iopub.status.idle":"2025-03-07T10:41:42.093646Z","shell.execute_reply.started":"2025-03-07T10:41:19.137383Z","shell.execute_reply":"2025-03-07T10:41:42.092306Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Responder6-Features distributions\nNow let's plot mutual distributions between 'reps 6' and some features","metadata":{}},{"cell_type":"code","source":"numerical_features=[]\nfor i in ['05', '06', '07', '08', '12', '15', '19', '32', '38', '39', '50', '51', '65', '66', '67']:\n numerical_features.append(f'feature_{i}') \n\ngs=600\nk=1;\ncol = 3\nrow = int(np.ceil(len(numerical_features) /3 ))\nsz=5\nw=sz*col\nh = w/col *row\nplt.figure(figsize=(w, h))\n\nfig, axs = plt.subplots(figsize=(w, h))\n\nfor i in numerical_features:\n \n plt.subplot(row, col, k)\n plt.hexbin(sample_df['responder_6'], sample_df[i], gridsize=gs, cmap='CMRmap', bins='log', alpha = 0.3)\n \n plt.xlabel(f'{i}')\n plt.ylabel('responder_6')\n plt.tick_params(axis='x', labelsize=6)\n plt.tick_params(axis='y', labelsize=6)\n k=k+1\n\nfig.patch.set_linewidth(3)\nfig.patch.set_edgecolor('#000000')\nfig.patch.set_facecolor('#eeeeee') \nplt.show() ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:41:42.095126Z","iopub.execute_input":"2025-03-07T10:41:42.095660Z","iopub.status.idle":"2025-03-07T10:42:24.538768Z","shell.execute_reply.started":"2025-03-07T10:41:42.095607Z","shell.execute_reply":"2025-03-07T10:42:24.537533Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"numerical_features=[]\n\nfor i in range(5,9):\n numerical_features.append(f'feature_0{i}') \nfor i in range(15,20):\n numerical_features.append(f'feature_{i}') \n \na=0; k=1;\nn=3; \n\nfig, axs = plt.subplots(figsize=(15, 4))\nfor i in numerical_features[:-1]:\n a=a+1\n for j in numerical_features[a:]:\n plt.subplot(1,n, k)\n plt.hexbin(sample_df[i], sample_df[j], gridsize=200, cmap='CMRmap', bins='log', alpha = 1)\n plt.grid()\n plt.xlabel(f'{i}', fontsize = 14)\n plt.ylabel(f'{j}', fontsize = 14)\n plt.tick_params(axis='x', labelsize=6)\n plt.tick_params(axis='y', labelsize=6)\n \n k=k+1\n if k == (n+1): \n k=1\n plt.show()\n plt.figure(figsize=(15, 4)) ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:42:24.539918Z","iopub.execute_input":"2025-03-07T10:42:24.540290Z","iopub.status.idle":"2025-03-07T10:42:57.444037Z","shell.execute_reply.started":"2025-03-07T10:42:24.540258Z","shell.execute_reply":"2025-03-07T10:42:57.442888Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- There are many nonlinear and non-trivial distributions","metadata":{}},{"cell_type":"markdown","source":"#
MODELLING
","metadata":{}},{"cell_type":"markdown","source":"
GO BACK","metadata":{}},{"cell_type":"markdown","source":"This is jsut a first approach to modelling and experiment\nSetting of ensemble:","metadata":{}},{"cell_type":"code","source":"ENSEMBLE_SOLUTIONS = ['SOLUTION_14','SOLUTION_5']\nOPTION,__WTS = 'option 91',[0.899, 0.28]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:45.325280Z","iopub.execute_input":"2025-03-07T10:52:45.325606Z","iopub.status.idle":"2025-03-07T10:52:45.329141Z","shell.execute_reply.started":"2025-03-07T10:52:45.325583Z","shell.execute_reply":"2025-03-07T10:52:45.328341Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def predict(test:pl.DataFrame, lags:pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame: \n pdB = predict_14(test,lags).to_pandas()\n pdC = predict_5 (test,lags).to_pandas()\n\n pdB = pdB.rename(columns={'responder_6':'responder_B'})\n pdC = pdC.rename(columns={'responder_6':'responder_C'})\n pds = pd.merge(pdB,pdC, on=['row_id'])\n pds['responder_6'] =\\\n pds['responder_B'] *__WTS[0] +\\\n pds['responder_C'] *__WTS[1] \n\n display(pds)\n predictions = test.select('row_id', pl.lit(0.0).alias('responder_6'))\n pred = pds['responder_6'].to_numpy()\n predictions = predictions.with_columns(pl.Series('responder_6', pred.ravel()))\n return predictions","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:45.355459Z","iopub.execute_input":"2025-03-07T10:52:45.355753Z","iopub.status.idle":"2025-03-07T10:52:45.360955Z","shell.execute_reply.started":"2025-03-07T10:52:45.355728Z","shell.execute_reply":"2025-03-07T10:52:45.360063Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":" 5. [JS Ridge baseline](https://www.kaggle.com/code/yunsuxiaozi/js-ridge-baseline) Lb=0.0026\n [yunsuxiaozi](https://www.kaggle.com/yunsuxiaozi)","metadata":{"papermill":{"duration":0.022173,"end_time":"2024-10-26T03:27:39.285949","exception":false,"start_time":"2024-10-26T03:27:39.263776","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if 'SOLUTION_5' in ENSEMBLE_SOLUTIONS:\n \n def predict_5(test,lags):\n cols=[f'feature_0{i}' if i<10 else f'feature_{i}' for i in range(79)]\n predictions = test.select(\n 'row_id',\n pl.lit(0.0).alias('responder_6'),\n )\n test_preds=model_5.predict(test[cols].to_pandas().fillna(3).values)\n predictions = predictions.with_columns(pl.Series('responder_6', test_preds.ravel()))\n return predictions\n\nif 'SOLUTION_5' in ENSEMBLE_SOLUTIONS:\n from sklearn.linear_model import BayesianRidge\n import joblib\n model_5 = joblib.load('/kaggle/input/jane-street-5-and-7_/other/default/1/ridge_model_5(1).pkl')","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.031534,"end_time":"2024-10-26T03:27:39.453592","exception":false,"start_time":"2024-10-26T03:27:39.422058","status":"completed"},"tags":[],"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:45.362129Z","iopub.execute_input":"2025-03-07T10:52:45.362455Z","iopub.status.idle":"2025-03-07T10:52:45.388872Z","shell.execute_reply.started":"2025-03-07T10:52:45.362424Z","shell.execute_reply":"2025-03-07T10:52:45.388173Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"14. [Jane Street RMF NN + XGB](https://www.kaggle.com/code/voix97/jane-street-rmf-nn-xgb), Lb=0.0056\n [Xiang Sheng](https://www.kaggle.com/voix97)","metadata":{}},{"cell_type":"markdown","source":"### NN + XGB inference\n\n### Configurations","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS: \n \n class CONFIG:\n seed = 42\n target_col = \"responder_6\"\n # feature_cols = [\"symbol_id\", \"time_id\"] + [f\"feature_{idx:02d}\" for idx in range(79)]+ [f\"responder_{idx}_lag_1\" for idx in range(9)]\n feature_cols = [f\"feature_{idx:02d}\" for idx in range(79)]+ [f\"responder_{idx}_lag_1\" for idx in range(9)]\n\n model_paths = [\n #\"/kaggle/input/js24-train-gbdt-model-with-lags-singlemodel/result.pkl\",\n #\"/kaggle/input/js24-trained-gbdt-model/result.pkl\",\n \"/kaggle/input/js-xs-nn-trained-model\",\n \"/kaggle/input/js-with-lags-trained-xgb/result.pkl\",\n ]\n\nif 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n \n valid = pl.scan_parquet(\n f\"/kaggle/input/js24-preprocessing-create-lags/validation.parquet/\"\n ).collect().to_pandas()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:45.390139Z","iopub.execute_input":"2025-03-07T10:52:45.390456Z","iopub.status.idle":"2025-03-07T10:52:48.456780Z","shell.execute_reply.started":"2025-03-07T10:52:45.390425Z","shell.execute_reply":"2025-03-07T10:52:48.455809Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Load model","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS: \n \n xgb_model = None\n model_path = CONFIG.model_paths[1]\n with open( model_path, \"rb\") as fp:\n result = pickle.load(fp)\n xgb_model = result[\"model\"]\n\n xgb_feature_cols = [\"symbol_id\", \"time_id\"] + CONFIG.feature_cols\n\n # Show model\n #display(xgb_model)\n\nif 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n \n # Custom R2 metric for validation\n def r2_val(y_true, y_pred, sample_weight):\n r2 = 1 - np.average((y_pred - y_true) ** 2, weights=sample_weight) / (np.average((y_true) ** 2, weights=sample_weight) + 1e-38)\n return r2\n\n\n class NN(LightningModule):\n def __init__(self, input_dim, hidden_dims, dropouts, lr, weight_decay):\n super().__init__()\n self.save_hyperparameters()\n layers = []\n in_dim = input_dim\n for i, hidden_dim in enumerate(hidden_dims):\n layers.append(nn.BatchNorm1d(in_dim))\n if i > 0:\n layers.append(nn.SiLU())\n if i < len(dropouts):\n layers.append(nn.Dropout(dropouts[i]))\n layers.append(nn.Linear(in_dim, hidden_dim))\n # layers.append(nn.ReLU())\n in_dim = hidden_dim\n layers.append(nn.Linear(in_dim, 1)) \n layers.append(nn.Tanh())\n self.model = nn.Sequential(*layers)\n self.lr = lr\n self.weight_decay = weight_decay\n self.validation_step_outputs = []\n\n def forward(self, x):\n return 5 * self.model(x).squeeze(-1) \n\n def training_step(self, batch):\n x, y, w = batch\n y_hat = self(x)\n loss = F.mse_loss(y_hat, y, reduction='none') * w \n loss = loss.mean()\n self.log('train_loss', loss, on_step=False, on_epoch=True, batch_size=x.size(0))\n return loss\n\n def validation_step(self, batch):\n x, y, w = batch\n y_hat = self(x)\n loss = F.mse_loss(y_hat, y, reduction='none') * w\n loss = loss.mean()\n self.log('val_loss', loss, on_step=False, on_epoch=True, batch_size=x.size(0))\n self.validation_step_outputs.append((y_hat, y, w))\n return loss\n\n def on_validation_epoch_end(self):\n \"\"\"Calculate validation WRMSE at the end of the epoch.\"\"\"\n y = torch.cat([x[1] for x in self.validation_step_outputs]).cpu().numpy()\n if self.trainer.sanity_checking:\n prob = torch.cat([x[0] for x in self.validation_step_outputs]).cpu().numpy()\n else:\n prob = torch.cat([x[0] for x in self.validation_step_outputs]).cpu().numpy()\n weights = torch.cat([x[2] for x in self.validation_step_outputs]).cpu().numpy()\n # r2_val\n val_r_square = r2_val(y, prob, weights)\n self.log(\"val_r_square\", val_r_square, prog_bar=True, on_step=False, on_epoch=True)\n self.validation_step_outputs.clear()\n\n def configure_optimizers(self):\n optimizer = torch.optim.Adam(self.parameters(), lr=self.lr, weight_decay=self.weight_decay)\n scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=5,\n verbose=True)\n return {\n 'optimizer': optimizer,\n 'lr_scheduler': {\n 'scheduler': scheduler,\n 'monitor': 'val_loss',\n }\n }\n\n def on_train_epoch_end(self):\n if self.trainer.sanity_checking:\n return\n epoch = self.trainer.current_epoch\n metrics = {k: v.item() if isinstance(v, torch.Tensor) else v for k, v in self.trainer.logged_metrics.items()}\n formatted_metrics = {k: f\"{v:.5f}\" for k, v in metrics.items()}\n print(f\"Epoch {epoch}: {formatted_metrics}\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:48.457936Z","iopub.execute_input":"2025-03-07T10:52:48.458174Z","iopub.status.idle":"2025-03-07T10:52:48.565475Z","shell.execute_reply.started":"2025-03-07T10:52:48.458155Z","shell.execute_reply":"2025-03-07T10:52:48.564739Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n \n N_folds = 5\n models = []\n for fold in range(N_folds):\n checkpoint_path = f\"{CONFIG.model_paths[0]}/nn_{fold}.model\"\n model = NN.load_from_checkpoint(checkpoint_path)\n models.append(model.to(\"cuda:0\"))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:48.566646Z","iopub.execute_input":"2025-03-07T10:52:48.566894Z","iopub.status.idle":"2025-03-07T10:52:49.423059Z","shell.execute_reply.started":"2025-03-07T10:52:48.566873Z","shell.execute_reply":"2025-03-07T10:52:49.422250Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### CV Score","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS: \n \n X_valid = valid[ xgb_feature_cols ]\n y_valid = valid[ CONFIG.target_col ]\n w_valid = valid[ \"weight\" ]\n y_pred_valid_xgb = xgb_model.predict(X_valid)\n valid_score = r2_score( y_valid, y_pred_valid_xgb, sample_weight=w_valid )\n valid_score","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:49.424396Z","iopub.execute_input":"2025-03-07T10:52:49.424696Z","iopub.status.idle":"2025-03-07T10:52:52.135509Z","shell.execute_reply.started":"2025-03-07T10:52:49.424668Z","shell.execute_reply":"2025-03-07T10:52:52.134750Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n X_valid = valid[ CONFIG.feature_cols ]\n y_valid = valid[ CONFIG.target_col ]\n w_valid = valid[ \"weight\" ]\n X_valid = X_valid.fillna(method = 'ffill').fillna(0)\n X_valid.shape, y_valid.shape, w_valid.shape\n\nif 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n y_pred_valid_nn = np.zeros(y_valid.shape)\n with torch.no_grad():\n for model in models:\n model.eval()\n y_pred_valid_nn += model(torch.FloatTensor(X_valid.values).to(\"cuda:0\")).cpu().numpy() / len(models)\n valid_score = r2_score( y_valid, y_pred_valid_nn, sample_weight=w_valid )\n valid_score\n\nif 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n y_pred_valid_ensemble = 0.5 * (y_pred_valid_xgb + y_pred_valid_nn)\n valid_score = r2_score( y_valid, y_pred_valid_ensemble, sample_weight=w_valid )\n valid_score\n\nif 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n del valid, X_valid, y_valid, w_valid\n gc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-07T10:52:52.136171Z","iopub.execute_input":"2025-03-07T10:52:52.136412Z","iopub.status.idle":"2025-03-07T10:53:00.812280Z","shell.execute_reply.started":"2025-03-07T10:52:52.136391Z","shell.execute_reply":"2025-03-07T10:53:00.811542Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS: \n \n lags_ : pl.DataFrame | None = None\n\n def predict_14(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:\n global lags_\n if lags is not None:\n lags_ = lags\n\n predictions_14 = test.select(\n 'row_id',\n pl.lit(0.0).alias('responder_6'),\n )\n symbol_ids = test.select('symbol_id').to_numpy()[:, 0]\n\n if not lags is None:\n lags = lags.group_by([\"date_id\", \"symbol_id\"], maintain_order=True).last() # pick up last record of previous date\n test = test.join(lags, on=[\"date_id\", \"symbol_id\"], how=\"left\")\n else:\n test = test.with_columns(\n ( pl.lit(0.0).alias(f'responder_{idx}_lag_1') for idx in range(9) )\n )\n\n preds = np.zeros((test.shape[0],))\n preds += xgb_model.predict(test[xgb_feature_cols].to_pandas()) / 2\n test_input = test[CONFIG.feature_cols].to_pandas()\n test_input = test_input.fillna(method = 'ffill').fillna(0)\n test_input = torch.FloatTensor(test_input.values).to(\"cuda:0\")\n with torch.no_grad():\n for i, nn_model in enumerate(tqdm(models)):\n nn_model.eval()\n preds += nn_model(test_input).cpu().numpy() / 10\n print(f\"predict> preds.shape =\", preds.shape)\n\n predictions_14 = \\\n test.select('row_id').\\\n with_columns(\n pl.Series(\n name = 'responder_6', \n values = np.clip(preds, a_min = -5, a_max = 5),\n dtype = pl.Float64,\n )\n )\n\n # The predict function must return a DataFrame\n #assert isinstance(predictions, pl.DataFrame | pd.DataFrame)\n # with columns 'row_id', 'responer_6'\n #assert list(predictions.columns) == ['row_id', 'responder_6']\n # and as many rows as the test data.\n #assert len(predictions) == len(test)\n\n return predictions_14","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-07T10:53:00.813131Z","iopub.execute_input":"2025-03-07T10:53:00.813439Z","iopub.status.idle":"2025-03-07T10:53:00.821205Z","shell.execute_reply.started":"2025-03-07T10:53:00.813410Z","shell.execute_reply":"2025-03-07T10:53:00.820335Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
SUB TO SERVER
","metadata":{}},{"cell_type":"markdown","source":"
GO BACK","metadata":{}},{"cell_type":"code","source":"#sdf\ninference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)\n\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n inference_server.serve()\nelse:\n inference_server.run_local_gateway(\n (\n '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n )\n )","metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"papermill":{"duration":0.351292,"end_time":"2024-10-26T03:27:42.101707","exception":false,"start_time":"2024-10-26T03:27:41.750415","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null}]}
![](\"https://i.postimg.cc/J0S0d25V/1234-Untitled.png\")
\n