{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":84493,"databundleVersionId":9871156,"sourceType":"competition"},{"sourceId":9625192,"sourceType":"datasetVersion","datasetId":5875295},{"sourceId":9631435,"sourceType":"datasetVersion","datasetId":5879957},{"sourceId":9640394,"sourceType":"datasetVersion","datasetId":5882430},{"sourceId":9671528,"sourceType":"datasetVersion","datasetId":5887570},{"sourceId":9756372,"sourceType":"datasetVersion","datasetId":5973876},{"sourceId":9761005,"sourceType":"datasetVersion","datasetId":5931288},{"sourceId":9801075,"sourceType":"datasetVersion","datasetId":6006872},{"sourceId":9806342,"sourceType":"datasetVersion","datasetId":6010899},{"sourceId":9838282,"sourceType":"datasetVersion","datasetId":6035149},{"sourceId":201255000,"sourceType":"kernelVersion"},{"sourceId":201579302,"sourceType":"kernelVersion"},{"sourceId":203781885,"sourceType":"kernelVersion"},{"sourceId":203900450,"sourceType":"kernelVersion"},{"sourceId":145414,"sourceType":"modelInstanceVersion","modelInstanceId":123281,"modelId":146350},{"sourceId":171905,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":146319,"modelId":168862}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":80.344101,"end_time":"2024-10-26T03:27:42.952247","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-10-26T03:26:22.608146","version":"2.6.0"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"252dd2de87de42f4becd877fbbafc26b":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_ff55584ae0ab4f39ae471f314eaad988","placeholder":"​","style":"IPY_MODEL_b8d338c473aa4dc3ba25f37a997a9037","value":" 1/1 [00:00&lt;00:00, 33.79it/s]"}},"48a2731fb59b4ce8ace5b53d6f0e3337":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_a8dbc79c7c5a48318466a102c55801bb","placeholder":"​","style":"IPY_MODEL_ebd06aaaa7024a1684ba1b9fe89358cf","value":"100%"}},"56da652f3eeb42aca986e6fd815629dc":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"5bb4e0df01c44716afebab25aafe9f5f":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_f4e41234b0cc4f3e9313d971f5aadc9f","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_6408698650f74dd699bcc914b850e396","value":1}},"6408698650f74dd699bcc914b850e396":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"9cb493ec04fc4ed391f5ac28cc84500e":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_48a2731fb59b4ce8ace5b53d6f0e3337","IPY_MODEL_5bb4e0df01c44716afebab25aafe9f5f","IPY_MODEL_252dd2de87de42f4becd877fbbafc26b"],"layout":"IPY_MODEL_56da652f3eeb42aca986e6fd815629dc"}},"a8dbc79c7c5a48318466a102c55801bb":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"b8d338c473aa4dc3ba25f37a997a9037":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"ebd06aaaa7024a1684ba1b9fe89358cf":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"f4e41234b0cc4f3e9313d971f5aadc9f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"ff55584ae0ab4f39ae471f314eaad988":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}}},"version_major":2,"version_minor":0}}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"ENSEMBLE_SOLUTIONS = ['SOLUTION_7','SOLUTION_14','SOLUTION_5']\nOPTION,__WTS = 'option 91',[0.0017, 0.7183, 0.2800]\n\n# ENSEMBLE_SOLUTIONS = ['SOLUTION_7','SOLUTION_12','SOLUTION_5']\n# OPTION,__WTS = 'option 90',[0.0017, 0.7183, 0.2800]\n\n\n#ENSEMBLE_SOLUTIONS = ['SOLUTION_7','SOLUTION_10','SOLUTION_5']\n#OPTION,__WTS = 'option 80',[0.0500, 0.7193, 0.2307] #  option.80 < option.79\n#OPTION,__WTS = 'option 81',[0.0500, 0.7195, 0.2305] #  stop\n#OPTION,__WTS = 'option 81',[0.0111, 0.7189, 0.2700]\n#OPTION,__WTS = 'option 82',[0.0113, 0.7187, 0.2700]\n#OPTION,__WTS = 'option 83',[0.0115, 0.7185, 0.2700]\n#OPTION,__WTS = 'option 84',[0.0118, 0.7182, 0.2700]\n#OPTION,__WTS = 'option 85',[0.0121, 0.7179, 0.2700]\n#OPTION,__WTS = 'option 86',[0.0017, 0.7183, 0.2800] #  BEST\n#OPTION,__WTS = 'option 87',[0.0119, 0.7181, 0.2700]","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"papermill":{"duration":0.035727,"end_time":"2024-10-26T03:26:25.661447","exception":false,"start_time":"2024-10-26T03:26:25.62572","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-11-19T14:28:03.932268Z","iopub.execute_input":"2024-11-19T14:28:03.932527Z","iopub.status.idle":"2024-11-19T14:28:03.940245Z","shell.execute_reply.started":"2024-11-19T14:28:03.932499Z","shell.execute_reply":"2024-11-19T14:28:03.939383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !pip install polars\n# !pip install xgboost\n# !pip install lightgbm\n# !pip install catboost","metadata":{"execution":{"iopub.status.busy":"2024-11-19T14:28:03.944875Z","iopub.execute_input":"2024-11-19T14:28:03.945767Z","iopub.status.idle":"2024-11-19T14:28:03.950480Z","shell.execute_reply.started":"2024-11-19T14:28:03.945718Z","shell.execute_reply":"2024-11-19T14:28:03.949740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensemble of solution","metadata":{}},{"cell_type":"code","source":"import pandas as pd, polars as pl\n\n\ndef predict(test:pl.DataFrame, lags:pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:    \n    pdA = predict_7 (test,lags).to_pandas()\n    pdB = predict_14(test,lags).to_pandas()\n    pdC = predict_5 (test,lags).to_pandas()\n    pdA = pdA.rename(columns={'responder_6':'responder_A'})\n    pdB = pdB.rename(columns={'responder_6':'responder_B'})\n    pdC = pdC.rename(columns={'responder_6':'responder_C'})\n    pds = pd.merge(pdA,pdB, on=['row_id'])\n    pds = pd.merge(pds,pdC, on=['row_id'])\n    pds['responder_6'] =\\\n        pds['responder_A'] *__WTS[0] +\\\n        pds['responder_B'] *__WTS[1] +\\\n        pds['responder_C'] *__WTS[2]\n    display(pds)\n    predictions = test.select('row_id', pl.lit(0.0).alias('responder_6'))\n    pred = pds['responder_6'].to_numpy()\n    predictions = predictions.with_columns(pl.Series('responder_6', pred.ravel()))\n    return predictions\n","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:03.959939Z","iopub.execute_input":"2024-11-19T14:28:03.960332Z","iopub.status.idle":"2024-11-19T14:28:05.025160Z","shell.execute_reply.started":"2024-11-19T14:28:03.960307Z","shell.execute_reply":"2024-11-19T14:28:05.024262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thank you for visiting my notebook. If you liked the notebook and it was useful for you, don't forget to give it a plus. Thank you!","metadata":{}},{"cell_type":"markdown","source":"## 5. [JS Ridge baseline](https://www.kaggle.com/code/yunsuxiaozi/js-ridge-baseline) Lb=0.0026\n### [yunsuxiaozi](https://www.kaggle.com/yunsuxiaozi)","metadata":{"papermill":{"duration":0.022173,"end_time":"2024-10-26T03:27:39.285949","exception":false,"start_time":"2024-10-26T03:27:39.263776","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if 'SOLUTION_5' in ENSEMBLE_SOLUTIONS:\n    \n    def predict_5(test,lags):\n        cols=[f'feature_0{i}' if i<10 else f'feature_{i}' for i in range(79)]\n        predictions = test.select(\n            'row_id',\n            pl.lit(0.0).alias('responder_6'),\n        )\n        test_preds=model_5.predict(test[cols].to_pandas().fillna(3).values)\n        predictions = predictions.with_columns(pl.Series('responder_6', test_preds.ravel()))\n        return predictions","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.031534,"end_time":"2024-10-26T03:27:39.453592","exception":false,"start_time":"2024-10-26T03:27:39.422058","status":"completed"},"tags":[],"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:05.026686Z","iopub.execute_input":"2024-11-19T14:28:05.027047Z","iopub.status.idle":"2024-11-19T14:28:05.032036Z","shell.execute_reply.started":"2024-11-19T14:28:05.027020Z","shell.execute_reply":"2024-11-19T14:28:05.031209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nif 'SOLUTION_5' in ENSEMBLE_SOLUTIONS:\n    from sklearn.linear_model import BayesianRidge\n    import joblib\n    model_5 = joblib.load('/kaggle/input/jane-street-5-and-7_/other/default/1/ridge_model_5(1).pkl')\n\n","metadata":{"execution":{"iopub.status.busy":"2024-11-19T14:28:05.032955Z","iopub.execute_input":"2024-11-19T14:28:05.033187Z","iopub.status.idle":"2024-11-19T14:28:06.258163Z","shell.execute_reply.started":"2024-11-19T14:28:05.033164Z","shell.execute_reply":"2024-11-19T14:28:06.257058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7. [JS BayesianRidge baseline](https://www.kaggle.com/code/vyacheslavbolotin/js-bayesianridge-baseline) Lb=0.0027\n### [invisible](https://www.kaggle.com/vyacheslavbolotin/)","metadata":{"papermill":{"duration":0.023164,"end_time":"2024-10-26T03:27:40.561582","exception":false,"start_time":"2024-10-26T03:27:40.538418","status":"completed"},"tags":[]}},{"cell_type":"code","source":"    def predict_7(test,lags):\n        cols=[f'feature_0{i}' if i<10 else f'feature_{i}' for i in range(79)]\n        predictions = test.select(\n            'row_id',\n            pl.lit(0.0).alias('responder_6'),\n        )\n        test_preds=model_7.predict(test[cols].to_pandas().fillna(3).values)\n        predictions = predictions.with_columns(pl.Series('responder_6', test_preds.ravel()))\n        return predictions","metadata":{"execution":{"iopub.status.busy":"2024-11-19T14:28:06.260488Z","iopub.execute_input":"2024-11-19T14:28:06.261013Z","iopub.status.idle":"2024-11-19T14:28:06.266297Z","shell.execute_reply.started":"2024-11-19T14:28:06.260981Z","shell.execute_reply":"2024-11-19T14:28:06.265485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.linear_model import BayesianRidge\nimport joblib\nif 'SOLUTION_7' in ENSEMBLE_SOLUTIONS:\n    model_7 = joblib.load('/kaggle/input/jane-street-5-and-7_/other/default/1/bayesian_ridge_model_7(1).pkl')\n    \n    \n\n","metadata":{"execution":{"iopub.status.busy":"2024-11-19T14:28:06.267304Z","iopub.execute_input":"2024-11-19T14:28:06.267556Z","iopub.status.idle":"2024-11-19T14:28:06.285565Z","shell.execute_reply.started":"2024-11-19T14:28:06.267531Z","shell.execute_reply":"2024-11-19T14:28:06.284537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 14. [Jane Street RMF NN + XGB](https://www.kaggle.com/code/voix97/jane-street-rmf-nn-xgb), Lb=0.0056\n### [Xiang Sheng](https://www.kaggle.com/voix97)","metadata":{}},{"cell_type":"markdown","source":"### Useful notebooks:\n\n- Preprocessing : https://www.kaggle.com/code/motono0223/js24-preprocessing-create-lags\n- Training (XGB) : https://www.kaggle.com/code/motono0223/js24-train-gbdt-model-with-lags-singlemodel\n  - trained XGB model : https://www.kaggle.com/datasets/motono0223/js24-trained-gbdt-model\n  - trained NN model : https://www.kaggle.com/datasets/voix97/js-xs-nn-trained-model\n- Inference of NN : https://www.kaggle.com/code/voix97/jane-street-rmf-nn-with-pytorch-lightning\n- Inference of NN+XGB: **this notebook**  https://www.kaggle.com/code/voix97/jane-street-rmf-nn-xgb\n- EDA(1) : https://www.kaggle.com/code/motono0223/eda-jane-street-real-time-market-data-forecasting\n- EDA(2) : https://www.kaggle.com/code/motono0223/eda-v2-jane-street-real-time-market-forecasting","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    import pandas as pd\n    import polars as pl\n    import numpy as np\n    import os, gc\n    from tqdm.auto import tqdm\n    from matplotlib import pyplot as plt\n    import pickle\n\n    import torch\n    import torch.nn as nn\n    import torch.nn.functional as F\n    from pytorch_lightning import (LightningDataModule, LightningModule, Trainer)\n    from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint, Timer\n\n    import pandas as pd\n    import numpy as np\n    from sklearn.metrics import r2_score\n    from sklearn.model_selection import train_test_split\n    from torch.utils.data import Dataset, DataLoader\n\n\n    from sklearn.metrics import r2_score\n    from lightgbm import LGBMRegressor\n    import lightgbm as lgb\n    from xgboost import XGBRegressor\n    from catboost import CatBoostRegressor\n    from sklearn.ensemble import VotingRegressor\n\n    import warnings\n    warnings.filterwarnings('ignore')\n    pd.options.display.max_columns = None\n\n    import kaggle_evaluation.jane_street_inference_server","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:06.286559Z","iopub.execute_input":"2024-11-19T14:28:06.286946Z","iopub.status.idle":"2024-11-19T14:28:15.370749Z","shell.execute_reply.started":"2024-11-19T14:28:06.286920Z","shell.execute_reply":"2024-11-19T14:28:15.369746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NN + XGB inference\n\n### Configurations","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:    \n    \n    class CONFIG:\n        seed = 42\n        target_col = \"responder_6\"\n        # feature_cols = [\"symbol_id\", \"time_id\"] + [f\"feature_{idx:02d}\" for idx in range(79)]+ [f\"responder_{idx}_lag_1\" for idx in range(9)]\n        feature_cols = [f\"feature_{idx:02d}\" for idx in range(79)]+ [f\"responder_{idx}_lag_1\" for idx in range(9)]\n\n        model_paths = [\n            #\"/kaggle/input/js24-train-gbdt-model-with-lags-singlemodel/result.pkl\",\n            #\"/kaggle/input/js24-trained-gbdt-model/result.pkl\",\n            \"/kaggle/input/js-xs-nn-trained-model\",\n            \"/kaggle/input/js-with-lags-trained-xgb/result.pkl\",\n        ]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:15.372078Z","iopub.execute_input":"2024-11-19T14:28:15.373097Z","iopub.status.idle":"2024-11-19T14:28:15.377959Z","shell.execute_reply.started":"2024-11-19T14:28:15.373055Z","shell.execute_reply":"2024-11-19T14:28:15.377028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load preprocessed data (to calculate CV)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    valid = pl.scan_parquet(\n        f\"/kaggle/input/js24-preprocessing-create-lags/validation.parquet/\"\n    ).collect().to_pandas()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:15.379097Z","iopub.execute_input":"2024-11-19T14:28:15.379728Z","iopub.status.idle":"2024-11-19T14:28:18.219535Z","shell.execute_reply.started":"2024-11-19T14:28:15.379666Z","shell.execute_reply":"2024-11-19T14:28:18.218862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load model","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS: \n    \n    xgb_model = None\n    model_path = CONFIG.model_paths[1]\n    with open( model_path, \"rb\") as fp:\n        result = pickle.load(fp)\n        xgb_model = result[\"model\"]\n\n    xgb_feature_cols = [\"symbol_id\", \"time_id\"] + CONFIG.feature_cols\n\n    # Show model\n    display(xgb_model)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:18.220990Z","iopub.execute_input":"2024-11-19T14:28:18.221429Z","iopub.status.idle":"2024-11-19T14:28:18.317155Z","shell.execute_reply.started":"2024-11-19T14:28:18.221367Z","shell.execute_reply":"2024-11-19T14:28:18.316209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    # Custom R2 metric for validation\n    def r2_val(y_true, y_pred, sample_weight):\n        r2 = 1 - np.average((y_pred - y_true) ** 2, weights=sample_weight) / (np.average((y_true) ** 2, weights=sample_weight) + 1e-38)\n        return r2\n\n\n    class NN(LightningModule):\n        def __init__(self, input_dim, hidden_dims, dropouts, lr, weight_decay):\n            super().__init__()\n            self.save_hyperparameters()\n            layers = []\n            in_dim = input_dim\n            for i, hidden_dim in enumerate(hidden_dims):\n                layers.append(nn.BatchNorm1d(in_dim))\n                if i > 0:\n                    layers.append(nn.SiLU())\n                if i < len(dropouts):\n                    layers.append(nn.Dropout(dropouts[i]))\n                layers.append(nn.Linear(in_dim, hidden_dim))\n                # layers.append(nn.ReLU())\n                in_dim = hidden_dim\n            layers.append(nn.Linear(in_dim, 1))  # 输出层\n            layers.append(nn.Tanh())\n            self.model = nn.Sequential(*layers)\n            self.lr = lr\n            self.weight_decay = weight_decay\n            self.validation_step_outputs = []\n\n        def forward(self, x):\n            return 5 * self.model(x).squeeze(-1)  # 输出为一维张量\n\n        def training_step(self, batch):\n            x, y, w = batch\n            y_hat = self(x)\n            loss = F.mse_loss(y_hat, y, reduction='none') * w  # 考虑样本权重\n            loss = loss.mean()\n            self.log('train_loss', loss, on_step=False, on_epoch=True, batch_size=x.size(0))\n            return loss\n\n        def validation_step(self, batch):\n            x, y, w = batch\n            y_hat = self(x)\n            loss = F.mse_loss(y_hat, y, reduction='none') * w\n            loss = loss.mean()\n            self.log('val_loss', loss, on_step=False, on_epoch=True, batch_size=x.size(0))\n            self.validation_step_outputs.append((y_hat, y, w))\n            return loss\n\n        def on_validation_epoch_end(self):\n            \"\"\"Calculate validation WRMSE at the end of the epoch.\"\"\"\n            y = torch.cat([x[1] for x in self.validation_step_outputs]).cpu().numpy()\n            if self.trainer.sanity_checking:\n                prob = torch.cat([x[0] for x in self.validation_step_outputs]).cpu().numpy()\n            else:\n                prob = torch.cat([x[0] for x in self.validation_step_outputs]).cpu().numpy()\n                weights = torch.cat([x[2] for x in self.validation_step_outputs]).cpu().numpy()\n                # r2_val\n                val_r_square = r2_val(y, prob, weights)\n                self.log(\"val_r_square\", val_r_square, prog_bar=True, on_step=False, on_epoch=True)\n            self.validation_step_outputs.clear()\n\n        def configure_optimizers(self):\n            optimizer = torch.optim.Adam(self.parameters(), lr=self.lr, weight_decay=self.weight_decay)\n            scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=5,\n                                                                   verbose=True)\n            return {\n                'optimizer': optimizer,\n                'lr_scheduler': {\n                    'scheduler': scheduler,\n                    'monitor': 'val_loss',\n                }\n            }\n\n        def on_train_epoch_end(self):\n            if self.trainer.sanity_checking:\n                return\n            epoch = self.trainer.current_epoch\n            metrics = {k: v.item() if isinstance(v, torch.Tensor) else v for k, v in self.trainer.logged_metrics.items()}\n            formatted_metrics = {k: f\"{v:.5f}\" for k, v in metrics.items()}\n            print(f\"Epoch {epoch}: {formatted_metrics}\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:18.320045Z","iopub.execute_input":"2024-11-19T14:28:18.320743Z","iopub.status.idle":"2024-11-19T14:28:18.340823Z","shell.execute_reply.started":"2024-11-19T14:28:18.320708Z","shell.execute_reply":"2024-11-19T14:28:18.339992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    N_folds = 5\n    # 加载最佳模型\n    models = []\n    for fold in range(N_folds):\n        checkpoint_path = f\"{CONFIG.model_paths[0]}/nn_{fold}.model\"\n        model = NN.load_from_checkpoint(checkpoint_path)\n        models.append(model.to(\"cuda:0\"))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:18.341877Z","iopub.execute_input":"2024-11-19T14:28:18.342170Z","iopub.status.idle":"2024-11-19T14:28:19.302778Z","shell.execute_reply.started":"2024-11-19T14:28:18.342146Z","shell.execute_reply":"2024-11-19T14:28:19.301999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### CV Score","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS: \n    \n    X_valid = valid[ xgb_feature_cols ]\n    y_valid = valid[ CONFIG.target_col ]\n    w_valid = valid[ \"weight\" ]\n    y_pred_valid_xgb = xgb_model.predict(X_valid)\n    valid_score = r2_score( y_valid, y_pred_valid_xgb, sample_weight=w_valid )\n    valid_score","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:19.303780Z","iopub.execute_input":"2024-11-19T14:28:19.304024Z","iopub.status.idle":"2024-11-19T14:28:22.059235Z","shell.execute_reply.started":"2024-11-19T14:28:19.303999Z","shell.execute_reply":"2024-11-19T14:28:22.058390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    X_valid = valid[ CONFIG.feature_cols ]\n    y_valid = valid[ CONFIG.target_col ]\n    w_valid = valid[ \"weight\" ]\n    X_valid = X_valid.fillna(method = 'ffill').fillna(0)\n    X_valid.shape, y_valid.shape, w_valid.shape","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:22.060266Z","iopub.execute_input":"2024-11-19T14:28:22.060514Z","iopub.status.idle":"2024-11-19T14:28:24.028395Z","shell.execute_reply.started":"2024-11-19T14:28:22.060489Z","shell.execute_reply":"2024-11-19T14:28:24.027640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    y_pred_valid_nn = np.zeros(y_valid.shape)\n    with torch.no_grad():\n        for model in models:\n            model.eval()\n            y_pred_valid_nn += model(torch.FloatTensor(X_valid.values).to(\"cuda:0\")).cpu().numpy() / len(models)\n    valid_score = r2_score( y_valid, y_pred_valid_nn, sample_weight=w_valid )\n    valid_score","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:24.029382Z","iopub.execute_input":"2024-11-19T14:28:24.029672Z","iopub.status.idle":"2024-11-19T14:28:30.560269Z","shell.execute_reply.started":"2024-11-19T14:28:24.029647Z","shell.execute_reply":"2024-11-19T14:28:30.559503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    y_pred_valid_ensemble = 0.5 * (y_pred_valid_xgb + y_pred_valid_nn)\n    valid_score = r2_score( y_valid, y_pred_valid_ensemble, sample_weight=w_valid )\n    valid_score","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:30.561215Z","iopub.execute_input":"2024-11-19T14:28:30.561486Z","iopub.status.idle":"2024-11-19T14:28:30.580763Z","shell.execute_reply.started":"2024-11-19T14:28:30.561461Z","shell.execute_reply":"2024-11-19T14:28:30.579999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:\n    \n    del valid, X_valid, y_valid, w_valid\n    gc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:30.581839Z","iopub.execute_input":"2024-11-19T14:28:30.582108Z","iopub.status.idle":"2024-11-19T14:28:30.779104Z","shell.execute_reply.started":"2024-11-19T14:28:30.582083Z","shell.execute_reply":"2024-11-19T14:28:30.778220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### There seems to be bug in official code, can only submit polars dataframe","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"if 'SOLUTION_14' in ENSEMBLE_SOLUTIONS:    \n    \n    lags_ : pl.DataFrame | None = None\n\n    def predict_14(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:\n        global lags_\n        if lags is not None:\n            lags_ = lags\n\n        predictions_14 = test.select(\n            'row_id',\n            pl.lit(0.0).alias('responder_6'),\n        )\n        symbol_ids = test.select('symbol_id').to_numpy()[:, 0]\n\n        if not lags is None:\n            lags = lags.group_by([\"date_id\", \"symbol_id\"], maintain_order=True).last() # pick up last record of previous date\n            test = test.join(lags, on=[\"date_id\", \"symbol_id\"],  how=\"left\")\n        else:\n            test = test.with_columns(\n                ( pl.lit(0.0).alias(f'responder_{idx}_lag_1') for idx in range(9) )\n            )\n\n        preds = np.zeros((test.shape[0],))\n        preds += xgb_model.predict(test[xgb_feature_cols].to_pandas()) / 2\n        test_input = test[CONFIG.feature_cols].to_pandas()\n        test_input = test_input.fillna(method = 'ffill').fillna(0)\n        test_input = torch.FloatTensor(test_input.values).to(\"cuda:0\")\n        with torch.no_grad():\n            for i, nn_model in enumerate(tqdm(models)):\n                nn_model.eval()\n                preds += nn_model(test_input).cpu().numpy() / 10\n        print(f\"predict> preds.shape =\", preds.shape)\n\n        predictions_14 = \\\n        test.select('row_id').\\\n        with_columns(\n            pl.Series(\n                name   = 'responder_6', \n                values = np.clip(preds, a_min = -5, a_max = 5),\n                dtype  = pl.Float64,\n            )\n        )\n\n        # The predict function must return a DataFrame\n        #assert isinstance(predictions, pl.DataFrame | pd.DataFrame)\n        # with columns 'row_id', 'responer_6'\n        #assert list(predictions.columns) == ['row_id', 'responder_6']\n        # and as many rows as the test data.\n        #assert len(predictions) == len(test)\n\n        return predictions_14","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-11-19T14:28:30.780345Z","iopub.execute_input":"2024-11-19T14:28:30.780650Z","iopub.status.idle":"2024-11-19T14:28:30.791999Z","shell.execute_reply.started":"2024-11-19T14:28:30.780623Z","shell.execute_reply":"2024-11-19T14:28:30.791125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When your notebook is run on the hidden test set, inference_server.serve must be called within 15 minutes of the notebook starting or the gateway will throw an error. If you need more than 15 minutes to load your model you can do so during the very first `predict` call, which does not have the usual 10 minute response deadline.","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"markdown","source":"## inference_server","metadata":{"papermill":{"duration":0.023299,"end_time":"2024-10-26T03:27:41.615214","exception":false,"start_time":"2024-10-26T03:27:41.591915","status":"completed"},"tags":[]}},{"cell_type":"code","source":"inference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)\n\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n    inference_server.serve()\nelse:\n    inference_server.run_local_gateway(\n        (\n            '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n            '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n        )\n    )","metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"papermill":{"duration":0.351292,"end_time":"2024-10-26T03:27:42.101707","exception":false,"start_time":"2024-10-26T03:27:41.750415","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-11-19T14:28:48.726519Z","iopub.execute_input":"2024-11-19T14:28:48.727127Z","iopub.status.idle":"2024-11-19T14:28:48.858684Z","shell.execute_reply.started":"2024-11-19T14:28:48.727090Z","shell.execute_reply":"2024-11-19T14:28:48.856289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}