{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.14"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84493,"databundleVersionId":9871156,"sourceType":"competition"},{"sourceId":9921468,"sourceType":"datasetVersion","datasetId":6090371},{"sourceId":207221536,"sourceType":"kernelVersion"}],"dockerImageVersionId":30786,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":4.669361,"end_time":"2024-10-10T13:05:46.686069","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-10-10T13:05:42.016708","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Jane Street RMF: TensorFlow Custom Training Loop\n\n## Overview\n\nThis notebook tackles a real-time market data forecasting challenge using TensorFlow. The primary goal is to predict a target variable (responder_6) based on 79 anonymized features. The notebook incorporates a complete workflow for data loading, model training, validation, and submission. It uses a custom training loop and implements advanced techniques like a custom weighted loss function and performance metrics. The focus is on handling the complexities of financial market data, including non-stationarity, fat-tailed distributions, and dynamic behavior.\n\nThe solution is built around a feedforward neural network, with enhancements including weighted loss, custom R² metric, and checkpointing for improved training and evaluation. Additionally, the notebook provides an inference pipeline and a local server setup for testing prediction requests.\n\n## Key Components\n\n**1.Configuration**\n\nA configuration class (CFG) defines global parameters, such as training mode and other configurations for experimentation.\n\n**2.Data Loading**\n- **Features**: The dataset includes 79 feature columns (feature_00 to feature_78).\n- **Target**: The target column is responder_6.\n- **Training and Validation Data**: Data is loaded from partitioned Parquet files and combined into training (X_train, y_train) and validation sets (X_val, y_val).\n\n**3.Data Generators**\n\nA generator function (get_generator) yields batches of data for efficient training and validation.\n\n**4.TensorFlow Dataset**\n\nTensorFlow datasets (train_ds and valid_ds) are constructed from the generators, enabling seamless integration with TensorFlow’s training framework.\n\n**5.Custom Loss and Metrics**\n\n• **Weighted MSE Loss**: A custom loss function (weighted_mse) accounts for sample weights to handle the varying importance of data points.\n\n• **Custom R² Metric**: A custom metric (R2Metric) is used to evaluate model performance beyond standard metrics.\n\n**6.Model Architecture**\n\nThe model architecture is a feedforward neural network with the following layers:\n- **Input layer**: Accepts 79 features.\n- **Hidden layers**:\n- 512 units with Swish activation and 30% dropout.\n- 256 units with Swish activation and 20% dropout.\n- 128 units with Swish activation and 10% dropout.\n- 32 units with Swish activation and L2 regularization.\n- **Output layer**: A single neuron for regression.\n\nThe model is compiled with:\n- **Loss**: Weighted MSE (weighted_mse).\n- **Optimizer**: Adam with a learning rate of 5e-5.\n- **Metric**: Custom R² metric (R2Metric).\n\n**7.Custom Training Loop**\n\nThe training loop incorporates advanced features such as:\n\n- Real-time batch-wise metric computation.\n- Handling NaN loss scenarios.\n- Dynamic time monitoring to avoid prolonged training.\n- Early stopping with patience to prevent overfitting.\n- Checkpointing based on the best validation R² score.\n\n**8.Validation and Checkpointing**\nValidation is performed after every epoch, with the best model saved based on validation R². Early stopping halts training if performance stagnates.\n\n**9. Predict Function**\nA predict function handles missing values, makes predictions using the trained model, and formats outputs as required for submission.\n\n**10.Inference Server**\nThe notebook sets up an inference server (JSInferenceServer) to process prediction requests, adapting seamlessly to both competition rerun environments and local testing setups.\n\n## Dataset Description\n\nThe dataset reflects the complexities of real-world financial markets. It presents challenges such as non-stationarity, fat-tailed distributions, and dynamic behaviors driven by economic, geopolitical, and societal changes. Features and targets are anonymized to balance relevance and data security, creating a realistic simulation of the quantitative problems tackled in modern financial trading.\n\nJane Street’s automated trading strategies rely on robust machine learning models. This challenge exemplifies the type of modeling innovation that supports their daily operations across thousands of financial products and venues.","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport polars as pl\nimport kaggle_evaluation.jane_street_inference_server\nimport tensorflow as tf\nimport numpy as np\nimport pickle\nimport gc\nimport time","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:42:13.885987Z","iopub.execute_input":"2024-11-15T01:42:13.887771Z","iopub.status.idle":"2024-11-15T01:42:13.894132Z","shell.execute_reply.started":"2024-11-15T01:42:13.887695Z","shell.execute_reply":"2024-11-15T01:42:13.892818Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1. Configuration","metadata":{}},{"cell_type":"code","source":"class CFG:\n    \n    is_training = False\n    \n    feature_columns =  [f\"feature_{i:02d}\" for i in range(79)]\n    \n    target_column = \"responder_6\"","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:03:32.269142Z","iopub.execute_input":"2024-11-15T01:03:32.269575Z","iopub.status.idle":"2024-11-15T01:03:32.276453Z","shell.execute_reply.started":"2024-11-15T01:03:32.269531Z","shell.execute_reply":"2024-11-15T01:03:32.275020Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Load training and validation data","metadata":{}},{"cell_type":"markdown","source":"### Load statics values","metadata":{}},{"cell_type":"code","source":"with open(\"/kaggle/input/jane-street-rmf-computing-feature-statistics/max.pkl\", \"rb\") as f:\n    max_values = pickle.load(f)\nwith open(\"/kaggle/input/jane-street-rmf-computing-feature-statistics/min.pkl\", \"rb\") as f:\n    min_values = pickle.load(f)\nwith open(\"/kaggle/input/jane-street-rmf-computing-feature-statistics/mean.pkl\", \"rb\") as f:\n    mean_values = pickle.load(f)\nwith open(\"/kaggle/input/jane-street-rmf-computing-feature-statistics/mean_pandas.pkl\", \"rb\") as f:\n    mean_pandas = pickle.load(f)\nmax_values = max_values.astype(np.float16)\nmin_values = min_values.astype(np.float16)\nmax_min_diff = max_values - min_values\nmean_values = mean_values.astype(np.float16)\nmean_pandas = mean_pandas.astype(np.float16)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-15T01:03:34.202090Z","iopub.execute_input":"2024-11-15T01:03:34.202543Z","iopub.status.idle":"2024-11-15T01:03:34.243139Z","shell.execute_reply.started":"2024-11-15T01:03:34.202499Z","shell.execute_reply":"2024-11-15T01:03:34.241783Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Load training data","metadata":{}},{"cell_type":"code","source":"if CFG.is_training:\n    X_train = None\n    y_train = None\n    train_weights = None\n    for i in range(9):\n        print(\"=\" * 30)\n        print(f\"Partition {i}\")\n        print(\"=\" * 30)\n        df = pd.read_parquet(f\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id={i}/part-0.parquet\")\n        X_part = df[CFG.feature_columns].astype(np.float16)\n        X_part.fillna(mean_pandas, inplace=True)\n        X_part = (X_part - min_values) / max_min_diff\n        targets = df[CFG.target_column]\n        weights = df[\"weight\"].astype(np.float16)\n        if X_train is None:\n            X_train = X_part\n            y_train = targets\n            train_weights = weights\n        else:\n            X_train = pd.concat([X_train, X_part])\n            y_train = pd.concat([y_train, targets])\n            train_weights = pd.concat([train_weights, weights])\n        del X_part\n        del targets\n        del weights\n        gc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:03:39.904831Z","iopub.execute_input":"2024-11-15T01:03:39.905298Z","iopub.status.idle":"2024-11-15T01:07:22.254144Z","shell.execute_reply.started":"2024-11-15T01:03:39.905256Z","shell.execute_reply":"2024-11-15T01:07:22.252834Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Load Validation data","metadata":{}},{"cell_type":"code","source":"X_val = None\ny_val = None\nval_weights = None\nfor i in range(9, 10):\n    print(\"=\" * 30)\n    print(f\"Partition {i}\")\n    print(\"=\" * 30)\n    df = pd.read_parquet(f\"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id={i}/part-0.parquet\")\n    X_part = df[CFG.feature_columns].astype(np.float16)\n    X_part.fillna(mean_pandas, inplace=True)\n    X_part = (X_part - min_values) / max_min_diff\n    targets = df[CFG.target_column]\n    weights = df[\"weight\"].astype(np.float16)\n    if X_val is None:\n        X_val = X_part\n        y_val = targets\n        val_weights = weights\n    else:\n        X_val = pd.concat([X_val, X_part])\n        y_val = pd.concat([y_val, targets])\n        val_weights = pd.concat([val_weights, weights])\n    del X_part\n    del weights\n    del targets\n    del df\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:08:36.930962Z","iopub.execute_input":"2024-11-15T01:08:36.931507Z","iopub.status.idle":"2024-11-15T01:09:05.928282Z","shell.execute_reply.started":"2024-11-15T01:08:36.931458Z","shell.execute_reply":"2024-11-15T01:09:05.926998Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Create Training and Validation dataset","metadata":{}},{"cell_type":"code","source":"def get_generator(X, y, weights, shuffle=True, batch_size=4096):\n    def generator():\n        # Create a shuffled index for random access\n        if shuffle:\n            indices = np.random.permutation(len(X))\n        else:\n            indices = np.arange(len(X))\n        num_batch = len(indices) // batch_size + (1 if len(indices) % batch_size > 0 else 0)\n        for i in range(num_batch):\n            start_index = i * batch_size\n            end_index = min((i + 1) * batch_size, len(X))\n            current_indices = indices[start_index: end_index]\n            features = X.iloc[current_indices].values\n            label = y.iloc[current_indices]\n            weight_batch = weights.iloc[current_indices].values\n            yield {\n                \"features\": features,\n                \"weights\":  np.array(weight_batch).reshape(-1, 1)\n            }, np.array(label).reshape(-1, 1)\n    return generator\n        \ndef create_dataset(X, y, weights, shuffle=True, batch_size=4096):\n    output_signature = (\n        {\n            \"features\": tf.TensorSpec(shape=(None, len(CFG.feature_columns),), dtype=tf.float32),\n            \"weights\": tf.TensorSpec(shape=(None, 1), dtype=tf.float32)\n        }, # Adjust shape to number of features\n        tf.TensorSpec(shape=(None, 1), dtype=tf.float32)\n    )\n    # Create a TensorFlow Dataset from the generator\n    ds = tf.data.Dataset.from_generator(\n        get_generator(X, y, weights, shuffle=shuffle, batch_size=batch_size),\n        output_signature=output_signature\n    )\n    return ds","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:29:59.972242Z","iopub.execute_input":"2024-11-15T01:29:59.972748Z","iopub.status.idle":"2024-11-15T01:29:59.987369Z","shell.execute_reply.started":"2024-11-15T01:29:59.972703Z","shell.execute_reply":"2024-11-15T01:29:59.986074Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.is_training:\n    train_ds = create_dataset(X_train, y_train, train_weights, shuffle=True)","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:30:05.021219Z","iopub.execute_input":"2024-11-15T01:30:05.021742Z","iopub.status.idle":"2024-11-15T01:30:05.073035Z","shell.execute_reply.started":"2024-11-15T01:30:05.021699Z","shell.execute_reply":"2024-11-15T01:30:05.071424Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"valid_ds = create_dataset(X_val, y_val, val_weights, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:30:06.204973Z","iopub.execute_input":"2024-11-15T01:30:06.205627Z","iopub.status.idle":"2024-11-15T01:30:06.255710Z","shell.execute_reply.started":"2024-11-15T01:30:06.205571Z","shell.execute_reply":"2024-11-15T01:30:06.254156Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Check data format","metadata":{}},{"cell_type":"code","source":"for batch in valid_ds:\n    print(batch)\n    break","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:30:07.841568Z","iopub.execute_input":"2024-11-15T01:30:07.842853Z","iopub.status.idle":"2024-11-15T01:30:07.909366Z","shell.execute_reply.started":"2024-11-15T01:30:07.842776Z","shell.execute_reply":"2024-11-15T01:30:07.907935Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Modeling","metadata":{}},{"cell_type":"code","source":"class R2Metric(tf.keras.metrics.Metric):\n    def __init__(self, name='r2', **kwargs):\n        super(R2Metric, self).__init__(name=name, **kwargs)\n        self.squared_residuals_sum = self.add_weight(name='squared_residuals_sum', initializer='zeros')\n        self.total_sum_squares = self.add_weight(name='total_sum_squares', initializer='zeros')\n        self.count = self.add_weight(name='count', initializer='zeros')\n\n    def update_state(self, y_true, y_pred, sample_weight=None):\n        # Flatten tensors to ensure shape compatibility\n        y_true = tf.reshape(y_true, (-1,))\n        y_pred = tf.reshape(y_pred, (-1,))\n        weights  = tf.reshape(sample_weight, (-1,))\n        # Residual Sum of Squares\n        residuals = y_true - y_pred\n\n        #print(\"Res:\", weights * tf.reduce_sum(tf.square(residuals)))\n        self.squared_residuals_sum.assign_add(tf.reduce_sum(weights * tf.square(residuals)))\n        \n        # Total Sum of Squares\n        self.total_sum_squares.assign_add(tf.reduce_sum(weights * tf.square(y_true)))\n        \n        # Increment count\n        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))\n\n    def result(self):\n        # Calculate R2: 1 - (SS_res / SS_tot)\n        return 1 - (self.squared_residuals_sum / (self.total_sum_squares + tf.keras.backend.epsilon()))\n\n    def reset_states(self):\n        # Reset all variables at the beginning of each epoch\n        self.squared_residuals_sum.assign(0.0)\n        self.total_sum_squares.assign(0.0)\n        self.count.assign(0.0)\n","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:38:01.950489Z","iopub.execute_input":"2024-11-15T01:38:01.951454Z","iopub.status.idle":"2024-11-15T01:38:01.965643Z","shell.execute_reply.started":"2024-11-15T01:38:01.951402Z","shell.execute_reply":"2024-11-15T01:38:01.964325Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Weighted MSE Loss\ndef weighted_mse(y_true, y_pred, sample_weight=None):\n    mse = tf.square(y_true - y_pred)\n    if sample_weight is not None:\n        mse = sample_weight * mse\n    return tf.reduce_mean(mse)\n    \ndef get_model():\n    inputs = tf.keras.Input(shape=(79, ), dtype=tf.float16, name=\"features\")\n    x = tf.keras.layers.Dense(512, activation=\"swish\")(inputs)\n    x = tf.keras.layers.Dropout(0.3)(x)\n    x = tf.keras.layers.Dense(256, activation=\"swish\")(x)\n    x = tf.keras.layers.Dropout(0.2)(x)\n    x = tf.keras.layers.Dense(128, activation=\"swish\")(x)\n    x = tf.keras.layers.Dropout(0.1)(x)\n    x = tf.keras.layers.Dense(32, activation=\"swish\", kernel_regularizer=\"l2\")(x)\n    outputs = tf.keras.layers.Dense(1)(x)\n    model = tf.keras.Model(inputs=inputs, outputs=outputs)\n    optimizer = tf.keras.optimizers.Adam(5e-5)\n    model.compile(loss=weighted_mse, optimizer=optimizer, metrics=[R2Metric()])\n    return model","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:38:03.866635Z","iopub.execute_input":"2024-11-15T01:38:03.867167Z","iopub.status.idle":"2024-11-15T01:38:03.878428Z","shell.execute_reply.started":"2024-11-15T01:38:03.867121Z","shell.execute_reply":"2024-11-15T01:38:03.877023Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = get_model()\nmodel.summary()\ntf.keras.utils.plot_model(model, show_shapes=True, show_dtype=True)","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:38:06.259114Z","iopub.execute_input":"2024-11-15T01:38:06.259611Z","iopub.status.idle":"2024-11-15T01:38:06.616168Z","shell.execute_reply.started":"2024-11-15T01:38:06.259557Z","shell.execute_reply":"2024-11-15T01:38:06.614729Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Training or loading model","metadata":{}},{"cell_type":"code","source":"def train_model(train_ds, valid_ds):\n    # Training loop parameters\n    model = get_model()\n    optimizer = tf.keras.optimizers.Adam()\n    r2_metric = R2Metric()\n    epochs = 30\n    patience = 5\n    checkpoint_path = 'model.keras'\n    best_val_r2 = -np.inf  # Initialize to track the best R2 score\n    wait = 0\n    time_out = 9 * 3600\n    total_begin_time = time.time()\n    train_steps = X_train.shape[0] // 4096\n    should_stop_training = False\n    # Custom training loop\n    for epoch in range(epochs):\n        print(f\"\\nEpoch {epoch + 1}/{epochs}\")\n        # Reset the metric states at the beginning of each epoch\n        r2_metric.reset_states()\n        begin_time = time.time()\n        for step, (batch_features, batch_targets) in enumerate(train_ds):\n            with tf.GradientTape() as tape:\n                y_pred = model(batch_features[\"features\"], training=True)\n                loss = weighted_mse(batch_targets, y_pred, batch_features[\"weights\"])\n                # Check for NaN loss\n                if tf.math.is_nan(loss):\n                    print(\"NaN encountered, skip to next batch.\")\n                    continue\n    \n            # Compute gradients and update weights\n            grads = tape.gradient(loss, model.trainable_weights)\n            optimizer.apply_gradients(zip(grads, model.trainable_weights))\n    \n            # Update R2 metric with training batch\n            r2_metric.update_state(batch_targets, y_pred, batch_features[\"weights\"])\n            if step > 0 and step % 100 == 0:\n                elapsed = time.time() - begin_time\n                estimated = (elapsed / step) * train_steps\n                print(f\"Elapsed time: {elapsed:.4f}/{estimated:.4f}s\")\n                if time.time() - total_begin_time > time_out:\n                    print(\"Exceed timeout, stop training.\")\n                    should_stop_training = True\n                    break\n        \n        # Print training R2 for the epoch\n        train_r2 = r2_metric.result().numpy()\n        print(f\"Training R2: {train_r2:.4f}\")\n    \n        # Validation step\n        val_losses = []\n        r2_metric.reset_states()\n        for (batch_features, batch_targets) in valid_ds:\n            y_pred_val = model(batch_features[\"features\"], training=False)\n            val_loss = weighted_mse(batch_targets, y_pred_val,  batch_features[\"weights\"])\n            val_losses.append(val_loss)\n            # Update R2 metric with validation batch\n            r2_metric.update_state(batch_targets, y_pred_val, batch_features[\"weights\"])\n    \n        # Compute average validation loss and R2\n        avg_val_loss = np.mean(val_losses)\n        avg_val_r2 = r2_metric.result().numpy()\n        print(f\"Validation loss: {avg_val_loss:.4f}, Validation R2: {avg_val_r2:.4f}\")\n    \n        # Checkpoint saving based on validation R2\n        model.save(f\"model_{epoch}.keras\")\n        if avg_val_r2 > best_val_r2:\n            best_val_r2 = avg_val_r2\n            model.save(checkpoint_path)\n            print(f\"Model saved with R2: {best_val_r2:.4f}\")\n            wait = 0  # Reset patience counter\n        else:\n            wait += 1  # Increment patience counter\n    \n        # Early stopping\n        if wait >= patience:\n            print(\"Early stopping triggered.\")\n            break\n        if should_stop_training:\n            break\n    return {\n        \"model\": model,\n        \"val_r2\": best_val_r2\n    }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-15T01:48:06.440497Z","iopub.execute_input":"2024-11-15T01:48:06.441035Z","iopub.status.idle":"2024-11-15T01:48:06.460232Z","shell.execute_reply.started":"2024-11-15T01:48:06.440987Z","shell.execute_reply":"2024-11-15T01:48:06.458856Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.is_training:\n    # Model, optimizer, and metric setup\n    results = train_model(train_ds, valid_ds)\n    model = results[\"model\"]\nelse:\n    model_path = \"/kaggle/input/jane-street-rmf-tensorflow-model-v3/model.keras\"\n    model = tf.keras.models.load_model(model_path, custom_objects={\n        \"R2Metric\": R2Metric,\n        \"weighted_mse\": weighted_mse\n    })\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2024-11-15T01:48:09.355693Z","iopub.execute_input":"2024-11-15T01:48:09.356265Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. Model Evaluation","metadata":{}},{"cell_type":"code","source":"def calculate_r2(y_true, y_pred, weights):\n    \"\"\"\n    Calculate the sample weighted zero-mean R-squared score (R2).\n\n    Parameters:\n    - y_true (pd.Series or np.array): Ground truth values.\n    - y_pred (pd.Series or np.array): Predicted values.\n    - weights (pd.Series or np.array): Sample weights.\n\n    Returns:\n    - float: R2 score.\n    \"\"\"\n    numerator = np.sum(weights * (y_true - y_pred) ** 2)\n    denominator = np.sum(weights * (y_true ** 2))\n    r2_score = 1 - (numerator / denominator)\n    return r2_score","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T17:09:26.396823Z","iopub.execute_input":"2024-11-07T17:09:26.397703Z","iopub.status.idle":"2024-11-07T17:09:26.404725Z","shell.execute_reply.started":"2024-11-07T17:09:26.397647Z","shell.execute_reply":"2024-11-07T17:09:26.403368Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"y_pred = model.predict(valid_ds, verbose=0).reshape(-1)\nsample_weight_val = val_weights.values.reshape(-1)\nr2 = calculate_r2(y_val, y_pred, sample_weight_val)\nprint(f\"Validation R2:{r2:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T17:07:48.121013Z","iopub.execute_input":"2024-11-07T17:07:48.122468Z","iopub.status.idle":"2024-11-07T17:07:48.127858Z","shell.execute_reply.started":"2024-11-07T17:07:48.122412Z","shell.execute_reply":"2024-11-07T17:07:48.126489Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7. Create submission pipeline","metadata":{}},{"cell_type":"markdown","source":"\n\n\nYour code will always have access to the published copies of the files.","metadata":{"papermill":{"duration":0.002051,"end_time":"2024-10-10T13:05:45.83073","exception":false,"start_time":"2024-10-10T13:05:45.828679","status":"completed"},"tags":[]}},{"cell_type":"code","source":"lags_ : pl.DataFrame | None = None\n\n\n# Replace this function with your inference code.\n# You can return either a Pandas or Polars dataframe, though Polars is recommended.\n# Each batch of predictions (except the very first) must be returned within 10 minutes of the batch features being provided.\ndef predict(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:\n    \"\"\"Make a prediction.\"\"\"\n    # All the responders from the previous day are passed in at time_id == 0. We save them in a global variable for access at every time_id.\n    # Use them as extra features, if you like.\n    global lags_\n    if lags is not None:\n        lags_ = lags\n    # 1. Select the required feature columns and convert to numpy array for Keras\n    X_test = test.select(CFG.feature_columns).to_numpy()\n    X_test = np.where(np.isnan(X_test), mean_values, X_test)\n    X_test = (X_test - min_values) / max_min_diff\n    # 2. Make predictions using the Keras model\n    y_pred = model.predict(X_test, batch_size=4096)\n    \n    # 3. Prepare the DataFrame for output\n    predictions = test.select('row_id').with_columns(\n        pl.Series(\"responder_6\", y_pred.flatten())\n    )\n    # The predict function must return a DataFrame\n    assert isinstance(predictions, pl.DataFrame | pd.DataFrame)\n    # with columns 'row_id', 'responer_6'\n    assert predictions.columns == ['row_id', 'responder_6']\n    # and as many rows as the test data.\n    assert len(predictions) == len(test)\n\n    return predictions","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.015917,"end_time":"2024-10-10T13:05:45.848958","exception":false,"start_time":"2024-10-10T13:05:45.833041","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-11-02T11:24:56.448353Z","iopub.execute_input":"2024-11-02T11:24:56.448955Z","iopub.status.idle":"2024-11-02T11:24:56.45851Z","shell.execute_reply.started":"2024-11-02T11:24:56.448905Z","shell.execute_reply":"2024-11-02T11:24:56.457142Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.is_training == False:\n\n    inference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)\n\n    if os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n        inference_server.serve()\n    else:\n        inference_server.run_local_gateway(\n            (\n                '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n                '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n            )\n        )","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.308219,"end_time":"2024-10-10T13:05:46.163573","exception":false,"start_time":"2024-10-10T13:05:45.855354","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-11-02T11:24:59.078687Z","iopub.execute_input":"2024-11-02T11:24:59.079183Z","iopub.status.idle":"2024-11-02T11:24:59.211405Z","shell.execute_reply.started":"2024-11-02T11:24:59.079113Z","shell.execute_reply":"2024-11-02T11:24:59.209899Z"},"trusted":true},"outputs":[],"execution_count":null}]}