{ "id": 402864, "title": "20th place solution", "url": "/competitions/icecube-neutrinos-in-deep-ice/writeups/ice-team-20th-place-solution", "author_name": "", "post_date": "2023-05-02T21:01:16.527Z", "votes": 18, "comment_count": 2, "views": 0, "content": "

Team members are junseonglee11 (@junseonglee11), Ayaan Jang(@ayaanjang).
\nWe ensembled 6 LSTM models (2 different versions).
\nWe modified Robin Smith's and Robert Hatch's notebooks.

\n

Our notebooks:

\n

Inference: https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged
\nTrain: https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train
\nDataset (TFRecord): https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1

\n

References

\n

Robin Smith's: notebooks:
\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-inference
\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-training-tpu
\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-data-preprocessor/notebook I modified his notebook

\n

Robert Hatch's notebook
\nhttps://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars
\nIt was crucial to improving our score. Used the results of this notebook as additional inputs in our model.

\n

Seungmoklee's notebook
\nhttps://www.kaggle.com/code/seungmoklee/lstm-preprocessing-point-picker

\n

Data preprocessing

\n

Our code: [https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1]

\n
    \n

  1. Reference part: Preprocessed data to generate 96 time-series data with 6 features including:

    \n
    2. \n

  2. Performed feature engineering on the 6 features to improve the prediction accuracy of the RNN (Residual Neural Network) model (trained on 90 data files instead of the entire dataset, then experimented with feature transformation)

    \n
    3. \n

  3. Convert all the inputs above to TensorFlow TFRecord format to minimize CPU memory usage and accelerate training.

    4. \n
\n

Model training and inference

\n

Our code (model): https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train
\n\"\"

\n\n

Data postprocessing

\n

Our code (Data postprocessing): https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged
\nOriginal code: Power value not squared in the code below.
\n Perform a weighted average of predicted probabilities for each category and the direction of the particle represented by each category to
\n calculate the azimuth and zenith directions of a neutron.
\nChanged code: Add a square of the power value to the predicted value obtained from the model.
\n It's not optimal to simply derive the results based on the direction for each category and the predicted model probability.
\n Attempt various modifications to the predicted probability using exponential, logarithmic functions, activation functions, etc. to improve the
\n post-processing stage.
\nImprovement: When the model-predicted category probabilities were squared by 1.35, there was a decrease of about 0.002 in mean angular error.
\n→ This gave appropriate additional weight to the category with a high probability in the predicted result.

", "messages": [ { "id": "2227753", "postDate": "04/20/2023 02:08:55", "content": "

Team members are junseonglee11 (@junseonglee11), Ayaan Jang(@ayaanjang).
\nWe ensembled 6 LSTM models (2 different versions).
\nWe modified Robin Smith's and Robert Hatch's notebooks.

\n

Our notebooks:

\n

Inference: https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged
\nTrain: https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train
\nDataset (TFRecord): https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1

\n

References

\n

Robin Smith's: notebooks:
\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-inference
\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-training-tpu
\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-data-preprocessor/notebook I modified his notebook

\n

Robert Hatch's notebook
\nhttps://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars
\nIt was crucial to improving our score. Used the results of this notebook as additional inputs in our model.

\n

Seungmoklee's notebook
\nhttps://www.kaggle.com/code/seungmoklee/lstm-preprocessing-point-picker

\n

Data preprocessing

\n

Our code: [https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1]

\n
    \n

  1. Reference part: Preprocessed data to generate 96 time-series data with 6 features including:

    \n
    2. \n

  2. Performed feature engineering on the 6 features to improve the prediction accuracy of the RNN (Residual Neural Network) model (trained on 90 data files instead of the entire dataset, then experimented with feature transformation)

    \n
    3. \n

  3. Convert all the inputs above to TensorFlow TFRecord format to minimize CPU memory usage and accelerate training.

    4. \n
\n

Model training and inference

\n

Our code (model): https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train
\n\"\"

\n\n

Data postprocessing

\n

Our code (Data postprocessing): https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged
\nOriginal code: Power value not squared in the code below.
\n Perform a weighted average of predicted probabilities for each category and the direction of the particle represented by each category to
\n calculate the azimuth and zenith directions of a neutron.
\nChanged code: Add a square of the power value to the predicted value obtained from the model.
\n It's not optimal to simply derive the results based on the direction for each category and the predicted model probability.
\n Attempt various modifications to the predicted probability using exponential, logarithmic functions, activation functions, etc. to improve the
\n post-processing stage.
\nImprovement: When the model-predicted category probabilities were squared by 1.35, there was a decrease of about 0.002 in mean angular error.
\n→ This gave appropriate additional weight to the category with a high probability in the predicted result.

", "rawMarkdown": "Team members are junseonglee11 (@junseonglee11), Ayaan Jang(@ayaanjang). \nWe ensembled 6 LSTM models (2 different versions).\nWe modified Robin Smith's and Robert Hatch's notebooks. \n \n# **Our notebooks:**\nInference: https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged\nTrain: https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train\nDataset (TFRecord): https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1 \n\n# **References**\nRobin Smith's: notebooks:\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-inference\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-training-tpu\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-data-preprocessor/notebook I modified his notebook \n \nRobert Hatch's notebook\nhttps://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars\nIt was crucial to improving our score. Used the results of this notebook as additional inputs in our model.\n\nSeungmoklee's notebook\nhttps://www.kaggle.com/code/seungmoklee/lstm-preprocessing-point-picker\n\n# **Data preprocessing**\n**Our code:** [https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1]\n1. Reference part: Preprocessed data to generate 96 time-series data with 6 features including:\n - Sensor signal measurement time\n - Sensor signal strength\n - Sensor signal quality\n - X, Y, Z coordinates of received sensor (3 features)\n\n2. Performed feature engineering on the 6 features to improve the prediction accuracy of the RNN (Residual Neural Network) model (trained on 90 data files instead of the entire dataset, then experimented with feature transformation)\n - **Using original features: LB 1.015**\n - **Adding Time Difference: LB 1.0128**\n - **Replaced sensor signal measurement time with time interval (time difference between next and current measurement time, Time diff):** \n The difference in sensor position between current and next time points has different meanings depending on the time interval (for example, \n moving 10m in 1 second vs. 1m in 10 seconds has a 10-fold speed difference). However, the absolute measurement time of the sensor signal \n can not reflect this, so it was determined that time interval is a more appropriate input than the measurement time.\n - **Adding Coordinate Difference: LB 1.0115** \n **Added three features with the difference in X, Y, Z coordinates between the next and current sensor positions:** Similar to the time \n interval feature, it was determined that the difference in coordinate values between time points can better reflect the direction information of \n the neutral particle. However, when replacing the sensor's xyz coordinates with coordinate difference values, the accuracy decreased. This is \n because the difference in sensor coordinate values between non-adjacent time points has important information.\n3. Convert all the inputs above to TensorFlow TFRecord format to minimize CPU memory usage and accelerate training.\n\n\n# **Model training and inference**\n**Our code (model):** https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7429668%2Feea4665a122c77a5c2e3aa60a38e65b1%2F.png?generation=1683061023145992&alt=media)\n\n- **Model Overview**: Bidirectional LSTM (long short-term memory) 7 layer+ batch normalization + Line fitting features concatenation + Dense layer\n- **Activation Function**: GELU(Gaussian Error Linear Unit) activation layer\n- **Loss Function**: Sparse_categorical_crossentropy loss\n- **Training Metric**: Accuracy\n- **Optimizer:** RAdam (Rectified Adam) optimizer\n- **Ensemble (2 versions of the model)**\n- Model trained on fold 0 of the dataset divided into 10 folds\n - Train: 594 dataset, Valid: 66 dataset: model trained using only 1 validation dataset\n - Train: 659 dataset, Valid: 1 dataset: four models with the lowest Mean Angular Error values are selected from each version of the model (total of 8 models)\n - Optimal ensemble weights are determined through cross-validation\n\n\n# **Data postprocessing**\n **Our code (Data postprocessing):** [https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged](https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged)\n**Original code:** Power value not squared in the code below.\n Perform a weighted average of predicted probabilities for each category and the direction of the particle represented by each category to \n calculate the azimuth and zenith directions of a neutron.\n**Changed code:** Add a square of the power value to the predicted value obtained from the model.\n It's not optimal to simply derive the results based on the direction for each category and the predicted model probability.\n Attempt various modifications to the predicted probability using exponential, logarithmic functions, activation functions, etc. to improve the \n post-processing stage.\n**Improvement:** When the model-predicted category probabilities were squared by 1.35, there was a decrease of about 0.002 in mean angular error.\n→ This gave appropriate additional weight to the category with a high probability in the predicted result.", "votes": null }, { "id": "2228452", "postDate": "04/20/2023 15:00:35", "content": "

@junseonglee11 nice to see, that you pushed LSTMs that high. Could you explain a bit more this item: Additional inputs and some preprocessing? I tried many experiments with LSTMs, but the best ensemble was around 0.998.

", "rawMarkdown": "junseonglee11 nice to see, that you pushed LSTMs that high. Could you explain a bit more this item: *Additional inputs and some preprocessing*? I tried many experiments with LSTMs, but the best ensemble was around 0.998.", "votes": null }, { "id": "2228849", "postDate": "04/20/2023 21:43:18", "content": "

@manwithaflower additional preprocessing is as follows.

\n
    \n
  1. Used time interval (next time point - current time point) instead of time point (becuase for RNN, I felt that interval is more natural.)
    2. \n
  2. Added difference of sensor position of next and current time point (because I thought that it may contain direction information)
    3. \n
  3. Added line fit result of Robert Hatch's notebook and concatenated it to the layer after the last LSTM layer.
    4. \n
\n

Also, stacking 7 LSTM layers and using all data with TFRecord format were crucial.

\n

Thank you!

", "rawMarkdown": "manwithaflower additional preprocessing is as follows.\n1. Used time interval (next time point - current time point) instead of time point (becuase for RNN, I felt that interval is more natural.)\n2. Added difference of sensor position of next and current time point (because I thought that it may contain direction information)\n3. Added line fit result of Robert Hatch's notebook and concatenated it to the layer after the last LSTM layer.\n\nAlso, stacking 7 LSTM layers and using all data with TFRecord format were crucial.\n\nThank you!", "votes": null } ], "comments": [ { "id": 2228452, "author_name": "manwithaflower", "author_url": "", "post_date": "04/20/2023 15:00:35", "content": "

@junseonglee11 nice to see, that you pushed LSTMs that high. Could you explain a bit more this item: Additional inputs and some preprocessing? I tried many experiments with LSTMs, but the best ensemble was around 0.998.

", "votes": null, "replies": [ { "id": 2228849, "author_name": "junseonglee11", "author_url": "", "post_date": "04/20/2023 21:43:18", "content": "

@manwithaflower additional preprocessing is as follows.

\n
    \n
  1. Used time interval (next time point - current time point) instead of time point (becuase for RNN, I felt that interval is more natural.)
    2. \n
  2. Added difference of sensor position of next and current time point (because I thought that it may contain direction information)
    3. \n
  3. Added line fit result of Robert Hatch's notebook and concatenated it to the layer after the last LSTM layer.
    4. \n
\n

Also, stacking 7 LSTM layers and using all data with TFRecord format were crucial.

\n

Thank you!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2227753": "Team members are junseonglee11 (@junseonglee11), Ayaan Jang(@ayaanjang). \nWe ensembled 6 LSTM models (2 different versions).\nWe modified Robin Smith's and Robert Hatch's notebooks. \n \n# **Our notebooks:**\nInference: https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged\nTrain: https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train\nDataset (TFRecord): https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1 \n\n# **References**\nRobin Smith's: notebooks:\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-inference\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-training-tpu\nhttps://www.kaggle.com/code/rsmits/tensorflow-lstm-model-data-preprocessor/notebook I modified his notebook \n \nRobert Hatch's notebook\nhttps://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars\nIt was crucial to improving our score. Used the results of this notebook as additional inputs in our model.\n\nSeungmoklee's notebook\nhttps://www.kaggle.com/code/seungmoklee/lstm-preprocessing-point-picker\n\n# **Data preprocessing**\n**Our code:** [https://www.kaggle.com/code/junseonglee11/icecube-data-to-tfrecord-v2-1]\n1. Reference part: Preprocessed data to generate 96 time-series data with 6 features including:\n - Sensor signal measurement time\n - Sensor signal strength\n - Sensor signal quality\n - X, Y, Z coordinates of received sensor (3 features)\n\n2. Performed feature engineering on the 6 features to improve the prediction accuracy of the RNN (Residual Neural Network) model (trained on 90 data files instead of the entire dataset, then experimented with feature transformation)\n - **Using original features: LB 1.015**\n - **Adding Time Difference: LB 1.0128**\n - **Replaced sensor signal measurement time with time interval (time difference between next and current measurement time, Time diff):** \n The difference in sensor position between current and next time points has different meanings depending on the time interval (for example, \n moving 10m in 1 second vs. 1m in 10 seconds has a 10-fold speed difference). However, the absolute measurement time of the sensor signal \n can not reflect this, so it was determined that time interval is a more appropriate input than the measurement time.\n - **Adding Coordinate Difference: LB 1.0115** \n **Added three features with the difference in X, Y, Z coordinates between the next and current sensor positions:** Similar to the time \n interval feature, it was determined that the difference in coordinate values between time points can better reflect the direction information of \n the neutral particle. However, when replacing the sensor's xyz coordinates with coordinate difference values, the accuracy decreased. This is \n because the difference in sensor coordinate values between non-adjacent time points has important information.\n3. Convert all the inputs above to TensorFlow TFRecord format to minimize CPU memory usage and accelerate training.\n\n\n# **Model training and inference**\n**Our code (model):** https://www.kaggle.com/code/junseonglee11/20th-tensorflow-tfrecord-tpu-lstm-line-fit-train\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7429668%2Feea4665a122c77a5c2e3aa60a38e65b1%2F.png?generation=1683061023145992&alt=media)\n\n- **Model Overview**: Bidirectional LSTM (long short-term memory) 7 layer+ batch normalization + Line fitting features concatenation + Dense layer\n- **Activation Function**: GELU(Gaussian Error Linear Unit) activation layer\n- **Loss Function**: Sparse_categorical_crossentropy loss\n- **Training Metric**: Accuracy\n- **Optimizer:** RAdam (Rectified Adam) optimizer\n- **Ensemble (2 versions of the model)**\n- Model trained on fold 0 of the dataset divided into 10 folds\n - Train: 594 dataset, Valid: 66 dataset: model trained using only 1 validation dataset\n - Train: 659 dataset, Valid: 1 dataset: four models with the lowest Mean Angular Error values are selected from each version of the model (total of 8 models)\n - Optimal ensemble weights are determined through cross-validation\n\n\n# **Data postprocessing**\n **Our code (Data postprocessing):** [https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged](https://www.kaggle.com/code/ayaanjang/20th-tensorflow-lstm-model-inference-merged)\n**Original code:** Power value not squared in the code below.\n Perform a weighted average of predicted probabilities for each category and the direction of the particle represented by each category to \n calculate the azimuth and zenith directions of a neutron.\n**Changed code:** Add a square of the power value to the predicted value obtained from the model.\n It's not optimal to simply derive the results based on the direction for each category and the predicted model probability.\n Attempt various modifications to the predicted probability using exponential, logarithmic functions, activation functions, etc. to improve the \n post-processing stage.\n**Improvement:** When the model-predicted category probabilities were squared by 1.35, there was a decrease of about 0.002 in mean angular error.\n→ This gave appropriate additional weight to the category with a high probability in the predicted result.", "2228452": "junseonglee11 nice to see, that you pushed LSTMs that high. Could you explain a bit more this item: *Additional inputs and some preprocessing*? I tried many experiments with LSTMs, but the best ensemble was around 0.998.", "2228849": "manwithaflower additional preprocessing is as follows.\n1. Used time interval (next time point - current time point) instead of time point (becuase for RNN, I felt that interval is more natural.)\n2. Added difference of sensor position of next and current time point (because I thought that it may contain direction information)\n3. Added line fit result of Robert Hatch's notebook and concatenated it to the layer after the last LSTM layer.\n\nAlso, stacking 7 LSTM layers and using all data with TFRecord format were crucial.\n\nThank you!" }, "source": "meta" }