{ "id": 403153, "title": "6th solution ", "url": "/competitions/icecube-neutrinos-in-deep-ice/writeups/oaf-6th-solution", "author_name": "", "post_date": "2023-04-21T15:37:12.787Z", "votes": 13, "comment_count": 2, "views": 0, "content": "

inference(normal ensemble,0.978):https://www.kaggle.com/code/chihantsai/ens-gnn-v4-5-2-and-lstm-v8-and-tformer-v18-up300
\ninference(group ensemble,0.975):https://www.kaggle.com/code/chihantsai/ens-group-gnn-v4-5-2-and-lstm-v8-and-tformer-v17

\n

GNN part :

\n

https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880

\n

Transformer part :

\n

https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880#2227949

\n

LSTM:

\n

Based on LSTM model by @rsmits.
\nloss :
\nCustom cross-entropy loss can be enhanced for regression tasks by adding a nearby area loss to improve training. The weights of the nearby loss used are shown in the figure below, inspired by this.

\n

\"weights
\nWe believe that our custom cross-entropy loss can cover a larger area than the original cross-entropy loss, which enables us to divide the spheres into more classes
\nbin_num : 48 (Spheres are classified into 48*48 classes)
\npulse_count(the number of pulses selected per event) : 192 and 300
\noptimizer: AdamW
\ntrain step:
\nStep 1: set the learning rate to 1e-3 and use pulse_count 192 to train the dataset for 3 rounds
\nStep 2: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 4 rounds
\nStep 3: use the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 3 rounds
\nStep 4: use the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 300

\n

SAKT(LSTM CNN transformer) :

\n

(from https://www.kaggle.com/code/shujun717/1-solution-lstm-cnn-transformer-1-fold/notebook)
\nloss : as same as LSTM model
\nbin_num : 48 (Spheres are classified into 48 classes)
\npulse_count(the number of pulses selected per event) : 192
\noptimizer: AdamW
\ntrain step:
\nStep 1:use learning rate to 1e-3 , train the dataset for 3 rounds
\nStep 2:apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration) , train the dataset for 4 rounds
\nStep 3: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration)

\n

Ensemble

\n

Validation score of the model in batch 657 658 659
\nLstm : 0.9899
\nSakt : 0.9927
\nGnn : 0.9927
\nTransformer : 0.9869

\n

Normal ensemble :

\n

validation score : 0.9803
\nlb : 0.978

\n

Group ensemble :

\n

I used the results of the line-fit and metadata to classify the events into 6 groups

\n
group1 :  the number of pulses  aux= >=   zenith(prediction by line-fit) >= pi/ (~%) \ngroup2 :  the number of pulses  aux= >=   zenith(prediction by line-fit) < pi/ (~%)\ngroup3: the number of pulses  aux= <  , zenith(prediction by line-fit) >= pi/  the pulses of aux= are  on a vertical string (~%)\ngroup4 : the number of pulses  aux= <  , zenith(prediction by line-fit) < pi/  the pulses of aux= are  on a vertical string (~%)\ngroup5 : the number of pulses  aux= <  , zenith(prediction by line-fit) >= pi/  the pulses of aux= are  on multiple strings (~%)\ngroup6 : the number of pulses  aux= <  , zenith(prediction by line-fit) < pi/  the pulses of aux= are  on multiple strings (~%)\n
\n

\"Validation

\n

we can observe that the validation scores of different groups vary considerably, and each model performs better on different groups. This is logical since atmospheric muons always travel from top to bottom, so the zenith (predicted by line-fit) < pi/2 is prone to more noise. Moreover, pulses of aux=False located on a vertical string have a higher probability of containing noise. Additionally, events with more pulses are better predicted.

\n

To enhance our results, we used different ensemble weights for each group, leading to an improvement of 0.003 in both validation score and lb (0.975). We further divided the events into 13 groups based on additional criteria, resulting in a gain of 0.001 in both validation score and lb (0.974, private score 0.975).

\n

Perhaps using an MLP for ensemble could get better results, but unfortunately, we did not have enough time to investigate this approach.

\n

P.S. My English is not very fluent, so please let me know if my explanation is not clear enough.

", "messages": [ { "id": "2229603", "postDate": "04/21/2023 14:05:32", "content": "

inference(normal ensemble,0.978):https://www.kaggle.com/code/chihantsai/ens-gnn-v4-5-2-and-lstm-v8-and-tformer-v18-up300
\ninference(group ensemble,0.975):https://www.kaggle.com/code/chihantsai/ens-group-gnn-v4-5-2-and-lstm-v8-and-tformer-v17

\n

GNN part :

\n

https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880

\n

Transformer part :

\n

https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880#2227949

\n

LSTM:

\n

Based on LSTM model by @rsmits.
\nloss :
\nCustom cross-entropy loss can be enhanced for regression tasks by adding a nearby area loss to improve training. The weights of the nearby loss used are shown in the figure below, inspired by this.

\n

\"weights
\nWe believe that our custom cross-entropy loss can cover a larger area than the original cross-entropy loss, which enables us to divide the spheres into more classes
\nbin_num : 48 (Spheres are classified into 48*48 classes)
\npulse_count(the number of pulses selected per event) : 192 and 300
\noptimizer: AdamW
\ntrain step:
\nStep 1: set the learning rate to 1e-3 and use pulse_count 192 to train the dataset for 3 rounds
\nStep 2: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 4 rounds
\nStep 3: use the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 3 rounds
\nStep 4: use the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 300

\n

SAKT(LSTM CNN transformer) :

\n

(from https://www.kaggle.com/code/shujun717/1-solution-lstm-cnn-transformer-1-fold/notebook)
\nloss : as same as LSTM model
\nbin_num : 48 (Spheres are classified into 48 classes)
\npulse_count(the number of pulses selected per event) : 192
\noptimizer: AdamW
\ntrain step:
\nStep 1:use learning rate to 1e-3 , train the dataset for 3 rounds
\nStep 2:apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration) , train the dataset for 4 rounds
\nStep 3: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration)

\n

Ensemble

\n

Validation score of the model in batch 657 658 659
\nLstm : 0.9899
\nSakt : 0.9927
\nGnn : 0.9927
\nTransformer : 0.9869

\n

Normal ensemble :

\n

validation score : 0.9803
\nlb : 0.978

\n

Group ensemble :

\n

I used the results of the line-fit and metadata to classify the events into 6 groups

\n
group1 :  the number of pulses  aux= >=   zenith(prediction by line-fit) >= pi/ (~%) \ngroup2 :  the number of pulses  aux= >=   zenith(prediction by line-fit) < pi/ (~%)\ngroup3: the number of pulses  aux= <  , zenith(prediction by line-fit) >= pi/  the pulses of aux= are  on a vertical string (~%)\ngroup4 : the number of pulses  aux= <  , zenith(prediction by line-fit) < pi/  the pulses of aux= are  on a vertical string (~%)\ngroup5 : the number of pulses  aux= <  , zenith(prediction by line-fit) >= pi/  the pulses of aux= are  on multiple strings (~%)\ngroup6 : the number of pulses  aux= <  , zenith(prediction by line-fit) < pi/  the pulses of aux= are  on multiple strings (~%)\n
\n

\"Validation

\n

we can observe that the validation scores of different groups vary considerably, and each model performs better on different groups. This is logical since atmospheric muons always travel from top to bottom, so the zenith (predicted by line-fit) < pi/2 is prone to more noise. Moreover, pulses of aux=False located on a vertical string have a higher probability of containing noise. Additionally, events with more pulses are better predicted.

\n

To enhance our results, we used different ensemble weights for each group, leading to an improvement of 0.003 in both validation score and lb (0.975). We further divided the events into 13 groups based on additional criteria, resulting in a gain of 0.001 in both validation score and lb (0.974, private score 0.975).

\n

Perhaps using an MLP for ensemble could get better results, but unfortunately, we did not have enough time to investigate this approach.

\n

P.S. My English is not very fluent, so please let me know if my explanation is not clear enough.

", "rawMarkdown": "**inference(normal ensemble,0.978)**:https://www.kaggle.com/code/chihantsai/ens-gnn-v4-5-2-and-lstm-v8-and-tformer-v18-up300 \n**inference(group ensemble,0.975):**https://www.kaggle.com/code/chihantsai/ens-group-gnn-v4-5-2-and-lstm-v8-and-tformer-v17 \n\n### **GNN part** : \nhttps://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880\n### **Transformer part** : \nhttps://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880#2227949\n\n\n### **LSTM**:\nBased on [LSTM model](https://www.kaggle.com/code/rsmits/tensorflow-lstm-model-training-tpu) by @rsmits.\nloss : \nCustom cross-entropy loss can be enhanced for regression tasks by adding a nearby area loss to improve training. The weights of the nearby loss used are shown in the figure below, inspired by [this](https://www.kaggle.com/competitions/ventilator-pressure-prediction/discussion/278362).\n\n![weights of nearly loss](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4184342%2F937cb92c0b9387a371379130f211eee4%2Fnearby_weight.png?generation=1682086516402080&alt=media)\nWe believe that our custom cross-entropy loss can cover a larger area than the original cross-entropy loss, which enables us to divide the spheres into more classes\nbin_num : 48 (Spheres are classified into 48*48 classes)\npulse_count(the number of pulses selected per event) : 192 and 300\noptimizer: AdamW \ntrain step:\nStep 1: set the learning rate to 1e-3 and use pulse_count 192 to train the dataset for 3 rounds\nStep 2: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 4 rounds\nStep 3: use the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 3 rounds\nStep 4: use the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 300\n \n### **SAKT**(LSTM CNN transformer) : \n(from https://www.kaggle.com/code/shujun717/1-solution-lstm-cnn-transformer-1-fold/notebook)\nloss : as same as LSTM model\nbin_num : 48 (Spheres are classified into 48 classes)\npulse_count(the number of pulses selected per event) : 192 \noptimizer: AdamW \ntrain step:\nStep 1:use learning rate to 1e-3 , train the dataset for 3 rounds\nStep 2:apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration) , train the dataset for 4 rounds\nStep 3: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration)\n\n# **Ensemble**\nValidation score of the model in batch 657 658 659\nLstm : 0.9899\nSakt : 0.9927\nGnn : 0.9927\nTransformer : 0.9869\n\n## **Normal ensemble** :\nvalidation score : 0.9803\nlb : 0.978\n\n## **Group ensemble** :\nI used the results of the [line-fit](https://www.kaggle.com/code/solverworld/icecube-neutrino-path-least-squares-1-214) and metadata to classify the events into 6 groups \n\n```python\ngroup1 : the number of pulses for aux=False >= 300 and zenith(prediction by line-fit) >= pi/2 (~1%) \ngroup2 : the number of pulses for aux=False >= 300 and zenith(prediction by line-fit) < pi/2 (~1.58%)\ngroup3: the number of pulses for aux=False < 300 , zenith(prediction by line-fit) >= pi/2 and the pulses of aux=False are all on a vertical string (~6.03%)\ngroup4 : the number of pulses for aux=False < 300 , zenith(prediction by line-fit) < pi/2 and the pulses of aux=False are all on a vertical string (~11.34%)\ngroup5 : the number of pulses for aux=False < 300 , zenith(prediction by line-fit) >= pi/2 and the pulses of aux=False are all on multiple strings (~21.67%)\ngroup6 : the number of pulses for aux=False < 300 , zenith(prediction by line-fit) < pi/2 and the pulses of aux=False are all on multiple strings (~58.37%)\n```\n![Validation score](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4184342%2F48225ff16bf6c8a8bf69ef01029cc4e1%2FScreenshot%20from%202023-04-21%2022-04-42.png?generation=1682085909903995&alt=media)\n\n\n\nwe can observe that the validation scores of different groups vary considerably, and each model performs better on different groups. This is logical since atmospheric muons always travel from top to bottom, so the zenith (predicted by line-fit) < pi/2 is prone to more noise. Moreover, pulses of aux=False located on a vertical string have a higher probability of containing noise. Additionally, events with more pulses are better predicted.\n \nTo enhance our results, we used different ensemble weights for each group, leading to an improvement of 0.003 in both validation score and lb (0.975). We further divided the events into 13 groups based on additional criteria, resulting in a gain of 0.001 in both validation score and lb (0.974, private score 0.975).\n\nPerhaps using an MLP for ensemble could get better results, but unfortunately, we did not have enough time to investigate this approach.\n\nP.S. My English is not very fluent, so please let me know if my explanation is not clear enough.", "votes": null }, { "id": "2229718", "postDate": "04/21/2023 16:17:34", "content": "

Hi @fate Nice solution and a very good result!

\n

And don't worry about your English….it is good enough for me to understand your solution clearly!

", "rawMarkdown": "Hi @fate Nice solution and a very good result!\n\nAnd don't worry about your English....it is good enough for me to understand your solution clearly!", "votes": null }, { "id": "2229751", "postDate": "04/21/2023 16:41:58", "content": "

Congratulations on being awarded the Gold Medal! Your commitment and hard work have yielded fruitful results, and it is truly motivating to witness your attainment of this achievement.

", "rawMarkdown": "Congratulations on being awarded the Gold Medal! Your commitment and hard work have yielded fruitful results, and it is truly motivating to witness your attainment of this achievement.", "votes": null } ], "comments": [ { "id": 2229718, "author_name": "rsmits", "author_url": "", "post_date": "04/21/2023 16:17:34", "content": "

Hi @fate Nice solution and a very good result!

\n

And don't worry about your English….it is good enough for me to understand your solution clearly!

", "votes": null, "replies": [] }, { "id": 2229751, "author_name": "atom1231", "author_url": "", "post_date": "04/21/2023 16:41:58", "content": "

Congratulations on being awarded the Gold Medal! Your commitment and hard work have yielded fruitful results, and it is truly motivating to witness your attainment of this achievement.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2229603": "**inference(normal ensemble,0.978)**:https://www.kaggle.com/code/chihantsai/ens-gnn-v4-5-2-and-lstm-v8-and-tformer-v18-up300 \n**inference(group ensemble,0.975):**https://www.kaggle.com/code/chihantsai/ens-group-gnn-v4-5-2-and-lstm-v8-and-tformer-v17 \n\n### **GNN part** : \nhttps://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880\n### **Transformer part** : \nhttps://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402880#2227949\n\n\n### **LSTM**:\nBased on [LSTM model](https://www.kaggle.com/code/rsmits/tensorflow-lstm-model-training-tpu) by @rsmits.\nloss : \nCustom cross-entropy loss can be enhanced for regression tasks by adding a nearby area loss to improve training. The weights of the nearby loss used are shown in the figure below, inspired by [this](https://www.kaggle.com/competitions/ventilator-pressure-prediction/discussion/278362).\n\n![weights of nearly loss](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4184342%2F937cb92c0b9387a371379130f211eee4%2Fnearby_weight.png?generation=1682086516402080&alt=media)\nWe believe that our custom cross-entropy loss can cover a larger area than the original cross-entropy loss, which enables us to divide the spheres into more classes\nbin_num : 48 (Spheres are classified into 48*48 classes)\npulse_count(the number of pulses selected per event) : 192 and 300\noptimizer: AdamW \ntrain step:\nStep 1: set the learning rate to 1e-3 and use pulse_count 192 to train the dataset for 3 rounds\nStep 2: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 4 rounds\nStep 3: use the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration), and pulse_count 192. We train the dataset for 3 rounds\nStep 4: use the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration), and pulse_count 300\n \n### **SAKT**(LSTM CNN transformer) : \n(from https://www.kaggle.com/code/shujun717/1-solution-lstm-cnn-transformer-1-fold/notebook)\nloss : as same as LSTM model\nbin_num : 48 (Spheres are classified into 48 classes)\npulse_count(the number of pulses selected per event) : 192 \noptimizer: AdamW \ntrain step:\nStep 1:use learning rate to 1e-3 , train the dataset for 3 rounds\nStep 2:apply the CosineAnnealingLR scheduler with a learning rate range of 1e-4 to 1e-6, t_max=4003 (an epoch iteration) , train the dataset for 4 rounds\nStep 3: apply the CosineAnnealingLR scheduler with a learning rate range of 1e-5 to 1e-7, t_max=4003 (an epoch iteration)\n\n# **Ensemble**\nValidation score of the model in batch 657 658 659\nLstm : 0.9899\nSakt : 0.9927\nGnn : 0.9927\nTransformer : 0.9869\n\n## **Normal ensemble** :\nvalidation score : 0.9803\nlb : 0.978\n\n## **Group ensemble** :\nI used the results of the [line-fit](https://www.kaggle.com/code/solverworld/icecube-neutrino-path-least-squares-1-214) and metadata to classify the events into 6 groups \n\n```python\ngroup1 : the number of pulses for aux=False >= 300 and zenith(prediction by line-fit) >= pi/2 (~1%) \ngroup2 : the number of pulses for aux=False >= 300 and zenith(prediction by line-fit) < pi/2 (~1.58%)\ngroup3: the number of pulses for aux=False < 300 , zenith(prediction by line-fit) >= pi/2 and the pulses of aux=False are all on a vertical string (~6.03%)\ngroup4 : the number of pulses for aux=False < 300 , zenith(prediction by line-fit) < pi/2 and the pulses of aux=False are all on a vertical string (~11.34%)\ngroup5 : the number of pulses for aux=False < 300 , zenith(prediction by line-fit) >= pi/2 and the pulses of aux=False are all on multiple strings (~21.67%)\ngroup6 : the number of pulses for aux=False < 300 , zenith(prediction by line-fit) < pi/2 and the pulses of aux=False are all on multiple strings (~58.37%)\n```\n![Validation score](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4184342%2F48225ff16bf6c8a8bf69ef01029cc4e1%2FScreenshot%20from%202023-04-21%2022-04-42.png?generation=1682085909903995&alt=media)\n\n\n\nwe can observe that the validation scores of different groups vary considerably, and each model performs better on different groups. This is logical since atmospheric muons always travel from top to bottom, so the zenith (predicted by line-fit) < pi/2 is prone to more noise. Moreover, pulses of aux=False located on a vertical string have a higher probability of containing noise. Additionally, events with more pulses are better predicted.\n \nTo enhance our results, we used different ensemble weights for each group, leading to an improvement of 0.003 in both validation score and lb (0.975). We further divided the events into 13 groups based on additional criteria, resulting in a gain of 0.001 in both validation score and lb (0.974, private score 0.975).\n\nPerhaps using an MLP for ensemble could get better results, but unfortunately, we did not have enough time to investigate this approach.\n\nP.S. My English is not very fluent, so please let me know if my explanation is not clear enough.", "2229718": "Hi @fate Nice solution and a very good result!\n\nAnd don't worry about your English....it is good enough for me to understand your solution clearly!", "2229751": "Congratulations on being awarded the Gold Medal! Your commitment and hard work have yielded fruitful results, and it is truly motivating to witness your attainment of this achievement." }, "source": "meta" }