{
  "id": 209600,
  "title": "Private 2nd Public 1st rank Solution",
  "url": "/competitions/predict-volcanic-eruptions-ingv-oe/writeups/byungsunbae-private-2nd-public-1st-rank-solution",
  "author_name": "",
  "post_date": "2021-01-08T01:40:33.100Z",
  "votes": 28,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I had a really good experience thanks to the INGV competition. Thank you to the organization that hosted this event.</p>\n<p>When I made the solution (= model) in this contest, I went through the following sequence.</p>\n<p><strong>1. Data Preprocessing</strong></p>\n<ul>\n<li>NA -&gt; 0</li>\n<li>Normalizing Data per segment_id<ul>\n<li>Normalization was performed for each column. The execution method is: <br>\n1) Subtract the average to find the deviation.<br>\n2) Divide the deviation by the maximum of the absolute deviation value. Add 0.00001 to prevent the denominator from becoming zero.</li></ul></li>\n<li>Feature Extraction : I extract features from raw data with <code>librosa</code> module in python.<ul>\n<li>rms</li>\n<li>spectral_centroid</li>\n<li>spectral_bandwidth</li>\n<li>zero_crossing_rate</li>\n<li>mel frequency cepstral coefficient</li>\n<li>poly_features</li>\n<li>spectral_contrast</li>\n<li>chroma_stft after applying stft (or not)</li></ul></li>\n<li>My dataset is divided as follows depending on whether chroma_stft is applied or not.<ul>\n<li>First set : chroma_stft X, shape of data per segment_id : (Channel, Time) =&gt; (340, 118) </li>\n<li>Second set : chroma_stft O shape of data per segment_id : (Channel, Time) =&gt; (460, 118)</li></ul></li>\n<li>Save to numpy array per segment_id ( ex 212423.csv -&gt; 212423.npy )<ul>\n<li>It takes 3 hours to configure one set. ( Total 6 hours contain train and test )</li></ul></li>\n<li>In fact, I didn't really understand the Waveform dataset. If I had understood it much better, I would have been able to create a model with better performance than it is now.</li>\n</ul>\n<p><strong>2. Training Model</strong></p>\n<ul>\n<li>Model Architecture\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F505929%2F0b54666a68d49b435695714a429172e1%2F2021-01-08%20%2010.35.28.png?generation=1610069767443741&amp;alt=media\" alt=\"\"><ul>\n<li>Activation Function : Leaky ReLU except Bi-LSTM and FC layer</li>\n<li>From : <ul>\n<li><a href=\"https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py\" target=\"_blank\">https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py</a></li>\n<li><a href=\"https://amaarora.github.io/2020/07/24/SeNet.html\" target=\"_blank\">https://amaarora.github.io/2020/07/24/SeNet.html</a></li></ul></li></ul></li>\n<li>Using Stratified 5-Fold Cross Validation through K-Means based on train's Target <code>time-to-eruption</code><ul>\n<li>After doing K-Means with K=5, Stratified sampling was performed using sklearn's StratifiedKFold.</li>\n<li>I proceeded with the idea that the target values ​​were evenly distributed for each hold-out set.</li></ul></li>\n<li>Loss Function : LogCoshLoss<ul>\n<li>I also used Huber loss. but not better than LogCoshLoss</li>\n<li>From : <a href=\"https://github.com/tuantle/regression-losses-pytorch\" target=\"_blank\">https://github.com/tuantle/regression-losses-pytorch</a></li></ul></li>\n<li>Optimizing<ul>\n<li>AdamW with weight_decay 0.001</li>\n<li>Learning Rate Scheduler : CosineAnnealingWarmRestarts (period : 10 epochs)</li></ul></li>\n<li>I make 2 models with differences like below:<ul>\n<li>(1) First set =&gt; Epochs 1200, base lr = 0.005 ( First Model )</li>\n<li>(2) Second set =&gt; Epochs 1000, base lr = 0.004 ( Second Model )</li></ul></li>\n<li>Training models with different seed ( 5 different seed )</li>\n</ul>\n<p><strong>3. Blending Results</strong></p>\n<ul>\n<li>Simple Averaging two results.</li>\n</ul>\n<p>Everyone has a hard time. Thank you for reading.</p>",
  "messages": [
    {
      "id": "1143630",
      "postDate": "01/08/2021 01:21:41",
      "content": "<p>I had a really good experience thanks to the INGV competition. Thank you to the organization that hosted this event.</p>\n<p>When I made the solution (= model) in this contest, I went through the following sequence.</p>\n<p><strong>1. Data Preprocessing</strong></p>\n<ul>\n<li>NA -&gt; 0</li>\n<li>Normalizing Data per segment_id<ul>\n<li>Normalization was performed for each column. The execution method is: <br>\n1) Subtract the average to find the deviation.<br>\n2) Divide the deviation by the maximum of the absolute deviation value. Add 0.00001 to prevent the denominator from becoming zero.</li></ul></li>\n<li>Feature Extraction : I extract features from raw data with <code>librosa</code> module in python.<ul>\n<li>rms</li>\n<li>spectral_centroid</li>\n<li>spectral_bandwidth</li>\n<li>zero_crossing_rate</li>\n<li>mel frequency cepstral coefficient</li>\n<li>poly_features</li>\n<li>spectral_contrast</li>\n<li>chroma_stft after applying stft (or not)</li></ul></li>\n<li>My dataset is divided as follows depending on whether chroma_stft is applied or not.<ul>\n<li>First set : chroma_stft X, shape of data per segment_id : (Channel, Time) =&gt; (340, 118) </li>\n<li>Second set : chroma_stft O shape of data per segment_id : (Channel, Time) =&gt; (460, 118)</li></ul></li>\n<li>Save to numpy array per segment_id ( ex 212423.csv -&gt; 212423.npy )<ul>\n<li>It takes 3 hours to configure one set. ( Total 6 hours contain train and test )</li></ul></li>\n<li>In fact, I didn't really understand the Waveform dataset. If I had understood it much better, I would have been able to create a model with better performance than it is now.</li>\n</ul>\n<p><strong>2. Training Model</strong></p>\n<ul>\n<li>Model Architecture\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F505929%2F0b54666a68d49b435695714a429172e1%2F2021-01-08%20%2010.35.28.png?generation=1610069767443741&amp;alt=media\" alt=\"\"><ul>\n<li>Activation Function : Leaky ReLU except Bi-LSTM and FC layer</li>\n<li>From : <ul>\n<li><a href=\"https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py\" target=\"_blank\">https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py</a></li>\n<li><a href=\"https://amaarora.github.io/2020/07/24/SeNet.html\" target=\"_blank\">https://amaarora.github.io/2020/07/24/SeNet.html</a></li></ul></li></ul></li>\n<li>Using Stratified 5-Fold Cross Validation through K-Means based on train's Target <code>time-to-eruption</code><ul>\n<li>After doing K-Means with K=5, Stratified sampling was performed using sklearn's StratifiedKFold.</li>\n<li>I proceeded with the idea that the target values ​​were evenly distributed for each hold-out set.</li></ul></li>\n<li>Loss Function : LogCoshLoss<ul>\n<li>I also used Huber loss. but not better than LogCoshLoss</li>\n<li>From : <a href=\"https://github.com/tuantle/regression-losses-pytorch\" target=\"_blank\">https://github.com/tuantle/regression-losses-pytorch</a></li></ul></li>\n<li>Optimizing<ul>\n<li>AdamW with weight_decay 0.001</li>\n<li>Learning Rate Scheduler : CosineAnnealingWarmRestarts (period : 10 epochs)</li></ul></li>\n<li>I make 2 models with differences like below:<ul>\n<li>(1) First set =&gt; Epochs 1200, base lr = 0.005 ( First Model )</li>\n<li>(2) Second set =&gt; Epochs 1000, base lr = 0.004 ( Second Model )</li></ul></li>\n<li>Training models with different seed ( 5 different seed )</li>\n</ul>\n<p><strong>3. Blending Results</strong></p>\n<ul>\n<li>Simple Averaging two results.</li>\n</ul>\n<p>Everyone has a hard time. Thank you for reading.</p>",
      "rawMarkdown": "I had a really good experience thanks to the INGV competition. Thank you to the organization that hosted this event.\n\nWhen I made the solution (= model) in this contest, I went through the following sequence.\n\n**1. Data Preprocessing**\n  - NA -> 0\n  - Normalizing Data per segment_id\n     + Normalization was performed for each column. The execution method is: \n        1) Subtract the average to find the deviation.\n        2) Divide the deviation by the maximum of the absolute deviation value. Add 0.00001 to prevent the denominator from becoming zero.\n  - Feature Extraction : I extract features from raw data with `librosa` module in python.\n    + rms\n    + spectral_centroid\n    + spectral_bandwidth\n    + zero_crossing_rate\n    + mel frequency cepstral coefficient\n    + poly_features\n    + spectral_contrast\n    + chroma_stft after applying stft (or not)\n  - My dataset is divided as follows depending on whether chroma_stft is applied or not.\n    + First set : chroma_stft X, shape of data per segment_id : (Channel, Time) => (340, 118) \n    + Second set : chroma_stft O shape of data per segment_id : (Channel, Time) => (460, 118)\n  - Save to numpy array per segment_id ( ex 212423.csv -> 212423.npy )\n    + It takes 3 hours to configure one set. ( Total 6 hours contain train and test )\n  - In fact, I didn't really understand the Waveform dataset. If I had understood it much better, I would have been able to create a model with better performance than it is now.\n\n**2. Training Model**\n  - Model Architecture\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F505929%2F0b54666a68d49b435695714a429172e1%2F2021-01-08%20%2010.35.28.png?generation=1610069767443741&alt=media)\n    + Activation Function : Leaky ReLU except Bi-LSTM and FC layer\n    + From : \n       + https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py\n       + https://amaarora.github.io/2020/07/24/SeNet.html\n  - Using Stratified 5-Fold Cross Validation through K-Means based on train's Target `time-to-eruption`\n    + After doing K-Means with K=5, Stratified sampling was performed using sklearn's StratifiedKFold.\n    + I proceeded with the idea that the target values ​​were evenly distributed for each hold-out set.\n  - Loss Function : LogCoshLoss\n    + I also used Huber loss. but not better than LogCoshLoss\n    + From : https://github.com/tuantle/regression-losses-pytorch\n  - Optimizing\n    + AdamW with weight_decay 0.001\n    + Learning Rate Scheduler : CosineAnnealingWarmRestarts (period : 10 epochs)\n  - I make 2 models with differences like below:\n    + (1) First set => Epochs 1200, base lr = 0.005 ( First Model )\n    + (2) Second set => Epochs 1000, base lr = 0.004 ( Second Model )\n  - Training models with different seed ( 5 different seed )\n\n**3. Blending Results**\n  - Simple Averaging two results.\n\nEveryone has a hard time. Thank you for reading.",
      "votes": null
    },
    {
      "id": "1143852",
      "postDate": "01/08/2021 05:17:05",
      "content": "<p>good nn solution, thanks for your sharing</p>",
      "rawMarkdown": "good nn solution, thanks for your sharing",
      "votes": null
    },
    {
      "id": "1144163",
      "postDate": "01/08/2021 09:18:18",
      "content": "<p>Great work. Congrats on the rank and thank you for sharing. </p>",
      "rawMarkdown": "Great work. Congrats on the rank and thank you for sharing.",
      "votes": null
    },
    {
      "id": "1147096",
      "postDate": "01/10/2021 09:33:57",
      "content": "<p>thx for sharing. look forward to understanding and learning from your solution when i have a moment.</p>",
      "rawMarkdown": "thx for sharing. look forward to understanding and learning from your solution when i have a moment.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1143852,
      "author_name": "laplaceplanet",
      "author_url": "",
      "post_date": "01/08/2021 05:17:05",
      "content": "<p>good nn solution, thanks for your sharing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1144163,
      "author_name": "obougacha",
      "author_url": "",
      "post_date": "01/08/2021 09:18:18",
      "content": "<p>Great work. Congrats on the rank and thank you for sharing. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1147096,
      "author_name": "davidedwards1",
      "author_url": "",
      "post_date": "01/10/2021 09:33:57",
      "content": "<p>thx for sharing. look forward to understanding and learning from your solution when i have a moment.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1143630": "I had a really good experience thanks to the INGV competition. Thank you to the organization that hosted this event.\n\nWhen I made the solution (= model) in this contest, I went through the following sequence.\n\n**1. Data Preprocessing**\n  - NA -> 0\n  - Normalizing Data per segment_id\n     + Normalization was performed for each column. The execution method is: \n        1) Subtract the average to find the deviation.\n        2) Divide the deviation by the maximum of the absolute deviation value. Add 0.00001 to prevent the denominator from becoming zero.\n  - Feature Extraction : I extract features from raw data with `librosa` module in python.\n    + rms\n    + spectral_centroid\n    + spectral_bandwidth\n    + zero_crossing_rate\n    + mel frequency cepstral coefficient\n    + poly_features\n    + spectral_contrast\n    + chroma_stft after applying stft (or not)\n  - My dataset is divided as follows depending on whether chroma_stft is applied or not.\n    + First set : chroma_stft X, shape of data per segment_id : (Channel, Time) => (340, 118) \n    + Second set : chroma_stft O shape of data per segment_id : (Channel, Time) => (460, 118)\n  - Save to numpy array per segment_id ( ex 212423.csv -> 212423.npy )\n    + It takes 3 hours to configure one set. ( Total 6 hours contain train and test )\n  - In fact, I didn't really understand the Waveform dataset. If I had understood it much better, I would have been able to create a model with better performance than it is now.\n\n**2. Training Model**\n  - Model Architecture\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F505929%2F0b54666a68d49b435695714a429172e1%2F2021-01-08%20%2010.35.28.png?generation=1610069767443741&alt=media)\n    + Activation Function : Leaky ReLU except Bi-LSTM and FC layer\n    + From : \n       + https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py\n       + https://amaarora.github.io/2020/07/24/SeNet.html\n  - Using Stratified 5-Fold Cross Validation through K-Means based on train's Target `time-to-eruption`\n    + After doing K-Means with K=5, Stratified sampling was performed using sklearn's StratifiedKFold.\n    + I proceeded with the idea that the target values ​​were evenly distributed for each hold-out set.\n  - Loss Function : LogCoshLoss\n    + I also used Huber loss. but not better than LogCoshLoss\n    + From : https://github.com/tuantle/regression-losses-pytorch\n  - Optimizing\n    + AdamW with weight_decay 0.001\n    + Learning Rate Scheduler : CosineAnnealingWarmRestarts (period : 10 epochs)\n  - I make 2 models with differences like below:\n    + (1) First set => Epochs 1200, base lr = 0.005 ( First Model )\n    + (2) Second set => Epochs 1000, base lr = 0.004 ( Second Model )\n  - Training models with different seed ( 5 different seed )\n\n**3. Blending Results**\n  - Simple Averaging two results.\n\nEveryone has a hard time. Thank you for reading.",
    "1143852": "good nn solution, thanks for your sharing",
    "1144163": "Great work. Congrats on the rank and thank you for sharing.",
    "1147096": "thx for sharing. look forward to understanding and learning from your solution when i have a moment."
  },
  "source": "meta"
}