{
  "id": 85258,
  "title": "9th Place Solution Overview",
  "url": "/competitions/vsb-power-line-fault-detection/discussion/85258",
  "author_name": "ngyope",
  "post_date": "2019-03-22T14:30:17.096000",
  "votes": 31,
  "comment_count": 25,
  "views": 0,
  "content": "<p>Congratulations to all the winners! \nAnd thanks a lot to VSB/Enet Center and Kaggle for this exciting competition. Since this is my first Kaggle competition, I'm surprised and glad to this honorable result.<br><br></p>\n\n<p>I'm going to briefly share all of you my 10th place solution, which keeps the score of 0.688 in the private LB. <br><br></p>\n\n<p>First of all, I have to tell you that this solution is not my best public LB model (RNN), which is in the end 13th place in public LB (0.766). <br>\nI think that the keys of this competition are high-pass filtering and DWT-denoising. In this point, I owe a lot to Jack's <a href=\"https://www.kaggle.com/jackvial/dwt-signal-denoising\">kernel</a>.<br><br></p>\n\n<p>As many other guys mentioned in public kernel and discussion, the problem of this competition is the discrepancy between train / test distributions(and the unstability of CV / LB resulted from that).\nSo, I chose as a final submission more <em>conservative</em> RNN model, which kept modest score in public LB but might be more robust in my view.<br><br></p>\n\n<p>In order to choose a more robust model, I spent time to decrease val_auc score in adversarial validation.\nI'm going to show what I thought and tried as below.<br><br></p>\n\n<ol>\n<li>Without any filtering / denoising, the discrepancy between train / test distributions was huge (val-auc was over 0.96 in adversarial validation. So, I thought that a model without any preprocessing was in danger of shake.</li>\n<li>With high-pass filtered and DWT-denoised, val-auc decreased to under 0.78. Therefore, I selected high-pass filtered and DWT-denoised model as a base one (high-pass filtering had the stronger influence).</li>\n<li>Next, I cut down features which contributed to increase val-auc in adversarial validation, and finally succeeded to decrease val_auc to under 0.7. (Notwithstanding, the instability of CV / LB was not completely solved...)<br><br></li>\n</ol>\n\n<p>The architecture of my RNN model is simple and not novel. \nSimilar to other guys, I selected <a href=\"https://www.kaggle.com/braquino/5-fold-lstm-attention-fully-commented-0-694\">Bruno-based</a> BiLSTM x 2 + Attention model and <a href=\"https://www.kaggle.com/tarunpaparaju/vsb-competition-attention-bilstm-with-features\">Tarun-based</a> BiLSTM x 2 + Attention + feature concatenation model.<br><br></p>\n\n<p>And I hard-voted 8 predictions (each resulted from stratified 5- or 4-fold CV models) which were a little bit different from each other in terms of features and random seeds.\nWhen I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).</p>\n\n<p>Moreover, I used recently released AdaBound optimizer. Although It contributed to increase local CV score, I don't know it is a good choice especially in this easily-overfitting competition.</p>",
  "messages": [
    {
      "id": 496742,
      "postDate": "2019-03-22T14:30:17.097Z",
      "content": "<p>Congratulations to all the winners! \nAnd thanks a lot to VSB/Enet Center and Kaggle for this exciting competition. Since this is my first Kaggle competition, I'm surprised and glad to this honorable result.<br><br></p>\n\n<p>I'm going to briefly share all of you my 10th place solution, which keeps the score of 0.688 in the private LB. <br><br></p>\n\n<p>First of all, I have to tell you that this solution is not my best public LB model (RNN), which is in the end 13th place in public LB (0.766). <br>\nI think that the keys of this competition are high-pass filtering and DWT-denoising. In this point, I owe a lot to Jack's <a href=\"https://www.kaggle.com/jackvial/dwt-signal-denoising\">kernel</a>.<br><br></p>\n\n<p>As many other guys mentioned in public kernel and discussion, the problem of this competition is the discrepancy between train / test distributions(and the unstability of CV / LB resulted from that).\nSo, I chose as a final submission more <em>conservative</em> RNN model, which kept modest score in public LB but might be more robust in my view.<br><br></p>\n\n<p>In order to choose a more robust model, I spent time to decrease val_auc score in adversarial validation.\nI'm going to show what I thought and tried as below.<br><br></p>\n\n<ol>\n<li>Without any filtering / denoising, the discrepancy between train / test distributions was huge (val-auc was over 0.96 in adversarial validation. So, I thought that a model without any preprocessing was in danger of shake.</li>\n<li>With high-pass filtered and DWT-denoised, val-auc decreased to under 0.78. Therefore, I selected high-pass filtered and DWT-denoised model as a base one (high-pass filtering had the stronger influence).</li>\n<li>Next, I cut down features which contributed to increase val-auc in adversarial validation, and finally succeeded to decrease val_auc to under 0.7. (Notwithstanding, the instability of CV / LB was not completely solved...)<br><br></li>\n</ol>\n\n<p>The architecture of my RNN model is simple and not novel. \nSimilar to other guys, I selected <a href=\"https://www.kaggle.com/braquino/5-fold-lstm-attention-fully-commented-0-694\">Bruno-based</a> BiLSTM x 2 + Attention model and <a href=\"https://www.kaggle.com/tarunpaparaju/vsb-competition-attention-bilstm-with-features\">Tarun-based</a> BiLSTM x 2 + Attention + feature concatenation model.<br><br></p>\n\n<p>And I hard-voted 8 predictions (each resulted from stratified 5- or 4-fold CV models) which were a little bit different from each other in terms of features and random seeds.\nWhen I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).</p>\n\n<p>Moreover, I used recently released AdaBound optimizer. Although It contributed to increase local CV score, I don't know it is a good choice especially in this easily-overfitting competition.</p>",
      "rawMarkdown": "Congratulations to all the winners! \nAnd thanks a lot to VSB/Enet Center and Kaggle for this exciting competition. Since this is my first Kaggle competition, I'm surprised and glad to this honorable result.<br><br>\n \nI'm going to briefly share all of you my 10th place solution, which keeps the score of 0.688 in the private LB. <br><br>\n\n\nFirst of all, I have to tell you that this solution is not my best public LB model (RNN), which is in the end 13th place in public LB (0.766).  \nI think that the keys of this competition are high-pass filtering and DWT-denoising. In this point, I owe a lot to Jack's [kernel](https://www.kaggle.com/jackvial/dwt-signal-denoising).<br><br>\n\nAs many other guys mentioned in public kernel and discussion, the problem of this competition is the discrepancy between train / test distributions(and the unstability of CV / LB resulted from that).\nSo, I chose as a final submission more *conservative* RNN model, which kept modest score in public LB but might be more robust in my view.<br><br>\n\n\nIn order to choose a more robust model, I spent time to decrease val_auc score in adversarial validation.\nI'm going to show what I thought and tried as below.<br><br>\n\n1. Without any filtering / denoising, the discrepancy between train / test distributions was huge (val-auc was over 0.96 in adversarial validation. So, I thought that a model without any preprocessing was in danger of shake.\n2. With high-pass filtered and DWT-denoised, val-auc decreased to under 0.78. Therefore, I selected high-pass filtered and DWT-denoised model as a base one (high-pass filtering had the stronger influence).\n3. Next, I cut down features which contributed to increase val-auc in adversarial validation, and finally succeeded to decrease val_auc to under 0.7. (Notwithstanding, the instability of CV / LB was not completely solved...)<br><br>\n\n\nThe architecture of my RNN model is simple and not novel. \nSimilar to other guys, I selected [Bruno-based](https://www.kaggle.com/braquino/5-fold-lstm-attention-fully-commented-0-694) BiLSTM x 2 + Attention model and [Tarun-based](https://www.kaggle.com/tarunpaparaju/vsb-competition-attention-bilstm-with-features) BiLSTM x 2 + Attention + feature concatenation model.<br><br>\n\nAnd I hard-voted 8 predictions (each resulted from stratified 5- or 4-fold CV models) which were a little bit different from each other in terms of features and random seeds.\nWhen I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).\n\nMoreover, I used recently released AdaBound optimizer. Although It contributed to increase local CV score, I don't know it is a good choice especially in this easily-overfitting competition.",
      "votes": 31
    },
    {
      "id": 498939,
      "postDate": "2019-03-24T04:19:03.750Z",
      "content": "<p>Hi again <a href=\"/yukinkgwa\">@yukinkgwa</a> , I would love to reimplement your approach next week, and I have a few more questions. It would be truly grateful if you can help a bit more : )</p>\n\n<p>1) Above, you mentioned that after filtered / denoised , you also removed some features in order to reduce the val_auc . Could you please give me examples what kind of features that you removed?</p>\n\n<p>2) Was the input to your Bi-LSTM has shape 2904 x 160 x 57 as in public kernel ?</p>\n\n<p>3) How did the ‘removed features’ in 1) affect the Bi-LSTM inputs in 2) ?</p>\n\n<p>4) Could you please let me know the final performance on <strong>public LB</strong> of your best private LB model (0.68837) ? </p>\n\n<p>Thank you again , and sorry for this many questions!</p>",
      "rawMarkdown": "Hi again @yukinkgwa , I would love to reimplement your approach next week, and I have a few more questions. It would be truly grateful if you can help a bit more : )\n\n1) Above, you mentioned that after filtered / denoised , you also removed some features in order to reduce the val_auc . Could you please give me examples what kind of features that you removed?\n\n2) Was the input to your Bi-LSTM has shape 2904 x 160 x 57 as in public kernel ?\n\n3) How did the ‘removed features’ in 1) affect the Bi-LSTM inputs in 2) ?\n\n4) Could you please let me know the final performance on **public LB** of your best private LB model (0.68837) ? \n\nThank you again , and sorry for this many questions!",
      "votes": 1,
      "replies": [
        {
          "id": 498971,
          "postDate": "2019-03-24T06:03:07.267Z",
          "content": "<p>Not at all, <a href=\"/ratthachat\">@ratthachat</a> ! Thank you for your question.\nMy answers are below.<br><br></p>\n\n<p>1)\nfeatures I used\n* mean\n* max_range\n* percentile (0, 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1)\n* relative_percentile (0, 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1)<br></p>\n\n<p>I added percentile 0.05 and 0.95 but they affected the val_auc little.<br><br></p>\n\n<p>And features below which Tarun provided us in a public kernel\n* perm_entropy\n* svd_entropy\n* app_entropy\n* sample_entropy\n* katz_fd\n* higuchi_fd<br></p>\n\n<p>Features which Tarun provided us affected effectively to decrease the val_auc score.<br><br></p>\n\n<p>features I didn't use\n* std\n* std_top\n* std_bot\n* petrosian_fd<br><br></p>\n\n<p>2)\nMy inputs are <br>\n* input_1: 2904 x 160 x 60(Bruno-based features)\n* input_2: 2904 x 6(Tarun-based features, I mean entropies and fds above.)<br></p>\n\n<p>I concatenated a and b below, and then input them to the Dense layers. \na: input_1 -&gt; BiLSTM -&gt; BiLSTM -&gt; Attention\nb: input_2<br><br></p>\n\n<p>3)\nSince I didn't have enough time and submission data to make an exact experiment how they affected to the scores, I simply didn't use the removed features.<br><br></p>\n\n<p>4)\nThe public LB score is 0.72213.</p>",
          "rawMarkdown": "Not at all, @ratthachat ! Thank you for your question.\nMy answers are below.<br><br>\n\n1)\nfeatures I used\n* mean\n* max_range\n* percentile (0, 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1)\n* relative_percentile (0, 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1)<br>\n\nI added percentile 0.05 and 0.95 but they affected the val\\_auc little.<br><br>\n\nAnd features below which Tarun provided us in a public kernel\n* perm_entropy\n* svd_entropy\n* app_entropy\n* sample_entropy\n* katz_fd\n* higuchi_fd<br>\n\nFeatures which Tarun provided us affected effectively to decrease the val\\_auc score.<br><br>\n\n\nfeatures I didn't use\n* std\n* std_top\n* std_bot\n* petrosian_fd<br><br>\n\n\n2)\nMy inputs are  \n* input_1: 2904 x 160 x 60(Bruno-based features)\n* input_2: 2904 x 6(Tarun-based features, I mean entropies and fds above.)<br>\n\nI concatenated a and b below, and then input them to the Dense layers. \na: input\\_1 -&gt; BiLSTM -&gt; BiLSTM -&gt; Attention\nb: input\\_2<br><br>\n\n\n3)\nSince I didn't have enough time and submission data to make an exact experiment how they affected to the scores, I simply didn't use the removed features.<br><br>\n\n\n4)\nThe public LB score is 0.72213.",
          "votes": 3
        },
        {
          "id": 498979,
          "postDate": "2019-03-24T06:18:55.960Z",
          "content": "<p><a href=\"/ratthachat\">@ratthachat</a>, can you share your code when you repeat this experiment?</p>",
          "rawMarkdown": "@ratthachat, can you share your code when you repeat this experiment?",
          "votes": 1
        },
        {
          "id": 499005,
          "postDate": "2019-03-24T07:18:53.590Z",
          "content": "<p>Thanks again <a href=\"/yukinkgwa\">@yukinkgwa</a> ! <a href=\"/sheriytm\">@sheriytm</a> YaGana, sure!, if I success in reproducing a good result, I will share it here.</p>",
          "rawMarkdown": "Thanks again @yukinkgwa ! @sheriytm YaGana, sure!, if I success in reproducing a good result, I will share it here."
        },
        {
          "id": 499084,
          "postDate": "2019-03-24T09:58:12.010Z",
          "content": "<p><a href=\"/yukinkgwa\">@yukinkgwa</a> I'm so happy to see that features from my kernel helped you win gold in this competition !\nCongratulations and looking forward to many more medals to come in the future !</p>",
          "rawMarkdown": "@yukinkgwa I'm so happy to see that features from my kernel helped you win gold in this competition !\nCongratulations and looking forward to many more medals to come in the future !",
          "votes": 1
        },
        {
          "id": 499085,
          "postDate": "2019-03-24T09:59:49.593Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 499086,
          "postDate": "2019-03-24T09:59:53.993Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 499101,
          "postDate": "2019-03-24T10:28:26.960Z",
          "content": "<p>Your kernel helps a lot of us here Tarun ;)</p>",
          "rawMarkdown": "Your kernel helps a lot of us here Tarun ;)",
          "votes": 1
        },
        {
          "id": 499121,
          "postDate": "2019-03-24T11:08:35.320Z",
          "content": "<p>Thanks <a href=\"/ratthachat\">@ratthachat</a> ! I hope I can keep sharing and helping participants in other competitions also !</p>",
          "rawMarkdown": "Thanks @ratthachat ! I hope I can keep sharing and helping participants in other competitions also !",
          "votes": 1
        },
        {
          "id": 499159,
          "postDate": "2019-03-24T12:14:16.287Z",
          "content": "<p>Thanks a lot, <a href=\"/tarunpaparaju\">@tarunpaparaju</a> !\nYou really made a great contribution for this competition!</p>",
          "rawMarkdown": "Thanks a lot, @tarunpaparaju !\nYou really made a great contribution for this competition!",
          "votes": 1
        },
        {
          "id": 499162,
          "postDate": "2019-03-24T12:17:39.127Z",
          "content": "<p>Thanks <a href=\"/yukinkgwa\">@yukinkgwa</a> ! It really means a lot to me !</p>",
          "rawMarkdown": "Thanks @yukinkgwa ! It really means a lot to me !"
        }
      ]
    },
    {
      "id": 496872,
      "postDate": "2019-03-22T17:31:28.363Z",
      "content": "<p>Congrats on the great finish and for sharing your approach. Great to see you were able to put my kernel to good use!</p>",
      "rawMarkdown": "Congrats on the great finish and for sharing your approach. Great to see you were able to put my kernel to good use!",
      "votes": 1,
      "replies": [
        {
          "id": 496911,
          "postDate": "2019-03-22T18:32:34.403Z",
          "content": "<p>Thanks, Jack.\n I learned a lot from your excellent kernel!</p>",
          "rawMarkdown": " Thanks, Jack.\n I learned a lot from your excellent kernel!",
          "votes": 1
        }
      ]
    },
    {
      "id": 496790,
      "postDate": "2019-03-22T15:29:01.680Z",
      "content": "<p>Could youe explaine \"When I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).\"</p>\n\n<p>So you mean choose the model with smaller CV gap?</p>",
      "rawMarkdown": "Could youe explaine \"When I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).\"\n\nSo you mean choose the model with smaller CV gap?",
      "votes": 1,
      "replies": [
        {
          "id": 496834,
          "postDate": "2019-03-22T16:31:49.803Z",
          "content": "<p>Thanks, Strideradu!<br><br></p>\n\n<p>I mean that I simply kept in mind to avoid the overfit to the train.\n(i.e. I didn't choose high local CV / low public LB models)</p>",
          "rawMarkdown": "Thanks, Strideradu!<br><br>\n\nI mean that I simply kept in mind to avoid the overfit to the train.\n(i.e. I didn't choose high local CV / low public LB models)",
          "votes": 3
        }
      ]
    },
    {
      "id": 497052,
      "postDate": "2019-03-22T22:28:16.673Z",
      "content": "<p>Congrats <a href=\"/yukinkgwa\">@yukinkgwa</a> on your solo gold medal and thanks for sharing.</p>\n\n<p><a href=\"https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146#496224\">https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146#496224</a></p>",
      "rawMarkdown": "Congrats @yukinkgwa on your solo gold medal and thanks for sharing.\n\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146#496224",
      "replies": [
        {
          "id": 497220,
          "postDate": "2019-03-23T06:37:01.517Z",
          "content": "<p>You too!  Thanks, YaGana.</p>",
          "rawMarkdown": "You too!  Thanks, YaGana."
        }
      ]
    },
    {
      "id": 497014,
      "postDate": "2019-03-22T20:49:41.693Z",
      "content": "<p>Congrats! Watched you rising in ranking, and stayed high after the shakeup!</p>\n\n<p>Interesting to see you \"conservative\" way of selecting submission.  Honestly, my submissions are all over the place, no clear indication what would help me to choose for private LB. My best private LB comes from a fairly complex TCN. </p>\n\n<p>And the public LB is a data set much larger than training set. </p>\n\n<p>Will investigate a little more, but it is Friday! TGIF~</p>",
      "rawMarkdown": "Congrats! Watched you rising in ranking, and stayed high after the shakeup!\n\nInteresting to see you \"conservative\" way of selecting submission.  Honestly, my submissions are all over the place, no clear indication what would help me to choose for private LB. My best private LB comes from a fairly complex TCN. \n\nAnd the public LB is a data set much larger than training set. \n\nWill investigate a little more, but it is Friday! TGIF~"
    },
    {
      "id": 496803,
      "postDate": "2019-03-22T15:44:36.287Z",
      "content": "<p>Congrats! Could you explain what are the hgih-pass filter and denoising parameters? Are they those default ones in the public kernel or modified somehow? And are the inputs to the adversarial validation and RNN the same? Thank you!</p>",
      "rawMarkdown": "Congrats! Could you explain what are the hgih-pass filter and denoising parameters? Are they those default ones in the public kernel or modified somehow? And are the inputs to the adversarial validation and RNN the same? Thank you!",
      "replies": [
        {
          "id": 496839,
          "postDate": "2019-03-22T16:37:57.383Z",
          "content": "<p>Thanks, lucaskg.<br></p>\n\n<p>As I'm not an expert to signal processing, I used just the same parameters as Jack's.\nAnd I used the same inputs and the same RNN architecture.</p>",
          "rawMarkdown": "Thanks, lucaskg.<br>\n\nAs I'm not an expert to signal processing, I used just the same parameters as Jack's.\nAnd I used the same inputs and the same RNN architecture."
        }
      ]
    },
    {
      "id": 496748,
      "postDate": "2019-03-22T14:44:32.473Z",
      "content": "<p>First of all, Congratulation! And thanks for sharing <a href=\"/yukinkgwa\">@yukinkgwa</a>, it is a very nice approach to know! </p>\n\n<p>Please allow me to ask some questions:\n1) So after you successfully decreased the val_auc to 0.7, how was your BiLSTM doing ?\nCould you please let us know CV vs. Public LB vs. Private LB ?</p>\n\n<p>2) And how is your best public LB (0.766) performing in CV / private?</p>\n\n<p>3) You said that you select a model with 'relatively small train/valid losses', so that mean it is not a usual keras callback 'save best weight due to a <code>valid_mcc</code>', right? How did you exactly decide to stop the training process?</p>",
      "rawMarkdown": "First of all, Congratulation! And thanks for sharing @yukinkgwa, it is a very nice approach to know! \n\nPlease allow me to ask some questions:\n1) So after you successfully decreased the val_auc to 0.7, how was your BiLSTM doing ?\nCould you please let us know CV vs. Public LB vs. Private LB ?\n\n2) And how is your best public LB (0.766) performing in CV / private?\n\n3) You said that you select a model with 'relatively small train/valid losses', so that mean it is not a usual keras callback 'save best weight due to a `valid_mcc`', right? How did you exactly decide to stop the training process?",
      "replies": [
        {
          "id": 496775,
          "postDate": "2019-03-22T15:12:41Z",
          "content": "<p>Thanks for your comment!<br><br></p>\n\n<p>Unfortunately, after I decreased val-auc to 0.7, the problem was not completely solved. \nI don't know whether this result is from the gap of neg/pos-ratio between train and test.<br><br></p>\n\n<p>But compared to un-filtered / un-denoised models, the gap (especially between public LB and private LB) seems to get small.<br><br></p>\n\n<p><strong>High-pass filtering and DWT-denoising</strong>\n<em>local cv: public lb: private lb</em>\n0.765: 0.688: 0.674\n0.728: 0.677: 0.673\n0.710: 0.677: 0.677<br><br></p>\n\n<p><strong>Without any filtering and denoising</strong>\n<em>local cv: public lb: private lb</em>\n0.753: 0.682: 0.611<br><br></p>\n\n<p>My best public LB (0.766) keeps 0.680 in private.\nSince this score is resulted from repetitive hard-votings, CV score is missing.</p>",
          "rawMarkdown": "Thanks for your comment!<br><br>\n\nUnfortunately, after I decreased val-auc to 0.7, the problem was not completely solved. \nI don't know whether this result is from the gap of neg/pos-ratio between train and test.<br><br>\n\nBut compared to un-filtered / un-denoised models, the gap (especially between public LB and private LB) seems to get small.<br><br>\n\n**High-pass filtering and DWT-denoising**\n*local cv: public lb: private lb*\n0.765: 0.688: 0.674\n0.728: 0.677: 0.673\n0.710: 0.677: 0.677<br><br>\n\n**Without any filtering and denoising**\n*local cv: public lb: private lb*\n0.753: 0.682: 0.611<br><br>\n\n\nMy best public LB (0.766) keeps 0.680 in private.\nSince this score is resulted from repetitive hard-votings, CV score is missing.",
          "votes": 2
        },
        {
          "id": 496797,
          "postDate": "2019-03-22T15:33:34.677Z",
          "content": "<p>Thank you so much!!</p>\n\n<p>By the way would you mind answer question 3) above? (I added the question later, so perhaps you missed it)</p>\n\n<blockquote>\n  <p>3) You said that you select a model with 'relatively small train/valid losses', so that mean it is not a usual keras callback 'save best weight due to a valid_mcc', right? How did you exactly decide to stop the training process?</p>\n</blockquote>",
          "rawMarkdown": "Thank you so much!!\n\nBy the way would you mind answer question 3) above? (I added the question later, so perhaps you missed it)\n\n&gt; 3) You said that you select a model with 'relatively small train/valid losses', so that mean it is not a usual keras callback 'save best weight due to a valid_mcc', right? How did you exactly decide to stop the training process?"
        },
        {
          "id": 496860,
          "postDate": "2019-03-22T17:13:15.870Z",
          "content": "<p>Oh, sorry for missing 3).<br><br></p>\n\n<p>I used keras ModelCheckpoint callbacks in a usual way. <em>(monitor=\"val_matthews_corr_coeff\", save_best_only=True, mode=\"max\")</em>\nWhat I mean above is that I tried to adjust the num of epochs so as to avoid the risk of the overfitting to the train.<br><br></p>\n\n<p>The more the num of epochs I set (I mean <em>thousands</em> of epochs instead of tens or hundreds), the higher the local CV score got. But simultaneously, the gap between train / validation losses widened, and the public LB of such model got lower.\nI adjusted the num of epochs lower than the threshold from which the gap began widening.</p>",
          "rawMarkdown": "Oh, sorry for missing 3).<br><br>\n\nI used keras ModelCheckpoint callbacks in a usual way. *(monitor=\"val\\_matthews\\_corr\\_coeff\", save\\_best\\_only=True, mode=\"max\")*\nWhat I mean above is that I tried to adjust the num of epochs so as to avoid the risk of the overfitting to the train.<br><br>\n\nThe more the num of epochs I set (I mean *thousands* of epochs instead of tens or hundreds), the higher the local CV score got. But simultaneously, the gap between train / validation losses widened, and the public LB of such model got lower.\nI adjusted the num of epochs lower than the threshold from which the gap began widening.",
          "votes": 1
        },
        {
          "id": 497109,
          "postDate": "2019-03-23T00:57:36.733Z",
          "content": "<p>Thank you for your kind answer!</p>",
          "rawMarkdown": "Thank you for your kind answer!"
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 498939,
      "author_name": "Neuron Engineer",
      "author_url": "",
      "post_date": "2019-03-24T04:19:03.750000",
      "content": "<p>Hi again <a href=\"/yukinkgwa\">@yukinkgwa</a> , I would love to reimplement your approach next week, and I have a few more questions. It would be truly grateful if you can help a bit more : )</p>\n\n<p>1) Above, you mentioned that after filtered / denoised , you also removed some features in order to reduce the val_auc . Could you please give me examples what kind of features that you removed?</p>\n\n<p>2) Was the input to your Bi-LSTM has shape 2904 x 160 x 57 as in public kernel ?</p>\n\n<p>3) How did the ‘removed features’ in 1) affect the Bi-LSTM inputs in 2) ?</p>\n\n<p>4) Could you please let me know the final performance on <strong>public LB</strong> of your best private LB model (0.68837) ? </p>\n\n<p>Thank you again , and sorry for this many questions!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 498971,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-24T06:03:07.267000",
          "content": "<p>Not at all, <a href=\"/ratthachat\">@ratthachat</a> ! Thank you for your question.\nMy answers are below.<br><br></p>\n\n<p>1)\nfeatures I used\n* mean\n* max_range\n* percentile (0, 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1)\n* relative_percentile (0, 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1)<br></p>\n\n<p>I added percentile 0.05 and 0.95 but they affected the val_auc little.<br><br></p>\n\n<p>And features below which Tarun provided us in a public kernel\n* perm_entropy\n* svd_entropy\n* app_entropy\n* sample_entropy\n* katz_fd\n* higuchi_fd<br></p>\n\n<p>Features which Tarun provided us affected effectively to decrease the val_auc score.<br><br></p>\n\n<p>features I didn't use\n* std\n* std_top\n* std_bot\n* petrosian_fd<br><br></p>\n\n<p>2)\nMy inputs are <br>\n* input_1: 2904 x 160 x 60(Bruno-based features)\n* input_2: 2904 x 6(Tarun-based features, I mean entropies and fds above.)<br></p>\n\n<p>I concatenated a and b below, and then input them to the Dense layers. \na: input_1 -&gt; BiLSTM -&gt; BiLSTM -&gt; Attention\nb: input_2<br><br></p>\n\n<p>3)\nSince I didn't have enough time and submission data to make an exact experiment how they affected to the scores, I simply didn't use the removed features.<br><br></p>\n\n<p>4)\nThe public LB score is 0.72213.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 498979,
          "author_name": "YaGana Sheriff-Hussaini",
          "author_url": "",
          "post_date": "2019-03-24T06:18:55.960000",
          "content": "<p><a href=\"/ratthachat\">@ratthachat</a>, can you share your code when you repeat this experiment?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 499005,
          "author_name": "Neuron Engineer",
          "author_url": "",
          "post_date": "2019-03-24T07:18:53.590000",
          "content": "<p>Thanks again <a href=\"/yukinkgwa\">@yukinkgwa</a> ! <a href=\"/sheriytm\">@sheriytm</a> YaGana, sure!, if I success in reproducing a good result, I will share it here.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 499084,
          "author_name": "Tarun Paparaju",
          "author_url": "",
          "post_date": "2019-03-24T09:58:12.010000",
          "content": "<p><a href=\"/yukinkgwa\">@yukinkgwa</a> I'm so happy to see that features from my kernel helped you win gold in this competition !\nCongratulations and looking forward to many more medals to come in the future !</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 499085,
          "author_name": "",
          "author_url": "",
          "post_date": "2019-03-24T09:59:49.593000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 499086,
          "author_name": "",
          "author_url": "",
          "post_date": "2019-03-24T09:59:53.993000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 499101,
          "author_name": "Neuron Engineer",
          "author_url": "",
          "post_date": "2019-03-24T10:28:26.960000",
          "content": "<p>Your kernel helps a lot of us here Tarun ;)</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 499121,
          "author_name": "Tarun Paparaju",
          "author_url": "",
          "post_date": "2019-03-24T11:08:35.320000",
          "content": "<p>Thanks <a href=\"/ratthachat\">@ratthachat</a> ! I hope I can keep sharing and helping participants in other competitions also !</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 499159,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-24T12:14:16.287000",
          "content": "<p>Thanks a lot, <a href=\"/tarunpaparaju\">@tarunpaparaju</a> !\nYou really made a great contribution for this competition!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 499162,
          "author_name": "Tarun Paparaju",
          "author_url": "",
          "post_date": "2019-03-24T12:17:39.127000",
          "content": "<p>Thanks <a href=\"/yukinkgwa\">@yukinkgwa</a> ! It really means a lot to me !</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 496872,
      "author_name": "Jack Vial",
      "author_url": "",
      "post_date": "2019-03-22T17:31:28.363000",
      "content": "<p>Congrats on the great finish and for sharing your approach. Great to see you were able to put my kernel to good use!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 496911,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-22T18:32:34.403000",
          "content": "<p>Thanks, Jack.\n I learned a lot from your excellent kernel!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 496790,
      "author_name": "Strideradu",
      "author_url": "",
      "post_date": "2019-03-22T15:29:01.680000",
      "content": "<p>Could youe explaine \"When I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).\"</p>\n\n<p>So you mean choose the model with smaller CV gap?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 496834,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-22T16:31:49.803000",
          "content": "<p>Thanks, Strideradu!<br><br></p>\n\n<p>I mean that I simply kept in mind to avoid the overfit to the train.\n(i.e. I didn't choose high local CV / low public LB models)</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 497052,
      "author_name": "YaGana Sheriff-Hussaini",
      "author_url": "",
      "post_date": "2019-03-22T22:28:16.673000",
      "content": "<p>Congrats <a href=\"/yukinkgwa\">@yukinkgwa</a> on your solo gold medal and thanks for sharing.</p>\n\n<p><a href=\"https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146#496224\">https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146#496224</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 497220,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-23T06:37:01.517000",
          "content": "<p>You too!  Thanks, YaGana.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 497014,
      "author_name": "Yunwei Hu",
      "author_url": "",
      "post_date": "2019-03-22T20:49:41.693000",
      "content": "<p>Congrats! Watched you rising in ranking, and stayed high after the shakeup!</p>\n\n<p>Interesting to see you \"conservative\" way of selecting submission.  Honestly, my submissions are all over the place, no clear indication what would help me to choose for private LB. My best private LB comes from a fairly complex TCN. </p>\n\n<p>And the public LB is a data set much larger than training set. </p>\n\n<p>Will investigate a little more, but it is Friday! TGIF~</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 496803,
      "author_name": "lucaskg",
      "author_url": "",
      "post_date": "2019-03-22T15:44:36.287000",
      "content": "<p>Congrats! Could you explain what are the hgih-pass filter and denoising parameters? Are they those default ones in the public kernel or modified somehow? And are the inputs to the adversarial validation and RNN the same? Thank you!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 496839,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-22T16:37:57.383000",
          "content": "<p>Thanks, lucaskg.<br></p>\n\n<p>As I'm not an expert to signal processing, I used just the same parameters as Jack's.\nAnd I used the same inputs and the same RNN architecture.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 496748,
      "author_name": "Neuron Engineer",
      "author_url": "",
      "post_date": "2019-03-22T14:44:32.473000",
      "content": "<p>First of all, Congratulation! And thanks for sharing <a href=\"/yukinkgwa\">@yukinkgwa</a>, it is a very nice approach to know! </p>\n\n<p>Please allow me to ask some questions:\n1) So after you successfully decreased the val_auc to 0.7, how was your BiLSTM doing ?\nCould you please let us know CV vs. Public LB vs. Private LB ?</p>\n\n<p>2) And how is your best public LB (0.766) performing in CV / private?</p>\n\n<p>3) You said that you select a model with 'relatively small train/valid losses', so that mean it is not a usual keras callback 'save best weight due to a <code>valid_mcc</code>', right? How did you exactly decide to stop the training process?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 496775,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-22T15:12:41",
          "content": "<p>Thanks for your comment!<br><br></p>\n\n<p>Unfortunately, after I decreased val-auc to 0.7, the problem was not completely solved. \nI don't know whether this result is from the gap of neg/pos-ratio between train and test.<br><br></p>\n\n<p>But compared to un-filtered / un-denoised models, the gap (especially between public LB and private LB) seems to get small.<br><br></p>\n\n<p><strong>High-pass filtering and DWT-denoising</strong>\n<em>local cv: public lb: private lb</em>\n0.765: 0.688: 0.674\n0.728: 0.677: 0.673\n0.710: 0.677: 0.677<br><br></p>\n\n<p><strong>Without any filtering and denoising</strong>\n<em>local cv: public lb: private lb</em>\n0.753: 0.682: 0.611<br><br></p>\n\n<p>My best public LB (0.766) keeps 0.680 in private.\nSince this score is resulted from repetitive hard-votings, CV score is missing.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 496797,
          "author_name": "Neuron Engineer",
          "author_url": "",
          "post_date": "2019-03-22T15:33:34.677000",
          "content": "<p>Thank you so much!!</p>\n\n<p>By the way would you mind answer question 3) above? (I added the question later, so perhaps you missed it)</p>\n\n<blockquote>\n  <p>3) You said that you select a model with 'relatively small train/valid losses', so that mean it is not a usual keras callback 'save best weight due to a valid_mcc', right? How did you exactly decide to stop the training process?</p>\n</blockquote>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 496860,
          "author_name": "ngyope",
          "author_url": "",
          "post_date": "2019-03-22T17:13:15.870000",
          "content": "<p>Oh, sorry for missing 3).<br><br></p>\n\n<p>I used keras ModelCheckpoint callbacks in a usual way. <em>(monitor=\"val_matthews_corr_coeff\", save_best_only=True, mode=\"max\")</em>\nWhat I mean above is that I tried to adjust the num of epochs so as to avoid the risk of the overfitting to the train.<br><br></p>\n\n<p>The more the num of epochs I set (I mean <em>thousands</em> of epochs instead of tens or hundreds), the higher the local CV score got. But simultaneously, the gap between train / validation losses widened, and the public LB of such model got lower.\nI adjusted the num of epochs lower than the threshold from which the gap began widening.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 497109,
          "author_name": "Neuron Engineer",
          "author_url": "",
          "post_date": "2019-03-23T00:57:36.733000",
          "content": "<p>Thank you for your kind answer!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "496742": "Congratulations to all the winners! \nAnd thanks a lot to VSB/Enet Center and Kaggle for this exciting competition. Since this is my first Kaggle competition, I'm surprised and glad to this honorable result.<br><br>\n \nI'm going to briefly share all of you my 10th place solution, which keeps the score of 0.688 in the private LB. <br><br>\n\n\nFirst of all, I have to tell you that this solution is not my best public LB model (RNN), which is in the end 13th place in public LB (0.766).  \nI think that the keys of this competition are high-pass filtering and DWT-denoising. In this point, I owe a lot to Jack's [kernel](https://www.kaggle.com/jackvial/dwt-signal-denoising).<br><br>\n\nAs many other guys mentioned in public kernel and discussion, the problem of this competition is the discrepancy between train / test distributions(and the unstability of CV / LB resulted from that).\nSo, I chose as a final submission more *conservative* RNN model, which kept modest score in public LB but might be more robust in my view.<br><br>\n\n\nIn order to choose a more robust model, I spent time to decrease val_auc score in adversarial validation.\nI'm going to show what I thought and tried as below.<br><br>\n\n1. Without any filtering / denoising, the discrepancy between train / test distributions was huge (val-auc was over 0.96 in adversarial validation. So, I thought that a model without any preprocessing was in danger of shake.\n2. With high-pass filtered and DWT-denoised, val-auc decreased to under 0.78. Therefore, I selected high-pass filtered and DWT-denoised model as a base one (high-pass filtering had the stronger influence).\n3. Next, I cut down features which contributed to increase val-auc in adversarial validation, and finally succeeded to decrease val_auc to under 0.7. (Notwithstanding, the instability of CV / LB was not completely solved...)<br><br>\n\n\nThe architecture of my RNN model is simple and not novel. \nSimilar to other guys, I selected [Bruno-based](https://www.kaggle.com/braquino/5-fold-lstm-attention-fully-commented-0-694) BiLSTM x 2 + Attention model and [Tarun-based](https://www.kaggle.com/tarunpaparaju/vsb-competition-attention-bilstm-with-features) BiLSTM x 2 + Attention + feature concatenation model.<br><br>\n\nAnd I hard-voted 8 predictions (each resulted from stratified 5- or 4-fold CV models) which were a little bit different from each other in terms of features and random seeds.\nWhen I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).\n\nMoreover, I used recently released AdaBound optimizer. Although It contributed to increase local CV score, I don't know it is a good choice especially in this easily-overfitting competition.",
    "498939": "Hi again @yukinkgwa , I would love to reimplement your approach next week, and I have a few more questions. It would be truly grateful if you can help a bit more : )\n\n1) Above, you mentioned that after filtered / denoised , you also removed some features in order to reduce the val_auc . Could you please give me examples what kind of features that you removed?\n\n2) Was the input to your Bi-LSTM has shape 2904 x 160 x 57 as in public kernel ?\n\n3) How did the ‘removed features’ in 1) affect the Bi-LSTM inputs in 2) ?\n\n4) Could you please let me know the final performance on **public LB** of your best private LB model (0.68837) ? \n\nThank you again , and sorry for this many questions!",
    "496872": "Congrats on the great finish and for sharing your approach. Great to see you were able to put my kernel to good use!",
    "496790": "Could youe explaine \"When I selected models, I kept in mind to choose ones whose train / validation losses were relatively small(because of CV / LB scores' instability).\"\n\nSo you mean choose the model with smaller CV gap?",
    "497052": "Congrats @yukinkgwa on your solo gold medal and thanks for sharing.\n\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/85146#496224",
    "497014": "Congrats! Watched you rising in ranking, and stayed high after the shakeup!\n\nInteresting to see you \"conservative\" way of selecting submission.  Honestly, my submissions are all over the place, no clear indication what would help me to choose for private LB. My best private LB comes from a fairly complex TCN. \n\nAnd the public LB is a data set much larger than training set. \n\nWill investigate a little more, but it is Friday! TGIF~",
    "496803": "Congrats! Could you explain what are the hgih-pass filter and denoising parameters? Are they those default ones in the public kernel or modified somehow? And are the inputs to the adversarial validation and RNN the same? Thank you!",
    "496748": "First of all, Congratulation! And thanks for sharing @yukinkgwa, it is a very nice approach to know! \n\nPlease allow me to ask some questions:\n1) So after you successfully decreased the val_auc to 0.7, how was your BiLSTM doing ?\nCould you please let us know CV vs. Public LB vs. Private LB ?\n\n2) And how is your best public LB (0.766) performing in CV / private?\n\n3) You said that you select a model with 'relatively small train/valid losses', so that mean it is not a usual keras callback 'save best weight due to a `valid_mcc`', right? How did you exactly decide to stop the training process?"
  }
}