{
  "id": 253233,
  "title": "The result of the best public kernels on new data",
  "url": "/competitions/seti-breakthrough-listen/discussion/253233",
  "author_name": "Kramarenko Vladislav",
  "post_date": "2021-07-15T14:18:03.901000",
  "votes": 33,
  "comment_count": 11,
  "views": 0,
  "content": "<table>\n<thead>\n<tr>\n<th>link</th>\n<th>new cv</th>\n<th>old</th>\n<th>new</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><a href=\"https://www.kaggle.com/xuxu1234/lb-0-980-efficientnet-b0-more-epoch\" target=\"_blank\">[LB:0.980]efficientnet_b0 More epoch</a></td>\n<td>0.85</td>\n<td>0.980</td>\n<td>0.750</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/ttahara/seti-e-t-volo-d1-baseline-inference\" target=\"_blank\">SETI-E.T. : VOLO-D1 Baseline [Inference]</a></td>\n<td>0.816</td>\n<td>0.974</td>\n<td>0.719</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/fauzanalfariz/seti-e-t-efficientnet-b4-signal-detection\" target=\"_blank\">[SETI E.T] EfficientNet B4 - Signal Detection</a></td>\n<td>-</td>\n<td>0.974</td>\n<td>0.717</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97\" target=\"_blank\">[Custom_Head + Gradual_Warmup] Single Fold 0.97</a></td>\n<td>-</td>\n<td>0.97</td>\n<td>0.730</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97\" target=\"_blank\">Pytorch Lightning Single Fold Training [LB 0.97]</a></td>\n<td>0.84+</td>\n<td>0.97</td>\n<td>0.728</td>\n</tr>\n<tr>\n<td>My_train_old_data</td>\n<td>0.83</td>\n<td>0.984</td>\n<td>0.742</td>\n</tr>\n<tr>\n<td>My_train_new_data</td>\n<td>0.84</td>\n<td>-</td>\n<td>0.752</td>\n</tr>\n</tbody>\n</table>",
  "messages": [
    {
      "id": 1389202,
      "postDate": "2021-07-15T14:18:03.900Z",
      "content": "<table>\n<thead>\n<tr>\n<th>link</th>\n<th>new cv</th>\n<th>old</th>\n<th>new</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><a href=\"https://www.kaggle.com/xuxu1234/lb-0-980-efficientnet-b0-more-epoch\" target=\"_blank\">[LB:0.980]efficientnet_b0 More epoch</a></td>\n<td>0.85</td>\n<td>0.980</td>\n<td>0.750</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/ttahara/seti-e-t-volo-d1-baseline-inference\" target=\"_blank\">SETI-E.T. : VOLO-D1 Baseline [Inference]</a></td>\n<td>0.816</td>\n<td>0.974</td>\n<td>0.719</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/fauzanalfariz/seti-e-t-efficientnet-b4-signal-detection\" target=\"_blank\">[SETI E.T] EfficientNet B4 - Signal Detection</a></td>\n<td>-</td>\n<td>0.974</td>\n<td>0.717</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97\" target=\"_blank\">[Custom_Head + Gradual_Warmup] Single Fold 0.97</a></td>\n<td>-</td>\n<td>0.97</td>\n<td>0.730</td>\n</tr>\n<tr>\n<td><a href=\"https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97\" target=\"_blank\">Pytorch Lightning Single Fold Training [LB 0.97]</a></td>\n<td>0.84+</td>\n<td>0.97</td>\n<td>0.728</td>\n</tr>\n<tr>\n<td>My_train_old_data</td>\n<td>0.83</td>\n<td>0.984</td>\n<td>0.742</td>\n</tr>\n<tr>\n<td>My_train_new_data</td>\n<td>0.84</td>\n<td>-</td>\n<td>0.752</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "| link | new cv | old | new |\n| --- | --- | --- | --- |\n| [[LB:0.980]efficientnet_b0 More epoch](https://www.kaggle.com/xuxu1234/lb-0-980-efficientnet-b0-more-epoch) | 0.85 | 0.980 | 0.750 |\n| [SETI-E.T. : VOLO-D1 Baseline [Inference]](https://www.kaggle.com/ttahara/seti-e-t-volo-d1-baseline-inference) | 0.816 | 0.974 |  0.719 |\n|  [[SETI E.T] EfficientNet B4 - Signal Detection](https://www.kaggle.com/fauzanalfariz/seti-e-t-efficientnet-b4-signal-detection)  | - | 0.974 | 0.717 |\n|  [[Custom_Head + Gradual_Warmup] Single Fold 0.97](https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97)  | - | 0.97 | 0.730 |\n|  [Pytorch Lightning Single Fold Training [LB 0.97]](https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97)  | 0.84+ | 0.97 | 0.728 |\n| My_train_old_data | 0.83 | 0.984 | 0.742 |\n| My_train_new_data | 0.84 | - | 0.752 |",
      "votes": 32
    },
    {
      "id": 1389266,
      "postDate": "2021-07-15T15:05:34.297Z",
      "content": "<p><a href=\"https://www.kaggle.com/vlomme\" target=\"_blank\">@vlomme</a> thanks for testing the kernels! So the new data is indeed more difficult. I hope there is some correlation though so that (some of) the previous experiment results remain valid in terms of the relative performance.</p>",
      "rawMarkdown": "@vlomme thanks for testing the kernels! So the new data is indeed more difficult. I hope there is some correlation though so that (some of) the previous experiment results remain valid in terms of the relative performance.",
      "votes": 3
    },
    {
      "id": 1389298,
      "postDate": "2021-07-15T15:35:42.173Z",
      "content": "<p><a href=\"https://www.kaggle.com/vlomme\" target=\"_blank\">@vlomme</a> thanks! </p>\n<p>EDIT: New CV scores for [LB 0.980] public nb are as follows: </p>\n<pre><code>    fold    metric\n    0   0.855494\n    1   0.854027\n    2   0.847246\n    3   0.852948\n   oof    0.849163\n</code></pre>",
      "rawMarkdown": "@vlomme thanks! ~~can you also share the CV scores of 1st one [LB0.980] ? ~~\n\nEDIT: New CV scores for [LB 0.980] public nb are as follows: \n\n```\n\tfold\tmetric\n\t0\t0.855494\n\t1\t0.854027\n\t2\t0.847246\n\t3\t0.852948\n   oof\t0.849163\n```",
      "votes": 1
    },
    {
      "id": 1398921,
      "postDate": "2021-07-24T16:44:38.903Z",
      "content": "<p>And 0.750 notebook is deleted. It doesn't seem fair in the spirit of competition. It may be huge disadvantage for people joining late. If someone can highlight key ideas of the notebook it will be great.</p>",
      "rawMarkdown": "And 0.750 notebook is deleted. It doesn't seem fair in the spirit of competition. It may be huge disadvantage for people joining late. If someone can highlight key ideas of the notebook it will be great.",
      "votes": 2,
      "replies": [
        {
          "id": 1398984,
          "postDate": "2021-07-24T18:18:40.547Z",
          "content": "<p>If I remember correctly, 0.750 notebook use same method below.<br>\n<a href=\"https://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline?scriptVersionId=68360235\" target=\"_blank\">https://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline?scriptVersionId=68360235</a><br>\nAnd change baseline ResNet to Efficientnet b0 with 2 layer of the fully connected layer.<br>\nIt achive old_roc_auc 0.978 near 17 epochs to 30 epochs. And final learning rate (that has meaning on training) is almost 0.5e-6. And It could get 0.980 - 0.984 if we ensemble 4 fold models.<br>\nAnd there were some other notebook using L1 loss with BCE loss together. It also got high score.</p>\n<p>ResNext, NFNet, Efficientnet had similar results.</p>\n<p>If we use leaky_old dataset as an extended dataset, then we may get 0.750 in only 2 epochs.<br>\nHowever, I think current leaderboard have a possibility of overfitting.<br>\nSo, we need to explore more general method.</p>",
          "rawMarkdown": "If I remember correctly, 0.750 notebook use same method below.\nhttps://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline?scriptVersionId=68360235\nAnd change baseline ResNet to Efficientnet b0 with 2 layer of the fully connected layer.\nIt achive old_roc_auc 0.978 near 17 epochs to 30 epochs. And final learning rate (that has meaning on training) is almost 0.5e-6. And It could get 0.980 - 0.984 if we ensemble 4 fold models.\nAnd there were some other notebook using L1 loss with BCE loss together. It also got high score.\n\nResNext, NFNet, Efficientnet had similar results.\n\nIf we use leaky_old dataset as an extended dataset, then we may get 0.750 in only 2 epochs.\nHowever, I think current leaderboard have a possibility of overfitting.\nSo, we need to explore more general method.",
          "votes": 5
        },
        {
          "id": 1399006,
          "postDate": "2021-07-24T18:57:10.653Z",
          "content": "<p>Deleting a public kernel makes all the ones who used it or clones it use private sharing, which violates competition rules.  I have raised this many times, but Kaggle staff doe snot care about this use case apparently.</p>",
          "rawMarkdown": "Deleting a public kernel makes all the ones who used it or clones it use private sharing, which violates competition rules.  I have raised this many times, but Kaggle staff doe snot care about this use case apparently.",
          "votes": 1
        },
        {
          "id": 1399011,
          "postDate": "2021-07-24T19:00:17.393Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 1399028,
          "postDate": "2021-07-24T19:26:52.397Z",
          "content": "<p>I also don't understand why a public notebook would be deleted. Link to copy <a href=\"https://www.kaggle.com/vlomme/lb-0-980-efficientnet-b0-more-epoch\" target=\"_blank\">https://www.kaggle.com/vlomme/lb-0-980-efficientnet-b0-more-epoch</a></p>",
          "rawMarkdown": "I also don't understand why a public notebook would be deleted. Link to copy https://www.kaggle.com/vlomme/lb-0-980-efficientnet-b0-more-epoch",
          "votes": 6
        }
      ]
    },
    {
      "id": 1389388,
      "postDate": "2021-07-15T17:01:06.293Z",
      "content": "<p>I tested my single fold baseline <a href=\"https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97\" target=\"_blank\">Pytorch Lightning Single Fold Training [LB 0.97]</a> it used to have [LB 0.97] now I got [LB 0.728] and single fold CV 0.84+</p>\n<p>I hope you find that info useful, thanks for sharing again…</p>",
      "rawMarkdown": "I tested my single fold baseline [Pytorch Lightning Single Fold Training [LB 0.97]](https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97) it used to have [LB 0.97] now I got [LB 0.728] and single fold CV 0.84+\n\nI hope you find that info useful, thanks for sharing again...",
      "votes": 2
    },
    {
      "id": 1389275,
      "postDate": "2021-07-15T15:13:42.017Z",
      "content": "<p><a href=\"https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97\" target=\"_blank\">[Custom_Head + Gradual_Warmup] Single Fold 0.97 </a>  Old: 0.970 New: 0.730</p>",
      "rawMarkdown": "[[Custom_Head + Gradual_Warmup] Single Fold 0.97 ](https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97)  Old: 0.970 New: 0.730",
      "votes": 2
    },
    {
      "id": 1390275,
      "postDate": "2021-07-16T14:07:36.257Z",
      "content": "<p>Thanks.  We now know why so many people are at 0.750</p>",
      "rawMarkdown": "Thanks.  We now know why so many people are at 0.750"
    },
    {
      "id": 1389209,
      "postDate": "2021-07-15T14:22:56.030Z",
      "content": "<p>me after seeing this results <img src=\"https://media.wired.com/photos/5f87340d114b38fa1f8339f9/master/w_1600%2Cc_limit/Ideas_Surprised_Pikachu_HD.jpg\" alt=\"\"></p>",
      "rawMarkdown": "me after seeing this results ![](https://media.wired.com/photos/5f87340d114b38fa1f8339f9/master/w_1600%2Cc_limit/Ideas_Surprised_Pikachu_HD.jpg)"
    }
  ],
  "comments": [
    {
      "id": 1389266,
      "author_name": "Nikita Kozodoi",
      "author_url": "",
      "post_date": "2021-07-15T15:05:34.297000",
      "content": "<p><a href=\"https://www.kaggle.com/vlomme\" target=\"_blank\">@vlomme</a> thanks for testing the kernels! So the new data is indeed more difficult. I hope there is some correlation though so that (some of) the previous experiment results remain valid in terms of the relative performance.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1389298,
      "author_name": "Ioannis M",
      "author_url": "",
      "post_date": "2021-07-15T15:35:42.173000",
      "content": "<p><a href=\"https://www.kaggle.com/vlomme\" target=\"_blank\">@vlomme</a> thanks! </p>\n<p>EDIT: New CV scores for [LB 0.980] public nb are as follows: </p>\n<pre><code>    fold    metric\n    0   0.855494\n    1   0.854027\n    2   0.847246\n    3   0.852948\n   oof    0.849163\n</code></pre>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1398921,
      "author_name": "Kumar Shubham",
      "author_url": "",
      "post_date": "2021-07-24T16:44:38.903000",
      "content": "<p>And 0.750 notebook is deleted. It doesn't seem fair in the spirit of competition. It may be huge disadvantage for people joining late. If someone can highlight key ideas of the notebook it will be great.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1398984,
          "author_name": "WOOSUNG YOON",
          "author_url": "",
          "post_date": "2021-07-24T18:18:40.547000",
          "content": "<p>If I remember correctly, 0.750 notebook use same method below.<br>\n<a href=\"https://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline?scriptVersionId=68360235\" target=\"_blank\">https://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline?scriptVersionId=68360235</a><br>\nAnd change baseline ResNet to Efficientnet b0 with 2 layer of the fully connected layer.<br>\nIt achive old_roc_auc 0.978 near 17 epochs to 30 epochs. And final learning rate (that has meaning on training) is almost 0.5e-6. And It could get 0.980 - 0.984 if we ensemble 4 fold models.<br>\nAnd there were some other notebook using L1 loss with BCE loss together. It also got high score.</p>\n<p>ResNext, NFNet, Efficientnet had similar results.</p>\n<p>If we use leaky_old dataset as an extended dataset, then we may get 0.750 in only 2 epochs.<br>\nHowever, I think current leaderboard have a possibility of overfitting.<br>\nSo, we need to explore more general method.</p>",
          "votes": 5,
          "replies": []
        },
        {
          "id": 1399006,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2021-07-24T18:57:10.653000",
          "content": "<p>Deleting a public kernel makes all the ones who used it or clones it use private sharing, which violates competition rules.  I have raised this many times, but Kaggle staff doe snot care about this use case apparently.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1399011,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-07-24T19:00:17.393000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1399028,
          "author_name": "Kramarenko Vladislav",
          "author_url": "",
          "post_date": "2021-07-24T19:26:52.397000",
          "content": "<p>I also don't understand why a public notebook would be deleted. Link to copy <a href=\"https://www.kaggle.com/vlomme/lb-0-980-efficientnet-b0-more-epoch\" target=\"_blank\">https://www.kaggle.com/vlomme/lb-0-980-efficientnet-b0-more-epoch</a></p>",
          "votes": 6,
          "replies": []
        }
      ]
    },
    {
      "id": 1389388,
      "author_name": "Ertuğrul Demir",
      "author_url": "",
      "post_date": "2021-07-15T17:01:06.293000",
      "content": "<p>I tested my single fold baseline <a href=\"https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97\" target=\"_blank\">Pytorch Lightning Single Fold Training [LB 0.97]</a> it used to have [LB 0.97] now I got [LB 0.728] and single fold CV 0.84+</p>\n<p>I hope you find that info useful, thanks for sharing again…</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1389275,
      "author_name": "Bruce Young",
      "author_url": "",
      "post_date": "2021-07-15T15:13:42.017000",
      "content": "<p><a href=\"https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97\" target=\"_blank\">[Custom_Head + Gradual_Warmup] Single Fold 0.97 </a>  Old: 0.970 New: 0.730</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1390275,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2021-07-16T14:07:36.257000",
      "content": "<p>Thanks.  We now know why so many people are at 0.750</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1389209,
      "author_name": "Mithil Salunkhe",
      "author_url": "",
      "post_date": "2021-07-15T14:22:56.030000",
      "content": "<p>me after seeing this results <img src=\"https://media.wired.com/photos/5f87340d114b38fa1f8339f9/master/w_1600%2Cc_limit/Ideas_Surprised_Pikachu_HD.jpg\" alt=\"\"></p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1389202": "| link | new cv | old | new |\n| --- | --- | --- | --- |\n| [[LB:0.980]efficientnet_b0 More epoch](https://www.kaggle.com/xuxu1234/lb-0-980-efficientnet-b0-more-epoch) | 0.85 | 0.980 | 0.750 |\n| [SETI-E.T. : VOLO-D1 Baseline [Inference]](https://www.kaggle.com/ttahara/seti-e-t-volo-d1-baseline-inference) | 0.816 | 0.974 |  0.719 |\n|  [[SETI E.T] EfficientNet B4 - Signal Detection](https://www.kaggle.com/fauzanalfariz/seti-e-t-efficientnet-b4-signal-detection)  | - | 0.974 | 0.717 |\n|  [[Custom_Head + Gradual_Warmup] Single Fold 0.97](https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97)  | - | 0.97 | 0.730 |\n|  [Pytorch Lightning Single Fold Training [LB 0.97]](https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97)  | 0.84+ | 0.97 | 0.728 |\n| My_train_old_data | 0.83 | 0.984 | 0.742 |\n| My_train_new_data | 0.84 | - | 0.752 |",
    "1389266": "@vlomme thanks for testing the kernels! So the new data is indeed more difficult. I hope there is some correlation though so that (some of) the previous experiment results remain valid in terms of the relative performance.",
    "1389298": "@vlomme thanks! ~~can you also share the CV scores of 1st one [LB0.980] ? ~~\n\nEDIT: New CV scores for [LB 0.980] public nb are as follows: \n\n```\n\tfold\tmetric\n\t0\t0.855494\n\t1\t0.854027\n\t2\t0.847246\n\t3\t0.852948\n   oof\t0.849163\n```",
    "1398921": "And 0.750 notebook is deleted. It doesn't seem fair in the spirit of competition. It may be huge disadvantage for people joining late. If someone can highlight key ideas of the notebook it will be great.",
    "1389388": "I tested my single fold baseline [Pytorch Lightning Single Fold Training [LB 0.97]](https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97) it used to have [LB 0.97] now I got [LB 0.728] and single fold CV 0.84+\n\nI hope you find that info useful, thanks for sharing again...",
    "1389275": "[[Custom_Head + Gradual_Warmup] Single Fold 0.97 ](https://www.kaggle.com/reighns/custom-head-gradual-warmup-single-fold-0-97)  Old: 0.970 New: 0.730",
    "1390275": "Thanks.  We now know why so many people are at 0.750",
    "1389209": "me after seeing this results ![](https://media.wired.com/photos/5f87340d114b38fa1f8339f9/master/w_1600%2Cc_limit/Ideas_Surprised_Pikachu_HD.jpg)"
  }
}