{
  "id": 260778,
  "title": "Test score doesn't match validation score",
  "url": "/competitions/g2net-gravitational-wave-detection/discussion/260778",
  "author_name": "",
  "post_date": "2021-08-03T19:14:55.993916200Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello everyone. I just submitted my first attempt and I think I'm doing something wrong with my submission. My validation set AUC score is around .8, but when I load the test set, make the predictions and submit the csv, the score is less than .5. I check the test predictions against the data by hand and it seems to be predicting fine, at least for the obvious samples. My submission.csv is formatted exactly as the sample submissions, with probability predictions in the target column. Any idea what I'm doing wrong?</p>",
  "messages": [
    {
      "id": "1440925",
      "postDate": "08/03/2021 19:14:55",
      "content": "<p>Hello everyone. I just submitted my first attempt and I think I'm doing something wrong with my submission. My validation set AUC score is around .8, but when I load the test set, make the predictions and submit the csv, the score is less than .5. I check the test predictions against the data by hand and it seems to be predicting fine, at least for the obvious samples. My submission.csv is formatted exactly as the sample submissions, with probability predictions in the target column. Any idea what I'm doing wrong?</p>",
      "rawMarkdown": "Hello everyone. I just submitted my first attempt and I think I'm doing something wrong with my submission. My validation set AUC score is around .8, but when I load the test set, make the predictions and submit the csv, the score is less than .5. I check the test predictions against the data by hand and it seems to be predicting fine, at least for the obvious samples. My submission.csv is formatted exactly as the sample submissions, with probability predictions in the target column. Any idea what I'm doing wrong?",
      "votes": null
    },
    {
      "id": "1442641",
      "postDate": "08/03/2021 23:40:41",
      "content": "<p>As you know, ROC AUC score less than .5 means worse performance than random predictions. I suggest you make sure your test samples are not shuffled e.g. <code>shuffle</code> parameter in data loader etc.</p>",
      "rawMarkdown": "As you know, ROC AUC score less than .5 means worse performance than random predictions. I suggest you make sure your test samples are not shuffled e.g. `shuffle` parameter in data loader etc.",
      "votes": null
    },
    {
      "id": "1448050",
      "postDate": "08/04/2021 16:35:59",
      "content": "<p>That was the problem. Thanks!</p>",
      "rawMarkdown": "That was the problem. Thanks!",
      "votes": null
    },
    {
      "id": "1561297",
      "postDate": "10/27/2021 13:13:01",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1442641,
      "author_name": "analokamus",
      "author_url": "",
      "post_date": "08/03/2021 23:40:41",
      "content": "<p>As you know, ROC AUC score less than .5 means worse performance than random predictions. I suggest you make sure your test samples are not shuffled e.g. <code>shuffle</code> parameter in data loader etc.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1448050,
          "author_name": "tonydavis629",
          "author_url": "",
          "post_date": "08/04/2021 16:35:59",
          "content": "<p>That was the problem. Thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1561297,
      "author_name": "zerafachris",
      "author_url": "",
      "post_date": "10/27/2021 13:13:01",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1440925": "Hello everyone. I just submitted my first attempt and I think I'm doing something wrong with my submission. My validation set AUC score is around .8, but when I load the test set, make the predictions and submit the csv, the score is less than .5. I check the test predictions against the data by hand and it seems to be predicting fine, at least for the obvious samples. My submission.csv is formatted exactly as the sample submissions, with probability predictions in the target column. Any idea what I'm doing wrong?",
    "1442641": "As you know, ROC AUC score less than .5 means worse performance than random predictions. I suggest you make sure your test samples are not shuffled e.g. `shuffle` parameter in data loader etc.",
    "1448050": "That was the problem. Thanks!",
    "1561297": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris"
  },
  "source": "meta"
}