{
  "id": 270868,
  "title": "Huge difference between the CV AUC and public score",
  "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/270868",
  "author_name": "",
  "post_date": "2021-09-07T11:31:38.118562Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I'm observing a systematically large difference between CV AUC ~0.88 (and accuracy ~0.83) and score calculated on the public LB (~0.47). Such difference is quite reproducible for different train/validation divisions of the training dataset.</p>\n<p>My first guess was it is a submission artifact like in a few challenges before. But I never met such a discussion in the forum yet, it seems everybody use sample_submission.csv like me.</p>\n<p>Do anybody had an experience like that and how did you cure it?</p>",
  "messages": [
    {
      "id": "1505567",
      "postDate": "09/07/2021 11:31:38",
      "content": "<p>I'm observing a systematically large difference between CV AUC ~0.88 (and accuracy ~0.83) and score calculated on the public LB (~0.47). Such difference is quite reproducible for different train/validation divisions of the training dataset.</p>\n<p>My first guess was it is a submission artifact like in a few challenges before. But I never met such a discussion in the forum yet, it seems everybody use sample_submission.csv like me.</p>\n<p>Do anybody had an experience like that and how did you cure it?</p>",
      "rawMarkdown": "I'm observing a systematically large difference between CV AUC ~0.88 (and accuracy ~0.83) and score calculated on the public LB (~0.47). Such difference is quite reproducible for different train/validation divisions of the training dataset.\n\nMy first guess was it is a submission artifact like in a few challenges before. But I never met such a discussion in the forum yet, it seems everybody use sample_submission.csv like me.\n\nDo anybody had an experience like that and how did you cure it?",
      "votes": null
    },
    {
      "id": "1505848",
      "postDate": "09/07/2021 15:21:33",
      "content": "<p><a href=\"https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352\" target=\"_blank\">https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352</a> you can see some related discussions here</p>",
      "rawMarkdown": "https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352 you can see some related discussions here",
      "votes": null
    },
    {
      "id": "1505944",
      "postDate": "09/07/2021 16:48:37",
      "content": "<p>Validation AUC 0.88 is interesting.</p>",
      "rawMarkdown": "Validation AUC 0.88 is interesting.",
      "votes": null
    },
    {
      "id": "1506153",
      "postDate": "09/08/2021 00:12:59",
      "content": "<p>DeepUnderstanding, thank you for pointing this! <br>\nInteresting discussion.</p>",
      "rawMarkdown": "DeepUnderstanding, thank you for pointing this! \nInteresting discussion.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1505848,
      "author_name": "mrinath",
      "author_url": "",
      "post_date": "09/07/2021 15:21:33",
      "content": "<p><a href=\"https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352\" target=\"_blank\">https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352</a> you can see some related discussions here</p>",
      "votes": null,
      "replies": [
        {
          "id": 1506153,
          "author_name": "polishch",
          "author_url": "",
          "post_date": "09/08/2021 00:12:59",
          "content": "<p>DeepUnderstanding, thank you for pointing this! <br>\nInteresting discussion.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1505944,
      "author_name": "drtausamaru",
      "author_url": "",
      "post_date": "09/07/2021 16:48:37",
      "content": "<p>Validation AUC 0.88 is interesting.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1505567": "I'm observing a systematically large difference between CV AUC ~0.88 (and accuracy ~0.83) and score calculated on the public LB (~0.47). Such difference is quite reproducible for different train/validation divisions of the training dataset.\n\nMy first guess was it is a submission artifact like in a few challenges before. But I never met such a discussion in the forum yet, it seems everybody use sample_submission.csv like me.\n\nDo anybody had an experience like that and how did you cure it?",
    "1505848": "https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352 you can see some related discussions here",
    "1505944": "Validation AUC 0.88 is interesting.",
    "1506153": "DeepUnderstanding, thank you for pointing this! \nInteresting discussion."
  },
  "source": "meta"
}