{
  "id": 134851,
  "title": "Huge difference between CV and LB",
  "url": "/competitions/bengaliai-cv19/discussion/134851",
  "author_name": "",
  "post_date": "2020-03-10T19:06:28.862659600Z",
  "votes": 2,
  "comment_count": 9,
  "views": 0,
  "content": "<p>I trained my densenet121 for about 250 epochs. My CV score is around 0.99 but my LB score is in the range of 0.92-0.93.\nI tried reducing the Variance by incorporating Dropout and some augmentations. But it still did not work. Any suggestions would be immensely helpful 🙏 </p>",
  "messages": [
    {
      "id": "768413",
      "postDate": "03/10/2020 19:06:28",
      "content": "<p>I trained my densenet121 for about 250 epochs. My CV score is around 0.99 but my LB score is in the range of 0.92-0.93.\nI tried reducing the Variance by incorporating Dropout and some augmentations. But it still did not work. Any suggestions would be immensely helpful 🙏 </p>",
      "rawMarkdown": "I trained my densenet121 for about 250 epochs. My CV score is around 0.99 but my LB score is in the range of 0.92-0.93.\nI tried reducing the Variance by incorporating Dropout and some augmentations. But it still did not work. Any suggestions would be immensely helpful 🙏",
      "votes": null
    },
    {
      "id": "768451",
      "postDate": "03/10/2020 20:17:49",
      "content": "<p>U can try out using adaptive pooling methods to tune your model </p>",
      "rawMarkdown": "U can try out using adaptive pooling methods to tune your model",
      "votes": null
    },
    {
      "id": "768479",
      "postDate": "03/10/2020 21:06:24",
      "content": "<p>tripple check your inference kernel,,your model seems ok to me,,it's definitely your inference kernel that is causing problem,,,maybe the  images that were used to train is different than test images,,maybe you are missing normalization during inference or maybe you  are not doing proper validation,,,but to me it seems like your inference kernel is causing problem,,,check that carefully,,if that's okay then  maybe the  images you are using for training, not good</p>",
      "rawMarkdown": "tripple check your inference kernel,,your model seems ok to me,,it's definitely your inference kernel that is causing problem,,,maybe the  images that were used to train is different than test images,,maybe you are missing normalization during inference or maybe you  are not doing proper validation,,,but to me it seems like your inference kernel is causing problem,,,check that carefully,,if that's okay then  maybe the  images you are using for training, not good",
      "votes": null
    },
    {
      "id": "768612",
      "postDate": "03/11/2020 02:47:39",
      "content": "<p>That's a very big difference. I suspect something is wrong. Either your CV is leaking and/or computed wrong, or there is a problem with your inference notebook. Make sure you apply the same image preprocess on test before inference that you perform on train. And make sure your validation is computing 0.5/0.25/0.25 macro recall with no leakage from CutMix/MixUp random image selection between train and validation.</p>",
      "rawMarkdown": "That's a very big difference. I suspect something is wrong. Either your CV is leaking and/or computed wrong, or there is a problem with your inference notebook. Make sure you apply the same image preprocess on test before inference that you perform on train. And make sure your validation is computing 0.5/0.25/0.25 macro recall with no leakage from CutMix/MixUp random image selection between train and validation.",
      "votes": null
    },
    {
      "id": "768695",
      "postDate": "03/11/2020 05:13:23",
      "content": "<p>Will check it out</p>",
      "rawMarkdown": "Will check it out",
      "votes": null
    },
    {
      "id": "768696",
      "postDate": "03/11/2020 05:13:31",
      "content": "<p>Will check it out</p>",
      "rawMarkdown": "Will check it out",
      "votes": null
    },
    {
      "id": "768812",
      "postDate": "03/11/2020 08:02:44",
      "content": "<p>I don't think there is an issue with the CV computation. The training and validation losses are in the range of 0.02 and 0.1 respectively. </p>",
      "rawMarkdown": "I don't think there is an issue with the CV computation. The training and validation losses are in the range of 0.02 and 0.1 respectively.",
      "votes": null
    },
    {
      "id": "769398",
      "postDate": "03/11/2020 21:36:14",
      "content": "<p>The possible cause can be improper split into train/validation.\nBy improper, I mean that class occurrence frequencies can be different in validation and test sets.\nThus you get really good numbers in CV but worse in LB.</p>",
      "rawMarkdown": "The possible cause can be improper split into train/validation.\nBy improper, I mean that class occurrence frequencies can be different in validation and test sets.\nThus you get really good numbers in CV but worse in LB.",
      "votes": null
    },
    {
      "id": "769628",
      "postDate": "03/12/2020 05:17:32",
      "content": "<p>Check whether your preprocessing/postprocessing is correct in the kaggle kernel. One easy way to make sure is to predict on a bunch of validation examples on kaggle and compare with your local computer results.</p>\n\n<p>If this is not the case, check whether your CV is leaking (for example you shuffle your indexes wrongly and sample validation examples during training). </p>",
      "rawMarkdown": "Check whether your preprocessing/postprocessing is correct in the kaggle kernel. One easy way to make sure is to predict on a bunch of validation examples on kaggle and compare with your local computer results.\n\nIf this is not the case, check whether your CV is leaking (for example you shuffle your indexes wrongly and sample validation examples during training).",
      "votes": null
    },
    {
      "id": "769667",
      "postDate": "03/12/2020 06:11:31",
      "content": "<p>That's too much. Check the input image resolution for prediction and if you give augmentation to the test.  I made these mistakes before, giving a very low LB score.</p>",
      "rawMarkdown": "That's too much. Check the input image resolution for prediction and if you give augmentation to the test.  I made these mistakes before, giving a very low LB score.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 768451,
      "author_name": "vineeth1999",
      "author_url": "",
      "post_date": "03/10/2020 20:17:49",
      "content": "<p>U can try out using adaptive pooling methods to tune your model </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 768479,
      "author_name": "mobassir",
      "author_url": "",
      "post_date": "03/10/2020 21:06:24",
      "content": "<p>tripple check your inference kernel,,your model seems ok to me,,it's definitely your inference kernel that is causing problem,,,maybe the  images that were used to train is different than test images,,maybe you are missing normalization during inference or maybe you  are not doing proper validation,,,but to me it seems like your inference kernel is causing problem,,,check that carefully,,if that's okay then  maybe the  images you are using for training, not good</p>",
      "votes": null,
      "replies": [
        {
          "id": 768695,
          "author_name": "venky2506",
          "author_url": "",
          "post_date": "03/11/2020 05:13:23",
          "content": "<p>Will check it out</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 768612,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "03/11/2020 02:47:39",
      "content": "<p>That's a very big difference. I suspect something is wrong. Either your CV is leaking and/or computed wrong, or there is a problem with your inference notebook. Make sure you apply the same image preprocess on test before inference that you perform on train. And make sure your validation is computing 0.5/0.25/0.25 macro recall with no leakage from CutMix/MixUp random image selection between train and validation.</p>",
      "votes": null,
      "replies": [
        {
          "id": 768696,
          "author_name": "venky2506",
          "author_url": "",
          "post_date": "03/11/2020 05:13:31",
          "content": "<p>Will check it out</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 768812,
          "author_name": "venky2506",
          "author_url": "",
          "post_date": "03/11/2020 08:02:44",
          "content": "<p>I don't think there is an issue with the CV computation. The training and validation losses are in the range of 0.02 and 0.1 respectively. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 769398,
      "author_name": "dgrechka",
      "author_url": "",
      "post_date": "03/11/2020 21:36:14",
      "content": "<p>The possible cause can be improper split into train/validation.\nBy improper, I mean that class occurrence frequencies can be different in validation and test sets.\nThus you get really good numbers in CV but worse in LB.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 769628,
      "author_name": "dipamc77",
      "author_url": "",
      "post_date": "03/12/2020 05:17:32",
      "content": "<p>Check whether your preprocessing/postprocessing is correct in the kaggle kernel. One easy way to make sure is to predict on a bunch of validation examples on kaggle and compare with your local computer results.</p>\n\n<p>If this is not the case, check whether your CV is leaking (for example you shuffle your indexes wrongly and sample validation examples during training). </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 769667,
      "author_name": "yuanlin08",
      "author_url": "",
      "post_date": "03/12/2020 06:11:31",
      "content": "<p>That's too much. Check the input image resolution for prediction and if you give augmentation to the test.  I made these mistakes before, giving a very low LB score.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "768413": "I trained my densenet121 for about 250 epochs. My CV score is around 0.99 but my LB score is in the range of 0.92-0.93.\nI tried reducing the Variance by incorporating Dropout and some augmentations. But it still did not work. Any suggestions would be immensely helpful 🙏",
    "768451": "U can try out using adaptive pooling methods to tune your model",
    "768479": "tripple check your inference kernel,,your model seems ok to me,,it's definitely your inference kernel that is causing problem,,,maybe the  images that were used to train is different than test images,,maybe you are missing normalization during inference or maybe you  are not doing proper validation,,,but to me it seems like your inference kernel is causing problem,,,check that carefully,,if that's okay then  maybe the  images you are using for training, not good",
    "768612": "That's a very big difference. I suspect something is wrong. Either your CV is leaking and/or computed wrong, or there is a problem with your inference notebook. Make sure you apply the same image preprocess on test before inference that you perform on train. And make sure your validation is computing 0.5/0.25/0.25 macro recall with no leakage from CutMix/MixUp random image selection between train and validation.",
    "768695": "Will check it out",
    "768696": "Will check it out",
    "768812": "I don't think there is an issue with the CV computation. The training and validation losses are in the range of 0.02 and 0.1 respectively.",
    "769398": "The possible cause can be improper split into train/validation.\nBy improper, I mean that class occurrence frequencies can be different in validation and test sets.\nThus you get really good numbers in CV but worse in LB.",
    "769628": "Check whether your preprocessing/postprocessing is correct in the kaggle kernel. One easy way to make sure is to predict on a bunch of validation examples on kaggle and compare with your local computer results.\n\nIf this is not the case, check whether your CV is leaking (for example you shuffle your indexes wrongly and sample validation examples during training).",
    "769667": "That's too much. Check the input image resolution for prediction and if you give augmentation to the test.  I made these mistakes before, giving a very low LB score."
  },
  "source": "meta"
}