{
  "id": 132652,
  "title": "LB/CV gap due to models/augumentations/hyperparams?",
  "url": "/competitions/bengaliai-cv19/discussion/132652",
  "author_name": "",
  "post_date": "2020-02-27T05:07:32.468177100Z",
  "votes": 2,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I see some models have similar LB/CV (0.2% gap) and some with large gaps (1%).\nWhy do these gaps occur? From model choice or augmentations?</p>\n\n<p>Will be interesting if we can see if good augmentations (like autoaug) will provide smaller LB/CV gaps.</p>\n\n<p>Both my res34 models had a big gap\nCV 0.980\nLB 0.970\nwhile some people report smaller gaps!</p>",
  "messages": [
    {
      "id": "757779",
      "postDate": "02/27/2020 05:07:32",
      "content": "<p>I see some models have similar LB/CV (0.2% gap) and some with large gaps (1%).\nWhy do these gaps occur? From model choice or augmentations?</p>\n\n<p>Will be interesting if we can see if good augmentations (like autoaug) will provide smaller LB/CV gaps.</p>\n\n<p>Both my res34 models had a big gap\nCV 0.980\nLB 0.970\nwhile some people report smaller gaps!</p>",
      "rawMarkdown": "I see some models have similar LB/CV (0.2% gap) and some with large gaps (1%).\nWhy do these gaps occur? From model choice or augmentations?\n\nWill be interesting if we can see if good augmentations (like autoaug) will provide smaller LB/CV gaps.\n\nBoth my res34 models had a big gap\nCV 0.980\nLB 0.970\nwhile some people report smaller gaps!",
      "votes": null
    },
    {
      "id": "757892",
      "postDate": "02/27/2020 08:09:09",
      "content": "<p>Same issue here,  from some of my experiments:</p>\n\n<p>SeResNext50,101 -&gt; 0.9~1.1% gap\nEfficientNet b3-b5 -&gt; 0.9~1.1% gap\nWideRes50,101 -&gt; 1.0~1.3% gap</p>\n\n<p>It may be caused by different distribution and unseen combinations mentioned in this <a href=\"https://www.kaggle.com/c/bengaliai-cv19/discussion/131078\">post</a>,\nI'v tried several augmentation methods but seems didn't work,\nStill looking for a way to deal with it.</p>",
      "rawMarkdown": "Same issue here,  from some of my experiments:\n\nSeResNext50,101 -&gt; 0.9~1.1% gap\nEfficientNet b3-b5 -&gt; 0.9~1.1% gap\nWideRes50,101 -&gt; 1.0~1.3% gap\n\nIt may be caused by different distribution and unseen combinations mentioned in this [post](https://www.kaggle.com/c/bengaliai-cv19/discussion/131078),\nI'v tried several augmentation methods but seems didn't work,\nStill looking for a way to deal with it.",
      "votes": null
    },
    {
      "id": "757906",
      "postDate": "02/27/2020 08:22:04",
      "content": "<p>My gap is smaller. Single model CV 0.989 and LB 0.984. </p>",
      "rawMarkdown": "My gap is smaller. Single model CV 0.989 and LB 0.984.",
      "votes": null
    },
    {
      "id": "757924",
      "postDate": "02/27/2020 08:51:30",
      "content": "<p>Wow, the gap is really small!</p>\n\n<p>It is interesting that some people have smaller gaps.\nMy best model performs CV 0.99 but LB 0.98, just thought if minimizing this gap may move my LB up</p>",
      "rawMarkdown": "Wow, the gap is really small!\n\nIt is interesting that some people have smaller gaps.\nMy best model performs CV 0.99 but LB 0.98, just thought if minimizing this gap may move my LB up",
      "votes": null
    },
    {
      "id": "758562",
      "postDate": "02/27/2020 23:24:15",
      "content": "<p>thank you for sharing the experiments.\nso it may be normal to have a CV-LB gap, regradress of model architectures.</p>\n\n<p>I was wondering if augumentations contribute to these gaps or not.\nBut good augumentations will both push up LB+CV, so it may not be able to split the effects.</p>",
      "rawMarkdown": "thank you for sharing the experiments.\nso it may be normal to have a CV-LB gap, regradress of model architectures.\n\nI was wondering if augumentations contribute to these gaps or not.\nBut good augumentations will both push up LB+CV, so it may not be able to split the effects.",
      "votes": null
    },
    {
      "id": "758578",
      "postDate": "02/28/2020 00:10:12",
      "content": "<p>I have really big gap. cv0.994, lb 0.9770.</p>",
      "rawMarkdown": "I have really big gap. cv0.994, lb 0.9770.",
      "votes": null
    },
    {
      "id": "758581",
      "postDate": "02/28/2020 00:14:58",
      "content": "<p>I tried a few different random seeds but none of them make my gaps smaller. Although augmentation seems to be an important factor, I don't quite understand how it could make an impact in CV/LB difference.</p>",
      "rawMarkdown": "I tried a few different random seeds but none of them make my gaps smaller. Although augmentation seems to be an important factor, I don't quite understand how it could make an impact in CV/LB difference.",
      "votes": null
    },
    {
      "id": "758643",
      "postDate": "02/28/2020 02:05:24",
      "content": "<p>I think your gap seems very normal, you can refer to <a href=\"https://www.kaggle.com/c/bengaliai-cv19/discussion/123198\">here</a></p>",
      "rawMarkdown": "I think your gap seems very normal, you can refer to [here](https://www.kaggle.com/c/bengaliai-cv19/discussion/123198)",
      "votes": null
    },
    {
      "id": "758646",
      "postDate": "02/28/2020 02:13:21",
      "content": "<p>your results are promising, thanks for sharing.\nI guess one approach is trusting my CV and try to push it as much as possible; LB will follow. </p>",
      "rawMarkdown": "your results are promising, thanks for sharing.\nI guess one approach is trusting my CV and try to push it as much as possible; LB will follow.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 757892,
      "author_name": "ccchang801023",
      "author_url": "",
      "post_date": "02/27/2020 08:09:09",
      "content": "<p>Same issue here,  from some of my experiments:</p>\n\n<p>SeResNext50,101 -&gt; 0.9~1.1% gap\nEfficientNet b3-b5 -&gt; 0.9~1.1% gap\nWideRes50,101 -&gt; 1.0~1.3% gap</p>\n\n<p>It may be caused by different distribution and unseen combinations mentioned in this <a href=\"https://www.kaggle.com/c/bengaliai-cv19/discussion/131078\">post</a>,\nI'v tried several augmentation methods but seems didn't work,\nStill looking for a way to deal with it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 758562,
          "author_name": "kyoshioka47",
          "author_url": "",
          "post_date": "02/27/2020 23:24:15",
          "content": "<p>thank you for sharing the experiments.\nso it may be normal to have a CV-LB gap, regradress of model architectures.</p>\n\n<p>I was wondering if augumentations contribute to these gaps or not.\nBut good augumentations will both push up LB+CV, so it may not be able to split the effects.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 758581,
          "author_name": "tonychenxyz",
          "author_url": "",
          "post_date": "02/28/2020 00:14:58",
          "content": "<p>I tried a few different random seeds but none of them make my gaps smaller. Although augmentation seems to be an important factor, I don't quite understand how it could make an impact in CV/LB difference.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 757906,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "02/27/2020 08:22:04",
      "content": "<p>My gap is smaller. Single model CV 0.989 and LB 0.984. </p>",
      "votes": null,
      "replies": [
        {
          "id": 757924,
          "author_name": "kyoshioka47",
          "author_url": "",
          "post_date": "02/27/2020 08:51:30",
          "content": "<p>Wow, the gap is really small!</p>\n\n<p>It is interesting that some people have smaller gaps.\nMy best model performs CV 0.99 but LB 0.98, just thought if minimizing this gap may move my LB up</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 758578,
      "author_name": "tonychenxyz",
      "author_url": "",
      "post_date": "02/28/2020 00:10:12",
      "content": "<p>I have really big gap. cv0.994, lb 0.9770.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 758643,
      "author_name": "garybios",
      "author_url": "",
      "post_date": "02/28/2020 02:05:24",
      "content": "<p>I think your gap seems very normal, you can refer to <a href=\"https://www.kaggle.com/c/bengaliai-cv19/discussion/123198\">here</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 758646,
          "author_name": "kyoshioka47",
          "author_url": "",
          "post_date": "02/28/2020 02:13:21",
          "content": "<p>your results are promising, thanks for sharing.\nI guess one approach is trusting my CV and try to push it as much as possible; LB will follow. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "757779": "I see some models have similar LB/CV (0.2% gap) and some with large gaps (1%).\nWhy do these gaps occur? From model choice or augmentations?\n\nWill be interesting if we can see if good augmentations (like autoaug) will provide smaller LB/CV gaps.\n\nBoth my res34 models had a big gap\nCV 0.980\nLB 0.970\nwhile some people report smaller gaps!",
    "757892": "Same issue here,  from some of my experiments:\n\nSeResNext50,101 -&gt; 0.9~1.1% gap\nEfficientNet b3-b5 -&gt; 0.9~1.1% gap\nWideRes50,101 -&gt; 1.0~1.3% gap\n\nIt may be caused by different distribution and unseen combinations mentioned in this [post](https://www.kaggle.com/c/bengaliai-cv19/discussion/131078),\nI'v tried several augmentation methods but seems didn't work,\nStill looking for a way to deal with it.",
    "757906": "My gap is smaller. Single model CV 0.989 and LB 0.984.",
    "757924": "Wow, the gap is really small!\n\nIt is interesting that some people have smaller gaps.\nMy best model performs CV 0.99 but LB 0.98, just thought if minimizing this gap may move my LB up",
    "758562": "thank you for sharing the experiments.\nso it may be normal to have a CV-LB gap, regradress of model architectures.\n\nI was wondering if augumentations contribute to these gaps or not.\nBut good augumentations will both push up LB+CV, so it may not be able to split the effects.",
    "758578": "I have really big gap. cv0.994, lb 0.9770.",
    "758581": "I tried a few different random seeds but none of them make my gaps smaller. Although augmentation seems to be an important factor, I don't quite understand how it could make an impact in CV/LB difference.",
    "758643": "I think your gap seems very normal, you can refer to [here](https://www.kaggle.com/c/bengaliai-cv19/discussion/123198)",
    "758646": "your results are promising, thanks for sharing.\nI guess one approach is trusting my CV and try to push it as much as possible; LB will follow."
  },
  "source": "meta"
}