{
  "id": 185798,
  "title": "How about your CV and LB?",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/185798",
  "author_name": "",
  "post_date": "2020-09-22T06:30:44.405274Z",
  "votes": 6,
  "comment_count": 11,
  "views": 0,
  "content": "<p>Now it's the end of this competition.<br>\nI tried several methods and my local CV is around -6.6 in most methods.(Leak free CV)<br>\nAnd the Public LB is about -6.88~-6.91<br>\nIt's seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)<br>\nWhat will you try in the next 2 weeks?</p>",
  "messages": [
    {
      "id": "1021818",
      "postDate": "09/22/2020 06:30:44",
      "content": "<p>Now it's the end of this competition.<br>\nI tried several methods and my local CV is around -6.6 in most methods.(Leak free CV)<br>\nAnd the Public LB is about -6.88~-6.91<br>\nIt's seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)<br>\nWhat will you try in the next 2 weeks?</p>",
      "rawMarkdown": "Now it's the end of this competition.\nI tried several methods and my local CV is around -6.6 in most methods.(Leak free CV)\nAnd the Public LB is about -6.88~-6.91\nIt's seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)\nWhat will you try in the next 2 weeks?",
      "votes": null
    },
    {
      "id": "1021839",
      "postDate": "09/22/2020 06:52:40",
      "content": "<p>May be recheck the values for each parameter and hyperparameter!</p>",
      "rawMarkdown": "May be recheck the values for each parameter and hyperparameter!",
      "votes": null
    },
    {
      "id": "1022939",
      "postDate": "09/22/2020 20:38:03",
      "content": "<p>Same here. my local CV is coming out to be 6.7 approx but my public LB is 6.89.<br>\nAlso, can we sort of like confirm that the image data models are not performing better in this competition?</p>",
      "rawMarkdown": "Same here. my local CV is coming out to be 6.7 approx but my public LB is 6.89.\nAlso, can we sort of like confirm that the image data models are not performing better in this competition?",
      "votes": null
    },
    {
      "id": "1023184",
      "postDate": "09/23/2020 02:33:18",
      "content": "<p>Which method of CV have you used <a href=\"https://www.kaggle.com/akshatshreemali91\" target=\"_blank\">@akshatshreemali91</a> <a href=\"https://www.kaggle.com/gzl0506\" target=\"_blank\">@gzl0506</a> <a href=\"https://www.kaggle.com/alifrahman\" target=\"_blank\">@alifrahman</a> ?</p>",
      "rawMarkdown": "Which method of CV have you used @akshatshreemali91 @gzl0506 @alifrahman ?",
      "votes": null
    },
    {
      "id": "1023191",
      "postDate": "09/23/2020 02:52:19",
      "content": "<p>Groupkfold</p>",
      "rawMarkdown": "Groupkfold",
      "votes": null
    },
    {
      "id": "1023215",
      "postDate": "09/23/2020 03:29:44",
      "content": "<p>groupkfold</p>",
      "rawMarkdown": "groupkfold",
      "votes": null
    },
    {
      "id": "1023898",
      "postDate": "09/23/2020 14:15:37",
      "content": "<p>That's really weird I've made some tricks and got a very promising CV with tabnet : -6.24<br>\nBut LB is arround -7 ..</p>",
      "rawMarkdown": "That's really weird I've made some tricks and got a very promising CV with tabnet : -6.24\nBut LB is arround -7 ..",
      "votes": null
    },
    {
      "id": "1024013",
      "postDate": "09/23/2020 15:31:05",
      "content": "<p>In my opinion, while calculating the CV score we should only consider the last 3 predictions for each patient and then compare that with LB scores. I have seen people saying that CV and LB do not correlate, I feel this is because they calculate CV based on all weeks and the LB is only scored on last 3 weeks. After computing the CV based on last three weeks I see a good correlation between CV and LB. </p>\n<blockquote>\n  <p>It seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)</p>\n</blockquote>\n<p>I strongly believe these models are overfitting because even seed changes lead to a big drop in the score. I have built a few ensembles which have a much better CV than models in public notebook but score around -6.88xx on LB. </p>",
      "rawMarkdown": "In my opinion, while calculating the CV score we should only consider the last 3 predictions for each patient and then compare that with LB scores. I have seen people saying that CV and LB do not correlate, I feel this is because they calculate CV based on all weeks and the LB is only scored on last 3 weeks. After computing the CV based on last three weeks I see a good correlation between CV and LB. \n\n> It seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)\n\nI strongly believe these models are overfitting because even seed changes lead to a big drop in the score. I have built a few ensembles which have a much better CV than models in public notebook but score around -6.88xx on LB.",
      "votes": null
    },
    {
      "id": "1024072",
      "postDate": "09/23/2020 16:18:16",
      "content": "<p>thanks! Let me try to explore this route. I am getting similar results on public LB. though my cv is 6.6 on avg</p>",
      "rawMarkdown": "thanks! Let me try to explore this route. I am getting similar results on public LB. though my cv is 6.6 on avg",
      "votes": null
    },
    {
      "id": "1024107",
      "postDate": "09/23/2020 16:48:17",
      "content": "<p>For my easy models, CV and LB correlates very well. But that's only true to a LB score of about 6.90</p>",
      "rawMarkdown": "For my easy models, CV and LB correlates very well. But that's only true to a LB score of about 6.90",
      "votes": null
    },
    {
      "id": "1024299",
      "postDate": "09/23/2020 18:35:31",
      "content": "<p>Yeah! I am slightly worried about what private LB would be. </p>",
      "rawMarkdown": "Yeah! I am slightly worried about what private LB would be.",
      "votes": null
    },
    {
      "id": "1024589",
      "postDate": "09/24/2020 02:12:14",
      "content": "<p>Image features improve the CV scores.</p>",
      "rawMarkdown": "Image features improve the CV scores.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1021839,
      "author_name": "alifrahman",
      "author_url": "",
      "post_date": "09/22/2020 06:52:40",
      "content": "<p>May be recheck the values for each parameter and hyperparameter!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1022939,
      "author_name": "akshatshreemali91",
      "author_url": "",
      "post_date": "09/22/2020 20:38:03",
      "content": "<p>Same here. my local CV is coming out to be 6.7 approx but my public LB is 6.89.<br>\nAlso, can we sort of like confirm that the image data models are not performing better in this competition?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1024589,
          "author_name": "abhishekgbhat",
          "author_url": "",
          "post_date": "09/24/2020 02:12:14",
          "content": "<p>Image features improve the CV scores.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1023184,
      "author_name": "kurianbenoy",
      "author_url": "",
      "post_date": "09/23/2020 02:33:18",
      "content": "<p>Which method of CV have you used <a href=\"https://www.kaggle.com/akshatshreemali91\" target=\"_blank\">@akshatshreemali91</a> <a href=\"https://www.kaggle.com/gzl0506\" target=\"_blank\">@gzl0506</a> <a href=\"https://www.kaggle.com/alifrahman\" target=\"_blank\">@alifrahman</a> ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1023191,
          "author_name": "gzl0506",
          "author_url": "",
          "post_date": "09/23/2020 02:52:19",
          "content": "<p>Groupkfold</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1023215,
          "author_name": "akshatshreemali91",
          "author_url": "",
          "post_date": "09/23/2020 03:29:44",
          "content": "<p>groupkfold</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1023898,
      "author_name": "alexj21",
      "author_url": "",
      "post_date": "09/23/2020 14:15:37",
      "content": "<p>That's really weird I've made some tricks and got a very promising CV with tabnet : -6.24<br>\nBut LB is arround -7 ..</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1024013,
      "author_name": "abhishekgbhat",
      "author_url": "",
      "post_date": "09/23/2020 15:31:05",
      "content": "<p>In my opinion, while calculating the CV score we should only consider the last 3 predictions for each patient and then compare that with LB scores. I have seen people saying that CV and LB do not correlate, I feel this is because they calculate CV based on all weeks and the LB is only scored on last 3 weeks. After computing the CV based on last three weeks I see a good correlation between CV and LB. </p>\n<blockquote>\n  <p>It seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)</p>\n</blockquote>\n<p>I strongly believe these models are overfitting because even seed changes lead to a big drop in the score. I have built a few ensembles which have a much better CV than models in public notebook but score around -6.88xx on LB. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1024072,
          "author_name": "akshatshreemali91",
          "author_url": "",
          "post_date": "09/23/2020 16:18:16",
          "content": "<p>thanks! Let me try to explore this route. I am getting similar results on public LB. though my cv is 6.6 on avg</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1024107,
      "author_name": "ilu000",
      "author_url": "",
      "post_date": "09/23/2020 16:48:17",
      "content": "<p>For my easy models, CV and LB correlates very well. But that's only true to a LB score of about 6.90</p>",
      "votes": null,
      "replies": [
        {
          "id": 1024299,
          "author_name": "akshatshreemali91",
          "author_url": "",
          "post_date": "09/23/2020 18:35:31",
          "content": "<p>Yeah! I am slightly worried about what private LB would be. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1021818": "Now it's the end of this competition.\nI tried several methods and my local CV is around -6.6 in most methods.(Leak free CV)\nAnd the Public LB is about -6.88~-6.91\nIt's seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)\nWhat will you try in the next 2 weeks?",
    "1021839": "May be recheck the values for each parameter and hyperparameter!",
    "1022939": "Same here. my local CV is coming out to be 6.7 approx but my public LB is 6.89.\nAlso, can we sort of like confirm that the image data models are not performing better in this competition?",
    "1023184": "Which method of CV have you used @akshatshreemali91 @gzl0506 @alifrahman ?",
    "1023191": "Groupkfold",
    "1023215": "groupkfold",
    "1023898": "That's really weird I've made some tricks and got a very promising CV with tabnet : -6.24\nBut LB is arround -7 ..",
    "1024013": "In my opinion, while calculating the CV score we should only consider the last 3 predictions for each patient and then compare that with LB scores. I have seen people saying that CV and LB do not correlate, I feel this is because they calculate CV based on all weeks and the LB is only scored on last 3 weeks. After computing the CV based on last three weeks I see a good correlation between CV and LB. \n\n> It seems that it's difficult to get about -6.8 but several public NB get there with simple model.(Maybe over-fitting?)\n\nI strongly believe these models are overfitting because even seed changes lead to a big drop in the score. I have built a few ensembles which have a much better CV than models in public notebook but score around -6.88xx on LB.",
    "1024072": "thanks! Let me try to explore this route. I am getting similar results on public LB. though my cv is 6.6 on avg",
    "1024107": "For my easy models, CV and LB correlates very well. But that's only true to a LB score of about 6.90",
    "1024299": "Yeah! I am slightly worried about what private LB would be.",
    "1024589": "Image features improve the CV scores."
  },
  "source": "meta"
}