{
  "id": 172816,
  "title": "Inconsistant CV and LB when adding Metadata",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/172816",
  "author_name": "",
  "post_date": "2020-08-06T15:03:32.143199900Z",
  "votes": 9,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I wonder if anyone achieved an improvement of both LB and local CV scores while combining metadata part with image part ?</p>\n\n<p>With models trained using images only I got better LB as well as CV when blending. Then I blended this result with metadata part, the results published by Giba (0.6928 LB), I continued to increase my LB score.</p>\n\n<p>Next, I generated an oof using Giba's rules and Chris's Triple Stratified Split on training data in order to find the best combination on local. The metadata CV is scored 0.64-0.65 which seems consistant with its LB. </p>\n\n<p>However, adding the metadata oof into image oofs my CV decreased a lot ! About 0.94 to 0.90 ! </p>\n\n<p>Any explanation why the same manipulation result in completely different behaviors ? I do not think it is a simple overfitting to LB because the first solution in ISIC also combined metadata with images.</p>",
  "messages": [
    {
      "id": "960640",
      "postDate": "08/06/2020 15:03:32",
      "content": "<p>I wonder if anyone achieved an improvement of both LB and local CV scores while combining metadata part with image part ?</p>\n\n<p>With models trained using images only I got better LB as well as CV when blending. Then I blended this result with metadata part, the results published by Giba (0.6928 LB), I continued to increase my LB score.</p>\n\n<p>Next, I generated an oof using Giba's rules and Chris's Triple Stratified Split on training data in order to find the best combination on local. The metadata CV is scored 0.64-0.65 which seems consistant with its LB. </p>\n\n<p>However, adding the metadata oof into image oofs my CV decreased a lot ! About 0.94 to 0.90 ! </p>\n\n<p>Any explanation why the same manipulation result in completely different behaviors ? I do not think it is a simple overfitting to LB because the first solution in ISIC also combined metadata with images.</p>",
      "rawMarkdown": "I wonder if anyone achieved an improvement of both LB and local CV scores while combining metadata part with image part ?\n\nWith models trained using images only I got better LB as well as CV when blending. Then I blended this result with metadata part, the results published by Giba (0.6928 LB), I continued to increase my LB score.\n\nNext, I generated an oof using Giba's rules and Chris's Triple Stratified Split on training data in order to find the best combination on local. The metadata CV is scored 0.64-0.65 which seems consistant with its LB. \n\nHowever, adding the metadata oof into image oofs my CV decreased a lot ! About 0.94 to 0.90 ! \n\nAny explanation why the same manipulation result in completely different behaviors ? I do not think it is a simple overfitting to LB because the first solution in ISIC also combined metadata with images.",
      "votes": null
    },
    {
      "id": "960946",
      "postDate": "08/06/2020 20:05:44",
      "content": "<p>There is completely new set of test images not included in train images at all so when you CV it's based on image features in your train data but meta predictions somewhat more generalized (independent from image attributes) so it does little bit better on unseen data? </p>",
      "rawMarkdown": "There is completely new set of test images not included in train images at all so when you CV it's based on image features in your train data but meta predictions somewhat more generalized (independent from image attributes) so it does little bit better on unseen data?",
      "votes": null
    },
    {
      "id": "961022",
      "postDate": "08/06/2020 21:10:24",
      "content": "<p>This could be a possibility. I will check that as using weighted average to combine metadata is expensive. The weights need to be adjusted each time when adding a new model.\nBtw, thanks for your public kernels.</p>",
      "rawMarkdown": "This could be a possibility. I will check that as using weighted average to combine metadata is expensive. The weights need to be adjusted each time when adding a new model.\nBtw, thanks for your public kernels.",
      "votes": null
    },
    {
      "id": "961370",
      "postDate": "08/07/2020 05:59:14",
      "content": "<p>In discussions noted that metadata inconsistent in Test. For example the same images have different annatom site or even patient_id. Maybe it is also impact on public lb</p>",
      "rawMarkdown": "In discussions noted that metadata inconsistent in Test. For example the same images have different annatom site or even patient_id. Maybe it is also impact on public lb",
      "votes": null
    },
    {
      "id": "961467",
      "postDate": "08/07/2020 07:48:19",
      "content": "<p>Maybe. CV or LB, so far I don't know which one to trust... \nEven CV gap is a question in this comp.</p>",
      "rawMarkdown": "Maybe. CV or LB, so far I don't know which one to trust... \nEven CV gap is a question in this comp.",
      "votes": null
    },
    {
      "id": "963653",
      "postDate": "08/09/2020 07:35:45",
      "content": "<p>When I combine meta-data, CV drops by 0.0015, but LB increases by 0.0015, fair trade I guess. (Use my own model , gradient boosted trees on meta data).</p>",
      "rawMarkdown": "When I combine meta-data, CV drops by 0.0015, but LB increases by 0.0015, fair trade I guess. (Use my own model , gradient boosted trees on meta data).",
      "votes": null
    },
    {
      "id": "964178",
      "postDate": "08/09/2020 16:44:54",
      "content": "<p>I have been suffering similar symptoms like you are suffering</p>",
      "rawMarkdown": "I have been suffering similar symptoms like you are suffering",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 960946,
      "author_name": "datafan07",
      "author_url": "",
      "post_date": "08/06/2020 20:05:44",
      "content": "<p>There is completely new set of test images not included in train images at all so when you CV it's based on image features in your train data but meta predictions somewhat more generalized (independent from image attributes) so it does little bit better on unseen data? </p>",
      "votes": null,
      "replies": [
        {
          "id": 961022,
          "author_name": "vicioussong",
          "author_url": "",
          "post_date": "08/06/2020 21:10:24",
          "content": "<p>This could be a possibility. I will check that as using weighted average to combine metadata is expensive. The weights need to be adjusted each time when adding a new model.\nBtw, thanks for your public kernels.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 961370,
      "author_name": "aybatov",
      "author_url": "",
      "post_date": "08/07/2020 05:59:14",
      "content": "<p>In discussions noted that metadata inconsistent in Test. For example the same images have different annatom site or even patient_id. Maybe it is also impact on public lb</p>",
      "votes": null,
      "replies": [
        {
          "id": 961467,
          "author_name": "vicioussong",
          "author_url": "",
          "post_date": "08/07/2020 07:48:19",
          "content": "<p>Maybe. CV or LB, so far I don't know which one to trust... \nEven CV gap is a question in this comp.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 963653,
      "author_name": "manjeshg03",
      "author_url": "",
      "post_date": "08/09/2020 07:35:45",
      "content": "<p>When I combine meta-data, CV drops by 0.0015, but LB increases by 0.0015, fair trade I guess. (Use my own model , gradient boosted trees on meta data).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 964178,
      "author_name": "deepkim",
      "author_url": "",
      "post_date": "08/09/2020 16:44:54",
      "content": "<p>I have been suffering similar symptoms like you are suffering</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "960640": "I wonder if anyone achieved an improvement of both LB and local CV scores while combining metadata part with image part ?\n\nWith models trained using images only I got better LB as well as CV when blending. Then I blended this result with metadata part, the results published by Giba (0.6928 LB), I continued to increase my LB score.\n\nNext, I generated an oof using Giba's rules and Chris's Triple Stratified Split on training data in order to find the best combination on local. The metadata CV is scored 0.64-0.65 which seems consistant with its LB. \n\nHowever, adding the metadata oof into image oofs my CV decreased a lot ! About 0.94 to 0.90 ! \n\nAny explanation why the same manipulation result in completely different behaviors ? I do not think it is a simple overfitting to LB because the first solution in ISIC also combined metadata with images.",
    "960946": "There is completely new set of test images not included in train images at all so when you CV it's based on image features in your train data but meta predictions somewhat more generalized (independent from image attributes) so it does little bit better on unseen data?",
    "961022": "This could be a possibility. I will check that as using weighted average to combine metadata is expensive. The weights need to be adjusted each time when adding a new model.\nBtw, thanks for your public kernels.",
    "961370": "In discussions noted that metadata inconsistent in Test. For example the same images have different annatom site or even patient_id. Maybe it is also impact on public lb",
    "961467": "Maybe. CV or LB, so far I don't know which one to trust... \nEven CV gap is a question in this comp.",
    "963653": "When I combine meta-data, CV drops by 0.0015, but LB increases by 0.0015, fair trade I guess. (Use my own model , gradient boosted trees on meta data).",
    "964178": "I have been suffering similar symptoms like you are suffering"
  },
  "source": "meta"
}