{
  "id": 98606,
  "title": "My LB score is too low compared to validation score",
  "url": "/competitions/aptos2019-blindness-detection/discussion/98606",
  "author_name": "",
  "post_date": "2019-07-05T01:02:13.619534600Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I trained resnet50 via cross validation. (with 4 stratified validation set)\ncross entropy for loss</p>\n\n<p>Final validation loss is about 0.5 \nFinal validation kappa is about 0.85</p>\n\n<p>But after submitting, the score is under 0 (~-0.02)</p>\n\n<p>what happened in my case?</p>",
  "messages": [
    {
      "id": "568463",
      "postDate": "07/05/2019 01:02:13",
      "content": "<p>I trained resnet50 via cross validation. (with 4 stratified validation set)\ncross entropy for loss</p>\n\n<p>Final validation loss is about 0.5 \nFinal validation kappa is about 0.85</p>\n\n<p>But after submitting, the score is under 0 (~-0.02)</p>\n\n<p>what happened in my case?</p>",
      "rawMarkdown": "I trained resnet50 via cross validation. (with 4 stratified validation set)\ncross entropy for loss\n\nFinal validation loss is about 0.5 \nFinal validation kappa is about 0.85\n\nBut after submitting, the score is under 0 (~-0.02)\n\nwhat happened in my case?",
      "votes": null
    },
    {
      "id": "568483",
      "postDate": "07/05/2019 01:59:21",
      "content": "<p>Some possible reasons:</p>\n\n<ol>\n<li>leak between folds: Check whether your validation loss becomes lower and lower in each fold.</li>\n<li>The inference part differs from the training part: Check whether you perform the same preprocessing on the testing data as the processing techniques in your training procedure.</li>\n</ol>",
      "rawMarkdown": "Some possible reasons:\n\n1. leak between folds: Check whether your validation loss becomes lower and lower in each fold.\n2. The inference part differs from the training part: Check whether you perform the same preprocessing on the testing data as the processing techniques in your training procedure.",
      "votes": null
    },
    {
      "id": "568677",
      "postDate": "07/05/2019 09:44:53",
      "content": "<p>I think ot's becuase of data leaking between valid and train set, since two loss graph's fluctuation looks same</p>\n\n<p>But, I can't figure out leakage on the following code:</p>\n\n<p>for train_index, valid_index in skf.split(IDs, labels):\n    print(\"============fold {}============\".format(fold))\n    K.clear_session()</p>\n\n<pre><code>train_IDs, train_labels = IDs[train_index],labels[train_index]\nvalid_IDs, valid_labels = IDs[valid_index],labels[valid_index]\ntrain_labels = {ID : value for ID,value in zip(train_IDs,train_labels)}\nvalid_labels = {ID : value for ID,value in zip(valid_IDs,valid_labels)}\n</code></pre>\n\n<p>..... more codes</p>",
      "rawMarkdown": "I think ot's becuase of data leaking between valid and train set, since two loss graph's fluctuation looks same\n\nBut, I can't figure out leakage on the following code:\n\nfor train_index, valid_index in skf.split(IDs, labels):\n    print(\"============fold {}============\".format(fold))\n    K.clear_session()\n\n    train_IDs, train_labels = IDs[train_index],labels[train_index]\n    valid_IDs, valid_labels = IDs[valid_index],labels[valid_index]\n    train_labels = {ID : value for ID,value in zip(train_IDs,train_labels)}\n    valid_labels = {ID : value for ID,value in zip(valid_IDs,valid_labels)}\n\n..... more codes",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 568483,
      "author_name": "playif1",
      "author_url": "",
      "post_date": "07/05/2019 01:59:21",
      "content": "<p>Some possible reasons:</p>\n\n<ol>\n<li>leak between folds: Check whether your validation loss becomes lower and lower in each fold.</li>\n<li>The inference part differs from the training part: Check whether you perform the same preprocessing on the testing data as the processing techniques in your training procedure.</li>\n</ol>",
      "votes": null,
      "replies": [
        {
          "id": 568677,
          "author_name": "woongjo",
          "author_url": "",
          "post_date": "07/05/2019 09:44:53",
          "content": "<p>I think ot's becuase of data leaking between valid and train set, since two loss graph's fluctuation looks same</p>\n\n<p>But, I can't figure out leakage on the following code:</p>\n\n<p>for train_index, valid_index in skf.split(IDs, labels):\n    print(\"============fold {}============\".format(fold))\n    K.clear_session()</p>\n\n<pre><code>train_IDs, train_labels = IDs[train_index],labels[train_index]\nvalid_IDs, valid_labels = IDs[valid_index],labels[valid_index]\ntrain_labels = {ID : value for ID,value in zip(train_IDs,train_labels)}\nvalid_labels = {ID : value for ID,value in zip(valid_IDs,valid_labels)}\n</code></pre>\n\n<p>..... more codes</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "568463": "I trained resnet50 via cross validation. (with 4 stratified validation set)\ncross entropy for loss\n\nFinal validation loss is about 0.5 \nFinal validation kappa is about 0.85\n\nBut after submitting, the score is under 0 (~-0.02)\n\nwhat happened in my case?",
    "568483": "Some possible reasons:\n\n1. leak between folds: Check whether your validation loss becomes lower and lower in each fold.\n2. The inference part differs from the training part: Check whether you perform the same preprocessing on the testing data as the processing techniques in your training procedure.",
    "568677": "I think ot's becuase of data leaking between valid and train set, since two loss graph's fluctuation looks same\n\nBut, I can't figure out leakage on the following code:\n\nfor train_index, valid_index in skf.split(IDs, labels):\n    print(\"============fold {}============\".format(fold))\n    K.clear_session()\n\n    train_IDs, train_labels = IDs[train_index],labels[train_index]\n    valid_IDs, valid_labels = IDs[valid_index],labels[valid_index]\n    train_labels = {ID : value for ID,value in zip(train_IDs,train_labels)}\n    valid_labels = {ID : value for ID,value in zip(valid_IDs,valid_labels)}\n\n..... more codes"
  },
  "source": "meta"
}