{
  "id": 122016,
  "title": "How do you trust your local CV?",
  "url": "/competitions/pku-autonomous-driving/discussion/122016",
  "author_name": "",
  "post_date": "2019-12-17T05:38:53.949566600Z",
  "votes": 6,
  "comment_count": 11,
  "views": 0,
  "content": "<p>Previously I found my <strong>cv=0.14 model could only score 0.045 lb</strong>...but the story behind cv=0.14 is like:\n- you can use raw images for evaluation\n- you can also add image augmentation before evaluation\n- you can add different image augmentations  </p>\n\n<p><strong>For the same model, tuning augmentation seed result in local cv varying from 0.11~0.14.</strong></p>\n\n<p>And you may wonder why even adding augmentation on valid-set? Like I said in the notebook, the test images we are predicting is different from what we are training and evaluating locally.\n<a href=\"https://www.kaggle.com/niuddd/image-augmentations-for-pku-self-driving-car\">image-augmentations-for-pku-self-driving-car</a></p>\n\n<p>Is your local cv on raw images from train_images folder? And how much is it consistent with lb score?</p>",
  "messages": [
    {
      "id": "696829",
      "postDate": "12/17/2019 05:38:53",
      "content": "<p>Previously I found my <strong>cv=0.14 model could only score 0.045 lb</strong>...but the story behind cv=0.14 is like:\n- you can use raw images for evaluation\n- you can also add image augmentation before evaluation\n- you can add different image augmentations  </p>\n\n<p><strong>For the same model, tuning augmentation seed result in local cv varying from 0.11~0.14.</strong></p>\n\n<p>And you may wonder why even adding augmentation on valid-set? Like I said in the notebook, the test images we are predicting is different from what we are training and evaluating locally.\n<a href=\"https://www.kaggle.com/niuddd/image-augmentations-for-pku-self-driving-car\">image-augmentations-for-pku-self-driving-car</a></p>\n\n<p>Is your local cv on raw images from train_images folder? And how much is it consistent with lb score?</p>",
      "rawMarkdown": "Previously I found my **cv=0.14 model could only score 0.045 lb**...but the story behind cv=0.14 is like:\n- you can use raw images for evaluation\n- you can also add image augmentation before evaluation\n- you can add different image augmentations  \n\n**For the same model, tuning augmentation seed result in local cv varying from 0.11~0.14.**\n\nAnd you may wonder why even adding augmentation on valid-set? Like I said in the notebook, the test images we are predicting is different from what we are training and evaluating locally.\n[image-augmentations-for-pku-self-driving-car](https://www.kaggle.com/niuddd/image-augmentations-for-pku-self-driving-car)\n\nIs your local cv on raw images from train_images folder? And how much is it consistent with lb score?",
      "votes": null
    },
    {
      "id": "698265",
      "postDate": "12/19/2019 01:56:39",
      "content": "<p>Did you use the way that in public kernel to calculate the local CV? </p>",
      "rawMarkdown": "Did you use the way that in public kernel to calculate the local CV?",
      "votes": null
    },
    {
      "id": "698283",
      "postDate": "12/19/2019 02:40:08",
      "content": "<p>Yes. And also tried different thresholds for evaluation.</p>",
      "rawMarkdown": "Yes. And also tried different thresholds for evaluation.",
      "votes": null
    },
    {
      "id": "707606",
      "postDate": "01/01/2020 06:16:41",
      "content": "<p>My CV was quite consistent with LB before. But after I got LB over 0.07, the CV drops a lot...</p>\n\n<p>LB 0.056 CV 0.103\nLB 0.06 CV 0.113\nLB 0.064, CV 0.122\nLB 0.069, CV 0.15\nLB 0.076, CV 0.12</p>\n\n<p>But I tend to trust the CV though..</p>",
      "rawMarkdown": "My CV was quite consistent with LB before. But after I got LB over 0.07, the CV drops a lot...\n\nLB 0.056 CV 0.103\nLB 0.06 CV 0.113\nLB 0.064, CV 0.122\nLB 0.069, CV 0.15\nLB 0.076, CV 0.12\n\nBut I tend to trust the CV though..",
      "votes": null
    },
    {
      "id": "707624",
      "postDate": "01/01/2020 07:21:00",
      "content": "<p>we can also augument test images during inference.\nhave you computed the LB with/without augumentations? (I might try that..)</p>",
      "rawMarkdown": "we can also augument test images during inference.\nhave you computed the LB with/without augumentations? (I might try that..)",
      "votes": null
    },
    {
      "id": "707671",
      "postDate": "01/01/2020 09:19:13",
      "content": "<p>I didn't perform augmentation on validation or testing set. Testing set seems has augmentation already?</p>",
      "rawMarkdown": "I didn't perform augmentation on validation or testing set. Testing set seems has augmentation already?",
      "votes": null
    },
    {
      "id": "707767",
      "postDate": "01/01/2020 13:04:29",
      "content": "<p>Does your CV evaluation penalty for false positives? Does your comparison at same level of predicted positives? <a href=\"/xiejialun\">@xiejialun</a> </p>",
      "rawMarkdown": "Does your CV evaluation penalty for false positives? Does your comparison at same level of predicted positives? @xiejialun",
      "votes": null
    },
    {
      "id": "707814",
      "postDate": "01/01/2020 14:22:10",
      "content": "<p>Did you mean false negative? I use the same public kernel you used to calculate the map, and the number is based on score including false negative.</p>\n\n<p>I don't understand the second question, could you explain to me?</p>",
      "rawMarkdown": "Did you mean false negative? I use the same public kernel you used to calculate the map, and the number is based on score including false negative.\n\nI don't understand the second question, could you explain to me?",
      "votes": null
    },
    {
      "id": "708490",
      "postDate": "01/02/2020 10:53:29",
      "content": "<p>Similar to your results　<a href=\"/xiejialun\">@xiejialun</a> </p>",
      "rawMarkdown": "Similar to your results　@xiejialun",
      "votes": null
    },
    {
      "id": "708583",
      "postDate": "01/02/2020 12:57:58",
      "content": "<p>Hi, thanks for your notebook. is there any improvement in your notebook by using image augmentations?</p>",
      "rawMarkdown": "Hi, thanks for your notebook. is there any improvement in your notebook by using image augmentations?",
      "votes": null
    },
    {
      "id": "717864",
      "postDate": "01/13/2020 18:02:01",
      "content": "<p>Which mAP calculation notebook did you use?</p>",
      "rawMarkdown": "Which mAP calculation notebook did you use?",
      "votes": null
    },
    {
      "id": "717867",
      "postDate": "01/13/2020 18:02:35",
      "content": "<p><a href=\"/xiejialun\">@xiejialun</a> Which metric calculation kernel did you use?</p>",
      "rawMarkdown": "xiejialun Which metric calculation kernel did you use?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 698265,
      "author_name": "xiejialun",
      "author_url": "",
      "post_date": "12/19/2019 01:56:39",
      "content": "<p>Did you use the way that in public kernel to calculate the local CV? </p>",
      "votes": null,
      "replies": [
        {
          "id": 698283,
          "author_name": "niuddd",
          "author_url": "",
          "post_date": "12/19/2019 02:40:08",
          "content": "<p>Yes. And also tried different thresholds for evaluation.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 707606,
          "author_name": "xiejialun",
          "author_url": "",
          "post_date": "01/01/2020 06:16:41",
          "content": "<p>My CV was quite consistent with LB before. But after I got LB over 0.07, the CV drops a lot...</p>\n\n<p>LB 0.056 CV 0.103\nLB 0.06 CV 0.113\nLB 0.064, CV 0.122\nLB 0.069, CV 0.15\nLB 0.076, CV 0.12</p>\n\n<p>But I tend to trust the CV though..</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 707624,
          "author_name": "kyoshioka47",
          "author_url": "",
          "post_date": "01/01/2020 07:21:00",
          "content": "<p>we can also augument test images during inference.\nhave you computed the LB with/without augumentations? (I might try that..)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 707671,
          "author_name": "xiejialun",
          "author_url": "",
          "post_date": "01/01/2020 09:19:13",
          "content": "<p>I didn't perform augmentation on validation or testing set. Testing set seems has augmentation already?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 707767,
          "author_name": "niuddd",
          "author_url": "",
          "post_date": "01/01/2020 13:04:29",
          "content": "<p>Does your CV evaluation penalty for false positives? Does your comparison at same level of predicted positives? <a href=\"/xiejialun\">@xiejialun</a> </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 707814,
          "author_name": "xiejialun",
          "author_url": "",
          "post_date": "01/01/2020 14:22:10",
          "content": "<p>Did you mean false negative? I use the same public kernel you used to calculate the map, and the number is based on score including false negative.</p>\n\n<p>I don't understand the second question, could you explain to me?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 717867,
          "author_name": "tonychenxyz",
          "author_url": "",
          "post_date": "01/13/2020 18:02:35",
          "content": "<p><a href=\"/xiejialun\">@xiejialun</a> Which metric calculation kernel did you use?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 708490,
      "author_name": "miraclebcool",
      "author_url": "",
      "post_date": "01/02/2020 10:53:29",
      "content": "<p>Similar to your results　<a href=\"/xiejialun\">@xiejialun</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 708583,
      "author_name": "uestctubiao",
      "author_url": "",
      "post_date": "01/02/2020 12:57:58",
      "content": "<p>Hi, thanks for your notebook. is there any improvement in your notebook by using image augmentations?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 717864,
      "author_name": "tonychenxyz",
      "author_url": "",
      "post_date": "01/13/2020 18:02:01",
      "content": "<p>Which mAP calculation notebook did you use?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "696829": "Previously I found my **cv=0.14 model could only score 0.045 lb**...but the story behind cv=0.14 is like:\n- you can use raw images for evaluation\n- you can also add image augmentation before evaluation\n- you can add different image augmentations  \n\n**For the same model, tuning augmentation seed result in local cv varying from 0.11~0.14.**\n\nAnd you may wonder why even adding augmentation on valid-set? Like I said in the notebook, the test images we are predicting is different from what we are training and evaluating locally.\n[image-augmentations-for-pku-self-driving-car](https://www.kaggle.com/niuddd/image-augmentations-for-pku-self-driving-car)\n\nIs your local cv on raw images from train_images folder? And how much is it consistent with lb score?",
    "698265": "Did you use the way that in public kernel to calculate the local CV?",
    "698283": "Yes. And also tried different thresholds for evaluation.",
    "707606": "My CV was quite consistent with LB before. But after I got LB over 0.07, the CV drops a lot...\n\nLB 0.056 CV 0.103\nLB 0.06 CV 0.113\nLB 0.064, CV 0.122\nLB 0.069, CV 0.15\nLB 0.076, CV 0.12\n\nBut I tend to trust the CV though..",
    "707624": "we can also augument test images during inference.\nhave you computed the LB with/without augumentations? (I might try that..)",
    "707671": "I didn't perform augmentation on validation or testing set. Testing set seems has augmentation already?",
    "707767": "Does your CV evaluation penalty for false positives? Does your comparison at same level of predicted positives? @xiejialun",
    "707814": "Did you mean false negative? I use the same public kernel you used to calculate the map, and the number is based on score including false negative.\n\nI don't understand the second question, could you explain to me?",
    "708490": "Similar to your results　@xiejialun",
    "708583": "Hi, thanks for your notebook. is there any improvement in your notebook by using image augmentations?",
    "717864": "Which mAP calculation notebook did you use?",
    "717867": "xiejialun Which metric calculation kernel did you use?"
  },
  "source": "meta"
}