{
  "id": 37229,
  "title": "Mask errors and winning strategies",
  "url": "/competitions/carvana-image-masking-challenge/discussion/37229",
  "author_name": "",
  "post_date": "2017-07-29T10:42:31.398193700Z",
  "votes": 19,
  "comment_count": 43,
  "views": 0,
  "content": "<p>There are, understandably, small errors in the supplied training masks - see 2cb06c1f5bb1_05 for example:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/208304/6919/2cb06c1f5bb1_05_diffs_sm.jpg\" alt=\"segmentation differences for 2cb06c1f5bb1_05\" title=\"\"></p>\n\n<p>Given the high quality of the segmentation algorithms that we are likely to see, the top leaderboard margins could be very narrow. This has already been pointed out in topics elsewhere.  I wonder if winning the competition might come down to the algorithms that best emulate these human errors, rather than coming up with the optimal algorithm for the original purpose of the project.</p>\n\n<p>Perhaps I'm worrying too much, but it might be nice if competitors consider publishing, if possible, an alternative algorithm that is optimal as well as the one intended to win.</p>",
  "messages": [
    {
      "id": "208304",
      "postDate": "07/29/2017 10:42:31",
      "content": "<p>There are, understandably, small errors in the supplied training masks - see 2cb06c1f5bb1_05 for example:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/208304/6919/2cb06c1f5bb1_05_diffs_sm.jpg\" alt=\"segmentation differences for 2cb06c1f5bb1_05\" title=\"\"></p>\n\n<p>Given the high quality of the segmentation algorithms that we are likely to see, the top leaderboard margins could be very narrow. This has already been pointed out in topics elsewhere.  I wonder if winning the competition might come down to the algorithms that best emulate these human errors, rather than coming up with the optimal algorithm for the original purpose of the project.</p>\n\n<p>Perhaps I'm worrying too much, but it might be nice if competitors consider publishing, if possible, an alternative algorithm that is optimal as well as the one intended to win.</p>",
      "rawMarkdown": "There are, understandably, small errors in the supplied training masks - see 2cb06c1f5bb1_05 for example:\n\n![segmentation differences for 2cb06c1f5bb1_05][1]\n\nGiven the high quality of the segmentation algorithms that we are likely to see, the top leaderboard margins could be very narrow. This has already been pointed out in topics elsewhere.  I wonder if winning the competition might come down to the algorithms that best emulate these human errors, rather than coming up with the optimal algorithm for the original purpose of the project.\n\nPerhaps I'm worrying too much, but it might be nice if competitors consider publishing, if possible, an alternative algorithm that is optimal as well as the one intended to win.\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/208304/6919/2cb06c1f5bb1_05_diffs_sm.jpg",
      "votes": null
    },
    {
      "id": "208446",
      "postDate": "07/29/2017 20:10:15",
      "content": "<p>Great attention to detail!</p>\n\n<p>I wish our manual masks were more accurate and consistent. I would caution against over-fitting to the human errors found in the training set, as the errors in the final scoring set will be different. I've seen a couple of instances of a very unexpected part of vehicles being cut out, but it's rare enough that it would be detrimental to emulate and affect non-emulators uniformly.</p>\n\n<p>It's amazing to see results exceeding 0.99 already, but we still have a ways to go before details as small as these human errors come into play.</p>\n\n<p>I'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?</p>",
      "rawMarkdown": "Great attention to detail!\n\nI wish our manual masks were more accurate and consistent. I would caution against over-fitting to the human errors found in the training set, as the errors in the final scoring set will be different. I've seen a couple of instances of a very unexpected part of vehicles being cut out, but it's rare enough that it would be detrimental to emulate and affect non-emulators uniformly.\n\nIt's amazing to see results exceeding 0.99 already, but we still have a ways to go before details as small as these human errors come into play.\n\nI'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?",
      "votes": null
    },
    {
      "id": "208461",
      "postDate": "07/29/2017 21:10:49",
      "content": "<blockquote>\n  <p>I'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?</p>\n</blockquote>\n\n<p>Well that's good question. I can't think of a way without using an even better segmenter. If there was a way, it's a fair bet that there'd be a GAN involved somewhere! I just hope people don't lose focus on the real goal, as I've seen in some other competitions.</p>",
      "rawMarkdown": "&gt; I'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?\n\nWell that's good question. I can't think of a way without using an even better segmenter. If there was a way, it's a fair bet that there'd be a GAN involved somewhere! I just hope people don't lose focus on the real goal, as I've seen in some other competitions.",
      "votes": null
    },
    {
      "id": "208881",
      "postDate": "07/31/2017 13:34:48",
      "content": "<p>Hi, <br>\nYou have say that the errors on the test set will be different, \nCould you elaborate more?</p>\n\n<p>Have the images on the test set been labelled by another person for example?</p>",
      "rawMarkdown": "Hi,  \nYou have say that the errors on the test set will be different, \nCould you elaborate more?\n\nHave the images on the test set been labelled by another person for example?",
      "votes": null
    },
    {
      "id": "208883",
      "postDate": "07/31/2017 13:46:26",
      "content": "<p>There are no intentional errors, they are not labeled, and the data was completely randomized.</p>\n\n<p>If there are very few instances of a type of defect, they may by chance exist disproportionately in the training set, which would mean over-fitting for human error may result in worse outcomes in the scoring set.</p>\n\n<p>However, if the training masks commonly have a similar type of defect (preferring right angles, skipping gaps smaller than a specific size like the rightmost highlight above) then the highest scoring solution would need to have those traits even if an optimal algorithm doesn't.</p>",
      "rawMarkdown": "There are no intentional errors, they are not labeled, and the data was completely randomized.\n\nIf there are very few instances of a type of defect, they may by chance exist disproportionately in the training set, which would mean over-fitting for human error may result in worse outcomes in the scoring set.\n\nHowever, if the training masks commonly have a similar type of defect (preferring right angles, skipping gaps smaller than a specific size like the rightmost highlight above) then the highest scoring solution would need to have those traits even if an optimal algorithm doesn't.",
      "votes": null
    },
    {
      "id": "208938",
      "postDate": "07/31/2017 16:09:27",
      "content": "<p>Thanks for the info Brian</p>",
      "rawMarkdown": "Thanks for the info Brian",
      "votes": null
    },
    {
      "id": "209724",
      "postDate": "08/03/2017 07:03:14",
      "content": "<p>I also found small errors in following two train masks:\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif</p>",
      "rawMarkdown": "I also found small errors in following two train masks:\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif",
      "votes": null
    },
    {
      "id": "209742",
      "postDate": "08/03/2017 08:55:16",
      "content": "<p>Thanks, @JandJ! _01 is especially disappointing to see.</p>\n\n<p>I would be tempted to auto-fill fully enclosed gaps, but I'm not sure doing so wouldn't have unintended consequences (like the side runner/step on OP's truck)</p>",
      "rawMarkdown": "Thanks, @JandJ! _01 is especially disappointing to see.\n\nI would be tempted to auto-fill fully enclosed gaps, but I'm not sure doing so wouldn't have unintended consequences (like the side runner/step on OP's truck)",
      "votes": null
    },
    {
      "id": "210045",
      "postDate": "08/04/2017 06:12:01",
      "content": "<p>@JandJ\nYou have a script to automatically detect such error? I think we need to correct such labels by hand</p>",
      "rawMarkdown": "JandJ\nYou have a script to automatically detect such error? I think we need to correct such labels by hand",
      "votes": null
    },
    {
      "id": "210050",
      "postDate": "08/04/2017 06:19:41",
      "content": "<p>If you have camera parameters, a 3d reconstruction algorithm like visual hull extraction should be able to catch these label error.</p>\n\n<p>Also, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)</p>",
      "rawMarkdown": "If you have camera parameters, a 3d reconstruction algorithm like visual hull extraction should be able to catch these label error.\n\nAlso, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)",
      "votes": null
    },
    {
      "id": "210051",
      "postDate": "08/04/2017 06:20:32",
      "content": "<p>I just sorted output mask files from train set by DICE error - and it just stood out like this:\n(Yellow: false positive, Red: false negative)</p>\n\n<p><img src=\"https://image.ibb.co/cqUTCF/mask_err.png\" alt=\"enter image description here\" title=\"\"></p>",
      "rawMarkdown": "I just sorted output mask files from train set by DICE error - and it just stood out like this:\n(Yellow: false positive, Red: false negative)\n\n![enter image description here][1]\n\n  [1]: https://image.ibb.co/cqUTCF/mask_err.png",
      "votes": null
    },
    {
      "id": "211470",
      "postDate": "08/09/2017 05:05:35",
      "content": "<p>Whether to exclude the area between wheel spokes doesn't seem to be consistent. </p>\n\n<p>Another question: when making the ground truth, was there any semi-automated tool used, or was that fully manual?  </p>",
      "rawMarkdown": "Whether to exclude the area between wheel spokes doesn't seem to be consistent. \n\nAnother question: when making the ground truth, was there any semi-automated tool used, or was that fully manual?",
      "votes": null
    },
    {
      "id": "211532",
      "postDate": "08/09/2017 10:13:49",
      "content": "<p>We outsourced it. As far as we know, it was manual. Some of the mask errors do make it seem like a tool may have been used to assist.</p>",
      "rawMarkdown": "We outsourced it. As far as we know, it was manual. Some of the mask errors do make it seem like a tool may have been used to assist.",
      "votes": null
    },
    {
      "id": "211545",
      "postDate": "08/09/2017 10:34:01",
      "content": "<p>Did it take about 2-3 weeks to take the photos by any chance?</p>",
      "rawMarkdown": "Did it take about 2-3 weeks to take the photos by any chance?",
      "votes": null
    },
    {
      "id": "211555",
      "postDate": "08/09/2017 10:56:48",
      "content": "<p>No, the photos are from a long time period.</p>",
      "rawMarkdown": "No, the photos are from a long time period.",
      "votes": null
    },
    {
      "id": "211627",
      "postDate": "08/09/2017 15:42:49",
      "content": "<p>Interesting, most of variance of images averaged per angle is explained by under 20 principal components and after playing around a bit with those there seem to be similar amount of unique backgrounds ( different studios and camera angles ) . My estimate for number of those is 14</p>",
      "rawMarkdown": "Interesting, most of variance of images averaged per angle is explained by under 20 principal components and after playing around a bit with those there seem to be similar amount of unique backgrounds ( different studios and camera angles ) . My estimate for number of those is 14",
      "votes": null
    },
    {
      "id": "213016",
      "postDate": "08/13/2017 13:02:36",
      "content": "<p>some error train image masks:</p>\n\n<p>b98c63cd6102_02_mask</p>\n\n<p>d1a3af34e674_07_mask</p>\n\n<p>eaf9eb0b2293_01_mask</p>\n\n<p>eb91b1c659a0_10_mask</p>",
      "rawMarkdown": "some error train image masks:\n\nb98c63cd6102_02_mask\n\nd1a3af34e674_07_mask\n\neaf9eb0b2293_01_mask\n\neb91b1c659a0_10_mask",
      "votes": null
    },
    {
      "id": "213076",
      "postDate": "08/13/2017 16:38:51",
      "content": "<p>There are some images with large part of bottom of the  car excluded:\n0d53224da2b7_13\nc3dafdb02e7f_04</p>\n\n<p>Here is a comparison of two views one with error one without: 0d53224da2b7_13 and 0d53224da2b7_14.</p>",
      "rawMarkdown": "There are some images with large part of bottom of the  car excluded:\n0d53224da2b7_13\nc3dafdb02e7f_04\n\nHere is a comparison of two views one with error one without: 0d53224da2b7_13 and 0d53224da2b7_14.",
      "votes": null
    },
    {
      "id": "213081",
      "postDate": "08/13/2017 16:57:27",
      "content": "<p>in some cases the deep CNN outperform human annotations.</p>\n\n<p>here are some examples</p>\n\n<p>red outline = human annotation</p>\n\n<p>green outline = cnn prediction</p>\n\n<p>red area = miss pixels</p>\n\n<p>green area = false positive pixels</p>\n\n<p>grey area = true positive pixels</p>",
      "rawMarkdown": "in some cases the deep CNN outperform human annotations.\n\nhere are some examples\n\nred outline = human annotation\n\ngreen outline = cnn prediction\n\nred area = miss pixels\n\ngreen area = false positive pixels\n\ngrey area = true positive pixels",
      "votes": null
    },
    {
      "id": "213088",
      "postDate": "08/13/2017 17:29:34",
      "content": "<p>Updated with other posting(Total 31 cases) :</p>\n\n<pre>    \n0d1a9caf4350_02\n0d1a9caf4350_14\n0d53224da2b7_13 \n1390696b70b6_14\n189a2a32a615_02\n1ba84b81628e_06\n1e89e1af42e7_07\n23c088f6ec27_10\n2a4a8964ebf3_08\n2cb06c1f5bb1_04\n364923a5002f_03\n364fd5fd7569_06\n3cb21125f126_04\n3cb21125f126_05\n3cb21125f126_06\n3cb21125f126_12\n3cb21125f126_13\n3cb21125f126_14\n4e308ad8a254_14\n6ba36af67cb0_07\n791c1a9775be_06\n7bd1142155ae_08\n8d1a6723c458_01\n8d1a6723c458_08\nb98c63cd6102_02\nc3dafdb02e7f_04\nc6f50d44f141_09\nd1a3af34e674_07\neaf9eb0b2293_01\neb91b1c659a0_10\nfa613ac8eac5_02\n</pre>",
      "rawMarkdown": "Updated with other posting(Total 31 cases) :\n\n<pre>    \n0d1a9caf4350_02\n0d1a9caf4350_14\n0d53224da2b7_13 \n1390696b70b6_14\n189a2a32a615_02\n1ba84b81628e_06\n1e89e1af42e7_07\n23c088f6ec27_10\n2a4a8964ebf3_08\n2cb06c1f5bb1_04\n364923a5002f_03\n364fd5fd7569_06\n3cb21125f126_04\n3cb21125f126_05\n3cb21125f126_06\n3cb21125f126_12\n3cb21125f126_13\n3cb21125f126_14\n4e308ad8a254_14\n6ba36af67cb0_07\n791c1a9775be_06\n7bd1142155ae_08\n8d1a6723c458_01\n8d1a6723c458_08\nb98c63cd6102_02\nc3dafdb02e7f_04\nc6f50d44f141_09\nd1a3af34e674_07\neaf9eb0b2293_01\neb91b1c659a0_10\nfa613ac8eac5_02\n</pre>",
      "votes": null
    },
    {
      "id": "213092",
      "postDate": "08/13/2017 17:36:39",
      "content": "<p>Did anyone compare results before and after correcting human error in annotations? For me it seems that there is no visible improvement using original annotations and the corrected versions.</p>",
      "rawMarkdown": "Did anyone compare results before and after correcting human error in annotations? For me it seems that there is no visible improvement using original annotations and the corrected versions.",
      "votes": null
    },
    {
      "id": "213098",
      "postDate": "08/13/2017 18:02:19",
      "content": "<p>There are many more correct samples for similar shaped cases, so I guess this errors will not make much difference. I haven't tried to remove these cases from my train set yet.</p>",
      "rawMarkdown": "There are many more correct samples for similar shaped cases, so I guess this errors will not make much difference. I haven't tried to remove these cases from my train set yet.",
      "votes": null
    },
    {
      "id": "213103",
      "postDate": "08/13/2017 18:19:38",
      "content": "<p>I guess we are allowed to skip these during training, but do you know if we are allowed to fix them and then use them?</p>",
      "rawMarkdown": "I guess we are allowed to skip these during training, but do you know if we are allowed to fix them and then use them?",
      "votes": null
    },
    {
      "id": "213146",
      "postDate": "08/13/2017 21:19:14",
      "content": "<p>As far as I know from other competitions you can do with the train data whatever you want...</p>",
      "rawMarkdown": "As far as I know from other competitions you can do with the train data whatever you want...",
      "votes": null
    },
    {
      "id": "213313",
      "postDate": "08/14/2017 12:51:22",
      "content": "<p>Hi Brian,</p>\n\n<p>As there are already users that have generated masks with better quality than the originals, I would suggest to use these as the labels for the test. This comes with some heavy drawbacks for your team, though:</p>\n\n<ul>\n<li>Most part of submissions would need to be reevaluated, as their punctuation might change and place them in a relevant ranking position.</li>\n<li>Every time any submission gets a punctuation close to the better one, a visual inspection of the masks should be made in order to decide whether these new masks should replace the older ones as reference.</li>\n</ul>\n\n<p>This seems to me too difficult to overcome, but perhaps you find it feasible.</p>",
      "rawMarkdown": "Hi Brian,\n\nAs there are already users that have generated masks with better quality than the originals, I would suggest to use these as the labels for the test. This comes with some heavy drawbacks for your team, though:\n\n - Most part of submissions would need to be reevaluated, as their punctuation might change and place them in a relevant ranking position.\n - Every time any submission gets a punctuation close to the better one, a visual inspection of the masks should be made in order to decide whether these new masks should replace the older ones as reference.\n\nThis seems to me too difficult to overcome, but perhaps you find it feasible.",
      "votes": null
    },
    {
      "id": "213366",
      "postDate": "08/14/2017 15:56:04",
      "content": "<p>Here are my examples of worst and best predicted results (0.9972 on cross-validation)\nNotice that not all of bad result caused by mislabeling.</p>",
      "rawMarkdown": "Here are my examples of worst and best predicted results (0.9972 on cross-validation)\nNotice that not all of bad result caused by mislabeling.",
      "votes": null
    },
    {
      "id": "213398",
      "postDate": "08/14/2017 17:24:26",
      "content": "<p>i corrected some gif files. I will correct the rest when i am free. You are welcome to add more corrected files to the share google drive. </p>\n\n<p>I also provide some code for the visualization of the error.</p>\n\n<p><a href=\"https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM\">https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM</a></p>",
      "rawMarkdown": "i corrected some gif files. I will correct the rest when i am free. You are welcome to add more corrected files to the share google drive. \n\nI also provide some code for the visualization of the error.\n\nhttps://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM",
      "votes": null
    },
    {
      "id": "213484",
      "postDate": "08/14/2017 22:41:06",
      "content": "<p>Thanks for providing the correct labels. But I wonder if such small number of noisy samples would actually affect the results. Considering the number of problematic images are probably less than 50 (less than 1% of the training data) and the ratio of mislabeled pixels are somewhere around 1-5%, the total number of mislabeled pixels would be less than 1 in 10000.</p>",
      "rawMarkdown": "Thanks for providing the correct labels. But I wonder if such small number of noisy samples would actually affect the results. Considering the number of problematic images are probably less than 50 (less than 1% of the training data) and the ratio of mislabeled pixels are somewhere around 1-5%, the total number of mislabeled pixels would be less than 1 in 10000.",
      "votes": null
    },
    {
      "id": "213949",
      "postDate": "08/15/2017 15:03:46",
      "content": "<p>i don't think there will be large differences in results. But i would make my error analysis easier.</p>",
      "rawMarkdown": "i don't think there will be large differences in results. But i would make my error analysis easier.",
      "votes": null
    },
    {
      "id": "214134",
      "postDate": "08/16/2017 03:12:48",
      "content": "<p>I tried to upload some images and it says I don't have access. </p>",
      "rawMarkdown": "I tried to upload some images and it says I don't have access.",
      "votes": null
    },
    {
      "id": "214136",
      "postDate": "08/16/2017 03:14:43",
      "content": "<p>Fixed mask for 13 cases (updated 0d1a9caf4350_14_mask.gif - black pixel value wasn't (0,0,0)) :</p>\n\n<pre>0d1a9caf4350_14_mask.gif\n189a2a32a615_02_mask.gif\n23c088f6ec27_10_mask.gif\n2a4a8964ebf3_08_mask.gif\n6ba36af67cb0_07_mask.gif\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif\nb98c63cd6102_02_mask.gif\nc6f50d44f141_09_mask.gif\nd1a3af34e674_07_mask.gif\neaf9eb0b2293_01_mask.gif\neb91b1c659a0_10_mask.gif\nfa613ac8eac5_02_mask.gif\n</pre>",
      "rawMarkdown": "Fixed mask for 13 cases (updated 0d1a9caf4350_14_mask.gif - black pixel value wasn't (0,0,0)) :\n<pre>0d1a9caf4350_14_mask.gif\n189a2a32a615_02_mask.gif\n23c088f6ec27_10_mask.gif\n2a4a8964ebf3_08_mask.gif\n6ba36af67cb0_07_mask.gif\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif\nb98c63cd6102_02_mask.gif\nc6f50d44f141_09_mask.gif\nd1a3af34e674_07_mask.gif\neaf9eb0b2293_01_mask.gif\neb91b1c659a0_10_mask.gif\nfa613ac8eac5_02_mask.gif\n</pre>",
      "votes": null
    },
    {
      "id": "214248",
      "postDate": "08/16/2017 11:32:34",
      "content": "<p>thank you for the corrections. Now all 31 files are fixed by hand (but i may make some mistakes). please see:</p>\n\n<p><a href=\"https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM\">https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM</a></p>",
      "rawMarkdown": "thank you for the corrections. Now all 31 files are fixed by hand (but i may make some mistakes). please see:\n\n\nhttps://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM",
      "votes": null
    },
    {
      "id": "214512",
      "postDate": "08/17/2017 07:57:23",
      "content": "<p>you may want to confirm this, these file cannot be read by PIL, \"6ba36af67cb0_07_mask.gif\",\"189a2a32a615_02\", \"23c088f6ec27_10\". The below code gives me empty image:</p>\n\n<pre><code>    img_file = '/media/ssd/data/kaggle-carvana-cars-2017/annotations/train_gif/6ba36af67cb0_07_mask.gif'\n    img = PIL.Image.open(img_file)\n    img = np.array(img)*255\n\n    im_show('img',img, resize=0.25)\n    cv2.waitKey(0)\n</code></pre>",
      "rawMarkdown": "you may want to confirm this, these file cannot be read by PIL, \"6ba36af67cb0_07_mask.gif\",\"189a2a32a615_02\", \"23c088f6ec27_10\". The below code gives me empty image:\n\n        img_file = '/media/ssd/data/kaggle-carvana-cars-2017/annotations/train_gif/6ba36af67cb0_07_mask.gif'\n        img = PIL.Image.open(img_file)\n        img = np.array(img)*255\n\n        im_show('img',img, resize=0.25)\n        cv2.waitKey(0)",
      "votes": null
    },
    {
      "id": "214535",
      "postDate": "08/17/2017 08:48:50",
      "content": "<p>two more error image:</p>\n\n<p>3d7a1030deeb_02</p>\n\n<p>d61b6bfeabb2_02</p>",
      "rawMarkdown": "two more error image:\n\n3d7a1030deeb_02\n\nd61b6bfeabb2_02",
      "votes": null
    },
    {
      "id": "214675",
      "postDate": "08/17/2017 20:21:04",
      "content": "<p>It worked okay for me with following code (FYI, I'm using Windows):</p>\n\n<pre>from PIL import Image\nimport matplotlib.pyplot as plt\nimg_file = '6ba36af67cb0_07_mask.gif'\nimg = Image.open(img_file)\nplt.imshow(img)\nplt.show()\n</pre>",
      "rawMarkdown": "It worked okay for me with following code (FYI, I'm using Windows):\n<pre>from PIL import Image\nimport matplotlib.pyplot as plt\nimg_file = '6ba36af67cb0_07_mask.gif'\nimg = Image.open(img_file)\nplt.imshow(img)\nplt.show()\n</pre>",
      "votes": null
    },
    {
      "id": "214677",
      "postDate": "08/17/2017 20:24:56",
      "content": "<p>Here's Heng CherKeng's masks for 31 images in gif format - with one dot fix on 3cb21125f126_05_mask.</p>",
      "rawMarkdown": "Here's Heng CherKeng's masks for 31 images in gif format - with one dot fix on 3cb21125f126_05_mask.",
      "votes": null
    },
    {
      "id": "214718",
      "postDate": "08/18/2017 01:47:04",
      "content": "<blockquote>\n  <p>Also, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)</p>\n</blockquote>\n\n<p>Just wondering can we use photoshop for image processing even on test set?</p>",
      "rawMarkdown": "&gt;  Also, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)\n\nJust wondering can we use photoshop for image processing even on test set?",
      "votes": null
    },
    {
      "id": "214746",
      "postDate": "08/18/2017 05:22:09",
      "content": "<p>once you get good image pre-processing method it is not difficult to rewrite photoshop function in python (e.g. usual contrast stretch, histogram equalization, unshaped filter)</p>\n\n<p>try this for those don't have photoshop:\n<a href=\"https://www.photopea.com/\">https://www.photopea.com/</a></p>",
      "rawMarkdown": "once you get good image pre-processing method it is not difficult to rewrite photoshop function in python (e.g. usual contrast stretch, histogram equalization, unshaped filter)\n\ntry this for those don't have photoshop:\nhttps://www.photopea.com/",
      "votes": null
    },
    {
      "id": "215114",
      "postDate": "08/19/2017 20:36:38",
      "content": "<p>There is a dot on the background of 3cb21125f126_05_mask <a href=\"http://i.imgur.com/ZsZ4yUz.png\">http://i.imgur.com/ZsZ4yUz.png</a></p>",
      "rawMarkdown": "There is a dot on the background of 3cb21125f126_05_mask http://i.imgur.com/ZsZ4yUz.png",
      "votes": null
    },
    {
      "id": "215235",
      "postDate": "08/20/2017 16:42:27",
      "content": "<p>Started to get warnings (2 per epoch). Not sure if it's coming from your fixed pngs or mine.</p>\n\n<p>libpng warning: iCCP: known incorrect sRGB profile</p>",
      "rawMarkdown": "Started to get warnings (2 per epoch). Not sure if it's coming from your fixed pngs or mine.\n\nlibpng warning: iCCP: known incorrect sRGB profile",
      "votes": null
    },
    {
      "id": "215238",
      "postDate": "08/20/2017 16:45:08",
      "content": "<p>should be from my fixed png. you may want to resave them. I have two warning, but the reading seems ok. </p>",
      "rawMarkdown": "should be from my fixed png. you may want to resave them. I have two warning, but the reading seems ok.",
      "votes": null
    },
    {
      "id": "215997",
      "postDate": "08/23/2017 22:12:41",
      "content": "<p>Thanks a lot for Heng and JandJ!</p>",
      "rawMarkdown": "Thanks a lot for Heng and JandJ!",
      "votes": null
    },
    {
      "id": "221102",
      "postDate": "09/14/2017 05:32:24",
      "content": "<p>Hi JandJ, \nI tried using these masks for training from scratch. I was able to train a network that achieves 0.9939 local validation dice coefficient. However the same network seems to be predicting garbage values on the test set. I checked by visualizing some of the test masks. They are mostly all black with some random white lines. The same network (unet 1024) achieved 0.9953 local validation dice coefficient with the original data (data with no fixing of masks). Have you noticed this weird behaviour?</p>",
      "rawMarkdown": "Hi JandJ, \nI tried using these masks for training from scratch. I was able to train a network that achieves 0.9939 local validation dice coefficient. However the same network seems to be predicting garbage values on the test set. I checked by visualizing some of the test masks. They are mostly all black with some random white lines. The same network (unet 1024) achieved 0.9953 local validation dice coefficient with the original data (data with no fixing of masks). Have you noticed this weird behaviour?",
      "votes": null
    },
    {
      "id": "221108",
      "postDate": "09/14/2017 06:02:21",
      "content": "<p>No, I haven't seen that issue. I just noticed that some image's black values are 1 instead of 0 (not just edited part but all black pixels are 1 - maybe caused by image editor??) \nFor me, this doesn't matter since I'm using pixel value 127 as threshold. But your case may differ.</p>",
      "rawMarkdown": "No, I haven't seen that issue. I just noticed that some image's black values are 1 instead of 0 (not just edited part but all black pixels are 1 - maybe caused by image editor??) \nFor me, this doesn't matter since I'm using pixel value 127 as threshold. But your case may differ.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 208446,
      "author_name": "brianshaler",
      "author_url": "",
      "post_date": "07/29/2017 20:10:15",
      "content": "<p>Great attention to detail!</p>\n\n<p>I wish our manual masks were more accurate and consistent. I would caution against over-fitting to the human errors found in the training set, as the errors in the final scoring set will be different. I've seen a couple of instances of a very unexpected part of vehicles being cut out, but it's rare enough that it would be detrimental to emulate and affect non-emulators uniformly.</p>\n\n<p>It's amazing to see results exceeding 0.99 already, but we still have a ways to go before details as small as these human errors come into play.</p>\n\n<p>I'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?</p>",
      "votes": null,
      "replies": [
        {
          "id": 208461,
          "author_name": "stainsby",
          "author_url": "",
          "post_date": "07/29/2017 21:10:49",
          "content": "<blockquote>\n  <p>I'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?</p>\n</blockquote>\n\n<p>Well that's good question. I can't think of a way without using an even better segmenter. If there was a way, it's a fair bet that there'd be a GAN involved somewhere! I just hope people don't lose focus on the real goal, as I've seen in some other competitions.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 208881,
          "author_name": "ironbar",
          "author_url": "",
          "post_date": "07/31/2017 13:34:48",
          "content": "<p>Hi, <br>\nYou have say that the errors on the test set will be different, \nCould you elaborate more?</p>\n\n<p>Have the images on the test set been labelled by another person for example?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 208883,
          "author_name": "brianshaler",
          "author_url": "",
          "post_date": "07/31/2017 13:46:26",
          "content": "<p>There are no intentional errors, they are not labeled, and the data was completely randomized.</p>\n\n<p>If there are very few instances of a type of defect, they may by chance exist disproportionately in the training set, which would mean over-fitting for human error may result in worse outcomes in the scoring set.</p>\n\n<p>However, if the training masks commonly have a similar type of defect (preferring right angles, skipping gaps smaller than a specific size like the rightmost highlight above) then the highest scoring solution would need to have those traits even if an optimal algorithm doesn't.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 208938,
          "author_name": "ironbar",
          "author_url": "",
          "post_date": "07/31/2017 16:09:27",
          "content": "<p>Thanks for the info Brian</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 210050,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "08/04/2017 06:19:41",
          "content": "<p>If you have camera parameters, a 3d reconstruction algorithm like visual hull extraction should be able to catch these label error.</p>\n\n<p>Also, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 213313,
          "author_name": "dcasbol",
          "author_url": "",
          "post_date": "08/14/2017 12:51:22",
          "content": "<p>Hi Brian,</p>\n\n<p>As there are already users that have generated masks with better quality than the originals, I would suggest to use these as the labels for the test. This comes with some heavy drawbacks for your team, though:</p>\n\n<ul>\n<li>Most part of submissions would need to be reevaluated, as their punctuation might change and place them in a relevant ranking position.</li>\n<li>Every time any submission gets a punctuation close to the better one, a visual inspection of the masks should be made in order to decide whether these new masks should replace the older ones as reference.</li>\n</ul>\n\n<p>This seems to me too difficult to overcome, but perhaps you find it feasible.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 214718,
          "author_name": "outrunner",
          "author_url": "",
          "post_date": "08/18/2017 01:47:04",
          "content": "<blockquote>\n  <p>Also, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)</p>\n</blockquote>\n\n<p>Just wondering can we use photoshop for image processing even on test set?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 214746,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "08/18/2017 05:22:09",
          "content": "<p>once you get good image pre-processing method it is not difficult to rewrite photoshop function in python (e.g. usual contrast stretch, histogram equalization, unshaped filter)</p>\n\n<p>try this for those don't have photoshop:\n<a href=\"https://www.photopea.com/\">https://www.photopea.com/</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 209724,
      "author_name": "jandjenter",
      "author_url": "",
      "post_date": "08/03/2017 07:03:14",
      "content": "<p>I also found small errors in following two train masks:\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif</p>",
      "votes": null,
      "replies": [
        {
          "id": 209742,
          "author_name": "brianshaler",
          "author_url": "",
          "post_date": "08/03/2017 08:55:16",
          "content": "<p>Thanks, @JandJ! _01 is especially disappointing to see.</p>\n\n<p>I would be tempted to auto-fill fully enclosed gaps, but I'm not sure doing so wouldn't have unintended consequences (like the side runner/step on OP's truck)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 210045,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "08/04/2017 06:12:01",
          "content": "<p>@JandJ\nYou have a script to automatically detect such error? I think we need to correct such labels by hand</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 210051,
          "author_name": "jandjenter",
          "author_url": "",
          "post_date": "08/04/2017 06:20:32",
          "content": "<p>I just sorted output mask files from train set by DICE error - and it just stood out like this:\n(Yellow: false positive, Red: false negative)</p>\n\n<p><img src=\"https://image.ibb.co/cqUTCF/mask_err.png\" alt=\"enter image description here\" title=\"\"></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 211470,
      "author_name": "amitani",
      "author_url": "",
      "post_date": "08/09/2017 05:05:35",
      "content": "<p>Whether to exclude the area between wheel spokes doesn't seem to be consistent. </p>\n\n<p>Another question: when making the ground truth, was there any semi-automated tool used, or was that fully manual?  </p>",
      "votes": null,
      "replies": [
        {
          "id": 211532,
          "author_name": "brianshaler",
          "author_url": "",
          "post_date": "08/09/2017 10:13:49",
          "content": "<p>We outsourced it. As far as we know, it was manual. Some of the mask errors do make it seem like a tool may have been used to assist.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 211545,
          "author_name": "stimakov",
          "author_url": "",
          "post_date": "08/09/2017 10:34:01",
          "content": "<p>Did it take about 2-3 weeks to take the photos by any chance?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 211555,
          "author_name": "brianshaler",
          "author_url": "",
          "post_date": "08/09/2017 10:56:48",
          "content": "<p>No, the photos are from a long time period.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 211627,
          "author_name": "stimakov",
          "author_url": "",
          "post_date": "08/09/2017 15:42:49",
          "content": "<p>Interesting, most of variance of images averaged per angle is explained by under 20 principal components and after playing around a bit with those there seem to be similar amount of unique backgrounds ( different studios and camera angles ) . My estimate for number of those is 14</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 213016,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "08/13/2017 13:02:36",
      "content": "<p>some error train image masks:</p>\n\n<p>b98c63cd6102_02_mask</p>\n\n<p>d1a3af34e674_07_mask</p>\n\n<p>eaf9eb0b2293_01_mask</p>\n\n<p>eb91b1c659a0_10_mask</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 213076,
      "author_name": "jurand",
      "author_url": "",
      "post_date": "08/13/2017 16:38:51",
      "content": "<p>There are some images with large part of bottom of the  car excluded:\n0d53224da2b7_13\nc3dafdb02e7f_04</p>\n\n<p>Here is a comparison of two views one with error one without: 0d53224da2b7_13 and 0d53224da2b7_14.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 213081,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "08/13/2017 16:57:27",
      "content": "<p>in some cases the deep CNN outperform human annotations.</p>\n\n<p>here are some examples</p>\n\n<p>red outline = human annotation</p>\n\n<p>green outline = cnn prediction</p>\n\n<p>red area = miss pixels</p>\n\n<p>green area = false positive pixels</p>\n\n<p>grey area = true positive pixels</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 213088,
      "author_name": "jandjenter",
      "author_url": "",
      "post_date": "08/13/2017 17:29:34",
      "content": "<p>Updated with other posting(Total 31 cases) :</p>\n\n<pre>    \n0d1a9caf4350_02\n0d1a9caf4350_14\n0d53224da2b7_13 \n1390696b70b6_14\n189a2a32a615_02\n1ba84b81628e_06\n1e89e1af42e7_07\n23c088f6ec27_10\n2a4a8964ebf3_08\n2cb06c1f5bb1_04\n364923a5002f_03\n364fd5fd7569_06\n3cb21125f126_04\n3cb21125f126_05\n3cb21125f126_06\n3cb21125f126_12\n3cb21125f126_13\n3cb21125f126_14\n4e308ad8a254_14\n6ba36af67cb0_07\n791c1a9775be_06\n7bd1142155ae_08\n8d1a6723c458_01\n8d1a6723c458_08\nb98c63cd6102_02\nc3dafdb02e7f_04\nc6f50d44f141_09\nd1a3af34e674_07\neaf9eb0b2293_01\neb91b1c659a0_10\nfa613ac8eac5_02\n</pre>",
      "votes": null,
      "replies": [
        {
          "id": 213092,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "08/13/2017 17:36:39",
          "content": "<p>Did anyone compare results before and after correcting human error in annotations? For me it seems that there is no visible improvement using original annotations and the corrected versions.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 213098,
          "author_name": "jandjenter",
          "author_url": "",
          "post_date": "08/13/2017 18:02:19",
          "content": "<p>There are many more correct samples for similar shaped cases, so I guess this errors will not make much difference. I haven't tried to remove these cases from my train set yet.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 213103,
          "author_name": "adamhart",
          "author_url": "",
          "post_date": "08/13/2017 18:19:38",
          "content": "<p>I guess we are allowed to skip these during training, but do you know if we are allowed to fix them and then use them?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 213146,
          "author_name": "timjoseph",
          "author_url": "",
          "post_date": "08/13/2017 21:19:14",
          "content": "<p>As far as I know from other competitions you can do with the train data whatever you want...</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 213366,
      "author_name": "jetblack",
      "author_url": "",
      "post_date": "08/14/2017 15:56:04",
      "content": "<p>Here are my examples of worst and best predicted results (0.9972 on cross-validation)\nNotice that not all of bad result caused by mislabeling.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 213398,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "08/14/2017 17:24:26",
      "content": "<p>i corrected some gif files. I will correct the rest when i am free. You are welcome to add more corrected files to the share google drive. </p>\n\n<p>I also provide some code for the visualization of the error.</p>\n\n<p><a href=\"https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM\">https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 213484,
          "author_name": "harungunaydin",
          "author_url": "",
          "post_date": "08/14/2017 22:41:06",
          "content": "<p>Thanks for providing the correct labels. But I wonder if such small number of noisy samples would actually affect the results. Considering the number of problematic images are probably less than 50 (less than 1% of the training data) and the ratio of mislabeled pixels are somewhere around 1-5%, the total number of mislabeled pixels would be less than 1 in 10000.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 213949,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "08/15/2017 15:03:46",
          "content": "<p>i don't think there will be large differences in results. But i would make my error analysis easier.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 214134,
          "author_name": "jandjenter",
          "author_url": "",
          "post_date": "08/16/2017 03:12:48",
          "content": "<p>I tried to upload some images and it says I don't have access. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 214136,
      "author_name": "jandjenter",
      "author_url": "",
      "post_date": "08/16/2017 03:14:43",
      "content": "<p>Fixed mask for 13 cases (updated 0d1a9caf4350_14_mask.gif - black pixel value wasn't (0,0,0)) :</p>\n\n<pre>0d1a9caf4350_14_mask.gif\n189a2a32a615_02_mask.gif\n23c088f6ec27_10_mask.gif\n2a4a8964ebf3_08_mask.gif\n6ba36af67cb0_07_mask.gif\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif\nb98c63cd6102_02_mask.gif\nc6f50d44f141_09_mask.gif\nd1a3af34e674_07_mask.gif\neaf9eb0b2293_01_mask.gif\neb91b1c659a0_10_mask.gif\nfa613ac8eac5_02_mask.gif\n</pre>",
      "votes": null,
      "replies": [
        {
          "id": 214248,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "08/16/2017 11:32:34",
          "content": "<p>thank you for the corrections. Now all 31 files are fixed by hand (but i may make some mistakes). please see:</p>\n\n<p><a href=\"https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM\">https://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 215114,
          "author_name": "killthekitten",
          "author_url": "",
          "post_date": "08/19/2017 20:36:38",
          "content": "<p>There is a dot on the background of 3cb21125f126_05_mask <a href=\"http://i.imgur.com/ZsZ4yUz.png\">http://i.imgur.com/ZsZ4yUz.png</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 215235,
          "author_name": "killthekitten",
          "author_url": "",
          "post_date": "08/20/2017 16:42:27",
          "content": "<p>Started to get warnings (2 per epoch). Not sure if it's coming from your fixed pngs or mine.</p>\n\n<p>libpng warning: iCCP: known incorrect sRGB profile</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 215238,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "08/20/2017 16:45:08",
          "content": "<p>should be from my fixed png. you may want to resave them. I have two warning, but the reading seems ok. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 214512,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "08/17/2017 07:57:23",
      "content": "<p>you may want to confirm this, these file cannot be read by PIL, \"6ba36af67cb0_07_mask.gif\",\"189a2a32a615_02\", \"23c088f6ec27_10\". The below code gives me empty image:</p>\n\n<pre><code>    img_file = '/media/ssd/data/kaggle-carvana-cars-2017/annotations/train_gif/6ba36af67cb0_07_mask.gif'\n    img = PIL.Image.open(img_file)\n    img = np.array(img)*255\n\n    im_show('img',img, resize=0.25)\n    cv2.waitKey(0)\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 214675,
          "author_name": "jandjenter",
          "author_url": "",
          "post_date": "08/17/2017 20:21:04",
          "content": "<p>It worked okay for me with following code (FYI, I'm using Windows):</p>\n\n<pre>from PIL import Image\nimport matplotlib.pyplot as plt\nimg_file = '6ba36af67cb0_07_mask.gif'\nimg = Image.open(img_file)\nplt.imshow(img)\nplt.show()\n</pre>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 214535,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "08/17/2017 08:48:50",
      "content": "<p>two more error image:</p>\n\n<p>3d7a1030deeb_02</p>\n\n<p>d61b6bfeabb2_02</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 214677,
      "author_name": "jandjenter",
      "author_url": "",
      "post_date": "08/17/2017 20:24:56",
      "content": "<p>Here's Heng CherKeng's masks for 31 images in gif format - with one dot fix on 3cb21125f126_05_mask.</p>",
      "votes": null,
      "replies": [
        {
          "id": 221102,
          "author_name": "amalhotra",
          "author_url": "",
          "post_date": "09/14/2017 05:32:24",
          "content": "<p>Hi JandJ, \nI tried using these masks for training from scratch. I was able to train a network that achieves 0.9939 local validation dice coefficient. However the same network seems to be predicting garbage values on the test set. I checked by visualizing some of the test masks. They are mostly all black with some random white lines. The same network (unet 1024) achieved 0.9953 local validation dice coefficient with the original data (data with no fixing of masks). Have you noticed this weird behaviour?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 221108,
          "author_name": "jandjenter",
          "author_url": "",
          "post_date": "09/14/2017 06:02:21",
          "content": "<p>No, I haven't seen that issue. I just noticed that some image's black values are 1 instead of 0 (not just edited part but all black pixels are 1 - maybe caused by image editor??) \nFor me, this doesn't matter since I'm using pixel value 127 as threshold. But your case may differ.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 215997,
      "author_name": "noruen",
      "author_url": "",
      "post_date": "08/23/2017 22:12:41",
      "content": "<p>Thanks a lot for Heng and JandJ!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "208304": "There are, understandably, small errors in the supplied training masks - see 2cb06c1f5bb1_05 for example:\n\n![segmentation differences for 2cb06c1f5bb1_05][1]\n\nGiven the high quality of the segmentation algorithms that we are likely to see, the top leaderboard margins could be very narrow. This has already been pointed out in topics elsewhere.  I wonder if winning the competition might come down to the algorithms that best emulate these human errors, rather than coming up with the optimal algorithm for the original purpose of the project.\n\nPerhaps I'm worrying too much, but it might be nice if competitors consider publishing, if possible, an alternative algorithm that is optimal as well as the one intended to win.\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/208304/6919/2cb06c1f5bb1_05_diffs_sm.jpg",
    "208446": "Great attention to detail!\n\nI wish our manual masks were more accurate and consistent. I would caution against over-fitting to the human errors found in the training set, as the errors in the final scoring set will be different. I've seen a couple of instances of a very unexpected part of vehicles being cut out, but it's rare enough that it would be detrimental to emulate and affect non-emulators uniformly.\n\nIt's amazing to see results exceeding 0.99 already, but we still have a ways to go before details as small as these human errors come into play.\n\nI'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?",
    "208461": "&gt; I'm curious: How would we score, at scale, a more optimal algorithm without more accurate masks to compare against?\n\nWell that's good question. I can't think of a way without using an even better segmenter. If there was a way, it's a fair bet that there'd be a GAN involved somewhere! I just hope people don't lose focus on the real goal, as I've seen in some other competitions.",
    "208881": "Hi,  \nYou have say that the errors on the test set will be different, \nCould you elaborate more?\n\nHave the images on the test set been labelled by another person for example?",
    "208883": "There are no intentional errors, they are not labeled, and the data was completely randomized.\n\nIf there are very few instances of a type of defect, they may by chance exist disproportionately in the training set, which would mean over-fitting for human error may result in worse outcomes in the scoring set.\n\nHowever, if the training masks commonly have a similar type of defect (preferring right angles, skipping gaps smaller than a specific size like the rightmost highlight above) then the highest scoring solution would need to have those traits even if an optimal algorithm doesn't.",
    "208938": "Thanks for the info Brian",
    "209724": "I also found small errors in following two train masks:\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif",
    "209742": "Thanks, @JandJ! _01 is especially disappointing to see.\n\nI would be tempted to auto-fill fully enclosed gaps, but I'm not sure doing so wouldn't have unintended consequences (like the side runner/step on OP's truck)",
    "210045": "JandJ\nYou have a script to automatically detect such error? I think we need to correct such labels by hand",
    "210050": "If you have camera parameters, a 3d reconstruction algorithm like visual hull extraction should be able to catch these label error.\n\nAlso, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)",
    "210051": "I just sorted output mask files from train set by DICE error - and it just stood out like this:\n(Yellow: false positive, Red: false negative)\n\n![enter image description here][1]\n\n  [1]: https://image.ibb.co/cqUTCF/mask_err.png",
    "211470": "Whether to exclude the area between wheel spokes doesn't seem to be consistent. \n\nAnother question: when making the ground truth, was there any semi-automated tool used, or was that fully manual?",
    "211532": "We outsourced it. As far as we know, it was manual. Some of the mask errors do make it seem like a tool may have been used to assist.",
    "211545": "Did it take about 2-3 weeks to take the photos by any chance?",
    "211555": "No, the photos are from a long time period.",
    "211627": "Interesting, most of variance of images averaged per angle is explained by under 20 principal components and after playing around a bit with those there seem to be similar amount of unique backgrounds ( different studios and camera angles ) . My estimate for number of those is 14",
    "213016": "some error train image masks:\n\nb98c63cd6102_02_mask\n\nd1a3af34e674_07_mask\n\neaf9eb0b2293_01_mask\n\neb91b1c659a0_10_mask",
    "213076": "There are some images with large part of bottom of the  car excluded:\n0d53224da2b7_13\nc3dafdb02e7f_04\n\nHere is a comparison of two views one with error one without: 0d53224da2b7_13 and 0d53224da2b7_14.",
    "213081": "in some cases the deep CNN outperform human annotations.\n\nhere are some examples\n\nred outline = human annotation\n\ngreen outline = cnn prediction\n\nred area = miss pixels\n\ngreen area = false positive pixels\n\ngrey area = true positive pixels",
    "213088": "Updated with other posting(Total 31 cases) :\n\n<pre>    \n0d1a9caf4350_02\n0d1a9caf4350_14\n0d53224da2b7_13 \n1390696b70b6_14\n189a2a32a615_02\n1ba84b81628e_06\n1e89e1af42e7_07\n23c088f6ec27_10\n2a4a8964ebf3_08\n2cb06c1f5bb1_04\n364923a5002f_03\n364fd5fd7569_06\n3cb21125f126_04\n3cb21125f126_05\n3cb21125f126_06\n3cb21125f126_12\n3cb21125f126_13\n3cb21125f126_14\n4e308ad8a254_14\n6ba36af67cb0_07\n791c1a9775be_06\n7bd1142155ae_08\n8d1a6723c458_01\n8d1a6723c458_08\nb98c63cd6102_02\nc3dafdb02e7f_04\nc6f50d44f141_09\nd1a3af34e674_07\neaf9eb0b2293_01\neb91b1c659a0_10\nfa613ac8eac5_02\n</pre>",
    "213092": "Did anyone compare results before and after correcting human error in annotations? For me it seems that there is no visible improvement using original annotations and the corrected versions.",
    "213098": "There are many more correct samples for similar shaped cases, so I guess this errors will not make much difference. I haven't tried to remove these cases from my train set yet.",
    "213103": "I guess we are allowed to skip these during training, but do you know if we are allowed to fix them and then use them?",
    "213146": "As far as I know from other competitions you can do with the train data whatever you want...",
    "213313": "Hi Brian,\n\nAs there are already users that have generated masks with better quality than the originals, I would suggest to use these as the labels for the test. This comes with some heavy drawbacks for your team, though:\n\n - Most part of submissions would need to be reevaluated, as their punctuation might change and place them in a relevant ranking position.\n - Every time any submission gets a punctuation close to the better one, a visual inspection of the masks should be made in order to decide whether these new masks should replace the older ones as reference.\n\nThis seems to me too difficult to overcome, but perhaps you find it feasible.",
    "213366": "Here are my examples of worst and best predicted results (0.9972 on cross-validation)\nNotice that not all of bad result caused by mislabeling.",
    "213398": "i corrected some gif files. I will correct the rest when i am free. You are welcome to add more corrected files to the share google drive. \n\nI also provide some code for the visualization of the error.\n\nhttps://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM",
    "213484": "Thanks for providing the correct labels. But I wonder if such small number of noisy samples would actually affect the results. Considering the number of problematic images are probably less than 50 (less than 1% of the training data) and the ratio of mislabeled pixels are somewhere around 1-5%, the total number of mislabeled pixels would be less than 1 in 10000.",
    "213949": "i don't think there will be large differences in results. But i would make my error analysis easier.",
    "214134": "I tried to upload some images and it says I don't have access.",
    "214136": "Fixed mask for 13 cases (updated 0d1a9caf4350_14_mask.gif - black pixel value wasn't (0,0,0)) :\n<pre>0d1a9caf4350_14_mask.gif\n189a2a32a615_02_mask.gif\n23c088f6ec27_10_mask.gif\n2a4a8964ebf3_08_mask.gif\n6ba36af67cb0_07_mask.gif\n8d1a6723c458_01_mask.gif\n8d1a6723c458_08_mask.gif\nb98c63cd6102_02_mask.gif\nc6f50d44f141_09_mask.gif\nd1a3af34e674_07_mask.gif\neaf9eb0b2293_01_mask.gif\neb91b1c659a0_10_mask.gif\nfa613ac8eac5_02_mask.gif\n</pre>",
    "214248": "thank you for the corrections. Now all 31 files are fixed by hand (but i may make some mistakes). please see:\n\n\nhttps://drive.google.com/drive/folders/0B_DICebvRE-kN21fYVNxc1MyVXM",
    "214512": "you may want to confirm this, these file cannot be read by PIL, \"6ba36af67cb0_07_mask.gif\",\"189a2a32a615_02\", \"23c088f6ec27_10\". The below code gives me empty image:\n\n        img_file = '/media/ssd/data/kaggle-carvana-cars-2017/annotations/train_gif/6ba36af67cb0_07_mask.gif'\n        img = PIL.Image.open(img_file)\n        img = np.array(img)*255\n\n        im_show('img',img, resize=0.25)\n        cv2.waitKey(0)",
    "214535": "two more error image:\n\n3d7a1030deeb_02\n\nd61b6bfeabb2_02",
    "214675": "It worked okay for me with following code (FYI, I'm using Windows):\n<pre>from PIL import Image\nimport matplotlib.pyplot as plt\nimg_file = '6ba36af67cb0_07_mask.gif'\nimg = Image.open(img_file)\nplt.imshow(img)\nplt.show()\n</pre>",
    "214677": "Here's Heng CherKeng's masks for 31 images in gif format - with one dot fix on 3cb21125f126_05_mask.",
    "214718": "&gt;  Also, through this manual process, i hope to gain some domain knowledge about the data and maybe from that i can design a better algorithm. e.g., i need to enhance the contrast of some images to mark the boundary correctly in photoshop. This may give me a hint on how to preprocess the image :)\n\nJust wondering can we use photoshop for image processing even on test set?",
    "214746": "once you get good image pre-processing method it is not difficult to rewrite photoshop function in python (e.g. usual contrast stretch, histogram equalization, unshaped filter)\n\ntry this for those don't have photoshop:\nhttps://www.photopea.com/",
    "215114": "There is a dot on the background of 3cb21125f126_05_mask http://i.imgur.com/ZsZ4yUz.png",
    "215235": "Started to get warnings (2 per epoch). Not sure if it's coming from your fixed pngs or mine.\n\nlibpng warning: iCCP: known incorrect sRGB profile",
    "215238": "should be from my fixed png. you may want to resave them. I have two warning, but the reading seems ok.",
    "215997": "Thanks a lot for Heng and JandJ!",
    "221102": "Hi JandJ, \nI tried using these masks for training from scratch. I was able to train a network that achieves 0.9939 local validation dice coefficient. However the same network seems to be predicting garbage values on the test set. I checked by visualizing some of the test masks. They are mostly all black with some random white lines. The same network (unet 1024) achieved 0.9953 local validation dice coefficient with the original data (data with no fixing of masks). Have you noticed this weird behaviour?",
    "221108": "No, I haven't seen that issue. I just noticed that some image's black values are 1 instead of 0 (not just edited part but all black pixels are 1 - maybe caused by image editor??) \nFor me, this doesn't matter since I'm using pixel value 127 as threshold. But your case may differ."
  },
  "source": "meta"
}