{
  "id": 54804,
  "title": "Some analysis of current submissions, and test labels fixed",
  "url": "/competitions/cvpr-2018-autonomous-driving/discussion/54804",
  "author_name": "Wendy Kan",
  "post_date": "2018-04-18T00:10:29.237000",
  "votes": 2,
  "comment_count": 17,
  "views": 0,
  "content": "<p>Hi all, </p>\n\n<p>We have found a labeling error in our public leaderboard test data, to fix it, the following has been done:</p>\n\n<ul>\n<li>the 10 public leaderboard images have been marked as \"ignored\" so they don't count towards anything</li>\n<li>we chose another 10 images in the test set that have correct labels, moved them to be used as public leaderboard</li>\n<li>we rescored your submissions such that they reflect the correct scores</li>\n</ul>\n\n<p><strong>There is nothing to be done on your side</strong>, no additional data downloads, and all your past submissions are still valid. </p>\n\n<p>We also want to explain a little on the 0.0 public leaderboard scores. We looked at some submissions in order to make sure our evaluation metric code doesn't have any bugs, and the accuracy are indeed low, mostly because the starting threshold of object IoU is 0.5, so all the objects need to have at least 50% overlap with the ground truth for it to be correct, and that is perhaps difficult to achieve, hence the low scores. </p>\n\n<p>The other concern that the community shared was that 10 images was too few to have signal in the public leaderboard. While we agree with that statement, please understand that this is necessary since we are under quite a bit of compute constraint with this evaluation metric as it could take a long time to run, and as a result we had to cut down the size of the test set. Because of that, we needed to reserve almost all of the test set to private leaderboard, in order to have meaningful scores for everyone for the final ranking.</p>\n\n<p>The organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores. They are also working on providing tools to help converting masks to RLE format. We will keep you posted when the tools are ready. </p>\n\n<p><strong>EDIT</strong>: We have also increased daily submission limit to 3 submissions a day, in case any of you need extra quota to test out your past submissions. </p>\n\n<p>Kaggle admins+WAD/Baidu organizers</p>",
  "messages": [
    {
      "id": 320282,
      "postDate": "2018-04-28T06:17:05.057Z",
      "content": "<p>Hi,</p>\n\n<p>The filenames for 10 test images are missing from the md5-&gt;filename mapping (listed below). Is this deliberate, or was that information posted somewhere? Or are these the test images that were affected?</p>\n\n<ul>\n<li>b0a81467ffa2b260fa84fdda78d245a9.jpg</li>\n<li>b0a539102aba1dee708d14edaf499bc0.jpg </li>\n<li>e714994348b85004d09a447b2e0d5fc0.jpg</li>\n<li>9df73c4dd7d6efab6e39c9e91f4fe3ae.jpg</li>\n<li>12eeb997abcf94235b65f663d2cff198.jpg</li>\n<li>fa23f779c6c842dc2ad6445b41650707.jpg</li>\n<li>30226074f61ab422d3f0ddbb284fa852.jpg</li>\n<li>0d5920509771a8d99dcfc265ce046df6.jpg</li>\n<li>abffd3db151bca65c1a72281c98592f2.jpg</li>\n<li>c745d4d27662374bebc0ab0d1cd31dd6.jpg</li>\n</ul>",
      "rawMarkdown": "Hi,\n\nThe filenames for 10 test images are missing from the md5-&gt;filename mapping (listed below). Is this deliberate, or was that information posted somewhere? Or are these the test images that were affected?\n\n* b0a81467ffa2b260fa84fdda78d245a9.jpg\n* b0a539102aba1dee708d14edaf499bc0.jpg \n* e714994348b85004d09a447b2e0d5fc0.jpg\n* 9df73c4dd7d6efab6e39c9e91f4fe3ae.jpg\n* 12eeb997abcf94235b65f663d2cff198.jpg\n* fa23f779c6c842dc2ad6445b41650707.jpg\n* 30226074f61ab422d3f0ddbb284fa852.jpg\n* 0d5920509771a8d99dcfc265ce046df6.jpg\n* abffd3db151bca65c1a72281c98592f2.jpg\n* c745d4d27662374bebc0ab0d1cd31dd6.jpg",
      "votes": 1,
      "replies": [
        {
          "id": 321251,
          "postDate": "2018-04-30T22:04:00.913Z",
          "content": "<p>In my download this one is missing:  1e2b27e2c4b653f096e58b4c8f619abd.jpg</p>",
          "rawMarkdown": "In my download this one is missing:  1e2b27e2c4b653f096e58b4c8f619abd.jpg"
        },
        {
          "id": 321337,
          "postDate": "2018-05-01T04:17:16.043Z",
          "content": "<p>I misunderstook the post.  Deleting comment.</p>",
          "rawMarkdown": "I misunderstook the post.  Deleting comment.\n"
        },
        {
          "id": 321891,
          "postDate": "2018-05-02T05:17:49.690Z",
          "content": "<p>This problem should have been resolved weeks ago. Please download the testing images and lists again.</p>",
          "rawMarkdown": "This problem should have been resolved weeks ago. Please download the testing images and lists again."
        },
        {
          "id": 321905,
          "postDate": "2018-05-02T06:03:13.083Z",
          "content": "<p>md5 mappings were downloaded 20 hours ago, and test images re-downloaded just now.</p>",
          "rawMarkdown": "md5 mappings were downloaded 20 hours ago, and test images re-downloaded just now.",
          "votes": 1
        },
        {
          "id": 326360,
          "postDate": "2018-05-09T15:47:34.533Z",
          "content": "<p>Confirm, this IDs do not match:\n ['0d5920509771a8d99dcfc265ce046df6',\n '9df73c4dd7d6efab6e39c9e91f4fe3ae',\n 'c745d4d27662374bebc0ab0d1cd31dd6',\n '12eeb997abcf94235b65f663d2cff198',\n 'b0a81467ffa2b260fa84fdda78d245a9',\n 'e714994348b85004d09a447b2e0d5fc0',\n 'abffd3db151bca65c1a72281c98592f2',\n 'b0a539102aba1dee708d14edaf499bc0',\n '30226074f61ab422d3f0ddbb284fa852',\n 'fa23f779c6c842dc2ad6445b41650707']</p>\n\n<p>Downloaded test set and mappings today. </p>",
          "rawMarkdown": "Confirm, this IDs do not match:\n ['0d5920509771a8d99dcfc265ce046df6',\n '9df73c4dd7d6efab6e39c9e91f4fe3ae',\n 'c745d4d27662374bebc0ab0d1cd31dd6',\n '12eeb997abcf94235b65f663d2cff198',\n 'b0a81467ffa2b260fa84fdda78d245a9',\n 'e714994348b85004d09a447b2e0d5fc0',\n 'abffd3db151bca65c1a72281c98592f2',\n 'b0a539102aba1dee708d14edaf499bc0',\n '30226074f61ab422d3f0ddbb284fa852',\n 'fa23f779c6c842dc2ad6445b41650707']\n\nDownloaded test set and mappings today. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 316422,
      "postDate": "2018-04-19T02:08:03.467Z",
      "content": "<p>how to  simplify the shapes ?  Excuse me , the submission format is limited by csv, I don't know how to do this. Would mind give some details .</p>",
      "rawMarkdown": "how to  simplify the shapes ?  Excuse me , the submission format is limited by csv, I don't know how to do this. Would mind give some details .",
      "votes": 1,
      "replies": [
        {
          "id": 316716,
          "postDate": "2018-04-19T18:29:03.547Z",
          "content": "<p>Just my speculation: Baidu's metric code converts your RLE into masks first before performing semi-exhaustive object to object IoU, so if your RLE has many broken objects, it increases the complexity of the evaluation. Imagine you have N objects in ground truth and you predicted M objects, then it's O(NM), so the larger your M is, the more time it takes to compute, therefore it's easy to time out. </p>",
          "rawMarkdown": "Just my speculation: Baidu's metric code converts your RLE into masks first before performing semi-exhaustive object to object IoU, so if your RLE has many broken objects, it increases the complexity of the evaluation. Imagine you have N objects in ground truth and you predicted M objects, then it's O(NM), so the larger your M is, the more time it takes to compute, therefore it's easy to time out. ",
          "votes": 1
        },
        {
          "id": 316812,
          "postDate": "2018-04-20T01:02:20.417Z",
          "content": "<p>Great thinking. I appreciated.</p>",
          "rawMarkdown": "Great thinking. I appreciated."
        }
      ]
    },
    {
      "id": 316095,
      "postDate": "2018-04-18T07:30:44.100Z",
      "content": "<blockquote>\n  <p>The organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores.</p>\n</blockquote>\n\n<p>That would be awesome, thanks!</p>",
      "rawMarkdown": "&gt; The organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores.\n\nThat would be awesome, thanks!",
      "votes": 1
    },
    {
      "id": 316090,
      "postDate": "2018-04-18T07:21:26.480Z",
      "content": "<p>Hi, would you mind check the max-size for submission ? As proposals in one image may be larger, so that needing submit cvs whose size may be more than 100M. </p>",
      "rawMarkdown": "Hi, would you mind check the max-size for submission ? As proposals in one image may be larger, so that needing submit cvs whose size may be more than 100M. \n",
      "votes": 1,
      "replies": [
        {
          "id": 316339,
          "postDate": "2018-04-18T19:15:53.580Z",
          "content": "<p><a href=\"/zhuangyq\">@zhuangyq</a>, we don't have a size limit on submission file sizes (we have had submission files &gt;2GB). Your submissions timed out (&gt;15 mins) and you might need to simplify the shapes to make the computation faster. Please let me know if you're still having troubles with the compute time. </p>\n\n<p>I think our error message is misleading, so we will get those time out error messages fixed. </p>",
          "rawMarkdown": "@zhuangyq, we don't have a size limit on submission file sizes (we have had submission files &gt;2GB). Your submissions timed out (&gt;15 mins) and you might need to simplify the shapes to make the computation faster. Please let me know if you're still having troubles with the compute time. \n\nI think our error message is misleading, so we will get those time out error messages fixed. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 315989,
      "postDate": "2018-04-18T00:10:29.237Z",
      "content": "<p>Hi all, </p>\n\n<p>We have found a labeling error in our public leaderboard test data, to fix it, the following has been done:</p>\n\n<ul>\n<li>the 10 public leaderboard images have been marked as \"ignored\" so they don't count towards anything</li>\n<li>we chose another 10 images in the test set that have correct labels, moved them to be used as public leaderboard</li>\n<li>we rescored your submissions such that they reflect the correct scores</li>\n</ul>\n\n<p><strong>There is nothing to be done on your side</strong>, no additional data downloads, and all your past submissions are still valid. </p>\n\n<p>We also want to explain a little on the 0.0 public leaderboard scores. We looked at some submissions in order to make sure our evaluation metric code doesn't have any bugs, and the accuracy are indeed low, mostly because the starting threshold of object IoU is 0.5, so all the objects need to have at least 50% overlap with the ground truth for it to be correct, and that is perhaps difficult to achieve, hence the low scores. </p>\n\n<p>The other concern that the community shared was that 10 images was too few to have signal in the public leaderboard. While we agree with that statement, please understand that this is necessary since we are under quite a bit of compute constraint with this evaluation metric as it could take a long time to run, and as a result we had to cut down the size of the test set. Because of that, we needed to reserve almost all of the test set to private leaderboard, in order to have meaningful scores for everyone for the final ranking.</p>\n\n<p>The organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores. They are also working on providing tools to help converting masks to RLE format. We will keep you posted when the tools are ready. </p>\n\n<p><strong>EDIT</strong>: We have also increased daily submission limit to 3 submissions a day, in case any of you need extra quota to test out your past submissions. </p>\n\n<p>Kaggle admins+WAD/Baidu organizers</p>",
      "rawMarkdown": "Hi all, \n\nWe have found a labeling error in our public leaderboard test data, to fix it, the following has been done:\n\n- the 10 public leaderboard images have been marked as \"ignored\" so they don't count towards anything\n- we chose another 10 images in the test set that have correct labels, moved them to be used as public leaderboard\n- we rescored your submissions such that they reflect the correct scores\n\n**There is nothing to be done on your side**, no additional data downloads, and all your past submissions are still valid. \n\nWe also want to explain a little on the 0.0 public leaderboard scores. We looked at some submissions in order to make sure our evaluation metric code doesn't have any bugs, and the accuracy are indeed low, mostly because the starting threshold of object IoU is 0.5, so all the objects need to have at least 50% overlap with the ground truth for it to be correct, and that is perhaps difficult to achieve, hence the low scores. \n\nThe other concern that the community shared was that 10 images was too few to have signal in the public leaderboard. While we agree with that statement, please understand that this is necessary since we are under quite a bit of compute constraint with this evaluation metric as it could take a long time to run, and as a result we had to cut down the size of the test set. Because of that, we needed to reserve almost all of the test set to private leaderboard, in order to have meaningful scores for everyone for the final ranking.\n\nThe organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores. They are also working on providing tools to help converting masks to RLE format. We will keep you posted when the tools are ready. \n\n**EDIT**: We have also increased daily submission limit to 3 submissions a day, in case any of you need extra quota to test out your past submissions. \n\nKaggle admins+WAD/Baidu organizers",
      "votes": 2
    },
    {
      "id": 319425,
      "postDate": "2018-04-26T02:55:13.870Z",
      "rawMarkdown": ""
    },
    {
      "id": 318090,
      "postDate": "2018-04-23T07:13:00.010Z",
      "content": "<p>Maybe it is tricky to do, but it would be great to have unlimited 0.0000 submissions per day since quite a bit of the work is figuring out exactly how the format works. Once the LB score is above 0.0000 the submissions would start counting since something can be learned from them.</p>",
      "rawMarkdown": "Maybe it is tricky to do, but it would be great to have unlimited 0.0000 submissions per day since quite a bit of the work is figuring out exactly how the format works. Once the LB score is above 0.0000 the submissions would start counting since something can be learned from them.",
      "replies": [
        {
          "id": 319294,
          "postDate": "2018-04-25T17:56:44.480Z",
          "content": "<p>Sorry, it's not possible in our current set up to have a rule dependency on the score. </p>",
          "rawMarkdown": "Sorry, it's not possible in our current set up to have a rule dependency on the score. ",
          "votes": 1
        },
        {
          "id": 321261,
          "postDate": "2018-04-30T22:19:21.833Z",
          "content": "<p>Maybe you can set up a small test submission for one or few images. Just to check the format and size limits. I get a lot of errors and a lot of timeouts. \nI even tried to predict whole images as 33 label. The format is suppose to be simple,\nPixelCount is: with*height, and PixelEncoding is: \"0,width*height|\". I got an error.</p>",
          "rawMarkdown": "Maybe you can set up a small test submission for one or few images. Just to check the format and size limits. I get a lot of errors and a lot of timeouts. \nI even tried to predict whole images as 33 label. The format is suppose to be simple,\nPixelCount is: with*height, and PixelEncoding is: \"0,width*height|\". I got an error."
        }
      ]
    },
    {
      "id": 320280,
      "postDate": "2018-04-28T06:11:41.027Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 320282,
      "author_name": "Snorkle",
      "author_url": "",
      "post_date": "2018-04-28T06:17:05.057000",
      "content": "<p>Hi,</p>\n\n<p>The filenames for 10 test images are missing from the md5-&gt;filename mapping (listed below). Is this deliberate, or was that information posted somewhere? Or are these the test images that were affected?</p>\n\n<ul>\n<li>b0a81467ffa2b260fa84fdda78d245a9.jpg</li>\n<li>b0a539102aba1dee708d14edaf499bc0.jpg </li>\n<li>e714994348b85004d09a447b2e0d5fc0.jpg</li>\n<li>9df73c4dd7d6efab6e39c9e91f4fe3ae.jpg</li>\n<li>12eeb997abcf94235b65f663d2cff198.jpg</li>\n<li>fa23f779c6c842dc2ad6445b41650707.jpg</li>\n<li>30226074f61ab422d3f0ddbb284fa852.jpg</li>\n<li>0d5920509771a8d99dcfc265ce046df6.jpg</li>\n<li>abffd3db151bca65c1a72281c98592f2.jpg</li>\n<li>c745d4d27662374bebc0ab0d1cd31dd6.jpg</li>\n</ul>",
      "votes": 1,
      "replies": [
        {
          "id": 321251,
          "author_name": "NaomiFridman",
          "author_url": "",
          "post_date": "2018-04-30T22:04:00.913000",
          "content": "<p>In my download this one is missing:  1e2b27e2c4b653f096e58b4c8f619abd.jpg</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 321337,
          "author_name": "Wei Dong",
          "author_url": "",
          "post_date": "2018-05-01T04:17:16.043000",
          "content": "<p>I misunderstook the post.  Deleting comment.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 321891,
          "author_name": "Xinyu Huang",
          "author_url": "",
          "post_date": "2018-05-02T05:17:49.690000",
          "content": "<p>This problem should have been resolved weeks ago. Please download the testing images and lists again.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 321905,
          "author_name": "Snorkle",
          "author_url": "",
          "post_date": "2018-05-02T06:03:13.083000",
          "content": "<p>md5 mappings were downloaded 20 hours ago, and test images re-downloaded just now.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 326360,
          "author_name": "Azat Akhtyamov",
          "author_url": "",
          "post_date": "2018-05-09T15:47:34.533000",
          "content": "<p>Confirm, this IDs do not match:\n ['0d5920509771a8d99dcfc265ce046df6',\n '9df73c4dd7d6efab6e39c9e91f4fe3ae',\n 'c745d4d27662374bebc0ab0d1cd31dd6',\n '12eeb997abcf94235b65f663d2cff198',\n 'b0a81467ffa2b260fa84fdda78d245a9',\n 'e714994348b85004d09a447b2e0d5fc0',\n 'abffd3db151bca65c1a72281c98592f2',\n 'b0a539102aba1dee708d14edaf499bc0',\n '30226074f61ab422d3f0ddbb284fa852',\n 'fa23f779c6c842dc2ad6445b41650707']</p>\n\n<p>Downloaded test set and mappings today. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 316422,
      "author_name": "Tommy.Zhuang",
      "author_url": "",
      "post_date": "2018-04-19T02:08:03.467000",
      "content": "<p>how to  simplify the shapes ?  Excuse me , the submission format is limited by csv, I don't know how to do this. Would mind give some details .</p>",
      "votes": 1,
      "replies": [
        {
          "id": 316716,
          "author_name": "Wendy Kan",
          "author_url": "",
          "post_date": "2018-04-19T18:29:03.547000",
          "content": "<p>Just my speculation: Baidu's metric code converts your RLE into masks first before performing semi-exhaustive object to object IoU, so if your RLE has many broken objects, it increases the complexity of the evaluation. Imagine you have N objects in ground truth and you predicted M objects, then it's O(NM), so the larger your M is, the more time it takes to compute, therefore it's easy to time out. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 316812,
          "author_name": " Hermina Lannon",
          "author_url": "",
          "post_date": "2018-04-20T01:02:20.417000",
          "content": "<p>Great thinking. I appreciated.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 316095,
      "author_name": "Konstantin Lopukhin",
      "author_url": "",
      "post_date": "2018-04-18T07:30:44.100000",
      "content": "<blockquote>\n  <p>The organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores.</p>\n</blockquote>\n\n<p>That would be awesome, thanks!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 316090,
      "author_name": "Tommy.Zhuang",
      "author_url": "",
      "post_date": "2018-04-18T07:21:26.480000",
      "content": "<p>Hi, would you mind check the max-size for submission ? As proposals in one image may be larger, so that needing submit cvs whose size may be more than 100M. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 316339,
          "author_name": "Wendy Kan",
          "author_url": "",
          "post_date": "2018-04-18T19:15:53.580000",
          "content": "<p><a href=\"/zhuangyq\">@zhuangyq</a>, we don't have a size limit on submission file sizes (we have had submission files &gt;2GB). Your submissions timed out (&gt;15 mins) and you might need to simplify the shapes to make the computation faster. Please let me know if you're still having troubles with the compute time. </p>\n\n<p>I think our error message is misleading, so we will get those time out error messages fixed. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 319425,
      "author_name": "kekedan",
      "author_url": "",
      "post_date": "2018-04-26T02:55:13.870000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 318090,
      "author_name": "K Scott Mader",
      "author_url": "",
      "post_date": "2018-04-23T07:13:00.010000",
      "content": "<p>Maybe it is tricky to do, but it would be great to have unlimited 0.0000 submissions per day since quite a bit of the work is figuring out exactly how the format works. Once the LB score is above 0.0000 the submissions would start counting since something can be learned from them.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 319294,
          "author_name": "Wendy Kan",
          "author_url": "",
          "post_date": "2018-04-25T17:56:44.480000",
          "content": "<p>Sorry, it's not possible in our current set up to have a rule dependency on the score. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 321261,
          "author_name": "NaomiFridman",
          "author_url": "",
          "post_date": "2018-04-30T22:19:21.833000",
          "content": "<p>Maybe you can set up a small test submission for one or few images. Just to check the format and size limits. I get a lot of errors and a lot of timeouts. \nI even tried to predict whole images as 33 label. The format is suppose to be simple,\nPixelCount is: with*height, and PixelEncoding is: \"0,width*height|\". I got an error.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 320280,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-04-28T06:11:41.027000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "320282": "Hi,\n\nThe filenames for 10 test images are missing from the md5-&gt;filename mapping (listed below). Is this deliberate, or was that information posted somewhere? Or are these the test images that were affected?\n\n* b0a81467ffa2b260fa84fdda78d245a9.jpg\n* b0a539102aba1dee708d14edaf499bc0.jpg \n* e714994348b85004d09a447b2e0d5fc0.jpg\n* 9df73c4dd7d6efab6e39c9e91f4fe3ae.jpg\n* 12eeb997abcf94235b65f663d2cff198.jpg\n* fa23f779c6c842dc2ad6445b41650707.jpg\n* 30226074f61ab422d3f0ddbb284fa852.jpg\n* 0d5920509771a8d99dcfc265ce046df6.jpg\n* abffd3db151bca65c1a72281c98592f2.jpg\n* c745d4d27662374bebc0ab0d1cd31dd6.jpg",
    "316422": "how to  simplify the shapes ?  Excuse me , the submission format is limited by csv, I don't know how to do this. Would mind give some details .",
    "316095": "&gt; The organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores.\n\nThat would be awesome, thanks!",
    "316090": "Hi, would you mind check the max-size for submission ? As proposals in one image may be larger, so that needing submit cvs whose size may be more than 100M. \n",
    "315989": "Hi all, \n\nWe have found a labeling error in our public leaderboard test data, to fix it, the following has been done:\n\n- the 10 public leaderboard images have been marked as \"ignored\" so they don't count towards anything\n- we chose another 10 images in the test set that have correct labels, moved them to be used as public leaderboard\n- we rescored your submissions such that they reflect the correct scores\n\n**There is nothing to be done on your side**, no additional data downloads, and all your past submissions are still valid. \n\nWe also want to explain a little on the 0.0 public leaderboard scores. We looked at some submissions in order to make sure our evaluation metric code doesn't have any bugs, and the accuracy are indeed low, mostly because the starting threshold of object IoU is 0.5, so all the objects need to have at least 50% overlap with the ground truth for it to be correct, and that is perhaps difficult to achieve, hence the low scores. \n\nThe other concern that the community shared was that 10 images was too few to have signal in the public leaderboard. While we agree with that statement, please understand that this is necessary since we are under quite a bit of compute constraint with this evaluation metric as it could take a long time to run, and as a result we had to cut down the size of the test set. Because of that, we needed to reserve almost all of the test set to private leaderboard, in order to have meaningful scores for everyone for the final ranking.\n\nThe organizers at WAD/Baidu understand the sentiment from the community, and are working hard on open-sourcing some of their tools to help the community get cross-validation scores locally so you can get more feedback on the mAP scores. They are also working on providing tools to help converting masks to RLE format. We will keep you posted when the tools are ready. \n\n**EDIT**: We have also increased daily submission limit to 3 submissions a day, in case any of you need extra quota to test out your past submissions. \n\nKaggle admins+WAD/Baidu organizers",
    "319425": "",
    "318090": "Maybe it is tricky to do, but it would be great to have unlimited 0.0000 submissions per day since quite a bit of the work is figuring out exactly how the format works. Once the LB score is above 0.0000 the submissions would start counting since something can be learned from them.",
    "320280": ""
  }
}