![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16463%2F12e88f8210f6a324a8fec7ec104fe091%2FID_521f0b21f.jpg?generation=1579661774036365&alt=media\")
The private leaderboard has been released and congrats to the top winners.
\n\nAfter examine the private/public leaderboard, we have one last issue want to ask the organisor or the competition participants so that we can R.I.P....\n(A similar topic has been raised at: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653 )
\n\nWe have detected the cropped image from the test set (around 400 images) and set them as empty in the prediction. The surprising result is that comparing with non-zeros predictions, they generate the same private LB score. Those 400 ish images constitute around 20% of the total images which is a decent amount of test images.\nThe following is one example of the cropped image with our prediction:
\n\n
Hence, we would really appreciate if the organisor could clarify whether those cropped images are not considered into the evaluation (or they were annotated first and than cropped?)
\n\nP.S.: we will also release our solution soon.
", "messages": [ { "id": "725370", "postDate": "01/22/2020 03:10:19", "content": "The private leaderboard has been released and congrats to the top winners.
\n\nAfter examine the private/public leaderboard, we have one last issue want to ask the organisor or the competition participants so that we can R.I.P....\n(A similar topic has been raised at: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653 )
\n\nWe have detected the cropped image from the test set (around 400 images) and set them as empty in the prediction. The surprising result is that comparing with non-zeros predictions, they generate the same private LB score. Those 400 ish images constitute around 20% of the total images which is a decent amount of test images.\nThe following is one example of the cropped image with our prediction:
\n\n
Hence, we would really appreciate if the organisor could clarify whether those cropped images are not considered into the evaluation (or they were annotated first and than cropped?)
\n\nP.S.: we will also release our solution soon.
", "rawMarkdown": "The private leaderboard has been released and congrats to the top winners.\n\nAfter examine the private/public leaderboard, we have one last issue want to ask the organisor or the competition participants so that we can R.I.P....\n(A similar topic has been raised at: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653 )\n\nWe have detected the cropped image from the test set (around 400 images) and set them as empty in the prediction. The surprising result is that comparing with non-zeros predictions, they generate the same private LB score. Those 400 ish images constitute around 20% of the total images which is a decent amount of test images.\nThe following is one example of the cropped image with our prediction:\n\n\n\n\n\nHence, we would really appreciate if the organisor could clarify whether those cropped images are not considered into the evaluation (or they were annotated first and than cropped?)\n\nP.S.: we will also release our solution soon.", "votes": null }, { "id": "725378", "postDate": "01/22/2020 03:25:59", "content": "Congrats for 2nd place and thank you for sharing very interesting finding.
\nNow I understand why kaggle and host couldn't open the evaluation metric, their code might contain something related this.\nBut I still feel it's very unfair to do that without answering you and other participants questions....(if it's true).\nI hope we can get answer from organizer, but sadly we cannot expect that😔
Congrats for your Gold.
\n\nFrom my submissions, I guess they ignore flipped images as well(public and private LB)
", "rawMarkdown": "Congrats for your Gold.\n\nFrom my submissions, I guess they ignore flipped images as well(public and private LB)", "votes": null }, { "id": "725398", "postDate": "01/22/2020 03:44:40", "content": "I did something similar.\nI think flipped images and noized images in test data are dummy too.
", "rawMarkdown": "I did something similar.\nI think flipped images and noized images in test data are dummy too.", "votes": null }, { "id": "725463", "postDate": "01/22/2020 06:04:30", "content": "yep, a submission with all flipped and cropped images having an empty prediction string have the same result.
", "rawMarkdown": "yep, a submission with all flipped and cropped images having an empty prediction string have the same result.", "votes": null }, { "id": "725507", "postDate": "01/22/2020 07:30:34", "content": "Thank you for sharing your insightful study!\nI'm wondering if there was a need for the evaluation to be such a tricky one...
", "rawMarkdown": "Thank you for sharing your insightful study!\nI'm wondering if there was a need for the evaluation to be such a tricky one...", "votes": null }, { "id": "725598", "postDate": "01/22/2020 09:16:50", "content": "This makes sense, because otherwise that camera intrinsic matrix wont work, but also kinda ridiculous, because that is intentionally misleading competitors, in an also dishonest way.
", "rawMarkdown": "This makes sense, because otherwise that camera intrinsic matrix wont work, but also kinda ridiculous, because that is intentionally misleading competitors, in an also dishonest way.", "votes": null }, { "id": "725822", "postDate": "01/22/2020 14:19:11", "content": "Congratulations. \nI also did something to deal with the cropped or flipped samples. Some preprocess to recover the cropped images like this, bad to know they are ignored.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F979178%2Fe887a5e178efd5f59c2ed17cee1ed393%2FID_002c9d7ed.jpg?generation=1579702713489114&alt=media\")
\nAnyway Thanks for your insight findings and analysis. And Happy Chinese New Year.
\n\n\n(or they were annotated first and than cropped?)
\n
this is not the case either.\nI replaced every single picture in test set with the corresponding real image (and even tried with and without flipping) and it didn't change the score at all. Neither public, nor private LB
", "rawMarkdown": "> (or they were annotated first and than cropped?)\n\nthis is not the case either.\nI replaced every single picture in test set with the corresponding real image (and even tried with and without flipping) and it didn't change the score at all. Neither public, nor private LB", "votes": null }, { "id": "725998", "postDate": "01/22/2020 17:41:57", "content": "I feel you as I manually searched for the real picture for each cropped, resized or colored picture and replaced every single one of them ... If anyone is interested in the list: https://www.kaggle.com/c/pku-autonomous-driving/discussion/127162
", "rawMarkdown": "I feel you as I manually searched for the real picture for each cropped, resized or colored picture and replaced every single one of them ... If anyone is interested in the list: https://www.kaggle.com/c/pku-autonomous-driving/discussion/127162", "votes": null }, { "id": "726417", "postDate": "01/23/2020 02:12:53", "content": "I actually did not do any manual search. I firstly trained a classifier to classify normal, flip, crop, flip+crop. Then train a small keypoint detector to detect the two points(almost fixed in normal samples) like this, calculate out the scale factor and move the two keypoints to the positions they should be automatically. I recovered about 270 cropped or flipped+cropped test samples with this method. All the training data used to train the classifier and the keypoint detector is synthesized by cropping and flipping samples in the training dataset.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F979178%2F483b37bb2ad44d745f2f7b74e8fd0179%2FID_90b99dfc4.jpg?generation=1579745249176508&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2304617%2Fd34e872e8e8342fd583dfde13a771607%2F1072_ID_4d9c2171b.jpg?generation=1579780118503534&alt=media\")
\nI know the feeling. Here is my result.
Same here... but ultimately my own fault, should've paid more attention and probe lb carefully... all those effort just gone to waste...
", "rawMarkdown": "Same here... but ultimately my own fault, should've paid more attention and probe lb carefully... all those effort just gone to waste...", "votes": null }, { "id": "727277", "postDate": "01/23/2020 15:44:39", "content": "very thoughtful. well done.
", "rawMarkdown": "very thoughtful. well done.", "votes": null } ], "comments": [ { "id": 725378, "author_name": "bamps53", "author_url": "", "post_date": "01/22/2020 03:25:59", "content": "Congrats for 2nd place and thank you for sharing very interesting finding.
\nNow I understand why kaggle and host couldn't open the evaluation metric, their code might contain something related this.\nBut I still feel it's very unfair to do that without answering you and other participants questions....(if it's true).\nI hope we can get answer from organizer, but sadly we cannot expect that😔
Congrats for your Gold.
\n\nFrom my submissions, I guess they ignore flipped images as well(public and private LB)
", "votes": null, "replies": [] }, { "id": 725398, "author_name": "its7171", "author_url": "", "post_date": "01/22/2020 03:44:40", "content": "I did something similar.\nI think flipped images and noized images in test data are dummy too.
", "votes": null, "replies": [ { "id": 725463, "author_name": "ilu000", "author_url": "", "post_date": "01/22/2020 06:04:30", "content": "yep, a submission with all flipped and cropped images having an empty prediction string have the same result.
", "votes": null, "replies": [] } ] }, { "id": 725507, "author_name": "nihei123", "author_url": "", "post_date": "01/22/2020 07:30:34", "content": "Thank you for sharing your insightful study!\nI'm wondering if there was a need for the evaluation to be such a tricky one...
", "votes": null, "replies": [] }, { "id": 725598, "author_name": "yl1202", "author_url": "", "post_date": "01/22/2020 09:16:50", "content": "This makes sense, because otherwise that camera intrinsic matrix wont work, but also kinda ridiculous, because that is intentionally misleading competitors, in an also dishonest way.
", "votes": null, "replies": [] }, { "id": 725822, "author_name": "shihjwjj", "author_url": "", "post_date": "01/22/2020 14:19:11", "content": "Congratulations. \nI also did something to deal with the cropped or flipped samples. Some preprocess to recover the cropped images like this, bad to know they are ignored.\n\nAnyway Thanks for your insight findings and analysis. And Happy Chinese New Year.
I feel you as I manually searched for the real picture for each cropped, resized or colored picture and replaced every single one of them ... If anyone is interested in the list: https://www.kaggle.com/c/pku-autonomous-driving/discussion/127162
", "votes": null, "replies": [] }, { "id": 726417, "author_name": "shihjwjj", "author_url": "", "post_date": "01/23/2020 02:12:53", "content": "I actually did not do any manual search. I firstly trained a classifier to classify normal, flip, crop, flip+crop. Then train a small keypoint detector to detect the two points(almost fixed in normal samples) like this, calculate out the scale factor and move the two keypoints to the positions they should be automatically. I recovered about 270 cropped or flipped+cropped test samples with this method. All the training data used to train the classifier and the keypoint detector is synthesized by cropping and flipping samples in the training dataset.\n
\nI know the feeling. Here is my result.
Same here... but ultimately my own fault, should've paid more attention and probe lb carefully... all those effort just gone to waste...
", "votes": null, "replies": [] } ] }, { "id": 725978, "author_name": "ilu000", "author_url": "", "post_date": "01/22/2020 17:23:52", "content": "\n\n\n(or they were annotated first and than cropped?)
\n
this is not the case either.\nI replaced every single picture in test set with the corresponding real image (and even tried with and without flipping) and it didn't change the score at all. Neither public, nor private LB
", "votes": null, "replies": [ { "id": 727277, "author_name": "yl1202", "author_url": "", "post_date": "01/23/2020 15:44:39", "content": "very thoughtful. well done.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "725370": "The private leaderboard has been released and congrats to the top winners.\n\nAfter examine the private/public leaderboard, we have one last issue want to ask the organisor or the competition participants so that we can R.I.P....\n(A similar topic has been raised at: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653 )\n\nWe have detected the cropped image from the test set (around 400 images) and set them as empty in the prediction. The surprising result is that comparing with non-zeros predictions, they generate the same private LB score. Those 400 ish images constitute around 20% of the total images which is a decent amount of test images.\nThe following is one example of the cropped image with our prediction:\n\n\n\n\n\nHence, we would really appreciate if the organisor could clarify whether those cropped images are not considered into the evaluation (or they were annotated first and than cropped?)\n\nP.S.: we will also release our solution soon.", "725378": "Congrats for 2nd place and thank you for sharing very interesting finding. \nNow I understand why kaggle and host couldn't open the evaluation metric, their code might contain something related this.\nBut I still feel it's very unfair to do that without answering you and other participants questions....(if it's true).\nI hope we can get answer from organizer, but sadly we cannot expect that😔", "725394": "Congrats for your Gold.\n\nFrom my submissions, I guess they ignore flipped images as well(public and private LB)", "725398": "I did something similar.\nI think flipped images and noized images in test data are dummy too.", "725463": "yep, a submission with all flipped and cropped images having an empty prediction string have the same result.", "725507": "Thank you for sharing your insightful study!\nI'm wondering if there was a need for the evaluation to be such a tricky one...", "725598": "This makes sense, because otherwise that camera intrinsic matrix wont work, but also kinda ridiculous, because that is intentionally misleading competitors, in an also dishonest way.", "725822": "Congratulations. \nI also did something to deal with the cropped or flipped samples. Some preprocess to recover the cropped images like this, bad to know they are ignored.\n\nAnyway Thanks for your insight findings and analysis. And Happy Chinese New Year.", "725978": "> (or they were annotated first and than cropped?)\n\nthis is not the case either.\nI replaced every single picture in test set with the corresponding real image (and even tried with and without flipping) and it didn't change the score at all. Neither public, nor private LB", "725998": "I feel you as I manually searched for the real picture for each cropped, resized or colored picture and replaced every single one of them ... If anyone is interested in the list: https://www.kaggle.com/c/pku-autonomous-driving/discussion/127162", "726417": "I actually did not do any manual search. I firstly trained a classifier to classify normal, flip, crop, flip+crop. Then train a small keypoint detector to detect the two points(almost fixed in normal samples) like this, calculate out the scale factor and move the two keypoints to the positions they should be automatically. I recovered about 270 cropped or flipped+cropped test samples with this method. All the training data used to train the classifier and the keypoint detector is synthesized by cropping and flipping samples in the training dataset.\n", "727027": "\nI know the feeling. Here is my result.", "727274": "Same here... but ultimately my own fault, should've paid more attention and probe lb carefully... all those effort just gone to waste...", "727277": "very thoughtful. well done." }, "source": "meta" }