I followed https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#object-detection-track to evaluate the validation mAP and found the validation mAP Score is much higher than the online test OpenImagesObjectDetectionAP(about 0.10 mAP).
\n\nAnd I am sure the training set and validation set that I used is excatly the same with the offical recommeded split. So the gap between the validation mAP and the online test mAP really bothers me.
\n\nHas anyone else encountered the same place?
", "messages": [ { "id": 360706, "postDate": "2018-07-23T04:14:54.413Z", "content": "I followed https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#object-detection-track to evaluate the validation mAP and found the validation mAP Score is much higher than the online test OpenImagesObjectDetectionAP(about 0.10 mAP).
\n\nAnd I am sure the training set and validation set that I used is excatly the same with the offical recommeded split. So the gap between the validation mAP and the online test mAP really bothers me.
\n\nHas anyone else encountered the same place?
", "rawMarkdown": "I followed https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#object-detection-track to evaluate the validation mAP and found the validation mAP Score is much higher than the online test OpenImagesObjectDetectionAP(about 0.10 mAP). \n\nAnd I am sure the training set and validation set that I used is excatly the same with the offical recommeded split. So the gap between the validation mAP and the online test mAP really bothers me. \n\nHas anyone else encountered the same place?", "votes": 3 }, { "id": 367061, "postDate": "2018-08-07T02:14:35.583Z", "content": "same problem~\nHave you solved it ?\nlooking for your reply~
", "rawMarkdown": "same problem~\nHave you solved it ?\nlooking for your reply~", "replies": [ { "id": 367604, "postDate": "2018-08-08T04:41:05.233Z", "content": "The problem still exists, I think it is possibly a inherent property of the dataset.
", "rawMarkdown": "The problem still exists, I think it is possibly a inherent property of the dataset." }, { "id": 369465, "postDate": "2018-08-13T05:40:48.830Z", "content": "How much is your mAP gap between the two sets?
", "rawMarkdown": "How much is your mAP gap between the two sets?" }, { "id": 369549, "postDate": "2018-08-13T10:26:09.360Z", "content": "0.1map
", "rawMarkdown": "0.1map" } ] }, { "id": 364301, "postDate": "2018-07-31T08:57:44.477Z", "content": "Hi, where did you download your validation dataset? \nMy validation set was followed by the official recommended and have 100000 images, and I get nearly correct result~
", "rawMarkdown": "Hi, where did you download your validation dataset? \nMy validation set was followed by the official recommended and have 100000 images, and I get nearly correct result~", "replies": [ { "id": 364363, "postDate": "2018-07-31T11:34:28.343Z", "content": "I exactly use the official recommended validation set, and I follow the tutorial at https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md, I expand the bounding box annotations and image label annotations as illustrated. And my prediction list each bounding box in a new line with (ImageID,LabelName,Score,XMin,YMin,XMax,YMax), then I run the latest TF ObjectDetection API to evaluate it, concretely, we run the oid_od_challenge_evaluation.py script as advised, then I get a csv file and find that my result of validation is 0.10 higher compared to the test result on leaderboard. Any detailed difference with your evaluation process?
", "rawMarkdown": "I exactly use the official recommended validation set, and I follow the tutorial at https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md, I expand the bounding box annotations and image label annotations as illustrated. And my prediction list each bounding box in a new line with (ImageID,LabelName,Score,XMin,YMin,XMax,YMax), then I run the latest TF ObjectDetection API to evaluate it, concretely, we run the oid_od_challenge_evaluation.py script as advised, then I get a csv file and find that my result of validation is 0.10 higher compared to the test result on leaderboard. Any detailed difference with your evaluation process?", "votes": 1 }, { "id": 366780, "postDate": "2018-08-06T13:45:50.933Z", "content": "Do you use the script for expanding classes for both local validation and submission?
", "rawMarkdown": "Do you use the script for expanding classes for both local validation and submission?" }, { "id": 366788, "postDate": "2018-08-06T14:11:38.150Z", "content": "Yes, I have expanded the bounding boxes label and images label for both validation and test set with the offered python scripts
", "rawMarkdown": "Yes, I have expanded the bounding boxes label and images label for both validation and test set with the offered python scripts", "votes": 1 }, { "id": 367063, "postDate": "2018-08-07T02:19:06.790Z", "content": "I got the same problem...\nsomeone help?
", "rawMarkdown": "I got the same problem...\nsomeone help?" } ] }, { "id": 362418, "postDate": "2018-07-26T10:07:29.617Z", "content": "I found 10 points diff in Score metric, not mAP.
", "rawMarkdown": "I found 10 points diff in Score metric, not mAP." }, { "id": 362416, "postDate": "2018-07-26T10:05:53.473Z", "content": "same problem. any progress?
", "rawMarkdown": "same problem. any progress?" }, { "id": 369463, "postDate": "2018-08-13T05:36:47.943Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 374851, "postDate": "2018-08-24T01:48:49.660Z", "content": "你是中国人嘛
", "rawMarkdown": "你是中国人嘛" } ] } ], "comments": [ { "id": 367061, "author_name": "MagicWolf", "author_url": "", "post_date": "2018-08-07T02:14:35.583000", "content": "same problem~\nHave you solved it ?\nlooking for your reply~
", "votes": 0, "replies": [ { "id": 367604, "author_name": "ChristianFans", "author_url": "", "post_date": "2018-08-08T04:41:05.233000", "content": "The problem still exists, I think it is possibly a inherent property of the dataset.
", "votes": 0, "replies": [] }, { "id": 369465, "author_name": "Louis", "author_url": "", "post_date": "2018-08-13T05:40:48.830000", "content": "How much is your mAP gap between the two sets?
", "votes": 0, "replies": [] }, { "id": 369549, "author_name": "MagicWolf", "author_url": "", "post_date": "2018-08-13T10:26:09.360000", "content": "0.1map
", "votes": 0, "replies": [] } ] }, { "id": 364301, "author_name": "MagicWolf", "author_url": "", "post_date": "2018-07-31T08:57:44.477000", "content": "Hi, where did you download your validation dataset? \nMy validation set was followed by the official recommended and have 100000 images, and I get nearly correct result~
", "votes": 0, "replies": [ { "id": 364363, "author_name": "ChristianFans", "author_url": "", "post_date": "2018-07-31T11:34:28.343000", "content": "I exactly use the official recommended validation set, and I follow the tutorial at https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md, I expand the bounding box annotations and image label annotations as illustrated. And my prediction list each bounding box in a new line with (ImageID,LabelName,Score,XMin,YMin,XMax,YMax), then I run the latest TF ObjectDetection API to evaluate it, concretely, we run the oid_od_challenge_evaluation.py script as advised, then I get a csv file and find that my result of validation is 0.10 higher compared to the test result on leaderboard. Any detailed difference with your evaluation process?
", "votes": 1, "replies": [] }, { "id": 366780, "author_name": "taraspiotr", "author_url": "", "post_date": "2018-08-06T13:45:50.933000", "content": "Do you use the script for expanding classes for both local validation and submission?
", "votes": 0, "replies": [] }, { "id": 366788, "author_name": "ChristianFans", "author_url": "", "post_date": "2018-08-06T14:11:38.150000", "content": "Yes, I have expanded the bounding boxes label and images label for both validation and test set with the offered python scripts
", "votes": 1, "replies": [] }, { "id": 367063, "author_name": "MagicWolf", "author_url": "", "post_date": "2018-08-07T02:19:06.790000", "content": "I got the same problem...\nsomeone help?
", "votes": 0, "replies": [] } ] }, { "id": 362418, "author_name": "fromsea", "author_url": "", "post_date": "2018-07-26T10:07:29.617000", "content": "I found 10 points diff in Score metric, not mAP.
", "votes": 0, "replies": [] }, { "id": 362416, "author_name": "fromsea", "author_url": "", "post_date": "2018-07-26T10:05:53.473000", "content": "same problem. any progress?
", "votes": 0, "replies": [] }, { "id": 369463, "author_name": "", "author_url": "", "post_date": "2018-08-13T05:36:47.943000", "content": "", "votes": 0, "replies": [ { "id": 374851, "author_name": "MagicWolf", "author_url": "", "post_date": "2018-08-24T01:48:49.660000", "content": "你是中国人嘛
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "360706": "I followed https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#object-detection-track to evaluate the validation mAP and found the validation mAP Score is much higher than the online test OpenImagesObjectDetectionAP(about 0.10 mAP). \n\nAnd I am sure the training set and validation set that I used is excatly the same with the offical recommeded split. So the gap between the validation mAP and the online test mAP really bothers me. \n\nHas anyone else encountered the same place?", "367061": "same problem~\nHave you solved it ?\nlooking for your reply~", "364301": "Hi, where did you download your validation dataset? \nMy validation set was followed by the official recommended and have 100000 images, and I get nearly correct result~", "362418": "I found 10 points diff in Score metric, not mAP.", "362416": "same problem. any progress?", "369463": "" } }