I followed : https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#object-detection-track\nand expansion went well but when I run the evaluation script I get:
\n\nobject_detection/metrics/oid_od_challenge_evaluation.py:74: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\nof pandas will change to not sort by default.\n\nTo accept the future behavior, pass 'sort=False'.\n\nTo retain the current behavior and silence the warning, pass 'sort=True'.\n\n all_annotations = pd.concat([all_box_annotations, all_label_annotations])\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:52: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes[['YMin', 'XMin', 'YMax', 'XMax']].as_matrix(),\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:54: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes['LabelName'].map(lambda x: class_label_map[x]).as_matrix(),\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:56: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes['IsGroupOf'].as_matrix().astype(int),\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:58: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_labels['LabelName'].map(lambda x: class_label_map[x])\nTraceback (most recent call last):\n File \"object_detection/metrics/oid_od_challenge_evaluation.py\", line 128, in <module>\n main(args)\n File \"object_detection/metrics/oid_od_challenge_evaluation.py\", line 84, in main\n image_groundtruth, class_label_map)\n File \"/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py\", line 54, in build_groundtruth_boxes_dictionary\n data_boxes['LabelName'].map(lambda x: class_label_map[x]).as_matrix(),\n File \"/home/m/.virtualenvs/tf/local/lib/python2.7/site-packages/pandas/core/series.py\", line 2998, in map\n arg, na_action=na_action)\n File \"/home/m/.virtualenvs/tf/local/lib/python2.7/site-packages/pandas/core/base.py\", line 1004, in _map_values\n new_values = map_f(values, mapper)\n File \"pandas/_libs/src/inference.pyx\", line 1472, in pandas._libs.lib.map_infer\n File \"/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py\", line 54, in <lambda>\n data_boxes['LabelName'].map(lambda x: class_label_map[x]).as_matrix(),\nKeyError: 'LabelName'\nI followed : https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#object-detection-track\nand expansion went well but when I run the evaluation script I get:
\n\nobject_detection/metrics/oid_od_challenge_evaluation.py:74: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\nof pandas will change to not sort by default.\n\nTo accept the future behavior, pass 'sort=False'.\n\nTo retain the current behavior and silence the warning, pass 'sort=True'.\n\n all_annotations = pd.concat([all_box_annotations, all_label_annotations])\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:52: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes[['YMin', 'XMin', 'YMax', 'XMax']].as_matrix(),\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:54: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes['LabelName'].map(lambda x: class_label_map[x]).as_matrix(),\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:56: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes['IsGroupOf'].as_matrix().astype(int),\n/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:58: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_labels['LabelName'].map(lambda x: class_label_map[x])\nTraceback (most recent call last):\n File \"object_detection/metrics/oid_od_challenge_evaluation.py\", line 128, in <module>\n main(args)\n File \"object_detection/metrics/oid_od_challenge_evaluation.py\", line 84, in main\n image_groundtruth, class_label_map)\n File \"/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py\", line 54, in build_groundtruth_boxes_dictionary\n data_boxes['LabelName'].map(lambda x: class_label_map[x]).as_matrix(),\n File \"/home/m/.virtualenvs/tf/local/lib/python2.7/site-packages/pandas/core/series.py\", line 2998, in map\n arg, na_action=na_action)\n File \"/home/m/.virtualenvs/tf/local/lib/python2.7/site-packages/pandas/core/base.py\", line 1004, in _map_values\n new_values = map_f(values, mapper)\n File \"pandas/_libs/src/inference.pyx\", line 1472, in pandas._libs.lib.map_infer\n File \"/home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py\", line 54, in <lambda>\n data_boxes['LabelName'].map(lambda x: class_label_map[x]).as_matrix(),\nKeyError: 'LabelName'\nIn your challenge-2018-train-annotations-bbox_expanded.csv, is there a column LabelName. Maybe the column was not written into the expanded a csv incorrectly(misspelling?), or it was not written at all. To make sure the error doesn't happen again, make sure you have a LabelName column in your detections csv as well.
", "rawMarkdown": "In your challenge-2018-train-annotations-bbox_expanded.csv, is there a column LabelName. Maybe the column was not written into the expanded a csv incorrectly(misspelling?), or it was not written at all. To make sure the error doesn't happen again, make sure you have a LabelName column in your detections csv as well.", "votes": null }, { "id": "357920", "postDate": "07/17/2018 06:33:11", "content": "wait, i think i can see why...
", "rawMarkdown": "wait, i think i can see why...", "votes": null }, { "id": "357922", "postDate": "07/17/2018 06:42:36", "content": "nope, false hope. is this how your files looks like?
\n\n\n(tf) m@dl1:~/models/research/oid$ head my-bbox.csv\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.012500,0.195312,0.148438,0.587500,0,1,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.025000,0.276563,0.714063,0.948438,0,1,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.151562,0.310937,0.198437,0.590625,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.256250,0.429688,0.651563,0.925000,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.257812,0.346875,0.235938,0.385938,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.320312,0.368750,0.260938,0.328125,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.412500,0.945312,0.120312,0.475000,1,0,1,0,0\n000002b66c9c498e,xclick,/m/04bcr3,1,0.064062,0.492188,0.889063,0.993750,0,0,0,0,0\n000002b66c9c498e,xclick,/m/04bcr3,1,0.301562,0.990625,0.462500,0.595312,0,0,0,0,0\n(tf) m@dl1:~/models/research/oid$ head my-bbox_expanded.csv\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.012500,0.195312,0.148438,0.587500,0,1,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.025000,0.276563,0.714063,0.948438,0,1,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.151562,0.310937,0.198437,0.590625,1,0,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.256250,0.429688,0.651563,0.925000,1,0,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.257812,0.346875,0.235938,0.385938,1,0,0,0,0\n(tf) m@dl1:~/models/research/oid$ head my-human_expanded.csv\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/04bcr3,1\n000002b66c9c498e,verification,/m/0c_jw,1\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/06z37_,0\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/017ftj,0\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/029tx,0\nImageID,Source,LabelName,Confidence\n
Has anyone managed to make it work? is the prediction files in the format kaggle expects?
", "rawMarkdown": "Has anyone managed to make it work? is the prediction files in the format kaggle expects?", "votes": null }, { "id": "359666", "postDate": "07/20/2018 14:55:04", "content": "The expected file format is NOT the same as Kaggle format.
\n\nWhen using oid_od_challenge_evaluation_utils.py:
\n\nPrediction file format has the same spirit as groundtruth file format (i.e. has a box\nper row and contains columns
\n\nImageID,LabelName,Score,XMin,YMin,XMax,YMax\n6b5bbebdaa1d9cfe,/m/09tvcd,0.35969013,0.0508683,0.14803758,0.10902788,0.32285407\nec0ed974fcd48948,/m/09tvcd,0.74969375,0.08453516,0.76285881,0.1835296,0.97871614
When using offline_evaluation.py:\nThe data must be first converted to tf.Example format. Check out the tutorial here
", "rawMarkdown": "The expected file format is NOT the same as Kaggle format. \n\nWhen using oid_od_challenge_evaluation_utils.py:\n\n - Ground truth file format is expected to be the same as the\n ground-truth provided [on the challenge website][1] \n - Prediction file format has the same spirit as groundtruth file format (i.e. has a box\n per row and contains columns\n\n ImageID,LabelName,Score,XMin,YMin,XMax,YMax\n 6b5bbebdaa1d9cfe,/m/09tvcd,0.35969013,0.0508683,0.14803758,0.10902788,0.32285407\n ec0ed974fcd48948,/m/09tvcd,0.74969375,0.08453516,0.76285881,0.1835296,0.97871614\n\nWhen using offline_evaluation.py:\nThe data must be first converted to tf.Example format. Check out the tutorial here\n [1]: https://storage.googleapis.com/openimages/web/challenge.html", "votes": null }, { "id": "359709", "postDate": "07/20/2018 16:35:40", "content": "I did manage to get the offline challenge evaluation to give me a sensible-looking output.
\n\nThe snippet you posted looks close to what the script wants, but it seems like the column headers have gotten repeated after each image in my-bbox_expanded.csv. Also, my browser is rendering your snippet of my-bbox.csv as a single line, but that may just be a web formatting thing.
", "rawMarkdown": "I did manage to get the offline challenge evaluation to give me a sensible-looking output.\n\nThe snippet you posted looks close to what the script wants, but it seems like the column headers have gotten repeated after each image in my-bbox_expanded.csv. Also, my browser is rendering your snippet of my-bbox.csv as a single line, but that may just be a web formatting thing.", "votes": null }, { "id": "359815", "postDate": "07/20/2018 21:11:20", "content": "Thank you Alina, but still no joy.\nhere is a super simply colab notebook i made. it trims everything to just one ImageID, to make it super simple.\nI have checked and rechecked my steps. while stupid mistakes are still possible, I just can't find what goes wrong. I do not what to use the offline evaluation but the csv one. The strange part is the expansion. it seems to repeat the column headers.
", "rawMarkdown": "Thank you Alina, but still no joy.\n[here][1] is a super simply colab notebook i made. it trims everything to just one ImageID, to make it super simple.\nI have checked and rechecked my steps. while stupid mistakes are still possible, I just can't find what goes wrong. I do not what to use the offline evaluation but the csv one. The strange part is the expansion. it seems to repeat the column headers.\n\n [1]: https://gist.github.com/mosheliv/de4ec04cb31cbb0d786fe38eb4c4531b", "votes": null }, { "id": "359816", "postDate": "07/20/2018 21:13:04", "content": "it is just problems with preview. you can see it better here (colab notebook)
", "rawMarkdown": "it is just problems with preview. you can see it better [here][1] (colab notebook)\n\n\n [1]: https://gist.github.com/mosheliv/de4ec04cb31cbb0d786fe38eb4c4531b", "votes": null }, { "id": "359820", "postDate": "07/20/2018 21:18:32", "content": "also, there was no link tothe tutorial in the last \"here\" in you post. I might just be using the wrong tutorial...
", "rawMarkdown": "also, there was no link tothe tutorial in the last \"here\" in you post. I might just be using the wrong tutorial...", "votes": null }, { "id": "359845", "postDate": "07/20/2018 22:45:36", "content": "Looking at the colab notebook you linked, it's like you say in your \"step 4\" header: the headers are repeated in the expanded ground-truth csv file.
\n\nThe error message that python is giving you is a little misleading -- I think the key error it's complaining about is not from the data_boxes['LabelName'] lookup; the \"LabelName\" there is just a coincidence. I think the actual key error is coming from the class_label_map[x] lookup. It's reading one of those repeated-header lines, finding that the class label for this box is \"LabelName\" instead of an actual label name like \"m/01g317\" or whatever, and throwing an error when the lookup fails.
\n\nI should have mentioned in my previous post that I had to make a small change to the oid_hierarchical_labels_expansion.py script: on line 170, where the original code reads \"expanded_lines = [header] + expanded_lines\", I added an \"if\" statement to check if this is the first pass through the loop, and only add the header then. So my expanded inputs only have the headers in the first line of the .csv.
", "rawMarkdown": "Looking at the colab notebook you linked, it's like you say in your \"step 4\" header: the headers are repeated in the expanded ground-truth csv file. \n\nThe error message that python is giving you is a little misleading -- I think the key error it's complaining about is not from the data_boxes['LabelName'] lookup; the \"LabelName\" there is just a coincidence. I think the actual key error is coming from the class_label_map[x] lookup. It's reading one of those repeated-header lines, finding that the class label for this box is \"LabelName\" instead of an actual label name like \"m/01g317\" or whatever, and throwing an error when the lookup fails.\n\nI should have mentioned in my previous post that I had to make a small change to the oid_hierarchical_labels_expansion.py script: on line 170, where the original code reads \"expanded_lines = [header] + expanded_lines\", I added an \"if\" statement to check if this is the first pass through the loop, and only add the header then. So my expanded inputs only have the headers in the first line of the .csv.", "votes": null }, { "id": "359866", "postDate": "07/21/2018 00:04:55", "content": "Ahhhh that makes sense. I am trying in these cases to follow the vendors instructions to the letter so if they are wrong, they can fix it for the interest of humanity. Thanks for your explanation. I will do the same as i have wasted enough time on trying to be strict. Hopefully @alina will have someone correct the script and clarify the format of the prediction file in the tutorial.
", "rawMarkdown": "Ahhhh that makes sense. I am trying in these cases to follow the vendors instructions to the letter so if they are wrong, they can fix it for the interest of humanity. Thanks for your explanation. I will do the same as i have wasted enough time on trying to be strict. Hopefully @alina will have someone correct the script and clarify the format of the prediction file in the tutorial.", "votes": null }, { "id": "359873", "postDate": "07/21/2018 00:52:58", "content": "should anyone else encounter the problem:
\n\nsed -i \"171 i \\ \\ \\ \\ \\ \\ \\ \\ header=\\\"\\\"\" object_detection/dataset_tools/oid_hierarchical_labels_expansion.py
fixes it
\n\nmakes you wonder how many times they have checked their script....
", "rawMarkdown": "should anyone else encounter the problem:\n\n`sed -i \"171 i \\ \\ \\ \\ \\ \\ \\ \\ header=\\\"\\\"\" object_detection/dataset_tools/oid_hierarchical_labels_expansion.py`\n\nfixes it\n\nmakes you wonder how many times they have checked their script....", "votes": null } ], "comments": [ { "id": 357777, "author_name": "arpandhatt", "author_url": "", "post_date": "07/16/2018 20:32:23", "content": "In your challenge-2018-train-annotations-bbox_expanded.csv, is there a column LabelName. Maybe the column was not written into the expanded a csv incorrectly(misspelling?), or it was not written at all. To make sure the error doesn't happen again, make sure you have a LabelName column in your detections csv as well.
", "votes": null, "replies": [] }, { "id": 357920, "author_name": "moshel", "author_url": "", "post_date": "07/17/2018 06:33:11", "content": "wait, i think i can see why...
", "votes": null, "replies": [ { "id": 357922, "author_name": "moshel", "author_url": "", "post_date": "07/17/2018 06:42:36", "content": "nope, false hope. is this how your files looks like?
\n\n\n(tf) m@dl1:~/models/research/oid$ head my-bbox.csv\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.012500,0.195312,0.148438,0.587500,0,1,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.025000,0.276563,0.714063,0.948438,0,1,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.151562,0.310937,0.198437,0.590625,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.256250,0.429688,0.651563,0.925000,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.257812,0.346875,0.235938,0.385938,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.320312,0.368750,0.260938,0.328125,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.412500,0.945312,0.120312,0.475000,1,0,1,0,0\n000002b66c9c498e,xclick,/m/04bcr3,1,0.064062,0.492188,0.889063,0.993750,0,0,0,0,0\n000002b66c9c498e,xclick,/m/04bcr3,1,0.301562,0.990625,0.462500,0.595312,0,0,0,0,0\n(tf) m@dl1:~/models/research/oid$ head my-bbox_expanded.csv\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.012500,0.195312,0.148438,0.587500,0,1,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.025000,0.276563,0.714063,0.948438,0,1,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.151562,0.310937,0.198437,0.590625,1,0,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.256250,0.429688,0.651563,0.925000,1,0,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.257812,0.346875,0.235938,0.385938,1,0,0,0,0\n(tf) m@dl1:~/models/research/oid$ head my-human_expanded.csv\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/04bcr3,1\n000002b66c9c498e,verification,/m/0c_jw,1\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/06z37_,0\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/017ftj,0\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/029tx,0\nImageID,Source,LabelName,Confidence\n
I did manage to get the offline challenge evaluation to give me a sensible-looking output.
\n\nThe snippet you posted looks close to what the script wants, but it seems like the column headers have gotten repeated after each image in my-bbox_expanded.csv. Also, my browser is rendering your snippet of my-bbox.csv as a single line, but that may just be a web formatting thing.
", "votes": null, "replies": [] }, { "id": 359816, "author_name": "moshel", "author_url": "", "post_date": "07/20/2018 21:13:04", "content": "it is just problems with preview. you can see it better here (colab notebook)
", "votes": null, "replies": [] }, { "id": 359845, "author_name": "particlebbq", "author_url": "", "post_date": "07/20/2018 22:45:36", "content": "Looking at the colab notebook you linked, it's like you say in your \"step 4\" header: the headers are repeated in the expanded ground-truth csv file.
\n\nThe error message that python is giving you is a little misleading -- I think the key error it's complaining about is not from the data_boxes['LabelName'] lookup; the \"LabelName\" there is just a coincidence. I think the actual key error is coming from the class_label_map[x] lookup. It's reading one of those repeated-header lines, finding that the class label for this box is \"LabelName\" instead of an actual label name like \"m/01g317\" or whatever, and throwing an error when the lookup fails.
\n\nI should have mentioned in my previous post that I had to make a small change to the oid_hierarchical_labels_expansion.py script: on line 170, where the original code reads \"expanded_lines = [header] + expanded_lines\", I added an \"if\" statement to check if this is the first pass through the loop, and only add the header then. So my expanded inputs only have the headers in the first line of the .csv.
", "votes": null, "replies": [] }, { "id": 359866, "author_name": "moshel", "author_url": "", "post_date": "07/21/2018 00:04:55", "content": "Ahhhh that makes sense. I am trying in these cases to follow the vendors instructions to the letter so if they are wrong, they can fix it for the interest of humanity. Thanks for your explanation. I will do the same as i have wasted enough time on trying to be strict. Hopefully @alina will have someone correct the script and clarify the format of the prediction file in the tutorial.
", "votes": null, "replies": [] } ] }, { "id": 359585, "author_name": "moshel", "author_url": "", "post_date": "07/20/2018 11:56:54", "content": "Has anyone managed to make it work? is the prediction files in the format kaggle expects?
", "votes": null, "replies": [] }, { "id": 359666, "author_name": "akuznetsa", "author_url": "", "post_date": "07/20/2018 14:55:04", "content": "The expected file format is NOT the same as Kaggle format.
\n\nWhen using oid_od_challenge_evaluation_utils.py:
\n\nPrediction file format has the same spirit as groundtruth file format (i.e. has a box\nper row and contains columns
\n\nImageID,LabelName,Score,XMin,YMin,XMax,YMax\n6b5bbebdaa1d9cfe,/m/09tvcd,0.35969013,0.0508683,0.14803758,0.10902788,0.32285407\nec0ed974fcd48948,/m/09tvcd,0.74969375,0.08453516,0.76285881,0.1835296,0.97871614
When using offline_evaluation.py:\nThe data must be first converted to tf.Example format. Check out the tutorial here
", "votes": null, "replies": [ { "id": 359815, "author_name": "moshel", "author_url": "", "post_date": "07/20/2018 21:11:20", "content": "Thank you Alina, but still no joy.\nhere is a super simply colab notebook i made. it trims everything to just one ImageID, to make it super simple.\nI have checked and rechecked my steps. while stupid mistakes are still possible, I just can't find what goes wrong. I do not what to use the offline evaluation but the csv one. The strange part is the expansion. it seems to repeat the column headers.
", "votes": null, "replies": [] }, { "id": 359820, "author_name": "moshel", "author_url": "", "post_date": "07/20/2018 21:18:32", "content": "also, there was no link tothe tutorial in the last \"here\" in you post. I might just be using the wrong tutorial...
", "votes": null, "replies": [] } ] }, { "id": 359873, "author_name": "moshel", "author_url": "", "post_date": "07/21/2018 00:52:58", "content": "should anyone else encounter the problem:
\n\nsed -i \"171 i \\ \\ \\ \\ \\ \\ \\ \\ header=\\\"\\\"\" object_detection/dataset_tools/oid_hierarchical_labels_expansion.py
fixes it
\n\nmakes you wonder how many times they have checked their script....
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "356627": "I followed : https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#object-detection-track\nand expansion went well but when I run the evaluation script I get:\n\n\n object_detection/metrics/oid_od_challenge_evaluation.py:74: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n of pandas will change to not sort by default.\n \n To accept the future behavior, pass 'sort=False'.\n \n To retain the current behavior and silence the warning, pass 'sort=True'.\n \n all_annotations = pd.concat([all_box_annotations, all_label_annotations])\n /home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:52: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes[['YMin', 'XMin', 'YMax', 'XMax']].as_matrix(),\n /home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:54: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes['LabelName'].map(lambda x: class_label_map[x]).as_matrix(),\n /home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:56: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_boxes['IsGroupOf'].as_matrix().astype(int),\n /home/m/models/research/object_detection/metrics/oid_od_challenge_evaluation_utils.py:58: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n data_labels['LabelName'].map(lambda x: class_label_map[x])\n Traceback (most recent call last):\n File \"object_detection/metrics/oid_od_challenge_evaluation.py\", line 128, in", "357777": "In your challenge-2018-train-annotations-bbox_expanded.csv, is there a column LabelName. Maybe the column was not written into the expanded a csv incorrectly(misspelling?), or it was not written at all. To make sure the error doesn't happen again, make sure you have a LabelName column in your detections csv as well.", "357920": "wait, i think i can see why...", "357922": "nope, false hope. is this how your files looks like?\n\n```\n(tf) m@dl1:~/models/research/oid$ head my-bbox.csv\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.012500,0.195312,0.148438,0.587500,0,1,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.025000,0.276563,0.714063,0.948438,0,1,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.151562,0.310937,0.198437,0.590625,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.256250,0.429688,0.651563,0.925000,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.257812,0.346875,0.235938,0.385938,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.320312,0.368750,0.260938,0.328125,1,0,0,0,0\n000002b66c9c498e,xclick,/m/01g317,1,0.412500,0.945312,0.120312,0.475000,1,0,1,0,0\n000002b66c9c498e,xclick,/m/04bcr3,1,0.064062,0.492188,0.889063,0.993750,0,0,0,0,0\n000002b66c9c498e,xclick,/m/04bcr3,1,0.301562,0.990625,0.462500,0.595312,0,0,0,0,0\n(tf) m@dl1:~/models/research/oid$ head my-bbox_expanded.csv\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.012500,0.195312,0.148438,0.587500,0,1,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.025000,0.276563,0.714063,0.948438,0,1,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.151562,0.310937,0.198437,0.590625,1,0,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.256250,0.429688,0.651563,0.925000,1,0,0,0,0\nImageID,Source,LabelName,Confidence,XMin,XMax,YMin,YMax,IsOccluded,IsTruncated,IsGroupOf,IsDepiction,IsInside\n000002b66c9c498e,xclick,/m/01g317,1,0.257812,0.346875,0.235938,0.385938,1,0,0,0,0\n(tf) m@dl1:~/models/research/oid$ head my-human_expanded.csv\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/04bcr3,1\n000002b66c9c498e,verification,/m/0c_jw,1\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/06z37_,0\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/017ftj,0\nImageID,Source,LabelName,Confidence\n000002b66c9c498e,verification,/m/029tx,0\nImageID,Source,LabelName,Confidence\n```", "359585": "Has anyone managed to make it work? is the prediction files in the format kaggle expects?", "359666": "The expected file format is NOT the same as Kaggle format. \n\nWhen using oid_od_challenge_evaluation_utils.py:\n\n - Ground truth file format is expected to be the same as the\n ground-truth provided [on the challenge website][1] \n - Prediction file format has the same spirit as groundtruth file format (i.e. has a box\n per row and contains columns\n\n ImageID,LabelName,Score,XMin,YMin,XMax,YMax\n 6b5bbebdaa1d9cfe,/m/09tvcd,0.35969013,0.0508683,0.14803758,0.10902788,0.32285407\n ec0ed974fcd48948,/m/09tvcd,0.74969375,0.08453516,0.76285881,0.1835296,0.97871614\n\nWhen using offline_evaluation.py:\nThe data must be first converted to tf.Example format. Check out the tutorial here\n [1]: https://storage.googleapis.com/openimages/web/challenge.html", "359709": "I did manage to get the offline challenge evaluation to give me a sensible-looking output.\n\nThe snippet you posted looks close to what the script wants, but it seems like the column headers have gotten repeated after each image in my-bbox_expanded.csv. Also, my browser is rendering your snippet of my-bbox.csv as a single line, but that may just be a web formatting thing.", "359815": "Thank you Alina, but still no joy.\n[here][1] is a super simply colab notebook i made. it trims everything to just one ImageID, to make it super simple.\nI have checked and rechecked my steps. while stupid mistakes are still possible, I just can't find what goes wrong. I do not what to use the offline evaluation but the csv one. The strange part is the expansion. it seems to repeat the column headers.\n\n [1]: https://gist.github.com/mosheliv/de4ec04cb31cbb0d786fe38eb4c4531b", "359816": "it is just problems with preview. you can see it better [here][1] (colab notebook)\n\n\n [1]: https://gist.github.com/mosheliv/de4ec04cb31cbb0d786fe38eb4c4531b", "359820": "also, there was no link tothe tutorial in the last \"here\" in you post. I might just be using the wrong tutorial...", "359845": "Looking at the colab notebook you linked, it's like you say in your \"step 4\" header: the headers are repeated in the expanded ground-truth csv file. \n\nThe error message that python is giving you is a little misleading -- I think the key error it's complaining about is not from the data_boxes['LabelName'] lookup; the \"LabelName\" there is just a coincidence. I think the actual key error is coming from the class_label_map[x] lookup. It's reading one of those repeated-header lines, finding that the class label for this box is \"LabelName\" instead of an actual label name like \"m/01g317\" or whatever, and throwing an error when the lookup fails.\n\nI should have mentioned in my previous post that I had to make a small change to the oid_hierarchical_labels_expansion.py script: on line 170, where the original code reads \"expanded_lines = [header] + expanded_lines\", I added an \"if\" statement to check if this is the first pass through the loop, and only add the header then. So my expanded inputs only have the headers in the first line of the .csv.", "359866": "Ahhhh that makes sense. I am trying in these cases to follow the vendors instructions to the letter so if they are wrong, they can fix it for the interest of humanity. Thanks for your explanation. I will do the same as i have wasted enough time on trying to be strict. Hopefully @alina will have someone correct the script and clarify the format of the prediction file in the tutorial.", "359873": "should anyone else encounter the problem:\n\n`sed -i \"171 i \\ \\ \\ \\ \\ \\ \\ \\ header=\\\"\\\"\" object_detection/dataset_tools/oid_hierarchical_labels_expansion.py`\n\nfixes it\n\nmakes you wonder how many times they have checked their script...." }, "source": "meta" }