Hi, \nI wrote a discussion post about a week ago (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915) which showed the results we got and still received a 0 score from the evaluation. This also seems to be a widespread problem as lot of people seem to be getting this
\n\nWe have addressed all the suggestions made by @polosin but are still getting a zero score. Thanks Oleg for all the help!
\n\nLet's assume that the Mask RCNN or Resnet based attribute model we have trained in not trained enough. Even then based on the results we are getting (see image in linked post) how can the auto-grader give a zero score. I mean shouldn't it at least be something like 0.001?
\n\nWe are running our models for more epochs in hope of getting better results but it's odd that no even one image is correct. This leads me to think that issue might be with something in the submission format rather than training of the model.
\n\nClearly the hosts/creators of this competition don't care enough about the competition to participate in the discussion forum or give an evaluation script. So being at my wit's end I am asking the participants at the top of the leader-board to help out with suggestions that might explain our 0.0 score. Any help/suggestion would be greatly appreciated! Thanks in advance!\n@polosin @loushun @guiguzhixing @shraddhaamohan
\n\nPlease look at the previous discussion too (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915)
", "messages": [ { "id": "851758", "postDate": "05/17/2020 22:21:52", "content": "Hi, \nI wrote a discussion post about a week ago (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915) which showed the results we got and still received a 0 score from the evaluation. This also seems to be a widespread problem as lot of people seem to be getting this
\n\nWe have addressed all the suggestions made by @polosin but are still getting a zero score. Thanks Oleg for all the help!
\n\nLet's assume that the Mask RCNN or Resnet based attribute model we have trained in not trained enough. Even then based on the results we are getting (see image in linked post) how can the auto-grader give a zero score. I mean shouldn't it at least be something like 0.001?
\n\nWe are running our models for more epochs in hope of getting better results but it's odd that no even one image is correct. This leads me to think that issue might be with something in the submission format rather than training of the model.
\n\nClearly the hosts/creators of this competition don't care enough about the competition to participate in the discussion forum or give an evaluation script. So being at my wit's end I am asking the participants at the top of the leader-board to help out with suggestions that might explain our 0.0 score. Any help/suggestion would be greatly appreciated! Thanks in advance!\n@polosin @loushun @guiguzhixing @shraddhaamohan
\n\nPlease look at the previous discussion too (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915)
", "rawMarkdown": "Hi, \nI wrote a discussion post about a week ago (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915) which showed the results we got and still received a 0 score from the evaluation. This also seems to be a widespread problem as lot of people seem to be getting this \n\nWe have addressed all the suggestions made by @polosin but are still getting a zero score. Thanks Oleg for all the help!\n\nLet's assume that the Mask RCNN or Resnet based attribute model we have trained in not trained enough. Even then based on the results we are getting (see image in linked post) how can the auto-grader give a zero score. I mean shouldn't it at least be something like 0.001? \n\nWe are running our models for more epochs in hope of getting better results but it's odd that no even one image is correct. This leads me to think that issue might be with something in the submission format rather than training of the model.\n\nClearly the hosts/creators of this competition don't care enough about the competition to participate in the discussion forum or give an evaluation script. So being at my wit's end I am asking the participants at the top of the leader-board to help out with suggestions that might explain our 0.0 score. Any help/suggestion would be greatly appreciated! Thanks in advance!\n@polosin @loushun @guiguzhixing @shraddhaamohan \n\nPlease look at the previous discussion too (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915)", "votes": null }, { "id": "851782", "postDate": "05/17/2020 23:09:55", "content": "Did you compare your code for generating a submission file with this notebook by @tanreinama?
", "rawMarkdown": "Did you compare your code for generating a submission file with [this notebook](https://www.kaggle.com/tanreinama/lb0-1213-models-training-code) by @tanreinama?", "votes": null }, { "id": "851801", "postDate": "05/17/2020 23:50:27", "content": "Yeah, I even copied the _scale_image(img, long_size) function to resize the images as is from that notebook.\nCan't see something obviously different from our submission notebook, but maybe I should go over it line by line.\nI shared our submission creation notebook with you.
", "rawMarkdown": "Yeah, I even copied the _scale_image(img, long_size) function to resize the images as is from that notebook.\nCan't see something obviously different from our submission notebook, but maybe I should go over it line by line.\nI shared our submission creation notebook with you.", "votes": null }, { "id": "851825", "postDate": "05/18/2020 00:26:43", "content": "I think so.\nI have a question. @polosin @mihirmavalankar \nIs there any case where attribute id does not exist?
", "rawMarkdown": "I think so.\nI have a question. @polosin @mihirmavalankar \nIs there any case where attribute id does not exist?", "votes": null }, { "id": "851830", "postDate": "05/18/2020 00:35:53", "content": "Yeah, so if no attribute is above a threshold of prediction I just return a 'nan' string?\nIs that what you meant? @hongym7
", "rawMarkdown": "Yeah, so if no attribute is above a threshold of prediction I just return a 'nan' string?\nIs that what you meant? @hongym7", "votes": null }, { "id": "851845", "postDate": "05/18/2020 00:52:06", "content": "@mihirmavalankar 'nan' string?! You should just leave the field empty.
", "rawMarkdown": "mihirmavalankar 'nan' string?! You should just leave the field empty.", "votes": null }, { "id": "851849", "postDate": "05/18/2020 00:55:28", "content": "@hongym7 Just check the distribution of attributes per class in the training data. Some classes don't have attributes.
", "rawMarkdown": "hongym7 Just check the distribution of attributes per class in the training data. Some classes don't have attributes.", "votes": null }, { "id": "851851", "postDate": "05/18/2020 00:58:01", "content": "@polosin \nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]
\n\nHowever, do these classes have no attribute values in test data?\nUm... 😑
", "rawMarkdown": "polosin \nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]\n\nHowever, do these classes have no attribute values in test data?\nUm... 😑", "votes": null }, { "id": "851854", "postDate": "05/18/2020 01:03:42", "content": "This is not a good idea to share the notebook just with me. It can be considered as a private sharing. I would recommend you to create a public notebook, so everyone can see it.
", "rawMarkdown": "This is not a good idea to share the notebook just with me. It can be considered as a private sharing. I would recommend you to create a public notebook, so everyone can see it.", "votes": null }, { "id": "851887", "postDate": "05/18/2020 02:12:15", "content": "I also got a zero score once as well because I forget to resize the longest side into 1024. Here are my suggestion;\n1. check the resize params according to evaluation page\n2.check the class from 0-45 as mentioned by @polosin \n3.visual the results to check whether the submitted results match the test images\nGood Luck!
", "rawMarkdown": "I also got a zero score once as well because I forget to resize the longest side into 1024. Here are my suggestion;\n1. check the resize params according to evaluation page\n2.check the class from 0-45 as mentioned by @polosin \n3.visual the results to check whether the submitted results match the test images\nGood Luck!", "votes": null }, { "id": "851964", "postDate": "05/18/2020 04:29:41", "content": "Sorry about that didn't realize it! Not sharing anymore
", "rawMarkdown": "Sorry about that didn't realize it! Not sharing anymore", "votes": null }, { "id": "851974", "postDate": "05/18/2020 04:53:42", "content": "@polosin is right. If there is no attributes for a certain mask, you should leave it empty, instead of \"nan\" string.
", "rawMarkdown": "polosin is right. If there is no attributes for a certain mask, you should leave it empty, instead of \"nan\" string.", "votes": null }, { "id": "851976", "postDate": "05/18/2020 04:55:58", "content": "\n\nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]
\n\nHowever, do these classes have no attribute values in test data?
", "rawMarkdown": "makeitworkjml \n\nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]\n\nHowever, do these classes have no attribute values in test data?", "votes": null }, { "id": "851982", "postDate": "05/18/2020 05:01:43", "content": "To understand this task better, a true positive should satisfy the following three condition: 1). the apparel category is correct, 2). attributes F1 score, 3). the masks are overlapped with the groundtruth masks. So it's possible that with longer epoch, the model might learn 1), or 2), but not all of them.
\n\nBut I agree with all the discussion, you need to post-process the submission file, i.e. image sizes and 0-indexed class ids.
", "rawMarkdown": "To understand this task better, a true positive should satisfy the following three condition: 1). the apparel category is correct, 2). attributes F1 score, 3). the masks are overlapped with the groundtruth masks. So it's possible that with longer epoch, the model might learn 1), or 2), but not all of them.\n\nBut I agree with all the discussion, you need to post-process the submission file, i.e. image sizes and 0-indexed class ids.", "votes": null }, { "id": "852044", "postDate": "05/18/2020 06:14:54", "content": "@polosin @makeitworkjml \n I tried putting empty quotes, null value and np.nan instead of writing 'nan' in the Attribute id column but it's still a zero score. I even checked that the csv had blanks in there as you told.\nThanks for pointing this out though Oleg!
\n\nBut that means that there is still something not right with the submission.
", "rawMarkdown": "polosin @makeitworkjml \n I tried putting empty quotes, null value and np.nan instead of writing 'nan' in the Attribute id column but it's still a zero score. I even checked that the csv had blanks in there as you told.\nThanks for pointing this out though Oleg!\n\nBut that means that there is still something not right with the submission.", "votes": null }, { "id": "852063", "postDate": "05/18/2020 06:44:39", "content": "@polosin @loushun @guiguzhixing @shraddhaamohan\nMade the submission notebook public\nhttps://www.kaggle.com/mihirmavalankar/imet-createop-csv-sep-segments-public
\n\nIn case you can spot some obvious mistake, thanks again!
", "rawMarkdown": "polosin @loushun @guiguzhixing @shraddhaamohan\nMade the submission notebook public\nhttps://www.kaggle.com/mihirmavalankar/imet-createop-csv-sep-segments-public\n\nIn case you can spot some obvious mistake, thanks again!", "votes": null }, { "id": "852977", "postDate": "05/18/2020 21:14:06", "content": "Hi @makeitworkjml thanks for clearing up how a TP is counted. \nIt's puzzling to me that not a single one of the 3200 test images was correct after looking at the mask visualizations we got (see linked post) which seem quite decent. I mean the masks+classes+attributes (majority of which are Nan) didn't produce a single TP?
\n\nThis makes me think that something is wrong with our submission format. Would that be a correct assumption or do you think the zero score is still explained by a poorly trained model?
\n\nIf our submission format is wrong what might help if seeing an evaluation script or getting more details on the evaluation code I believe. There was also another post in the discussion forum that requested this. Would this be possible?
", "rawMarkdown": "Hi @makeitworkjml thanks for clearing up how a TP is counted. \nIt's puzzling to me that not a single one of the 3200 test images was correct after looking at the mask visualizations we got (see linked post) which seem quite decent. I mean the masks+classes+attributes (majority of which are Nan) didn't produce a single TP?\n\nThis makes me think that something is wrong with our submission format. Would that be a correct assumption or do you think the zero score is still explained by a poorly trained model? \n\nIf our submission format is wrong what might help if seeing an evaluation script or getting more details on the evaluation code I believe. There was also another post in the discussion forum that requested this. Would this be possible?", "votes": null } ], "comments": [ { "id": 851782, "author_name": "polosin", "author_url": "", "post_date": "05/17/2020 23:09:55", "content": "Did you compare your code for generating a submission file with this notebook by @tanreinama?
", "votes": null, "replies": [ { "id": 851801, "author_name": "mihirmavalankar", "author_url": "", "post_date": "05/17/2020 23:50:27", "content": "Yeah, I even copied the _scale_image(img, long_size) function to resize the images as is from that notebook.\nCan't see something obviously different from our submission notebook, but maybe I should go over it line by line.\nI shared our submission creation notebook with you.
", "votes": null, "replies": [] }, { "id": 851854, "author_name": "polosin", "author_url": "", "post_date": "05/18/2020 01:03:42", "content": "This is not a good idea to share the notebook just with me. It can be considered as a private sharing. I would recommend you to create a public notebook, so everyone can see it.
", "votes": null, "replies": [] }, { "id": 851964, "author_name": "mihirmavalankar", "author_url": "", "post_date": "05/18/2020 04:29:41", "content": "Sorry about that didn't realize it! Not sharing anymore
", "votes": null, "replies": [] }, { "id": 852063, "author_name": "mihirmavalankar", "author_url": "", "post_date": "05/18/2020 06:44:39", "content": "@polosin @loushun @guiguzhixing @shraddhaamohan\nMade the submission notebook public\nhttps://www.kaggle.com/mihirmavalankar/imet-createop-csv-sep-segments-public
\n\nIn case you can spot some obvious mistake, thanks again!
", "votes": null, "replies": [] } ] }, { "id": 851825, "author_name": "hongym7", "author_url": "", "post_date": "05/18/2020 00:26:43", "content": "I think so.\nI have a question. @polosin @mihirmavalankar \nIs there any case where attribute id does not exist?
", "votes": null, "replies": [ { "id": 851830, "author_name": "mihirmavalankar", "author_url": "", "post_date": "05/18/2020 00:35:53", "content": "Yeah, so if no attribute is above a threshold of prediction I just return a 'nan' string?\nIs that what you meant? @hongym7
", "votes": null, "replies": [] }, { "id": 851845, "author_name": "polosin", "author_url": "", "post_date": "05/18/2020 00:52:06", "content": "@mihirmavalankar 'nan' string?! You should just leave the field empty.
", "votes": null, "replies": [] }, { "id": 851849, "author_name": "polosin", "author_url": "", "post_date": "05/18/2020 00:55:28", "content": "@hongym7 Just check the distribution of attributes per class in the training data. Some classes don't have attributes.
", "votes": null, "replies": [] }, { "id": 851851, "author_name": "hongym7", "author_url": "", "post_date": "05/18/2020 00:58:01", "content": "@polosin \nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]
\n\nHowever, do these classes have no attribute values in test data?\nUm... 😑
", "votes": null, "replies": [] }, { "id": 851974, "author_name": "makeitworkjml", "author_url": "", "post_date": "05/18/2020 04:53:42", "content": "@polosin is right. If there is no attributes for a certain mask, you should leave it empty, instead of \"nan\" string.
", "votes": null, "replies": [] }, { "id": 851976, "author_name": "hongym7", "author_url": "", "post_date": "05/18/2020 04:55:58", "content": "\n\nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]
\n\nHowever, do these classes have no attribute values in test data?
", "votes": null, "replies": [] }, { "id": 852044, "author_name": "mihirmavalankar", "author_url": "", "post_date": "05/18/2020 06:14:54", "content": "@polosin @makeitworkjml \n I tried putting empty quotes, null value and np.nan instead of writing 'nan' in the Attribute id column but it's still a zero score. I even checked that the csv had blanks in there as you told.\nThanks for pointing this out though Oleg!
\n\nBut that means that there is still something not right with the submission.
", "votes": null, "replies": [] } ] }, { "id": 851887, "author_name": "guiguzhixing", "author_url": "", "post_date": "05/18/2020 02:12:15", "content": "I also got a zero score once as well because I forget to resize the longest side into 1024. Here are my suggestion;\n1. check the resize params according to evaluation page\n2.check the class from 0-45 as mentioned by @polosin \n3.visual the results to check whether the submitted results match the test images\nGood Luck!
", "votes": null, "replies": [] }, { "id": 851982, "author_name": "makeitworkjml", "author_url": "", "post_date": "05/18/2020 05:01:43", "content": "To understand this task better, a true positive should satisfy the following three condition: 1). the apparel category is correct, 2). attributes F1 score, 3). the masks are overlapped with the groundtruth masks. So it's possible that with longer epoch, the model might learn 1), or 2), but not all of them.
\n\nBut I agree with all the discussion, you need to post-process the submission file, i.e. image sizes and 0-indexed class ids.
", "votes": null, "replies": [] }, { "id": 852977, "author_name": "mihirmavalankar", "author_url": "", "post_date": "05/18/2020 21:14:06", "content": "Hi @makeitworkjml thanks for clearing up how a TP is counted. \nIt's puzzling to me that not a single one of the 3200 test images was correct after looking at the mask visualizations we got (see linked post) which seem quite decent. I mean the masks+classes+attributes (majority of which are Nan) didn't produce a single TP?
\n\nThis makes me think that something is wrong with our submission format. Would that be a correct assumption or do you think the zero score is still explained by a poorly trained model?
\n\nIf our submission format is wrong what might help if seeing an evaluation script or getting more details on the evaluation code I believe. There was also another post in the discussion forum that requested this. Would this be possible?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "851758": "Hi, \nI wrote a discussion post about a week ago (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915) which showed the results we got and still received a 0 score from the evaluation. This also seems to be a widespread problem as lot of people seem to be getting this \n\nWe have addressed all the suggestions made by @polosin but are still getting a zero score. Thanks Oleg for all the help!\n\nLet's assume that the Mask RCNN or Resnet based attribute model we have trained in not trained enough. Even then based on the results we are getting (see image in linked post) how can the auto-grader give a zero score. I mean shouldn't it at least be something like 0.001? \n\nWe are running our models for more epochs in hope of getting better results but it's odd that no even one image is correct. This leads me to think that issue might be with something in the submission format rather than training of the model.\n\nClearly the hosts/creators of this competition don't care enough about the competition to participate in the discussion forum or give an evaluation script. So being at my wit's end I am asking the participants at the top of the leader-board to help out with suggestions that might explain our 0.0 score. Any help/suggestion would be greatly appreciated! Thanks in advance!\n@polosin @loushun @guiguzhixing @shraddhaamohan \n\nPlease look at the previous discussion too (https://www.kaggle.com/c/imaterialist-fashion-2020-fgvc7/discussion/150915)", "851782": "Did you compare your code for generating a submission file with [this notebook](https://www.kaggle.com/tanreinama/lb0-1213-models-training-code) by @tanreinama?", "851801": "Yeah, I even copied the _scale_image(img, long_size) function to resize the images as is from that notebook.\nCan't see something obviously different from our submission notebook, but maybe I should go over it line by line.\nI shared our submission creation notebook with you.", "851825": "I think so.\nI have a question. @polosin @mihirmavalankar \nIs there any case where attribute id does not exist?", "851830": "Yeah, so if no attribute is above a threshold of prediction I just return a 'nan' string?\nIs that what you meant? @hongym7", "851845": "mihirmavalankar 'nan' string?! You should just leave the field empty.", "851849": "hongym7 Just check the distribution of attributes per class in the training data. Some classes don't have attributes.", "851851": "polosin \nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]\n\nHowever, do these classes have no attribute values in test data?\nUm... 😑", "851854": "This is not a good idea to share the notebook just with me. It can be considered as a private sharing. I would recommend you to create a public notebook, so everyone can see it.", "851887": "I also got a zero score once as well because I forget to resize the longest side into 1024. Here are my suggestion;\n1. check the resize params according to evaluation page\n2.check the class from 0-45 as mentioned by @polosin \n3.visual the results to check whether the submitted results match the test images\nGood Luck!", "851964": "Sorry about that didn't realize it! Not sharing anymore", "851974": "polosin is right. If there is no attributes for a certain mask, you should leave it empty, instead of \"nan\" string.", "851976": "makeitworkjml \n\nI checked training data.\nThe class below has no attribute value in training data.\n[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 30, 34, 38, 41, 42, 44, 45]\n\nHowever, do these classes have no attribute values in test data?", "851982": "To understand this task better, a true positive should satisfy the following three condition: 1). the apparel category is correct, 2). attributes F1 score, 3). the masks are overlapped with the groundtruth masks. So it's possible that with longer epoch, the model might learn 1), or 2), but not all of them.\n\nBut I agree with all the discussion, you need to post-process the submission file, i.e. image sizes and 0-indexed class ids.", "852044": "polosin @makeitworkjml \n I tried putting empty quotes, null value and np.nan instead of writing 'nan' in the Attribute id column but it's still a zero score. I even checked that the csv had blanks in there as you told.\nThanks for pointing this out though Oleg!\n\nBut that means that there is still something not right with the submission.", "852063": "polosin @loushun @guiguzhixing @shraddhaamohan\nMade the submission notebook public\nhttps://www.kaggle.com/mihirmavalankar/imet-createop-csv-sep-segments-public\n\nIn case you can spot some obvious mistake, thanks again!", "852977": "Hi @makeitworkjml thanks for clearing up how a TP is counted. \nIt's puzzling to me that not a single one of the 3200 test images was correct after looking at the mask visualizations we got (see linked post) which seem quite decent. I mean the masks+classes+attributes (majority of which are Nan) didn't produce a single TP?\n\nThis makes me think that something is wrong with our submission format. Would that be a correct assumption or do you think the zero score is still explained by a poorly trained model? \n\nIf our submission format is wrong what might help if seeing an evaluation script or getting more details on the evaluation code I believe. There was also another post in the discussion forum that requested this. Would this be possible?" }, "source": "meta" }