I have a solution that generates very good instance labeling. I have manually verified and my detection and segmentation are very good.\nHowever, when I submit the results, my scores come as low as 0.02 which is not believable to me. \nIt seems that I have some misunderstanding about generating the RLE encoding and the issue is there is no clear explanation about it in the website. There are contradictory information in these forums which add to the confusion. \nI think the easiest approach would be to provide an example. An image with two instances and the corresponding acceptable encoding. The \"sample_submission.csv\" contains an example encoding, but it lacks the images the examples were generated from.
\n\nI have spent a lot of time for this challenge, and I think it is not fair that we can not be evaluated to our merits, only because the rules are not well explained.
", "messages": [ { "id": "340893", "postDate": "06/10/2018 15:13:46", "content": "I have a solution that generates very good instance labeling. I have manually verified and my detection and segmentation are very good.\nHowever, when I submit the results, my scores come as low as 0.02 which is not believable to me. \nIt seems that I have some misunderstanding about generating the RLE encoding and the issue is there is no clear explanation about it in the website. There are contradictory information in these forums which add to the confusion. \nI think the easiest approach would be to provide an example. An image with two instances and the corresponding acceptable encoding. The \"sample_submission.csv\" contains an example encoding, but it lacks the images the examples were generated from.
\n\nI have spent a lot of time for this challenge, and I think it is not fair that we can not be evaluated to our merits, only because the rules are not well explained.
", "rawMarkdown": "I have a solution that generates very good instance labeling. I have manually verified and my detection and segmentation are very good.\nHowever, when I submit the results, my scores come as low as 0.02 which is not believable to me. \nIt seems that I have some misunderstanding about generating the RLE encoding and the issue is there is no clear explanation about it in the website. There are contradictory information in these forums which add to the confusion. \nI think the easiest approach would be to provide an example. An image with two instances and the corresponding acceptable encoding. The \"sample_submission.csv\" contains an example encoding, but it lacks the images the examples were generated from.\n\nI have spent a lot of time for this challenge, and I think it is not fair that we can not be evaluated to our merits, only because the rules are not well explained.", "votes": null }, { "id": "340909", "postDate": "06/10/2018 15:54:21", "content": "Are u using provided ConvertVideotoCsv.py? I got very low scores initially but soon I found I was using wrong order for row and column. I am using a modified version of ConvertVideotoCsv.py (I added multi-process to make it faster) and can encode my submission without any issue.\nHave you manually checked you submitted file? Do they look similar to sample submission?
", "rawMarkdown": "Are u using provided ConvertVideotoCsv.py? I got very low scores initially but soon I found I was using wrong order for row and column. I am using a modified version of ConvertVideotoCsv.py (I added multi-process to make it faster) and can encode my submission without any issue.\nHave you manually checked you submitted file? Do they look similar to sample submission?", "votes": null }, { "id": "340935", "postDate": "06/10/2018 17:07:48", "content": "I had written my own script before finding \"ConvertVideotoCsv.py\". But I later checked and it seemed in agreement with mine. I have checked manually at least 100 files and they look OK. ANd my submission does look like the sample submission so I am really confused.
\n\nHere is a sample from my submission. Could you please tell me if anything seems wrong? I would highly appreciate it.\n(added the dashes to make it more readable).
\n\nImageId---------------------------------------LabelId ----Confidence--PixelCount---EncodedPixels\n001bbdb5c4f43bd4dc2b3e3a08b7202a--33000--- 1 ------------- 18192 --- 5888380 36|5891764 36|......|6274100 152|6277484 152|
", "rawMarkdown": "I had written my own script before finding \"ConvertVideotoCsv.py\". But I later checked and it seemed in agreement with mine. I have checked manually at least 100 files and they look OK. ANd my submission does look like the sample submission so I am really confused.\n\n\n Here is a sample from my submission. Could you please tell me if anything seems wrong? I would highly appreciate it.\n(added the dashes to make it more readable).\n\n ImageId---------------------------------------LabelId ----Confidence--PixelCount---EncodedPixels\n001bbdb5c4f43bd4dc2b3e3a08b7202a--33000--- 1\t ------------- 18192 --- 5888380 36|5891764 36|......|6274100 152|6277484 152|", "votes": null }, { "id": "340940", "postDate": "06/10/2018 17:11:47", "content": "\n\n\nLabelId is the class of that object (car, person, etc),
\n
So you labelid should be 33, not 33000
", "rawMarkdown": "> LabelId is the class of that object (car, person, etc),\n\nSo you labelid should be 33, not 33000", "votes": null }, { "id": "340942", "postDate": "06/10/2018 17:19:19", "content": "I was initially doing that, but later found in \"Case10Solution.csv\" in \"EvaluationScriptsAndExamples.zip\" that when there are multiple instances of the same object, they were labeled like 33000,330001,33002...
\n\nSo do you mean if I have lets say 5 cars in an image, I label them all as 33? or 33000,33001,...?
", "rawMarkdown": "I was initially doing that, but later found in \"Case10Solution.csv\" in \"EvaluationScriptsAndExamples.zip\" that when there are multiple instances of the same object, they were labeled like 33000,330001,33002... \n\nSo do you mean if I have lets say 5 cars in an image, I label them all as 33? or 33000,33001,...?", "votes": null }, { "id": "340948", "postDate": "06/10/2018 17:29:29", "content": "You are looking at \"Case10Solution.csv\". The sample submission is \"Case10Submission_0.51484858989715575.csv\". \nYes, you should label them all as 33, since they are in same class. But you should list them as individual instance in rle, i.e. put them in different lines.
", "rawMarkdown": "You are looking at \"Case10Solution.csv\". The sample submission is \"Case10Submission_0.51484858989715575.csv\". \nYes, you should label them all as 33, since they are in same class. But you should list them as individual instance in rle, i.e. put them in different lines.", "votes": null }, { "id": "340955", "postDate": "06/10/2018 17:39:35", "content": "I have already tried like that too anyways. There must be something else I am missing.\nThanks so much for your help.
", "rawMarkdown": "I have already tried like that too anyways. There must be something else I am missing.\nThanks so much for your help.", "votes": null }, { "id": "341022", "postDate": "06/10/2018 20:55:37", "content": "Does the order in which instances are listed matter? should they appear in the list sorted from the top to bottom of image?
", "rawMarkdown": "Does the order in which instances are listed matter? should they appear in the list sorted from the top to bottom of image?", "votes": null }, { "id": "341027", "postDate": "06/10/2018 21:03:55", "content": "That should not matter. What kind of model are you using? How much are you training it? I had the wrong approach in the beginning (not using Mask RCNN), and I could not get a better score than 0.05. So it may very well be that your approach is very limited.
", "rawMarkdown": "That should not matter. What kind of model are you using? How much are you training it? I had the wrong approach in the beginning (not using Mask RCNN), and I could not get a better score than 0.05. So it may very well be that your approach is very limited.", "votes": null }, { "id": "341030", "postDate": "06/10/2018 21:11:31", "content": "But I am doing manual checking. Today I looked at almost 100 test cases and I am observing very good match (7 out of 10 instances show a very good match on the image). To make sure, I am decoding my RLE encoding and overlay the area on the actual test image and I see very good match both for large and small objects.\nThat's why I am confused and thinking maybe my encoding is not matching well with the competition format.
\n\nI am using FCN with heuristic post-filtering. It is trained for two days and seems to be detecting well.
", "rawMarkdown": "But I am doing manual checking. Today I looked at almost 100 test cases and I am observing very good match (7 out of 10 instances show a very good match on the image). To make sure, I am decoding my RLE encoding and overlay the area on the actual test image and I see very good match both for large and small objects.\nThat's why I am confused and thinking maybe my encoding is not matching well with the competition format.\n\nI am using FCN with heuristic post-filtering. It is trained for two days and seems to be detecting well.", "votes": null }, { "id": "341045", "postDate": "06/10/2018 21:59:29", "content": "Manual checking is not very helpful in general. Unless you have a very obvious error such as bad training data. You should try to use a more systematic way of evaluation, such as using a local validation set and calculate the mAP of your predictions.
\n\nTwo days is not very long, especially if you are using a small machine with few GPUs. What is your backbone? If it is anything smaller/less sophisticated than a ResNext101 it will be hard to score well
", "rawMarkdown": "Manual checking is not very helpful in general. Unless you have a very obvious error such as bad training data. You should try to use a more systematic way of evaluation, such as using a local validation set and calculate the mAP of your predictions.\n\nTwo days is not very long, especially if you are using a small machine with few GPUs. What is your backbone? If it is anything smaller/less sophisticated than a ResNext101 it will be hard to score well", "votes": null }, { "id": "341050", "postDate": "06/10/2018 22:21:14", "content": "I am confident about basics of my approach. I can tell when a network is trained well using available performance measures. \nWhat I am not sure about is the evaluation criteria for this challenge.
", "rawMarkdown": "I am confident about basics of my approach. I can tell when a network is trained well using available performance measures. \nWhat I am not sure about is the evaluation criteria for this challenge.", "votes": null }, { "id": "341058", "postDate": "06/10/2018 22:36:24", "content": "Well as they explain somewhere in this forum, they only evaluate on 10 images for the public leaderboard due to resource constraints, which is not that much if you want to have a good signal. Even if you choose them well, so that might be an issue too, but I would doubt that.
", "rawMarkdown": "Well as they explain somewhere in this forum, they only evaluate on 10 images for the public leaderboard due to resource constraints, which is not that much if you want to have a good signal. Even if you choose them well, so that might be an issue too, but I would doubt that.", "votes": null }, { "id": "341356", "postDate": "06/11/2018 13:37:47", "content": "Would it be possible for people that achieve over 0.15 to share encodings for one image (with not so many objects)? I think this would be the best way to see if the encodings we are getting is the correct one.
", "rawMarkdown": "Would it be possible for people that achieve over 0.15 to share encodings for one image (with not so many objects)? I think this would be the best way to see if the encodings we are getting is the correct one.", "votes": null }, { "id": "341376", "postDate": "06/11/2018 13:58:01", "content": "def rle_encoding(x):\n \"\"\" Run-length encoding based on\n https://www.kaggle.com/rakhlin/fast-run-length-encoding-python\n Modified by Konstantin, https://www.kaggle.com/lopuhin\n x: binary mask,[H, W]\n \"\"\"\n assert x.dtype == np.bool\n # dots = np.where(x.T.flatten() == 1)[0]\n # print('Transpose')\n dots = np.where(x.flatten() == 1)[0]\n run_lengths = []\n prev = -2\n for b in dots:\n if b > prev + 1:\n run_lengths.append([b, 0])\n run_lengths[-1][1] += 1\n prev = b\n return '|'.join('{} {}'.format(*pair) for pair in run_lengths) + '|'\nYes, this is what I used. However, I used Facebook detectron, which does not output binary masks but rather a compressed RLE format. So, I first convert output into binary masks and then use this code to output rle into submission format. Thus, if I would have from your submission.csv the rows of only one image from the test set, I could say with more certainty if my format is really correct. And I think one image would not make any difference in the score.
", "rawMarkdown": "Yes, this is what I used. However, I used Facebook detectron, which does not output binary masks but rather a compressed RLE format. So, I first convert output into binary masks and then use this code to output rle into submission format. Thus, if I would have from your submission.csv the rows of only one image from the test set, I could say with more certainty if my format is really correct. And I think one image would not make any difference in the score.", "votes": null } ], "comments": [ { "id": 340909, "author_name": "woodywang", "author_url": "", "post_date": "06/10/2018 15:54:21", "content": "Are u using provided ConvertVideotoCsv.py? I got very low scores initially but soon I found I was using wrong order for row and column. I am using a modified version of ConvertVideotoCsv.py (I added multi-process to make it faster) and can encode my submission without any issue.\nHave you manually checked you submitted file? Do they look similar to sample submission?
", "votes": null, "replies": [] }, { "id": 340935, "author_name": "vahidkh62", "author_url": "", "post_date": "06/10/2018 17:07:48", "content": "I had written my own script before finding \"ConvertVideotoCsv.py\". But I later checked and it seemed in agreement with mine. I have checked manually at least 100 files and they look OK. ANd my submission does look like the sample submission so I am really confused.
\n\nHere is a sample from my submission. Could you please tell me if anything seems wrong? I would highly appreciate it.\n(added the dashes to make it more readable).
\n\nImageId---------------------------------------LabelId ----Confidence--PixelCount---EncodedPixels\n001bbdb5c4f43bd4dc2b3e3a08b7202a--33000--- 1 ------------- 18192 --- 5888380 36|5891764 36|......|6274100 152|6277484 152|
", "votes": null, "replies": [ { "id": 340940, "author_name": "woodywang", "author_url": "", "post_date": "06/10/2018 17:11:47", "content": "\n\n\nLabelId is the class of that object (car, person, etc),
\n
So you labelid should be 33, not 33000
", "votes": null, "replies": [] }, { "id": 340942, "author_name": "vahidkh62", "author_url": "", "post_date": "06/10/2018 17:19:19", "content": "I was initially doing that, but later found in \"Case10Solution.csv\" in \"EvaluationScriptsAndExamples.zip\" that when there are multiple instances of the same object, they were labeled like 33000,330001,33002...
\n\nSo do you mean if I have lets say 5 cars in an image, I label them all as 33? or 33000,33001,...?
", "votes": null, "replies": [] }, { "id": 340948, "author_name": "woodywang", "author_url": "", "post_date": "06/10/2018 17:29:29", "content": "You are looking at \"Case10Solution.csv\". The sample submission is \"Case10Submission_0.51484858989715575.csv\". \nYes, you should label them all as 33, since they are in same class. But you should list them as individual instance in rle, i.e. put them in different lines.
", "votes": null, "replies": [] }, { "id": 340955, "author_name": "vahidkh62", "author_url": "", "post_date": "06/10/2018 17:39:35", "content": "I have already tried like that too anyways. There must be something else I am missing.\nThanks so much for your help.
", "votes": null, "replies": [] }, { "id": 341022, "author_name": "vahidkh62", "author_url": "", "post_date": "06/10/2018 20:55:37", "content": "Does the order in which instances are listed matter? should they appear in the list sorted from the top to bottom of image?
", "votes": null, "replies": [] }, { "id": 341027, "author_name": "michaelheinzer", "author_url": "", "post_date": "06/10/2018 21:03:55", "content": "That should not matter. What kind of model are you using? How much are you training it? I had the wrong approach in the beginning (not using Mask RCNN), and I could not get a better score than 0.05. So it may very well be that your approach is very limited.
", "votes": null, "replies": [] }, { "id": 341030, "author_name": "vahidkh62", "author_url": "", "post_date": "06/10/2018 21:11:31", "content": "But I am doing manual checking. Today I looked at almost 100 test cases and I am observing very good match (7 out of 10 instances show a very good match on the image). To make sure, I am decoding my RLE encoding and overlay the area on the actual test image and I see very good match both for large and small objects.\nThat's why I am confused and thinking maybe my encoding is not matching well with the competition format.
\n\nI am using FCN with heuristic post-filtering. It is trained for two days and seems to be detecting well.
", "votes": null, "replies": [] }, { "id": 341045, "author_name": "michaelheinzer", "author_url": "", "post_date": "06/10/2018 21:59:29", "content": "Manual checking is not very helpful in general. Unless you have a very obvious error such as bad training data. You should try to use a more systematic way of evaluation, such as using a local validation set and calculate the mAP of your predictions.
\n\nTwo days is not very long, especially if you are using a small machine with few GPUs. What is your backbone? If it is anything smaller/less sophisticated than a ResNext101 it will be hard to score well
", "votes": null, "replies": [] }, { "id": 341050, "author_name": "vahidkh62", "author_url": "", "post_date": "06/10/2018 22:21:14", "content": "I am confident about basics of my approach. I can tell when a network is trained well using available performance measures. \nWhat I am not sure about is the evaluation criteria for this challenge.
", "votes": null, "replies": [] }, { "id": 341058, "author_name": "michaelheinzer", "author_url": "", "post_date": "06/10/2018 22:36:24", "content": "Well as they explain somewhere in this forum, they only evaluate on 10 images for the public leaderboard due to resource constraints, which is not that much if you want to have a good signal. Even if you choose them well, so that might be an issue too, but I would doubt that.
", "votes": null, "replies": [] } ] }, { "id": 341356, "author_name": "natasa1", "author_url": "", "post_date": "06/11/2018 13:37:47", "content": "Would it be possible for people that achieve over 0.15 to share encodings for one image (with not so many objects)? I think this would be the best way to see if the encodings we are getting is the correct one.
", "votes": null, "replies": [ { "id": 341376, "author_name": "mhttxkaggle", "author_url": "", "post_date": "06/11/2018 13:58:01", "content": "def rle_encoding(x):\n \"\"\" Run-length encoding based on\n https://www.kaggle.com/rakhlin/fast-run-length-encoding-python\n Modified by Konstantin, https://www.kaggle.com/lopuhin\n x: binary mask,[H, W]\n \"\"\"\n assert x.dtype == np.bool\n # dots = np.where(x.T.flatten() == 1)[0]\n # print('Transpose')\n dots = np.where(x.flatten() == 1)[0]\n run_lengths = []\n prev = -2\n for b in dots:\n if b > prev + 1:\n run_lengths.append([b, 0])\n run_lengths[-1][1] += 1\n prev = b\n return '|'.join('{} {}'.format(*pair) for pair in run_lengths) + '|'\nYes, this is what I used. However, I used Facebook detectron, which does not output binary masks but rather a compressed RLE format. So, I first convert output into binary masks and then use this code to output rle into submission format. Thus, if I would have from your submission.csv the rows of only one image from the test set, I could say with more certainty if my format is really correct. And I think one image would not make any difference in the score.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "340893": "I have a solution that generates very good instance labeling. I have manually verified and my detection and segmentation are very good.\nHowever, when I submit the results, my scores come as low as 0.02 which is not believable to me. \nIt seems that I have some misunderstanding about generating the RLE encoding and the issue is there is no clear explanation about it in the website. There are contradictory information in these forums which add to the confusion. \nI think the easiest approach would be to provide an example. An image with two instances and the corresponding acceptable encoding. The \"sample_submission.csv\" contains an example encoding, but it lacks the images the examples were generated from.\n\nI have spent a lot of time for this challenge, and I think it is not fair that we can not be evaluated to our merits, only because the rules are not well explained.", "340909": "Are u using provided ConvertVideotoCsv.py? I got very low scores initially but soon I found I was using wrong order for row and column. I am using a modified version of ConvertVideotoCsv.py (I added multi-process to make it faster) and can encode my submission without any issue.\nHave you manually checked you submitted file? Do they look similar to sample submission?", "340935": "I had written my own script before finding \"ConvertVideotoCsv.py\". But I later checked and it seemed in agreement with mine. I have checked manually at least 100 files and they look OK. ANd my submission does look like the sample submission so I am really confused.\n\n\n Here is a sample from my submission. Could you please tell me if anything seems wrong? I would highly appreciate it.\n(added the dashes to make it more readable).\n\n ImageId---------------------------------------LabelId ----Confidence--PixelCount---EncodedPixels\n001bbdb5c4f43bd4dc2b3e3a08b7202a--33000--- 1\t ------------- 18192 --- 5888380 36|5891764 36|......|6274100 152|6277484 152|", "340940": "> LabelId is the class of that object (car, person, etc),\n\nSo you labelid should be 33, not 33000", "340942": "I was initially doing that, but later found in \"Case10Solution.csv\" in \"EvaluationScriptsAndExamples.zip\" that when there are multiple instances of the same object, they were labeled like 33000,330001,33002... \n\nSo do you mean if I have lets say 5 cars in an image, I label them all as 33? or 33000,33001,...?", "340948": "You are looking at \"Case10Solution.csv\". The sample submission is \"Case10Submission_0.51484858989715575.csv\". \nYes, you should label them all as 33, since they are in same class. But you should list them as individual instance in rle, i.e. put them in different lines.", "340955": "I have already tried like that too anyways. There must be something else I am missing.\nThanks so much for your help.", "341022": "Does the order in which instances are listed matter? should they appear in the list sorted from the top to bottom of image?", "341027": "That should not matter. What kind of model are you using? How much are you training it? I had the wrong approach in the beginning (not using Mask RCNN), and I could not get a better score than 0.05. So it may very well be that your approach is very limited.", "341030": "But I am doing manual checking. Today I looked at almost 100 test cases and I am observing very good match (7 out of 10 instances show a very good match on the image). To make sure, I am decoding my RLE encoding and overlay the area on the actual test image and I see very good match both for large and small objects.\nThat's why I am confused and thinking maybe my encoding is not matching well with the competition format.\n\nI am using FCN with heuristic post-filtering. It is trained for two days and seems to be detecting well.", "341045": "Manual checking is not very helpful in general. Unless you have a very obvious error such as bad training data. You should try to use a more systematic way of evaluation, such as using a local validation set and calculate the mAP of your predictions.\n\nTwo days is not very long, especially if you are using a small machine with few GPUs. What is your backbone? If it is anything smaller/less sophisticated than a ResNext101 it will be hard to score well", "341050": "I am confident about basics of my approach. I can tell when a network is trained well using available performance measures. \nWhat I am not sure about is the evaluation criteria for this challenge.", "341058": "Well as they explain somewhere in this forum, they only evaluate on 10 images for the public leaderboard due to resource constraints, which is not that much if you want to have a good signal. Even if you choose them well, so that might be an issue too, but I would doubt that.", "341356": "Would it be possible for people that achieve over 0.15 to share encodings for one image (with not so many objects)? I think this would be the best way to see if the encodings we are getting is the correct one.", "341376": "def rle_encoding(x):\n \"\"\" Run-length encoding based on\n https://www.kaggle.com/rakhlin/fast-run-length-encoding-python\n Modified by Konstantin, https://www.kaggle.com/lopuhin\n x: binary mask,[H, W]\n \"\"\"\n assert x.dtype == np.bool\n # dots = np.where(x.T.flatten() == 1)[0]\n # print('Transpose')\n dots = np.where(x.flatten() == 1)[0]\n run_lengths = []\n prev = -2\n for b in dots:\n if b > prev + 1:\n run_lengths.append([b, 0])\n run_lengths[-1][1] += 1\n prev = b\n return '|'.join('{} {}'.format(*pair) for pair in run_lengths) + '|'", "341398": "Yes, this is what I used. However, I used Facebook detectron, which does not output binary masks but rather a compressed RLE format. So, I first convert output into binary masks and then use this code to output rle into submission format. Thus, if I would have from your submission.csv the rows of only one image from the test set, I could say with more certainty if my format is really correct. And I think one image would not make any difference in the score." }, "source": "meta" }