I am reading all the topics with results from other teams and I am a very excited but not surprised about how important it was to have many GPU cards. Since I was dreaming about having my own GPU during the competition, I'd like to tell you our team's story and show what our final solution consists of.
\n\nI've started to participate in the competition because I had no experience with fully convolutional neural networks, so it seemed to be a great chance to learn something new. Moreover this is my first DL competition on Kaggle and I am really proud to be in top-100 on a public leaderboard.
\n\nWe didn't have our own GPU at all. All I was able to do is to run training overnight on my server with a single GTX 1080 at work. Night is the only time when no one else is training their own models there. Let's say we started to contribute about 3 weeks ago, that means we had 20 nights including weekends :D
\n\nAnyway, this our solution is:
\n\nAs a final solution we use an ensemble of 3 models:
\n\na) uNet 1024 trained on low quality images using bce_dice_loss
\n\nb) uNet 1024 trained on low quality images using weighted_bce_dice_loss
\n\nc) uNet 1024 trained on high quality images using bce_dice_loss
That is pretty it ;)
\n\nBy the way, even uNet 256 with the same configuration gives about 0.9963
\n\nI am very glad to have our model that small and simple, so it could be trained on a single GPU without any problems. It makes me a bit happy when I read about all these 40-GPUs solutions :)\nAnd I'd like to thank everyone who contributed during the competition, it helped me to eventually learn FCNs.
\n\nCheers!
", "messages": [ { "id": "224956", "postDate": "09/28/2017 00:55:48", "content": "I am reading all the topics with results from other teams and I am a very excited but not surprised about how important it was to have many GPU cards. Since I was dreaming about having my own GPU during the competition, I'd like to tell you our team's story and show what our final solution consists of.
\n\nI've started to participate in the competition because I had no experience with fully convolutional neural networks, so it seemed to be a great chance to learn something new. Moreover this is my first DL competition on Kaggle and I am really proud to be in top-100 on a public leaderboard.
\n\nWe didn't have our own GPU at all. All I was able to do is to run training overnight on my server with a single GTX 1080 at work. Night is the only time when no one else is training their own models there. Let's say we started to contribute about 3 weeks ago, that means we had 20 nights including weekends :D
\n\nAnyway, this our solution is:
\n\nAs a final solution we use an ensemble of 3 models:
\n\na) uNet 1024 trained on low quality images using bce_dice_loss
\n\nb) uNet 1024 trained on low quality images using weighted_bce_dice_loss
\n\nc) uNet 1024 trained on high quality images using bce_dice_loss
That is pretty it ;)
\n\nBy the way, even uNet 256 with the same configuration gives about 0.9963
\n\nI am very glad to have our model that small and simple, so it could be trained on a single GPU without any problems. It makes me a bit happy when I read about all these 40-GPUs solutions :)\nAnd I'd like to thank everyone who contributed during the competition, it helped me to eventually learn FCNs.
\n\nCheers!
", "rawMarkdown": "I am reading all the topics with results from other teams and I am a very excited but not surprised about how important it was to have many GPU cards. Since I was dreaming about having my own GPU during the competition, I'd like to tell you our team's story and show what our final solution consists of.\n\nI've started to participate in the competition because I had no experience with fully convolutional neural networks, so it seemed to be a great chance to learn something new. Moreover this is my first DL competition on Kaggle and I am really proud to be in top-100 on a public leaderboard.\n\nWe didn't have our own GPU at all. All I was able to do is to run training overnight on my server with a single GTX 1080 at work. Night is the only time when no one else is training their own models there. Let's say we started to contribute about 3 weeks ago, that means we had 20 nights including weekends :D\n\nAnyway, this our solution is:\n\n 1. Train uNet 1024 on the raw input. This is used to predict bounding boxes.\n 2. Cut a car out of a predicted mask with a 40 pixels padding. It gives approximately 56% of a car on the picture. Then we use cutted images only.\n 3. As a final solution we use an ensemble of 3 models:\n\n a) uNet 1024 trained on low quality images using bce_dice_loss\n\n b) uNet 1024 trained on low quality images using weighted_bce_dice_loss\n\n c) uNet 1024 trained on high quality images using bce_dice_loss\n 4. Ensemble models with weights [0.3, 0.1, 0.6] accordingly.\n\nThat is pretty it ;)\n\nBy the way, even uNet 256 with the same configuration gives about 0.9963\n\nI am very glad to have our model that small and simple, so it could be trained on a single GPU without any problems. It makes me a bit happy when I read about all these 40-GPUs solutions :)\nAnd I'd like to thank everyone who contributed during the competition, it helped me to eventually learn FCNs.\n\nCheers!", "votes": null }, { "id": "224959", "postDate": "09/28/2017 01:14:34", "content": "To make it a bit more clear, we also used:
\n\nThank you guys!
\n\nP. S. I will update my code very soon https://github.com/sergeyshilin/Kaggle-Carvana-Image-Masking-Challenge
", "rawMarkdown": "To make it a bit more clear, we also used:\n\n 1. Absolutely the same uNet from here: https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37523\n 2. Train-time augmentation from there: https://www.kaggle.com/gaborfodor/augmentation-methods\n\nThank you guys!\n\nP. S. I will update my code very soon https://github.com/sergeyshilin/Kaggle-Carvana-Image-Masking-Challenge", "votes": null }, { "id": "224963", "postDate": "09/28/2017 01:45:11", "content": "very nice work!
", "rawMarkdown": "very nice work!", "votes": null }, { "id": "224964", "postDate": "09/28/2017 01:48:23", "content": "We've learned so much from you! Such a hard work, that I am sure everyone appreciates. Thank you!
", "rawMarkdown": "We've learned so much from you! Such a hard work, that I am sure everyone appreciates. Thank you!", "votes": null }, { "id": "225047", "postDate": "09/28/2017 04:48:49", "content": "@true_pk thanks for sharing. By the time I finished training and predicting with my my 3rd and best model today, I did not have enough time to do any ensembling. I am at least glad to have moved from the sub 300 to a position lower than 200.
\n\nHow did you do the ensemble? Are you in a position to share your ensemble code?
", "rawMarkdown": "true_pk thanks for sharing. By the time I finished training and predicting with my my 3rd and best model today, I did not have enough time to do any ensembling. I am at least glad to have moved from the sub 300 to a position lower than 200. \n\nHow did you do the ensemble? Are you in a position to share your ensemble code?", "votes": null }, { "id": "225078", "postDate": "09/28/2017 07:23:46", "content": "Really elegant solution, great job!
", "rawMarkdown": "Really elegant solution, great job!", "votes": null }, { "id": "225128", "postDate": "09/28/2017 10:24:20", "content": "Good results! Also had only one GTX 1080 and used similar idea with cutting a car + ensembling (7 models). Got 0.9968 on private.
", "rawMarkdown": "Good results! Also had only one GTX 1080 and used similar idea with cutting a car + ensembling (7 models). Got 0.9968 on private.", "votes": null }, { "id": "225232", "postDate": "09/28/2017 15:07:19", "content": "That's a pretty good result! By the way, our single model (3-c from the list) itself gave us 0.996845 on the private LB that is actually only 4 positions lower than our current.
", "rawMarkdown": "That's a pretty good result! By the way, our single model (3-c from the list) itself gave us 0.996845 on the private LB that is actually only 4 positions lower than our current.", "votes": null }, { "id": "225241", "postDate": "09/28/2017 15:15:20", "content": "I will share the code very soon, so you can follow my github repo from the above to be aware. We firstly tried three different configurations of uNet1024. The best of them gave us 0.996845 that is a bit worse than our current result. Ensembling was an easy thing to do, so you basically load three different models with their weights and predict a test batch three times. After that you can take an average or to sum predictions with their weights (as we did)
", "rawMarkdown": "I will share the code very soon, so you can follow my github repo from the above to be aware. We firstly tried three different configurations of uNet1024. The best of them gave us 0.996845 that is a bit worse than our current result. Ensembling was an easy thing to do, so you basically load three different models with their weights and predict a test batch three times. After that you can take an average or to sum predictions with their weights (as we did)", "votes": null }, { "id": "225250", "postDate": "09/28/2017 15:36:49", "content": "What was your training time per epoch? I see you used batch size 3 for 1024, did you add batches together like Heng mentioned? What improvement did you get from cutting out the bounding boxes?
", "rawMarkdown": "What was your training time per epoch? I see you used batch size 3 for 1024, did you add batches together like Heng mentioned? What improvement did you get from cutting out the bounding boxes?", "votes": null }, { "id": "225262", "postDate": "09/28/2017 16:00:26", "content": "Hey Paul, thanks for the comment. Training time was about 35 minutes per epoch with a batch of 3, as you said. I've tried to use accumulating trick that Heng mentioned a few days before the deadline and gave up as it didn't want to work.
\n\n\n\n\nWhat improvement did you get from cutting out the bounding boxes
\n
It gives a huge improvement for models with a small input size like 128 or 256. For example considering uNet256 it boosted us up from 0.9943 to 0.9961. For uNet512 it is 0.9966 instead of 0.9955. For uNet1024 it also gives a boost but not so huge though
", "rawMarkdown": "Hey Paul, thanks for the comment. Training time was about 35 minutes per epoch with a batch of 3, as you said. I've tried to use accumulating trick that Heng mentioned a few days before the deadline and gave up as it didn't want to work.\n\n> What improvement did you get from cutting out the bounding boxes\n\nIt gives a huge improvement for models with a small input size like 128 or 256. For example considering uNet256 it boosted us up from 0.9943 to 0.9961. For uNet512 it is 0.9966 instead of 0.9955. For uNet1024 it also gives a boost but not so huge though", "votes": null }, { "id": "225282", "postDate": "09/28/2017 16:54:47", "content": "Thanks @true_pk and congratulations.
", "rawMarkdown": "Thanks @true_pk and congratulations.", "votes": null } ], "comments": [ { "id": 224959, "author_name": "truepk", "author_url": "", "post_date": "09/28/2017 01:14:34", "content": "To make it a bit more clear, we also used:
\n\nThank you guys!
\n\nP. S. I will update my code very soon https://github.com/sergeyshilin/Kaggle-Carvana-Image-Masking-Challenge
", "votes": null, "replies": [] }, { "id": 224963, "author_name": "hengck23", "author_url": "", "post_date": "09/28/2017 01:45:11", "content": "very nice work!
", "votes": null, "replies": [ { "id": 224964, "author_name": "truepk", "author_url": "", "post_date": "09/28/2017 01:48:23", "content": "We've learned so much from you! Such a hard work, that I am sure everyone appreciates. Thank you!
", "votes": null, "replies": [] } ] }, { "id": 225047, "author_name": "sheriytm", "author_url": "", "post_date": "09/28/2017 04:48:49", "content": "@true_pk thanks for sharing. By the time I finished training and predicting with my my 3rd and best model today, I did not have enough time to do any ensembling. I am at least glad to have moved from the sub 300 to a position lower than 200.
\n\nHow did you do the ensemble? Are you in a position to share your ensemble code?
", "votes": null, "replies": [ { "id": 225241, "author_name": "truepk", "author_url": "", "post_date": "09/28/2017 15:15:20", "content": "I will share the code very soon, so you can follow my github repo from the above to be aware. We firstly tried three different configurations of uNet1024. The best of them gave us 0.996845 that is a bit worse than our current result. Ensembling was an easy thing to do, so you basically load three different models with their weights and predict a test batch three times. After that you can take an average or to sum predictions with their weights (as we did)
", "votes": null, "replies": [] }, { "id": 225282, "author_name": "sheriytm", "author_url": "", "post_date": "09/28/2017 16:54:47", "content": "Thanks @true_pk and congratulations.
", "votes": null, "replies": [] } ] }, { "id": 225078, "author_name": "ikibardin", "author_url": "", "post_date": "09/28/2017 07:23:46", "content": "Really elegant solution, great job!
", "votes": null, "replies": [] }, { "id": 225128, "author_name": "klimovich", "author_url": "", "post_date": "09/28/2017 10:24:20", "content": "Good results! Also had only one GTX 1080 and used similar idea with cutting a car + ensembling (7 models). Got 0.9968 on private.
", "votes": null, "replies": [ { "id": 225232, "author_name": "truepk", "author_url": "", "post_date": "09/28/2017 15:07:19", "content": "That's a pretty good result! By the way, our single model (3-c from the list) itself gave us 0.996845 on the private LB that is actually only 4 positions lower than our current.
", "votes": null, "replies": [] } ] }, { "id": 225250, "author_name": "dezmoanded", "author_url": "", "post_date": "09/28/2017 15:36:49", "content": "What was your training time per epoch? I see you used batch size 3 for 1024, did you add batches together like Heng mentioned? What improvement did you get from cutting out the bounding boxes?
", "votes": null, "replies": [ { "id": 225262, "author_name": "truepk", "author_url": "", "post_date": "09/28/2017 16:00:26", "content": "Hey Paul, thanks for the comment. Training time was about 35 minutes per epoch with a batch of 3, as you said. I've tried to use accumulating trick that Heng mentioned a few days before the deadline and gave up as it didn't want to work.
\n\n\n\n\nWhat improvement did you get from cutting out the bounding boxes
\n
It gives a huge improvement for models with a small input size like 128 or 256. For example considering uNet256 it boosted us up from 0.9943 to 0.9961. For uNet512 it is 0.9966 instead of 0.9955. For uNet1024 it also gives a boost but not so huge though
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "224956": "I am reading all the topics with results from other teams and I am a very excited but not surprised about how important it was to have many GPU cards. Since I was dreaming about having my own GPU during the competition, I'd like to tell you our team's story and show what our final solution consists of.\n\nI've started to participate in the competition because I had no experience with fully convolutional neural networks, so it seemed to be a great chance to learn something new. Moreover this is my first DL competition on Kaggle and I am really proud to be in top-100 on a public leaderboard.\n\nWe didn't have our own GPU at all. All I was able to do is to run training overnight on my server with a single GTX 1080 at work. Night is the only time when no one else is training their own models there. Let's say we started to contribute about 3 weeks ago, that means we had 20 nights including weekends :D\n\nAnyway, this our solution is:\n\n 1. Train uNet 1024 on the raw input. This is used to predict bounding boxes.\n 2. Cut a car out of a predicted mask with a 40 pixels padding. It gives approximately 56% of a car on the picture. Then we use cutted images only.\n 3. As a final solution we use an ensemble of 3 models:\n\n a) uNet 1024 trained on low quality images using bce_dice_loss\n\n b) uNet 1024 trained on low quality images using weighted_bce_dice_loss\n\n c) uNet 1024 trained on high quality images using bce_dice_loss\n 4. Ensemble models with weights [0.3, 0.1, 0.6] accordingly.\n\nThat is pretty it ;)\n\nBy the way, even uNet 256 with the same configuration gives about 0.9963\n\nI am very glad to have our model that small and simple, so it could be trained on a single GPU without any problems. It makes me a bit happy when I read about all these 40-GPUs solutions :)\nAnd I'd like to thank everyone who contributed during the competition, it helped me to eventually learn FCNs.\n\nCheers!", "224959": "To make it a bit more clear, we also used:\n\n 1. Absolutely the same uNet from here: https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37523\n 2. Train-time augmentation from there: https://www.kaggle.com/gaborfodor/augmentation-methods\n\nThank you guys!\n\nP. S. I will update my code very soon https://github.com/sergeyshilin/Kaggle-Carvana-Image-Masking-Challenge", "224963": "very nice work!", "224964": "We've learned so much from you! Such a hard work, that I am sure everyone appreciates. Thank you!", "225047": "true_pk thanks for sharing. By the time I finished training and predicting with my my 3rd and best model today, I did not have enough time to do any ensembling. I am at least glad to have moved from the sub 300 to a position lower than 200. \n\nHow did you do the ensemble? Are you in a position to share your ensemble code?", "225078": "Really elegant solution, great job!", "225128": "Good results! Also had only one GTX 1080 and used similar idea with cutting a car + ensembling (7 models). Got 0.9968 on private.", "225232": "That's a pretty good result! By the way, our single model (3-c from the list) itself gave us 0.996845 on the private LB that is actually only 4 positions lower than our current.", "225241": "I will share the code very soon, so you can follow my github repo from the above to be aware. We firstly tried three different configurations of uNet1024. The best of them gave us 0.996845 that is a bit worse than our current result. Ensembling was an easy thing to do, so you basically load three different models with their weights and predict a test batch three times. After that you can take an average or to sum predictions with their weights (as we did)", "225250": "What was your training time per epoch? I see you used batch size 3 for 1024, did you add batches together like Heng mentioned? What improvement did you get from cutting out the bounding boxes?", "225262": "Hey Paul, thanks for the comment. Training time was about 35 minutes per epoch with a batch of 3, as you said. I've tried to use accumulating trick that Heng mentioned a few days before the deadline and gave up as it didn't want to work.\n\n> What improvement did you get from cutting out the bounding boxes\n\nIt gives a huge improvement for models with a small input size like 128 or 256. For example considering uNet256 it boosted us up from 0.9943 to 0.9961. For uNet512 it is 0.9966 instead of 0.9955. For uNet1024 it also gives a boost but not so huge though", "225282": "Thanks @true_pk and congratulations." }, "source": "meta" }