I don't have much experience on model tuning, so I need help.
\n\nI built a U-Net model with the pretrained VGG16 (remove fully connected layers) for downsampling and some transposed convolutional layers for upsampling. I also add skip connections. I freeze the downsampling part and train only the upsampling layers on images with at least one ship. But the submission result is even worse than the empty prediction. So I am wondering how I can make improvements? Is there anything else I can try other than tuning hyper-parameters? i didn't do data processing except augmentation by flipping, rotation, shifting.
\n\nAny suggestion will be appreciated!
", "messages": [ { "id": "383605", "postDate": "09/09/2018 04:21:21", "content": "I don't have much experience on model tuning, so I need help.
\n\nI built a U-Net model with the pretrained VGG16 (remove fully connected layers) for downsampling and some transposed convolutional layers for upsampling. I also add skip connections. I freeze the downsampling part and train only the upsampling layers on images with at least one ship. But the submission result is even worse than the empty prediction. So I am wondering how I can make improvements? Is there anything else I can try other than tuning hyper-parameters? i didn't do data processing except augmentation by flipping, rotation, shifting.
\n\nAny suggestion will be appreciated!
", "rawMarkdown": "I don't have much experience on model tuning, so I need help. \n\nI built a U-Net model with the pretrained VGG16 (remove fully connected layers) for downsampling and some transposed convolutional layers for upsampling. I also add skip connections. I freeze the downsampling part and train only the upsampling layers on images with at least one ship. But the submission result is even worse than the empty prediction. So I am wondering how I can make improvements? Is there anything else I can try other than tuning hyper-parameters? i didn't do data processing except augmentation by flipping, rotation, shifting. \n\nAny suggestion will be appreciated!", "votes": null }, { "id": "383625", "postDate": "09/09/2018 06:02:45", "content": "U-net is not really the best model for this competition if you want to get high score (check my comment on https://www.kaggle.com/c/airbus-ship-detection/discussion/64730). I think the upper bound for this model is ~0.93 score on public LB, check https://www.kaggle.com/iafoss/unet34-submission-0-89-public-lb/comments for more details. I got ~0.92 with some creative post-processing but without TTA. The model is essentially the same as in my post, but I trained it further. To improve the score one may consider models that are based on detection of boxes since in the competition you are asked for individual pixalized bounding boxes rather than ship masks. Also U-net has problems with separating several ships closely located to each other.
\n\nWhat is validation dice or IoU of your model?
", "rawMarkdown": "U-net is not really the best model for this competition if you want to get high score (check my comment on https://www.kaggle.com/c/airbus-ship-detection/discussion/64730). I think the upper bound for this model is ~0.93 score on public LB, check https://www.kaggle.com/iafoss/unet34-submission-0-89-public-lb/comments for more details. I got ~0.92 with some creative post-processing but without TTA. The model is essentially the same as in my post, but I trained it further. To improve the score one may consider models that are based on detection of boxes since in the competition you are asked for individual pixalized bounding boxes rather than ship masks. Also U-net has problems with separating several ships closely located to each other.\n\nWhat is validation dice or IoU of your model?", "votes": null }, { "id": "383994", "postDate": "09/10/2018 05:27:36", "content": "Thanks!\nMy IoU can only get to around 0.4. Why is it that U-net couldn't separate several ships? is there an intuitive reason for that?
", "rawMarkdown": "Thanks!\nMy IoU can only get to around 0.4. Why is it that U-net couldn't separate several ships? is there an intuitive reason for that?", "votes": null }, { "id": "385208", "postDate": "09/10/2018 14:42:00", "content": "Even for a quite accurate model the masks for ships located very close to each other will overlap. So, it will be difficult to get individual masks for those ships. However, if ships are far away from each other, the masks of individual ships can be readily separated.
\n\nHowever, I think, your model can be improved. Did you train entire model after you trained the decoder part? It will boost your score since ImageNet is quite different from images in this competition. Did you do training and following predictions on full size images or rescaled ones? The results should be better if you get predictions from a full size images. You should check the loss function you use and think what are the problems and how it can be improved. Also you can check how your model performs on prediction for images without ships, pick 3-5k false positives, and add them to your training set. Finally, you can stack your model with one that predicts ship/nothing.
", "rawMarkdown": "Even for a quite accurate model the masks for ships located very close to each other will overlap. So, it will be difficult to get individual masks for those ships. However, if ships are far away from each other, the masks of individual ships can be readily separated.\n\nHowever, I think, your model can be improved. Did you train entire model after you trained the decoder part? It will boost your score since ImageNet is quite different from images in this competition. Did you do training and following predictions on full size images or rescaled ones? The results should be better if you get predictions from a full size images. You should check the loss function you use and think what are the problems and how it can be improved. Also you can check how your model performs on prediction for images without ships, pick 3-5k false positives, and add them to your training set. Finally, you can stack your model with one that predicts ship/nothing.", "votes": null }, { "id": "385366", "postDate": "09/10/2018 21:13:47", "content": "I used the full size images for training. I didn't train the entire model after the decoder part. Should I use a much smaller learning rate for it? I think my problem might be false positive. Now I am trying the focal loss you mentioned in your kernel. Thanks for the tips.
", "rawMarkdown": "I used the full size images for training. I didn't train the entire model after the decoder part. Should I use a much smaller learning rate for it? I think my problem might be false positive. Now I am trying the focal loss you mentioned in your kernel. Thanks for the tips.", "votes": null }, { "id": "385389", "postDate": "09/10/2018 23:01:16", "content": "Yes, otherwise you may face some problems during training. When I'm training entire net, I'm using different learning rate for different layers. For upper layer it may be quite high, while for pretrained layers lr should be lower. But if you cannot do it, you may try to decrease your lr by 10 times and check if training is stable.
", "rawMarkdown": "Yes, otherwise you may face some problems during training. When I'm training entire net, I'm using different learning rate for different layers. For upper layer it may be quite high, while for pretrained layers lr should be lower. But if you cannot do it, you may try to decrease your lr by 10 times and check if training is stable.", "votes": null } ], "comments": [ { "id": 383625, "author_name": "iafoss", "author_url": "", "post_date": "09/09/2018 06:02:45", "content": "U-net is not really the best model for this competition if you want to get high score (check my comment on https://www.kaggle.com/c/airbus-ship-detection/discussion/64730). I think the upper bound for this model is ~0.93 score on public LB, check https://www.kaggle.com/iafoss/unet34-submission-0-89-public-lb/comments for more details. I got ~0.92 with some creative post-processing but without TTA. The model is essentially the same as in my post, but I trained it further. To improve the score one may consider models that are based on detection of boxes since in the competition you are asked for individual pixalized bounding boxes rather than ship masks. Also U-net has problems with separating several ships closely located to each other.
\n\nWhat is validation dice or IoU of your model?
", "votes": null, "replies": [ { "id": 383994, "author_name": "zhengrui315", "author_url": "", "post_date": "09/10/2018 05:27:36", "content": "Thanks!\nMy IoU can only get to around 0.4. Why is it that U-net couldn't separate several ships? is there an intuitive reason for that?
", "votes": null, "replies": [] }, { "id": 385208, "author_name": "iafoss", "author_url": "", "post_date": "09/10/2018 14:42:00", "content": "Even for a quite accurate model the masks for ships located very close to each other will overlap. So, it will be difficult to get individual masks for those ships. However, if ships are far away from each other, the masks of individual ships can be readily separated.
\n\nHowever, I think, your model can be improved. Did you train entire model after you trained the decoder part? It will boost your score since ImageNet is quite different from images in this competition. Did you do training and following predictions on full size images or rescaled ones? The results should be better if you get predictions from a full size images. You should check the loss function you use and think what are the problems and how it can be improved. Also you can check how your model performs on prediction for images without ships, pick 3-5k false positives, and add them to your training set. Finally, you can stack your model with one that predicts ship/nothing.
", "votes": null, "replies": [] }, { "id": 385366, "author_name": "zhengrui315", "author_url": "", "post_date": "09/10/2018 21:13:47", "content": "I used the full size images for training. I didn't train the entire model after the decoder part. Should I use a much smaller learning rate for it? I think my problem might be false positive. Now I am trying the focal loss you mentioned in your kernel. Thanks for the tips.
", "votes": null, "replies": [] }, { "id": 385389, "author_name": "iafoss", "author_url": "", "post_date": "09/10/2018 23:01:16", "content": "Yes, otherwise you may face some problems during training. When I'm training entire net, I'm using different learning rate for different layers. For upper layer it may be quite high, while for pretrained layers lr should be lower. But if you cannot do it, you may try to decrease your lr by 10 times and check if training is stable.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "383605": "I don't have much experience on model tuning, so I need help. \n\nI built a U-Net model with the pretrained VGG16 (remove fully connected layers) for downsampling and some transposed convolutional layers for upsampling. I also add skip connections. I freeze the downsampling part and train only the upsampling layers on images with at least one ship. But the submission result is even worse than the empty prediction. So I am wondering how I can make improvements? Is there anything else I can try other than tuning hyper-parameters? i didn't do data processing except augmentation by flipping, rotation, shifting. \n\nAny suggestion will be appreciated!", "383625": "U-net is not really the best model for this competition if you want to get high score (check my comment on https://www.kaggle.com/c/airbus-ship-detection/discussion/64730). I think the upper bound for this model is ~0.93 score on public LB, check https://www.kaggle.com/iafoss/unet34-submission-0-89-public-lb/comments for more details. I got ~0.92 with some creative post-processing but without TTA. The model is essentially the same as in my post, but I trained it further. To improve the score one may consider models that are based on detection of boxes since in the competition you are asked for individual pixalized bounding boxes rather than ship masks. Also U-net has problems with separating several ships closely located to each other.\n\nWhat is validation dice or IoU of your model?", "383994": "Thanks!\nMy IoU can only get to around 0.4. Why is it that U-net couldn't separate several ships? is there an intuitive reason for that?", "385208": "Even for a quite accurate model the masks for ships located very close to each other will overlap. So, it will be difficult to get individual masks for those ships. However, if ships are far away from each other, the masks of individual ships can be readily separated.\n\nHowever, I think, your model can be improved. Did you train entire model after you trained the decoder part? It will boost your score since ImageNet is quite different from images in this competition. Did you do training and following predictions on full size images or rescaled ones? The results should be better if you get predictions from a full size images. You should check the loss function you use and think what are the problems and how it can be improved. Also you can check how your model performs on prediction for images without ships, pick 3-5k false positives, and add them to your training set. Finally, you can stack your model with one that predicts ship/nothing.", "385366": "I used the full size images for training. I didn't train the entire model after the decoder part. Should I use a much smaller learning rate for it? I think my problem might be false positive. Now I am trying the focal loss you mentioned in your kernel. Thanks for the tips.", "385389": "Yes, otherwise you may face some problems during training. When I'm training entire net, I'm using different learning rate for different layers. For upper layer it may be quite high, while for pretrained layers lr should be lower. But if you cannot do it, you may try to decrease your lr by 10 times and check if training is stable." }, "source": "meta" }