I'm using keras.\nAfter looking some kernels and dicussions, I have thought that unet+resnet34 will work.\nThen, I've tried unet+resnet34 and unet+resnext50, but the score is not good.\nunet+resnet34 gave me a 0.699. unet+resnext50 is not converging.
\n\nmy code is quiet similar with this my kernel. https://www.kaggle.com/youhanlee/1st-solution-reproducing-1-unet-resnet34-se. (using qubvel's library)
\n\nThere are some choices.\n(1) using pretrained weight or just using architecture without pretrained weights.\nI tried both, but it didn't work.
\n\n(2) loss function\ndice_loss, focal_loss
\n\nI referred some kernels. and used below loss function.
\n\nfrom keras.losses import binary_crossentropy\nfrom keras import backend as K
\n\n def dice_coef(y_true, y_pred):\n y_true_f = K.flatten(y_true)\n y_pred = K.cast(y_pred, 'float32')\n y_pred_f = K.cast(K.greater(K.flatten(y_pred), 0.5), 'float32')\n intersection = y_true_f * y_pred_f\n score = 2. * K.sum(intersection) / (K.sum(y_true_f) + K.sum(y_pred_f))\n return score\n\n def dice_loss(y_true, y_pred):\n smooth = 1.\n y_true_f = K.flatten(y_true)\n y_pred_f = K.flatten(y_pred)\n intersection = y_true_f * y_pred_f\n score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n return 1. - score\n\n def bce_dice_loss(y_true, y_pred):\n return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)\n(3) Number of epochs.\nI learned unet+resnet34 with ~300 epochs. Is it not enough?
\n\n(4) augmentation\nI used keras imageDataAugmentation.
\n\ndg_args = dict(featurewise_center = False,\n samplewise_center = False,\n rotation_range = 20,\n width_shift_range = 0.2,\n height_shift_range = 0.2,\n shear_range = 0.02,\n zoom_range = [0.8, 1.25],\n horizontal_flip = True,\n vertical_flip = True,\n fill_mode = 'reflect',\n data_format = 'channels_last')\n(5) data selection\nI ignored the non-zero images.
\n\nI think my loss function is wrong. \nHow do you, kagglers, think about my process?\nIf you have a time, please give me your advice. Thanks!
", "messages": [ { "id": "415446", "postDate": "11/05/2018 06:13:28", "content": "I'm using keras.\nAfter looking some kernels and dicussions, I have thought that unet+resnet34 will work.\nThen, I've tried unet+resnet34 and unet+resnext50, but the score is not good.\nunet+resnet34 gave me a 0.699. unet+resnext50 is not converging.
\n\nmy code is quiet similar with this my kernel. https://www.kaggle.com/youhanlee/1st-solution-reproducing-1-unet-resnet34-se. (using qubvel's library)
\n\nThere are some choices.\n(1) using pretrained weight or just using architecture without pretrained weights.\nI tried both, but it didn't work.
\n\n(2) loss function\ndice_loss, focal_loss
\n\nI referred some kernels. and used below loss function.
\n\nfrom keras.losses import binary_crossentropy\nfrom keras import backend as K
\n\n def dice_coef(y_true, y_pred):\n y_true_f = K.flatten(y_true)\n y_pred = K.cast(y_pred, 'float32')\n y_pred_f = K.cast(K.greater(K.flatten(y_pred), 0.5), 'float32')\n intersection = y_true_f * y_pred_f\n score = 2. * K.sum(intersection) / (K.sum(y_true_f) + K.sum(y_pred_f))\n return score\n\n def dice_loss(y_true, y_pred):\n smooth = 1.\n y_true_f = K.flatten(y_true)\n y_pred_f = K.flatten(y_pred)\n intersection = y_true_f * y_pred_f\n score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n return 1. - score\n\n def bce_dice_loss(y_true, y_pred):\n return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)\n(3) Number of epochs.\nI learned unet+resnet34 with ~300 epochs. Is it not enough?
\n\n(4) augmentation\nI used keras imageDataAugmentation.
\n\ndg_args = dict(featurewise_center = False,\n samplewise_center = False,\n rotation_range = 20,\n width_shift_range = 0.2,\n height_shift_range = 0.2,\n shear_range = 0.02,\n zoom_range = [0.8, 1.25],\n horizontal_flip = True,\n vertical_flip = True,\n fill_mode = 'reflect',\n data_format = 'channels_last')\n(5) data selection\nI ignored the non-zero images.
\n\nI think my loss function is wrong. \nHow do you, kagglers, think about my process?\nIf you have a time, please give me your advice. Thanks!
", "rawMarkdown": "I'm using keras.\nAfter looking some kernels and dicussions, I have thought that unet+resnet34 will work.\nThen, I've tried unet+resnet34 and unet+resnext50, but the score is not good.\nunet+resnet34 gave me a 0.699. unet+resnext50 is not converging.\n\nmy code is quiet similar with this my kernel. https://www.kaggle.com/youhanlee/1st-solution-reproducing-1-unet-resnet34-se. (using qubvel's library)\n\nThere are some choices.\n(1) using pretrained weight or just using architecture without pretrained weights.\nI tried both, but it didn't work.\n\n(2) loss function\ndice_loss, focal_loss\n\n I referred some kernels. and used below loss function.\n\n\n\n\nfrom keras.losses import binary_crossentropy\nfrom keras import backend as K\n \n def dice_coef(y_true, y_pred):\n y_true_f = K.flatten(y_true)\n y_pred = K.cast(y_pred, 'float32')\n y_pred_f = K.cast(K.greater(K.flatten(y_pred), 0.5), 'float32')\n intersection = y_true_f * y_pred_f\n score = 2. * K.sum(intersection) / (K.sum(y_true_f) + K.sum(y_pred_f))\n return score\n \n def dice_loss(y_true, y_pred):\n smooth = 1.\n y_true_f = K.flatten(y_true)\n y_pred_f = K.flatten(y_pred)\n intersection = y_true_f * y_pred_f\n score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n return 1. - score\n \n def bce_dice_loss(y_true, y_pred):\n return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)\n\n(3) Number of epochs.\nI learned unet+resnet34 with ~300 epochs. Is it not enough? \n\n\n(4) augmentation\nI used keras imageDataAugmentation.\n\n dg_args = dict(featurewise_center = False,\n samplewise_center = False,\n rotation_range = 20,\n width_shift_range = 0.2,\n height_shift_range = 0.2,\n shear_range = 0.02,\n zoom_range = [0.8, 1.25],\n horizontal_flip = True,\n vertical_flip = True,\n fill_mode = 'reflect',\n data_format = 'channels_last')\n\n\n(5) data selection\nI ignored the non-zero images.\n\n\nI think my loss function is wrong. \nHow do you, kagglers, think about my process?\nIf you have a time, please give me your advice. Thanks!", "votes": null }, { "id": "415490", "postDate": "11/05/2018 08:14:56", "content": "For what it’s worth, my 0.741 score is a simple unet-resnet34, ~30 epochs, using a single GPU. I suspect there are many routes to similar scores with this challenge.
", "rawMarkdown": "For what it’s worth, my 0.741 score is a simple unet-resnet34, ~30 epochs, using a single GPU. I suspect there are many routes to similar scores with this challenge.", "votes": null }, { "id": "415492", "postDate": "11/05/2018 08:19:45", "content": "Oh, amazing. I think Deep learning requires quite experience and technique.\nDid you use the pretrained-weight for unet+resnet34?
", "rawMarkdown": "Oh, amazing. I think Deep learning requires quite experience and technique.\nDid you use the pretrained-weight for unet+resnet34?", "votes": null }, { "id": "415494", "postDate": "11/05/2018 08:21:37", "content": "1.) Both should work! Pretrained a little better. I am using the same library.
\n\n2.) keras.losses.binary_crossentropy just reduces along the last (axis=-1) axis. So due to broadcasting binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred) will still be a (batch_size, height, width) tensor. You probably want K.mean(binary_crossentropy(y_true, y_pred)) + dice_loss(y_true, y_pred).
3.) 300 epochs is enough. I get LB 736 with ResNet18 based encoder and 155 epochs.
\n\n4.) I never used ImageDataAugmentation. Just make sure the masks get augmented in the same fashion.
\n\n5.) Training only on images with ship got me ~705 LB score. You can start by training on ship images. Then predict the non-ship -> add false positives.
", "rawMarkdown": "1.) Both should work! Pretrained a little better. I am using the same library.\n\n2.) `keras.losses.binary_crossentropy` just reduces along the last (`axis=-1`) axis. So due to broadcasting `binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)` will still be a (batch_size, height, width) tensor. You probably want `K.mean(binary_crossentropy(y_true, y_pred)) + dice_loss(y_true, y_pred)`.\n\n3.) 300 epochs is enough. I get LB 736 with ResNet18 based encoder and 155 epochs.\n\n4.) I never used ImageDataAugmentation. Just make sure the masks get augmented in the same fashion.\n\n5.) Training only on images with ship got me ~705 LB score. You can start by training on ship images. Then predict the non-ship -> add false positives.", "votes": null }, { "id": "415513", "postDate": "11/05/2018 08:47:55", "content": "Thanks for very kind and detailed comments.
\n\nI applied your 2nd suggestion and 4th suggestion in my process. I'll talk to you After learning.
\n\nCould I ask one more question?\nDid you use both pseudo-labeling and deep-supervision?
\n\nI've tried deep-supervision, but didn't work.
", "rawMarkdown": "Thanks for very kind and detailed comments.\n\nI applied your 2nd suggestion and 4th suggestion in my process. I'll talk to you After learning.\n\nCould I ask one more question?\nDid you use both pseudo-labeling and deep-supervision? \n\nI've tried deep-supervision, but didn't work.", "votes": null }, { "id": "415521", "postDate": "11/05/2018 08:57:38", "content": "Not yet. The model is a pretrained ResNet18 + FPN. I will try pseudo-labeling later on.
", "rawMarkdown": "Not yet. The model is a pretrained ResNet18 + FPN. I will try pseudo-labeling later on.", "votes": null }, { "id": "415535", "postDate": "11/05/2018 09:35:38", "content": "@robga, if I may ask, are you training on 768 pixels or are you using 256 pixels and mosaic techniques for inference?
", "rawMarkdown": "robga, if I may ask, are you training on 768 pixels or are you using 256 pixels and mosaic techniques for inference?", "votes": null }, { "id": "415536", "postDate": "11/05/2018 09:39:50", "content": "@seesee, if I may ask, do you train at 768 pixels or 256 crops? What about inference? Direct on 768 pixels or do you use mosaic techniques?
", "rawMarkdown": "seesee, if I may ask, do you train at 768 pixels or 256 crops? What about inference? Direct on 768 pixels or do you use mosaic techniques?", "votes": null }, { "id": "415565", "postDate": "11/05/2018 10:25:16", "content": "I started to train only on unique 256x256 patches. However, I got many FPs. I think this is due to very small cropped boats that the network sees during training. Now, I am training with 384x384 random crops. Bigger crops might be even better. For submission & validation I am using the full 768 images. I just use the mosaic to select the training/validation splits.
", "rawMarkdown": "I started to train only on unique 256x256 patches. However, I got many FPs. I think this is due to very small cropped boats that the network sees during training. Now, I am training with 384x384 random crops. Bigger crops might be even better. For submission & validation I am using the full 768 images. I just use the mosaic to select the training/validation splits.", "votes": null }, { "id": "415585", "postDate": "11/05/2018 11:05:34", "content": "Thanks! I have this weird behavior that if I train on 256 pixels crops and try to infer on 768 pixels I get completely wrong results. So right now I'm training on 768 pixels but it takes ages..
", "rawMarkdown": "Thanks! I have this weird behavior that if I train on 256 pixels crops and try to infer on 768 pixels I get completely wrong results. So right now I'm training on 768 pixels but it takes ages..", "votes": null }, { "id": "415674", "postDate": "11/05/2018 13:24:41", "content": "@arc144 You may try training on 256x256 crops (or even smaller) and then train for additional 10-15 epochs on full size.
", "rawMarkdown": "arc144 You may try training on 256x256 crops (or even smaller) and then train for additional 10-15 epochs on full size.", "votes": null }, { "id": "415694", "postDate": "11/05/2018 14:11:24", "content": "Hi~ @robga Did consider make a team?\nWe now have 10+ gpu and 0.731 score ~\nsome teammate are the 5th player in the TGS \nsome experience of us may help you get a better score boost
", "rawMarkdown": "Hi~ @robga Did consider make a team?\nWe now have 10+ gpu and 0.731 score ~\nsome teammate are the 5th player in the TGS \nsome experience of us may help you get a better score boost", "votes": null }, { "id": "415716", "postDate": "11/05/2018 14:49:55", "content": "@see-- Thanks for the reply! With your precious comments, I'm testing the Unet+Resnet34. Thanks!
", "rawMarkdown": "see-- Thanks for the reply! With your precious comments, I'm testing the Unet+Resnet34. Thanks!", "votes": null }, { "id": "420591", "postDate": "11/13/2018 21:19:01", "content": "@YouHan Lee, can you share the methods that improved ur score up from 0.699
", "rawMarkdown": "YouHan Lee, can you share the methods that improved ur score up from 0.699", "votes": null }, { "id": "420593", "postDate": "11/13/2018 21:20:04", "content": "@robga, would u mind sharing ur approach (and even better the code) after the competition is over.
", "rawMarkdown": "robga, would u mind sharing ur approach (and even better the code) after the competition is over.", "votes": null } ], "comments": [ { "id": 415490, "author_name": "robga", "author_url": "", "post_date": "11/05/2018 08:14:56", "content": "For what it’s worth, my 0.741 score is a simple unet-resnet34, ~30 epochs, using a single GPU. I suspect there are many routes to similar scores with this challenge.
", "votes": null, "replies": [ { "id": 415492, "author_name": "youhanlee", "author_url": "", "post_date": "11/05/2018 08:19:45", "content": "Oh, amazing. I think Deep learning requires quite experience and technique.\nDid you use the pretrained-weight for unet+resnet34?
", "votes": null, "replies": [] }, { "id": 415535, "author_name": "arc144", "author_url": "", "post_date": "11/05/2018 09:35:38", "content": "@robga, if I may ask, are you training on 768 pixels or are you using 256 pixels and mosaic techniques for inference?
", "votes": null, "replies": [] }, { "id": 415694, "author_name": "asd3032801", "author_url": "", "post_date": "11/05/2018 14:11:24", "content": "Hi~ @robga Did consider make a team?\nWe now have 10+ gpu and 0.731 score ~\nsome teammate are the 5th player in the TGS \nsome experience of us may help you get a better score boost
", "votes": null, "replies": [] }, { "id": 420593, "author_name": "ravivadapalli", "author_url": "", "post_date": "11/13/2018 21:20:04", "content": "@robga, would u mind sharing ur approach (and even better the code) after the competition is over.
", "votes": null, "replies": [] } ] }, { "id": 415494, "author_name": "seesee", "author_url": "", "post_date": "11/05/2018 08:21:37", "content": "1.) Both should work! Pretrained a little better. I am using the same library.
\n\n2.) keras.losses.binary_crossentropy just reduces along the last (axis=-1) axis. So due to broadcasting binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred) will still be a (batch_size, height, width) tensor. You probably want K.mean(binary_crossentropy(y_true, y_pred)) + dice_loss(y_true, y_pred).
3.) 300 epochs is enough. I get LB 736 with ResNet18 based encoder and 155 epochs.
\n\n4.) I never used ImageDataAugmentation. Just make sure the masks get augmented in the same fashion.
\n\n5.) Training only on images with ship got me ~705 LB score. You can start by training on ship images. Then predict the non-ship -> add false positives.
", "votes": null, "replies": [ { "id": 415513, "author_name": "youhanlee", "author_url": "", "post_date": "11/05/2018 08:47:55", "content": "Thanks for very kind and detailed comments.
\n\nI applied your 2nd suggestion and 4th suggestion in my process. I'll talk to you After learning.
\n\nCould I ask one more question?\nDid you use both pseudo-labeling and deep-supervision?
\n\nI've tried deep-supervision, but didn't work.
", "votes": null, "replies": [] }, { "id": 415521, "author_name": "seesee", "author_url": "", "post_date": "11/05/2018 08:57:38", "content": "Not yet. The model is a pretrained ResNet18 + FPN. I will try pseudo-labeling later on.
", "votes": null, "replies": [] }, { "id": 415536, "author_name": "arc144", "author_url": "", "post_date": "11/05/2018 09:39:50", "content": "@seesee, if I may ask, do you train at 768 pixels or 256 crops? What about inference? Direct on 768 pixels or do you use mosaic techniques?
", "votes": null, "replies": [] }, { "id": 415565, "author_name": "seesee", "author_url": "", "post_date": "11/05/2018 10:25:16", "content": "I started to train only on unique 256x256 patches. However, I got many FPs. I think this is due to very small cropped boats that the network sees during training. Now, I am training with 384x384 random crops. Bigger crops might be even better. For submission & validation I am using the full 768 images. I just use the mosaic to select the training/validation splits.
", "votes": null, "replies": [] }, { "id": 415585, "author_name": "arc144", "author_url": "", "post_date": "11/05/2018 11:05:34", "content": "Thanks! I have this weird behavior that if I train on 256 pixels crops and try to infer on 768 pixels I get completely wrong results. So right now I'm training on 768 pixels but it takes ages..
", "votes": null, "replies": [] }, { "id": 415674, "author_name": "ddanevskyi", "author_url": "", "post_date": "11/05/2018 13:24:41", "content": "@arc144 You may try training on 256x256 crops (or even smaller) and then train for additional 10-15 epochs on full size.
", "votes": null, "replies": [] }, { "id": 415716, "author_name": "youhanlee", "author_url": "", "post_date": "11/05/2018 14:49:55", "content": "@see-- Thanks for the reply! With your precious comments, I'm testing the Unet+Resnet34. Thanks!
", "votes": null, "replies": [] }, { "id": 420591, "author_name": "ravivadapalli", "author_url": "", "post_date": "11/13/2018 21:19:01", "content": "@YouHan Lee, can you share the methods that improved ur score up from 0.699
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "415446": "I'm using keras.\nAfter looking some kernels and dicussions, I have thought that unet+resnet34 will work.\nThen, I've tried unet+resnet34 and unet+resnext50, but the score is not good.\nunet+resnet34 gave me a 0.699. unet+resnext50 is not converging.\n\nmy code is quiet similar with this my kernel. https://www.kaggle.com/youhanlee/1st-solution-reproducing-1-unet-resnet34-se. (using qubvel's library)\n\nThere are some choices.\n(1) using pretrained weight or just using architecture without pretrained weights.\nI tried both, but it didn't work.\n\n(2) loss function\ndice_loss, focal_loss\n\n I referred some kernels. and used below loss function.\n\n\n\n\nfrom keras.losses import binary_crossentropy\nfrom keras import backend as K\n \n def dice_coef(y_true, y_pred):\n y_true_f = K.flatten(y_true)\n y_pred = K.cast(y_pred, 'float32')\n y_pred_f = K.cast(K.greater(K.flatten(y_pred), 0.5), 'float32')\n intersection = y_true_f * y_pred_f\n score = 2. * K.sum(intersection) / (K.sum(y_true_f) + K.sum(y_pred_f))\n return score\n \n def dice_loss(y_true, y_pred):\n smooth = 1.\n y_true_f = K.flatten(y_true)\n y_pred_f = K.flatten(y_pred)\n intersection = y_true_f * y_pred_f\n score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n return 1. - score\n \n def bce_dice_loss(y_true, y_pred):\n return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)\n\n(3) Number of epochs.\nI learned unet+resnet34 with ~300 epochs. Is it not enough? \n\n\n(4) augmentation\nI used keras imageDataAugmentation.\n\n dg_args = dict(featurewise_center = False,\n samplewise_center = False,\n rotation_range = 20,\n width_shift_range = 0.2,\n height_shift_range = 0.2,\n shear_range = 0.02,\n zoom_range = [0.8, 1.25],\n horizontal_flip = True,\n vertical_flip = True,\n fill_mode = 'reflect',\n data_format = 'channels_last')\n\n\n(5) data selection\nI ignored the non-zero images.\n\n\nI think my loss function is wrong. \nHow do you, kagglers, think about my process?\nIf you have a time, please give me your advice. Thanks!", "415490": "For what it’s worth, my 0.741 score is a simple unet-resnet34, ~30 epochs, using a single GPU. I suspect there are many routes to similar scores with this challenge.", "415492": "Oh, amazing. I think Deep learning requires quite experience and technique.\nDid you use the pretrained-weight for unet+resnet34?", "415494": "1.) Both should work! Pretrained a little better. I am using the same library.\n\n2.) `keras.losses.binary_crossentropy` just reduces along the last (`axis=-1`) axis. So due to broadcasting `binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)` will still be a (batch_size, height, width) tensor. You probably want `K.mean(binary_crossentropy(y_true, y_pred)) + dice_loss(y_true, y_pred)`.\n\n3.) 300 epochs is enough. I get LB 736 with ResNet18 based encoder and 155 epochs.\n\n4.) I never used ImageDataAugmentation. Just make sure the masks get augmented in the same fashion.\n\n5.) Training only on images with ship got me ~705 LB score. You can start by training on ship images. Then predict the non-ship -> add false positives.", "415513": "Thanks for very kind and detailed comments.\n\nI applied your 2nd suggestion and 4th suggestion in my process. I'll talk to you After learning.\n\nCould I ask one more question?\nDid you use both pseudo-labeling and deep-supervision? \n\nI've tried deep-supervision, but didn't work.", "415521": "Not yet. The model is a pretrained ResNet18 + FPN. I will try pseudo-labeling later on.", "415535": "robga, if I may ask, are you training on 768 pixels or are you using 256 pixels and mosaic techniques for inference?", "415536": "seesee, if I may ask, do you train at 768 pixels or 256 crops? What about inference? Direct on 768 pixels or do you use mosaic techniques?", "415565": "I started to train only on unique 256x256 patches. However, I got many FPs. I think this is due to very small cropped boats that the network sees during training. Now, I am training with 384x384 random crops. Bigger crops might be even better. For submission & validation I am using the full 768 images. I just use the mosaic to select the training/validation splits.", "415585": "Thanks! I have this weird behavior that if I train on 256 pixels crops and try to infer on 768 pixels I get completely wrong results. So right now I'm training on 768 pixels but it takes ages..", "415674": "arc144 You may try training on 256x256 crops (or even smaller) and then train for additional 10-15 epochs on full size.", "415694": "Hi~ @robga Did consider make a team?\nWe now have 10+ gpu and 0.731 score ~\nsome teammate are the 5th player in the TGS \nsome experience of us may help you get a better score boost", "415716": "see-- Thanks for the reply! With your precious comments, I'm testing the Unet+Resnet34. Thanks!", "420591": "YouHan Lee, can you share the methods that improved ur score up from 0.699", "420593": "robga, would u mind sharing ur approach (and even better the code) after the competition is over." }, "source": "meta" }