Hi guys, \nwhile working on this competition, I have kind of hit the wall. I cant get my network working. \nAny help about it will make my day. :)
\n\nHere are the details - \nI am trying to implement Unet with input as image tiles (cut out from full images) and output as 5-channel masks. The masks are created by making small squares ( should be changed to something else later) at places where 5 types of sea lions are located.\nI have tried various layer feature combinations, but the network is not learning at all. After some amount of training(3k - 4k patches) , the most frequent class in train data becomes white and all other become black. The weights are not changing at all with the training.\nI am using Dice coeff. with momentum optimizer. \nBelow are the tensorboard outputs for a run - \n(For ground truth and prediction images in tensorboard, output of 5 classes are concatenated horizontally.)
\n\nI mean , should I increase the learning rate? Should I decrease momentum value? Is regularizer (or dropout) hindering the learning? Or is is just that I have to train it longer to get reasonable results?
\n\nAny help about it would really make my day. Thanks
", "messages": [ { "id": "194116", "postDate": "06/19/2017 12:14:50", "content": "Hi guys, \nwhile working on this competition, I have kind of hit the wall. I cant get my network working. \nAny help about it will make my day. :)
\n\nHere are the details - \nI am trying to implement Unet with input as image tiles (cut out from full images) and output as 5-channel masks. The masks are created by making small squares ( should be changed to something else later) at places where 5 types of sea lions are located.\nI have tried various layer feature combinations, but the network is not learning at all. After some amount of training(3k - 4k patches) , the most frequent class in train data becomes white and all other become black. The weights are not changing at all with the training.\nI am using Dice coeff. with momentum optimizer. \nBelow are the tensorboard outputs for a run - \n(For ground truth and prediction images in tensorboard, output of 5 classes are concatenated horizontally.)
\n\nI mean , should I increase the learning rate? Should I decrease momentum value? Is regularizer (or dropout) hindering the learning? Or is is just that I have to train it longer to get reasonable results?
\n\nAny help about it would really make my day. Thanks
", "rawMarkdown": "Hi guys, \nwhile working on this competition, I have kind of hit the wall. I cant get my network working. \nAny help about it will make my day. :) \n\nHere are the details - \nI am trying to implement Unet with input as image tiles (cut out from full images) and output as 5-channel masks. The masks are created by making small squares ( should be changed to something else later) at places where 5 types of sea lions are located.\nI have tried various layer feature combinations, but the network is not learning at all. After some amount of training(3k - 4k patches) , the most frequent class in train data becomes white and all other become black. The weights are not changing at all with the training.\nI am using Dice coeff. with momentum optimizer. \nBelow are the tensorboard outputs for a run - \n(For ground truth and prediction images in tensorboard, output of 5 classes are concatenated horizontally.)\n\nI mean , should I increase the learning rate? Should I decrease momentum value? Is regularizer (or dropout) hindering the learning? Or is is just that I have to train it longer to get reasonable results?\n\nAny help about it would really make my day. Thanks", "votes": null }, { "id": "194128", "postDate": "06/19/2017 12:46:29", "content": "What's your current learning rate? Could you try using a simpler loss function (such as crossentropy) with an optimiser such as SGD or Adam (default params) to see if it works then - just to rule out if it is an issue with optimisation.
", "rawMarkdown": "What's your current learning rate? Could you try using a simpler loss function (such as crossentropy) with an optimiser such as SGD or Adam (default params) to see if it works then - just to rule out if it is an issue with optimisation.", "votes": null }, { "id": "194145", "postDate": "06/19/2017 13:41:59", "content": "May be worth to try softmax instead over 6 types including background, it should penalize more completely wrong prediction
", "rawMarkdown": "May be worth to try softmax instead over 6 types including background, it should penalize more completely wrong prediction", "votes": null }, { "id": "194148", "postDate": "06/19/2017 13:47:46", "content": "Thanks, @anokas. My current learning rate is 0.003. I have tried with other learning rates too ( 0.1 - 0.001) but with no luck.I have also used Cross Entropy with Momentum optimizer but got similar results.\nI will try Cross Entropy with Adam, and post an update here when I get the results.
", "rawMarkdown": "Thanks, @anokas. My current learning rate is 0.003. I have tried with other learning rates too ( 0.1 - 0.001) but with no luck.I have also used Cross Entropy with Momentum optimizer but got similar results.\nI will try Cross Entropy with Adam, and post an update here when I get the results.", "votes": null }, { "id": "194149", "postDate": "06/19/2017 13:48:29", "content": "Kapil,\nFirst follow @anokas' advise.\nThen, since I got the same problem than you where most the sealions but the pups are identified as adt_fem and the damned weights won't change, so try to input the full pictures with heavy but controlled augmentation instead of a reduced tile. I am using 1 layer output (6 values) with sparse cross entropy.
", "rawMarkdown": "Kapil,\nFirst follow @anokas' advise.\nThen, since I got the same problem than you where most the sealions but the pups are identified as adt_fem and the damned weights won't change, so try to input the full pictures with heavy but controlled augmentation instead of a reduced tile. I am using 1 layer output (6 values) with sparse cross entropy.", "votes": null }, { "id": "194169", "postDate": "06/19/2017 14:39:04", "content": "@eagle4 I have tried the tiramisu network with 1 channel output having 6 classes with taarget being small squares at the places of sealions but all it could predict is the background when I use softmax and it becomes more unstable if I try to use it for single class using sigmoid. Can you help me out here?
\n\nI have also tried loss functions like jaccard, sparse categorical crossentropy and tversky.
", "rawMarkdown": "eagle4 I have tried the tiramisu network with 1 channel output having 6 classes with taarget being small squares at the places of sealions but all it could predict is the background when I use softmax and it becomes more unstable if I try to use it for single class using sigmoid. Can you help me out here?\n\nI have also tried loss functions like jaccard, sparse categorical crossentropy and tversky.", "votes": null }, { "id": "194175", "postDate": "06/19/2017 15:08:49", "content": "@Daft Vader,\nI tried tiramisu but how did you run it ? did you implemented tiles of 224x224 with a batch of 3 or 4?
\n\nMy understanding after working on this competition for a while is that we can't downsize the full pictures too much or you loose the details of what make a sealions compared to the background and if you feed a model with a large majority of tiles from the original images that contains only the background pixels, the model pretty much doesn't learn anything.
\n\nI may be wrong but in the case of tiramisu, in the original model, all random tiles contain several categories of items whereas, in our cases of the sealions, if you take random tiles, a large of the pixels would be only background if you cut the full picture in small tiles. It may be doable if you manage to feed the model with a balanced mix of background and sealions but the reasons for which I quickly gave up is the time required to feed the 18,000 pictures of the test set by cutting the picture 5400x3400 in 224x224 tiles.
\n\nLet me know your thoughts.
", "rawMarkdown": "Daft Vader,\nI tried tiramisu but how did you run it ? did you implemented tiles of 224x224 with a batch of 3 or 4? \n\nMy understanding after working on this competition for a while is that we can't downsize the full pictures too much or you loose the details of what make a sealions compared to the background and if you feed a model with a large majority of tiles from the original images that contains only the background pixels, the model pretty much doesn't learn anything.\n\nI may be wrong but in the case of tiramisu, in the original model, all random tiles contain several categories of items whereas, in our cases of the sealions, if you take random tiles, a large of the pixels would be only background if you cut the full picture in small tiles. It may be doable if you manage to feed the model with a balanced mix of background and sealions but the reasons for which I quickly gave up is the time required to feed the 18,000 pictures of the test set by cutting the picture 5400x3400 in 224x224 tiles.\n\nLet me know your thoughts.", "votes": null }, { "id": "194180", "postDate": "06/19/2017 15:33:31", "content": "@eagle4 I tried tiramisu with image of size 512*512 by slicing it into 224*224 with condition being atleast one dot/square must be present in that slice, still if I use softmax for all the classes + void/background model gives the maximum probability of the output being the background/void class.
", "rawMarkdown": "eagle4 I tried tiramisu with image of size 512*512 by slicing it into 224*224 with condition being atleast one dot/square must be present in that slice, still if I use softmax for all the classes + void/background model gives the maximum probability of the output being the background/void class.", "votes": null }, { "id": "194193", "postDate": "06/19/2017 16:10:15", "content": "Unfortunately, I don't have an answer to your question. Can you try feeding the model with the tiles with only high concentrations of dots with some augmentation and see if it gives something else than background ?
", "rawMarkdown": "Unfortunately, I don't have an answer to your question. Can you try feeding the model with the tiles with only high concentrations of dots with some augmentation and see if it gives something else than background ?", "votes": null }, { "id": "194226", "postDate": "06/19/2017 19:04:12", "content": "I know this doesn't contribute to the discussion at hand, but --
\n\nI just thought this image was hilarious. A pup on it's mom's back. One final 'screw you' to our classifiers, lol.
", "rawMarkdown": "I know this doesn't contribute to the discussion at hand, but --\n\nI just thought [this image][1] was hilarious. A pup on it's mom's back. One final 'screw you' to our classifiers, lol.\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/194116/6696/Screen%20Shot%202017-06-19%20at%205.29.51%20PM.png", "votes": null }, { "id": "194232", "postDate": "06/19/2017 19:42:42", "content": "Heh, nothing a UNet with sigmoid output couldn't handle :)
", "rawMarkdown": "Heh, nothing a UNet with sigmoid output couldn't handle :)", "votes": null }, { "id": "194442", "postDate": "06/20/2017 16:07:53", "content": "Seems you have all zeros (all background) prediction and the reason can be unbalanced dataset/batch, so try to reduce number of background tiles in batch.
", "rawMarkdown": "Seems you have all zeros (all background) prediction and the reason can be unbalanced dataset/batch, so try to reduce number of background tiles in batch.", "votes": null }, { "id": "194552", "postDate": "06/21/2017 01:47:03", "content": "in my U-net experiment, the Adam is better than SGD. please check your code whether div 255 or color channels according to your mentioned questions!
", "rawMarkdown": "in my U-net experiment, the Adam is better than SGD. please check your code whether div 255 or color channels according to your mentioned questions!", "votes": null }, { "id": "194553", "postDate": "06/21/2017 01:53:11", "content": "Hi, can you share your ideas about how to improve your testing point from 15.2 up tp 14.2. I have been staying at 15.1 for long time. In my opions, the main problem is how to classfy the females and juvs, This may contribute at least 9~10 points to the last sum loss.
", "rawMarkdown": "Hi, can you share your ideas about how to improve your testing point from 15.2 up tp 14.2. I have been staying at 15.1 for long time. In my opions, the main problem is how to classfy the females and juvs, This may contribute at least 9~10 points to the last sum loss.", "votes": null }, { "id": "194642", "postDate": "06/21/2017 09:01:43", "content": "@pandav5 Hi, yeah I agree that lion misclassification is the major problem. I doubt that I have any significant tricks that you don't already know about, will post all details after the end, cheers!
", "rawMarkdown": "pandav5 Hi, yeah I agree that lion misclassification is the major problem. I doubt that I have any significant tricks that you don't already know about, will post all details after the end, cheers!", "votes": null }, { "id": "195815", "postDate": "06/25/2017 04:47:39", "content": "Wow,Good luck~~I prefer to your details~HaHa~
", "rawMarkdown": "Wow,Good luck~~I prefer to your details~HaHa~", "votes": null } ], "comments": [ { "id": 194128, "author_name": "anokas", "author_url": "", "post_date": "06/19/2017 12:46:29", "content": "What's your current learning rate? Could you try using a simpler loss function (such as crossentropy) with an optimiser such as SGD or Adam (default params) to see if it works then - just to rule out if it is an issue with optimisation.
", "votes": null, "replies": [ { "id": 194148, "author_name": "kapilyadav", "author_url": "", "post_date": "06/19/2017 13:47:46", "content": "Thanks, @anokas. My current learning rate is 0.003. I have tried with other learning rates too ( 0.1 - 0.001) but with no luck.I have also used Cross Entropy with Momentum optimizer but got similar results.\nI will try Cross Entropy with Adam, and post an update here when I get the results.
", "votes": null, "replies": [] }, { "id": 194552, "author_name": "pandav5", "author_url": "", "post_date": "06/21/2017 01:47:03", "content": "in my U-net experiment, the Adam is better than SGD. please check your code whether div 255 or color channels according to your mentioned questions!
", "votes": null, "replies": [] } ] }, { "id": 194145, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "06/19/2017 13:41:59", "content": "May be worth to try softmax instead over 6 types including background, it should penalize more completely wrong prediction
", "votes": null, "replies": [] }, { "id": 194149, "author_name": "chabir", "author_url": "", "post_date": "06/19/2017 13:48:29", "content": "Kapil,\nFirst follow @anokas' advise.\nThen, since I got the same problem than you where most the sealions but the pups are identified as adt_fem and the damned weights won't change, so try to input the full pictures with heavy but controlled augmentation instead of a reduced tile. I am using 1 layer output (6 values) with sparse cross entropy.
", "votes": null, "replies": [ { "id": 194169, "author_name": "syeddanish", "author_url": "", "post_date": "06/19/2017 14:39:04", "content": "@eagle4 I have tried the tiramisu network with 1 channel output having 6 classes with taarget being small squares at the places of sealions but all it could predict is the background when I use softmax and it becomes more unstable if I try to use it for single class using sigmoid. Can you help me out here?
\n\nI have also tried loss functions like jaccard, sparse categorical crossentropy and tversky.
", "votes": null, "replies": [] }, { "id": 194175, "author_name": "chabir", "author_url": "", "post_date": "06/19/2017 15:08:49", "content": "@Daft Vader,\nI tried tiramisu but how did you run it ? did you implemented tiles of 224x224 with a batch of 3 or 4?
\n\nMy understanding after working on this competition for a while is that we can't downsize the full pictures too much or you loose the details of what make a sealions compared to the background and if you feed a model with a large majority of tiles from the original images that contains only the background pixels, the model pretty much doesn't learn anything.
\n\nI may be wrong but in the case of tiramisu, in the original model, all random tiles contain several categories of items whereas, in our cases of the sealions, if you take random tiles, a large of the pixels would be only background if you cut the full picture in small tiles. It may be doable if you manage to feed the model with a balanced mix of background and sealions but the reasons for which I quickly gave up is the time required to feed the 18,000 pictures of the test set by cutting the picture 5400x3400 in 224x224 tiles.
\n\nLet me know your thoughts.
", "votes": null, "replies": [] }, { "id": 194180, "author_name": "syeddanish", "author_url": "", "post_date": "06/19/2017 15:33:31", "content": "@eagle4 I tried tiramisu with image of size 512*512 by slicing it into 224*224 with condition being atleast one dot/square must be present in that slice, still if I use softmax for all the classes + void/background model gives the maximum probability of the output being the background/void class.
", "votes": null, "replies": [] }, { "id": 194193, "author_name": "chabir", "author_url": "", "post_date": "06/19/2017 16:10:15", "content": "Unfortunately, I don't have an answer to your question. Can you try feeding the model with the tiles with only high concentrations of dots with some augmentation and see if it gives something else than background ?
", "votes": null, "replies": [] } ] }, { "id": 194226, "author_name": "authman", "author_url": "", "post_date": "06/19/2017 19:04:12", "content": "I know this doesn't contribute to the discussion at hand, but --
\n\nI just thought this image was hilarious. A pup on it's mom's back. One final 'screw you' to our classifiers, lol.
", "votes": null, "replies": [ { "id": 194232, "author_name": "lopuhin", "author_url": "", "post_date": "06/19/2017 19:42:42", "content": "Heh, nothing a UNet with sigmoid output couldn't handle :)
", "votes": null, "replies": [] }, { "id": 194553, "author_name": "pandav5", "author_url": "", "post_date": "06/21/2017 01:53:11", "content": "Hi, can you share your ideas about how to improve your testing point from 15.2 up tp 14.2. I have been staying at 15.1 for long time. In my opions, the main problem is how to classfy the females and juvs, This may contribute at least 9~10 points to the last sum loss.
", "votes": null, "replies": [] }, { "id": 194642, "author_name": "lopuhin", "author_url": "", "post_date": "06/21/2017 09:01:43", "content": "@pandav5 Hi, yeah I agree that lion misclassification is the major problem. I doubt that I have any significant tricks that you don't already know about, will post all details after the end, cheers!
", "votes": null, "replies": [] }, { "id": 195815, "author_name": "pandav5", "author_url": "", "post_date": "06/25/2017 04:47:39", "content": "Wow,Good luck~~I prefer to your details~HaHa~
", "votes": null, "replies": [] } ] }, { "id": 194442, "author_name": "mrgloom", "author_url": "", "post_date": "06/20/2017 16:07:53", "content": "Seems you have all zeros (all background) prediction and the reason can be unbalanced dataset/batch, so try to reduce number of background tiles in batch.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "194116": "Hi guys, \nwhile working on this competition, I have kind of hit the wall. I cant get my network working. \nAny help about it will make my day. :) \n\nHere are the details - \nI am trying to implement Unet with input as image tiles (cut out from full images) and output as 5-channel masks. The masks are created by making small squares ( should be changed to something else later) at places where 5 types of sea lions are located.\nI have tried various layer feature combinations, but the network is not learning at all. After some amount of training(3k - 4k patches) , the most frequent class in train data becomes white and all other become black. The weights are not changing at all with the training.\nI am using Dice coeff. with momentum optimizer. \nBelow are the tensorboard outputs for a run - \n(For ground truth and prediction images in tensorboard, output of 5 classes are concatenated horizontally.)\n\nI mean , should I increase the learning rate? Should I decrease momentum value? Is regularizer (or dropout) hindering the learning? Or is is just that I have to train it longer to get reasonable results?\n\nAny help about it would really make my day. Thanks", "194128": "What's your current learning rate? Could you try using a simpler loss function (such as crossentropy) with an optimiser such as SGD or Adam (default params) to see if it works then - just to rule out if it is an issue with optimisation.", "194145": "May be worth to try softmax instead over 6 types including background, it should penalize more completely wrong prediction", "194148": "Thanks, @anokas. My current learning rate is 0.003. I have tried with other learning rates too ( 0.1 - 0.001) but with no luck.I have also used Cross Entropy with Momentum optimizer but got similar results.\nI will try Cross Entropy with Adam, and post an update here when I get the results.", "194149": "Kapil,\nFirst follow @anokas' advise.\nThen, since I got the same problem than you where most the sealions but the pups are identified as adt_fem and the damned weights won't change, so try to input the full pictures with heavy but controlled augmentation instead of a reduced tile. I am using 1 layer output (6 values) with sparse cross entropy.", "194169": "eagle4 I have tried the tiramisu network with 1 channel output having 6 classes with taarget being small squares at the places of sealions but all it could predict is the background when I use softmax and it becomes more unstable if I try to use it for single class using sigmoid. Can you help me out here?\n\nI have also tried loss functions like jaccard, sparse categorical crossentropy and tversky.", "194175": "Daft Vader,\nI tried tiramisu but how did you run it ? did you implemented tiles of 224x224 with a batch of 3 or 4? \n\nMy understanding after working on this competition for a while is that we can't downsize the full pictures too much or you loose the details of what make a sealions compared to the background and if you feed a model with a large majority of tiles from the original images that contains only the background pixels, the model pretty much doesn't learn anything.\n\nI may be wrong but in the case of tiramisu, in the original model, all random tiles contain several categories of items whereas, in our cases of the sealions, if you take random tiles, a large of the pixels would be only background if you cut the full picture in small tiles. It may be doable if you manage to feed the model with a balanced mix of background and sealions but the reasons for which I quickly gave up is the time required to feed the 18,000 pictures of the test set by cutting the picture 5400x3400 in 224x224 tiles.\n\nLet me know your thoughts.", "194180": "eagle4 I tried tiramisu with image of size 512*512 by slicing it into 224*224 with condition being atleast one dot/square must be present in that slice, still if I use softmax for all the classes + void/background model gives the maximum probability of the output being the background/void class.", "194193": "Unfortunately, I don't have an answer to your question. Can you try feeding the model with the tiles with only high concentrations of dots with some augmentation and see if it gives something else than background ?", "194226": "I know this doesn't contribute to the discussion at hand, but --\n\nI just thought [this image][1] was hilarious. A pup on it's mom's back. One final 'screw you' to our classifiers, lol.\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/194116/6696/Screen%20Shot%202017-06-19%20at%205.29.51%20PM.png", "194232": "Heh, nothing a UNet with sigmoid output couldn't handle :)", "194442": "Seems you have all zeros (all background) prediction and the reason can be unbalanced dataset/batch, so try to reduce number of background tiles in batch.", "194552": "in my U-net experiment, the Adam is better than SGD. please check your code whether div 255 or color channels according to your mentioned questions!", "194553": "Hi, can you share your ideas about how to improve your testing point from 15.2 up tp 14.2. I have been staying at 15.1 for long time. In my opions, the main problem is how to classfy the females and juvs, This may contribute at least 9~10 points to the last sum loss.", "194642": "pandav5 Hi, yeah I agree that lion misclassification is the major problem. I doubt that I have any significant tricks that you don't already know about, will post all details after the end, cheers!", "195815": "Wow,Good luck~~I prefer to your details~HaHa~" }, "source": "meta" }![this image](\"https://storage.googleapis.com/kaggle-forum-message-attachments/194116/6696/Screen%20Shot%202017-06-19%20at%205.29.51%20PM.png\"){kind=link}