Hello everyone!
\n\nI'm still relatively new to Kaggle, so I'm still learning a lot. I'm getting on this competition pretty late, but I've been working on it for the past 2 weeks roughly. I've been stuck since then on a super low score, and I just can't figure out what the problem is. Here is my Kernel which is based upon several other kernels, but mostly the Keras-Starter Kernel which is basically what I was going for anyways.\nIn a nutshell, I'm using an InceptionResNetV2, with a last layer with 1103 nodes with a softmax activation.\nI think something is wrong with the way my model is learning as from what I display, the validation F2 score is already higher than the training one on the first epochs, which doesn't make much sense. The training F2 score is getting better as the epoch go on, and the training loss goes down, which seems correct. However the validation score and loss just don't seem to make sense, and I can't figure out why; hence after muliple days of searching, turning to the forum. If anybody would be willing to take a look at let me know what they think, I'd really appreciate it!
\n\nI have some ideas I would have wanted to try out for this competition, but if I can't even get a simple model working, then it's not even worth trying those! My timing wasn't great on this one.\nThanks for the help!
", "messages": [ { "id": "541763", "postDate": "06/03/2019 01:58:15", "content": "Hello everyone!
\n\nI'm still relatively new to Kaggle, so I'm still learning a lot. I'm getting on this competition pretty late, but I've been working on it for the past 2 weeks roughly. I've been stuck since then on a super low score, and I just can't figure out what the problem is. Here is my Kernel which is based upon several other kernels, but mostly the Keras-Starter Kernel which is basically what I was going for anyways.\nIn a nutshell, I'm using an InceptionResNetV2, with a last layer with 1103 nodes with a softmax activation.\nI think something is wrong with the way my model is learning as from what I display, the validation F2 score is already higher than the training one on the first epochs, which doesn't make much sense. The training F2 score is getting better as the epoch go on, and the training loss goes down, which seems correct. However the validation score and loss just don't seem to make sense, and I can't figure out why; hence after muliple days of searching, turning to the forum. If anybody would be willing to take a look at let me know what they think, I'd really appreciate it!
\n\nI have some ideas I would have wanted to try out for this competition, but if I can't even get a simple model working, then it's not even worth trying those! My timing wasn't great on this one.\nThanks for the help!
", "rawMarkdown": "Hello everyone!\n\nI'm still relatively new to Kaggle, so I'm still learning a lot. I'm getting on this competition pretty late, but I've been working on it for the past 2 weeks roughly. I've been stuck since then on a super low score, and I just can't figure out what the problem is. Here is [my Kernel ](https://www.kaggle.com/maxlenormand/act-softmax-incresnetv2) which is based upon several other kernels, but mostly [the Keras-Starter Kernel](https://www.kaggle.com/ateplyuk/keras-starter) which is basically what I was going for anyways.\nIn a nutshell, I'm using an InceptionResNetV2, with a last layer with 1103 nodes with a softmax activation.\nI think something is wrong with the way my model is learning as from what I display, the validation F2 score is already higher than the training one on the first epochs, which doesn't make much sense. The training F2 score is getting better as the epoch go on, and the training loss goes down, which seems correct. However the validation score and loss just don't seem to make sense, and I can't figure out why; hence after muliple days of searching, turning to the forum. If anybody would be willing to take a look at let me know what they think, I'd really appreciate it!\n\nI have some ideas I would have wanted to try out for this competition, but if I can't even get a simple model working, then it's not even worth trying those! My timing wasn't great on this one.\nThanks for the help!", "votes": null }, { "id": "541766", "postDate": "06/03/2019 02:09:14", "content": "Your early-stopping stops the model very early so it only trains for 4 epochs. It's probably best to get rid of the early stopping and train as long as you can
", "rawMarkdown": "Your early-stopping stops the model very early so it only trains for 4 epochs. It's probably best to get rid of the early stopping and train as long as you can", "votes": null }, { "id": "541805", "postDate": "06/03/2019 03:43:07", "content": "Yep, my early stopping isn't working very well, I agree.\nHowever I did train it over 15 epoch, and the result was no better, the score was still around 0.001 ~ 0.004. So the problem doesn't (only) come from that. And still, it doesn't explain why my validation loss and F2 score are higher than my train.
", "rawMarkdown": "Yep, my early stopping isn't working very well, I agree.\nHowever I did train it over 15 epoch, and the result was no better, the score was still around 0.001 ~ 0.004. So the problem doesn't (only) come from that. And still, it doesn't explain why my validation loss and F2 score are higher than my train.", "votes": null }, { "id": "541824", "postDate": "06/03/2019 04:35:21", "content": "Have you tried increasing your learning rate? Maybe 1e-2?
", "rawMarkdown": "Have you tried increasing your learning rate? Maybe 1e-2?", "votes": null }, { "id": "541828", "postDate": "06/03/2019 04:41:59", "content": "It's also weird that your validation f2 is higher than your training, but the loss is also higher.
", "rawMarkdown": "It's also weird that your validation f2 is higher than your training, but the loss is also higher.", "votes": null }, { "id": "541840", "postDate": "06/03/2019 05:13:41", "content": "Are you using pretrained imagenet weights or not?
\n\nI also built one Siamese Net solution which was not properly learning without pretrained imagenet weights
", "rawMarkdown": "Are you using pretrained imagenet weights or not?\n\nI also built one Siamese Net solution which was not properly learning without pretrained imagenet weights", "votes": null }, { "id": "541842", "postDate": "06/03/2019 05:16:13", "content": "I did try at 0.001 and 0.0001, but not at 0.01. I could try it just to see, even though I don't think that's the core of the problem.\nAnd yes, the fact that the validation metrics are higher is what concerns me the most, and I can't figure what's wrong with it.
", "rawMarkdown": "I did try at 0.001 and 0.0001, but not at 0.01. I could try it just to see, even though I don't think that's the core of the problem.\nAnd yes, the fact that the validation metrics are higher is what concerns me the most, and I can't figure what's wrong with it.", "votes": null }, { "id": "541845", "postDate": "06/03/2019 05:18:05", "content": "Yep, using imagenet weights. But maybe not properly.\nI'll have another look!
", "rawMarkdown": "Yep, using imagenet weights. But maybe not properly.\nI'll have another look!", "votes": null }, { "id": "541846", "postDate": "06/03/2019 05:18:27", "content": "If you are not using imagenet weights, that could explain a lot of problems. Because training from scratch would take a really long time. See if there is any problems with that and let us know.
", "rawMarkdown": "If you are not using imagenet weights, that could explain a lot of problems. Because training from scratch would take a really long time. See if there is any problems with that and let us know.", "votes": null }, { "id": "541847", "postDate": "06/03/2019 05:22:47", "content": "I know that, but I am using the imagenet weights :-)
", "rawMarkdown": "I know that, but I am using the imagenet weights :-)", "votes": null }, { "id": "541848", "postDate": "06/03/2019 05:27:19", "content": "Are you generating labels for your preds properly? there could be a bug there
", "rawMarkdown": "Are you generating labels for your preds properly? there could be a bug there", "votes": null }, { "id": "541858", "postDate": "06/03/2019 06:02:49", "content": "I think that might be where the problem is coming from yeah\nBecause I tweaked my model, and the result looks much better already. (you can see it in this Kernel or in the image attached)
", "rawMarkdown": "I think that might be where the problem is coming from yeah\nBecause I tweaked my model, and the result looks much better already. (you can see it [in this Kernel](https://www.kaggle.com/maxlenormand/change-model-act-softmax-incresnetv2) or in the image attached)", "votes": null }, { "id": "541868", "postDate": "06/03/2019 06:16:21", "content": "Indeed this looks better. BTW the kernel you linked is private.
", "rawMarkdown": "Indeed this looks better. BTW the kernel you linked is private.", "votes": null }, { "id": "541882", "postDate": "06/03/2019 06:40:37", "content": "My bad!\nAll public now!
", "rawMarkdown": "My bad!\nAll public now!", "votes": null }, { "id": "541938", "postDate": "06/03/2019 08:09:33", "content": "Just clone a working kernel. Btw, 4 epochs are not enough for training anyway. And you don't need InceptionResNetV2 for the start, take ResNet50 and make sure your training works as expected.
\n\nFYI, InceptionResNetV2 achieves 0.605 on validation for me.
", "rawMarkdown": "Just clone a working kernel. Btw, 4 epochs are not enough for training anyway. And you don't need InceptionResNetV2 for the start, take ResNet50 and make sure your training works as expected. \n\nFYI, InceptionResNetV2 achieves 0.605 on validation for me.", "votes": null }, { "id": "542138", "postDate": "06/03/2019 13:52:50", "content": "Sure I could do that, but I don't want to just clone a kernel. I want to be able to make on from scratch and understand it along the way. I've still pretty new to all of this, so I want to take it as a learning experience.\nAnd I just took InceptionResNet because that was one of the ones I used in a previous competition, so I just copied the code from there.\nAnd yeah, 4 epochs are not enough indeed, but at least they can start showing a trend. For debugging quickly I thought it might be a could way to go. Might not be though.
", "rawMarkdown": "Sure I could do that, but I don't want to just clone a kernel. I want to be able to make on from scratch and understand it along the way. I've still pretty new to all of this, so I want to take it as a learning experience.\nAnd I just took InceptionResNet because that was one of the ones I used in a previous competition, so I just copied the code from there.\nAnd yeah, 4 epochs are not enough indeed, but at least they can start showing a trend. For debugging quickly I thought it might be a could way to go. Might not be though.", "votes": null }, { "id": "542174", "postDate": "06/03/2019 14:38:01", "content": "I see 2 different points:
\n\n1) You are not giving enough iterations to your model to learn. You probably need more than 4 epochs especially for softmax - in my experience with this dataset at least. Increase the patience of your early stopping should improve this.
\n\n2) Even though your validation score is low, there is an important gap (> x10) between your validation score and your test score, the latter being very close to 0 anyway. It probably means that you are not submitting your results properly. I suspect that you have the same issue I went through caused by a bad order of your labels in the final dataframe you submit.
\n\nIn my case, it was caused by a mismatch between the alphabetic order and the numerical order: the numerical labels where converted to string and thus ordered by alphabetic order (\"0\", \"1\", \"10\", \"100\", etc.). As a consequence, I was submitting the results of the 10th label for the 3nd etc.
\n\nLook at the train_generator.class_indices to see the mapping between labels and their order and check that it's coherent. Another quick check is to look at your most frequently predicted class. It should be 813 (\"tag::men\"). Having a quick look at your submission, that does not seem to be the case here.
", "rawMarkdown": "I see 2 different points: \n\n1) You are not giving enough iterations to your model to learn. You probably need more than 4 epochs especially for softmax - in my experience with this dataset at least. Increase the patience of your early stopping should improve this.\n\n\n2) Even though your validation score is low, there is an important gap (> x10) between your validation score and your test score, the latter being very close to 0 anyway. It probably means that you are not submitting your results properly. I suspect that you have the same issue I went through caused by a bad order of your labels in the final dataframe you submit. \n\nIn my case, it was caused by a mismatch between the alphabetic order and the numerical order: the numerical labels where converted to string and thus ordered by alphabetic order (\"0\", \"1\", \"10\", \"100\", etc.). As a consequence, I was submitting the results of the 10th label for the 3nd etc. \n\nLook at the train\\_generator.class\\_indices to see the mapping between labels and their order and check that it's coherent. Another quick check is to look at your most frequently predicted class. It should be 813 (\"tag::men\"). Having a quick look at your submission, that does not seem to be the case here.", "votes": null }, { "id": "542245", "postDate": "06/03/2019 15:56:23", "content": "I had some similar issues and started the same way on this. This is my first image classification competition, just trying to learn how to do it all. I started by copying parts of a kernel using InceptionResNetV2. Now I have been using ResNet50 to just get started, but quickly running out of time since I only started on this 2 weeks ago and my first image learning one.
\n\nI had some issues initially to get the labels correct, as also mentioned by @kiurtis. I ended up one-hot encoding the target variables, and checking closely that they are actually correctly encoded (printing out specific indices in the one-hot encoded arrays to see they match the numbers from the labels dataframe). I only had a quick look at your kernel but you don't seem to be one-hot encoding it. Not sure how much difference it makes, as I said, just trying to learn and see myself. And limited time to experiment.
\n\nAlso, maybe use the inception_resnet_v2.preprocess_input() function on your images. It should give decent results anyway, but this should ensure at least the input has a similar distribution (I guess that is the correct term..).
", "rawMarkdown": "I had some similar issues and started the same way on this. This is my first image classification competition, just trying to learn how to do it all. I started by copying parts of a kernel using InceptionResNetV2. Now I have been using ResNet50 to just get started, but quickly running out of time since I only started on this 2 weeks ago and my first image learning one.\n\nI had some issues initially to get the labels correct, as also mentioned by @kiurtis. I ended up one-hot encoding the target variables, and checking closely that they are actually correctly encoded (printing out specific indices in the one-hot encoded arrays to see they match the numbers from the labels dataframe). I only had a quick look at your kernel but you don't seem to be one-hot encoding it. Not sure how much difference it makes, as I said, just trying to learn and see myself. And limited time to experiment.\n\nAlso, maybe use the inception_resnet_v2.preprocess_input() function on your images. It should give decent results anyway, but this should ensure at least the input has a similar distribution (I guess that is the correct term..).", "votes": null }, { "id": "542355", "postDate": "06/03/2019 18:36:01", "content": "I'm not saying copy-pasting code from kernels is good. But you are given many working examples. It's up to you to find a difference between your code and any of the working implementations. A good ML engineer must be good at coding.
\n\nI see your point though. I made my solution from scratch.
\n\nYou must have some serious error somewhere in your pipeline, like wrong image normalization. Another thing you could try is to take the code from any image classification tutorial and make it use competition data.
\n\nProblem is, if you change a lot of things at once and then click Run, it wouldn't work. Try doing small incremental changes while keeping an eye on validation metrics.
", "rawMarkdown": "I'm not saying copy-pasting code from kernels is good. But you are given many working examples. It's up to you to find a difference between your code and any of the working implementations. A good ML engineer must be good at coding.\n\nI see your point though. I made my solution from scratch.\n\nYou must have some serious error somewhere in your pipeline, like wrong image normalization. Another thing you could try is to take the code from any image classification tutorial and make it use competition data.\n\nProblem is, if you change a lot of things at once and then click Run, it wouldn't work. Try doing small incremental changes while keeping an eye on validation metrics.", "votes": null }, { "id": "542428", "postDate": "06/03/2019 22:49:41", "content": "Regarding this: \"It's also weird that your validation f2 is higher than your training, but the loss is also higher.\"\nThis is absolutely normal when augmentations are in place. Especially with mixup.
", "rawMarkdown": "Regarding this: \"It's also weird that your validation f2 is higher than your training, but the loss is also higher.\"\nThis is absolutely normal when augmentations are in place. Especially with mixup.", "votes": null }, { "id": "542630", "postDate": "06/04/2019 02:28:35", "content": "Well, I saw one-hot encoding was one option out of it, but I did want to try to do it without\nI saw that it could work without doing it like that, so I wanted to try.\nI think I got onboard this competition a bit too late to have time to solve any of these issues. But thanks for the ideas, I'll have a look at those!
", "rawMarkdown": "Well, I saw one-hot encoding was one option out of it, but I did want to try to do it without\nI saw that it could work without doing it like that, so I wanted to try.\nI think I got onboard this competition a bit too late to have time to solve any of these issues. But thanks for the ideas, I'll have a look at those!", "votes": null }, { "id": "542636", "postDate": "06/04/2019 02:30:30", "content": "Totally agree, and that's what I'm doing, tweaking one thing at a time (Kaggle allowing to run 4 kernels at once does help).\nI think my issue here is a lack of understanding at some point of the entire pipeline. I do understand most of it, but you only need one error for it not to work.
", "rawMarkdown": "Totally agree, and that's what I'm doing, tweaking one thing at a time (Kaggle allowing to run 4 kernels at once does help).\nI think my issue here is a lack of understanding at some point of the entire pipeline. I do understand most of it, but you only need one error for it not to work.", "votes": null }, { "id": "542637", "postDate": "06/04/2019 02:30:50", "content": "Okay, thanks! Good to know that!
", "rawMarkdown": "Okay, thanks! Good to know that!", "votes": null }, { "id": "542718", "postDate": "06/04/2019 03:59:39", "content": "Hi @maxlenormand ,
\n\nI think one issue could be this step: datagen=ImageDataGenerator(rescale=1./255.,validation_split=0.2)
\n\nSince we use imageNet weights we need to apply same normalization to the images as ImageNet did. \nIt may converge without this normalization but it will take a much longer time.
\n\nfor normalization you need to find mean value from ImageNet and subtract it from training and testing. \nI will try and see if i can fork your kernel and make this change and test my theory :-)
\n\nHappy Debugging ! (which in a way is one of the most fun part of ML )
", "rawMarkdown": "Hi @maxlenormand , \n\nI think one issue could be this step: datagen=ImageDataGenerator(rescale=1./255.,validation_split=0.2)\n\nSince we use imageNet weights we need to *apply same normalization to the images as ImageNet did*. \nIt may converge without this normalization but it will take a much longer time. \n\nfor normalization you need to find mean value from ImageNet and subtract it from training and testing. \nI will try and see if i can fork your kernel and make this change and test my theory :-) \n\nHappy Debugging ! (which in a way is one of the most fun part of ML )", "votes": null }, { "id": "542719", "postDate": "06/04/2019 04:00:48", "content": "also just saw averagemn's comment. i am suggesting the same as him. \"Also, maybe use the inceptionresnetv2.preprocess_input() function on your images\"
", "rawMarkdown": "also just saw averagemn's comment. i am suggesting the same as him. \"Also, maybe use the inceptionresnetv2.preprocess_input() function on your images\"", "votes": null }, { "id": "542874", "postDate": "06/04/2019 06:47:52", "content": "Just would like to thanks @kiurtis which has helped me in the background find a fix, it indeed was the predicitons that was not working properly\nI jumped up to 0.472 now thanks to that\nThanks to everyone for the help, really appreciate it! :-)
", "rawMarkdown": "Just would like to thanks @kiurtis which has helped me in the background find a fix, it indeed was the predicitons that was not working properly\nI jumped up to 0.472 now thanks to that\nThanks to everyone for the help, really appreciate it! :-)", "votes": null }, { "id": "542876", "postDate": "06/04/2019 06:49:21", "content": "Interesting, I didn't know about that. I'm not sure how to apply the same normalization as Imagenet, so I'd definitely be interested in seeing that, even after the competition is over!
", "rawMarkdown": "Interesting, I didn't know about that. I'm not sure how to apply the same normalization as Imagenet, so I'd definitely be interested in seeing that, even after the competition is over!", "votes": null } ], "comments": [ { "id": 541766, "author_name": "tanlikesmath", "author_url": "", "post_date": "06/03/2019 02:09:14", "content": "Your early-stopping stops the model very early so it only trains for 4 epochs. It's probably best to get rid of the early stopping and train as long as you can
", "votes": null, "replies": [ { "id": 541805, "author_name": "maxlenormand", "author_url": "", "post_date": "06/03/2019 03:43:07", "content": "Yep, my early stopping isn't working very well, I agree.\nHowever I did train it over 15 epoch, and the result was no better, the score was still around 0.001 ~ 0.004. So the problem doesn't (only) come from that. And still, it doesn't explain why my validation loss and F2 score are higher than my train.
", "votes": null, "replies": [] }, { "id": 541824, "author_name": "tanlikesmath", "author_url": "", "post_date": "06/03/2019 04:35:21", "content": "Have you tried increasing your learning rate? Maybe 1e-2?
", "votes": null, "replies": [] }, { "id": 541828, "author_name": "tanlikesmath", "author_url": "", "post_date": "06/03/2019 04:41:59", "content": "It's also weird that your validation f2 is higher than your training, but the loss is also higher.
", "votes": null, "replies": [] }, { "id": 541840, "author_name": "axel81", "author_url": "", "post_date": "06/03/2019 05:13:41", "content": "Are you using pretrained imagenet weights or not?
\n\nI also built one Siamese Net solution which was not properly learning without pretrained imagenet weights
", "votes": null, "replies": [] }, { "id": 541842, "author_name": "maxlenormand", "author_url": "", "post_date": "06/03/2019 05:16:13", "content": "I did try at 0.001 and 0.0001, but not at 0.01. I could try it just to see, even though I don't think that's the core of the problem.\nAnd yes, the fact that the validation metrics are higher is what concerns me the most, and I can't figure what's wrong with it.
", "votes": null, "replies": [] }, { "id": 541845, "author_name": "maxlenormand", "author_url": "", "post_date": "06/03/2019 05:18:05", "content": "Yep, using imagenet weights. But maybe not properly.\nI'll have another look!
", "votes": null, "replies": [] }, { "id": 541846, "author_name": "tanlikesmath", "author_url": "", "post_date": "06/03/2019 05:18:27", "content": "If you are not using imagenet weights, that could explain a lot of problems. Because training from scratch would take a really long time. See if there is any problems with that and let us know.
", "votes": null, "replies": [] }, { "id": 541847, "author_name": "maxlenormand", "author_url": "", "post_date": "06/03/2019 05:22:47", "content": "I know that, but I am using the imagenet weights :-)
", "votes": null, "replies": [] }, { "id": 541848, "author_name": "axel81", "author_url": "", "post_date": "06/03/2019 05:27:19", "content": "Are you generating labels for your preds properly? there could be a bug there
", "votes": null, "replies": [] }, { "id": 541858, "author_name": "maxlenormand", "author_url": "", "post_date": "06/03/2019 06:02:49", "content": "I think that might be where the problem is coming from yeah\nBecause I tweaked my model, and the result looks much better already. (you can see it in this Kernel or in the image attached)
", "votes": null, "replies": [] }, { "id": 541868, "author_name": "tanlikesmath", "author_url": "", "post_date": "06/03/2019 06:16:21", "content": "Indeed this looks better. BTW the kernel you linked is private.
", "votes": null, "replies": [] }, { "id": 541882, "author_name": "maxlenormand", "author_url": "", "post_date": "06/03/2019 06:40:37", "content": "My bad!\nAll public now!
", "votes": null, "replies": [] }, { "id": 542428, "author_name": "artyomp", "author_url": "", "post_date": "06/03/2019 22:49:41", "content": "Regarding this: \"It's also weird that your validation f2 is higher than your training, but the loss is also higher.\"\nThis is absolutely normal when augmentations are in place. Especially with mixup.
", "votes": null, "replies": [] }, { "id": 542637, "author_name": "maxlenormand", "author_url": "", "post_date": "06/04/2019 02:30:50", "content": "Okay, thanks! Good to know that!
", "votes": null, "replies": [] } ] }, { "id": 541938, "author_name": "artyomp", "author_url": "", "post_date": "06/03/2019 08:09:33", "content": "Just clone a working kernel. Btw, 4 epochs are not enough for training anyway. And you don't need InceptionResNetV2 for the start, take ResNet50 and make sure your training works as expected.
\n\nFYI, InceptionResNetV2 achieves 0.605 on validation for me.
", "votes": null, "replies": [ { "id": 542138, "author_name": "maxlenormand", "author_url": "", "post_date": "06/03/2019 13:52:50", "content": "Sure I could do that, but I don't want to just clone a kernel. I want to be able to make on from scratch and understand it along the way. I've still pretty new to all of this, so I want to take it as a learning experience.\nAnd I just took InceptionResNet because that was one of the ones I used in a previous competition, so I just copied the code from there.\nAnd yeah, 4 epochs are not enough indeed, but at least they can start showing a trend. For debugging quickly I thought it might be a could way to go. Might not be though.
", "votes": null, "replies": [] }, { "id": 542355, "author_name": "artyomp", "author_url": "", "post_date": "06/03/2019 18:36:01", "content": "I'm not saying copy-pasting code from kernels is good. But you are given many working examples. It's up to you to find a difference between your code and any of the working implementations. A good ML engineer must be good at coding.
\n\nI see your point though. I made my solution from scratch.
\n\nYou must have some serious error somewhere in your pipeline, like wrong image normalization. Another thing you could try is to take the code from any image classification tutorial and make it use competition data.
\n\nProblem is, if you change a lot of things at once and then click Run, it wouldn't work. Try doing small incremental changes while keeping an eye on validation metrics.
", "votes": null, "replies": [] }, { "id": 542636, "author_name": "maxlenormand", "author_url": "", "post_date": "06/04/2019 02:30:30", "content": "Totally agree, and that's what I'm doing, tweaking one thing at a time (Kaggle allowing to run 4 kernels at once does help).\nI think my issue here is a lack of understanding at some point of the entire pipeline. I do understand most of it, but you only need one error for it not to work.
", "votes": null, "replies": [] } ] }, { "id": 542174, "author_name": "kiurtis", "author_url": "", "post_date": "06/03/2019 14:38:01", "content": "I see 2 different points:
\n\n1) You are not giving enough iterations to your model to learn. You probably need more than 4 epochs especially for softmax - in my experience with this dataset at least. Increase the patience of your early stopping should improve this.
\n\n2) Even though your validation score is low, there is an important gap (> x10) between your validation score and your test score, the latter being very close to 0 anyway. It probably means that you are not submitting your results properly. I suspect that you have the same issue I went through caused by a bad order of your labels in the final dataframe you submit.
\n\nIn my case, it was caused by a mismatch between the alphabetic order and the numerical order: the numerical labels where converted to string and thus ordered by alphabetic order (\"0\", \"1\", \"10\", \"100\", etc.). As a consequence, I was submitting the results of the 10th label for the 3nd etc.
\n\nLook at the train_generator.class_indices to see the mapping between labels and their order and check that it's coherent. Another quick check is to look at your most frequently predicted class. It should be 813 (\"tag::men\"). Having a quick look at your submission, that does not seem to be the case here.
", "votes": null, "replies": [] }, { "id": 542245, "author_name": "donkeys", "author_url": "", "post_date": "06/03/2019 15:56:23", "content": "I had some similar issues and started the same way on this. This is my first image classification competition, just trying to learn how to do it all. I started by copying parts of a kernel using InceptionResNetV2. Now I have been using ResNet50 to just get started, but quickly running out of time since I only started on this 2 weeks ago and my first image learning one.
\n\nI had some issues initially to get the labels correct, as also mentioned by @kiurtis. I ended up one-hot encoding the target variables, and checking closely that they are actually correctly encoded (printing out specific indices in the one-hot encoded arrays to see they match the numbers from the labels dataframe). I only had a quick look at your kernel but you don't seem to be one-hot encoding it. Not sure how much difference it makes, as I said, just trying to learn and see myself. And limited time to experiment.
\n\nAlso, maybe use the inception_resnet_v2.preprocess_input() function on your images. It should give decent results anyway, but this should ensure at least the input has a similar distribution (I guess that is the correct term..).
", "votes": null, "replies": [ { "id": 542630, "author_name": "maxlenormand", "author_url": "", "post_date": "06/04/2019 02:28:35", "content": "Well, I saw one-hot encoding was one option out of it, but I did want to try to do it without\nI saw that it could work without doing it like that, so I wanted to try.\nI think I got onboard this competition a bit too late to have time to solve any of these issues. But thanks for the ideas, I'll have a look at those!
", "votes": null, "replies": [] } ] }, { "id": 542718, "author_name": "mpsampat", "author_url": "", "post_date": "06/04/2019 03:59:39", "content": "Hi @maxlenormand ,
\n\nI think one issue could be this step: datagen=ImageDataGenerator(rescale=1./255.,validation_split=0.2)
\n\nSince we use imageNet weights we need to apply same normalization to the images as ImageNet did. \nIt may converge without this normalization but it will take a much longer time.
\n\nfor normalization you need to find mean value from ImageNet and subtract it from training and testing. \nI will try and see if i can fork your kernel and make this change and test my theory :-)
\n\nHappy Debugging ! (which in a way is one of the most fun part of ML )
", "votes": null, "replies": [ { "id": 542719, "author_name": "mpsampat", "author_url": "", "post_date": "06/04/2019 04:00:48", "content": "also just saw averagemn's comment. i am suggesting the same as him. \"Also, maybe use the inceptionresnetv2.preprocess_input() function on your images\"
", "votes": null, "replies": [] }, { "id": 542876, "author_name": "maxlenormand", "author_url": "", "post_date": "06/04/2019 06:49:21", "content": "Interesting, I didn't know about that. I'm not sure how to apply the same normalization as Imagenet, so I'd definitely be interested in seeing that, even after the competition is over!
", "votes": null, "replies": [] } ] }, { "id": 542874, "author_name": "maxlenormand", "author_url": "", "post_date": "06/04/2019 06:47:52", "content": "Just would like to thanks @kiurtis which has helped me in the background find a fix, it indeed was the predicitons that was not working properly\nI jumped up to 0.472 now thanks to that\nThanks to everyone for the help, really appreciate it! :-)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "541763": "Hello everyone!\n\nI'm still relatively new to Kaggle, so I'm still learning a lot. I'm getting on this competition pretty late, but I've been working on it for the past 2 weeks roughly. I've been stuck since then on a super low score, and I just can't figure out what the problem is. Here is [my Kernel ](https://www.kaggle.com/maxlenormand/act-softmax-incresnetv2) which is based upon several other kernels, but mostly [the Keras-Starter Kernel](https://www.kaggle.com/ateplyuk/keras-starter) which is basically what I was going for anyways.\nIn a nutshell, I'm using an InceptionResNetV2, with a last layer with 1103 nodes with a softmax activation.\nI think something is wrong with the way my model is learning as from what I display, the validation F2 score is already higher than the training one on the first epochs, which doesn't make much sense. The training F2 score is getting better as the epoch go on, and the training loss goes down, which seems correct. However the validation score and loss just don't seem to make sense, and I can't figure out why; hence after muliple days of searching, turning to the forum. If anybody would be willing to take a look at let me know what they think, I'd really appreciate it!\n\nI have some ideas I would have wanted to try out for this competition, but if I can't even get a simple model working, then it's not even worth trying those! My timing wasn't great on this one.\nThanks for the help!", "541766": "Your early-stopping stops the model very early so it only trains for 4 epochs. It's probably best to get rid of the early stopping and train as long as you can", "541805": "Yep, my early stopping isn't working very well, I agree.\nHowever I did train it over 15 epoch, and the result was no better, the score was still around 0.001 ~ 0.004. So the problem doesn't (only) come from that. And still, it doesn't explain why my validation loss and F2 score are higher than my train.", "541824": "Have you tried increasing your learning rate? Maybe 1e-2?", "541828": "It's also weird that your validation f2 is higher than your training, but the loss is also higher.", "541840": "Are you using pretrained imagenet weights or not?\n\nI also built one Siamese Net solution which was not properly learning without pretrained imagenet weights", "541842": "I did try at 0.001 and 0.0001, but not at 0.01. I could try it just to see, even though I don't think that's the core of the problem.\nAnd yes, the fact that the validation metrics are higher is what concerns me the most, and I can't figure what's wrong with it.", "541845": "Yep, using imagenet weights. But maybe not properly.\nI'll have another look!", "541846": "If you are not using imagenet weights, that could explain a lot of problems. Because training from scratch would take a really long time. See if there is any problems with that and let us know.", "541847": "I know that, but I am using the imagenet weights :-)", "541848": "Are you generating labels for your preds properly? there could be a bug there", "541858": "I think that might be where the problem is coming from yeah\nBecause I tweaked my model, and the result looks much better already. (you can see it [in this Kernel](https://www.kaggle.com/maxlenormand/change-model-act-softmax-incresnetv2) or in the image attached)", "541868": "Indeed this looks better. BTW the kernel you linked is private.", "541882": "My bad!\nAll public now!", "541938": "Just clone a working kernel. Btw, 4 epochs are not enough for training anyway. And you don't need InceptionResNetV2 for the start, take ResNet50 and make sure your training works as expected. \n\nFYI, InceptionResNetV2 achieves 0.605 on validation for me.", "542138": "Sure I could do that, but I don't want to just clone a kernel. I want to be able to make on from scratch and understand it along the way. I've still pretty new to all of this, so I want to take it as a learning experience.\nAnd I just took InceptionResNet because that was one of the ones I used in a previous competition, so I just copied the code from there.\nAnd yeah, 4 epochs are not enough indeed, but at least they can start showing a trend. For debugging quickly I thought it might be a could way to go. Might not be though.", "542174": "I see 2 different points: \n\n1) You are not giving enough iterations to your model to learn. You probably need more than 4 epochs especially for softmax - in my experience with this dataset at least. Increase the patience of your early stopping should improve this.\n\n\n2) Even though your validation score is low, there is an important gap (> x10) between your validation score and your test score, the latter being very close to 0 anyway. It probably means that you are not submitting your results properly. I suspect that you have the same issue I went through caused by a bad order of your labels in the final dataframe you submit. \n\nIn my case, it was caused by a mismatch between the alphabetic order and the numerical order: the numerical labels where converted to string and thus ordered by alphabetic order (\"0\", \"1\", \"10\", \"100\", etc.). As a consequence, I was submitting the results of the 10th label for the 3nd etc. \n\nLook at the train\\_generator.class\\_indices to see the mapping between labels and their order and check that it's coherent. Another quick check is to look at your most frequently predicted class. It should be 813 (\"tag::men\"). Having a quick look at your submission, that does not seem to be the case here.", "542245": "I had some similar issues and started the same way on this. This is my first image classification competition, just trying to learn how to do it all. I started by copying parts of a kernel using InceptionResNetV2. Now I have been using ResNet50 to just get started, but quickly running out of time since I only started on this 2 weeks ago and my first image learning one.\n\nI had some issues initially to get the labels correct, as also mentioned by @kiurtis. I ended up one-hot encoding the target variables, and checking closely that they are actually correctly encoded (printing out specific indices in the one-hot encoded arrays to see they match the numbers from the labels dataframe). I only had a quick look at your kernel but you don't seem to be one-hot encoding it. Not sure how much difference it makes, as I said, just trying to learn and see myself. And limited time to experiment.\n\nAlso, maybe use the inception_resnet_v2.preprocess_input() function on your images. It should give decent results anyway, but this should ensure at least the input has a similar distribution (I guess that is the correct term..).", "542355": "I'm not saying copy-pasting code from kernels is good. But you are given many working examples. It's up to you to find a difference between your code and any of the working implementations. A good ML engineer must be good at coding.\n\nI see your point though. I made my solution from scratch.\n\nYou must have some serious error somewhere in your pipeline, like wrong image normalization. Another thing you could try is to take the code from any image classification tutorial and make it use competition data.\n\nProblem is, if you change a lot of things at once and then click Run, it wouldn't work. Try doing small incremental changes while keeping an eye on validation metrics.", "542428": "Regarding this: \"It's also weird that your validation f2 is higher than your training, but the loss is also higher.\"\nThis is absolutely normal when augmentations are in place. Especially with mixup.", "542630": "Well, I saw one-hot encoding was one option out of it, but I did want to try to do it without\nI saw that it could work without doing it like that, so I wanted to try.\nI think I got onboard this competition a bit too late to have time to solve any of these issues. But thanks for the ideas, I'll have a look at those!", "542636": "Totally agree, and that's what I'm doing, tweaking one thing at a time (Kaggle allowing to run 4 kernels at once does help).\nI think my issue here is a lack of understanding at some point of the entire pipeline. I do understand most of it, but you only need one error for it not to work.", "542637": "Okay, thanks! Good to know that!", "542718": "Hi @maxlenormand , \n\nI think one issue could be this step: datagen=ImageDataGenerator(rescale=1./255.,validation_split=0.2)\n\nSince we use imageNet weights we need to *apply same normalization to the images as ImageNet did*. \nIt may converge without this normalization but it will take a much longer time. \n\nfor normalization you need to find mean value from ImageNet and subtract it from training and testing. \nI will try and see if i can fork your kernel and make this change and test my theory :-) \n\nHappy Debugging ! (which in a way is one of the most fun part of ML )", "542719": "also just saw averagemn's comment. i am suggesting the same as him. \"Also, maybe use the inceptionresnetv2.preprocess_input() function on your images\"", "542874": "Just would like to thanks @kiurtis which has helped me in the background find a fix, it indeed was the predicitons that was not working properly\nI jumped up to 0.472 now thanks to that\nThanks to everyone for the help, really appreciate it! :-)", "542876": "Interesting, I didn't know about that. I'm not sure how to apply the same normalization as Imagenet, so I'd definitely be interested in seeing that, even after the competition is over!" }, "source": "meta" }